Voice AI in Production: Handling Real Accents, Real Noise, Real Failure Modes
What it takes to ship a voice AI agent on real phone lines: latency budgets, accents, code-switching, noise, interruptions and handover to humans.

Voice AI demos are impressive. A clear speaker in a quiet room asks a tidy question and gets a fluent answer. Then the system goes live on a phone line, and callers ring from traffic, speak with accents the demo never heard, switch languages mid-sentence, interrupt, go silent, and ask for things nobody planned for.
The difference between a demo and a production voice agent is almost entirely in handling those conditions.
Latency: the budget that shapes everything
In human conversation, the gap between one person finishing and the other starting is typically around 200 milliseconds, a pattern research has found across many languages. Callers are sensitive to delay. Too slow, and they talk over the agent or assume it has failed.
A voice agent's delay accumulates across four stages:
- Speech recognition: detecting that the caller has finished and transcribing what they said.
- Reasoning: the language model deciding what to say, plus any lookups.
- Speech synthesis: turning the response into audio.
- Network: getting audio to and from the phone system.
We measure each stage separately and budget for it. Techniques that help: streaming recognition and synthesis so work starts before each stage finishes, smaller models for simple turns, fast lookups, short first sentences, and hosting close to the telephony provider.
Phone audio is not studio audio
Standard telephone lines carry narrowband audio sampled at 8 kHz, compressed and often noisy. Speech models that perform well on clean, wideband recordings can lose significant accuracy on real phone calls. Always test on the channel you will actually use, with recordings from real callers.
Accents and code-switching
Benchmarks rarely reflect your callers. Indian English, Gulf Arabic-accented English, Scottish and regional UK accents, Canadian French speakers and many others are underrepresented in some training data. Many callers in India also switch between Hindi and English mid-sentence.
What works:
- Measure recognition accuracy on your own recordings before choosing a speech model.
- Add custom vocabulary for names, places, product terms and codes.
- Confirm critical details such as numbers, dates, names and addresses back to the caller.
- Design prompts that tolerate imperfect transcripts, asking for clarification rather than guessing.
Turn-taking and interruptions
Real callers interrupt. They say "sorry, what?", start answering before the question ends, or pause mid-thought. A production agent needs:
- Barge-in: detecting when the caller speaks, stopping immediately and listening.
- Sensible end-of-turn detection, so a pause to think is not treated as the end of an answer.
- Handling silence, with a gentle prompt rather than a hang-up.
- Backchannel tolerance, so "uh-huh" and "okay" do not derail the conversation.
Poor turn-taking makes an agent feel robotic even when every answer is correct.
Noise
Callers ring from cars, streets, kitchens and shop floors. Noise suppression helps, but the conversation design matters too: short, confirmable steps, and a quick route to a person when recognition confidence stays low.
Handover to humans
Design the handover first, not last. Callers should be able to ask for a person at any time, and some situations, such as repeated misunderstanding, distress or complaints, should trigger a handover automatically. The human agent should receive the transcript, detected intent and any collected details so the caller never repeats themselves.
Use cases that work today
Narrow, high-volume calls with clear outcomes:
- Appointment booking, changes and reminders.
- Order and delivery status.
- Simple account questions after verification.
- Lead qualification.
- After-hours triage and routing.
Complex complaints, sensitive conversations and anything with significant financial or medical consequences should go to people.
Consent and data
Tell callers when a call is recorded or handled by an automated system. Several US states require all parties to consent to call recording, and data protection laws such as GDPR, UK GDPR, India's DPDP Act and the UAE PDPL apply to recordings and transcripts. Set retention limits, redact sensitive details from logs, and take legal advice for your markets.
Evaluate on real calls
Build an evaluation set from anonymised real calls covering your callers' accents, lines and typical problems. Measure recognition accuracy, task completion, latency at each stage, handover rate and caller satisfaction. Every production failure becomes a new test case. For the text side of the same discipline, see AI chatbots that resolve tickets.
A latency budget example
To keep the gap after the caller stops speaking to around a second, a budget might look like this:
| Stage | Target |
|---|---|
| End-of-turn detection | A few hundred milliseconds |
| Speech recognition (streaming) | Mostly overlapped with speech |
| Reasoning and lookups | A few hundred milliseconds |
| First audio from synthesis | A couple of hundred milliseconds |
| Network and telephony | Kept low by hosting close to the provider |
The exact numbers depend on your stack and providers. The discipline is measuring each stage separately, so you know where time goes when calls feel slow.
Questions to ask a voice AI vendor
- Can you show recognition accuracy on recordings of our callers, on our phone lines?
- How do you handle interruptions and long pauses?
- What is the measured delay at each stage, not just the model's?
- How does handover to a person work, and what context is passed?
- How are recordings and transcripts stored, redacted and deleted?
Start with your recordings
Our first fixed-scope project is a feasibility check on your anonymised recordings: we measure recognition on your callers, map failure modes and estimate latency, then recommend whether and how to pilot. Two to four weeks. We build voice AI for teams across India, the United States, the United Kingdom, the UAE and Canada.
Voice AI, in numbers
Blurbee's assessment of the share of failures by cause in voice pilots we have reviewed, indicative. Opinion, not a survey.
Why teams build voice AI with Blurbee
We test on your real callers, lines and accents before promising anything, design the handover to a human first, and measure latency at every stage. Voice agents fail in the gaps; we build for the gaps.
- Fixed-scope first projectTwo to four weeks, priced as a number rather than an estimate, so you have evidence before a longer commitment.
- One delivery lead, namedYou always know who to ask. No account manager relaying questions to a team you never meet.
- Code and IP are yoursAssigned in writing, handed over with repositories, infrastructure definitions and documentation whether or not you continue with us.
- Performance budgets in CICore Web Vitals and bundle size gate the build. Slow does not ship.
- Accessibility to WCAG 2.2 AAReviewed with automated and manual passes, not claimed on a slide.
- Written overlap disciplineStanding calls inside your working window, everything else asynchronous with a decision log.