Voice AI in Production: Handling Real Accents, Real Noise, Real Failure Modes

What it takes to ship a voice AI agent on real phone lines: latency budgets, accents, code-switching, noise, interruptions and handover to humans.

Voice AI in Production: Handling Real Accents, Real Noise, Real Failure Modes — Blurbee Solutions

Voice AI demos are impressive. A clear speaker in a quiet room asks a tidy question and gets a fluent answer. Then the system goes live on a phone line, and callers ring from traffic, speak with accents the demo never heard, switch languages mid-sentence, interrupt, go silent, and ask for things nobody planned for.

The difference between a demo and a production voice agent is almost entirely in handling those conditions.

Latency: the budget that shapes everything

In human conversation, the gap between one person finishing and the other starting is typically around 200 milliseconds, a pattern research has found across many languages. Callers are sensitive to delay. Too slow, and they talk over the agent or assume it has failed.

A voice agent's delay accumulates across four stages:

  1. Speech recognition: detecting that the caller has finished and transcribing what they said.
  2. Reasoning: the language model deciding what to say, plus any lookups.
  3. Speech synthesis: turning the response into audio.
  4. Network: getting audio to and from the phone system.

We measure each stage separately and budget for it. Techniques that help: streaming recognition and synthesis so work starts before each stage finishes, smaller models for simple turns, fast lookups, short first sentences, and hosting close to the telephony provider.

Phone audio is not studio audio

Standard telephone lines carry narrowband audio sampled at 8 kHz, compressed and often noisy. Speech models that perform well on clean, wideband recordings can lose significant accuracy on real phone calls. Always test on the channel you will actually use, with recordings from real callers.

Accents and code-switching

Benchmarks rarely reflect your callers. Indian English, Gulf Arabic-accented English, Scottish and regional UK accents, Canadian French speakers and many others are underrepresented in some training data. Many callers in India also switch between Hindi and English mid-sentence.

What works:

  • Measure recognition accuracy on your own recordings before choosing a speech model.
  • Add custom vocabulary for names, places, product terms and codes.
  • Confirm critical details such as numbers, dates, names and addresses back to the caller.
  • Design prompts that tolerate imperfect transcripts, asking for clarification rather than guessing.

Turn-taking and interruptions

Real callers interrupt. They say "sorry, what?", start answering before the question ends, or pause mid-thought. A production agent needs:

  • Barge-in: detecting when the caller speaks, stopping immediately and listening.
  • Sensible end-of-turn detection, so a pause to think is not treated as the end of an answer.
  • Handling silence, with a gentle prompt rather than a hang-up.
  • Backchannel tolerance, so "uh-huh" and "okay" do not derail the conversation.

Poor turn-taking makes an agent feel robotic even when every answer is correct.

Noise

Callers ring from cars, streets, kitchens and shop floors. Noise suppression helps, but the conversation design matters too: short, confirmable steps, and a quick route to a person when recognition confidence stays low.

Handover to humans

Design the handover first, not last. Callers should be able to ask for a person at any time, and some situations, such as repeated misunderstanding, distress or complaints, should trigger a handover automatically. The human agent should receive the transcript, detected intent and any collected details so the caller never repeats themselves.

Use cases that work today

Narrow, high-volume calls with clear outcomes:

  • Appointment booking, changes and reminders.
  • Order and delivery status.
  • Simple account questions after verification.
  • Lead qualification.
  • After-hours triage and routing.

Complex complaints, sensitive conversations and anything with significant financial or medical consequences should go to people.

Tell callers when a call is recorded or handled by an automated system. Several US states require all parties to consent to call recording, and data protection laws such as GDPR, UK GDPR, India's DPDP Act and the UAE PDPL apply to recordings and transcripts. Set retention limits, redact sensitive details from logs, and take legal advice for your markets.

Evaluate on real calls

Build an evaluation set from anonymised real calls covering your callers' accents, lines and typical problems. Measure recognition accuracy, task completion, latency at each stage, handover rate and caller satisfaction. Every production failure becomes a new test case. For the text side of the same discipline, see AI chatbots that resolve tickets.

A latency budget example

To keep the gap after the caller stops speaking to around a second, a budget might look like this:

Stage Target
End-of-turn detection A few hundred milliseconds
Speech recognition (streaming) Mostly overlapped with speech
Reasoning and lookups A few hundred milliseconds
First audio from synthesis A couple of hundred milliseconds
Network and telephony Kept low by hosting close to the provider

The exact numbers depend on your stack and providers. The discipline is measuring each stage separately, so you know where time goes when calls feel slow.

Questions to ask a voice AI vendor

  • Can you show recognition accuracy on recordings of our callers, on our phone lines?
  • How do you handle interruptions and long pauses?
  • What is the measured delay at each stage, not just the model's?
  • How does handover to a person work, and what context is passed?
  • How are recordings and transcripts stored, redacted and deleted?

Start with your recordings

Our first fixed-scope project is a feasibility check on your anonymised recordings: we measure recognition on your callers, map failure modes and estimate latency, then recommend whether and how to pilot. Two to four weeks. We build voice AI for teams across India, the United States, the United Kingdom, the UAE and Canada.

Voice AI, in numbers

~200 msTypical gap between speaking turns in human conversation across languagesStivers et al., PNAS, 2009
8 kHzSampling rate of standard narrowband telephone audio, far less detail than a studio recordingITU-T G.711
4 stagesWhere latency accumulates: speech recognition, reasoning, speech synthesis and the networkBlurbee delivery practice
1 stepHow far away a human handover should always be in a voice agentBlurbee delivery standard
Where voice agents fail in production, in our experience

Blurbee's assessment of the share of failures by cause in voice pilots we have reviewed, indicative. Opinion, not a survey.

  • Recognition errors (accent, noise, line) ~35%
  • Turn-taking and interruptions ~20%
  • Latency too high ~15%
  • Unhandled intents ~15%
  • Integration and data errors ~15%
Share of failures

Why teams build voice AI with Blurbee

We test on your real callers, lines and accents before promising anything, design the handover to a human first, and measure latency at every stage. Voice agents fail in the gaps; we build for the gaps.

  • Fixed-scope first projectTwo to four weeks, priced as a number rather than an estimate, so you have evidence before a longer commitment.
  • One delivery lead, namedYou always know who to ask. No account manager relaying questions to a team you never meet.
  • Code and IP are yoursAssigned in writing, handed over with repositories, infrastructure definitions and documentation whether or not you continue with us.
  • Performance budgets in CICore Web Vitals and bundle size gate the build. Slow does not ship.
  • Accessibility to WCAG 2.2 AAReviewed with automated and manual passes, not claimed on a slide.
  • Written overlap disciplineStanding calls inside your working window, everything else asynchronous with a decision log.

Frequently asked questions

Frequently asked questions

Why do voice AI agents fail on real calls?

How fast does a voice agent need to respond?

How do you handle accents in speech recognition?
Can voice AI handle Hindi-English code-switching?
What is barge-in?
Why is phone audio harder than app audio?
How should a voice agent hand over to a human?
Do we need consent to record or process calls?
What use cases work well for voice AI today?
How long does a voice AI pilot take?

Start with a fixed-scope first project

Two to four weeks, priced as a number. Code, accounts and IP stay in your name.