Voice AI turn-taking: a policy decision, not a model decision

Lessons from four enterprise voice deployments

A resident calls a rental company's line late at night. There is a smell of gas in the apartment. The agent recognizes the emergency and begins the one sentence it exists to say: "This is a critical, life-threatening emergency. Please hang up and call 911 immediately."

The resident is frightened and talks over it. The agent does what voice agents are built to do: it stops and listens. The sentence that mattered is cut in half.

We have seen the same failure in other forms. A hotel guest on speakerphone, with a fan running, loses the welcome message to noise, and the agent answers a question nobody asked. A bank customer says "okay" after a hold message, and the agent treats it as a new request. A pharmacy member reads out a date of birth with a pause in the middle, and the agent takes the first half as the whole answer.

Four deployments, four industries, one question: who decides when the caller has the floor?

Why turn-taking is hard on the phone

In a chat window, turns are explicit: the customer presses send. On a phone line, every turn has to be inferred from audio, in real time, before anyone knows what the words mean.

The decision to stop talking is acoustic, and the voice gateway makes it in a fraction of a second. Whether the sound was a person, a fan or a television only becomes clear later. So does whether a person's words were an interruption, an acknowledgement or half an answer. By then the agent has already stopped, or already committed the fragment.

The obvious fixes trade one failure for another. Switching interruption off for the whole call makes every other exchange feel robotic. A longer silence timer adds delay to every turn. Asking the model to decide adds a round-trip, and puts a probabilistic judgment on a safety line.

Four decisions, and who should own them

We stopped treating turn-taking as one setting and split it into four decisions, each with a clear owner.

What must be heard is the author's decision.

An agent author can mark a single response, a template or a flow step as one the caller cannot interrupt:

VOICE:
  BARGE_IN: disallow

The most specific setting wins, and a misspelled value fails the build. When a protected message plays, the runtime keeps streaming speech and pauses the gateway's listening for that message only. It waits for confirmation that playback finished, then restores normal interruption. The rental company's emergency line now plays to the end; it was tested on real calls, with background music. New voice channels protect the welcome message by default, because the sentence that lists a caller's options is the one noise cuts off most.

What counts as an interruption requires evidence.

A barge-in counts only when there is evidence the caller actually spoke. Noise isolation at the gateway filters fans and traffic before they reach speech recognition. On the noisiest audio we tested, it improved word error rate by 37%.

What counts as a request is a classification, not a guess.

"Okay", "sure", "haan" and "theek hai" after a hold message are backchannels: the caller is listening, not asking. Every language has its own, and they vary by region, register and speaker, so no word list can keep up. We fine-tuned a small language model for this one job, and it recognizes backchannels in more than 200 languages. Every decision is recorded in the trace, so a reviewer can see why the agent did or didn't respond.

When the caller has finished depends on what was asked.

A yes-or-no confirmation needs a short silence window; a date of birth read in chunks needs a longer one. End-of-speech timing is now set by the expected answer, from 650 ms for a confirmation to 1,600 ms for an explanation. The runtime also separates a transport retry, which it runs once, from a second real utterance, which it keeps and puts in order.

What we got wrong first

Two of our first fixes failed, and both taught us something.

Our first per-sentence protection passed its tests, then failed on a real call. The opening words still escaped, because the first chunk of streamed audio left before the runtime knew which policy applied. Now the first chunk waits until the policy is settled. If streaming fails, a protected non-streamed message takes over, so the sentence is never left half-said.

On a speech-to-speech deployment, we tried ignoring interruptions that arrived without a transcript, since the model sometimes reports interrupting itself. Real barge-ins turned out to arrive the same way, with the transcript more than a second behind. That fix silenced genuine callers, and we walked it back. A provider's "interrupted" signal doesn't tell you who interrupted, and neither does a missing transcript.

The principle‍

The model generates language. The runtime owns the floor: when the agent speaks, what it must finish, and what counts as the caller's turn. Authors decide what must be heard, and evidence decides what counts as speech. The model decides neither.

“Before turn-taking policy, we relied on tuning sensitivity settings and hoped for better results. Now turn-taking is a policy we can read. When a caller says the agent talked over them, we can see exactly what it heard, why it kept the floor, and fix the rule, not the model,” says the head of the contact center of a retail bank. 

This matters more as agents take on real work. An agent that yields to every sound is polite and unsafe. An agent that never yields is safe and unbearable. Treating turn-taking as policy gives you an agent that is courteous and dependable, and can show which it was being at any moment.