Your voice agent needs a policy for when to stop talking

This is a part 1 of the "Voice has no undo" series we launched last week.

Lessons from enterprise voice deployments

A resident calls a rental company’s line late at night. There is a smell of gas in the apartment. The voice agent recognizes the emergency and starts the one sentence it exists to say: “This is a critical, life-threatening emergency. Please hang up and call 911 immediately.”

The resident is frightened and talks over it. The agent stops to listen, as voice agents are designed to do. The sentence that mattered is cut in half.

We’ve seen the same problem in other forms. A hotel guest on speakerphone, with a fan running, loses the welcome message to the noise, and the agent answers a question nobody asked. A bank customer says “okay” after a hold message, and the agent treats it as a new request. A pharmacy member pauses halfway through a date of birth, and the agent takes the first half as the whole answer.

Each of these is a decision about whose turn it is to talk, and in each case the agent made it on its own, in a fraction of a second. We see versions of these four problems across our enterprise deployments, and our view is that your business should decide when the agent keeps talking and when it gives way, the same way you decide what it says. That is what we mean by a turn-taking policy.

On the phone, the agent has to guess whose turn it is

In a chat window, the customer presses send, so the turn is obvious. On the phone, the agent has to work out from sound alone, as it happens, when the caller wants to speak.

The agent makes that decision in a fraction of a second, before anyone knows what the sound was. It could be the caller, a fan or a television. If it was the caller, it could be an interruption, an “okay” or half an answer.

Three common fixes, and why each one fails

The usual fixes swap one problem for another, because each one applies a single answer to the whole call.

Turn interruptions off for the whole call. he emergency message would play to the end. So would everything else, and every other exchange feels robotic.

Make the agent wait longer before replying. The pharmacy member would have time to finish the date of birth. Every other turn gets slower too, including a simple yes or no.

Let the AI that holds the conversation decide each time. Every decision adds a round-trip, so replies slow down. The gas-leak message would also depend on a judgment that is probably right, which isn’t good enough for a safety message.

The problems in the opening each needed a different answer at a different moment. The emergency message had to finish. The hotel guest’s fan should have been ignored. The bank customer’s “okay” needed to be heard as listening, not as a request. The date of birth needed time. None of the three fixes does anything about the fan. Getting all four right takes rules that change with the moment, decided in advance by the people responsible for the call.

A turn-taking policy answers four questions on every call

A turn-taking policy is a short set of rules that settle when the agent keeps talking and when it gives the caller the floor. You already approve what your agent says. These rules decide what it must finish saying, and what counts as the caller’s turn.

Your team sets the rules. Small models, each built for one narrow job, answer questions inside those rules: was that a person, was that a request, is the answer finished? None of them can override a protected message, and the AI that holds the conversation decides none of it.

Rules beat settings tuned by feel because you can read them. Before a call, you know what the agent will do with a protected message. After a call, you can see which rule applied, and change the rule instead of retraining the AI.

We break turn-taking into four questions. The table shows who answers each one in our approach, and what each rule prevents.

1. Must the caller hear this in full? Your team decides.

Your team can mark any message as one the caller can’t interrupt: an emergency instruction, a required disclosure, the opening line of an outbound call. While it plays, the agent keeps talking and stops listening for interruptions. Normal back-and-forth resumes once it finishes.

The rental company’s emergency message now plays to the end. It was tested on real calls, with background music. New voice channels protect the welcome message by default, because the message that lists a caller’s options is the one noise cuts off most.

2. Was that the caller, or the room? Only evidence counts.

Typical voice AI models can mistake background noise, such as a fan, a television or traffic, for the caller speaking. When that happens, the agent creates a false turn: it stops mid-sentence, or answers something nobody said, as it did for the hotel guest on speakerphone.

The problem is common. In one study of a large deployed phone system, 89% of the interruptions it detected weren’t real: they were listening noises, background sound, the system’s own voice echoing back, or a turn taken too early (Alibaba, KDD 2022).

The agent stops for an interruption only when there is evidence a person spoke. That evidence comes from a dedicated model in the Kore.ai voice platform that separates a human voice from fans, traffic and other background sounds before speech recognition hears them. On the noisiest calls we tested, it cut speech-recognition errors by 37%.

3. Was that a request, or just “okay”? A dedicated model decides.

“Okay”, “sure”, “haan” and “theek hai” after a hold message mean “I’m listening”, not “I want something”. Every language has its own versions, and they vary by region and speaker, so no word list keeps up.

We trained a small AI model for this one job. It reads the agent’s sentence and what the caller has said over it, and asks one question: would the caller be better served if the agent stopped now? An answer, a correction, an objection, “hold on” or a request for a person stops the agent. “Hmm”, “okay”, “go on” or a thank-you while the agent is still explaining does not. It recognizes these listening words in more than 200 languages, and it is separate from the AI that holds the conversation.

The usual approach is a word list: the agent stops for any word not on it. In our tests on generated English and Hindi calls, when callers said nothing that needed a response, a typical list still stopped the agent 94.5% of the time while it was explaining something. The model stopped it 2.9% of the time.

When the model isn’t sure, the agent finishes its sentence, and nothing is lost: every word the caller said is kept and passed on once the agent is done. Stopping only silences the agent; whether the caller has finished speaking is rule 4’s job. The model never overrides a protected message, and every decision it makes is logged, so your team can see why the agent did or didn’t respond.

4. Has the caller finished? It depends on the question.

A yes or no needs only a short pause before the agent replies. A date of birth read out in chunks needs a longer one. A fixed timer can’t tell a thinking pause from a finished answer, so we built a multi-model pipeline that judges both sound and meaning.

The pipeline fuses voice-activity signals from the audio with a turn-detection model. The model reads the agent’s last question alongside the caller’s words as they are transcribed, and decides whether the answer is complete for that question. It counts digits against what the question asked for, so half a card number reads as unfinished. A date of birth with only the day and month isn’t finished; a “yes” is. The model checks the words while the caller is still speaking, and each check takes a few hundredths of a second, so it doesn’t add a noticeable wait before the agent replies.

As a backstop, a time limit still applies, set by the answer the agent expects: 0.65 seconds of silence for a confirmation, up to 1.6 seconds for an explanation.

If an answer still arrives in two parts, the agent keeps both and puts them in order.

Two early fixes that failed on real calls

The first mistake was in protected messages. Our first version passed its tests, then failed on a real call. The opening words slipped out before the system knew the message was protected. Now nothing plays until that is settled, and if the audio stream fails, a backup copy of the message plays in full.

The second mistake was about evidence. On another deployment, the provider’s voice AI, which produced the agent’s speech, sometimes flagged an interruption it had caused itself. So we started ignoring interruptions that arrived without the caller’s words attached. Real callers’ interruptions arrived the same way, with the words more than a second behind, so we were ignoring real people. We reversed the change.

An AI provider’s “interrupted” signal doesn’t tell you who interrupted. Your system needs its own rules for that.

Callers notice turn-taking only when it goes wrong

Callers rarely notice turn-taking when it works. They notice when the agent talks over them, answers something they didn’t say, or cuts off the one sentence that mattered. Each of those moments costs trust.

This matters more as voice agents take on real work. An agent that yields to every sound is polite and unsafe. An agent that never yields is safe and unbearable. A turn-taking policy gives you an agent that is courteous and dependable, with a record of when it yielded, when it held the floor, and why.

On the rental company’s line, the emergency sentence now plays to the end, and the rest of the call still runs like a conversation.

‍

Want to build trusted voice AI agent?