Enterprise voice AI: the model speaks, the runtime decides

‍Six problems, one architecture: what enterprise voice deployments taught us

Text can be edited. Voice can't be taken back.

That one difference has shaped more of our engineering over the past year than any model release. We have put AI agents on the phone lines of a Canadian rental company, banks in India and the Middle East, a US health insurer, a specialty pharmacy, a global hotel brand, a European tire chain and a Dubai-based conglomerate. The problems we met there fill the six essays in this series. Together they make one argument, and this essay states it up front.

Six problems, one question

• A frightened resident talked over a 911 instruction, and the agent stopped to listen.
• A pharmacy member heard the agent read its own routing error aloud.
• A tire booking ran three times, and the caller was given a part number as an order number.
• A caller waited fourteen seconds while six model calls, and a summary nobody was counting, ran one after another.
• A single Hinglish (Hindi + English) sentence became three turns, and the agent answered twice at once.
• A greeting cached as an empty file played silence to caller after caller.

None of these was a bad model. In every case the model had been left to decide something it should never have been asked to decide: when to stop talking, what the caller may hear, whether a booking happened, how long a turn may take, which language the call is in, and whether a sound was actually heard.

Two owners for every turn

Language models are extraordinary at conversation. They understand what people mean, find the right words and work out which tool might help. They are not built to be a system of record, a clock or a gatekeeper. On a phone line those jobs happen in real time, and mistakes can't be undone.

So we divided the work.

The model owns the conversation. The runtime owns everything that can't be undone: the floor, the output, the commit, the clock, the language and the delivery. The model proposes; the runtime decides.

Why prompts can't do this

The tempting alternative is to write the rules into the model's instructions: always confirm before a payment, never read out internal errors, always finish the safety message. Every team that has shipped a voice agent has written sentences like these. They work in the demo.

They fail in production for an arithmetical reason. When rules live in prompts and handler code, every rule has to be right at every point where it applies, in every journey, maintained by every team, across every model upgrade. Correctness becomes multiplication: rules times call sites. That is why these failures never appear in testing. They appear on the one path nobody wired, on the turn that starts with "actually".

When rules live in the runtime, correctness is addition. Each rule is declared once and enforced wherever it applies. For this reason, Artemis agents are written in Agent Blueprint Language and compiled. A rule becomes a declaration the runtime enforces and a reviewer can read. And because every trace event carries the identifiers of the agent's definition, a failure points to the line that caused it.

There is a strategic reason too. Models will keep improving, and keep changing. When the rules live in the runtime, a new model changes what an agent says, not what it is allowed to do. Enterprises can move to a better model sooner, because the rules don't move with it.

What it costs

This architecture isn't free, and the essays that follow are honest about that.

Every check has a price in latency, and on a phone line latency is audible. So we hold output only after the first suspicious signal, and we remove work before we try to speed it up. The runtime enforces what is declared, so authors still have to classify their tools; the compiler warns them, but it can't decide for them. And many of our first fixes were local, a filter here and a language fix there. Some created new problems: the triple booking was caused by our own fix for silence. Every time, the lasting fix was structural.

I would rather publish that than pretend we got it right the first time. The failures are where the architecture came from.

Where this goes

Agents are moving from answering questions to taking action: refunds, refills, bookings, payments and transfers. Each step raises the cost of a wrong decision, and the number of decisions in a single call. Voice is where this shows first, because voice has no undo. The same split applies on every channel.

When it holds, the results follow. A state Medicaid program answers routine inquiries by natural speech and resolves 89% of them, across about 250,000 calls a year. A US retail pharmacy chain handles more than a million pharmacy calls a day, across more than 9,000 stores. Neither deployment appears in this series, and their numbers are their own, not the result of any fix described here. They show what becomes possible when customers can trust what an agent says, and businesses can trust what it does.

Let the model speak. Let the runtime decide.

Coming in this series

1. Voice AI turn-taking: a policy decision, not a model decision
2. Multi-agent output governance: the caller should hear one agent
3. Transaction safety for voice agents: the model proposes, the runtime commits
4. Voice AI latency: measure it at the caller's ear
5. Multilingual voice AI: the conversation owns the language
6. Voice agent observability: delivered is not heard