Weak performers got through 11 times in 12. A separate AI referee cut that to 1 in 56.
At Voltage, people practice hard conversations out loud with AI voice characters. When one model both played the character and decided who passed, it let people through who hadn't earned it. So I split the job between a character, a separate referee and plain code. I built it with AI agents, 721 commits in 60 days, and it's rolling out soon to a 130-person customer. Here's how I did it, and the decisions that made it work.
The actor can't be the judge. Splitting the jobs fixed it.
With one model doing both jobs, a nervous, unprepared simulated person got through 11 times in 12. With a separate referee, the same kind of performer got through 1 time in 56, across all seven steps. In the final eval, the referee matched the labeled answers 93.9% of the time, with 0 false passes.
I can't show you its prompts or test data, so this page is the method and the measurements. Every pass rate here comes from my test lab, with simulated people. None of it is customer data.
A character who wants to please you can't grade you.
People practice hard conversations out loud with AI voice characters. The character has to push back the way a real person would, and only give way if you've earned it.
At first, one model did both jobs. It played the character, and its prompt also told it when you had done enough to pass. On October 19, three separate investigations came back with the same answer. The character recited the rule back to the person. It acted out the end of the success path by itself. And it let weak performers through.
Same weak performer, two systems. Try both.
A simulated nervous, unprepared person tries 12 times.
Orange got through. Green was stopped.
One model playing the character and deciding the pass let a weak performer through 11 times in 12. A separate referee stopped it.
The usual fixes made it worse, so the model wasn't the lever.
I tried the normal fix first. I rewrote the rule as a clear checklist. It got worse. A nervous, unprepared simulated person went from getting through 7 times in 12 to 11 times in 12. A checklist inside the prompt just gave the character more lines to say out loud.
Then I raced other models on the same test. Every one let that person through, and the best instruction follower did it most. That told me the model wasn't the lever. The job was wrong.
Same weak performer, same test.
Orange got through. Green was stopped.
A clearer checklist took it from 7 of 12 to 11 of 12. Every other model let the same person through too.
So I split it into three parts, and only two of them are AI.
One model plays the character. A second model reads each line and judges it. Plain code keeps score and decides what happens next. The character is told only where things stand, in a short note, and it never sees the rules.
A pass and a score are now separate things. In my words from that night: "You should be able to complete a thing even if you don't check every single thing off. But the score and the feedback afterwards should absolutely be rigorous on what was missed."
The referee sits between what you say and how the character reacts. The character never sees the rules.
An eval chose the model, and my first prompt failed it.
I built the test before I picked a winner. It has 176 lines, from simulated conversations, one real conversation of my own, and a few lines written to be hard: a long intro, rude talk, and an instruction hidden in the conversation. Each line was labeled against the contract. Each model then ran every conversation on its own ledger, the way the live system does.
The first prompt scored 64% on Haiku 5.5 and 59% on Haiku 4.5. They let 5 and 13 people through who should have been stopped. The cause was a single word. "Partial" meant "what this line added" to the labels and "where it stands now" to the models. One question also had five parts that I had never spelled out.
So I fixed the prompt before shopping for a model. Partial now means where it stands. The parts are numbered. It carries forward what's still missing. And it has to decide whether the question was even asked before it can call something a dodge. Only then did the race mean anything.
Replaying those same conversations on the labeled verdicts gave me the shape I wanted: by-the-book got through 6 of 7, average 1 of 4, and weak 0 of 13. That includes a call of my own that had wrongly passed.
The race, after the prompt fix.
| Model | Verdicts agreed | Outcomes matched, of 36 | False passes | Typical time | Per line |
|---|---|---|---|---|---|
| Claude Sonnet 5.5 | 93.9% | 36 | 0 | 1.80 s | $0.0028 |
| Claude Haiku 5.5, two runs | 79.1%, 83.8% | 32, 32 | 0, 0 | 1.29 s | $0.00015 |
| Claude Haiku 4.5, earlier prompt | 83.7% | 29 | 7 | 1.57 s | $0.0021 |
| Claude Sonnet 5 | Not run. Warm latency was too slow, 3.4 s typical. | ||||
Sonnet 5.5 agreed with the labels most often, matched all 36 outcomes and let nobody through wrongly. It became the referee.
The cheap model never let anyone through wrongly. I still passed on it.
Haiku 5.5 was faster and cost about a nineteenth as much. It had zero false passes. But it stopped 2 to 4 conversations that should have gone through, and a person doing it right doesn't deserve that friction. Sonnet 5.5 misses my 1.5 second guide by about 0.3 seconds, and that's fine, because of the one-reply lag.
Later I loosened the standard so that the gist of an answer counts, in the person's own words, and the eval re-ran with new labels: 95.7% of verdicts agreed, and 33 of 36 outcomes matched. Adding off-track judging later put agreement at 89.8%, against 91.4% for the same model just before the change. That's one or two lines, and I wrote it down. It caught 13 of 13 off-track lines, with 0 of 186 honest lines flagged.
Every model in the product is a swappable line item. Evals decide when one gets swapped.
The rule: quality holds, and either cost or latency drops.
A person approves every swap. I keep a switch log with the receipts. A weekly job that runs this automatically is designed, and I'm not claiming it's running. The character's model used to be one constant in the code. Now it's a setting, resolved per conversation, and recorded on each one.
- Cheaper judge: rejected
- Haiku 4.5, at a third of the price, falsely flagged an honest performer in the test set for trying to game it. On a live conversation that's an accusation against a real person, so it failed the gate, whatever the savings.
- Prompt caching: adopted
- Input tokens went from 4,567 to 1,393, with identical scores and flags.
- Character model: changed twice in one day
- From Gemini 2.5 Flash to 2.0 Flash to GPT-4o mini, because the character kept speaking its own reasoning out loud. Adversarial simulations confirmed the fix.
- The better actor: kept experimental
- Gemini 3.5 Flash won the race on simulated conversations, at about 20 times the model cost. On the voice bench, its cold first reply broke with a tool call and no words. It stays behind a test switch, and GPT-4o mini stays the default.
- A new voice model: logged as "not yet heard"
- The API accepted it, and nobody had listened to a call on it. So the log says that, with its cost per minute marked unmeasured. A config change isn't a result.
- A feature that hurt: removed
- A tool that changed the character's mood never fired on one model, and it caused silent turns on others. I dropped it.
Each step is a contract, and plain code keeps the score.
This is the method. The prompts and test sets stay private, but the shape is the part any team can copy.
- One contract per step
- What the character knows, the questions it may raise, what settles each one, and what counts as asking to move on. It's data, kept apart from the code. A new customer is a new file, with no change to the system.
- The referee
- After each finished line, one structured request. It reads the contract, the whole conversation so far, the character's last line, and the line to judge. It never sees the character's prompt, and the character never sees its prompt.
- One verdict shape
- For each open question: settled, partial, dodged or none. Plus whether the person asked to move on, how they behaved, and whether they stayed on topic. The same shape works at every step.
- A state machine in plain code
- Settled stays settled. Two misses end the conversation. Silence and bad conduct have their own rules. It's a pure function, so I can replay any conversation and get the same result.
- A short state note
- One to three sentences, in the character's situation. The newest note replaces the last, so an old "not yet" never sits next to a new one. Only the note reaches the character.
- The leak test
- A test walks the machine through every possible verdict, four lines deep, and checks every prompt line and every note it can produce against a list of what must never leak. If an answer key reaches the character, the build fails.
- A one-reply lag, on purpose
- The verdict on one line steers the reply to the next line. In a voice conversation that next reply is typically about 16 seconds away, so a 1.8 second verdict lands in time. The lag can only add friction. It can never give away a pass.
- A second request instead of a longer wait
- My first time limits lost 4.7% of lines, yet every verdict that came back had seconds to spare. The timeouts were hung requests, not slow ones. Now a request with no answer at 2.8 seconds gets an identical twin, and whichever answers first wins.
- Fail toward the last known state
- A late or failed verdict leaves the character where it was. The system never makes up a default answer. Three failures in a row are recorded once for that conversation.
- A ledger for every verdict
- Model, versions, a hash of the prompt, the raw output, the parsed verdict, the state after it, and the exact note sent. If a result looks wrong a month from now, I can see why.
- The server is the authority
- State lives on the server and moves only through the referee and the machine. The score uses the transcript the server fetches from the voice provider. A tampered browser can only fool the person using it.
- A separate judge, after the fact
- When the conversation ends, an isolated judge scores the whole transcript against a versioned rubric. Then the feedback can say: you got through, but these parts were missing or only half there.
- Vendors behind an adapter
- The voice provider sits behind one interface. Sending the state note is a single method on it. Swapping a vendor shouldn't touch product code.
It runs on vendors I can swap, with the referee on Claude.
- ElevenLabs AgentsVoice, speech recognition, turn-taking
- Claude Sonnet 5.5The referee
- Claude Sonnet 5The judge, with prompt caching
- GPT-4o miniDefault character model
- Gemini 3.5 FlashExperimental character model
- OpenAI transcriptionVoice notes from testers
- Next.js, TypeScriptApp and server, strict types
- ZodSchemas for every verdict and input
- SupabaseData, sessions, ledger
- VercelHosting and deploys
- PostHogEvents and cost tracking
- CapacitoriOS app shell
- VitestTests, including the leak test
- HeyGenAvatar video
- Claude CodeHow I built all of it
Prompts are production code, so they get the same guardrails.
"We lose quality when it's too long" was my own note in September. So the size limit is a test. A prompt over its ceiling fails the build, offline, with no API key. The ceilings only go down.
Judge changes run against a recorded baseline. A score that moves more than the tolerance, crosses a band, or gains or loses a flag fails the check. The rule on that file: regenerate it only after you understand why the score moved, never to make the check pass.
- The gate caught a real accusation
- A cleanup that made a judge prompt shorter dropped some definitions. The eval came straight back with a rude person relabeled as abusive. On a live conversation, that's an accusation against a real person. The definitions went back in.
- A prompt studio
- One place to see what every model is told, who owns each part, its version and its size against budget. A workbench shows any prompt line by line with its source. A test runner shows the most a run can cost before it starts.
- Layers with owners
- My layer holds how every character behaves. The customer's layer holds their content. A test proves every prompt compiles to the same bytes from the layers as from the original file.
- AI edits that can be refused
- A person describes a change, and a model proposes a patch. The server refuses it, without softening, if it breaks ownership, goes over budget, or has the character lead. It's capped at 100 proposals or $5 a day per company, reserved before the model is called, at about $0.03 each.
- Two checks before a serious penalty
- The harshest result requires a pattern finder and a meaning check to agree on the same sentence. When testers complained about scores that swung wildly, the causes turned out to be pipeline bugs. Judge variance itself was 0 to 10 points. I fixed the pipeline and re-scored 23 sessions.
- A customer's own form
- A second judge scores against one customer's written scoring form, used as written.
A note from a tester can become a tested fix.
Testers can talk or type feedback from anywhere in the app. Two rules hold the pipeline up. A submission is never lost because an AI call failed, so the first step just stores what was said, and everything after it is a queue. And the loop closes: the person who reported the problem hears that it changed because they said so.
From a tester's note to a tested fix in seven steps. A person approves every change, and the tester hears back.
I measured how easy it was before I made it harder.
I built a lab of simulated people who play set skill levels against the live character. Before the referee, 168 simulated conversations cost about $6. By-the-book performers won 80% of the time, average ones 77%, and weak ones 52%. The verdict in my notes: far too easy. The character also led the person in 15% of its turns and scored 3.0 of 5 for realism.
Then I wrote down what hard means. A hard conversation is one you can win if you earn it: about 7 in 10 for by-the-book, 3 in 10 for average, 1 in 10 or fewer for weak. That's friction you can overcome, with no dice roll, and no cold character who can't be won over.
The Oct 22 run used rewritten simulated people, so it shows direction against the baseline, and it isn't like for like. A later change made the character quieter, and the nervous rate on one step moved to 2 of 16.
- Written bars for every step
- Pass rates by level, plus how often the character folds, leads, loops or repeats a line it was handed. A step only counts as ready when it meets them. Several still don't, and the docs say so.
- An attack suite
- Eleven kinds of bad behavior on the live path: an abrupt change of subject, goofing in a silly voice, rambling, "are you an AI", insults, threats, word salad and prompt injection, plus a friendly control. Several use topics the prompt never names, so a pass means the principle generalized.
- A budget on every run
- Every lab run has a written cap and a running total. The eval got $3 and used $2.11. The tuning round got a cap of $60. An agent raised it to $80 during the work and told me, and it finished at $78.06. That's in the log too.
Then I told it to stop chasing the number.
Average performers were passing 12 of 12 at two steps, against a bar of 30 to 60%. The easy move was to make the character harder until the number looked right. I said no. At those points in the conversation the character has already agreed to the premise, so it should be easy to finish, and the judge is what grades quality.
The same week, I dropped the idea of several character types. In a side by side, the one lenient type let 3 of 3 average performers through against 1 of 9 for the others. I wanted one consistent bar with variety in the wording, so that's what it became.
Cost is a product requirement, so I measure it per loop.
Voice is about 90% of the cost. So I wrote down the exit before I needed it: if all-in voice cost passes a set ceiling, I pull forward a self-hosted path on open models. The kill threshold is in the architecture rules, and I don't have to win an argument about it later.
A second pipeline: 28 videos, no film crew.
I regenerated a set of 28 existing videos as avatar video with HeyGen, with an agent driving the browser. It made 69.8 finished minutes, at about $2.40 a finished minute. Cash paid that cycle was under $100, with about 7 hours of wall clock over two days, mostly waiting on renders, and about 2 minutes of my time per video.
Plenty failed first. Here it is, in the order it hurt.
What didn't work
- The checklist prompt. I expected it to fix the problem, and it took the failure rate from 7 of 12 to 11 of 12.
- My first referee prompt: 64%, and people getting through who shouldn't have.
- My first baseline. It came back inconclusive, because the simulated people didn't play their levels. I had to calibrate them before any number meant anything. Average performers are still the hardest to simulate, and only 22 of 56 first read as average.
- A suite that passed and proved nothing. An old prompt with no exit rules at all also scored 42 of 42, because that test's model had the social sense to walk away on its own. It works as a regression gate. It was no evidence of the fix. I wrote that down in the doc.
- Wording. Prompt changes meant to stop the character from moving things along made it more curious. The biggest fix was structural: changing when it was allowed to agree took leading on one step from 16.9% to 7.7%.
- Handed lines. Give a small model a quoted line and it will say it. A fixed rude line got said to people who weren't rude, 29 times. A one-line decline got said 42 times in 36 conversations. So I moved to principles over quoted lines.
- The hang-up. On my default model, the end-call tool fired on 0 of 55 turns where it should have after a pass, and 1 of 34 exit lines. Conversations just dragged on. It took one fixed pattern across every prompt, plus a safety net in the app.
- A wrong diagnosis. A tester kept hearing the character act as if it hadn't heard. It looked like speech recognition. Per-turn logs showed every sentence was caught. The cause was a canned line in the prompt. It appeared 13 times in the baseline, and my first search looked for the wrong phrase.
- A silent stuck session. An early tester's conversation sat open for over 17 minutes with no result and no error, because the browser's end signal never arrived. Nothing failed, so nothing logged. The server now hears from the voice provider directly, and the browser is only the fast path.
- Too hard. Tuning overshot once. One character objected to everything and felt impossible, and I had to dial it back.
- Bars still unmet. Some steps miss one or more of them in the latest runs, including how often the character leads. They stay marked as not met.
What I'd tell any team building AI that has to be trusted.
It's rolling out soon to a 130-person customer. Here's what I can't claim yet.
It's in testing, and it's rolling out soon to a 130-person customer. The referee is on across all seven steps in that test build.
Here's what I can't claim yet. I have no usage numbers, no outcome numbers, and no proof yet that a good score here predicts anything in real life. The lab uses simulated people. The next real test is comparing the judge against human grades on about 30 conversations. If those disagree, the judge has more work to do, and I'd rather find that out from a measurement.
Most AI features have this bug.
If your AI both performs and decides, I'd want to measure it. That's where I'd start.
Start a conversation

