David MonsonTalk to me

Weak performers got through 11 times in 12. A separate AI referee cut that to 1 in 56.

At Voltage, people practice hard conversations out loud with AI voice characters. When one model both played the character and decided who passed, it let people through who hadn't earned it. So I split the job between a character, a separate referee and plain code. I built it with AI agents, 721 commits in 60 days, and it's rolling out soon to a 130-person customer. Here's how I did it, and the decisions that made it work.

Voltage, a voice-AI startup. Founder. In testing, rolling out soon to a 130-person customer.

The actor can't be the judge. Splitting the jobs fixed it.

With one model doing both jobs, a nervous, unprepared simulated person got through 11 times in 12. With a separate referee, the same kind of performer got through 1 time in 56, across all seven steps. In the final eval, the referee matched the labeled answers 93.9% of the time, with 0 false passes.

I can't show you its prompts or test data, so this page is the method and the measurements. Every pass rate here comes from my test lab, with simulated people. None of it is customer data.

11 of 12Times a weak performer got through when one model did both jobs
1 of 56Same kind of performer, across all seven steps, with a separate referee
93.9%Referee verdicts that matched the labeled answers
0False passes in the final eval

A character who wants to please you can't grade you.

People practice hard conversations out loud with AI voice characters. The character has to push back the way a real person would, and only give way if you've earned it.

At first, one model did both jobs. It played the character, and its prompt also told it when you had done enough to pass. On October 19, three separate investigations came back with the same answer. The character recited the rule back to the person. It acted out the end of the success path by itself. And it let weak performers through.

Same weak performer, two systems. Try both.

A simulated nervous, unprepared person tries 12 times.

Orange got through. Green was stopped.

One model playing the character and deciding the pass let a weak performer through 11 times in 12. A separate referee stopped it.

The usual fixes made it worse, so the model wasn't the lever.

I tried the normal fix first. I rewrote the rule as a clear checklist. It got worse. A nervous, unprepared simulated person went from getting through 7 times in 12 to 11 times in 12. A checklist inside the prompt just gave the character more lines to say out loud.

Then I raced other models on the same test. Every one let that person through, and the best instruction follower did it most. That told me the model wasn't the lever. The job was wrong.

Same weak performer, same test.

Rule in the prompt7 of 12
Rule rewritten as a checklist11 of 12
GPT-4o mini6 of 8
Gemini 3.5 Flash4 of 8
Claude Haiku 5.54 of 8
GPT-4.17 of 8

Orange got through. Green was stopped.

A clearer checklist took it from 7 of 12 to 11 of 12. Every other model let the same person through too.

So I split it into three parts, and only two of them are AI.

One model plays the character. A second model reads each line and judges it. Plain code keeps score and decides what happens next. The character is told only where things stand, in a short note, and it never sees the rules.

A pass and a score are now separate things. In my words from that night: "You should be able to complete a thing even if you don't check every single thing off. But the score and the feedback afterwards should absolutely be rigorous on what was missed."

You speakLive voice, transcribed.
The referee judgesA separate model checks each line against this step's contract.
State movesPlain code decides what happens next. No model involved.
The character reactsIt acts on the result and never knows the rules.

The referee sits between what you say and how the character reacts. The character never sees the rules.

An eval chose the model, and my first prompt failed it.

I built the test before I picked a winner. It has 176 lines, from simulated conversations, one real conversation of my own, and a few lines written to be hard: a long intro, rude talk, and an instruction hidden in the conversation. Each line was labeled against the contract. Each model then ran every conversation on its own ledger, the way the live system does.

The first prompt scored 64% on Haiku 5.5 and 59% on Haiku 4.5. They let 5 and 13 people through who should have been stopped. The cause was a single word. "Partial" meant "what this line added" to the labels and "where it stands now" to the models. One question also had five parts that I had never spelled out.

So I fixed the prompt before shopping for a model. Partial now means where it stands. The parts are numbered. It carries forward what's still missing. And it has to decide whether the question was even asked before it can call something a dodge. Only then did the race mean anything.

Replaying those same conversations on the labeled verdicts gave me the shape I wanted: by-the-book got through 6 of 7, average 1 of 4, and weak 0 of 13. That includes a call of my own that had wrongly passed.

The race, after the prompt fix.

ModelVerdicts agreedOutcomes matched, of 36False passesTypical timePer line
Claude Sonnet 5.593.9%3601.80 s$0.0028
Claude Haiku 5.5, two runs79.1%, 83.8%32, 320, 01.29 s$0.00015
Claude Haiku 4.5, earlier prompt83.7%2971.57 s$0.0021
Claude Sonnet 5Not run. Warm latency was too slow, 3.4 s typical.

Sonnet 5.5 agreed with the labels most often, matched all 36 outcomes and let nobody through wrongly. It became the referee.

176Lines labeled one by one for that eval
1.8 sTypical time to judge a line
$0.0028Cost to judge one line
$2.11Total spend on the eval that picked the model

The cheap model never let anyone through wrongly. I still passed on it.

Haiku 5.5 was faster and cost about a nineteenth as much. It had zero false passes. But it stopped 2 to 4 conversations that should have gone through, and a person doing it right doesn't deserve that friction. Sonnet 5.5 misses my 1.5 second guide by about 0.3 seconds, and that's fine, because of the one-reply lag.

Later I loosened the standard so that the gist of an answer counts, in the person's own words, and the eval re-ran with new labels: 95.7% of verdicts agreed, and 33 of 36 outcomes matched. Adding off-track judging later put agreement at 89.8%, against 91.4% for the same model just before the change. That's one or two lines, and I wrote it down. It caught 13 of 13 off-track lines, with 0 of 186 honest lines flagged.

Every model in the product is a swappable line item. Evals decide when one gets swapped.

The rule: quality holds, and either cost or latency drops.

A person approves every swap. I keep a switch log with the receipts. A weekly job that runs this automatically is designed, and I'm not claiming it's running. The character's model used to be one constant in the code. Now it's a setting, resolved per conversation, and recorded on each one.

Cheaper judge: rejected
Haiku 4.5, at a third of the price, falsely flagged an honest performer in the test set for trying to game it. On a live conversation that's an accusation against a real person, so it failed the gate, whatever the savings.
Prompt caching: adopted
Input tokens went from 4,567 to 1,393, with identical scores and flags.
Character model: changed twice in one day
From Gemini 2.5 Flash to 2.0 Flash to GPT-4o mini, because the character kept speaking its own reasoning out loud. Adversarial simulations confirmed the fix.
The better actor: kept experimental
Gemini 3.5 Flash won the race on simulated conversations, at about 20 times the model cost. On the voice bench, its cold first reply broke with a tool call and no words. It stays behind a test switch, and GPT-4o mini stays the default.
A new voice model: logged as "not yet heard"
The API accepted it, and nobody had listened to a call on it. So the log says that, with its cost per minute marked unmeasured. A config change isn't a result.
A feature that hurt: removed
A tool that changed the character's mood never fired on one model, and it caused silent turns on others. I dropped it.

Each step is a contract, and plain code keeps the score.

This is the method. The prompts and test sets stay private, but the shape is the part any team can copy.

One contract per step
What the character knows, the questions it may raise, what settles each one, and what counts as asking to move on. It's data, kept apart from the code. A new customer is a new file, with no change to the system.
The referee
After each finished line, one structured request. It reads the contract, the whole conversation so far, the character's last line, and the line to judge. It never sees the character's prompt, and the character never sees its prompt.
One verdict shape
For each open question: settled, partial, dodged or none. Plus whether the person asked to move on, how they behaved, and whether they stayed on topic. The same shape works at every step.
A state machine in plain code
Settled stays settled. Two misses end the conversation. Silence and bad conduct have their own rules. It's a pure function, so I can replay any conversation and get the same result.
A short state note
One to three sentences, in the character's situation. The newest note replaces the last, so an old "not yet" never sits next to a new one. Only the note reaches the character.
The leak test
A test walks the machine through every possible verdict, four lines deep, and checks every prompt line and every note it can produce against a list of what must never leak. If an answer key reaches the character, the build fails.
A one-reply lag, on purpose
The verdict on one line steers the reply to the next line. In a voice conversation that next reply is typically about 16 seconds away, so a 1.8 second verdict lands in time. The lag can only add friction. It can never give away a pass.
A second request instead of a longer wait
My first time limits lost 4.7% of lines, yet every verdict that came back had seconds to spare. The timeouts were hung requests, not slow ones. Now a request with no answer at 2.8 seconds gets an identical twin, and whichever answers first wins.
Fail toward the last known state
A late or failed verdict leaves the character where it was. The system never makes up a default answer. Three failures in a row are recorded once for that conversation.
A ledger for every verdict
Model, versions, a hash of the prompt, the raw output, the parsed verdict, the state after it, and the exact note sent. If a result looks wrong a month from now, I can see why.
The server is the authority
State lives on the server and moves only through the referee and the machine. The score uses the transcript the server fetches from the voice provider. A tampered browser can only fool the person using it.
A separate judge, after the fact
When the conversation ends, an isolated judge scores the whole transcript against a versioned rubric. Then the feedback can say: you got through, but these parts were missing or only half there.
Vendors behind an adapter
The voice provider sits behind one interface. Sending the state note is a single method on it. Swapping a vendor shouldn't touch product code.
The Voltage phone app's Train screen: four levels, Rookie and Pro in progress, Ace and Legend coming soon. The call analysis screen: a score of 78, objective met, a skill chart, and the coach's voice debrief pinned above Practice again.
The phone app, with demo data. Left, the levels a person works through. Right, what they see after a call: the separate judge's score and breakdown, with the coach's voice debrief pinned below.
The coach orb from the app, speaking. It's the voice that gives the debrief after a call.

It runs on vendors I can swap, with the referee on Claude.

Prompts are production code, so they get the same guardrails.

"We lose quality when it's too long" was my own note in September. So the size limit is a test. A prompt over its ceiling fails the build, offline, with no API key. The ceilings only go down.

Judge changes run against a recorded baseline. A score that moves more than the tolerance, crosses a band, or gains or loses a flag fails the check. The rule on that file: regenerate it only after you understand why the score moved, never to make the check pass.

The gate caught a real accusation
A cleanup that made a judge prompt shorter dropped some definitions. The eval came straight back with a rude person relabeled as abusive. On a live conversation, that's an accusation against a real person. The definitions went back in.
A prompt studio
One place to see what every model is told, who owns each part, its version and its size against budget. A workbench shows any prompt line by line with its source. A test runner shows the most a run can cost before it starts.
Layers with owners
My layer holds how every character behaves. The customer's layer holds their content. A test proves every prompt compiles to the same bytes from the layers as from the original file.
AI edits that can be refused
A person describes a change, and a model proposes a patch. The server refuses it, without softening, if it breaks ownership, goes over budget, or has the character lead. It's capped at 100 proposals or $5 a day per company, reserved before the model is called, at about $0.03 each.
Two checks before a serious penalty
The harshest result requires a pattern finder and a meaning check to agree on the same sentence. When testers complained about scores that swung wildly, the causes turned out to be pipeline bugs. Judge variance itself was 0 to 10 points. I fixed the pipeline and re-scored 23 sessions.
A customer's own form
A second judge scores against one customer's written scoring form, used as written.

A note from a tester can become a tested fix.

Testers can talk or type feedback from anywhere in the app. Two rules hold the pipeline up. A submission is never lost because an AI call failed, so the first step just stores what was said, and everything after it is a queue. And the loop closes: the person who reported the problem hears that it changed because they said so.

1. CaptureVoice or text, with the context filled in automatically.
2. Transcribe and classifySpeech to text on the server, then a strict structured classifier.
3. TriageI see everything that comes in, on one screen.
4. ProposeNotes cluster by theme, with one proposed change per cluster.
5. ApproveA person says yes or no.
6. Patch and evalOne small change, then the evals run on it.
7. Tell themThe author hears that it's done.

From a tester's note to a tested fix in seven steps. A person approves every change, and the tester hears back.

I measured how easy it was before I made it harder.

I built a lab of simulated people who play set skill levels against the live character. Before the referee, 168 simulated conversations cost about $6. By-the-book performers won 80% of the time, average ones 77%, and weak ones 52%. The verdict in my notes: far too easy. The character also led the person in 15% of its turns and scored 3.0 of 5 for realism.

Then I wrote down what hard means. A hard conversation is one you can win if you earn it: about 7 in 10 for by-the-book, 3 in 10 for average, 1 in 10 or fewer for weak. That's friction you can overcome, with no dice roll, and no cold character who can't be won over.

By-the-book got through, Oct 22, all seven steps52 of 56
Average got through36 of 56
Weak got through0 of 56
Nervous and weak got through1 of 56
Realism, latest checks, was 3.0 at baseline3.8 to 4.3
Handed lines repeated out of place, that run0
Bad-faith simulated conversations that won a pass0 of 80
Latest lab round, total spend$43.81

The Oct 22 run used rewritten simulated people, so it shows direction against the baseline, and it isn't like for like. A later change made the character quieter, and the nervous rate on one step moved to 2 of 16.

Written bars for every step
Pass rates by level, plus how often the character folds, leads, loops or repeats a line it was handed. A step only counts as ready when it meets them. Several still don't, and the docs say so.
An attack suite
Eleven kinds of bad behavior on the live path: an abrupt change of subject, goofing in a silly voice, rambling, "are you an AI", insults, threats, word salad and prompt injection, plus a friendly control. Several use topics the prompt never names, so a pass means the principle generalized.
A budget on every run
Every lab run has a written cap and a running total. The eval got $3 and used $2.11. The tuning round got a cap of $60. An agent raised it to $80 during the work and told me, and it finished at $78.06. That's in the log too.

Then I told it to stop chasing the number.

Average performers were passing 12 of 12 at two steps, against a bar of 30 to 60%. The easy move was to make the character harder until the number looked right. I said no. At those points in the conversation the character has already agreed to the premise, so it should be easy to finish, and the judge is what grades quality.

The same week, I dropped the idea of several character types. In a side by side, the one lenient type let 3 of 3 average performers through against 1 of 9 for the others. I wanted one consistent bar with variety in the wording, so that's what it became.

Cost is a product requirement, so I measure it per loop.

Voice is about 90% of the cost. So I wrote down the exit before I needed it: if all-in voice cost passes a set ceiling, I pull forward a self-hosted path on open models. The kill threshold is in the architecture rules, and I don't have to win an argument about it later.

One practice loopAbout $0.30
Share of that from voiceAbout 90%
Referee, per line$0.0028
Referee, typical conversationAbout $0.03
Judge, per session$0.013 to $0.02
Voice platform, per minute, when measuredAbout $0.08
The Voltage admin overview: what needs a decision, this week's practice minutes and scores, spend per practice minute, and the voice bill and reply time by model.
The admin view, with demo data. Cost sits beside quality on the first screen: spend per practice minute, and the voice bill and reply time for each voice model.

A second pipeline: 28 videos, no film crew.

I regenerated a set of 28 existing videos as avatar video with HeyGen, with an agent driving the browser. It made 69.8 finished minutes, at about $2.40 a finished minute. Cash paid that cycle was under $100, with about 7 hours of wall clock over two days, mostly waiting on renders, and about 2 minutes of my time per video.

Plenty failed first. Here it is, in the order it hurt.

What didn't work

What I'd tell any team building AI that has to be trusted.

Separate the actor from the judgeIf a model performs and also decides, it's grading itself. Anything you must trust needs its own evaluator and plain code to keep the state.
Measure the failure first11 of 12 made the decision easy. Without it, this would have stayed a feeling, and I'd have kept tuning words.
Fix the prompt before shopping for a modelOne ambiguous word helped hold my first prompt to 64%. Every model improved once it was fixed.
Use structure, then wordsWhen something matters, put it in code. Words moved the numbers a little. A change to the structure moved them a lot.
Small models say what you hand themQuoted lines come back out. Give it a principle and a reason, and keep the answer key out of its reach.
Count both kinds of errorA false pass and a false stop both cost you. So does a false accusation, and that one can cost you trust. Decide what each is worth before you see the results.
Check what your eval can't seeRun it on the old version. If the old one also passes, your test isn't measuring your fix.
Don't tune to a targetA number you chose is still only a stand-in for what you want. When it fights common sense, question the number.
Put the limits in writing firstA budget for each run, a ceiling for each prompt, a cost that triggers a switch. It's much easier to stick to a line you drew in advance.

It's rolling out soon to a 130-person customer. Here's what I can't claim yet.

It's in testing, and it's rolling out soon to a 130-person customer. The referee is on across all seven steps in that test build.

Here's what I can't claim yet. I have no usage numbers, no outcome numbers, and no proof yet that a good score here predicts anything in real life. The lab uses simulated people. The next real test is comparing the judge against human grades on about 30 conversations. If those disagree, the judge has more work to do, and I'd rather find that out from a measurement.

721Commits in 60 days, built by me with AI agents
130People at the customer it's rolling out to soon
About $0.30Cost of one practice loop, about 90% of it voice
0 of 80Bad-faith simulated conversations that won a pass

Most AI features have this bug.

If your AI both performs and decides, I'd want to measure it. That's where I'd start.

Start a conversation