A while back I built an LLM-as-judge pipeline to compare two versions of a prompt. The new version scored higher, so we went with it. A few weeks later it was clear the new prompt wasn't actually better. It wrote longer answers, and the judge liked longer answers.
Nothing was broken in the usual sense. The pipeline ran, the scores were stable, and the dashboard looked fine. The problem was that I had been treating the judge's score as a result, when it was really a measurement, and I had never checked what it was measuring.
That experience changed how I build evals. Most eval setups I've seen since then do measure something. It's just often not what the team thinks it is. This post covers the checks I now run before I trust an eval number, and why I think a lot of "it worked in the demo" failures start in the eval harness rather than the model.
Demos and evals are different things
A demo is a set of examples that work. An eval set is supposed to include the ones that don't.
In practice, the two get mixed up. The person who writes the prompt usually writes the first test cases too. They pick inputs they understand, phrased the way they'd phrase them, covering the cases they were thinking about while writing the prompt. So the eval set ends up looking a lot like the demo, and the prompt passes it for the same reason the demo works.
This isn't carelessness. It's just hard to write test cases for failure modes you haven't thought of yet.
The fix I've found most useful is simple: get test cases from somewhere other than the prompt author. Real user inputs are the best source, even a few dozen. Support tickets, logs from a beta, or inputs from a teammate who didn't see the prompt all work. The goal is an eval set that the prompt wasn't shaped around.
One accuracy number isn't enough
Most eval reports lead with one number. Accuracy, pass rate, average judge score. It's easy to track, easy to put in a slide, and easy to compare across versions.
The trouble is that an overall number can go up while the part you care about gets worse.
Here's a made-up but realistic example. Say your eval set has 1,000 cases, and 100 of them are high-stakes: requests where a wrong answer causes real problems for the user. Version A gets 90% overall and 98% on the high-stakes cases. Version B gets 92% overall and 85% on the high-stakes cases.
If you only look at the headline number, B looks like an improvement. It isn't. B got better at the easy cases and worse at the ones that matter.
This happens more than you'd expect, because prompt changes often trade one kind of behavior for another. A change that makes answers more concise might help on simple questions and hurt on complex ones. A change that makes the model more cautious might reduce bad answers and also cause it to refuse valid requests.
What I do now:
- Report by slice. Break results down by input type, failure type, and severity. The slices depend on your product, but you usually know what they are once you look at real inputs.
- Decide in advance which slices can block a release. If you decide after seeing the numbers, it's too easy to talk yourself into shipping.
- Look at the diffs, not just the totals. Which specific cases went from pass to fail? Reading twenty of those tells you more than the aggregate.
Where LLM judges go wrong
Using an LLM to grade another LLM's output is practical. Human review doesn't scale, and exact-match checks don't work for open-ended answers. I use LLM judges and I'd recommend them.
But a judge is a model, and it has its own tendencies. Three have caused me problems.
Position bias
In pairwise comparisons, where the judge sees two answers and picks the better one, judges tend to favor one position. Some prefer whichever answer comes first, some prefer the second. The effect is strongest when the two answers are close in quality, which is exactly when you most need the judge to be reliable.
The check is straightforward. Run each comparison twice with the order swapped. If the judge is consistent, it picks the same underlying answer both times. If it doesn't, that comparison isn't telling you much.
def position_flip_rate(judge, pairs):
"""
For each (prompt, answer_a, answer_b), ask the judge twice with the
order swapped. A consistent judge picks the same underlying answer
both times.
"""
flips = 0
for prompt, a, b in pairs:
first = judge(prompt, left=a, right=b) # returns "left" or "right"
second = judge(prompt, left=b, right=a)
winner_1 = a if first == "left" else b
winner_2 = b if second == "left" else a
if winner_1 != winner_2:
flips += 1
return flips / len(pairs)
rate = position_flip_rate(judge, gold_pairs)
if rate > 0.10:
raise RuntimeError(f"Judge flip rate {rate:.0%}: pairwise results not trustworthy")
I started with a 10% threshold. That was a judgment call, not a standard. Look at the flip rate on your own gold set first and set the threshold from there. Also worth doing: when you run pairwise evals for real, run both orders and only count a win when both agree. It doubles the cost, but it removes most of this problem.
Verbosity bias
Judges tend to give higher scores to longer answers, even when the extra length doesn't add anything. This is what bit me in the story at the top.
It makes some sense. A longer answer often looks more thorough, covers more ground, and includes more of the words the judge is looking for. But from a user's point of view, a longer answer that says the same thing is usually worse.
To check for this, take your gold set and look at judge score against answer length. If the correlation is strong, and your human labels don't show the same pattern, the judge is rewarding length. You can also run a simple test: take a set of good answers, pad them with a few sentences of reasonable-sounding filler, and see if the scores go up. If they do, you have your answer.
Fixes that have helped me: telling the judge explicitly that length is not a quality signal, asking it to score specific criteria separately instead of giving one overall score, and in some cases normalizing for length when comparing versions.
Self-preference
Judges tend to rate outputs from their own model family higher than outputs from other families. If you use the same model to generate and to judge, your scores are probably a bit inflated. If you're comparing your model against a competitor's using your model as the judge, the comparison is tilted.
The check here is to run a second judge from a different model family on the same cases and compare. If the two judges broadly agree, good. If one consistently prefers outputs from its own family, you've found the bias.
I don't think this means you can never use a same-family judge. It means you should know how much it matters for your setup, and be careful with comparisons across model families.
Check the judge against people
All three of those biases point to the same underlying issue. The judge is a component in your system, and like any component, you should know how often it's right before you depend on it.
This is the part of my platform-security background that carries over most directly. In security work you don't rely on a signal you haven't verified. A certificate means something because you checked the chain. A log entry means something because you know how it got written. An eval score should be treated the same way.
What this looks like in practice:
- Build a small human-labeled gold set. A few hundred examples is enough to start. Have people who understand the task label them. If possible, have two people label each one, so you know how often humans agree with each other. That gives you a realistic ceiling for the judge.
- Measure judge agreement against the gold set. How often does the judge's verdict match the human label? Look at it by slice too. A judge can be fine on easy cases and unreliable on hard ones.
- Re-run the check whenever the judge changes. New judge prompt, new judge model, new model version from your provider. Any of these can shift agreement, and you won't notice from the scores alone, because the scores will still look like scores.
The gold set takes some effort to build, but it's the thing that makes every other number in the harness meaningful. Without it, you know the judge produced a score. You don't know whether the score is right.
Prompt changes need the same discipline as code changes
Prompts are easy to change. You edit a line, try a few inputs, the outputs look better, and you ship. It feels low risk because it's just text.
But a prompt change can change behavior across every input the system sees. Trying three examples by hand and deciding they look better is the same as merging a code change after clicking around the app for a minute. Nobody would accept that for code, and it shouldn't be acceptable for prompts either.
The minimum bar I'd suggest:
- Prompts live in version control. Same review process as code.
- Every prompt change runs the full eval set. Not a sample, not the handful of cases you were thinking about when you made the change.
- Results are compared by slice against the current baseline. The reviewer sees what got better and what got worse, not just the total.
- A regression on a critical slice blocks the merge. Decided ahead of time, not argued case by case.
This does slow down prompt iteration somewhat. In my experience that's fine. Most of the speed you lose was speed you were spending on changes that didn't help.
What a working harness looks like
Putting this together, here's roughly how I structure an eval harness now:
The main idea is that the judge sits inside a loop that checks it, rather than at the end of the pipeline as the final word.
As a checklist:
- A fixed gold set with human labels, versioned so you can tell what changed between runs.
- A versioned judge. The judge prompt and model are pinned, and changing either one triggers a re-check against the gold set.
- Bias checks that run automatically, at least position and length. Cross-family checks can run less often since they cost more.
- Per-slice reporting with critical slices defined up front.
- A path from production back to evals. When something fails in production, it gets triaged, labeled, and added to the gold set. Over time this is where most of your useful test cases come from, because production traffic contains the inputs nobody thought to write.
None of this is complicated. Most of it is the kind of process discipline engineers already apply to code. It just tends to get skipped for prompts and evals, because they're newer and feel more like experimentation than engineering.
Back to that prompt
When I went back and looked at the prompt change from the opening, the model wasn't the problem. The new prompt did exactly what it was told. The eval harness also did exactly what it was built to do. I just hadn't checked whether the judge's score meant what I assumed it meant.
That's the main thing I'd pass on. Before you trust an eval number, find out how it's produced and check it against something you know is right. It's less exciting than trying a new model, but in my experience it's where most of the improvement actually comes from.