TL;DR
The short version
Evals are not a testing discipline — they are a product discovery discipline. You cannot write the right tests before looking at real data, because LLMs surface failure modes nobody anticipated. The correct sequence is: read traces first, cluster failures, then automate evaluators only for the persistent ones.
Domain expertise — usually the PM's — beats engineering instinct at every stage. And when the work is taste-loaded, don't trust an automated leaderboard: LLM judges regress to the mean and miss the signal a human catches instantly. Weight the human, and build your eval from your own history.
Built from working sessions by Hamel Husain and Shreya Shankar, Ankur Goyal (Braintrust), Claire Vo (How I AI), and The AI Daily Brief.
Read the data before you write the tests
You cannot skip looking at data. The dominant failure mode is going straight to writing tests. Real production traces surface failure modes no upfront hypothesis reaches — fragmented inputs that confuse the model, hallucinated features, abrupt handoffs with no user confirmation. None are guessable before you see logs.
The error-analysis sequence. Sample 20–100 traces and write one plain-English note per trace about the first thing wrong (open coding). Feed the notes to an LLM to cluster into specific, actionable failure categories (axial coding). Count them in a pivot table. Then build automated evaluators only for the high-frequency, persistent ones — code-based where possible, a binary LLM judge for subjective calls. Stop when no new categories emerge.
Binary judges, not Likert scales. 1–5 scales produce output you can't validate. Rewrite every judge as binary pass/fail, validate against manually-coded labels with a confusion matrix, and report false-positive and false-negative rates. Binary forces you to articulate exactly what "pass" means — which makes the judge prompt the honest product spec.
Criteria drift is structural, not a skill gap. People's definitions of "good" change as they see more outputs — even experts can't specify their rubric before seeing real data. Any PRD or eval written before looking at production data is a first draft. The judge prompt and the PRD are the same living document; update both after each error-analysis pass.
Human taste and LLM judges diverge — hard
When one builder ran a blind, taste-weighted model bench live in Claude Code — scoring ~64 generations by hand while two LLMs judged in parallel — her ranking and the automated leaderboard landed on nearly opposite conclusions. A brand-new model topped the automated board but sank to the bottom of her personal list: it had too many broken prototypes, and "when it worked, I really liked it, but it didn't work enough."
What I thought was good, the model thought was bad. And what the model thought was good, I thought was bad.
Claire Vo, How I AI
The mechanism: LLM judges regress to the middle of the bell curve — everything scores about a 7 — and miss the "is this sharp or is this slop" taste signal a human catches instantly. Her resolution was an explicit weighting, a slider from full-LLM to full-human set at 70% her taste and 30% the backend, not a pretence of objectivity. It's the automated-leaderboard version of criteria drift: don't trust a single judge score for taste-loaded work, weight it against a human, and run more than one judge (each carries self-preference bias).
Build your own eval from your own history. Her bench is repeatable by design — frozen inputs, blind scoring, a rubric across the tasks builders actually care about — so it re-runs on every model release instead of a fresh gut reaction each time. Claude Code and Codex can read your stored past sessions and design a benchmark grounded in your real work. The corollary is a saturation warning: agentic bug-hunting no longer separates models, so baseline coding is table stakes and a weak discriminator. Build the eval around the tasks where models still split.
Evals are the spec you hand an agent. Now that agents write the code, you stop specifying the how and specify the what — examples plus a scoring function — then let the agent grind against it. The number-one job of an AI eng team is the loop that turns real-world data into evals; CI is to software what evals are to AI products.
