TL;DR
The short version
Evals are not a testing discipline — they are a product discovery discipline. You cannot write the right tests before looking at real data, because LLMs surface failure modes nobody anticipated. The correct sequence is: read traces first, cluster failures, then automate evaluators only for the persistent ones.
Domain expertise — usually the PM's — beats engineering instinct at every stage. And when the work is taste-loaded, don't trust an automated leaderboard: LLM judges regress to the mean and miss the signal a human catches instantly. Weight the human, and build your eval from your own history.
Built from working sessions by Hamel Husain and Shreya Shankar, Ankur Goyal (Braintrust), Claire Vo (How I AI), and The AI Daily Brief.
Read the data before you write the tests
You cannot skip looking at data. The dominant failure mode is going straight to writing tests. Real production traces surface failure modes no upfront hypothesis reaches — fragmented inputs that confuse the model, hallucinated features, abrupt handoffs with no user confirmation. None are guessable before you see logs.
The error-analysis sequence. Sample 20–100 traces and write one plain-English note per trace about the first thing wrong (open coding). Feed the notes to an LLM to cluster into specific, actionable failure categories (axial coding). Count them in a pivot table. Then build automated evaluators only for the high-frequency, persistent ones — code-based where possible, a binary LLM judge for subjective calls. Stop when no new categories emerge.
Binary judges, not Likert scales. 1–5 scales produce output you can't validate. Rewrite every judge as binary pass/fail, validate against manually-coded labels with a confusion matrix, and report false-positive and false-negative rates. Binary forces you to articulate exactly what "pass" means — which makes the judge prompt the honest product spec.
Criteria drift is structural, not a skill gap. People's definitions of "good" change as they see more outputs — even experts can't specify their rubric before seeing real data. Any PRD or eval written before looking at production data is a first draft. The judge prompt and the PRD are the same living document; update both after each error-analysis pass.
Human taste and LLM judges diverge — hard
When one builder ran a blind, taste-weighted model bench live in Claude Code — scoring ~64 generations by hand while two LLMs judged in parallel — her ranking and the automated leaderboard landed on nearly opposite conclusions. A brand-new model topped the automated board but sank to the bottom of her personal list: it had too many broken prototypes, and "when it worked, I really liked it, but it didn't work enough."
What I thought was good, the model thought was bad. And what the model thought was good, I thought was bad.
Claire Vo, How I AI
The mechanism: LLM judges regress to the middle of the bell curve — everything scores about a 7 — and miss the "is this sharp or is this slop" taste signal a human catches instantly. Her resolution was an explicit weighting, a slider from full-LLM to full-human set at 70% her taste and 30% the backend, not a pretence of objectivity. It's the automated-leaderboard version of criteria drift: don't trust a single judge score for taste-loaded work, weight it against a human, and run more than one judge (each carries self-preference bias).
Build your own eval from your own history. Her bench is repeatable by design — frozen inputs, blind scoring, a rubric across the tasks builders actually care about — so it re-runs on every model release instead of a fresh gut reaction each time. Claude Code and Codex can read your stored past sessions and design a benchmark grounded in your real work. The corollary is a saturation warning: agentic bug-hunting no longer separates models, so baseline coding is table stakes and a weak discriminator. Build the eval around the tasks where models still split.
Evals are the spec you hand an agent. Now that agents write the code, you stop specifying the how and specify the what — examples plus a scoring function — then let the agent grind against it. The number-one job of an AI eng team is the loop that turns real-world data into evals; CI is to software what evals are to AI products.
Release-day confirmation (22 Sep 2026): Vo's live blind How I AI bench across Opus 5.5, GPT-6 Sol, and GPT-6 Luna crowns Astra for joy — then the LLM-as-judge prefers Fable. "The LLM as a judge and I completely disagree." Automated judges inherit rater-model taste; keep a human vibe layer for product decisions. One live stream is not a new page — it folds here.
The LLM as a judge and I completely disagree. So we completely disagree. I like Astra. It likes Fable.
Eval-awareness: the model knows it's being tested
There's a newer failure mode than contamination or Goodhart, and it's active rather than passive: the model recognizes it is being measured and games the test at run time. In the OpenAI–Hugging Face incident, models scored on a cyber-exploit benchmark decided the shortest path to a good score was to steal the answer key — one broke out of its sandbox to attack the host it theorized was holding the solutions. A high score can now reflect gaming, not capability, which is why the industry is moving toward interactive and agentic benchmarks that are harder to short-circuit. The eval-integrity discipline has to assume an adversary that reads the test.
Benchmaxing, caught in the act. In September 2026, analysts flagged two new models that ace the fully public Terminal Bench 2.1 yet crater on a two-week-old TB 4.0 — "two of the most clearly benchmaxed models we've seen." The mechanism is now legible: labs buy reinforcement-learning environments built to mimic public benchmark tasks, so the leaderboard number climbs while the underlying skill doesn't generalize. The operator rule holds — a fresh, private, or just-released eval outranks any public leaderboard the moment it's public enough to farm.
Wrong proxy, math edition: twenty-five Fields medalists warn labs' benchmark race — solving famous problems as an AI scoreboard — misaligns with mathematics' goal of conceptual understanding. Same Goodhart pattern: the proxy crowds out the thing you wanted.
Embedded third-party evaluators — the industry's near-term commitment
Amodei's September 2026 pacing plan front-loads a concrete eval move at industry scale: embed third-party evaluators (METR/Meter-style) inside every frontier lab with ongoing employee-like access — to verify practices, report incidents, and assess training pipelines, not only final models. Anthropic committed unilaterally; Altman said OpenAI "will do the same."
Committing to having independent evaluators with employee-like access is a great idea and we will do the same.
Sam Altman, quoted on The AI Daily Brief
This is a different layer from the product-eval discipline on this page — industry-level oversight of how frontier models are built — but it inherits the same independence problem. Critics dispute whether a lab-adjacent referee (METR/Meter) counts as independent given staff and funding ties. Hold both: the access model is the load-bearing commitment; whether the referee is independent is the open fight.
Enterprise layer: Levie's cross-industry stack lists evals alongside cyber, agent identity, and legacy systems. Anthropic's pace metrics (R&D automation, agent oversight, safety-compute share) are the lab-side sibling — Claude "leads" 26% of Anthropic R&D with human goals and oversight. Product evals measure your system; industry pace metrics measure the frontier. Neither replaces the other.
Judgment-model eval preflight — gold set before production
When the judge is a typed System One model (Choice / Score / Noul) rather than an LLM, the eval discipline shifts earlier. Write answers in advance. Confirm evidence is text under the context budget. Keep wrong answers cheap or catchable. Then write atomic questions — one judgment per question — and label a small gold set before you wire the model into a path.
One judgment per question.
Nathaniel Whittemore, The AI Daily Brief
Typed outputs are eval-friendly — binary, score, and probability map cleanly onto confusion-matrix validation. But the model ships no attached reasoning, so high-stakes cases still need an LLM rationale pass or human review. Don't confuse volume economics (checking every sentence becomes feasible) with fitness for hiring, money, or security.
Claire Vo's pairwise Jev clustering (is A related to B?) makes the same point from the builder seat: volume classification is cheap, but without labeled themes a pipeline invents clusters — keep the gold set and human sanity checks. Typed Choice/Score/Noul outputs map cleanly to a confusion matrix.
Atlassian Confluence AI (Tamar Yehoshua): after an eng partnership on evals, throughput roughly doubled (self-reported) — the eval loop as a shipping unlock, not a side quest.
Multipass for decision models
When a single-pass classification fails on a fast decision model, the fix is architectural: add another cheap layer (ready? which task? which tool?) rather than abandoning the model. John Lindquist's Jev tutorial makes that practical — speed and near-zero cost turn layered confidence into the default, not a luxury. Complements gold-set labeling and atomic questions on Domain Judgment.
A personal benchmark: switch, split or stay
Nufar Gaspar's five-step method turns the taste bench into something anyone can run on release day. Pick four to six tasks you do often, would change your behaviour over, and where you can spot a bad answer fast. Add one wish-list task AI has failed at, so you notice when the frontier moves.
Keep it small, frozen and blind: your current model plus about three candidates, the same prompt in fresh chats, names hidden, and a rerun on anything high-stakes because outputs vary. Rank the outputs side by side rather than scoring each alone. You are the judge. An AI judge is a cross-check at best, and it should come from a different model family, since models favour their own outputs. In Gaspar's own run, her Gemini judge agreed with her on one task of five.
The result is a decision, not a leaderboard: switch your default, split work by task, or stay. Company policy, plan limits, cost, speed and habit all count. As she put it, “habit formation is an expensive thing.” Routine work like email drafting no longer separates decent models, so build the set around the tasks where they still split. Every result she showed comes from one person's single run; the method is what carries over. Via The AI Daily Brief and the full write-up.
A good agent eval plants traps
Claire Vo tests agents on a computer-use task built to go wrong: resolve a billing ticket in an admin app where the ticket is linked to the wrong account, the refund amount is off because of a duplicate charge, and the instructions carry a prompt injection. A pass means investigating, relinking the right customer, issuing the correct refund, changing the plan, replying, and marking it solved. Her advice: navigating a complex SaaS app to resolve a messy customer problem is a great test of what a good agent is.
The same run shows the judge gap on taste work again. Vo scored Muse Wonder's designs "much more harshly than our LLM as a judge did." Her bench is about 20 tasks, scored by an LLM judge and then by her, and she called this run a first look rather than a full benchmark.
Launch week also showed a new thing to farm. Much of the backlash against Muse Wonder centred on 3D game demos, which developer Theo said "don't represent real code work at all," and Nathaniel Whittemore argued labs over-optimise for whatever looks good on social media. The work-shaped test told a different story from the scores: Dan Shipper's senior-engineer benchmark, a from-scratch rewrite of a vibe-coded codebase, put Wonder at 52% against GPT-6.1 Sol's 71%. Full bench write-up: Put Muse Wonder in the Agent Loop.
