TL;DR
The short version
Tired of one-off vibe checks, Claire Vo had Claude Code build a repeatable, taste-weighted model benchmark — blind-testing Sonnet 5, Sonnet 4.6, Opus 4.8, GPT-5.5, and Gemini 3 Pro on PRDs, prototypes, agentic work, and voice.
The useful finding wasn't a winner. Her hand-scores and the LLM-as-judge scores landed on nearly opposite conclusions: brand-new Sonnet 5 topped the automated leaderboard but finished bottom of her personal list.
Built on Claire Vo's How I AI episode. Sonnet 5's launch and pricing are independently verified; the leaderboard is Vo's own single-run eval.
Build the eval from your own history
The first move is worth copying on its own. Vo asked Claude Code to design a benchmark based on their past work together. Every Claude Code session is stored locally, so the agent can read your real history and propose tasks grounded in what you actually do — not a generic leaderboard.
The design principles it returned are the ones that matter: frozen inputs, blind scoring where possible, a clear rubric. She narrowed the tasks to four that map to real builder work — PRDs, prototypes, agentic multi-step, and agent voice — then ran five models blind.
Two scorers, two verdicts
Here's the design that surfaced the real insight. Vo hand-scored roughly 64 generations on gut feel — "Would I ship this? Does it sound like me?" — while two LLMs, Opus 4.8 and GPT-5.5, judged the same outputs in parallel.
The automated leaderboard put Gemini 3 Pro, Sonnet 5, and GPT-5.5 on top. Her taste put Sonnet 4.6 first and Gemini 3 Pro dead last. Not a small gap — close to inverted.
What I thought was good, the model thought was bad. And what the model thought was good, I thought was bad.
Claire Vo, How I AI
That divergence is the whole episode. If you're using an LLM to grade your outputs, you may not be measuring what you think you're measuring.
Why the judges are mushy
Vo's read: models regress to the middle of the bell curve. Everything gets a 7 out of 10. They're not spiky enough to separate great from fine, and they miss the taste signals a human catches in a glance — "this is cute," "this is sharp," "this is slop." The rubric didn't see what her eye saw on the first screenshot.
The reverse was also true, and more humbling. The automated scores caught broken working code and ignored-constraint failures in generations she'd rated highly on visuals alone. She'd been eyeballing the first screenshot; the model actually checked whether the thing ran. Human taste and machine rigor each caught what the other missed.
Sonnet 5, and the model-by-task answer
The verdict on the actual subject: Sonnet 5 topped the automated board but had too many broken prototypes. "When it worked, I really liked it, but it didn't work enough." It finished at the bottom of her personal list — a striking outcome for a model that, on public benchmarks, ranks near the top and launched June 30 at a steep discount to Opus ($2/$10 per million tokens through summer, versus Opus 4.8's $5/$25).
To reconcile the two verdicts, Vo built a "Claire-weighted index" — 70% her taste, 30% backend scores — and even that refused to crown one winner. The honest recommendation is by task: GPT-5.5 for a comprehensive PRD, Sonnet 4.6 for prototyping and good conversational vibes, Opus 4.8 and Sonnet 5 for dense codebase work.
The lesson already landed: an automated leaderboard and a person with judgment will not agree, and pretending otherwise hides the thing you actually needed to know.
