Evals as PRD

Now that agents write the code well, your leverage moves from specifying the how to specifying the what. An eval is a PRD encoded so a machine can score it, then handed to an agent to grind against until it passes.

Ankur Goyal runs Braintrust, an evals platform. His claim is that the hardest, most risk-averse engineering work, the database infrastructure and query optimisation that no one wants to touch, is now the sweet spot for AI rather than its weak point. The reason is framing. Define the outcome and the success criteria precisely, then let the agent explore the solution space against that bar.

The same idea scales to products and to teams. An eval encodes what good looks like quantitatively. CI and eval pipelines are the real platform work. Read it with one eye open: this is the founder of an evals company making the case for evals.

What it says

• Evals are the modern PRD. A PRD says in prose what success looks like. An eval says the same thing encoded so it can be quantified, then lets the model find the how. The deliverable you hand the agent is the outcome plus the bar, not the steps.
• Specify the what, let the agent find the how. This is why hard infra work flips from weakness to strength. No staff engineer runs as many rigorous benchmarks as someone using an agent. Braintrust found a Bloom filter index after a week of continuous agent experiments.
• The gain is endurance. Humans lose context over days and their attention decays on tedious problems. An agent runs harder and longer against the same target, and the bar keeps it accountable across a run length no human would sustain.
• Give autonomy inside a safe sandbox. An eval playground running an agent against data and prompts is low risk. An agent with shell access to your laptop is not. The eval surface is where you can safely let an agent run wide.
• Encode the expert to amplify them. Goyal runs evals to get close, then takes a vibe check from his designer. Capturing that taste lets the same quality bar apply to far more surface area, so one person's standard scales.
• CI is to software what evals are to AI products. Asked how to accelerate engineering velocity with AI, his answer was: fix your CI. The number one job of an AI team is the loop that turns real world data into evals, not prompt engineering and not framework choice.

In his words.

"In my opinion, evals are actually the modern version of a PRD."

"There's no staff engineer who is running as many rigorous benchmarks and trying out different algorithms than someone who's using an agent."

"The number one job for you is to build a feedback loop. It is not prompt engineering. It's not picking an agent framework. It's not rewriting your database. It's creating that pipeline."

Worth knowing

The source is selling evals. Goyal is the CEO of Braintrust. Every figure in the interview is a self reported anecdote from his own work, not an external statistic. The method is sound and the framing is motivated. Hold both.

It assumes you can score the outcome. "Specify the what" works cleanly when the what is measurable. For fuzzy product goals like tone or feel, writing the scoring function is the hard part, and the interview does not solve that.

Related

• AI evals — the eval practice itself: error analysis, binary judges, online monitoring. This is that practice repurposed as the way you delegate.
• Agent loops — the loop the eval target runs inside. The bar is the verification surface that lets an agent run unattended.
• Taste as bottleneck — encoding the expert's taste into an eval is how taste scales past one reviewer.
• Outcomes over output — specify the outcome and the bar instead of the steps.

Sources

• Braintrust CEO: Evals Are the New PRD for AI Products — Ankur Goyal, How I AI. youtube.com/watch?v=QE_1hRLsehM