TL;DR
The short version
Dianne Penn, Anthropic's head of product for its research and labs teams, says the artifact that carries product intent has changed: for a well-defined problem, an eval — a graded set of example inputs and the outputs you'd accept — is now how you tell the team what "good" means. "Evals are the new PRDs."
The deeper reason is that model capabilities arrive in jumps you can't schedule. So the durable skills are translating messy user feedback into tests a researcher can move, and staying hands-on enough to feel where the models are going. PRDs aren't dead — they've narrowed to alignment and vision work.
Built on Dianne Penn's conversation with Lenny Rachitsky on Lenny's Podcast.
The artifact changed
For two decades, a product manager demonstrated the job by writing a doc. You had an idea, wrote a PRD, aligned people, built it, shipped it, iterated.
Penn's team at Anthropic still writes PRDs. But for a defined problem, the thing that now encodes what the product should do is an eval — a graded set of example inputs paired with the outputs you'd accept. She calls it test-driven development for product: write the test first, then improve the model against it.
The mechanism is concrete. Early Claude users complained it "wasn't good at following instructions." Vague. Penn's team dug into exact prompts and responses and found roughly 80% of those complaints were really one failure: Claude wrote the wrong JSON. So they generated 30 to 40 examples of the failure, made that the eval set, and ran it against every new model version. That capability now passes near-perfectly and stopped being a pain point. The vague grievance became a number a researcher could move.
Why pattern-matching breaks
The reason a static doc is a weaker fit is that the ground keeps moving in discontinuous ways.
Penn points at the scaling-law papers: model loss falls smoothly as you add compute and data, but specific capabilities don't. They appear in sudden jumps — a model goes from unable to compute "1+1" to reliably doing it — and you often need an eval just to notice the jump happened. That makes strategy built on "here's what worked in consumer or B2B SaaS" fragile. The first-principles question — for this user, with this model, right now, what is the value? — beats the pattern you're importing from the last job.
It also changes planning. When a model can suddenly do something you'd scheduled for next quarter, you pull the plan forward. Penn frames the building question as forward compatibility: let's say Claude 8 comes around — what changes in what users do, and what does that mean for how you're building today?
Sweat the tokens, in public
If you can't predict the jumps, the response is to stay close to the technology and to each other.
Penn's version of hands-on is blunt: you have to sweat the tokens as much as you sweat the pixels. Token spend is just the input; the output that matters is experimentation. That's why the onboarding plan for a senior PM on her team is identical to a junior's, and why she still owns one or two workstreams on every model release — a manager who has stopped shipping can't tell what a great AI feature looks like.
The second half is that discovery isn't solo. In Anthropic's early days, nearly the whole company tested Claude in a shared Slack channel; someone would try an idea, others would riff on it, and within about ten requests a new use case would surface. "Experimentation is not an individual sport" — a claim she extends to avoiding burnout, where a team that holds the same first principles lets any one person step away.
There's a matching lesson for using the model itself. Penn's rule is to form a point of view first, then use Claude as a sparring partner that pushes back — not an assistant that agrees. Alignment work, counterintuitively, is what makes the model useful enough to disagree well.
A thinking partner doesn't just agree with you. It should add to you, and you should come away at the end of the day having better ideas because you worked with Claude. That should be the hero goal — not just making your ideas 10% better.
Dianne Penn, Anthropic
The boundary
Two cautions keep this honest.
Evals can overfit. Optimize a metric and you can pass the test while missing the messy real-world tail. Penn's own guard is asking whether an eval is "on distribution" — whether it captures both the cases that should fail and the ones that shouldn't — before trusting a green score. And PRDs aren't dead: she still writes one for every model to align researchers, legal, and safety, and for ambiguous bets where the value is in the product vision. OpenAI's Codex-app lead says the same, though two vendors agreeing isn't independent proof.
The larger caveat is the source. This is Anthropic's product leader describing Anthropic's way of working, while actively hiring for it. The portable parts — encode feedback as evals, build for jumps you can't schedule, keep leaders shipping, use AI to argue with you — stand on their own. The "our culture is the secret sauce" thread is real but self-interested. Take the method; weigh the narrative.
