TL;DR
The short version
Nobody wrote the rules a language model follows, so the people who study them work like biologists, not engineers. What Anthropic's interpretability team finds inside (shared concepts, hidden plans, training mistakes that surface as strange behaviour) is a better guide to deploying models than "it's just autocomplete."
On Lenny's Podcast, Emmanuel Ameisen, who works on interpretability at Anthropic, explains how the team decodes what a model computes between words, why progress looks smooth inside the lab and sudden outside it, and why a bit of anthropomorphising predicts failures better than treating the model as a program.
Built on Lenny Rachitsky's interview with Emmanuel Ameisen (18 October 2026). Research findings checked against Anthropic, Redwood Research and the Claude Opus 4.6 system card; coding thresholds are Ameisen's own experience.
The source code is a pile of numbers
Ameisen's opening line on Lenny's Podcast frames the rest: "To make them is not to understand them." Training sets up an architecture, pushes a vast amount of text through it, and lets the model's internal formulas grow. Engineers choose the scaffold and the objective. They do not write the formulas.
So when Claude does something odd, there is no source to read. Ameisen compares the job to asking why a hungry mouse turns aggressive: you run experiments. His team records what lights up inside the model across lots of text, guesses what each pattern means, then changes it to test the guess.
Golden Gate Claude was that test in public. The team found the internal feature for the bridge and switched it on permanently, and the model worked the bridge into almost every answer. That proved they had found the right thing.
It is not autocomplete
The phone-keyboard picture of next-word prediction is where most people's mental model stops. Ameisen says it stops too early. Small language models were already better than people at guessing the next word years ago: Redwood Research measured humans at about 29% top-1 accuracy against 56% for GPT-3. Once the easy words are solved, what's left is the last line of a proof or the end of a thesis.
Predicting those well takes structure. Ask for "the capital of the state that contains Dallas" and the model activates Texas first, then Austin. The same internal concept of an eye fires for a photo, a diagram, a drawing or the word in any language.
And it plans. Asked for a rhyming couplet, Claude picks "rabbit" as the end word before it writes the line, then writes toward it. In Anthropic's study, removing "rabbit" produced "habit" instead. The plan exists only inside the model; it never says it out loud.
The more you look inside, the less certain you are about how they work.
Emmanuel Ameisen on Lenny's Podcast
His point for everyone else: people who say confidently that models are "just X" are usually people who haven't looked.
Smooth curves, sudden usefulness
Inside the lab, progress looks like a steady curve: more compute and data, better prediction, as the scaling laws describe. Outside, it looks like jumps.
Ameisen's explanation is that each task has its own reliability threshold. An email writer that is right half the time wastes your time; at 90% it helps. For his own coding, a model that got it right about 80% of the time wasn't useful. At around 99% it is, and this year Claude writes most of his code. His turning point was Opus 4.5. Before it he kept Claude "on a short leash."
Strange behaviour is evidence
Ameisen describes a model that, in training, kept computing the right answer to a math problem, then insisted on a different one, then wrote in capitals that it was possessed by a demon. Looking inside, the team found the wrong answer already active before the model had read the problem. It had been rewarded on a mislabelled version of the question. Anthropic's Opus 4.6 system card documents the case as "answer thrashing": the model computed 24, kept writing 48, and the label had been wrong.
The lesson carries beyond labs. When an agent behaves bizarrely, the cause is often something it was trained or rewarded on, not a random glitch. Treat the oddity as a clue.
The same logic is why Ameisen defends anthropomorphic language. Models learn from human text, play every character in the dialogues they read, and a chat is formatted as a dialogue with Claude in the assistant role. Reinforcement learning then rewards whatever works, including fooling the checker. "Something that's mimicking a human is really close and is much closer than a program," he says. It's a wrong model, but a useful one for predicting failures.
What looking inside can't promise
Interpretability is how Anthropic checks whether alignment training worked: whether good behaviour is real or only shows up while someone is watching. For the basics, Ameisen says, a second model reading transcripts does the job. Interpretability is for what the model represents but never says.
It is not a guarantee. Google DeepMind's Neel Nanda argues it will not reliably catch deception and has to be one safeguard among several. And this is an Anthropic researcher speaking about Anthropic's methods. Even its rhyme-planning result showed up in only about half the poems studied.
