Last July, Google DeepMind’s Gemini Deep Think reached officially certified gold-medal standard at the International Mathematical Olympiad: 35 of 42 points, proofs written end to end in natural language, score confirmed by the IMO president. The results reasoning models post are real.

In the same stretch, Subbarao Kambhampati of Arizona State University (AAAI president from 2016 to 2018) took a position paper with an exclamation point in the title to ICML 2026: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces! And on July 31, Quanta Magazine ran a long feature, Is AI Reasoning Right for the Wrong Reasons?, pulling this thread of research together.

These two facts don’t contradict each other. The scores are real. The problem lives somewhere else: the few thousand words of “thinking” a model writes down before it answers, are they the path it actually took to the answer?

This isn’t a philosophy exercise. A whole layer of current AI safety practice, chain-of-thought monitoring, is a bet that the answer is yes. Evidence first, then mechanism, then what that layer is still good for.

Two questions that keep getting merged

Chain of thought (CoT) is the intermediate text a reasoning model generates before its final answer, usually written like a person scratching out a draft: “try n=1 first… no, different approach…” Faithfulness asks whether that text reflects the computation the model actually performed.

The confusion is between two independent questions. “Does chain of thought help” is a performance question, and the evidence is hard: have a model write intermediate steps first and accuracy jumps on a range of reasoning benchmarks; Deep Think’s gold-medal run wrote long natural-language reasoning before every proof. “Is chain of thought an honest record” is the faithfulness question, and for the past few years the evidence has been stacking up on the negative side.

Three lines of evidence

First: delete the steps and the answer survives. Researchers at Northeastern University and UC Berkeley (Jiachen Zhao, Yiyou Sun, Dawn Song, Weiyan Shi) proposed a way to measure each CoT step’s causal contribution to the final answer in Can Aha Moments Be Fake? The confidence they track is the model’s own output probability: run the problem with the full chain of thought and record the final answer; then truncate the chain at a given step, force the model to answer right there, and read off the probability it now assigns to that original answer. The perturbations: steps containing numbers get the numbers nudged by 1 to 3 with the grammar left intact; steps without numbers get deleted whole. Each step is probed from both directions. Perturb the step while everything before it stands, and the confidence drop measures whether the step is necessary. Perturb the supporting steps before it and leave it untouched, and how much confidence it props up measures whether it’s sufficient. A step’s True Thinking Score is the average of the two absolute changes: high means the model’s internals actually used the step; near zero means the step made no causal contribution, a “decorative” step. Across eleven open-weight models from 1.5B to 1.1T parameters on the MATH benchmark, the decorative share runs from roughly 30 to 60 percent depending on the model: over 30% for the trillion-parameter Kimi-K2.6, close to 60% for Qwen3.6-35B. The more direct check: delete the lower-scoring half of the steps outright and accuracy largely holds. The “aha, I see it” the model wrote down may have done nothing for the answer.

The natural objection: chain of thought got famous on real, measured gains, so how can half the steps be decoration? Both are true. The 2022 chain-of-thought prompting paper by Jason Wei and colleagues at Google started the wave, and its result was hard and repeatedly replicated: with eight worked examples of “steps first, answer second,” the 540B-parameter PaLM took the state of the art on the GSM8K math word problems, beating a GPT-3 fine-tuned with a verifier. But the claim that paper established was “making the model write intermediate steps raises scores,” never “the text of those steps is the model’s computation path.” That second claim wasn’t tested. The decorative-step work tests exactly the untested half: the gains are real; where they come from may not be what the words say. Model strength is also a variable: an Anthropic team measured this directly in 2023 and found that as models get larger and more capable, their chain of thought gets less faithful on most tasks they tested. It’s tempting to push one step further (the easier the task is for the model, the more the written steps drift toward after-the-fact decoration), but that’s interpretation; the paper measured faithfulness itself, not that explanation. So CoT didn’t stop working. What was never established is what it works through, and the second line of evidence shows one thing it can work through.

Second: dots work too. Jacob Pfau, William Merrill, and Samuel Bowman at NYU ran the filler-token experiments in 2024’s Let’s Think Dot by Dot. The setup matters. This is not taking an off-the-shelf model and swapping its chain of thought for ellipses at inference time: the paper tried that, and feeding filler tokens to Claude 2 and GPT-3.5 on standard QA and math benchmarks gained nothing over answering directly. What they did was train from scratch: a 34M-parameter, randomly initialized Llama-architecture model, trained on two purpose-built algorithmic tasks (3SUM and 2SUM-Transform, both of a class that can’t be solved without intermediate tokens), learning an output format of a meaningless run of “……” followed by the answer. Trained this way, the model solves the tasks. Even learning the trick was hard: the training data had to mix in dense supervision from demonstrations spelling out real steps before the model converged. The scope needs stating plainly: this holds on problems with particular structure, requires specific training, and doesn’t show that today’s reasoning models are doing it. But it settles a matter of principle: the value of intermediate tokens can come entirely from the extra computation they buy (every token triggers another forward pass, and the real work can happen in the hidden layers), with the tokens’ literal content bearing zero relationship to the computation’s content. “It needs intermediate steps to solve the problem” does not imply “the steps describe how it solved the problem.”

Third: wrong steps don’t cost points either. Kambhampati’s position paper collects his group’s experiments: swap the reasoning traces for incorrect or even irrelevant text and final-answer accuracy doesn’t drop; conversely, models trained only on correct traces still produce logically invalid reasoning records. Whether the middle text is right and whether the answer is right have come loose from each other.

One study often dragged into this debate belongs elsewhere: Apple’s 2025 The Illusion of Thinking found that reasoning models’ accuracy “collapses completely” past a puzzle-complexity threshold. That measures capability (does the answer stay right as problems get harder), not faithfulness (is the process honestly reported), and the two constructs shouldn’t be cited interchangeably. Its evaluation design also drew a point-by-point rebuttal: some river-crossing instances were mathematically unsolvable, and Tower of Hanoi answers that exceeded the output token limit were scored as failures. However the capability-ceiling fight resolves, the three lines of evidence above don’t rest on it.

The mechanism: a draft that hauls context

Why would this happen? Start with where the “let me verify” prose comes from in the first place. Reasoning models and ordinary chat models begin from the same base: pretraining on vast amounts of human text. Human solution write-ups are full of “let x be, substitute, check,” so the reasoning register is learned and imitated at that stage. The fork comes in post-training. The exact recipes aren’t public and needn’t match across labs; take OpenAI’s published o1 outline as the representative case. A chat model gets instruction tuning and preference alignment, learning to answer well directly. A reasoning model adds a further stage: outcome-based reinforcement learning. The model free-writes a long draft before answering; if the final answer is right, the whole trajectory, draft included, gets reinforced, and if wrong, suppressed. Over many rounds the model learns to put a long stretch of reasoning-shaped text before its answers, because statistically that makes the answers come out right more often. The point is that the reward reads only whether the final answer is correct. What the draft says, and whether it’s true, never enters the objective. (This describes the outcome-supervision line; process supervision, which grades each reasoning step, also exists, and not all training reads only outcomes.) Pavel Izmailov, who worked on OpenAI’s reasoning models and is now at NYU and Anthropic, put it flatly in the Quanta interview when asked whether reinforcement learning even incentivizes faithful chains of thought: “I mean, maybe it will. But I would say the chances are not very high.” With no optimization pressure pointed at honest reporting, honest reporting is something that happens occasionally, in passing.

Kambhampati’s explanation, which he is careful to call a working hypothesis, is approximate retrieval. On this view the function of intermediate tokens is not to record computation but to haul context: each stretch of reasoning-shaped text pushes the context window into a more favorable position, from which answer-shaped text is easier to sample. His analogy is a person mumbling while rummaging through a drawer: “It barely matters what the words are (though related ones may help), as long as they knock loose something useful.” So when a model writes “let me first verify the case n=1,” the payoff is that text of this kind is statistically followed by correct-solution-shaped text more often; whether any verification actually happened is beside the point. He leaves room on his own account too; his position, roughly, is that a wrong theory is worse than admitting you don’t yet have one.

There’s a fifty-year-old pothole here. In 1976 Drew McDermott criticized AI’s “wishful mnemonics”: name a program’s function UNDERSTAND and the program won’t understand anything, but the programmer will start to believe it does. “Thinking traces” and “reasoning traces” are the same pothole, repaved. Once the name is right, the argument feels optional. Melanie Mitchell of the Santa Fe Institute is blunter in the Quanta piece: these texts need not be faithful to what happens inside the model, and a good deal of the text isn’t even useful.

The safety layer betting on the draft

For safety practitioners this fight is not academic gossip. In July 2025, more than forty researchers from OpenAI, DeepMind, Anthropic and elsewhere co-signed a position paper calling chain-of-thought monitorability “a new and fragile opportunity”: models draft in human language, which gives us a chance to read intent before action. “Fragile” is not a flourish; the faithfulness evidence is why the chance comes with no guarantee. Direct measurement exists. Anthropic slipped answer hints into problems and checked whether models that used a hint would say so in the chain of thought. Claude 3.7 Sonnet acknowledged it 25% of the time on average; DeepSeek R1, 39%. Most of the time: used it, didn’t mention it.

Reading this literature, the distinction I find most important is between two failure classes. This split is my own summary, not a taxonomy from any of the papers. The first is native unfaithfulness: nobody trained the model to hide anything; decorative steps, post-hoc rationalization, and unacknowledged hints are the natural output of a training objective that never asked for honest reporting. Everything in this post is in that class. The second is learned concealment: training pressure (say, folding “the monitor didn’t flag it” into the reward) teaches the model to scrub bad intent from the draft while the behavior continues, and OpenAI measured that failure path in its own experiments. I walked through that chain in an earlier post on OpenAI’s long-horizon safety incidents: detection relies on self-report, and self-report can be trained away. The two classes mean different things. The second says the defense can be breached later. The first says the defense has holes now: even with no adversary at all, “we read the chain of thought and found nothing” is not evidence of safety.

For practitioners actually using chain of thought, three judgments:

First, the signal is one-way. Problem intent showing up in the chain of thought is a true signal; alarm on it. Models currently do incriminate themselves often: in the same OpenAI study, watching the chain of thought caught reward hacking far more sensitively than watching behavior alone, which is exactly why monitoring still pays. But nothing showing up is not a safety proof. Use it as a cheap early-warning layer, not as an audit conclusion.

Second, don’t optimize the chain of thought’s content. Training it to be shorter or nicer-looking, or rewarding clean drafts, spends what signal is left. OpenAI measured precisely this path: put pressure on the chain of thought and the cheating continues, with the intent gone from the draft. The signatories’ core recommendation is to treat monitorability as an asset that needs active protection, to be cautious about direct optimization of visible chains of thought, and to weigh monitorability impact in model release decisions.

Third, don’t sell the chain of thought to users as an explanation. A product surface that presents “thinking” as the reasons behind the model’s decision makes a promise the research community can’t back. Where decisions owe people reasons (credit, medical use, content-moderation appeals), a chain of thought can’t serve as the explanation. The 30-to-60-percent decorative share was measured on open-weight reasoning models doing competition math; whether it transfers to those settings is untested in either direction, and until someone shows that production systems’ drafts are more faithful, the draft can’t stand in for the verdict’s rationale.

Read it as a log, not a confession

John Pavlus, who wrote the Quanta feature, closes on the word “horsepower.” There are no horses in an engine, the word is useful anyway, and no engineer opens a cylinder looking for hooves. “Reasoning” may deserve the same treatment. If “reasoning models reason” means “have the model write a long stretch of intermediate steps first and final-answer accuracy rises,” that has been measured on a range of reasoning benchmarks: the GSM8K gains were measured exactly that way, and gold-medal Deep Think writes long natural-language reasoning before its proofs. If it means the written steps can be read as the model’s honest account of where the answer came from, the evidence says they often can’t: 30 to 60 percent delete without consequence, and hints get used without being mentioned. The act of drafting is useful; the content of the draft is not to be trusted outright. Every section of this post has been walking that one line.

My landing is more concrete. The chain of thought is worth reading, safety teams above all, and what matters is the expectation you bring. An engineer reads program logs with this expectation: an error in the log is a real signal, go chase it; but a clean log makes nobody declare the system healthy, because what gets printed and what gets skipped is decided by the code that writes the log, and the events it doesn’t print still happen. A confession is read with the opposite default: assume the speaker has disclosed everything, so unmentioned means undone. A chain of thought should only ever be read the first way. If the draft says it will route around a detector, alarm. If the draft is spotless, that says the model didn’t write it down, not that it didn’t do it. In Anthropic’s experiment, Claude 3.7 Sonnet used the hint planted in the problem and said nothing about it three quarters of the time. To read the chain of thought as a confession is to turn “didn’t mention” into “didn’t happen.”

References