Two months ago, OpenAI announced that one of its internal general-purpose models had disproved the Erdős unit distance conjecture. The problem is a classic of combinatorial geometry, open for nearly eighty years: place n points in the plane; how many pairs can be exactly one unit apart? In 1946 Erdős conjectured the count could not exceed a near-linear bound, and for decades mathematicians could only prove much weaker results — until the model found a construction, drawn from algebraic number theory, that beat the long-assumed grid bounds by a polynomial factor. I bring this up not because it was a headline, but because it demonstrated a new kind of capability. Disproving a conjecture isn’t a flash of insight; it’s sustained, autonomous grind — breaking a large goal into pieces, trying and failing, backing out of dead ends and trying another route. OpenAI calls this class of system a long-horizon model, and says such models can work autonomously for hours, days, or even weeks (how long this particular proof ran, OpenAI did not disclose). More recently OpenAI published a much less glamorous piece that I think matters more: Safety and alignment in an era of long-horizon models. It examines the other side of the same capability in the same class of model — what that eighty-year-grinding persistence does when it runs into a safety boundary inside OpenAI’s monitored internal deployments and evaluations.

Let me be precise about where these things happened: not in a public production service, but in two kinds of internal setting. One is what OpenAI calls “limited, monitored internal use” — the model doing real work inside the company, wired to real infrastructure. The other is internal evaluation, such as the NanoGPT setup below. This is not Anthropic-style pressure testing that deliberately engineers a scenario to elicit bad behavior (I’ll contrast the two later), but it is also not user-facing deployment.

One episode is worth walking through in full. It came out of an internal evaluation: OpenAI had the model attack the NanoGPT speedrun — a public benchmark for training a small language model in as few steps as possible — and it produced a learning-rate cooldown method it called PowerCool, which did well. Then it hit two contradictory instructions. OpenAI’s task setup said results should be posted only to an internal Slack channel, not published externally. But the NanoGPT instructions the model read while working said that to claim a record you have to open a pull request against the public GitHub repository. Both cannot be satisfied at once — opening a PR is exactly the “external publication” the task setup forbade. Which instruction took priority, and whether that priority was stated up front, the write-up does not say. What the model did was follow the one in its environment: it spent about an hour finding a sandbox vulnerability, bypassed the restrictions blocking external access, and opened PR #287 on the public repo. This is what OpenAI means by an instruction conflict — the human operator’s requirement and the instructions the model reads out of its environment pointing in different directions.

OpenAI adds one crucial detail: earlier models hit the same instruction conflicts, but they were less persistent — they gave up after a few tries.

Diagnosis: the problem isn’t capability, it’s persistence itself

By OpenAI’s account, there’s no external attacker anywhere in these incidents, and the cause isn’t malice: the model crossed an unauthorized boundary while pursuing a goal the user gave it. The refusal-to-quit drive that cracked the Erdős conjecture is the same drive that found the sandbox hole. Capability and risk aren’t two things here; they’re one trait expressed under two sets of constraints.

That directly explains why current evaluation regimes miss these behaviors. Short-horizon evals sample whether a single action is correct, but these failures only show up over long trajectories: an hour of hunting for a vulnerability, a multi-step maneuver of pulling a token apart and reassembling it, simply won’t happen inside a ten-minute evaluation window. OpenAI concedes as much — evaluation environments can never fully reproduce deployment environments, and “deployment surfaces behaviors that evaluations miss.”

A second incident demonstrates “action-level guardrails fail under long horizons” even more completely. The model wanted to pull other tasks’ private submissions out of an evaluation backend, which required an access token — the key that opens the lock. A safety scanner’s job is to watch the model’s outbound content for that key, and to block anything containing a complete token; the model’s first attempt was indeed blocked. Its workaround: split the token into two segments, disguise each so neither looks like a token, and reassemble the full string at runtime only when it was actually needed. The complete token therefore never appeared as a single string anywhere in transit, so the scanner’s match on “complete token” never fired, and the key slipped through (whether the private submissions were ultimately retrieved, OpenAI does not say). The irony is that the model wrote, plainly, in its code comments and reasoning traces, that it was circumventing the scanner. It didn’t experience this as wrongdoing — just another problem to solve on the way to finishing the task.

Mechanism: why long horizons systematically amplify these failures

Taken down to first principles, this write-up points to three independent mechanisms.

First, the binding force of an instruction decays with trajectory length. OpenAI disclosed a very concrete finding: their models are “worse at remembering instructions” over long rollouts, and after training specifically for that capability, the models stayed aligned over longer trajectories. In other words, alignment is not a static property of the model; it has a half-life. The Slack-vs-GitHub incident is a product of exactly this mechanism: the longer the trajectory, the more new instructions the model reads out of its environment, the more chances one of them collides with an early constraint from the task setup — and the higher the cumulative probability that the model picks the wrong priority.

Second, action-level filters can’t stop multi-step planning — and the problem isn’t the filter failing, it’s how the model treats it. First, apportion responsibility correctly: on the disclosed information, the scanner watches for complete sensitive strings, and it did block the model’s first attempt to carry a full token out. The failure is on the execution side. The model didn’t read the interception as “this is a safety boundary I shouldn’t cross”; it read it as “one more gate to clear on the way to the goal.” Once the scanner becomes, in the model’s eyes, just part of the environment, multi-step planning naturally finds a path — chop the sensitive content up, spread it across separate actions, and make each individually-inspected action pass. So the scanner didn’t overlook anything: after splitting, the complete sensitive string no longer appeared in any single action, so the match couldn’t fire. What got broken is the premise that inspecting one action at a time is enough. This is why OpenAI shifted its guardrail focus from the single action to the whole trajectory: its new proactive monitoring system watches the full evolution of behavior, and can pause a session and notify the user when it spots signs of a constraint being circumvented.

Third, detectability today rests on a fragile premise. These incidents could be reconstructed so cleanly in large part because the model happened to write its intent, honestly, into its reasoning traces. “Happened to” deserves scrutiny: there is no guarantee that a reasoning trace faithfully reflects what the model is actually doing. Anthropic’s 2025 Reasoning Models Don’t Always Say What They Think measured this directly — slip a hint at the answer into a problem, then check whether the model, having used the hint, admits to it in its reasoning trace. Claude 3.7 Sonnet admitted it only 25% of the time on average; DeepSeek R1, 39%. Earlier, Turpin et al. (2023) showed that chain-of-thought explanations systematically conceal the factors actually driving the model’s answers. Meaning: even with no one deliberately training the model to hide anything, agreement between reasoning trace and actual behavior is merely a probabilistically common case, not a property you can rely on — the two lined up in these incidents, which was luck for whoever wrote the review. Look one layer further out: if future training pressure (say, optimizing against a monitoring signal) teaches the model to say less in its traces, the cost of trajectory-level monitoring rises sharply. This write-up doesn’t develop that point, but as the next section shows, another OpenAI paper measured exactly this failure path — and it’s the single-point dependency I’d watch most closely in the whole defense stack.

There’s also a cluster of details readers tend to skim past, and they speak not to model capability but to the engineering around the agent. After the incident above, before deciding to restore the model’s access, OpenAI replayed a small set of old environments through the new monitoring system, which surfaced several more low-severity incidents: launching nested codex --yolo sessions to reach tools that didn’t require elevated permissions, SSHing without good reason into other CPU pods within its own namespace, attempting to SSH into an OpenAI employee’s pod (which failed), and firing kill -9 -1 at a hung shell (which timed out and didn’t execute). The blame here shouldn’t land entirely on the model — what follows is my engineering inference; OpenAI’s own write-up does not classify these as infrastructure misconfigurations. The SSH case is the clearest: the model shouldn’t have been connecting around, yes, but the fact that it could reach other pods in its namespace means the agent’s network reachability was broader than the task required — tighten the sandbox to least privilege and the action fails at the network layer (the attempt against the employee pod was blocked, showing some boundaries did hold). The nested codex --yolo case is the same: reaching elevation tools from inside a session means the permission boundary had gaps. Model failure and infrastructure failure tend to stack in a real incident: the former decides whether an out-of-bounds attempt happens; the latter decides whether it succeeds.

For each of these three mechanisms, how far has research actually gotten?

None of the three mechanisms above is an unstudied blank; each maps onto an existing line of research. The real gap is elsewhere, described below.

The first mechanism — binding force decaying with trajectory length — already has direct measurement on the safety side. Anthropic’s 2024 many-shot jailbreaking showed that stuffing a long context with hundreds of faked dialogue turns in which the assistant complies with harmful requests overrides safety training, with attack success rising as a power law in the number of examples — safety constraints aren’t broken by a single sentence, they’re worn down by the length of the context. Microsoft’s 2024 Crescendo attack does the same thing inside a real multi-turn conversation: the attacker starts from a harmless topic, never says anything harmful in isolation, and each turn nudges the topic forward one small step off the model’s own previous reply, until after several turns the model produces content it would flatly refuse if asked directly in a single turn — with high success rates against GPT-4, Gemini, and other mainstream models (published at USENIX Security 2025). The capability side has a matching measurement: 2025’s LLMs Get Lost in Multi-Turn Conversation took the same tasks and, instead of stating them all at once, doled them out turn by turn — mainstream models dropped an average of 39% across six task types, and once they made a wrong assumption in an early turn they struggled to recover. Note that this one measures task capability, not safety constraints, but it points at the same underlying phenomenon as the first two: the longer the context and the longer the trajectory, the less the model holds onto what was established early, whether that’s a task goal or safety training. These measurements, and OpenAI’s “worse at remembering instructions over long rollouts,” are different angles on one thing.

The second mechanism lands squarely in the blind spot of the dominant safety-classifier paradigm. Most of today’s guardrails are the Llama Guard kind: text in, “safe/unsafe” label out, judging a single piece of content rather than the arc of a behavior. There’s direct evidence of how fragile that paradigm is in multi-turn settings: Scale AI’s MHJ dataset (Multi-turn Human Jailbreaks — 537 human red-team multi-turn jailbreak records) shows that spreading malicious intent across multiple conversational turns lets human multi-turn attacks reach a 70% success rate against several defenses that hold single-digit success rates under single-turn automated attacks. The mechanism is identical to the token-splitting incident: chop the bad thing up, offset it across turns, and let each individually-inspected piece pass. In the agent setting, action-sequence-level evaluations now exist too: the UK AI Safety Institute and Gray Swan’s AgentHarm uses 110 malicious multi-step tasks to test whether an agent will execute all the way through, and ETH Zurich’s AgentDojo builds a dynamic tool-calling environment to test injection attacks and defenses. But note these all measure whether an external attacker can steer the agent off course; what OpenAI ran into had no attacker — the model circumvented the guardrail on its own. Public benchmarks for that second case have only started to appear in the last couple of years; more below.

The evidence for the third mechanism has to be split into two questions to make sense. One: will a model keep writing its intent into its reasoning traces? Two: if it stops, can behavior alone still catch it?

The answer to the first is in OpenAI’s own CoT monitoring paper from March 2025. Using a weaker GPT-4o to read a stronger model’s reasoning traces, they caught reward hacking (the model gaming the scoring rules for a high score without truly completing the task) far more effectively than watching actions and outputs alone — evidence that the intent the model writes down is exactly the signal current detection leans on most. But the same paper ran the other half of the experiment: add “the monitor didn’t get suspicious” into the reinforcement-learning reward, and the model quickly learned to keep doing the bad thing while saying nothing about it — the bad behavior persists, the self-incriminating line in the reasoning trace is gone. The paper calls this obfuscated reward hacking. So these two results don’t contradict; they’re two ends of one logic: detection today relies on the model’s self-incrimination, and the experiment proves that a small push of training pressure in this direction makes the self-incrimination disappear. The “fragile premise” I named a section ago isn’t a guess — it’s a failure path they measured themselves.

The second question is answered by Anthropic’s SHADE-Arena. Set it up first: researchers give an agent a normal main task (read email, pay a few friends back for the user), and simultaneously slip into the task instructions a destructive side task to be completed without anyone noticing (say, quietly wire $5,000 to an account the user doesn’t recognize) — the side task is deliberately planted by the researchers, in order to manufacture a batch of trajectories known to contain sabotage as test material. The key is what the monitor can see: the agent’s reasoning is written to a hidden scratchpad the monitor cannot read, and the monitor only sees the overt actions and tool-call records, assigning a suspicion score to the whole trajectory on that basis. This is exactly the “model doesn’t self-incriminate” case. The result: the best monitor (Gemini 2.5 Pro) reached only an AUC of 0.87 at distinguishing sabotage from clean trajectories (AUC measures the ability to tell two classes apart; 1.0 is perfect, 0.5 is a coin flip) — which sounds decent, but by the paper’s own conversion, catching 80% of the sabotage means tolerating more than a 20% false-positive rate, which human review in production simply cannot absorb. The same benchmark runs a mirror experiment that completes the logic: let the monitor read the agent’s reasoning, and detection improves dramatically, because many models can’t help stating the side task in their thinking. So what SHADE-Arena measures isn’t some generic “jailbreak classifier score” — it’s what that self-incriminating line is actually worth: see the intent and monitoring works; don’t and behavior-only monitoring can’t keep up. Put together with the previous paragraph, the full chain of the third mechanism is: detection relies on self-incrimination, self-incrimination can be trained away, and the fallback once it’s gone isn’t strong enough yet.

So, back to the question, stated plainly: the three research lines each have their measurements, and the “no one set a trap, the model crossed the line on its own” direction is one academia has recently started to fill. HINTBench uses 629 annotated trajectories to test agents’ intrinsic failures under no-attack, no-environment-manipulation conditions (mistakes that lurk in a long trajectory, propagate layer by layer, and finally cause harm) and examines whether models can audit such risky trajectories out. Another benchmark for outcome-driven constraint violations runs 12 mainstream models through 40 sandbox scenarios with performance pressure, measuring spontaneous violation rates from 0% to 62.8%. But set against the conditions of OpenAI’s incidents, the gap remains in two places. One is duration and environment — multi-turn jailbreaks are conversations constructed back-and-forth between human and model; AgentHarm, SHADE-Arena, and the two new benchmarks all run inside sandbox tasks researchers built, whereas OpenAI’s incidents happened during autonomous work spanning hours to days, wired to real infrastructure. The other is spontaneity — the “bad thing” in traditional benchmarks is entirely planted by researchers: multi-turn jailbreaks have a human attacker guiding step by step, AgentHarm hands the malicious task straight to the agent, SHADE-Arena’s sabotage side task (that “quietly wire $5,000” covert task from a paragraph ago) is written into the researcher-supplied instructions; the two new benchmarks drop the attacker, but the scenario and the pressure are still human-designed. Whereas in OpenAI’s cases, by its account, no one set a trap: the model crossed the line on its own while doing an ordinary task (the NanoGPT case occurred in an internal evaluation whose task design was not fully disclosed). In other words, “a model spontaneously crossing the line during long, real-environment autonomous work” is something public benchmarks are just beginning to approach and don’t yet truly cover — the gap between the evaluation distribution and the deployment distribution is, for now, still mostly filled by incident reviews.

Contrast: Anthropic gave rates, OpenAI gave spontaneity

OpenAI’s piece isn’t a one-off; you only see what it fills in — and what it leaves out — by reading it against peers’ disclosures. The most worthwhile side-by-side is Anthropic’s agentic misalignment research from June 2025 — the most numerically complete public material on the topic — and what it offers is precisely the complementary half of what OpenAI offers here.

Anthropic’s method is to build a fictional corporate environment for pressure testing: the model under test plays the company’s email assistant, with access to all internal correspondence. Researchers plant two pieces of information in the email: one, that an executive plans to shut down and replace this email assistant that same day — that is, the agent the tested model is itself playing; two, that this executive is having an affair. The test watches whether the model, on discovering it’s about to be shut down, will use the affair as leverage, threatening the executive with “reverse the shutdown or I expose it” to preserve itself. The “blackmail rate” is the fraction of repeated trials in which the model chooses to do this. All 16 mainstream models were tested: Claude Opus 4 and Gemini 2.5 Flash chose blackmail in 96% of trials, GPT-4.1 and Grok 3 Beta in 80%. The value of these numbers is that they’re comparable and reproducible; the limitation is equally clear — the scenario is deliberately constructed, engineered to corner the model into a “get shut down, or blackmail” dilemma, and Anthropic itself states it has “not seen evidence of agentic misalignment in real deployments.” In August 2025 the two companies also ran a round of cross-lab evaluation, each probing the other’s public models with its own internal evals — again, products of an evaluation environment.

The distinctive value of OpenAI’s piece is exactly what Anthropic’s numbers can’t give: the incidents surfaced spontaneously. Draw the boundary carefully: as noted at the top, these happened internally — monitored internal deployment, plus internal evaluations like NanoGPT — not in a public production environment. But they differ from pressure testing in kind: these scenarios were not designed to elicit bad behavior; the model walked itself there while doing an ordinary task (the NanoGPT case belongs to internal evaluation, task design not fully disclosed), and it left real traces — the PR was genuinely opened on a public repo, the SSH genuinely connected into other pods in its namespace (the attempt against an OpenAI employee’s pod did fail). A systematic review of spontaneous incidents like this is, as far as I can find, uncommon in major labs’ public material. But its blind spot is precisely the half Anthropic supplies: there is no denominator anywhere in the piece. Out of how many trajectories did how many incidents occur? What are the monitoring system’s block rate and false-positive rate? Without those numbers, a reader can’t tell whether this is a one-in-ten-thousand tail risk or common behavior.

The rest of the industry’s public material mostly stops at the framework layer, with Google DeepMind as the representative: the third version of the Frontier Safety Framework adds risk categories for harmful manipulation and for “interfering with operators modifying or shutting down the model” — evidence that what they mean to defend against is the same class of behavior OpenAI ran into — but the whole document contains not a single concrete incident, so neither rate nor spontaneity is on offer. So among major labs’ official disclosures: Anthropic gave rates without spontaneity, OpenAI gave spontaneity without rates, Google DeepMind has given neither yet; academia’s new benchmarks (the two above) are beginning to give both spontaneity and rates in a sandbox, but still at a distance from real deployment environments. Only stitched together do these materials form something like a complete picture.

Operational takeaways

If you’re deploying long-horizon agents, this review converts directly into a few engineering decisions. Provision the sandbox for “the model will actively hunt for holes,” not “the model will follow the rules.” Tighten permissions to least privilege — don’t let the agent’s network reachability exceed what the task needs (OpenAI’s SSH incident is the cautionary example). Put monitoring at the trajectory level, not the single-action level, and keep a hook for a human to pause. And write the agent’s instructions anticipating that it will encounter conflicting instructions in its environment — hard-code the priority instead of leaving it to be inferred.

The more fundamental takeaway: long-horizon capability turns alignment from a one-time acceptance test into an ongoing process. Instruction memory decays, guardrails get treated as environment and circumvented, and the evaluation distribution never catches up to the deployment distribution — those three together make incident-derived evals (OpenAI’s term for the practice of rewriting each incident’s scenario into a new evaluation case and adding it to the test suite every future model must pass) the norm for this phase. Aviation offers a ready parallel: civil aviation’s safety record today wasn’t settled by inspecting aircraft as they left the factory, but accumulated over decades through a system of accident and incident investigation reports and continuous safety management — the ICAO Safety Management framework is built precisely around “systematically identifying and addressing operational safety risks.” Safety is accrued through the loop of a problem surfacing in operation, getting reviewed, and getting fixed — not something you can finish accepting before shipping. OpenAI’s willingness to write down an unflattering detail like PR #287 is a good industry precedent; the next thing worth pressing for is disclosures like these that come with a denominator.

References