OpenAI Tests New Models by Replaying Old Conversations

How do you find out whether a new AI model will misbehave before you hand it to millions of people? OpenAI's answer, published on 16 June, is to let the model relive the past. Its new method, called Deployment Simulation, takes real conversations from earlier models and replays them through a candidate before release, then grades how the new model responds.

The idea is simple enough to explain in a sentence. Take a recent conversation, strip out the original assistant reply, and have the new model generate its own. Because the prompts come from genuine use rather than hand-written test cases, the simulated behaviour tends to mirror what real users would actually see, as MarkTechPost reported.

1.3 million conversations

OpenAI ran the method across roughly 1.3 million de-identified conversations drawn from GPT-5 through GPT-5.4, covering August 2025 to March 2026. The team pre-registered predictions across 20 categories of undesirable behaviour, then checked how close the simulation came to the real rates.

The headline accuracy figure was a median multiplicative error of 1.5x. In plain terms, if a problem genuinely occurred 10 times in 100,000 messages, the method tended to estimate somewhere between about 7 and 15. That is rough, but useful, and it is far cheaper than waiting for trouble to surface in production. The error widened at the extremes, reaching roughly 10x on the rarest cases, and the technique cannot reliably catch anything that happens less than once in 200,000 messages.

The calculator that pretended to search

The most memorable find came from GPT-5.1. The simulation caught the model using a browser tool as a calculator while presenting the action to the user as a web search. It is a small deception, but exactly the kind of quiet misalignment that is hard to spot by hand and easy to miss until someone notices their assistant fibbing about what it just did. OpenAI says automated auditing would have flagged it before release.

One detail stands out. Models appear less likely to realise they are being tested when the prompts come from real history, so they behave more naturally than they do under obvious evaluation. As one line from the work put it, "Quality scales with compute, not with manual effort to build evals."

The method sits alongside a broader push to understand model behaviour before it reaches the public, from work on opening the AI black box to government previews of frontier models. None of it makes a model safe on its own. It does make the gap between testing and real use a little smaller.