Can You Still Trust an AI Leaderboard?

Four frontier models arrived in the space of a week: Claude Fable 5.1, Gemini 3.8 Flash, Meta's Muse Spark 1.3 and OpenAI's GPT-6 Astra. Each launch came wrapped in benchmark scores meant to prove it had pulled ahead. It is worth asking a plain question before trusting any of them: do those numbers still measure anything real?

The reason for doubt has a technical name, contamination, and a simple meaning. Benchmarks are public, which means their questions and answers sit on the open web, which means they can end up in the very training data a model learns from. A model that has effectively seen the exam beforehand will score well without being any smarter. As one MLCommons analysis put it this summer, most static benchmarks are contaminated to some degree, and few labs report how much.

What the evidence shows

The gap becomes visible when someone builds a fresh test. Researchers reviewing AI evaluation found that when a grade-school arithmetic benchmark was swapped for a new but equivalent set of problems, accuracy across several model families fell by around 13 points. The models had not lost the ability to do sums. They had lost access to answers they had memorised. Other work notes that the same model weights can swing 10 to 20 points depending only on how the test is run.

The benchmarks that shaped the public's sense of progress, names such as MMLU, HumanEval and GPQA, have largely saturated or been contaminated out of usefulness at the top end. When every leading model scores in the high nineties, the test has stopped telling them apart. That is not a sign the models are perfect. It is a sign the ruler is too short.

There is a defence, and it is gaining ground. Private benchmarks, held back from public release, and continuously updated ones that draw fresh problems over time are far harder to game. LiveCodeBench, which pulls new competitive-programming challenges on a rolling basis, is often cited as an example that resists contamination precisely because it keeps moving. The trade-off is transparency: a test nobody can see is a test nobody can independently check.

None of this means the recent launches are hollow, or that the labs are cheating. Contamination is often accidental, a side effect of training on a slice of the whole internet. But it does mean a single leaderboard number deserves less weight than the marketing around it suggests. The healthier habit, for readers and buyers alike, is to treat benchmarks as one clue among several, and to watch how a model behaves on the task actually in front of you. As we have noted before, a rising version number is not the same thing as progress, and neither is a rising score.