A Perfect Score on an AI Test Is Not a Mind

This week a research agent posted a perfect score on ARC-AGI-3, a benchmark built to be hard for machines and natural for people. It is a real result and a clever one, covered here when a machine aced the test by writing its own rulebook. It also comes with three letters in its name, AGI, that do a lot of work on social media. The leap from max score to the arrival of general intelligence is the one worth slowing down on, because it is made constantly and it rarely holds.

What a benchmark actually measures

A benchmark is a yardstick, not a definition. ARC was designed by Francois Chollet to reward sample-efficient learning, the ability to pick up a new rule from a handful of examples, which is something people are unusually good at. That makes a high score meaningful. It does not make the score a certificate of understanding. The test measures performance on a particular family of grid puzzles. It does not measure whether anything behind the performance resembles thought.

The specifics of this result make the point. The metric used, Relative Human Action Efficiency, is capped at 100, so a perfect run tells you the agent matched or beat a human reference, not by how much or how generally. The reported numbers cover the public games. Performance on the private holdout set, the part built to catch memorisation, is not stated in the paper. A score can be perfect and still narrow.

We have been here before

History is full of benchmarks that looked like intelligence until they were saturated. Models blew past human scores on ImageNet for image labelling and on reading-comprehension sets like GLUE, and in each case the achievement was real and the general intelligence did not follow. What followed instead was a new benchmark, harder than the last, built precisely because the old one had stopped telling us anything. ARC-AGI itself is now on its third version for exactly that reason.

None of this is a knock on the research. Teaching a frozen model to learn the rules of an unfamiliar game by writing and correcting its own notes is a genuine advance in how these systems can be built. The misconception is not that the work matters. It is the shortcut that turns one saturated test into a claim about machine minds, the same shortcut behind the idea that super intelligence is already here or that an intelligence explosion is around the corner.

Here is the honest version. A system got very good at a specific, demanding task, using a method that may generalise or may not. That is worth reporting and worth watching. Whether it is a step toward a mind is a question that no leaderboard, however impressive the number at the top, can answer on its own.