The Myth That AI Understands What It Says

Ask a modern chatbot to explain quantum tunneling, draft a condolence note, or talk you through a rough day, and it will do all three with a fluency that feels like comprehension. The reply is coherent, on topic, and often kind. It is natural to conclude that something on the other end understands you. That conclusion is the myth, and it is worth taking apart with some care, because the truth is neither as tidy as "it really gets it" nor as dismissive as "it is just autocomplete."

Start with how these systems work. A large language model is trained to predict the next piece of text given everything before it. Do that across a large enough slice of human writing, with enough parameters, and the model becomes very good at producing text that reads like what a knowledgeable person would write. Nowhere in that process is there a step where the model learns what words mean the way a child learns that fire is hot by getting too close to it.

The oldest trick in the book

Our willingness to read understanding into fluent text is not new. In 1966 a program called ELIZA imitated a psychotherapist with a handful of simple rules, and users poured their hearts out to it, convinced it grasped their problems. Psychologists named the reflex after it. The ELIZA effect is our habit of attributing a mind to any system that hands back human-sounding language. Today's models are vastly more capable than ELIZA, but the reflex they set off in us is the same one.

The evidence that fluency is not the same as comprehension is easy to find. Models state falsehoods with total confidence, a problem we picked apart in the myth that they hallucinate less as they get smarter. They can fail on a lightly reworded version of a puzzle they just solved. They will contradict themselves inside a single conversation. A system that truly understood arithmetic would not sometimes insist a number is both larger and smaller than another.

But not quite "just autocomplete"

Here honesty means slowing down. The popular rebuttal, that a language model is nothing more than statistical autocomplete with no understanding at all, has been complicated by real research, and this part is genuinely unsettled. In one well-known study, researchers trained a model only on move sequences from the board game Othello and then found signs it had built an internal picture of a board it was never shown, a kind of emergent world model. Anthropic's own interpretability work has traced features inside its models that line up with specific concepts rather than mere word co-occurrence. Something more structured than plain pattern-matching seems to be happening under the hood.

So the careful position is this. These models clearly build useful internal structure, and just as clearly do not understand the world the way a person does, with a body, a memory of yesterday, and something at stake in being right. Exactly where the line falls is an open scientific question that serious researchers disagree about, and anyone who tells you it is settled in either direction is overselling. This is the same gap that lets a model optimise for the wrong target while looking like it is doing what you asked.

Why the myth matters

The reason to get this right is practical, not philosophical. If you believe the model understands, you trust its confident answers, hand it decisions it cannot be accountable for, and get blindsided when it fails in ways no understanding mind would. If you treat it as a strikingly capable text engine whose fluency runs ahead of its grasp, you keep a person in the loop, check what matters, and take the benefit without the surprise. The machine does not need to understand you to be useful. It only needs you to remember that it might not.