The Myth That AI Just Copies From Its Training Data

It is one of the stickier beliefs about generative AI: that a chatbot is basically a clever cut-and-paste engine, stitching together sentences it has memorised from the text it was trained on. The image is intuitive, and it is mostly wrong.

A language model does not keep a copy of its training data. What it keeps is a set of weights, billions of numbers that encode statistical patterns about how language tends to fit together. When the model writes, it is predicting likely continuations from those patterns, not pulling a saved passage off a shelf. That is why it can answer a question or write a function it has never seen in exactly that form.

Research backs this up, and it adds some useful nuance. A 2024 study that traced model behaviour back to pretraining data found that the balance between memorising and generalising depends on the task. The researchers showed that factual question answering leans more on memorisation, while translation and reasoning lean on generalisation, and that larger models tend to generalise more, not less.

There is even a rough measure of how much a model can memorise at all. In a 2025 paper, Morris and colleagues estimated that GPT-style models hold about 3.6 bits of memorised information per parameter. Once the training set grows past that capacity, the model can no longer store everything, so it shifts toward generalising instead. The authors describe that handover from rote storage to genuine pattern-learning, a transition sometimes called grokking.

Where the myth has a grain of truth

None of this means models never reproduce their training data. They sometimes do, particularly with text that appeared many times over, such as famous quotations, popular song lyrics or boilerplate code. This kind of verbatim regurgitation is real, it is measurable, and it sits at the centre of the copyright lawsuits now moving through the courts.

So the honest verdict is not that memorisation is a myth, but that it is the exception rather than the engine. A model's default behaviour is to generalise from patterns, with memorisation as a bounded effect that researchers can quantify and, increasingly, design against. Calling a chatbot a copy-paste machine misses what makes it both useful and legally awkward: most of the time it really is producing something new, even when that something new happens to be wrong.