The Soft Foundations Under AI Doomsday Arguments

Predictions that artificial intelligence will wipe out humanity have been a fixture of public discourse since at least 2023, when a one-sentence open statement signed by lab leaders compared the technology to nuclear weapons and pandemics. Three years on, the doom warnings have not gone quiet. If anything, they have been amplified by the rise of agentic systems and the steady upward march of model capability. And yet the technical foundations under those warnings are softer than the volume might suggest.

A clutch of papers published over the past eighteen months has been picking the foundations apart. A widely cited 2025 critique argued that artificial general intelligence, the construct at the centre of most extinction scenarios, is built on three contested moves. One is the idea that machine intelligence can scale to arbitrary generality without hitting domain-specific ceilings. The second is anthropomorphism, the unspoken assumption that a sufficiently capable system will develop goals and desires roughly resembling our own. The third is omnipotence, the leap from smarter than us to able to reshape the physical world at will.

None of those moves is technically settled. Researchers at MIT and elsewhere have pointed out that even highly capable systems still face significant social, political, and physical constraints. A model that knows everything about protein folding still needs a wet lab. A system that can write persuasive emails still needs an audience that reads them. AI capability does not translate directly into power, and the path between the two is shaped by laws, supply chains, regulators, and the messy texture of human institutions.

The other problem is the calculation itself. Surveys of researchers consistently find wide variance in p(doom), the probability that AI causes human extinction. A 2025 ArXiv paper looked at why experts disagree so sharply and concluded that the disagreement is not really about technical evidence. It is about deep priors regarding the AGI trajectory, the speed of recursive self-improvement, and what counts as a credible alignment strategy. The numbers are dressed up as probabilities, but underneath they are largely values disguised as forecasts.

None of this is an argument that AI is safe. The real risks are well documented. Model-amplified disinformation. Concentration of economic power. Surveillance creep. Weaponisation of generative tools. Structural displacement in specific occupations. These are concrete problems with sector-specific policy responses. Doomsday framing tends to crowd them out. When the discourse is dominated by extinction-level abstractions, it becomes harder to focus political energy on the AI harms that are already showing up in courts, hiring outcomes, and election cycles.

That is the durable cost of bad foresight. Doomsday scenarios sell. They get retweeted, and they show up in Congressional testimony. But the engineering under them is thinner than the conviction with which they are delivered. Treating extinction as the central concern of AI safety, instead of one possibility among many, has a real opportunity cost. It diverts attention from the present.