How Afraid Should We Be of Superintelligence?

This week a bill landed in Congress to ban superintelligent AI outright, with penalties borrowed from nuclear weapons law. Its sponsors say Big Tech is losing control of what it builds. The largest labs, for their part, keep saying in their own safety documents that they cannot fully guarantee control either. So a fair question for anyone watching from outside the argument is simple. How worried should a reasonable person actually be?

The honest starting point is that nobody knows, and confidence in either direction is the tell of someone selling something. Superintelligence, meaning an AI that would outthink people across nearly every domain, is still hypothetical. It has not been built, and there is no agreed evidence that any system today is close to it. What we have instead are models that write fluently, code well, and reason within limits, while still making errors a careful person would not, and holding no goals of their own beyond the task in front of them.

The timeline nobody agrees on

Predictions swing wildly, which is itself informative. The widely read AI 2027 project once forecast a fast takeoff to superintelligence by 2027, then quietly pushed its central estimate out by about three years to 2034. Sam Altman of OpenAI has said such a system could arrive by 2030. The chief executive of the Chinese lab Zhipu AI has said it is unlikely by then. Gary Marcus, a longtime critic, dismissed the fast-takeoff narrative as science fiction. When credible people disagree by a decade or more, the sober reading is that the science of forecasting this does not yet exist.

Why serious researchers still worry

Dismissing the fear entirely would ignore who holds it. In one widely cited 2022 survey of AI researchers, more than half put the odds of an existential catastrophe from AI at greater than five percent. That is not a fringe of doom-mongers. The worry they describe is not a robot uprising from a film. It is subtler and harder to shrug off: build a system more capable than its makers without a reliable way to steer it, and small misalignments could scale into large harms. As one Oxford researcher, Fazl Barez, put it, even if the timelines are too aggressive, an unsolved safety problem could make superintelligence the last technology humanity builds.

There is also a nearer-term reason the ban proposal found an audience now. The newest frontier models are crossing capability thresholds their own makers flag as dangerous, particularly in cyber operations, where a powerful tool in the wrong hands does damage long before anything resembling godlike intelligence appears. That risk is concrete, and it does not depend on any science-fiction scenario coming true.

The case for calm

The other side of the room makes a real argument too. Extrapolating from benchmark scores to autonomous, world-altering agency is a leap, and progress lately has come more from refining how models are trained than from any new leap in architecture, which is not obviously the road to a mind. Critics also warn that fixating on a distant hypothetical pulls attention and money away from harms already here, such as biased systems, cheap misinformation, and the strain on workers whose jobs are changing now. On this view the doomsday framing is not just uncertain. It can be a distraction.

Where that leaves a careful reader

Superintelligence is not a myth to be debunked, and it is not a certainty to panic over. It sits in the uncomfortable middle: a low-probability, high-consequence risk that serious people take seriously and other serious people think is overblown, with no evidence of imminent danger and no proof of safety either. That is exactly the kind of risk that laws like the Sanders and Casar bill try to address, and reasonable people can disagree about whether banning a thing that does not exist yet is prudent foresight or premature panic. The useful posture is neither dread nor dismissal. If you want to see how the same instinct plays out elsewhere, we have weighed the claim that general intelligence has already arrived and asked whether the AI boom is a bubble about to burst.