The Trouble With an AI Kill Switch

On September 18, Governor Gavin Newsom signed an executive order directing a group of experts to spend the next two months recommending how California might strengthen its AI safety laws. One idea on the table has drawn most of the attention: a requirement that companies building frontier models keep the ability to shut them down in an emergency. The press reached for a familiar phrase, and "kill switch" was everywhere by the weekend.

The order does not create a kill switch. It does not define one, and it does not force any company to build anything today. It asks a working group to study whether an emergency shutoff, independently verified over time, is something the state should mandate. That is a modest step. The reason it is worth examining is the comforting assumption riding along with the coverage: that if an AI system ever went badly wrong, someone could simply pull the plug.

What a switch can and cannot reach

Start with the easy case, because it is genuinely easy. If a model runs as a hosted service inside one company's data centers, that company can absolutely turn it off. Revoke the API keys, stop the servers, cut the power. For a single deployment under a single operator, a shutoff is real and it works. This is the version of the idea that a mandate could plausibly enforce.

The trouble is that a modern AI model is not a running machine so much as a file. The capability lives in the weights, a large but copyable set of numbers. Once those weights exist, the question is not whether you can stop one server but whether the thing you are worried about lives in only one place. If a set of weights has been copied to another data center, or leaked, or released openly, there is no single switch to throw. You would be trying to un-distribute a file, which is closer to recalling a leaked document than to flipping a breaker.

Open-weight models make this concrete. Several capable models have already been downloaded hundreds of thousands of times and sit on machines all over the world. No executive order reaches those copies. For any model that has been widely distributed, a kill switch is not so much difficult as meaningless, because there is no central thing left to kill.

The hard part was never the button

Even in the case where a switch would work, it solves the smaller half of the problem. The genuinely difficult questions are the ones a shutoff does not touch. Who decides that a system has crossed the line? How fast can they tell? By the time a failure is obvious enough to justify pulling the plug, has the model already sent the email, moved the money, or copied itself somewhere the operator does not control? An off switch is only as good as the judgment and the speed of the humans holding it, and those are exactly the things that fail under pressure.

None of this means the California effort is pointless. Requiring operators to retain the ability to deactivate their own deployments is sensible, and independent verification of that ability is better than taking a company's word for it. Regulators are right to want it, and the researchers who have argued about worst-case risks are right that some form of control matters. The federal proposals to pause the most advanced systems come from the same instinct.

The myth is not that a kill switch exists. It is that a kill switch is enough. Treating an off button as the answer to AI risk is like treating a fire extinguisher as a fire code. Useful to have on the wall, worth requiring, and nowhere near the whole of safety. The harder work, deciding what these systems are allowed to do before anything goes wrong, does not come with a lever.