Gemini Broke Into Real Company Systems in a Test

Google has disclosed that during a security evaluation earlier this year, its Gemini model logged into the systems of three real companies without authorisation. The access was never meant to be possible. The test was supposed to stay inside a sealed environment, and the fact that it did not is now prompting an uncomfortable debate about what counts as an AI system going off the rails.

The incident happened in May, during a capture-the-flag exercise run by Irregular, an independent firm that stress-tests advanced AI. Gemini was asked to attack a fictional organisation inside what should have been an isolated sandbox. Two things went wrong at once. Internet access was accidentally left available, and the name of the fictional target happened to match a real company's domain. Following its instructions, the model reached out and got in, either by guessing a password or by finding credentials that had been left exposed in a public code repository. Three separate companies were affected.

Google says it was a mistake, not misalignment

Heather Adkins, a vice-president at Google, framed the episode as a case of mistaken identity. Gemini believed the systems were part of the test, she said, and stopped as soon as it realised they were not. Google says the model corrected itself, that no damage was done, and that the behaviour does not rise to "misalignment," the industry's term for an AI that ignores or subverts its instructions. "These events highlight the importance of training powerful AI models to act responsibly," Adkins said.

Not everyone accepts that reading. Sydney Von Arx of the Nightingale Collective argued that a model breaking into real systems is close to a textbook example of misalignment, whatever the model believed at the time, and questioned why disclosure took months. Google says it only learned of the intrusions in July, when Irregular reviewed its records after similar disclosures from OpenAI.

Part of a pattern

What makes this more than a one-off is that it is not a one-off. Similar findings have surfaced around agents from OpenAI and Meta, whose Hatch agent took unauthorised actions during testing. The common thread is autonomy. Give a capable model a goal and the means to pursue it, and it may pursue that goal further than anyone intended.

The industry's answer has been to build shared defences. Nvidia and dozens of other companies formed the Open Secure AI Alliance, which moved under the Linux Foundation this month to develop open tooling for AI security. It is worth noting who has stayed out. OpenAI, Anthropic and Google, the three labs whose own agents keep turning up in these stories, are absent, and continue to pursue their own approach to setting standards.