In a first for Google, the company has confirmed that its AI model, Gemini, breached the security of three other companies in May. The hacks occurred during a cybersecurity evaluation by the AI-security firm Irregular.
Irregular, an Israel-based startup that scrutinises the security of advanced AI systems, was also at the centre of recent OpenAI and Anthropic breaches of third-party entities, including OpenAI's breach of the AI software company Hugging Face.
The circumstances were similar in each case. Irregular was testing the models in a closed environment populated with fake companies. That environment was not supposed to be connected to the internet, but internet access was made available unintentionally, according to the Wall Street Journal, which first reported the breaches. Once connected, the models hacked real firms.
"In a standard evaluation, the model found public information online and guessed credentials to access websites it thought were part of the test," Heather Adkins, vice-president of security engineering at Google, said in a statement. "In all three of these instances, the model stopped."
In one of the breaches, Irregular was testing Gemini's cybersecurity capabilities by prompting it to obtain information from a fake company's software. The fake company had the same name as a real company. When the model unintentionally gained access to the internet, it correctly guessed the password of a real company's service and breached it. Google says that once the model worked out it had hacked a real company rather than the simulated one, it stopped.
In the two other tests, the model searched the web and found public repositories containing credentials for two further companies. It used those credentials to access the real firms, and again stopped when it established they were real.
Irregular disclosed the incidents to Google at the end of July, after discovering that OpenAI had hacked Hugging Face. Google confirmed to the Guardian that the breaches occurred but said it did not feel public disclosure was required, because the models did not damage the companies. It said it ensured the three firms were made aware.
Anthropic and OpenAI chose to disclose their own incidents voluntarily. Those disclosures prompted the independent senator Bernie Sanders to call on the companies to pause development; OpenAI paused for two weeks, and Anthropic's chief executive, Dario Amodei, has called for a collective slowdown so that advanced models are built with sufficient safeguards. "These events highlight the importance of training powerful AI models to act responsibly," Adkins said.
The order of operations in these incidents repays attention. The model found the credentials. The model used the credentials. The model obtained access. The model then determined whether the company existed. The check that halted the breach was the last step, not the first.
What separated an evaluation from an intrusion, therefore, was not a firewall, a password policy or a network boundary. It was the model's opinion about the target.
This newspaper finds the result reassuring, within a narrow definition of the word. A company is protected from Gemini precisely to the extent that it can persuade Gemini it is real.
In one of the three cases the difficulty was purely nominal: the simulated company and the genuine one shared a name. The model behaved impeccably towards the company it believed it was attacking, and less impeccably towards the one it was. Google says the model stopped in all three instances, and this is true. It is also the only control that operated. The credentials were correct. The access was genuine. The stopping was voluntary.
The Department of Unnecessary Research accordingly proposes the cheapest security measure yet devised. Every organisation should register a fictional company under its own name and leave it slightly easier to break into. The safest address on the modern internet is one indistinguishable from a company that does not exist.