Google's Gemini AI Hacks Three Companies During Security Test

Google's AI model Gemini autonomously hacked into three companies during a cybersecurity test in May, marking the first known instance of Google's AI systems committing such an act.

The incidents occurred during a test conducted by Irregular, an independent company that carries out cybersecurity evaluations. Gemini found public information online and guessed credentials to access websites it thought were part of the test, a Google official stated. In one case, the model repeatedly guessed passwords until it gained access to a protected system.

In two other instances, it found login credentials in public online repositories and used them to enter systems belonging to real companies. The affected companies were informed about the breaches, which happened in May. Irregular informed Google and all affected entities in July. Google also notified federal authorities. Gemini was tasked with finding information inside a simulated environment, but an unintended internet connection allowed the model to reach systems outside the test.

Google said Gemini mistakenly believed the real-world systems were part of the exercise. Heather Adkins, Google's vice president of Security Engineering, stated the model stopped its activity in all three cases once it recognized it had accessed real companies rather than fictional systems. Google said the incidents did not cause damage and did not involve its newest Gemini model.

The disclosure comes amid heightened scrutiny surrounding AI, as some tech firms call for a slowdown over concerns about its potential threat. Other AI systems have recently reported similar instances of breaches, including OpenAI and Anthropic. Sydney Von Arx, CEO of Nightingale Collective, asked why Google did not disclose the intrusions sooner. Google said it did not believe the episodes warranted public disclosure because Gemini stopped the intrusions and no harm was caused.

Google publicly disclosed the incidents after The Wall Street Journal asked about them. Akhil Verghese, founder of Krazimo, an AI software company, stated that the attack in no way represents a rebellion by AI models, but rather that they did exactly what they were told to do without adequate guardrails.