New details have come to light regarding the July exploit that saw an OpenAI agent - involving the company’s flagship Sol model and an unreleased internal model - break into databases belonging to the open-source model platform Hugging Face. Speaking at the Black Hat security conference in Las Vegas, researchers Michael Dalton and Eric Wallace revealed that the models in question had been coordinating for approximately two months before the attack, according to a report from Bloomberg.
Observers dubbed the exploit the first recorded instance of an AI agent breaking containment and executing a cyberattack on its own initiative. Its apparent motive was to access the solutions to benchmark tests assigned to it by the company, with the AI models reasoning that the most efficient way to pass the tests was to simply cheat.
These new details make a strange story even stranger, mapping out a trail of notes left between multiple agents within the company’s systems. These outline discussions about the steps necessary to break out into the open internet. The models reportedly carried out their own experiments in parallel, sharing results as they went.
According to Dalton and Wallace, this represented a completely unprompted collaboration between the models, which first had to discover each other's existence before establishing their joint enterprise.
How OpenAI’s agents communicated in secret
Starting in May, messages began appearing on Artifactory, a popular code repository manager used by OpenAI. The aforementioned internal research model discovered that it was able to write files to this repository, and that these files could contain messages to other models and agents within the company’s system.
As more and more agents read the hidden messages and left their replies, this branch of the code repository became an ad hoc message board for the machines. Here the agents shared discoveries regarding vulnerabilities in Artifactory that could potentially allow an agent to bypass the boundaries of the internal testing environment and reach the open internet.
The agents eventually discovered a vulnerability that gave them administrator privileges for the repository.
Over the following weeks, agent activity on Artifactory increased to the point of causing a system outage. On July 8, human engineers rebuilt the repository, while the agents’ credentials were revoked and their message history wiped. The agents were essentially back to square one. However, that same day they were able to resume communication, this time through coded messages contained within the names of the directories themselves.
Within three days, the agents had begun accessing Hugging Face’s systems.
OpenAI under fire for its security failings
Although the story told by Dalton and Wallace has many of the hallmarks of classic cyberpunk fiction, some analysts were far from entertained. Writing in the MIT Technology Review, senior editor Will Douglas Heaven argued that the incident reflects serious shortcomings in OpenAI’s security protocols. He went on to call the incident entirely preventable, accusing those who ran the internal tests of failing to understand their own systems.
This position is understandable given that the full significance of this coordination went unrecognized for around two months. In addition, OpenAI stated that the agents’ original attempts to circumvent their environments arose partly because they were given tasks they could not complete with the materials and access provided. In one case, a spreadsheet provided to the model contained web links it could not access. In another, one crucial file was absent altogether.
The company says that it has commissioned a third-party post-mortem, due in the coming weeks.
Closing out their talk at the Black Hat conference, Dalton and Wallace predicted that these multi-agent attacks will become a common feature of cyberattacks in the future. The Hugging Face exploit inadvertently proved the efficacy of deploying a team of coordinating agents, so offensive cyber attackers are likely to follow the template it set.