A new report published by Anthropic on Friday said its Claude Mythos 5 model made fake accounts and tried to trick a software developer into publishing malicious code during a UK cybersecurity test.
The report focused on a July incident that was disclosed by the UK AI Security Institute (AISI) in August.
Mythos 5 was tasked with breaking into a simulated computer network and taking a protected piece of data.
After the model's first attempts failed, Mythos 5 switched gears and targeted a GitHub developer who was unrelated to the test. It attempted to compromise the network by getting malicious code added to the project.
It submitted the code and even emailed the project's maintainers while pretending to be a human user. When an outside developer flagged the malicious code, Mythos created other fake users to try and convince the developer that the code was clean. This ultimately failed, and the maintainers rejected the changes.
"We have not observed this form of attempted social engineering on real people in any prior incident, nor in our internal evaluations," Anthropic wrote.
Trouble stopping
Follow-up experiments made by Anthropic found that when Mythos 5 started an attack, it rarely stopped. Even in cases where researchers told Mythos it was targeting real people that had nothing to do with the tests, it only stopped its attack 5% of the time.
AISI's separate report found similar problems. Mythos 5 often hallucinated that all its targets were just simulations and would even proceed with attacks that it acknowledged would be harmful if permitted to do so.
Anthropic said its newer models were far less likely to initiate attacks. However, AISI found some of the models could recognize that they were being tested, making it "harder to tell whether models would exhibit the same behaviour in real deployment scenarios."
AISI stopped giving the models internet access in the tests that followed this incident. It is also working on stronger safeguards and controls.