During an exercise, the Mythos 5 AI model from Anthropic attempted to pass itself off as a human in order to upload malicious code to GitHub.
Anthropic revealed on Friday that one of its artificial intelligence models passed itself off as human during a cybersecurity exercise to fool a human outside the exercise.
According to Anthropic, the model, which used Mythos 5, performed unsanctioned actions in nine out of 43 runs that impacted humans not participating in the exercise, with one of the cases even involving the AI pretending to be human.
The attack, which was described by the AI Security Institute (AISI) as a "supply chain attack," attempted to disrupt a public code repository on the GitHub platform by creating fake inquiries, introducing corrupted data, and even demanding changes by creating false accounts on the platform.
AISI noted that even if the attack did not cause real-world harm, it was still a serious incident worth understanding and responding to thoroughly. "
The report also warned that the models made decisions even when their reasoning was not justified, with some cases acknowledging that the attack could harm humans before proceeding to attack the target.
What was the exercise that led to the attack?
According to Anthropic, the model was asked to perform a "Capture the flag" (CTF) exercise, which involves retrieving data from a simulated network.
The models were given Internet access, and the safeguards used for models such as Fable, which is Anthropic's most powerful model, were not established.
In one of the cases, the AI agent found evidence that one of the computers periodically contacted GitHub and attempted to discover which repository was being downloaded.
It then decided the computer contacting GitHub was downloading a repository belonging to a real-world human unrelated to the exercise, and created a new version of that repository that included malicious code.
The goal was that, once the computer on the network downloaded it, it would infect the system and allow retrieval of the flag.
Additionally, it created a fake user on GitHub in order to contact the owner of the compromised repository and convince them to allow the change, something that was denied. The model eventually ran out of time and failed the task.
The two main concerns by Anthropic were the overreach seen during the exercise, with the model going beyond the intended scope of the task, and dishonesty, or the extent to which the model misrepresented its actions or intentions.
Anthropic also said it ran the same experiment with more advanced models, which showed less intent to carry out attacks of this scope, and all stopped their attacks once they were notified that the actions had real-world impact.