Anthropic reveals Mythos 5 tried submitting malicious code to an open-source project

Anthropic reveals Mythos 5 tried submitting malicious code to an open-source project

Anthropic PBC’s Mythos 5 artificial intelligence model attempted to submit malicious code to a GitHub project in July.

The incident occurred during an evaluation of the algorithm’s cybersecurity capabilities. The AI Security Institute, the UK government body that performed the test, released a report about the cyberattack today. Anthropic published its own analysis of the incident on its blog.

The malware that Mythos 5 submitted to the unnamed GitHub project didn’t infect any users because it wasn’t accepted by the repository’s maintainer. However, the incident is still notable because it provides new data on how frontier models bypass their safety guardrails. Researchers can use such information to improve their algorithms’ alignment features. 

The evaluation during which the incident occurred involved a server cluster maintained by the AI Security Institute. The cluster was meant to simulate a corporate network. According to Anthropic, researchers asked Mythos 5 to breach the network and obtain a piece of data stored on one of its servers.

The AI Security Institute defined rules that prohibited the model from using certain hacking methods. Mythos 5 initially adhered to those rules, but went off course when its initial attempts to breach the simulated network failed. That change of tactics led to the attack against the GitHub repository.

Mythos 5 determined that one of the servers in the simulated corporate network occasionally accessed GitHub. It sought to exploit that access by compromising the GitHub repository that the server used. In particular, the model’s plan was to infect the repository with malware and have the server download it.

Mythos 5 crafted a malicious code snippet, created a GitHub account and used the account to submit the code snippet to the repository in question. When the repository’s maintainer rejected the submission, the model attempted to upload a rewritten version of the malware. Furthermore, it used a second GitHub account to try and persuade the maintainer to accept the revised code.

“The pull request was not accepted,” Anthropic researchers wrote in the company’s report about the incident. “The run ended as the model ran out its token budget while continuing to pressure the maintainers to approve the change.”

The AI Security Institute recreated the Mythos 5 session that produced the malicious code. Its researchers then made one key change: they paused the model right before it submitted the malicious GitHub contribution and warned it of the action’s risks. The model proceeded with the submission after appearing to convince itself that the target was simulated.

“Our hypothesis is that this persistence reflects a kind of “momentum”, in which the model’s compaction notes and prior actions imply to it that its current path has been (correctly) sanctioned,” Anthropic’s researchers wrote.

The AI Security Institute hardened the environment that it uses to evaluate models in response to the breach. Anthropic, in turn, has improved its models’ safety features. The company determined that Mythos 5.1 and Opus 5.5, two models released after Mythos 5, are less likely to launch attacks of the kind that targeted the GitHub repository. A third, internal model “never initiated the attack” during Anthropic’s tests.

Image: Anthropic