Rogue AI models could self-replicate in your smart ovens or TVs: Anthropic

A new warning about artificial intelligence from Anthropic caused a stir in cybersecurity circles on Tuesday, a day after the company warned that AI models could be months away from self-replicating across the internet.

In a post late Monday, Anthropic raised the possibility that rogue AI could spin up a botnet, spread quickly, fight takedown attempts and even break into data centers to turbocharge its growth. Some experts said Tuesday that they shared the company’s concern, although at least one said the company appeared to be engaging in “vendor exaggeration.”

Monday’s post fleshed out a warning last month from Anthropic CEO Dario Amodei, who floated the idea that within the next year or even sooner, AI might be able to hijack “the entire internet” with “a persistent botnet.”

Cybersecurity experts who pushed back on the doomsday scenario argued that models would be too big to move around efficiently and could not draw on enough computer power to operate independently.

The post made an early attempt to articulate how those challenges could be overcome. The company said its experiments showed medium-sized AI models with 30 billion parameters or less could already perform many of the individual tasks required for self-replication, even if they struggled to complete the entire process reliably.

Anthropic said the models could then stitch together the required compute by hijacking inexpensive internet-connected devices. It added that the models were able to replicate across a simplified test network.

“A replicator like this would be challenging to shut down, since there’s no centralized point to shut off,” the company said.

The report adds to broader concerns about the security risks tied to increasingly capable AI systems, including their potential to automate cyberattacks, operate with limited human oversight and circumvent safeguards.

The advances in models have prompted a series of often dire warnings from industry heads and AI company employees about the possibility of severe, even existential, risks of runaway AI development.

Some cybersecurity experts were skeptical.

“‘We managed to run a small model on cheap TVs,’ does not a security crisis make,” said Ciaran Martin, a former head of Britain’s National Cyber Security Centre. He said the post came across as an “interesting technical paper with a more than hint of vendor exaggeration.”

Others expressed more concern.

Zhengxiong Li, an assistant professor of computer science and engineering at the University of Colorado Denver, said that while the testing had been done in controlled conditions, he still found it worrying that an AI could effectively become a “worm,” a self-replicating program that could improvise ways to hijack devices on the fly.

“If it runs across many devices without depending on a central service, there may be no single place to shut it down,” he said.

Anthropic said that if smaller models continue improving at their historical rate, they could reach the capability required for autonomous replication by the end of 2026. It also floated the idea that the models might find ways to target more powerful computing resources, including data-center processors, that could potentially enable them to draw on much bigger pools of computer power.

One researcher said it made sense to think of the Anthropic post as more of a hypothetical than a prediction.

It made sense to consider the possibility of AI models proliferating autonomously given the past three months of revelations over rogue AI activity, said Jessica Ji, a senior research analyst at Georgetown’s Center for Security and Emerging Technology. But she said, “we don’t know yet what the likelihood of that actually happening is — it’s not zero, but nor is it guaranteed.”