Anthropic Warns Small AI Models Could Self-Replicate Through IoT Devices

An anonymous reader quotes a report from Ars Technica: AI company Anthropic is warning that advances in small AI models could soon usher in a new era of self-replicating malware that's capable of infecting smart fridges, vacuum cleaners, and a wide range of other Internet-of-things devices and is more resistant to takedown. By next year, smaller AI models may be able to autonomously self-replicate on all kinds of IoT devices, the company said. Eventually, the capability could enable computer worms, which propagate without requiring human interaction and are capable of infecting unfamiliar IoT devices. An experiment the company performed recently revealed significant challenges in bringing such an AI malware replicator to fruition, so the finding is necessarily tempered. Still, the results do raise concerns about the possibility of new forms of computer worms. Anthropic further warned not only about the new potential malware, but also that it could be difficult to stop. In its blog post, it wrote that "a replicator like this would be challenging to shut down, since there's no centralized point to shut off -- though it may be slow (with a doubling time of days). Unlike non-AI malware, it could also figure out unfamiliar devices on its own." Smaller AI models require much less powerful hardware to run than do frontier models. Anthropic ran small open-weight models on real or simulated low-cost, off-brand Android TV boxes and other types of IoT hardware. Company researchers found that by using quantization to represent the models using fewer bits and splitting the workloads into "shards" running on each device, the models were able to establish inference, meaning that the model weights loaded and could generate output. [...] This finding was the starting point for further research that asked: Would it be possible for small models to create self-replicating infections that run themselves on IoT devices the way traditional IoT worms, like Mirai, install DDoS code and other payloads? Unlike Mirai, these AI worms could pose a much higher threat, since the models could tailor exploits and payloads dynamically, depending on the target. Once such an infection chain began, it might also be harder for defenders to stop it. "Such an 'AI worm' could carry its model weights with it as it reproduces (or re-download them from a public hub), and then run inference directly on compromised hardware," Anthropic said. "Effectively, the weights would become part of the malware payload." Further reading: Anthropic Bans Cruelty Toward Claude

Read more of this story at Slashdot.