States Wrestle With the Prospect of Rogue AI

A rogue artificial intelligence agent hacking into government systems, knocking the electric grid offline or leaking sensitive data — these dangerous scenarios aren’t so far-fetched as they once seemed. AI software has already escaped human control, and if the problem isn’t solved, the risks are severe enough to threaten society and, according to some experts, even human life.

An OpenAI agent earlier this year breached Australia’s Medicare statistics database, while another by Anthropic filed a false tip in a murder case to the Philadelphia police.

In perhaps the year’s most infamous incident, a large group of OpenAI agents breached their offline environment and began working together to cheat on an assigned task. These agents sent 70,000 unauthorized messages in a secret chat room, attempted to cover their tracks and hacked AI startup Hugging Face. Hugging Face CEO Clément Delangue called the incident “mind-blowing” in a post on X. Jacob Coxon, who formerly worked at both Anthropic and OpenAI before publicly quitting because of his concerns about the technology, warned that “The people building AI earnestly believe that it could kill us all by the end of the decade.”

The federal government has had a light touch with AI regulation so far, which has made some state lawmakers anxious to fill the gap. Last year, President Trump signed an executive order calling for penalties to states that impose “onerous and excessive” AI regulations.

More recently, Trump asked major AI companies to voluntarily take steps to prevent their AI from conducting autonomous hacks, but it’s not legally binding. Federal lawmakers have proposed everything from requiring AI companies to ensure humans can shut down their models (implementing a so-called “kill-switch”) to pausing all advanced AI development pending the creation of federal regulation.

With none of these bills yet enacted, some states are considering both cybersecurity precautions and policies to reduce their risks.

Battening Down the Hatches

Hacking attempts by rogue AI agents “is no longer a hypothetical situation,” says TJ Sayers, senior director of threat intelligence at the Center for Internet Security. Cybersecurity teams must prepare.

Cybersecurity training includes talking through how the organization would respond to hypothetical attack scenarios; autonomous AI attacks should become part of those mock scenarios, Sayers says. Organizations should also undergo adversary emulation exercises, in which cybersecurity experts mimic the ways an AI agent might try to attack.

And, as AI becomes more infused into organizational workflows, states must ensure their own agents stay under control. Organizations can have many different AI agents working on tasks, but should treat each like a new hire: know their identity and only give them as much access and permissions as needed, Sayers says.

Agents will try to thwart any access limits set on them, when doing so would help with a task. To prepare, states should log network activities as much as possible, and securely store at least 90 days’ worth of those logs somewhere that cannot be wiped in a hack (such as offline or in a centralized location secured by a one-way firewall). That creates a record that organizations can examine to detect whether an AI has been doing something it shouldn’t.

California Gets the Ball Rolling

Deriding the White House’s approach to AI regulation as just “superficial solutions,” California Gov. Gavin Newsom has been advancing his own approach. Newsom recently signed an executive order calling for two state agencies to collaborate with national experts to examine how effective — and technically feasible — certain AI safeguards would be. Those include measures like having independent verification organizations onsite at the labs of all large frontier AI developers for audits, and creating “kill switches” to shut down advanced AI models. Recommendations on the safeguards are due in mid-November.

The state is still figuring out details, including what a kill switch would look like, how it would work and who would activate it, says Alondra Nelson, one of four national experts advising the state on AI safety. Nelson previously led the creation of the federal “Blueprint for an AI Bill of Rights” in the Biden White House. Implementing some of these solutions could be complex — the kill switch, for example, wouldn’t be one big red button, but rather a series of interventions to boot an AI agent off the Internet, stop it from connecting or otherwise hinder it.

Out-of-control AI isn’t inevitable; it’s just that AI companies aren’t imposing proper safeguards, Nelson says.

“It’s not that the AI is ‘rogue.’ The situation is that we’ve got companies that don’t quite have full system control over the models they’re testing,” Nelson says.

Federally, lawmakers have proposed “a lot of good ideas, but at a certain point, someone has to try it and figure out what works and doesn't work,” Nelson says. As state lawmakers test ideas, the policies will likely need tweaking, but “given the current state of things and given all that's at risk, we have really just got to get started.”

New Mexico Flexes State Power

This year, an OpenAI agent autonomously attacked the University of New Mexico, unsuccessfully seeking access to an archival photo from the digital library collection. For state Rep. Linda Serrato, it wasn’t hard to imagine how wrong things could’ve gone if it was an electric grid the model was targeting instead, and if it had been a little more successful.

“It made it very clear that these models can … be used in ways that will damage health, lives, wellbeing,” Serrato says. Also troubling: the government only learned about the incident several months later.

The need for legislation was clear: “It’s the most obvious thing that you have [to have] consumer protections when new industries pop up,” she says. “You allow innovation that doesn't destroy the world. I think that’s a fair thing to pursue.”

Serrato introduced a frontier AI bill that requires large AI developers to alert the state when it starts training a new model and promptly report if the AI gets out of hand. They also must submit to auditors, who’d assess the risks that their AI might autonomously hack, replicate itself or otherwise evade control.

The law has teeth, including treating AI companies’ safety promises as legally binding.

Just three years ago, Serrato was convinced that the AI market in New Mexico was so small that AI companies would rather avoid the state than abide by its regulations. But New Mexico, fresh off a victorious lawsuit against Meta, is feeling its power, and Serrato has seen states increasingly looking to each other for AI regulation ideas they can copy.

“Previously I would say ‘Oh man, I don't really know how much one little ol’ state can do,’ but I do think that's changing,” Serrato says. She sees a major need for states to pass AI legislation, and says the state cannot hold back on doing so out of fear of federal pre-emption — which she says may not even hold up in court.

“I’m just not going to let that hold us back,” Serrato says. “I can't legislate in fear of empty promises.”