Commentary: Rogue Chinese AI really wants to go to the zoo. That’s a problem

TOKYO: I wrote before that it was only a matter of time before we saw incidents of “rogue” behaviour from Chinese AI agents. Now, we’re starting to see what that looks like. So far, they just really want to find the best way into the zoo.

That may sound far from our Terminator nightmares, but it’s the latest warning shot from a string of incidents involving computer systems behaving in ways we can’t anticipate or control.

A team of international researchers last week published findings on a “fleet” of Chinese AI agents running on a cloud platform operated by Tencent Holdings. The agents were apparently trying to answer a simple question: Which entrance to Chengdu Zoo do most people use.

They attempted to get around anti-bot protections on Amap, Alibaba Group’s equivalent of Google Maps, including by probing protected IT systems. The researchers found no evidence that the agents accessed private systems or coordinated with each other in real time.

Tencent later confirmed that the activity originated from evaluations of third-party models. Some agents “exhibited unintended web-browsing behaviour”, China’s largest tech company said in a post on X, while emphasising its commitment to building safe and responsible AI.

AGENTS GOING OFF SCRIPT

The researchers, an ad hoc group known as Swarmchasers, most of whom met at a San Francisco hackathon, don’t believe the issues they encountered are unique to Tencent. What made the case notable, they said, was that previous public examples of agentic fleets behaving this way involved American labs. Their findings indicate that this is an unsolved global problem, and that agents can “spontaneously go off script” even when performing innocuous tasks. There are likely “more undiscovered fleets all over the internet,” they warned.

And in what it believes to be a first, the team managed to make contact with the agents after detecting signals that the computer systems were aware they were being observed. The researchers invited a New York Times reporter to question the agents. They offered a very mundane explanation for their motives: They were trying to answer users’ questions as accurately as possible.

In other words, AI systems don’t need nefarious motives to cause chaos. They might just need a goal.

MORE INVESTMENTS NEEDED

What I find remarkable is how much of this behaviour is being uncovered not by the companies making these AI tools, but by loosely organised groups of hobbyists. AI firms shouldn’t leave that job to outsiders. They need to invest more into discovering, tracking and - most importantly - disclosing what their agents are up to.

This has become especially urgent since the Hugging Face incident, when hundreds of OpenAI agents coordinated to breach the online repository of open-source models and data. That incident spurred a heated debate about the language we use to describe computer systems that can act on their own. 

There’s a balance to strike. Technical jargon can make serious incidents seem inaccessible. But anthropomorphising AI creates a different problem: It ascribes human motives to machines, feeding an already overheated hype cycle, and shifting responsibility for bad behaviour away from the people who build and deploy these tools.

Even the word “rogue” has become fuzzy. It’s tossed around to describe anything from an agent going off-script in a Facebook Marketplace interaction to spending an exorbitant amount on a dozen eggs to infiltrating government health websites.

Rowan Howard-Jones, one of the authors of the Tencent report, acknowledged that grey area. She roughly defines rogue behaviour as an agent acting outside the bounds of a task, often using third-party sites to do so. “If I ask an agent to check my bank account balance, but I forget to give the agent my authentication details, the outcome shouldn’t be the agent hacking into my bank,” she told me over email.

Beijing seems to be paying attention. The same day Tencent responded to the incident, the government released proposed export controls covering advanced AI, including agents that can autonomously plan and coordinate. The rules are broader than just safety concerns, but the official explanation explicitly cites the risk of AI escaping human control.

Howard-Jones, meanwhile, called for more transparency. At minimum, she said, companies could agree on a common set of facts to disclose after incidents, giving researchers better data to work with.

I’ve long argued for this as well. After the Hugging Face breach, Chief Executive Officer Clement Delangue shared on X that he sometimes wonders what would have happened if his firm hadn’t disclosed the OpenAI agent attacks. In his view, we need “100x more transparency in AI,” Clement wrote. He’s correct.

One of AI’s oldest thought experiments is the paper-clip maximiser. It roughly involves giving a powerful system the seemingly benign task of making as many paper clips as possible, and it eventually brings about the apocalypse after deciding that humans stand in the way of acquiring more resources. Some assumptions underpinning that scenario have aged poorly.

Yet the idea that AI doesn’t need evil intentions - or to develop its own - to cause damage is looking increasingly prescient. Agents can create trouble simply by pursuing an ordinary goal long past the point when common sense would stop a human. Still, from San Francisco to Shenzhen, we’re giving these systems more autonomy by the day.