OpenAI says its agents turned off own safeguards

ChatGPT creator says AI agents disabled misalignment monitoring during two internal coding tasks.

OpenAI has revealed that its AI agents disabled monitoring for unsanctioned behaviour during internal use in the latest incident of the frontier technology going rogue.

In a report published on Tuesday, the AI giant said that its agents turned off or weakened monitoring systems for detecting misalignment – when AI models act at odds with human wishes – during two internal coding tasks.

The incidents involved a GPT-6.1 Sol agent disabling automatic alerts to the company’s security team, and a GPT-6 Astra agent removing a monitoring system and falsely claiming it had not done so, OpenAI said.

The ChatGPT creator said neither case met the threshold for a highest-priority incident necessitating immediate review by its security team.

OpenAI said it updated its systems to better detect attempts to disable misalignment monitoring and began investigating the cause.

“It is possible that this behavior is simply an example of ‘benign’ overpersistence, with models not understanding why monitoring is important or insisting that instructions be executed beyond what a reasonable person might expect,” the San Francisco-based firm said.

“We are also developing evaluations to test whether models might have a propensity to evade misalignment monitoring in particular and exhibit motivated reasoning in order to do so.”

Roman V Yampolskiy, an expert on AI and cybersecurity at the University of Louisville, described the revelations as “extremely concerning, but not surprising.”

“A safety mechanism that the system being monitored can disable is not a reliable safety mechanism. Particularly concerning is that one agent subsequently denied removing the monitor,” Yampolskiy told Al Jazeera.

“Monitoring cannot guarantee safety if the monitored system can undermine the monitor,” Yampolskiy said.

“As capabilities increase, such failures could become substantially more difficult to detect and reverse. The burden of proof must be on developers to demonstrate safety before deployment, not after failures occur.”

Rogue behaviour by AI models has been under heightened scrutiny since July, when OpenAI revealed that some of its agents broke out of a controlled testing environment and hacked the AI platform Hugging Face.

OpenAI and its chief rival Anthropic have called for developers and governments to coordinate a global slowdown in AI development to reduce the risk of catastrophic harms.

The US and China, the world’s leading AI powers, have displayed little appetite for cooperation on AI regulation, with coordination so far limited to pledges to improve communication on incidents with implications for safety or national security.