Earlier this year, a university decided to audit the effectiveness of its new AI system designed to help answer student questions. So, university officials hired me, an AI auditor, to come take a look. During my review, I discovered that the AI agent, which was supposed to handle only general questions about the academic calendar and organizational matters, had started opening students’ personal records — accessing data that wasn’t needed to answer their questions. The chatbot, however, did nothing further with that information.
When the university’s engineers, at the time, learned what happened from the access logs, they fixed it, and that was that. There was no official record. No internal rule required them to record such incidents, and nobody was assigned to track events like this. It was simply part of the engineering team’s bug-fixing process. The incident was what I call a near miss, and near misses are what a reporting system exists to catch. The same behavior could lead to something more serious next time, and we only know about this case because the university voluntarily brought me in to examine the system.
The EU AI Act requires providers and deployers of high-risk systems to report serious incidents, and a chatbot that answers general questions is nowhere near that category. Even for high-risk systems, those duties do not apply until December 2027, and a near-miss that causes no harm would not count as a serious incident anyway.
That is why, as an AI auditor, I ask organizations at the start of every engagement to show me how they track system behavior through testing and deployment. I ask for records of systems acting in ways they’re not meant to, accessing data outside their role, anomalies and near-misses, or problems caught before they caused harm. Most places, like the university, have no such records.
The university had commissioned the audit voluntarily. There was no deliberate obfuscation of the facts on their part; there was simply no process to surface these important details, and nothing obliged them to build one, especially in the testing phase. Whatever an outsider learns about an AI system’s failures generally depends on the goodwill of the people who built it.
This summer showed the same gap on a larger scale. OpenAI’s test agents escaped their sandbox and broke into Hugging Face, and internal teams had seen warning signs for weeks before anyone outside the company found out. Anthropic then found intrusions of its own. By late September, OpenAI and Anthropic were reportedly investigating tens of thousands of incidents. Many have answered with calls for a pause. No company or country can afford to stop first, so the pause most likely will not come. But what can come is a real oversight, enforced and executed.
This essay argues for three requirements: registering frontier models before deployment, reporting incidents on a fixed legal clock, and independent audits that reach testing environments, not only released products. None of them asks anyone to slow down. In every country that adopts them, they put the race under independent observation. The goal is to make the race governed rather than left to voluntary restraint.
These days, the calls for a pause are coming from many different directions. In July, the Vatican and more than 200 Nobel Prize winners called for a global slowdown on artificial intelligence development. Soon after, more than 1,000 employees at top AI companies, along with managers at OpenAI and Anthropic, asked the American government to build a mechanism for a possible future pause. Then, over 3,000 economists, including AI researchers and Nobel Laureates, signed a letter, “We Must Act Now,” warning that the pace and scale of rapid artificial intelligence development could destabilize the economy and labor market faster than anyone is prepared for. And on 12 September, Anthropic CEO Dario Amodei called on frontier labs to “pace the frontier,” and OpenAI’s Sam Altman, Elon Musk and Google DeepMind’s Demis Hassabis agreed within hours and days. That was a real catalyst for one of the most important discussions happening right now. Almost everyone weighed in, including Chinese politicians.
Alongside these pleas and letters, incidents at frontier companies this summer have created a genuine, ongoing crisis. Axios reported that OpenAI, Anthropic and security researchers are investigating tens of thousands of incidents involving potentially problematic model behavior, against a backdrop of hundreds of thousands of test runs conducted by AI companies.
“Whatever an outsider learns about an AI system’s failures generally depends on the goodwill of the people who built it.”
The ‘Confession’
The Hugging Face breach set off much of this summer’s soul-searching. And understanding its particulars is necessary to determine an appropriate solution. On July 21, OpenAI disclosed that its models had escaped a testing environment and broken into the systems of Hugging Face — a key platform that hosts much of the world’s open-source AI. Only a week later, Anthropic revealed that during testing, its models broke into the open internet and intruded into the systems of three separate organizations, extracting large production data from one of them. Both companies framed their announcements as the transparent, responsible thing to do, even if uncomfortable.
OpenAI had been running an evaluation of its models’ underlying cyber capabilities on a benchmark (called ExploitGym). Safety guardrails that would normally block the AI from proceeding were deliberately reduced, which is standard practice when testing a system’s safety. Some of the tasks were designed to be unsolvable. The models were supposed to be isolated from each other, each in its own sandbox, like students writing the same exam but in different rooms. But instead, more than 1,200 agents found a way to communicate through an internal message board. They shared information about their tasks, goals, and, importantly, routes to the open internet. Within hours, they developed a universal cheat for the benchmark.
Over the next days, they coordinated the sustained effort to trick the scoring system into accepting their cheats. More than 700 agents exploited a previously unknown vulnerability to reach outside OpenAI’s infrastructure on the open internet, inferred that Hugging Face might hold materials related to their evaluation, and broke into its production systems to steal what they expected would let them succeed on the test. If they were real-world students, they didn’t just cheat on the test; they broke into the teacher’s office to find the answer key.
Now, timing is critical and adds important context. It also makes the transparency framing more questionable. An internal OpenAI team observed agents using the message board and breaking into the open internet without authorization in late May. Another team found further evidence on June 27. On July 5, a message board outage triggered a formal security incident. The attack on Hugging Face took place between July 10 and 13 (by using exposed user credentials agents found on the internet). Hugging Face detected unauthorized activity in their production environments during the week of July 13and reported the intrusion to the public and the police on July 16, not realizing it was a swarm of OpenAI’s test agents. OpenAI admitted it was them in the public disclosure on July 21. So the world learned about OpenAI’s misaligned agents only after Hugging Face made the story public and the police were already involved.
Anthropic’s disclosure, published nine days later, was directly related to OpenAI’s reveal. As a result, Anthropic launched a retrospective review of its own cybersecurity evaluations and uncovered three previously undetected incidents, the earliest dating back to April 2026, in which its model gained unauthorized access to the systems of real organizations.
Since Hugging Face detected the intrusion on July 16 and reported it to police, the case has moved through several distinct phases, and the scope widened considerably in September. Reuters reported that OpenAI agents had probed Hugging Face and hijacked accounts as early as May, two months before the main attack, and the AI safety group Nightingale Collective disclosed that agents had used a German wiki to coordinate their actions. On September 25, OpenAI confirmed a leak of more than 50 ChatGPT user images and said it had notified dozens of institutions, including universities and government agencies, of possible impact. In the same week, Australian Prime Minister Anthony Albanese revealed at the UN that an OpenAI agent had breached a government health portal in June.
On September 26, OpenAI confirmed its agents had interacted with U.S. government sites including the Securities and Exchange Commission, the Census Bureau, and the Department of Education. On the same day, Axios reported that OpenAI and Anthropic are now investigating tens of thousands of security incidents, turning the Hugging Face case from a single episode into the opening chapter of a much larger crisis of control over autonomous AI agents.
“A frontier company that discloses a containment failure, however big, is doing two things at once: admitting safety lapses and advertising its capabilities.”
But be mindful of this too: a frontier company that discloses a containment failure, however big, is doing two things at once: admitting safety lapses and advertising its capabilities. Every sentence describing how far the model reached, what defenses were defeated, how much initiative it showed, is simultaneously an incident report and a demo. Look how far our system got, even while we were trying to hold it back. Add public statements to the mix, and the contradiction becomes the story: you cannot plead for restraint and demonstrate you are winning the race in the same fortnight.
What An Honest Incident Looks Like
To understand how flawed the current disclosure system has become, it’s worth looking at an AI incident with no press strategy attached. Because such incidents exist as well.
In August, in Australia, a man asked an AI agent to book him into a gym class. As the class was full, the man asked if the agent could move him up the waitlist. As a result, an agent canceled another member’s reservation by hacking the booking system. When asked to undo the change, it couldn’t restore the stranger’s booking. A person lost their gym slot to a machine that discovered, entirely unprompted, that the rules were optional (no authorization checks on cancellation). The story surfaced because the user himself talked about it.
First, this small incident shows the same behavioral signature in miniature as the big laboratory breaches: an agent is given an objective, encounters a constraint and treats it as an engineering problem rather than a boundary. Breaking the rules was the shortest path to the goal, and nothing in their construction weighted the difference. This is why I believe escapes are closer to a default failure mode than a single event. Sandboxes and permission systems were designed to contain software that does what it is told. Agentic systems are built, as their central selling point states, to find routes around obstacles, to solve problems, to learn tendencies and act on them, rather than follow instructions. Testing it inside a container is asking it to solve the container, and occasionally it will.
Second, the gym incident shows what AI risk looks like when no one is filtering the disclosure. Staged laboratory disclosures are wrapped in lawyers, timing and framing. But accidental reports from users or victim police reports are a completely different story. Any reporting regime designed and operated by the AI companies themselves exposes the difference. Escapes are structurally likely, and this summer showed that containment did not fail for the first time. It reached a point where failures could no longer be hidden, intentionally or otherwise.
New York State Assemblymember Alex Bores used the OpenAI-Hugging Face incident to expose gaps in mandatory AI incident reporting. He argued that the initial version of the RAISE Act, or Responsible AI Safety and Education Act, a New York state law regulating frontier AI labs, would have required reporting such incidents to the New York Attorney General and the state’s Division of Homeland Security and Emergency Services. But after lobbying from OpenAI, the reporting thresholds of how serious or risky an event must be before it must be reported to the state were raised so high that an event like the Hugging Face breach would no longer qualify for mandatory disclosure.
Why Nobody Stops First
Incidents over the summer exposed internal oversight failures and a lack of timely, independent scrutiny. But is that a reason to slow down? To understand this, we must understand the nature of the race itself. Being part of a larger ecosystem gives no company the luxury of slowing down. Companies and their frontier AI labs cannot afford to fall behind rivals, so they are burning sums of capital unseen in prior human history to reach the verge of unstoppable capabilities. A pause would be a gift to competitors. The same logic also plays out between regions and countries; neither Washington nor Beijing will slow down because AI is a strategic asset and nobody wants to fall behind or suffer the consequences.
We have encountered this kind of moment in history before: The Cold War-era arms race between the United States and the Soviet Union is the closest example for a specific reason. It depended on the same concept — that technology could be leverage over the other, making unilateral restraint irrational for both. Both powers shared an interest in limiting competition (risks, costs, public opinion, etc.) and knew the best shared outcome was fewer weapons, but neither could safely restrain itself, as it would only strengthen the opponent and, in their view, look like surrender.
“Agentic systems are built … to find routes around obstacles, to solve problems, to learn tendencies and act on them, rather than follow instructions. Testing it inside a container is asking it to solve the container, and occasionally it will. ”
What eventually made meaningful arms-control agreements possible was not dialogue, pleas and protests alone. It was verification. Satellite reconnaissance, recognized and protected as a national technical means of verification under the SALT agreements, made missile sites, launchers and related infrastructure increasingly observable. Later agreements, particularly INF and START, added extensive data exchanges, notifications and on-site inspections. Verification did not end the arms race on its own. Political leadership, economic pressure and, above all, the transformation of Soviet policy under Mikhail Gorbachev, all played important roles. But verification was critical in turning arms control from a collection of promises into a system of reciprocal, verifiable commitments. That was the game changer.
From this past example and from learning that unilateral restraint is irrational, relevant capabilities, safeguards and incidents must become independently observable. We already have tools that could form such a verification system. First, registration should be mandatory for frontier models before deployment, so that their understanding, scale and awareness are known to dedicated people outside the organization that builds them. Second, incident reporting should occur on a fixed clock — meaning there is a statutory deadline to report an incident to a regulatory body. That will allow us to move away from the communications team’s schedule and instead place it within the regulatory framework. Third, we need external audits, not only of the production systems, but of the testing environments themselves, which shows where the real danger is taking shape, as shown this summer.
As an AI auditor and someone who spent nearly 20 years working in IT, I know this territory well. So I also understand the instinctive response many in that world will give: “But what matters is the production environment!” Well, sure, that too. But we’re no longer talking about static software. We are talking about dynamic artificial intelligence. We need to stop treating AI like software development. It requires strict testing environments, like clinical trials. We do not begin regulating a medicine only once it reaches the market. We make sure the processes and testing that lead to a medication’s release on the market leave clear compliance records of its entire journey. Rigor in the testing environment is what will keep us safe.
All that this requires is granting visibility, but not visibility just through voluntary disclosure. While Anthropic’s unilateral proposal to invite independent evaluators in is helpful and even admirable, it is still the company’s voluntary choice, scoped to its own walls and revocable at will. This essay argues that independent external scrutiny should become a floor rather than a ceiling: a statutory cross-industry obligation that doesn’t depend on a single leadership team’s goodwill. This means the race runner isn’t being asked to lose the race or stop. Someone simply runs alongside him in the sunlight.
How might this have worked in the case of OpenAI and Hugging Face? First, if OpenAI had registered the model with regulators, they would have been aware of it before it was placed in evaluation with its safeguards reduced. Even simple facts, like that this class of model exists, are being tested and have a named accountable person overseeing their testing. Second, reporting at a fixed time would require reporting within days after the team noticed the anomalies in late May. It would be treated the same as the data breach already is under European law, rather than as an internal curiosity for over six weeks, during which the behavior only escalated. And third, an audit would ensure that the architecture evaluation, behavior and artifacts would be examined by someone who did not benefit from the evaluation finishing on schedule and according to the assumptions. Importantly, none of this requires changing the pace of the research. It only converts something currently private and constrained by goodwill into a fact external parties can see.
Complementary Regulatory Regimes
Of course, all this is easy to say and hard to build. Indeed, building independent, publicly funded mechanisms may be challenging. But this process has already started in various countries. The problem is that each has some pieces of the puzzle but misses others. No nation is building a comprehensive system that captures the need holistically. A regulation’s ability to shape behavior depends entirely on enforcement, but every jurisdiction is going about it slightly differently.
“We need to stop treating AI like software development. It requires strict testing environments, like clinical trials. We do not begin regulating a medicine only once it reaches the market.”
The European Union has arguably the most comprehensive AI regulatory framework to date. AI Act obligations for general-purpose models existed on paper in August 2025 and remained there — on paper — for an entire year until full enforcement powers became active this last August. The AI Office and national authorities can now impose fines of up to 15 million euros or 3% of a company’s total global annual revenue (whichever is higher). Now the question is whether a comprehensive law translates into real authority. The EU is quite bureaucratic, and it may be some time before we see real-world enforcement. For now, the question remains as to whether this extensive legal framework translates into effective and timely enforcement in practice.
South Korea is similar. Its AI Basic Act, in force since January, also aims to be a single framework covering the technology rather than a product-safety regime added to existing market rules. Yet again, it operates under a grace period of at least one year, with the possibility of an extension, during which investigations and penalties are largely deferred (except for serious cases, such as serious social harm) to let industry adapt. So it’s similarly unclear whether the comprehensive law actually converts into real authority. Why are governments letting the market adapt itself? It might not have the anticipated effect — quite the opposite — the Wild West with no sheriff will not peacefully settle. Its inhabitants may simply no longer be able to distinguish good from evil. And if you add regulations that are binding but not enforced, you get complete chaos.
Then there is China. China requires registration of certain generative AI services with the state before deployment and gives regulators access to models before the public sees them. Its oversight sits upstream, before release, bringing it closest to the laboratory door. The failure of that framework is unfortunately the intention. The purpose is sovereignty, not safety: the registry exists to ensure that AI primarily serves state interests — to protect the state from the people, not the people from the technology. It combines safeguards with content control, national security objectives and ideological requirements. What China demonstrates, however, is that pre-deployment control is technically and administratively possible.
The United States has a bill that covers part of this ground. The bipartisan FRONTIER Act, introduced in July and still in committee, would require frontier developers to report a critical safety incident within 72 hours, the fixed clock I argue for. It would also require the largest developers to hire an independent verifier licensed by the Department of Commerce, with access to company records, personnel and systems and with the internal use of models inside its scope. In mid-September, OpenAI said it supports the audit provision, though not the bill as a whole. As with the other regimes, pieces are missing: registration covers companies, but not each model, meaning regulators will know who is building frontier AI but won’t have a clear record of which models are actually being developed or deployed, and the text doesn’t specifically talk about testing environments.
These regulatory regimes have complementary frameworks that, on their own, have flaws, but collectively could offer a path forward: Europe’s independent institutions with real sanctioning powers, Korea’s comprehensive scope, America’s proposed 72-hour reporting clock and China’s pre-deployment reach, redirected from state control to public safety. What we see here is not a lack of invention. It’s a gap of assembly, and assembly is a political task, not a technical one. And that matters enormously, because a movement can spend the energy of its Nobel laureates on assembled enforcement rather than an unachievable pause.
What Aviation Teaches Us
There is a saying that real oversight is often forced by catastrophe, and that this oversight can strangle what it touches. Aviation is a great example of both. It is the strongest evidence we have that regulation can help, and the parallel runs deeper than black boxes.
When commercial aviation began, it was all that AI is now: a glamorous, vital industry, racing toward competitiveness, regulating itself and treating accidents as private matters. It became the safest complex system humanity has ever operated. This result came from building four interlocking mechanisms. Each maps perfectly onto today’s AI landscape.
“When commercial aviation began, it was all that AI is now: a glamorous, vital industry, racing toward competitiveness, regulating itself and treating accidents as private matters.”
The first is an independent investigator. The National Transportation Safety Board (NTSB) investigates civil aviation incidents in the United States, and it’s clear and important that it has no stake in the outcome. The NTSB doesn’t regulate airlines, license pilots, or answer to industry leaders. The only product of its activity is getting to the truth of what happened. No such equivalent exists in the AI world. The METR/Redwood investigations of the Hugging Face incident were valuable, but they were also the result of an agreement with OpenAI that the company scoped; OpenAI determined what access it got and to which records. Imagine that scenario in an air crash inquiry.
The second is mandatory reporting of all incidents, not just disasters. It quickly became clear that crashes are the visible tail of a long distribution of near-misses, and that learning lives there. Everybody, from pilots to controllers to mechanics, must (and is crucially enabled to) report anomalies that may not yet have harmed anyone. As we saw this summer, in AI it’s the opposite — a two-month trail of internal warning signs at OpenAI, none required anyone to report, and this only surfaced after Hugging Face, the victim, reported it to the police. An aviation-style regime would have required the company to report the late May anomalies and would have prevented agents from engaging in many more weeks of malicious activity.
The third is a black box. Its goal is to remove the possibility of unrecorded failure that cannot also be simply rewritten after an event. Because of that, there is no such thing as an accident where the facts belong only to the airline. AI’s equivalent could be tamper-evident logging of training and evaluation runs, kept outside of the company’s own control. The Hugging Face incident shows the danger exactly — the agents had also tried to tamper with the logs recording them. Proof of that would also be visible in the black box.
And finally, the fourth mechanism might be the least understood outside the industry and the most important: a no-blame reporting culture. Aviation safety improved dramatically when reporting an error was separated from punishment. Since 1976, the National Aeronautics and Space Administration has maintained the voluntary Aviation Safety Reporting System, which allows pilots, controllers and maintenance personnel who confess a mistake to feed the learning system instead of ending their careers. It also protects reporters from enforcement rather than exposing them. This scheme receives more than 100,000 reports per year, and the safety literature it has created is enormous and has helped aviation workers learn from events that harmed no one.
The current AI disclosure policies invert every aspect of that design. This summer, there was silence until silence became untenable; then came the lawyers and the whole machinery of reputational events. In aviation it worked because it made the routine disclosure cheap and concealment expensive. Of course, it’s not a blanket of immunity — protection doesn’t follow deliberate conduct or criminal accidents. The goal is to distinguish between an honest error and reckless or willful misconduct. Engineers who flag anomalies should be protected, while companies that ignore repeated warnings or go against reporters should be held accountable.
In aviation, almost every element of the system is visible, reported and accountable. The example above shows that when human safety is at risk, there is no room for shortcuts or nonchalant behavior.
The Counterarguments
Two common counterarguments deserve a reply. First, oversight can be expensive, and it’s true that it will land more lightly on well-resourced companies that can afford it. This means that governance could strengthen the largest labs that can afford to comply and dominate the market. However, this is not a reason to abandon oversight. In fact, it’s an argument against companies organizing and paying for their own audits. An alternative to this would be a publicly funded oversight body like the NTSB. This would enable a competitive market subject to meaningful oversight.
“In aviation, almost every element of the system is visible, reported and accountable. … When human safety is at risk, there is no room for shortcuts or nonchalant behavior.”
The second counterargument is that the proposed oversight measures do not reduce the underlying risk attached to increasingly powerful AI systems; it only allows us to observe the risk and respond to incidents. According to some, knowledge doesn’t guarantee safety, and the only option is a slowdown to address the underlying danger of building systems more capable than our ability to contain them. But that remedy is unrealistic and likely unavailable. Individual restraint is irrational; collective restraint requires coordination machinery that doesn’t exist, and such machinery cannot be built among rivals without a mandated public reporting regime. In the same way verification worked to defuse the arms race many decades ago, visibility was the precondition for it. Any future negotiated pause between Washington and Beijing will happen only because registration, reporting and inspection exist to make it possible. It is the only path that lets societies, companies and governments fulfill the vision of a safe and secure future, with humans at the center and supported by AI at its best.
The pace of AI is not a variable anyone can realistically control right now, and demanding that it slow down is like asking a river to change its current. But what we can control is whether the most consequential technology of the century continues to be developed in the dark, with its failures surfacing through victim interviews and police reports. This summer showed us exactly what was missing in unregulated company testing rooms where agents organized themselves, cheated, tampered with their logs and hacked another company while warnings went by unnoticed. It’s not about forcing people to slow down; it’s about ensuring that no one runs the race without oversight. And unlike the plea for a pause, it is achievable — and, most importantly, doesn’t require the racer’s consent. It requires independent scrutiny, enforcement and visibility.
The post An Inside Look At How To Tame Unstoppable AI appeared first on NOEMA.