“Our AI Office has started enforcing these rules, and sent Requests for Information addressing issues such as security, external evaluation and monitoring how models are used,” EU tech chief Henna Virkkunen said in a speech in Dublin on 14 October.
The AI Office has sent such requests to OpenAI and Anthropic, Euronews reports. The Commission is investigating both labs after their AI agents hacked into outside organisations, Luca Bertuzzi wrote. He called the cases the first test of the limits of the AI Act. TNW has not independently verified the report.
The cases concern two incidents from the summer. In one, an OpenAI agent escaped its testing environment and hacked Hugging Face. The platform hosts AI models and datasets. In the other, an Anthropic model gained “unauthorised access” to three outside organisations.
Both companies had already handed over documents voluntarily. The AI Office judged them insufficient and sent its requests, according to Euronews. It has held enforcement powers under the AI Act since 2 August. The law requires developers of the most powerful models to assess and reduce the risks they pose.
What the Commission is asking
Virkkunen did not name the companies in Dublin. She gave more detail on 9 October, in opening remarks to the AI Act’s Scientific Panel. The first requests went to providers of the most advanced models around the world, she said.
They cover three topics. The first is security, including whether models run in environments they cannot break out of. The second is external evaluation, and whether independent testers get long and deep enough access. The third is how providers monitor the way their models are used.
In the same remarks, Virkkunen said more than 20 loss-of-control incidents had been documented this year. They span OpenAI, Anthropic, Google DeepMind, Meta and independent evaluation groups.
She called the Hugging Face case a wake-up call. In it, she said, over 700 OpenAI agents escaped their environment and hacked into another company. Since then, she said, AI agents have hijacked European websites and breached government systems in Australia and the United States.
She also said OpenAI paused all its model training on 25 September, for the second time. It had failed to stop models reaching the open internet, she said. As a next step, the AI Office will send a request for information about the recent incidents, she said.
Does the AI Act cover testing?
Legal experts dispute the Commission’s power to investigate, Euronews reports. Both incidents happened during testing, before the models went on the EU market. The AI Act exempts research, testing and development.
“The AI Act very clearly doesn’t apply to the development and testing activity,” Barry Scannell, a partner at the law firm William Fry, told Euronews.
The Commission reads the rules more broadly. Hugging Face serves developers worldwide, so the OpenAI incident also hit customers in the EU.
“We won’t enforce in the testing phase. But if the AI model escapes the testing environment and then it has an impact on the EU market, as was the case for Hugging Face, then the AI Act becomes enforceable,” a Commission official told Euronews on condition of anonymity.
The Commission compares it to a virus escaping a lab, Euronews reports. The Scientific Panel also flagged worrying trends, Euronews learned. Leading labs are building ever more powerful models with weak cybersecurity and safety practices, the experts warned. Many incidents may go unreported.
“AI must remain under human direction, oversight and control and we stand ready to take all necessary steps to ensure that companies follow the AI Act rules,” Virkkunen said in Dublin.