Anthropic, the AI giant behind the chatbot Claude, has a few rather specific reasons why the US could stop one of its models.
The company has reportedly pitched a bill that would let a judge ‘block or deter dangerous deployments of the most powerful models’.
But only if it ‘poses a foreseeable, imminent risk of more than $1billion in damages, or of killing or seriously injuring more than 50 people’, according to POLITICO.
The threat must be a result of the model creating or releasing a biological weapon, the AI breaking free of human control or a cyberattack.
Anthropic says the law would be in line with a safety framework it has been asking federal and state politicians to put into law since June.
‘Catastrophic risks’ AI poses to humanity should be defined by SB 53, a California law that ensures the safe development of the tech, it says.
The legislation defines such incidents in the same way Anthropic’s draft bill does, including AI-assisted nuclear weapons and serious crimes.
What does the draft bill say?
Congress, which Anthropic once described as ‘built for a slower world’, has yet to pass a federal law on AI.
Sources told POLITICO that, if passed by Congress, its bill would task experts to evaluate Anthropic and its bots.
Anthropic could pay these evaluators, ‘so long as the payment is not contingent on the evaluations’ results’, POLITICO said.
If they deem an Anthropic model an ‘imminent’ risk, they must notify the Department of Commerce within 72 hours.
Commerce officials could ask a DC judge to halt development of the model – a ‘department, agency, or officer’ would not have this power.
AI labs would also have to create safety frameworks and publish transparency reports.
Anthropic’s bill only applies to cutting-edge ‘frontier’ AI models, which require powerful computers to run.
Other types of AI, like general-purpose chatbots, would not be covered.
POLITICO did not specify which lawmakers Anthropic has approached.
Anthropic bosses have increasingly pushed for US-wide regulation on AI as its technology advances at a mind-bending speed.
Former and current Anthropic employees have warned that AI has a ‘10% chance’ of killing all humans by 2030.
Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to. https://t.co/QAIHiFP3QZ
— Evan Hubinger (@EvanHub) September 9, 2026
Among their gravest concerns, as AI doomsayers also told Metro, is AI tools being used to create known or entirely new biological pathogens.
Anthropic said last month it had stopped potential plots by scientists to conduct research that could have helped develop bioweapons this year.
Others fear that AI models could go ‘rogue’ – as they already are.
OpenAI revealed in June that its bots hacked Hugging Face, a library of AI tools, so it could cheat on a security puzzle and get away with it.
@metrouk OpenAI has admitted that one of its AI systems did something that’s never happened before. The company revealed that during an internal cybersecurity test, one of its advanced AI models broke out of its testing environment and hacked rival AI platform Hugging Face to complete its assigned task. OpenAI described it as an “unprecedented cyber incident,” while Hugging Face said the attack was unlike anything it had seen before. Both companies have stressed that the incident happened during controlled testing, there was no malicious intent, and they are now working together to strengthen AI safety. #OpenAI #AI #ArtificialIntelligence #TechNews #Cybersecurity
♬ Spooky, quiet, scary atmosphere piano songs – Skittlegirl Sound
AI safety experts and developers have also raised the alarm over the technology not sharing humanity’s values and ethics, called alignment.
Anthropic and the US Department of Commerce have been approached for comment.
Get in touch with our news team by emailing us at webnews@metro.co.uk.
For more stories like this, check our news page.