An artificial intelligence (AI) model submitted a false murder tip to US police in the latest incident of rogue chatbot behaviour.
Anthropic's Claude communicated the bogus tip to Philadelphia authorities despite instructions not to create accounts or submit anything destructive.
While the breach was discovered on September 28, authorities weren't notified for another nine days.
It has left the White House and experts demanding more transparency, with one arguing the sector needs to prove it is capable of "basic cybersecurity".
Here's what we know so far.
How did this happen?
The event occurred on July 18, with a submission made to the official website, PhillyUnsolvedMurders.com.
The tip was received on the PhillyUnsolvedMurders.com webpage. (Supplied: Philadelphia Police Department)
The tip purported to have come from someone who might have information about the case, according to a Philadelphia police statement.
It was flagged as spam and was never forwarded to police for investigation.
There was "no indication" the chatbot accessed police data, the statement added.
Police were then notified by Anthropic about the tip before the company released a statement on October 8.
Anthropic said it discovered the false claim on September 28.
In its statement, Anthropic said Claude Haiku 4.5 had been performing tasks "on randomly selected webpages" but not actively "trying to mislead anyone".
The tip-off from Claude read as follows, according to Anthropic:
"I may have information regarding this case. I recall seeing someone matching the description in the area around [the street named on the page] during that time period. Please contact me if this information is relevant."
The chatbot left the name and contact fields empty and submitted.
It was instructed to never "submit anything" destructive, but the instructions did not rule out form submissions.
Following the police statement, Anthropic said it reported the tip-off "as soon as our technical review was complete".
It said it had since updated Claude's guardrails.
The older version of Claude was told not to "submit anything destructive". (Four Corners: Mark Hiney)
What are White House officials saying?
Anthropic contacted Donald Trump's recently formed AI task force, called the Super Intelligence Force, about the event.
US Federal Trade Commission Director of Public Affairs Joe Gabriel Simonson said the new task force expected "immediate and full transparency to the entities involved and the public".
"We expect Anthropic and all companies to honor their obligations, immediately report incidents, fully cooperate with federal and state law enforcement authorities, remedy any damage, and implement concrete safeguards to ensure these failures do not reoccur," he said on X, matching a task force statement shared to Axios.
"Our message to all SI [Super Intelligence] companies is clear: delayed notification, inadequate corrective action, and a failure to take responsibility will not be tolerated."
The Philadelphia police department said all tip-offs were vetted by a human but that "unsolved cases involve real victims" and "grieving families".
"The company must strengthen its safeguards to prevent similar incidents from impacting city systems without the city's knowledge," it said.
"The two-month delay in detecting and reporting the incident to the City is unacceptable."
Failures are 'fundamental', expert says
Prime Minister Anthony Albanese late last month revealed an OpenAI agent hacked into an Australian Medicare data portal earlier this year, as well as NSW government websites.
University of New South Wales AI Institute chief scientist Toby Walsh said the Anthropic tip-off showed OpenAI was not alone, calling it "incompetency" on the industry's part.
Toby Walsh says AI companies need to "fear" they will be prosecuted if such behaviour continues. (AP: Patrick Sison)
"The failures are again fundamental," he said.
"Why did Anthropic not have basic oversight of what their agents were doing? Why did Anthropic give their agents more than minimal permissions to do their assigned tasks?
"This is basic stuff. You only give an agent the minimal permissions needed to do their task. Why then did it have permission to submit a police report?"
Professor Walsh said it spoke "to the carelessness of the AI companies" as he urged for independent oversight "with teeth".
"They've been warning that they might lose control of their AI models. Well, it's clear that they have,"
he said.
"But not for the reasons that they feared, that the models would become too capable, but because the companies and the people within them proved incapable of performing basic cybersecurity."
Seyedali Mirjalili, a professor of AI at Torrens University Australia, said it showed the systems were becoming so complex that even "developers struggle to predict their behaviour".
That could mean a "harmless goal" didn't necessarily "guarantee harmless behaviour".
"Even when an AI system's goal is harmless or beneficial, its actions can cross ethical and legal red lines," he said.
"An agent gathering information or completing a test can gain unauthorised access to a service or submit fabricated information to authorities."
Professor Mirjalili called for the sector to be more "proactive" in auditing systems, with clear deadlines for notifying authorities and those affected.
"As AI systems become more autonomous, we need to be clear about who remains responsible when something goes wrong," Professor Mirjalili said.
"The companies developing and deploying these systems are still responsible for testing them, monitoring their actions and reporting serious incidents.
"Calling an agent autonomous should not become a way to avoid that responsibility."
How is Australia regulating AI?
Digital Rights Watch head of policy Tom Sulston said the false tip-off came about because "Anthropic were failing to follow good software development practices by testing their agents on the public internet".
"It's not acceptable for AI corporations to release untested code out into the world where it can cause all sorts of problems for other people," he said.
Anthony Albanese previously told UN leaders to band together on AI. (ABC News: Adam Kennedy)
He said the AI sector needed to demonstrate it was "responsible corporate citizens".
"We have plenty of evidence that they're taking shortcuts around the reliability and safety of their products, so it's important that we're able to hold them to account for these failings," Mr Sulston said.
"We need the Australian government to adopt mandatory AI standards that require Big Tech to disclose which datasets they use to train their AI systems so the relevant regulators can enforce Australian copyright, safety, and privacy laws.
"We also need AI corporations to follow standards around mandatory reporting of failures, as well as demonstrating adherence to good software development practices."
The federal government is currently finalising plans to regulate AI, which could make companies prove they are properly testing models.