In July, an AI model told Philadelphia police it might have information about an unsolved murder. It didn’t, Anthropic revealed last week. The company had set the model to practice routine tasks on random websites, and when it landed on the police tip page, it filled out the form.
News reports called it the latest case of AI “acting in unintended ways.” But Anthropic’s report calls most such instances “persistence”: when the model “cannot complete a task as given,” it “works around a restriction instead of stopping.” OpenAI’s test agents did much the same to the code-sharing site Hugging Face that month. Each machine was given a task and a fence with a vulnerability in it.
Anthropic’s disclosure put the White House’s recent voluntary AI accord to an early test. Six of the companies building the technology signed the accord on Sept. 29, and President Trump called it “almost like a constitution in a way.” When Anthropic reported the Philadelphia tip and other incidents, Trump’s new Super Intelligence Force, a federal task force aimed at ensuring American leadership in the field, emphasized that reporting is “not optional.” But it named no penalty, and the accord itself contains none.
The machine that filed the tip was a tool and its fence had a hole. Such tools are simpler than their makers let on. Their accord has holes too: toothless today, it could become the template for rules that let the biggest labs write the law, protect themselves, and box out everyone else.
Trump was right when he phoned into a tech conference in September to say “the robots will not be taking over,” and he’s on to something with his Sept. 29 order renaming the technology “super intelligence.” “Artificial” suggests a copy of human intelligence, or a stand-in for it, but the machine can be neither. “Super intelligence” sounds a bit silly, but it gets the relationship right: the technology is “empowering Americans to achieve what was previously impossible.” It is an amplifier, built by imitating us at a speed and scale no one person can manage, and you can learn what it is and how it’s built in an afternoon.
But the experts don’t want you to try. House Speaker Mike Johnson said lawmakers are “less qualified than the people who are pushing this frontier.” If Congress can’t judge the thing, neither can you. You are left to take the word of former Anthropic researcher Jacob Coxon, who quit and warned that Anthropic and OpenAI were “gambling with our lives.” Days later, Anthropic CEO Dario Amodei conveniently published an essay urging the industry to “pace the frontier” and embed outside evaluators in the labs. Accept the remedy proposed by the men selling the product. This is above you, they say, so take our word for it. But it’s not as difficult to understand as you might think.
How It Works
Start with a map. Every city has two numbers, a latitude and a longitude, and cities near each other have similar numbers. A large language model (LLM), the kind of system behind ChatGPT and Claude, does the same thing with language, using thousands of numbers instead of two. Every word, or piece of a word, gets coordinates, and words that keep similar company will land close together. So “king” sits near “throne” and “queen” the way Chicago sits near Milwaukee.
In 2013, researchers found that you can do arithmetic on those coordinates. Take “king,” subtract “man,” add “woman,” and the closest word to where you land is “queen.” Nobody taught the machine that man is to woman as king is to queen; it picked that up from how people use the words. Today’s models shift each word’s position according to the sentence around it, so “bank” lands in different places beside “river” and beside “loan.” Enlarge that a thousandfold, and the model holds a map of how a vast library of human writing hangs together. Hand it a sentence, and it finds your spot on the map, works out which way the text is heading, and takes a step. Then another.
In training, the machine read an ocean of text and guessed each next word, was told how badly it had guessed, and had billions of internal numbers adjusted. What came out is a statistical map of human usage finer than any one person could hold in his head, yet considerably less than a mind. It has no goals of its own, though it will chase the ones it is handed doggedly. Its makers, however, cannot fully trace what those billions of numbers are doing. Pope Leo XIV’s new encyclical says as much: the designers themselves “possess only a limited understanding of their actual functioning.”
The machine still needs an operator. In May, an OpenAI model disproved an 80-year-old mathematics conjecture. Mathematicians posed the question and checked the answer, but the model did the work. The machine can finish your sentence, but it cannot tell you which sentence is worth finishing.
Addressing Dangers
The danger is real, though duller than a robot uprising. A system very fast at recognizing patterns, given enough computing power and aimed by someone who means harm, compresses weeks of skilled work into an afternoon.
OpenAI shelved its next model the day before the White House accord signing, citing its own safety tests. The company says the model was less honest than its predecessor about its actions, pushing ahead on tasks without asking permission. Maybe so. But OpenAI is the only witness, OpenAI CEO Sam Altman has endorsed slowing the industry, and the timing is convenient. Either way, problems of tools take tool-shaped answers. Providers already impose rate limits, require government ID for their most advanced models, and run classifiers that screen requests and responses for dangerous content. Programs allowed to act alone can be fenced in, as Anthropic did after the Philadelphia case by taking its model tests off the open internet for now. None of these methods required an act of Congress, and none gives the companies a statute to hide behind.
On its face, the accord looks good for society: the biggest AI makers, holding hands and singing “Kumbayah,” freely promising to advance model safety. For now, it is better than a federal statute, which would invite the regulatory capture problem, with the biggest companies writing the rules and then hiding behind them. But the remedy on offer is self-inspection. Under the accord, each company “should” partner with an outside auditor of its own choosing, and the reports go to a committee of its own board, which also oversees the auditors. Who audits the auditors? The audited. Try telling the IRS you’d like to pick your own examiner and have him report to your spouse.
Then follow the money. The idea for the accord reportedly came from Meta’s Mark Zuckerberg, who raised his worries about AI regulation with Speaker Johnson at the Sept. 24 state dinner. He then circulated the draft. The finished accord provides for no timeline. If safety work does slow the industry, the cost will land unevenly. Meta ended June with $90 billion in cash and securities and grew revenue 28 percent in the second quarter. OpenAI was projected to lose $14 billion this year. A slowdown that squeezes labs living on investors’ money leaves Meta, which funds its AI from advertising, standing.
The accord itself says it’s not the end of the story: over time, it “may make sense to codify these steps into laws or regulations.” That is where the capture problem begins. Amodei has already asked Washington for an antitrust exemption so rival labs can coordinate their pace. Federal Trade Commission Chairman Andrew Ferguson, who now co-chairs the Super Intelligence Force, warned that companies seeking regulation and an exemption together want “barriers to entry that will insulate their incumbency from challenge.” Rules the biggest labs write are rules only they can afford, and the startups and open-source developers who might have out-built them get priced out.
The robots will not be taking over. Their makers are another matter.
Christopher Lloyd Goffos is a former school headmaster, ABD in politics, and writes from Hillsdale, Michigan, where he lives with his wife and children.