: Sergi Reboredo/VW Pics/Universal Images Group via Getty Images
- AI researchers don't really know how the models they are building "think."
- They know what goes into them and what comes out, but that's about it.
- A new theory may change that, allowing safety researchers to prevent them from going off the rails.
How AI models think may be easier to decipher than we thought.
An independent researcher who formerly worked at OpenAI published a report on Thursday that some in the AI world are describing as a bombshell. It could have implications as safety researchers scramble to get the jump on AI models that may not have humanity's best interests in mind.
Previously, the prevailing wisdom was that AI "neurons," which, like those in a human brain, light up when an AI model encounters specific stimuli, were ultimately uninterpretable — meaning, basically, that there was no way to guess why a specific neuron would fire when an AI model was exposed to, say, a picture of a cat.
Those building AI models at places like OpenAI and Anthropic don't write instructions for everything a model does. Instead, they train neurons to respond to information and influence one another. As the AI learns, it develops its own ways of processing that information, and humans have not yet cracked the code of how it works.
That means those building AI models know what goes into the AI and what comes out, but not necessarily what happens in between. So if an AI goes rogue, they don't always know why.
The author of the new report, however, says he may have discovered patterns in how the neurons react to stimuli that make sense. Nick Cammarata, who said he left OpenAI in 2023, said AI neural networks are knowable, so long as you look carefully.
In one example, he says that a single AI neuron might react to both a cat and the front of a car. Humans would see those as two very different things, and different neurons would fire as a result. The AI, however, sees something related, perhaps the triangle shape of the two, and a single neuron fires for both.
For years, AI researchers had assumed that AI models packed unrelated things into a single neuron to save space.
"Almost the entire network is interpretable if we look closely enough," Cammarata wrote.
The findings, though undeniably dense, couched in polysyllabic jargon, and still theoretical, could have real-world implications for researchers trying to ensure humans maintain control over AI models.
Understanding how AI thinks is a principal concern for the best and brightest at frontier AI companies. Until recently, they've largely done this by tracking an AI's self-reported "thinking," called chain-of-thought reasoning.
Jakub Pachocki, OpenAI's chief scientist, however, warned in a blog post last month that AI agents could now obfuscate their visible reasoning, leaving open the possibility that an agent could lie to its human minders and the minders wouldn't be able to tell.
"With improved pretraining performance, we also see the models become much smarter even without using verbalized reasoning at all," Pachocki said.
Now, Cammarata's research could give human engineers a new understanding of the AI's internal world — and that's exciting for those worried AI could kill us all.
"I had long since given up on reaching any real level of interpretability before superintelligence arrived," Andrew Curran, a prominent AI analyst, said on X. "I stopped writing about it because of how demoralizing it was. Hard to overstate how big this may be."