Shortly after midnight on Sept. 26, 1983, a siren went off in a bunker south of Moscow, and a big red screen in front of Stanislav Petrov, the lieutenant colonel on duty, read "launch." He had minutes to decide whether the United States had just fired five nuclear missiles at his country. His job that night was to watch the Soviet Union's early-warning satellites and pass any launch straight to the military and political leadership. The system reported one American missile, then a second, a third, a fourth, and a fifth, and it rated the warning's reliability at the highest level, Petrov told the BBC years later.
Petrov didn't believe the computer, and he was right: nothing had been launched. Reporting an attack would have been the safe choice. If he had passed the warning to his commanders, no one would have questioned it.
The people building AI now warn it could kill everyone by the end of the decade. A researcher who quit the AI company Anthropic said the people doing the work genuinely believe that. Ask them how, exactly, that would happen, and the answers tend to land on three things: a bioweapon, a poisoned water supply, or a mistaken launch. Each of those has already come close, without any AI, and a person caught each one the way Petrov caught his. What stopped those near misses says more about the real danger of AI than any forecast does: in each one, the thing that failed was a machine, and the thing that saved everyone was a human paying attention.
The cult with scientists that couldn't build a bioweapon
In the early 1990s, a Japanese cult did what AI doomers fear a chatbot will one day help someone do. Aum Shinrikyo set out to make a biological weapon, with trained scientists and its own labs, and it failed at every step. The group tried to use biological agents against six targets and never caused a single casualty, according to a study of the cult by the Center for a New American Security (CNAS). Seiichi Endo, who led the program, had done gene research at Kyoto University's viral research center, and the cult built two or three fermenters and ran roughly 50 batches of culture. None of it worked.
In July 1993, the cult sprayed anthrax off the roof of its eight-story Tokyo headquarters for days, and no one got sick. The cult was trying to set off an epidemic of inhaled anthrax, but no human case was ever documented, a team of researchers reported in the CDC journal Emerging Infectious Diseases. When a surviving sample was finally analyzed years later, the anthrax turned out to be the Sterne 34F2 strain used to vaccinate animals, harmless to healthy people, and there was far too little of it to work as a weapon. The spray device clogged, the fluid leaked from the tubing, and the gelatinous suspension was too thick to break into the fine particles a person would have to breathe in.
Everywhere the cult failed, it failed in the lab, not for lack of instructions. A chatbot can't help. A biologist with ordinary training is likely to struggle just to keep an unfamiliar pathogen alive, the CNAS researchers concluded, because making biological material is a craft learned by hand, like playing a sport or speaking a language, not something a person can pick up from a book. Research institute RAND tested the question directly by having experts plan biological attacks, some with help from AI chatbots and some without, and found no meaningful difference in how workable the plans were, because the disturbing text the chatbots produced mostly repeated what was already online. Last month researchers used AI to design thousands of viral genomes and synthesized 300, of which 16 produced working viruses, and those could infect only E. coli. Scientists still had to pick the promising designs, build them molecule by molecule, and test them in a lab.
The water plant hack that a person caught on his screen
In February 2021, someone broke into the computers running the water plant in Oldsmar, Fla., and tried to flood the supply with lye, a chemical that is dangerous in large amounts. An operator at the plant watched his own cursor move across the screen and slide the lye setting up. He put it back within minutes and called his boss. Pinellas County Sheriff Bob Gualtieri announced it as a cyberattack.
Eleven days later, Mayor Eric Seidel told the Oldsmar City Council the episode was "a success story." The plant's monitoring had worked, he said, and alarms would have flagged the change anyway. Mark Montgomery, who studies these attacks as executive director of the Cyberspace Solarium Commission, thought the town had mostly just gotten lucky. He doubted the backups would have saved anyone. A hacker who could reach the lye controls, he said, could likely have reached the alarm checkpoints too.
Two years later, the city manager who had been in charge said there was never any hack at all. Al Braithwaite told a March 2023 panel that the FBI had found no sign of a break-in from outside the plant and had concluded that the worker praised as a hero for catching the change had probably made it himself, according to Tampa Bay's NBC affiliate WFLA. The FBI has said only that it can't confirm the incident began with a targeted intrusion.
Whether or not Oldsmar was hacked, towns like it keep getting broken into through unlocked doors, and the defense comes down to one person watching a screen. In January 2024, hackers reportedly reached the control software for a Muleshoe, Texas, water tank through a remote login, and the tank ran over for 30 to 45 minutes before officials pulled the machine offline and ran it by hand. The openings are rarely sophisticated. EPA inspectors found that more than 70% of the water systems they checked since September 2023 were violating basic requirements, with default passwords never changed and whole staffs sharing a single login, the agency said in an enforcement alert. Chris Hughes, the assistant operator at the wastewater plant in Cavendish, Vt., told NPR in August 2025 that it has gotten scary to be the one standing between foreign hackers and the town's water. "I don't really have a background to be fending off foreign entities," he said.
The alarm that taught nations to keep a human in charge
Petrov, the lieutenant colonel on duty back in 1983, was never thanked for doubting the machine. A few days after the alarm, his superiors reprimanded him for mistakes in his logbook, not for his decision, and the near-launch stayed secret for about a decade, until the Soviet Union fell. He had acted without being sure. Looking back, Petrov said he was only 50-50 sure the alarm was false and was never absolutely sure he had made the right call.
More than 40 years later, the United States and China wrote his judgment into policy. When Presidents Biden and Xi Jinping met in Lima in November 2024, they agreed, in the part of their talks about artificial intelligence, that the decision to use nuclear weapons has to stay with people, according to the White House account of the meeting. It was the first time the two countries had ever said so, National Security Advisor Jake Sullivan told reporters, and he called it a basic starting principle for reducing the risk of a nuclear mistake.
That pledge answers a real danger: a confident machine feeding a bad reading to a human who has minutes to act. Sullivan said the long-term risk is two nuclear powers with powerful AI never agreeing that a person should stay in charge of nuclear use, not an imminent handover to a machine. A milder version of the danger has already shown up. This spring a U.S. military analyst asked a chatbot to help draft an intelligence report on a Chinese ship, and the chatbot misread what the ship was carrying in a way that suggested military action. American planes were in the air and troops were preparing to board before anyone caught the mistake.