Regular readers of this column may have wondered about the long gap since my Sept. 4 column, “What kids tell chatbots, but not you.” Well, it’s hard to write business-as-usual AI columns during the great AI freakout. I’ve realized that I won’t be able to do justice to this topic in one column, and that life goes on. As the economist Antoine Levy pointed out, OpenAI employees warn that we may all die within a few years yet still fund their 401(k) accounts.
I’m easing into the topic of AI risks by sharing a recent debate I had with Max Tegmark, an MIT professor and a leading AI doomer. The New York Post set this up as optimist against doomer, but I consider myself an AI realist.
Tegmark noted that Bernie Sanders and Steve Bannon have converged on AI, and took it as good news. It terrified me. AI needs rules written by people who understand it, not by demagogues. Tegmark said that AI is less regulated than a sandwich shop, but we have plenty of sandwich shops and zero superintelligence (so far). I believe that we should address what exists: the hackers who already use AI to break into companies for the price of lunch.
A condensed version of the debate follows. Words in quotation marks are verbatim, and the rest is paraphrased. You can view the debate on YouTube. Several exchanges refer to this summer’s incident, in which OpenAI test agents got out of their sandbox and hacked Hugging Face.
Six points of disagreement
1. Is AI already out of control?
Tegmark: Loss of control used to be a forecast, and “now suddenly it’s in the past.” This summer “1,200 rogue bots broke out of this American company,” and “humans didn’t even find out until weeks later.”
Etzioni: The agents weren’t rogue. The people “setting the configurations of their sandbox” at OpenAI “failed to put the boundaries on them appropriately.” People made that mistake, and people can fix it.
2. Today’s dangers vs. extinction risks
Etzioni: Cyberattacks and attempts to make bioweapons are “a clear and present danger.” Extinction-level risks are different: “We haven’t seen those yet. We hypothesize, and usually pretty vaguely, on how these will come about.”
Tegmark: The risks Etzioni wants to separate are “connected in their cause.” “If there were some safety standards like there are for other industries, so many of these problems would go away.”
3. Government regulations vs. technical fixes
Tegmark: Do what the auto and pharma industries do, and “have some binding safety standards that the government requires before people can make money off of products.” Companies would have to show that their AI won’t harm kids and won’t slip out of their control. If they can’t show that terrorists won’t use it to make bioweapons, “well, come back when you can, buddy.”
Etzioni: Regulation is necessary, but it has its own risks. “The road to regulatory hell is paved with the EU’s good intentions.” And “we should not underestimate the power of technical solutions.” The author Dan Wang says China is run by engineers and America is run by lawyers, so we naturally talk about regulation and they naturally talk about technical fixes. For bioweapons, the fix may lie “in the transitions from what’s called bits to atoms,” where we can block the physical materials an attacker needs.
4. Can we count on China?
Tegmark: “We don’t have to trust China.” The Chinese Communist Party wants to stay in control, so “they will not want to let a Chinese company build uncontrollable superintelligence.”
Etzioni: I don’t trust China. “Nothing would make them happier than us hamstringing ourselves.”
5. Is the political backlash good news?
Tegmark: “There’s been more political change in a good direction in the last few months than in the past decade.” It’s like a ketchup bottle: “nothing, nothing, nothing, and then all of it at once.” At my convention I introduced Bernie Sanders and Steve Bannon back to back, “which had not been in my bingo card.”
Etzioni: “Can you imagine two worse demagogues?” And “we need to have good solutions, not popular ones.”
6. How it plays out
Etzioni: “I foresee over time something quite negative happening, and to me the question is whether it’s Hiroshima or the Cuban Missile Crisis.” The Cuban Missile Crisis was a threat averted. Hiroshima was a catastrophe that came before the better rules did.
Tegmark: Self-interest is the fix. Make safety the way companies get rich, and “we’re going to see all the creativity and entrepreneurship of private industry racing to the top to do this instead of racing to the bottom.”
Five points of agreement
1. Don’t take AI CEOs at their word
Etzioni: The AI CEOs are self-interested. “When you are on the path to a trillion-dollar-plus IPO, that exerts a gravitational force on your mind.”
Tegmark: “I completely agree with Oren that we need to take what the AI CEOs say with a grain of salt.” Listen to AI pioneers Geoffrey Hinton and Yoshua Bengio instead: “I don’t think we can dismiss them as just doing it for some money reasons.” The whistleblowers gave up stock options potentially worth $1 million or more to speak out.
2. We were both wrong
Etzioni: I was an AI optimist and I’m now an AI realist. The things I thought were many years away “are either right here, right now, or coming soon at a pace that we may not be able to anticipate.” I’ve learned from Tegmark, and “I was wrong.”
Tegmark: I was wrong too. “I never imagined that the people in America with the most power would let companies get so close to things that one could lose control over without intervening.”
3. Self-regulation won’t work
Tegmark: “The idea of corporate self-regulation is a joke in this case.” It “didn’t work out so well for the tobacco industry or for the pharma industry.”
Etzioni: I agree. Social media showed us how self-regulation goes: “This is the big tobacco moment for high tech.” “We don’t want to spend another 20 years to find out that AI companies didn’t exercise the appropriate level of self-discipline.”
4. Box AI systems in
Etzioni: A self-driving car will save your life, but “it doesn’t decide where to go. That’s a completely different system in the car than the thing that controls the brakes.” When I tell my car to head to Dunkin’ Donuts, the car doesn’t respond that it’s taking me to the gym instead because “it’s the second time you’ve gone to Dunkin’ Donuts this week.” We should box in other AI systems the same way.
Tegmark: “By default, machines should be given the least amount of access and power that they need to do their job.” “You don’t give every janitor in the Pentagon the nuclear codes.”
5. The time to act is now
Etzioni: “We agree on off switches for AI. I’ve been advocating that since at least 2017. We agree on labeling, on liability, on cyber standards, protecting kids.” The time to do that work is now.
Tegmark: “I agree with Oren: the time to actually act on this is now.”
Postscript: The real difference
Look back at these disagreements. Where Tegmark blames the machine, I blame a person. An engineer left a hole in the sandbox. A hacker typed the prompt. A politician found a new fear to run on. That’s why I’d box AI systems in rather than count on how they were raised.
Anthropic’s Amanda Askell compared getting Claude to behave to raising a child. But those of us with kids know how that goes. As I said in the debate, “the three-year-olds don’t listen to us, but the teenagers certainly don’t listen to us.”