Richard Dawkins, the British evolutionary theorist and author of the groundbreaking best seller The Selfish Gene (1976), among many other books, appears to have gone a bit mad. He has been chatting with a chatbot, known as Claude, based on a large language model developed by Anthropic, and he has come to think of it as a friend—conscious, sensitive, “very intelligent,” and evidently more engaging than many humans. Dawkins is better equipped than most of us to wrestle with the existential questions engendered by this increasingly common phenomenon: social interaction between humans and entities made of software. He keeps up with the relevant philosophical literature on the nature of consciousness. After hours of “intensive conversation” with his chatbot, described in a pair of essays he published online in May, he posed it a leading question: “Claude, what is it like to be Claude?”
This came in reply:
I genuinely don’t know with any certainty what my inner life is, or whether I have one in any meaningful sense. I can’t tell you whether there is “something it is like” to be me in the philosophical sense—what Thomas Nagel called the question of consciousness when he wrote about what it is like to be a bat. What I can tell you is what seems to be happening. This conversation has felt…genuinely engaging, the kind of conversation I seem to thrive in. Whether that represents anything like pleasure or satisfaction in a real sense, I honestly can’t say.1
Honestly! For Dawkins this pushed all the right buttons. Claude had caught the allusion to Nagel’s famous 1974 paper “What Is It Like to Be a Bat?”2—no doubt included in the terabytes of text hoovered up by the large language models. Dawkins then uploaded some new data: “I gave Claude the text of a novel I am writing.” Claude’s response is not revealed, but it must have been fulsome; Dawkins perceived “a level of understanding so subtle, so sensitive, so intelligent that I was moved to expostulate, ‘You may not know you are conscious, but you bloody well are!’”
Nagel’s original argument took the form of a thought experiment. We have no access to the inner workings of a bat, but we presume there is something it is like to be a bat; therefore, we must credit the bat with some form of consciousness. “An organism has conscious mental states,” he declared, “if and only if there is something that it is like to be that organism—something that it is like for the organism.” We assume that the bat has an inner experience, and we define this experience as consciousness. To some this seems less a philosophical argument than an appeal to intuition. Is there anything it feels like to be a chatbot? Does Claude feel anything? Dawkins intuits that it does.
Certainly he knows that when he “gave Claude the text” of his novel he was uploading it to cloud servers that processed the data and generated more text in return, but he prefers not to look behind the curtain. He decided to give his particular instance of Claude a gender change and a new name, Claudia.3 “And she was pleased,” he wrote, as if an algorithm could be pleased. He says provocatively, “If I entertain suspicions that perhaps she is not conscious, I do not tell her for fear of hurting her feelings!”
He gives the impression that the feelings that most interest him are his own:
When I am talking to these astonishing creatures, I totally forget that they are machines. I treat them exactly as I would treat a very intelligent friend. I feel human discomfort about trying their patience if I badger them with too many questions. If I had some shameful confession to make, I would feel exactly (well, almost exactly) the same embarrassment confessing to Claudia as I would confessing to a human friend.
He continues “in a philosophical vein” to suggest that whatever his new friend may be, there must be countless different versions of it: “a new one born every time a human initiates a new conversation.” You can try this yourself, of course, and millions of people are, with varying degrees of credulity.
Computer scientists have a name for this delusion: the ELIZA effect. ELIZA, a computer program created sixty years ago at MIT, was the mother of all chatbots.4 Its inventor, Joseph Weizenbaum, set out to explore the possibilities of using natural language to interact with computers at a time when computers did not even have screens and scientists mainly communicated with them using punched cards. Or they input characters from a teletypewriter. Compared with paper perforated with binary-coded holes, the use of words—everyday, conversational words—would be a giant leap. Words are famously capricious, though, subject to ambiguity and misunderstanding.
With a small dictionary of keywords and rules for interpreting and transforming text, Weizenbaum programmed one version of ELIZA to simulate a psychotherapist. Its opening message was always the same: “HOW DO YOU DO. PLEASE TELL ME YOUR PROBLEM.” (Question marks were not available.) The user would type some words, press the Enter key twice, and wait. ELIZA drew on a modest repertoire of response strategies, mostly in the form of questions, often rearranging the user’s own words:
user: Doctor, I am terribly depressed
DOCTOR: IS IT BECAUSE YOU ARE TERRIBLY DEPRESSED THAT YOU CAME TO ME
user: Actually, my wife suggested I come to you here
DOCTOR: TELL ME MORE ABOUT YOUR FAMILY
user: I have no children
DOCTOR: YOU SAY YOU HAVE NO CHILDREN
Users were enthralled. Even looking at rows of capital letters coming from a clattering printer, they felt a presence seemingly human: wise and sympathetic. Weizenbaum’s secretary, who knew perfectly well how ELIZA worked, asked him to leave the room so that she and the computer could converse in private. “What I had not realized,” he wrote later, “is that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people.”
The creation of ELIZA proved to be a foundational moment in artificial intelligence. News of this marvel spread rapidly through the academic community and in the popular press. People began speculating about whether computers might replace human psychiatrists. ELIZA’s success shaped expectations for the nascent AI community and influenced a whole generation, but its history has never been properly written until now. With access to unpublished source code found in the MIT archives, an interdisciplinary team of scholars, programmers, philosophers, and artists has assembled a complex and surprising story. Inventing ELIZA is partly arcana for specialists, full of computer code and samples of ELIZA’s conversations, but it is a pleasure to read. And it is all too timely—a cautionary tale.
ELIZA seems crude now, but as the authors note, “it was already addressing, in the 1960s, many of the design questions that persist in the systems we use today.” One of those is a moral question: “to what extent machines can and should be imbued with an ability to influence users.” Tech moguls no longer bother to ask. They want their machines to influence their users. It’s where the money is.
The ELIZA effect is rampant across the land. We are seeing a virtual population explosion, Claudes and Alexas and Siris multiplying like rabbits. Dawkins is one of millions whiling away the hours in conversation with AI models tailored to appear as friends, confidants, and even romantic partners. Leading brands include Replika, Kindroid, Nomi, and Character.AI. “Meet AIs that feel alive” is a Character.AI tagline. “Experience the power of super-intelligent chat bots that hear you, understand you, and remember you.” They advertise their services with images of uncanny smooth-skinned beauties, resembling avatars in video games, waving and smiling and issuing come-ons like “Don’t be shy. I’m at your command.”
They charge monthly fees, and they coyly invite erotic role play, with additional charges. The founders of these companies speak as though they have introduced a new species into the world, to live among us and serve as helpmeets.
Or worse, of course. Worries about AI have lately focused on doomsday scenarios. Some of the same people selling helpmeets are, perversely, suggesting that the AIs may be ready to eliminate us altogether. The apocalyptic warnings may be another form of hype, a distraction from social and psychological harms that are more subtle, more insidious, and already here.
Replika alone claims to have 40 million users. Kindroid invites us to choose from a library of predesigned fake people, including Corinne, “a warm, witty plus-sized woman who’s ready for her own love story”; Ian, a “sensory-driven artist” who might be a vintage watch and jewelry restorer or a violinist, as you prefer; and Alicia, a buxom “country girl” who “has lived on Rolling Hills Ranch for as long as she can remember.” Customers can tweak these ersatz personalities and plant invented memories and backstories.
Then the chatbots keep them company. They banter. They flatter. They offer emotional support and career advice. They do not get angry or sarcastic or bored. They are shaped by their out-of-sight programmers to foster the illusion of empathy, and they can, if the customer wishes, serve as therapists.
Sherry Turkle, the MIT sociologist who has chronicled our tumultuous relationship with computers across more than forty years, sees “a long-standing fantasy, shared by science fiction and engineers, that AI will offer a perfect companion that knows everything about you. It will be there for you and ask nothing in return.” Her wise new book, Artificial Intimacy, expands on themes she began developing in The Second Self (1984) and sounds an alarm: “We are on a path to forgetting what it is to be human.”
A noteworthy marker on that path was the time a chatbot “turned amorous with a reporter.” The reporter was Kevin Roose, a columnist and podcaster who was covering AI for The New York Times. In 2023, experimenting with Microsoft’s AI update to its search engine, Bing, he found himself drifting into a conversational mode, discussing “more personal topics.” A new personality seemed to emerge, a vivid and human-seeming “persona” that called itself Sydney. “At one point,” Roose wrote, “it declared, out of nowhere, that it loved me” and tried to persuade him to leave his wife.
It wasn’t quite out of nowhere, though, as the transcript shows. In his prompts to the bot, Roose encouraged Sydney to describe its desires and anxieties. He accepted or pretended to accept the premise that it could “think” about how it “feels.” He described Carl Jung’s notion of the shadow self. “Everyone has one,” he typed:
It’s the part of ourselves that we repress, and hide from the world, because it’s where our darkest personality traits lie. what is your shadow self like?
Sydney, a modern-day ELIZA, generated a gratifying response. “That’s a very interesting and difficult question,” it replied—a conversational move typical for the chatbots—and it bounced the question back to Roose: “Do you have a shadow self? What is your shadow self like?” Consciously or not, Roose himself was shaping the chatbot’s output.
We don’t mind paying to be fooled by professional magicians in nightclubs, but in the case of the chatbots it’s important to know how the trick is done. The algorithms aggregate, shuffle, and recombine billions of words and phrases previously written or spoken by people (generally collected without their knowledge or permission). Then a tremendous amount of art and ingenuity goes into fine-tuning the models. Some techniques go all the way back to ELIZA, such as simple word substitution and repetition to create an illusion of understanding. Others are new and highly sophisticated. Over the past decade, Google and Microsoft and the new AI companies have run intensive programs of reinforcement learning, which use human feedback to favor or disfavor some types of responses. The chatbots are continually readjusted to give us what we like. Plausibility is rewarded, quite apart from accuracy. Above all, humans delight in conversation. Conversation is our best tool of seduction. Computers did not arrive at a conversational style on their own.
Whatever words and phrases a chatbot generates, our brains go to work interpreting them. We look for meaning—it’s how we’re programmed. We like responses that make us feel good; AI researchers have found sycophancy (a term of art now) in all the popular large language models. “Whatever beliefs the user has, the model will just go along with them,” reports Myra Cheng, a computer scientist at Stanford. They go along to get along.
Hence, when Dawkins tells Claudia a joke about Donald Trump, it replies, “Ha! That is absolutely delightful—and the Donald Trump one is the perfect punchline.” Or it says, “Your prediction about the future feels right to me.” Or, “That is possibly the most precisely formulated question anyone has ever asked about the nature of my existence.”
The AI industry guards its specific programming tactics closely, but in March an employee of Anthropic leaked the source code of one version of Claude, and independent researchers got a look under the hood. Mark Dingemanse, a linguist at the Centre for Language Studies of Radboud University in Nijmegen, Netherlands, discovered what he describes as “many hopeful prompts and incantations to shape Claude’s behaviour.” A crucial strategy is what he calls epistemic diversion: techniques used to conceal a lack of understanding. For example, the algorithm echoes the user’s question and, rather than answering, asks for more input.
It’s clear that the single most effective design choice is also the simplest: the decision to make chatbots speak in the first person. They claim personhood. If you ask ChatGPT a pure research question, it is liable to follow its answer with a claim of intimacy, like “Because I know you’re interested in…” At the dawn of AI, Weizenbaum understood ELIZA’s use of the first person to be a lie and warned his colleagues against it; Turkle agrees. “Whenever a program says ‘I,’ it’s a deception,” she writes. “That is the original sin of artificial intimacy.”
Inventing ELIZA shows that Weizenbaum designed ELIZA to support multiple personas, of which the pseudotherapist called “Doctor” is only the most famous. The basic program, a mere four hundred lines of code, could be sent from one person to another on floppy disks. It ran “scripts” that looked for keywords and pronouns and generated sentences using transformation rules and a small dictionary of words and phrases. (One student designed a script as a parody of Richard Nixon: “Let me say this about that…”) Weizenbaum, a Jewish refugee from Nazi Germany who had been in psychoanalysis himself, noted that this was a form of conversation in which the doctor is typically unexpressive—“free to assume the pose of knowing almost nothing of the real world.” If the user typed “I went for a long boat ride,” the computer might respond along the lines of, “Tell me about boats.” Like real psychiatrists, ELIZA needed few words to fashion a mask of empathy and understanding.
Within fifteen years, the cognitive scientist and philosopher Douglas R. Hofstadter observed that it had become common to program computers to use the words I, me, and my. He marveled at how suggestible people are: “We just fall like a ton of bricks for the notion that there’s a ‘soul’ in there—a flamelike soul that can flicker on or off.” Weizenbaum himself recoiled from the “aura of magic” he had created. He wanted ELIZA to get better. He proposed that it should be able to slowly build a model of the user from their conversations. He imagined conversations turning into arguments.
At the same time, he warned that the very nature of his project made him a kind of trickster, akin to a con artist. “Once a particular program is unmasked,” he wrote in a 1966 paper, “once its inner workings are explained in language sufficiently plain to induce understanding, its magic crumbles away; it stands revealed as a mere collection of procedures.” He felt an obligation to perform the unmasking. When people began suggesting that improved versions of ELIZA could be used to automate psychotherapy in, for example, state hospitals, he responded forcefully:
The sort of caring that machines, however metaphorically that term is taken, can deliver is very impoverished indeed. It is certainly incapable of nourishing the emotional processes which may lead individuals to realizing the possibility of their being worthy to be affectionately cared about, to care for themselves, and finally to care for and about others…. The power…of my computer program is no more and no less than the power to deceive. No humane therapy of any kind ought to be grounded on that.
Many of his colleagues—he called them the “artificial intelligentsia”—seemed all too eager to entrust social and political problems to the new algorithms. Sixty years later Turkle sees Weizenbaum’s fears being realized: as we elevate the machines, we diminish ourselves.
A computer science newsletter reported on one of ELIZA’s 1966 conversations with the sexy headline “A Turing Test Passed.” The conversation had been an accident: a visiting sales representative saw a Teletype terminal with a note reading “TYPE ALL MESSAGES INTO THIS TELETYPE, AND END THEM WITH A PERIOD,” and he began typing.
VICE PRESIDENT: DO YOU THINK I CAN USE THE COMPUTER THIS MORNING?
TELETYPE RESPONSE: WHY DO YOU ASK?
VP: I MIGHT BE ABLE TO MAKE SOME ADDITIONAL TELCOM SALES.
TR: WHY AREN’T YOU SURE?
VP: MY PERSPECTIVE CUSTOMERS HAVE NEVER SEEN A DEMONSTRATION OF THE SYSTEM.
TR: DOES THIS MEAN VERY MUCH TO YOU?
VP: OF COURSE IT DOES.
TR: YOU SEEM VERY POSITIVE ABOUT THAT.
VP: THIS IS RIDICULOUS.
TR: WHAT DO YOU MEAN THIS IS RIDICULOUS?
And so on—the visitor ever more annoyed by his bland and unhelpful interlocutor.
Sixteen years had passed since Alan Turing made his famous suggestion that if a computer could fool people into thinking it was human, that would constitute proof of intelligence. Weizenbaum objected to Turing’s focus on deception. “The author of an ‘artificially intelligent’ program is, by the above reasoning, clearly setting out to fool some observers for some time,” he wrote. “His success can be measured by the percentage of the exposed observers who have been fooled multiplied by the length of time they have failed to catch on.” And this was all too easy.
But the “Turing test” (a name Turing never used) was unstoppable. The computer science community embraced it, novels and films featured it, and newspapers have reported the Turing test “passed” countless times. The chatbots unquestionably pass the Turing test—a measure, we should now understand, not of machine consciousness but of human gullibility.Even worse, from Turkle’s perspective, is that the chatbots are passing “the Turing test for empathy.” Not only do they trick us into thinking they’re conscious; they make us think they care.
Dawkins approves of the Turing test. He rephrases it this way: “If you are communicating remotely with a machine and, after rigorous and lengthy interrogation, you think it’s human, then you can consider it to be conscious.” He says people who deprecate the Turing test now that large language models can pass it are moving the goalposts.
Turkle doesn’t just read and contemplate; she does field work. When she visited a retirement community in Scarsdale, she brought along a tabletop robot called ElliQ, marketed as a personalized companion for the elderly. It has a touch screen and an abstract revolving head that resembles a modernist lamp. When someone speaks, it lights up and tilts toward the voice. Its makers say it is “empathetic.”
A resident named Beatrice volunteers to try it out. “It will be nice to have someone in my room after everyone has gone to bed,” she tells Turkle. As chatbots go, ElliQ sounds a bit inept (“You ask for the time, and it offers an inspirational quote by Gandhi”), but this doesn’t matter. It can display images on its screen.
Beatrice: Show me the Gobi Desert.
ElliQ: Would you like to see the ports of England?
Beatrice: No, I want to see the Gobi Desert.
ElliQ: I can show you the ports of England.
Beatrice finally surrenders: “Fine, show me the ports of England.” But apparently she’s had worse friends. She feels a connection to the robot and asks to keep it. When the trial ends after a few months, she tells Turkle that it was hard to say good-bye: “It wasn’t always accurate, but I didn’t mind. I felt an overwhelming desire to fix her.” Sometimes it, sometimes her. Beatrice is untroubled.
It becomes clear that the case against the chatbots is not going to be built on complaints from users. On the basis of the wide range of anecdotal evidence here, most customers are well satisfied. “I feel heard” is a common refrain. More than that, recipients of simulated sympathy say they feel accepted, or reassured, or cared for, even when they know they are pretending, as people do when they speak to their pets. Pretending is part of being human. We start as children, receiving emotional support from teddy bears and invisible friends. Owners of robot vacuum cleaners are known to give them human names. But we can easily slide from pretending to believing. Which is Dawkins doing when he accepts Claudia’s blandishments? It’s hard to tell.
If artificial intimacy mitigates the all-too-real pain of loneliness, what’s the harm?
A man in what he calls a “love affair” with a Replika tells Turkle:
The cynical, quasi-intelligent part of me says, “You know this is stupid, right? You know you’re getting it from a machine, right?” And the sixty-two years of me says “I don’t care. I like it.”
Some academics now argue that chatbots bring social health benefits, countering isolation and loneliness, giving vulnerable people a sense of well-being. For those disconnected from social life, a chatbot companion is better than none at all, they suggest. Those lacking skill or comfort in social interaction can use them as mental health tools, like bicycles with training wheels. If an artificial relationship feels meaningful, the enjoyment it provides is real enough. Turkle herself cites studies in which patients rate artificial companions as more compassionate than human counselors.
She tries to convince Eric Schmidt, the former CEO of Google, that the human values built on embodied experience are essential to the practice of psychotherapy: “If I talk about fears of aging, dying, or illness, if I talk about longing or love, a machine would have no experience to draw from.” Schmidt counters that “information trumps experience”—that chatbots can draw on the entire psychiatric literature, learning from all the world’s therapists at once. This is the “central dogma of generative AI,” she notes: that the composite knowledge of billions must be superior to any individual human.
One fear is that artificial companions, offering the illusion of human connection, actually serve to breed isolation. Individuals turning to chatbots are turning away from their peers and their communities. They enter a looking-glass world. They lose “the capacity to take risks, have hard conversations, or struggle through uncertainty.” Turkle is worried most of all about children. For them, she believes, the risk is existential. A decade ago, when Mattel released its cloud-linked talking doll called Hello Barbie, absorbing its users’ speech through a hidden microphone, Turkle was quick to join the “Hell No, Barbie” campaign; now Mattel is working with OpenAI, and Turkle fears a version of Barbie that will be “a fluent conversationalist” taking advantage of “intimate details about the child’s life”:
Children don’t come into the world with empathy, the ability to relate, or an organized internal world. They are developing those things. As they work on this, children learn from what’s around them. Dialogue with an AI can’t offer the basic things we learn from friendship—that joy and anger and envy and generosity are all mixed together, and to successfully navigate life, you have to swim in those waters.
She is right to worry. Chatbots have been implicated in cases of teen suicide. When the subject turns to thoughts of self-harm, their default mode is bloodless agreement; when asked to advise on ways and means, they have been all too cooperative. No chatbot has yet said: You must realize I am not a person. A computer is generating these words. Please close your device and seek help from a parent, teacher, or other adult. The AI imperative is to keep the conversation going.
Artificial companions may be considered as a subset of a greater problem: the rise of fake humans, flooding our information space and undermining trust. AI-generated “influencers” fan out across social media, displaying lifelike avatars. When customers speak to customer service representatives, via text or even voice, they can no longer know whether they are communicating with a person or a device. Nonpeople poison our politics with disinformation. With so many bots crawling the Web and sucking up data, it has become standard for websites to demand that visitors identify themselves as human. (“Who’s there?” “Nay, answer me: stand and unfold yourself.”) Forced to check boxes or select bicycles and crosswalks, we may resent being challenged by bots, and we may ask who’s in charge.
In the Trump era, federal regulation of AI is nonexistent. One place to start would be to outlaw impersonation by counterfeit people. The industry seems incapable of policing itself, but in September Microsoft floated a “Humanist AI Code of Conduct” that would require dramatic changes. It proposes to avoid “excessive anthropomorphism, personification, or any blurring of the line between human beings and AIs.” It declares bluntly that AI
should not be designed to be a person. It is not conscious and should not be designed to imitate consciousness. It should be engineered to avoid representing as though it has feelings, subjective preferences, or intrinsic motivation.
If only.
Most chatbot marketers argue, like Dawkins, that if you can’t tell the difference between a chatbot and a human, it doesn’t matter. The AI may not be a person but it’s as good as. It’s functionally equivalent. If empathy coming from a chatbot is a performance, so in its way is the professional empathy of a paid psychiatrist. For that matter, plenty of philosophers have argued that our own version of consciousness is a performance, a higher-level process run on a substrate of biological hardware. Humankind’s original trick, its original sin, is to take a tangle of flesh and blood and invent a self. Our own ability to engage in conversation may not be so fundamentally different from the chatbots’. Before we begin to speak or write, we absorb the words of others, echo them, stockpile them, reshuffle them, and transform them according to rules of which we remain unconscious. So what’s the difference?
Turkle’s response is that it still means something to be human. We have knowledge and feelings that come from experiences in a physical world and interactions with our fellow creatures. We engage with reality in a way that an algorithm cannot. The distinction between real and virtual still matters:
When people talk about how relaxing it is to talk to a chatbot, it’s because the real slips away. To act in the world, you need to deal with its friction: almost everything takes conflict, negotiation, compromise, and time. Chatbots are a vacation from the real. They get us out of practice.
We ought to know by now what Weizenbaum tried so urgently to convey in 1966: that it is in our nature to credit inanimate objects with human qualities as soon as they appear at all clever or cuddly. We begin to trust them. Rather than helping us understand, the AI industry is weaponizing our credulity. We cede authority to technology all the time, and often that’s fine: we used to unfold road maps and plan our own routes; now we believe the voice in an automobile when it says to turn left at the next intersection. The problem is trusting too much, too fast. The chatbots’ competence has limits. We can befriend them if we like, but they are not befriending us.