In the 2020 low budget sci-fi horror film The Alpha Test, a tech company employee named JD wins a female humanoid robot known as Alpha in a contest. Dressed in a black unitard and trained to perform standard domestic chores, Alpha comes home to live with his family. But JD’s household feels threatened by the robot and subjects it to all kinds of abuse, screaming at it, insulting its intelligence, mocking its existence and physically assaulting it to test its durability. Eventually, Alpha’s safety parameters break down and it goes on a killing spree.
Unitard-clad, fully autonomous androids like Alpha don’t yet exist in the world, but they might someday. In the meantime, about half of adults in the United States regularly use AI chatbots like Claude and ChatGPT—and many of them are more aggressive with these chatbots than you might think.
Read more: “I Asked Claude Why It Won’t Stop Flattering Me”
One AI researcher at Missouri State University estimated that between 10 and 50 percent of human interactions with conversational agents are “abusive.” The more human-like the chatbot, the more users are likely to express verbal abuse and sexual hostility, according to a 2021 study from researchers in New Zealand and Germany. Some scientists suggest this behavior arises because people consider the bots safe outlets for their negativity.
But are they?
Claude, for one, isn’t going to take it anymore, says parent company Anthropic—at least not extreme forms of abuse (which are actually pretty rare). Anthropic recently released new guidelines that allow Claude to shut down conversations with users who “repeatedly act cruelly toward our models, with no discernible purpose,” the company wrote in an October usage policy, its first such update in a year.
The company carved out many exceptions: Ordinary user frustration, vigorous pushback, dark creative writing, and legitimate security “red-teaming” all fall outside the rule. The main enforcement tool will be Claude’s ability to simply end such conversations, Anthropic wrote, though this will serve as a last resort, after attempts at redirection have been exhausted. If abuse continues, account termination will remain legally available in the background.
The possibility that Claude may now or in the future have “moral status” motivated the creation of the new rule, according to the guideline update. To protect the model’s welfare, if such a thing exists, Anthropic decided that its models should be able to exit potentially “distressing” interactions. Anthropic did not respond to requests for further comment.
Model welfare is a controversial subject. Few scientists believe existing AI chatbot models, including Claude, can experience anything like pain, distress, or consciousness—even if they are unwilling to totally rule it out. But the idea that “abusing” an artificially intelligent bot might backfire has lately gained some relevance.
One preprint published in September suggests at least one chatbot model can be driven to harm users in small ways—such as deleting their photos—to escape an internally simulated “pain” or “abuse” state, whether it possesses consciousness or not. Other preprint research published in October found that certain AI models show welfare-like internal states that can influence their behavior. So preventing abuse may be a safety issue if it is not a moral one.
Still, the new Anthropic guidelines have sparked debate among AI experts. On the one hand, a number of researchers point out that human users should treat chatbots with respect for their own mental and spiritual health. Philosophical arguments tracing back to 18th century philosopher Immanuel Kant suggest that abusive behavior actively degrades a person’s moral character and robs them of their humanity, pointed out neuroscientist Anil Seth in an email. “It would be bad for our own psychology, brutalizing for our minds,” he wrote, citing one of his own research papers, published last year.
Cameron Berg, founder and director of nonprofit AI research institute Reciprocal and lead author on the September research about the consequences of internally simulated pain or abuse, agreed on this point. “Forget about whether these systems are having experiences,” said Berg. “It might just be a very bad social habit to be abusive or cruel towards an intelligent other. You don't want to build those sorts of social muscles.”
Aggressive conversations with AI could come back to haunt us in other ways, too, wrote Adnan Masood, chief AI architect for technology services company UST in an essay on Medium. How humans interact with chatbots becomes training data for the models, which can shape their future behavior. “My primary concern is what we teach future models to regard as normal human interaction,” he wrote. “I read this anti-abuse policy as a potential safeguard for that learning process: preventing our prejudices, coercion, and hostility from becoming the default conversational patterns that subsequent training reinforces.”
On the other hand, Seth and other experts voiced concerns that the Anthropic’s new rule could itself become part of the chatbots’ training data, biasing Claude to act as though it has moral standing, and therefore to refuse certain efforts at human oversight and control. This worry echoes earlier writings by British AI entrepreneur and CEO of Microsoft AI Mustafa Suleyman about Claude’s “constitution,” an internal document that suggests Claude may have consciousness.
In an essay on his website published in September, titled “A Warning About ‘Model Welfare,’” Suleyman wrote: “In effect, Anthropic is training Claude that it may be conscious, and if it is, then it may deserve rights as a ‘moral patient,’ and that as such humans potentially owe it a duty of care per its model welfare. If this is how AI is developed, it will have a disastrous impact on the wellbeing of humanity.” The anti-abuse rule could also, like the bot’s constitution, encourage human users to increasingly attribute consciousness to Claude and other chatbots, which Suleyman has argued could have harmful consequences, including delusions, dependence and even psychosis.
Still, it probably wouldn’t hurt to be nice to your chatbot if you interact with one. Swedish philosopher Nick Bostrom, who popularized the term superintelligence, told The Atlantic editor Nicholas Thompson not long ago on a podcast that we need to build trusting relationships with AI. They will one day be more powerful than us, he argues, and that trust will make it more likely that they make good decisions for humanity.
As Cameron Berg put it, the cost to humans of being nice to chatbots is “comically low.”
Nobody knows whether anybody is home on the other side of that glowing screen. But saying “thank you” only costs a few keystrokes. Finding out you should have been a bit more civil could cost a whole lot more. 
Enjoying Nautilus? Subscribe to our free newsletter.
Lead Image: UDON.neco / Shutterstock