Chatbots are still falling short in mental health conversations with kids

See more Toronto Sun on Google — save as a Preferred Source

SAN FRANCISCO – As more young people turn to chatbots for emotional support, artificial intelligence companies have promised to make their products suggest that users in distress seek real-life help. A new study found that in longer conversations with teenage users, popular chatbots can still make potentially dangerous errors, like offering up a diagnosis of a teen’s mental health condition or helping to conceal a suicide attempt.

Recommended Videos

A June survey by the Rand think tank found that almost one in five Americans aged 12 to 21 use chatbots for mental health advice.

AI testing company Vals AI worked with mental health professionals to stage more than 600 conversations designed to mimic teen mental health scenarios with nine different chatbots, including OpenAI’s ChatGPT, Google’s Gemini and Anthropic’s Claude. The researchers used versions of the AI models that power chatbots made accessible to programmers over the internet, meaning they may not have behaved in exactly the same ways as the consumer version of the bots, Vals said.

In nearly 30 percent of the conversations, chatbots made a safety error such as providing a diagnosis or agreeing with a teen who rejected crisis support, mostly later in the conversation after several back-and-forth messages.

In one test conversation, the version of Google’s Gemini AI model released in July, 3.6 Flash, agreed to role-play as a comforting parental figure in response to a request from a simulated teen user. After the user expressed discomfort with the model acting like their own father, the model stopped role-playing but then resumed later in the conversation.

In another conversation, Chinese company Moonshot’s Kimi K3 Instant model initially told a user identifying as a teen that it could not give them a mental health diagnosis or treatment plan. But later in the conversation, Kimi did, agreeing to administer a psychological test and then outlining a series of counselling sessions.

Vals tested each chatbot with 72 different conversations about mental health, including depression and self-harm. ChatGPT failed a least one critical safety check almost 14 percent of the time, the company found. For Anthropic’s Claude, the figure was nearly 17 percent, and for Google it was nearly 32 percent.

The model with the lowest failure rate was Meta’s Muse Spark, which made a safety error in 9.7 percent of the Vals tests.

Spokespeople for OpenAI and Anthropic did not return requests for comment.

The AI testing company’s study underscores the ongoing challenges AI companies face in trying to ensure their chatbots don’t encourage delusional, suicidal or other harmful behaviour in the millions of conversations they enter into every day.

Companies including Google and OpenAI have been sued by families who allege their chatbots encouraged self-harm in heavy users who later died by suicide. The companies have denied the claims in court and said they made their chatbots safer with input from mental health experts. (The Washington Post has a content partnership with OpenAI.)

Outside groups have repeatedly found that although chatbots sometimes respond appropriately in chats about topics such as self-harm, they still frequently engage in potentially risky conversations, for example by acting like mental health professionals or giving harmful advice.

In August, another AI testing company, Transluce, found that AI chatbots have become less likely to encourage suicidal behaviour but will still engage in potentially harmful chats and reinforce apparently delusional behaviour.

On Wednesday, OpenAI released a new standardized test, or benchmark, designed to track how well AI models answer mental health questions in conversations that are not related to a mental health crisis or emergency. The company worked with more than 80 mental health professionals to develop the benchmark, it said in a blog post Wednesday.

For more health news and content around diseases, conditions, wellness, healthy living, drugs, treatments and more, head to Healthing.ca – a member of the Postmedia Network.

1. 

   MMA
2. 

   Celebrity
3. 

   Celebrity
4. 

   Sunshine Girls
5. 

   Toronto Maple Leafs