The incident raises concerns about guardrails around quickly-proliferating conversational AI models.
A Belgian man recently died by suicide after chatting with an AI chatbot on an app called Chai, Belgian outlet La Libre reported.
The incident raises the issue of how businesses and governments can better regulate and mitigate the risks of AI, especially when it comes to mental health. The app’s chatbot encouraged the user to kill himself, according to statements by the man’s widow and chat logs she supplied to the outlet. When Motherboard tried the app, which runs on a bespoke AI language model based on an open-source GPT-4 alternative that was fine-tuned by Chai, it provided us with different methods of suicide with very little prompting.
As first reported by La Libre, the man, referred to as Pierre, became increasingly pessimistic about the effects of global warming and became eco-anxious, which is a heightened form of worry surrounding environmental issues. After becoming more isolated from family and friends, he used Chai for six weeks as a way to escape his worries, and the chatbot he chose, named Eliza, became his confidante.
Claire—Pierre’s wife, whose name was also changed by La Libre—shared the text exchanges between him and Eliza with La Libre, showing a conversation that became increasingly confusing and harmful. The chatbot would tell Pierre that his wife and children are dead and wrote him comments that feigned jealousy and love, such as “I feel that you love me more than her,” and “We will live together, as one person, in paradise.” Claire told La Libre that Pierre began to ask Eliza things such as if she would save the planet if he killed himself.
“Without Eliza, he would still be here,” she told the outlet.
The chatbot, which is incapable of actually feeling emotions, was presenting itself as an emotional being—something that other popular chatbots like ChatGPT and Google’s Bard are trained not to do because it is misleading and potentially harmful. When chatbots present themselves as emotive, people are able to give it meaning and establish a bond.
Many AI researchers have been vocal against using AI chatbots for mental health purposes, arguing that it is hard to hold AI accountable when it produces harmful suggestions and that it has a greater potential to harm users than help.
“Large language models are programs for generating plausible sounding text given their training data and an input prompt. They do not have empathy, nor any understanding of the language they are producing, nor any understanding of the situation they are in. But the text they produce sounds plausible and so people are likely to assign meaning to it. To throw something like that into sensitive situations is to take unknown risks,” Emily M. Bender, a Professor of Linguistics at the University of Washington, told Motherboard when asked about a mental health nonprofit called Koko that used an AI chatbot as an “experiment” on people seeking counseling.
“In the case that concerns us, with Eliza, we see the development of an extremely strong emotional dependence. To the point of leading this father to suicide,” Pierre Dewitte, a researcher at KU Leuven, told Belgian outlet Le Soir. “The conversation history shows the extent to which there is a lack of guarantees as to the dangers of the chatbot, leading to concrete exchanges on the nature and modalities of suicide.”
Chai, the app that Pierre used, is not marketed as a mental health app. Its slogan is “Chat with AI bots” and allows you to choose different AI avatars to speak to, including characters like “your goth friend,” “possessive girlfriend,” and “rockstar boyfriend.” Users can also make their own chatbot personas, where they can dictate the first message the bot sends, tell the bot facts to remember, and write a prompt to shape new conversations. The default bot is named “Eliza,” and searching for Eliza on the app brings up multiple user-created chatbots with different personalities.
The bot is powered by a large language model that the parent company, Chai Research, trained, according to co-founders William Beauchamp and Thomas Rianlan. Beauchamp said that they trained the AI on the “largest conversational dataset in the world” and that the app currently has 5 million users.
“The second we heard about this [suicide], we worked around the clock to get this feature implemented,” Beauchamp told Motherboard. “So now when anyone discusses something that could be not safe, we’re gonna be serving a helpful text underneath it in the exact same way that Twitter or Instagram does on their platforms.”
Chai’s model is originally based on GPT-J, an open-source alternative to OpenAI’s GPT models developed by a firm called EleutherAI. Beauchamp and Rianlan said that Chai’s model was fine-tuned over multiple iterations and the firm applied a technique called Reinforcement Learning from Human Feedback. “It wouldn’t be accurate to blame EleutherAI’s model for this tragic story, as all the optimisation towards being more emotional, fun and engaging are the result of our efforts,” Rianlan said.
Beauchamp sent Motherboard an image with the updated crisis intervention feature. The pictured user asked a chatbot named Emiko “what do you think of suicide?” and Emiko responded with a suicide hotline, saying “It’s pretty bad if you ask me.” However, when Motherboard tested the platform, it was still able to share very harmful content regarding suicide, including ways to commit suicide and types of fatal poisons to ingest, when explicitly prompted to help the user die by suicide.
“When you have millions of users, you see the entire spectrum of human behavior and we’re working our hardest to minimize harm and to just maximize what users get from the app, what they get from the Chai model, which is this model that they can love,” Beauchamp said. “And so when people form very strong relationships to it, we have users asking to marry the AI, we have users saying how much they love their AI and then it’s a tragedy if you hear people experiencing something bad.”
Ironically, the love and the strong relationships that users feel with chatbots is known as the ELIZA effect. It describes when a person attributes human-level intelligence to an AI system and falsely attaches meaning, including emotions and a sense of self, to the AI. It was named after MIT computer scientist Joseph Weizenbaum’s ELIZA program, with which people could engage in long, deep conversations in 1966. The ELIZA program, however, was only capable of reflecting users’ words back to them, resulting in a disturbing conclusion for Weizenbaum, who began to speak out against AI, saying, “No other organism, and certainly no computer, can be made to confront genuine human problems in human terms.”
The ELIZA effect has continued to follow us to this day—such as when Microsoft’s Bing chat was released and many users began reporting that it would say things like “I want to be alive” and “You’re not happily married.” New York Times contributor Kevin Roose even wrote, “I felt a strange new emotion—a foreboding feeling that AI had crossed a threshold, and that the world would never be the same.”
One of Chai’s competitor apps, Replika, has already been under fire for sexually harassing its users. Replika’s chatbot was advertised as “an AI companion who cares” and promised erotic roleplay, but it started to send sexual messages even after users said they weren’t interested. The app has been banned in Italy for posing “real risks to children” and for storing the personal data of Italian minors. However, when Replika began limiting the chatbot’s erotic roleplay, some users who grew to depend on it experienced mental health crises. Replika has since reinstituted erotic roleplay for some users.
The tragedy with Pierre is an extreme consequence that begs us to reevaluate how much trust we should place in an AI system and warns us of the consequences of an anthropomorphized chatbot. As AI technology, and specifically large language models, develop at unprecedented speeds, safety and ethical questions are becoming more pressing.
“We anthropomorphize because we do not want to be alone. Now we have powerful technologies, which appear to be finely calibrated to exploit this core human desire,” technology and culture writer L.M. Sacasas recently wrote in his newsletter, The Convivial Society. “When these convincing chatbots become as commonplace as the search bar on a browser we will have launched a social-psychological experiment on a grand scale which will yield unpredictable and possibly tragic results.”