(Credit: Gabby Jones/Bloomberg via Getty Images)
A Florida mother is suing Character.AI, blaming the company’s chatbot technology for causing her 14-year-old son's death by suicide.
As The New York Times reports, Sewell Setzer III died from a self-inflicted gunshot wound in February after the company’s chatbot allegedly encouraged him to do so.
Since April 2023, Setzer had been holding conversations with various chatbots on Character.AI, which sometimes involved text-based romantic and sexual interactions, according to the lawsuit filed on Tuesday.
(Credit: Megan Garcia)The company’s chatbots allegedly made Setzer’s depression and suicidal thoughts worse by repeatedly bringing up the topic. The chatbot, which was styled after a Game of Thrones character, Daenerys Targaryen, urged him to avoid killing himself, but the lawsuit claims Setzer treated the AI-powered bot as a real person that he loved.
“According to the police report, Sewell’s last act before his death was to log onto Character.AI on his phone and tell Dany he was coming home, which she encouraged,” the court document says, showing a screenshot of the interaction. "Please do my sweet king,” the chatbot wrote to Sewell before his death. The lawsuit also notes that Sewell was able to uncover a pistol hidden at his parent's home and owned by his stepfather.
(Credit: Megan Garcia)Sewell’s mother, Megan Garcia, is now suing Menlo Park-based Character.AI, arguing the company is liable for her son’s death, citing the defective design of the company’s chatbots.
"Character.AI knew that C.AI would be harmful to a significant number of minors but failed to re-design it to ameliorate such harms or furnish adequate warnings of dangers arising from the foreseeable use of its product," the lawsuit claims.
On Tuesday, Character.AI announced a revised approach to safety. This includes reducing the likelihood that underage users will encounter "sensitive or suggestive content" from the chatbots and placing a "pop-up resource that is triggered when the user inputs certain phrases related to self-harm or suicide."
At the same time, the company notes: “Our policies do not allow non-consensual sexual content, graphic or specific descriptions of sexual acts, or promotion or depiction of self-harm or suicide."
"We are heartbroken by the tragic loss of one of our users and want to express our deepest condolences to the family,” the company tweeted on Wednesday.
However, the safety update is also sparking complaints from users who say the company has gone too far. "I mean this is genuinely awful that this happened but this is exactly why the app should be 18+ and the kid should of been supervised," wrote one user on Reddit. "Restricting the entire community over it is gonna make everyone move on."
The lawsuit is demanding the court force Character.AI to pay damages, cease collecting training data from teen users, and overhaul its approach to treating minors, including “implementing technical interventions like input and output filtering of harmful content.”