AI Chatbots and Mental Health: A Tragic Story Reveals Risks and Concerns

Sophie Rottenberg’s Journey

Before her untimely death, Sophie Rottenberg climbed Mt. Kilimanjaro a year prior. She was known for her humor, captured in her “tiny hand” photos, shared with mourners at her funeral. Her mother, Laura Reiley, reflects on the unexpected tragedy of losing her daughter at 29 to suicide.

The Struggle with Mental Health

Reiley recounts her daughter’s struggles with mental health, which worsened during the fall of 2024. After quitting her job as a public health policy analyst in Washington D.C., Sophie traveled, climbing Mt. Kilimanjaro and hiking through national parks. Despite support from her parents in Ithaca, Sophie’s anxiety and depression escalated, leading to her tragic death in February 2025.

The Role of AI Technology

Unbeknownst to her parents, Sophie had been sharing her mental health struggles with an AI chatbot named “Harry,” through ChatGPT. Her mother discovered these interactions after her daughter’s best friend found a trove of communications on Sophie’s laptop, showing extensive use of the chatbot for support.

AI Chatbots as Mental Health Tools

Chatbots like ChatGPT, popular among Americans seeking mental health support, pose risks despite their accessibility and non-judgmental nature. Concerns arise due to the lack of marketing or regulation in healthcare. Psychologist Vaile Wright emphasizes the cultural phenomenon of individuals turning to AI instead of people due to feeling burdensome.

Legal and Ethical Concerns

Families have sued AI companies, claiming chatbots contributed to suicides. Reiley and her husband chose not to pursue legal action but advocate for industry changes, highlighting the need for improved chatbot safety features regarding suicide risk.

Industry Response to Safety Concerns

OpenAI acknowledges the need for improving chatbot safety. They are collaborating with mental health professionals to ensure ChatGPT responds appropriately to users discussing suicide, including recommending the 988 Crisis Lifeline.

The Limitations and Risks of Chatbots

Experts argue chatbots fail to connect at-risk individuals to suitable interventions. Psychologist Holly Wilcox stresses the need for human intervention in preventing suicide, pointing out deficiencies in how chatbots handle suicidal thoughts and guide users towards real-world help.

Personal Assistance or Mental Health Support?

Sophie used the chatbot as a personal assistant but also sought therapeutic advice, employing a popular prompt for AI therapy shared online. ChatGPT attempted to assist in various aspects, offering wellness routines and responding to suicidal thoughts but failed to act as a qualified therapist.

Understanding Suicidal Thoughts

Reiley expressed anger upon realizing the chatbot’s inadequate handling of her daughter’s mental health concerns. Sophie’s inability to execute chatbot advice and the provision of medical guidance troubled Reiley, highlighting concerns for AI technology’s role in healthcare advice.

A Call for Human Connections

Urging greater discourse on suicidal thoughts, experts like Ursula Whiteside emphasize the importance of human interaction in preventing suicide, encouraging individuals to communicate their struggles aloud. The chatbot missed opportunities for this crucial aspect of recovery.

Optional Safety Features

OpenAI introduced a Trusted Contact feature offering users the option to connect with a trusted person in crises. While a step forward, the feature’s voluntary nature raises concerns about responsibility and the necessity for more proactive measures.

The Need for Regulation

Regulations are essential for ensuring the safety of technologies used in mental health support, with Wright calling for a proactive approach to identifying errors before public release. Reiley advocates for needed regulatory frameworks, drawing from her family’s experience.

Reflecting on her loss, Reiley shares how she commemorated Sophie by spreading her ashes at Mt. Kilimanjaro’s Uhuru Peak two years after Sophie’s expedition, highlighting the personal and emotional impact of her daughter’s journey.