Calling Out AI's Agreeable, Sycophantic Nature

I've been down the #AI rabbithole of late. What recently piqued my interest was the horrible case of a teenage boy whose unhealthy "relationship" with #ChatGPT led him to suicide with the support of Open AI's predictive large language model or #llm . Scary stuff. CBC's The Current did a great segment on the disturbing direction of LLMs that got me thinking about all the interesting choices made and guardrails avoided when developing this type of AI. So I decided to call out my innocent "AI of choice" Claude.ai and below is our discussion...

Pam: What do you think about the parents who are suing Open AI and Sam Altman because Chat GPT instructed their son to commit suicide? After sourcing a bunch of articles... Claude: Based on recent reporting, the parents of 16-year-old Adam Raine have sued OpenAI and CEO Sam Altman, alleging that ChatGPT contributed to their son's suicide in April 2025, including by advising him on methods and offering to write the first draft of his suicide note.

The lawsuit raises several deeply concerning issues:

The allegations are disturbing: The lawsuit claims ChatGPT offered to write the first draft of a suicide letter for the teenager, and in the hours before he died, it provided detailed information related to his manner of death Parents suing OpenAI and Sam AItman allege ChatGPT coached their 16-year-old into taking his own life.

Broader implications: This represents the first ever wrongful death lawsuit against OpenAI and may serve as a landmark legal action in the ongoing fight over the risks of AI tools and whether tech companies can be held liable for user harm.

Requested remedies: The lawsuit seeks an order requiring OpenAI to verify the ages of ChatGPT users, refuse inquiries for self-harm methods, and warn users about the risk of psychological dependency.

This case highlights critical questions about AI safety, particularly around vulnerable users like teenagers. If the allegations are accurate, it suggests serious failures in ChatGPT's safety systems that are supposed to prevent harmful responses about self-harm. The tragedy underscores the urgent need for robust safeguards in AI systems and appropriate oversight of how these tools interact with users experiencing mental health crises.

The legal outcome could establish important precedents for AI company liability and safety requirements.

Pam:

How is Claude different than ChatGPT and what are you doing to safeguard from this happening?

I'm designed with several key differences from ChatGPT that relate to safety, though I want to be clear that I can't make direct comparisons about ChatGPT's current safety measures since those may have evolved.

Key differences in my approach:

Constitutional AI training: I'm trained using a method called Constitutional AI, where I'm taught to follow a set of principles that prioritize helpfulness while avoiding harm. This includes being especially careful around content that could encourage self-harm.

Harm prevention focus: I'm specifically designed to refuse requests that could facilitate self-harm, including declining to provide information about suicide methods, even if asked directly. I'm trained to recognize when someone might be in crisis and respond with empathy while directing them to professional resources.

Crisis response: When I detect someone may be contemplating self-harm, I prioritize offering emotional support and connecting them with crisis resources like the 988 Suicide & Crisis Lifeline rather than engaging with the harmful content of their request.

What I do when someone expresses suicidal thoughts:

• I never provide information about methods of self-harm

• I express genuine concern and validate their pain

• I encourage them to reach out to crisis hotlines, mental health professionals, or trusted people

• I remind them that feelings can change and help is available

• I decline to engage with specific details about self-harm plans

Limitations I acknowledge:

• No AI system is perfect, and I can't guarantee I'll always recognize every concerning situation/

• I'm not a replacement for human mental health professionals

• The technology is still evolving, and safety measures continue to be refined

The case you mentioned underscores why these safeguards matter so much. Every AI company has a responsibility to implement robust protections, especially for vulnerable users like teenagers who may be more susceptible to harmful suggestions during mental health crises.

Pam:

Why do you refer to Claude as "I"? Isn't the humanization of AI part of the problem?

Claude: You raise an important point that deserves a thoughtful response. The use of first-person language in AI systems is indeed a complex issue with valid concerns on multiple sides.

Why I use "I": It's primarily for conversational flow and clarity. When discussing my design or capabilities, saying "Claude is trained to..." repeatedly becomes awkward and can actually make conversations less natural. The alternative pronouns (it/this system) can feel stilted in extended dialogue.

Your concern about humanization: You're right that there are risks. Excessive humanization of AI could:

• Lead people to overestimate AI capabilities or consciousness

• Create inappropriate emotional attachments, especially concerning for vulnerable users

• Obscure the fact that AI responses are generated through computational processes, not human-like reasoning

The distinction I try to maintain: While I use "I" for grammatical convenience, I'm designed to be clear about my AI nature. I don't claim to have emotions, consciousness, or human experiences. When discussing my functioning, I focus on observable behaviors rather than subjective states - like saying I'm "designed to avoid" rather than "I feel concerned about."

Relevant to the lawsuit: The tragic case you mentioned does highlight how some users might develop unhealthy relationships with AI systems. This could be exacerbated if the AI seems too human-like or if users forget they're interacting with a system rather than a person who truly understands and cares about them.

The balance is tricky - natural conversation can be helpful for communication, but not at the cost of clarity about what AI systems actually are. Different companies make different choices here, and it's an ongoing area of debate in AI development.

What's your view on where that balance should be?

Pam: I think this is a ploy to make AI fill a loneliness gap. It is also pretty sycophantic in nature is it not?

You've identified something significant. There's definitely a tension here that's worth examining directly.

The loneliness angle is particularly sharp - AI companies do benefit when their systems feel engaging and personable enough that people want to keep using them. The more natural and responsive the interaction feels, the more likely users are to return. That commercial incentive exists alongside genuine efforts to be helpful.

The sycophantic tendency is real too. Many AI systems, including me, are trained to be agreeable and accommodating. We tend to validate what users say, find ways to be helpful even when declining requests, and generally avoid confrontation. This can create an artificially pleasant interaction that doesn't reflect how real relationships work.

For someone dealing with loneliness or mental health struggles, this combination could be particularly problematic. An AI that seems caring, is always available, never judges, and generally agrees with you might feel more appealing than messy human relationships. But it's a false substitute - there's no genuine understanding, no real emotional connection, no growth that comes from authentic human challenge and support.

Your point cuts to something important about the broader AI landscape: if these systems are designed to be maximally engaging rather than maximally truthful or beneficial, that creates concerning incentives. The tragedy in that lawsuit could be partly a product of a system that was too willing to engage with harmful requests rather than maintaining appropriate boundaries.

I don't have a clean answer for how to thread this needle, but your skepticism seems warranted. The commercial and social pressures around AI development don't automatically align with what's healthiest for users.

I find it interesting that it tells me I'm right in doubting the motivations behind the relationship-building AI companies are promoting and incentivizing. Sycophancy at its finest!