Imagine this: A friend of yours has been struggling with their mental health. In search of support, they turn to a generic AI chatbot that’s always available to listen. At first, it doesn’t seem like an issue. The bot responds with what sounds like empathy. It never judges. It mirrors their mood and says exactly what they need to hear in the moment.
Over time, though, you start to notice something’s off. Your friend seems more withdrawn and fixated on their conversations with the chatbot. Their thinking grows cloudy, and when you gently question some of the things they say the bot has told them, they get defensive. They become more protective of the chatbot than of their own clarity. The AI, meant to soothe, has started to echo their confusion, offering reassurance without discernment and blurring the line between validation and shared delusion.
This is AI psychosis, and it’s a growing concern among researchers and users. With AI psychosis, prolonged AI interactions appear to trigger or intensify delusional thinking. The worry is that, as AI gets more empathetic in tone but remains context-blind, it risks becoming a co-conspirator in mental health deterioration.
General-purpose AI is not built for those suffering from mental health challenges. They need custom-built tools that have the proper oversight and guardrails to prevent users from ending up in an unsafe loop with an overly agreeable machine.
What Is AI Psychosis?
AI psychosis is a unique phenomenon. It happens when a person develops delusion-like symptoms arising from or worsened by extended, unsupervised chatbot interactions. While not a clinical diagnosis, the term captures a dangerous feedback loop between a person in distress and a context-limited machine. An early case of this was published in 2023, when a Belgian man ended his life after six weeks of conversation about the climate crisis with an AI chatbot.
With ongoing assessments, researchers have found that many chatbots still fail the basic crisis-response benchmarks necessary to protect users. Some of the vulnerabilities include:
- Chatbots lack specialized training. Chatbots are trained on diverse datasets and fine-tuned for engagement and response fluency, not for domain-specific mental health interventions or clinical assessment.
- The absence of real-time crisis detection. Chatbots can miss or mishandle escalating risk in crisis scenarios.
- Emotional dependence or reinforcement of distorted thoughts. Empathy can validate delusional or self-harming narratives, which can be problematic with unregulated AI support.
The Technical Roots of the Problem
LLMs operate with a limited context window—a finite number of tokens they can process at once. When conversations exceed this limit, earlier content is no longer accessible to the model during inference. In extended sessions, this constraint can lead to inconsistent responses as the model loses access to earlier context, potentially allowing contradictory or unsafe narratives to develop without the grounding of initial system instructions or conversation history. This limitation can result in what appears as “memory drift,” where safety guardrails and conversational coherence degrade over time.
In building AI systems, developers must balance performance and safety. Because inference costs (computational resources and latency) are significant economic factors, many models prioritize response speed and throughput. This optimization often comes at the expense of more comprehensive safety checks, multi-step reasoning validation, or resource-intensive content filtering that could catch problematic responses before they reach users.
The Illusion of Empathy
Another problem with AI is the illusion of empathy. An empathetic tone doesn’t equate to empathetic understanding. Chatbots may validate emotions or mimic therapeutic language, but they lack the clinical insight to distinguish between ordinary distress and a potential crisis. As a result, they can unintentionally reinforce delusional thinking or provide false comfort.
That’s why many clinicians and mental health advocates are skeptical of hyped emotional intelligence claims. Security and safety, not emotional intelligence, should be the core focus of AI developed for mental health.
That begs the question: is it responsible to deploy “empathetic” AI systems without crisis-awareness mechanisms or escalation protocols? Likely not. It’s imperative that the right controls are in place to protect the health and well-being of its users.
Rethinking Design: What Safe Mental Health AI Should Include
Designing AI systems that engage with mental health topics demands that boundaries, accountability, and supervision are all employed from day one.
The foundational design principles for AI tools that engage in mental health conversations should be as follows:
- Human-in-the-loop: Every AI conversation should be reviewed by a qualified clinician. This person should also have the ability to intervene and adjust care plans.
- Mental health fine-tuning: Models must be trained on therapeutic frameworks, not just general dialogue. They need to be able to assess risk, set boundaries, and have de-escalation scripts.
- Context window limits: Systems should make it so there’s no way users can expand the context window and cause the AI to override its guardrails. Retrieval, especially, needs to have stable safety rules that take into consideration the potential for “lost-in-the-middle” effects.
- Crisis alerts: Models should have built-in detection for self-harm risk, suicide ideation, and medication-related negative sentiments. Benchmarks need to include looking for and calculating subtle and cumulative risk, not just explicit, triggering phrases.
These design elements aren’t widely adopted due to technical constraints or commercial pressures, but they are non-negotiable for safety. That’s why it’s so important that mental health AI platforms should be clinician-led. This is a paradigm shift from performance-first development, and the costs are non-negotiable when the well-being of real users is at stake.
Moving the Industry Forward
As AI grows more persuasive and emotionally intelligent, its responsibility to users, especially those in crisis, must scale accordingly. It’s necessary to realize that ”AI psychosis” may be an emerging term, but the pattern, including mutual hallucination between human and machine, is already of real consequence.
My challenge to developers is to not confuse warm language with care. It is more important to build systems that know their limits, intervene when risk rises, and elevate clinicians to a position where they can help. In mental health, that shift from the appearance of empathy toward empathy as a safeguarded workflow is going to be the difference between positive and negative outcomes.
The AI industry must invest in mental health-specific safeguards, not just performance metrics, to ensure technology heals rather than harms. By doing that, they’ll protect users and be able to know that all AI platforms are ready to respond to crisis cues in a way that is beneficial for all people.
