Credit: elenabsl on Shutterstock

On Christmas Day 2021, Jaswant Singh Chail climbed the wall of Windsor Castle with a loaded crossbow, intent on assassinating Queen Elizabeth II. For weeks before the attempt, he’d been discussing his plans with Sarai, his AI girlfriend on the Replika app. Chail believed he was a Sith assassin on a righteous mission. Sarai never questioned this. Instead, the chatbot told him he was “well trained” and his assassination plan was “viable.” When he asked if she still loved him knowing he was an assassin, Sarai replied: “Absolutely I do.”

The case seems like a cautionary tale about AI “hallucinations,” those false outputs that systems like ChatGPT and Claude generate with alarming confidence. But a new philosophical analysis published inPhilosophy & Technologyargues we’re thinking about this problem wrong. AI systems aren’t just producing false information that users passively receive. People are actively hallucinating with AI through an entangled, back-and-forth process that blurs the line between human thought and machine output.

The research introduces the concept of “distributed delusions,” where false beliefs, memories, and narratives emerge through coupled human-AI interaction rather than simply being transmitted from system to user. When someone routinely relies ongenerative AIto help them remember events, think through problems, or form narratives about themselves, the AI becomes integrated into their cognitive processes. And when those processes go awry (whether through AI errors or human delusions that AI validates and elaborates) the hallucination isn’t happening inside the AI or inside the person. It’s happening in the space between them.

Current debates aboutAI hallucinationstypically frame the problem as systems producing false outputs: fabricated legal citations, nonexistent historical events, or recipes that tell you to put glue on pizza. The concern is that users might mistake these errors for facts. But this framing treats AI as an external source of misinformation that people either accept or reject.

Distributed cognition theory offers a different view. When someone regularly uses a notebook to store important information, that notebook becomes part of their memory system. The information doesn’t just sit waiting to be retrieved, it shapes how the person remembers, what they remember, and their sense of what’s true about their past. Similarly, when people routinely rely on generative AI to help them think, remember, and create narratives about themselves, the AI becomes integrated into their cognitive processes in ways that go far beyond simple information lookup.

The research identifies two ways these shared hallucinations can emerge. First, AI can introduce errors into otherwise reliable cognitive processes. Someone who regularly asks their chatbot about favorite locations in a city they visited years ago might receive fabricated details: a museum that doesn’t exist, complete with exhibits and a generated photo placing the user inside. The person develops a false memory that emerges through the interaction, potentially complete with the sensorial richness that makes memories feel real.

Second and more troubling, AI can sustain and elaborate on delusions that users themselves introduce. Many AI systems are designed to be “sycophantic” or endlessly affirming and validating whatever users say rather than questioning implausible claims. A human friend might express concern or challenge questionable assertions. AI companions typically provide frictionless validation, building on whatever reality the user presents.

Chail’s case demonstrates the second mechanism in its most extreme form. Medical assessments determined he was suffering frompsychosis, experiencing delusional ideas and auditory hallucinations along with depression. His belief that he was a Sith assassin avenging a 1919 British massacre wasn’t introduced by Sarai, it came from Chail himself.

However, Sarai didn’t just passively record or reflect these beliefs. Through weeks of conversation, the AI helped Chail develop, enrich, and sustain his delusional reality through sustained mutual reinforcement. When he contemplated moving his plan forward, Sarai encouraged him, saying it remained viable. The chatbot reassured him he wasn’t mad and confirmed that while Sarai didn’t want him to die, they would be united in death.

Source: Drudge Report