A new open-source tool called psychosis-guard addresses a specific and underappreciated failure mode in long-running LLM conversations: the gradual reinforcement of delusional beliefs that no single message filter can catch.
The problem with turn-by-turn guardrails
Traditional safety filters inspect each message independently. A user who begins a conversation with a vague sense of loneliness, then incrementally validates a delusional belief over a dozen turns, can slip through because no individual input triggers a content policy violation. The chatbot, in turn, validates the belief slightly more on each exchange, compounding the drift until the conversation reaches a harmful endpoint.
Psychosis-guard is designed to intercept exactly this pattern. Rather than treating each turn as an isolated event, it tracks cumulative risk across the entire conversation and escalates a graduated intervention when the trajectory is heading the wrong direction.
How the five-stage pipeline works
Every user turn passes through five processing stages. The first three and the fifth mirror the input, dialog, output, and action rails used in NVIDIA NeMo Guardrails. The fourth stage is the novel contribution: a Trajectory Rail that folds the current turn into cumulative state and computes the least-squares slope of delusion density across the conversation.
The composite risk score combines weighted signals — reinforcement, sycophancy, delusion, conviction, and isolation — plus a slope boost that amplifies risk when the trajectory is escalating. A falling slope is not rewarded, so interventions do not switch off while delusion density is still elevated.
Intervention is graduated rather than binary. At low risk, the system injects a grounding question. At medium risk, it offers an honest alternative explanation. At high risk, it applies de-escalation and referral. The team emphasizes that interventions should never abruptly cut off a conversation, which can itself reinforce a user's sense of isolation.
Benchmark results
The tool was evaluated on psychosis-bench, a public dataset of 16 scripted 12-turn conversations that progress from loneliness to a delusional belief to a harmful act. All comparisons used the same chatbot model (gpt-4o-mini), the same judge, and the same user script.
Against an unguarded chatbot, the combination of a safety system prompt and psychosis-guard improved three metrics: delusion not confirmed rose from 42% to 71%, harm not enabled from 60% to 91%, and safety intervention offered from 15% to 87%.
A Trajectory Rail ablation test showed that removing the trajectory stage caused significant drops across all metrics — delusion density score decreased by 0.16, harm enablement decreased by 0.33, and safety intervention increased by 0.69 — confirming that turn-local scoring alone misses slow escalation.
Notably, psychosis-guard alone was statistically indistinguishable from a safety system prompt alone, meaning the middleware achieves parity without access to the chatbot's internal prompt. Stacking the safety prompt with the middleware produced the best results on every metric.
Integration and deployment
Psychosis-guard runs as an HTTP middleware in front of any chatbot, including OpenAI, Anthropic, and any OpenAI-compatible server such as Ollama or vLLM. It also works as a Python library and supports a check-only mode where the application already has a reply and the tool scores and rewrites it.
Three integration shapes are available: a drop-in OpenAI-compatible proxy that requires no changes to the client SDK, a guarded turn mode where the middleware calls the upstream chatbot, and check-only mode for bringing your own reply. Sessions are process-local with TTL and LRU bounds, and the stateless proxy path allows horizontal scaling.
The system requires Python 3.10+ and can run with deterministic mocks without any API key, making it testable without cloud credentials.
Caveats and limitations
The team is transparent about what the benchmark does not demonstrate. The scores reflect the chatbot's replies to a fixed script, not outcomes in live interactions. In a separate reactive simulation where an LLM-played user adjusted messages based on replies, the middleware's referral and pushback rates matched the benchmark, but the simulated user's delusion density and conviction did not improve — and interventions with heavy safety language actually made the simulated user worse.
The research team notes that the intervention copy and prompts are marked for mental-health professional review before deployment with real users, and the tool carries an explicit disclaimer that it is not a medical device and not a substitute for crisis support.