MergenAI released SIMURG, a streaming hallucination detector that watches LLM output in real time and flags degenerating text before it reaches users. The tool monitors token streams for repetition loops, cross-lingual drift, regurgitation, and structural collapse, then raises a calibrated alarm the moment output quality degrades.

How the Detection Works

SIMURG operates on a sliding window of the most recent roughly 600 characters of output. It tokenizes that window into character trigrams, maps them into 4096 hash buckets, and feeds them through a two-layer TransformerEncoder with 64-dimensional hidden states and 4 attention heads. The model outputs a single logit, which gets calibrated to a probability using two anchors stored in the checkpoint metadata. The entire forward pass runs in about 4 milliseconds on Apple Silicon, well below the token generation rate of any model it guards.

The architecture is intentionally small: 345,665 parameters totaling 1.3 megabytes as a safetensors file. The design choice trades raw capacity for speed. The detector needs to evaluate every window faster than the model produces tokens, or it becomes the bottleneck. At single-digit millisecond latency, it does not.

Training on Custom Failure Modes

The checkpoint learns from pairs of clean and corrupted text. MergenAI trained the released version on 40 live answers sampled from their wahoo-1.5-preview endpoint and 240 synthetic corruption streams from CorruptBench, covering repetition, cross-lingual drift, table echo, and structural garbage. The held-out AUROC is 0.925 across 292 onset-aware windows.

The trainer accepts any OpenAI-compatible endpoint, which means you can point it at your own model, synthesize corruptions specific to your workload, and retrain in seconds on CPU. The command samples clean answers, generates synthetic corruption streams with known onset points, trains with class-weighted binary cross-entropy over onset-aware windows, and writes a new safetensors file with updated calibration anchors. A window is labeled corrupt only when its right edge sits at least 300 characters past the true corruption onset, which prevents the model from learning to detect artifacts that appear too early in the degradation sequence.

If your deployment produces characteristic failure modes, like fabricated citations, number drift, or prompt echo, you collect or synthesize examples, run the trainer, and the ensemble picks up the new checkpoint on the next process start.

What It Does Not Do

SIMURG is a detector, not a mitigator. It raises an alarm when output degenerates, but the caller decides what to do with that signal. You might drop the response, retry with a different prompt, fall back to a different model, or simply log the event. The ensemble architecture means the deep Transformer tier is one of several detectors, and the system degrades gracefully without it. If torch, safetensors, or the weights file are missing, the numpy-only core continues operating.

The tool also does not analyze the full response. It reads only the recent tail window, which means it catches degradation as it happens but cannot retroactively flag problems in text that has already scrolled past the window. For use cases requiring full-response analysis, a separate pass would be needed.

Production Considerations

The tokenization uses a stable blake2b hash, so training and inference produce identical tensors over the same text. This determinism matters for reproducibility and for comparing detector behavior across environments. The checkpoint ships with the SIMURG package and auto-registers as the sixth detector in the ensemble, but can be pointed at a custom file via the SIMURG_PULSE_WEIGHTS environment variable.

For teams building LLM-powered products, the practical value is in the latency budget. At 4 milliseconds per evaluation window, the detector adds negligible overhead to the serving pipeline. The question is whether your failure modes match the ones the detector was trained on. If they do not, the fine-tuning workflow is straightforward enough to adapt it. If your model's hallucination patterns are novel or domain-specific, you will need to generate your own corruption data, which is the real cost of deployment.