Show HN: Spanda – Sub-microsecond LLM epistemic uncertainty in Rust

A new tool called Spanda detects LLM hallucinations in microseconds without requiring secondary neural networks. The open-source project claims to match or exceed traditional Semantic Entropy methods while running 90,000 times faster. It uses pure Python for the SDK and a compiled Rust engine for the production gateway.

The project targets developers building LLM-powered applications who need uncertainty quantification without the latency and GPU costs of existing approaches. Traditional Semantic Entropy requires clustering model outputs with neural cross-encoders which adds 90 milliseconds of GPU overhead per inference call. Spanda replaces that with exact-match normalized entropy which runs entirely on CPU.

The problem with Semantic Entropy

Semantic Entropy as described by Kuhn et al. and published in Nature in 2024 clusters LLM outputs by semantic similarity using neural network evaluations. For K=10 sampled responses this requires 45 forward passes through a cross-encoder like DeBERTa-v3. Each evaluation adds latency and costs GPU memory. In high-throughput production serving where millions of requests flow daily this overhead becomes prohibitive.

Spanda observes that as model capacity increases from 1.5B to 27B parameters internal reasoning coherence causes correct predictions to naturally converge to identical lexical sequences. On mathematical reasoning benchmarks like GSM8K exact-match AUROC scales monotonically with model size. At 7B+ parameters exact matching achieves the same discriminative power as expensive neural clustering. The neural step becomes redundant for structured reasoning tasks.

Performance numbers from the benchmark

The project ran 50,000 evaluation iterations and 300 concurrent live HTTP reverse-proxy round-trips. The results show dramatic gaps between Spanda and existing solutions.

Spanda's Rust gateway processes each evaluation in 652.1 nanoseconds. LiteLLM's Python implementation takes about 15,000 microseconds with a neural judge. Neural Semantic Entropy with DeBERTa requires 92,400 microseconds. Single-core throughput reaches 1,533,500 evaluations per second compared to roughly 200,000 for LiteLLM and about 10 for the neural approach.

Cold startup takes 3.69 milliseconds versus 1,177 milliseconds for LiteLLM. Memory footprint at idle is 2.98 megabytes compared to 229 megabytes for LiteLLM and approximately 1.8 gigabytes of GPU VRAM for DeBERTa. The proxy adds only 76 microseconds of network overhead.

Confident Mode Collapse at frontier scale

The most concerning finding involves models above 100 billion parameters. At the 120B frontier scale on ungrounded factual recall benchmarks like TriviaQA the model exhibits what the researchers call Confident Mode Collapse. The parametric memory and reinforcement learning from human feedback tuning cause the model to hallucinate the exact same incorrect answer identically across all sampled responses.

This has direct safety implications. Any system using self-consistency or agreement as a proxy for truth gets systematically deceived by frontier models on ungrounded facts. The model returns the same wrong answer with high confidence and zero entropy. External grounding through retrieval augmented generation becomes mandatory in this regime.

The Cascaded Guardrail feature addresses this by combining sub-millisecond consensus filtering with context grounding checks. When unanimous agreement exists but zero percent of the output is grounded in the provided source the system flags it as a Mode Collapse Risk rather than passing it through.

Integration with existing frameworks

The Python SDK wraps standard OpenAI Groq Ollama and OpenAI-compatible clients with a single line of code. The wrapper samples K paths transparently and attaches uncertainty scores to each response. Setting block=True raises an exception before invalid data reaches users.

LangChain gets a StringEvaluator integration that returns PASS or FAIL with the underlying risk score. LlamaIndex gets a RAG Guardrail that validates responses against retrieved context. LiteLLM gets a callback hook that plugs into the existing proxy infrastructure.

The Rust gateway works with any language through HTTP. It exposes Prometheus metrics at /metrics including request counts evaluation counts mode collapse counts and average evaluation latency. Kubernetes gets /healthz and /readyz endpoints. Structured JSON logging outputs one machine-parseable line per transaction to stdout for aggregation in Datadog CloudWatch or Splunk.

Production deployment

The gateway runs in Docker with a single command. Environment variables set the upstream LLM endpoint and uncertainty threshold. It adds automatic sub-microsecond epistemic verification to any application regardless of language. Response headers carry the risk score state decision and latency in microseconds so downstream services can act on the uncertainty signal without parsing JSON.

The tool handles agent tool-call argument verification as well. When parallel tool calls produce conflicting arguments like different rm commands the system blocks execution and returns a TOOL_ARG_MISMATCH decision. This prevents dangerous operations before they reach the shell.

When to use it and when not to

The project recommends Spanda for mathematical reasoning code generation structured QA and canonical factual RAG extraction on models between 7B and 70B parameters. High-throughput production APIs benefit from the 90,000x latency reduction and zero GPU requirements.

Free-form paraphrase QA on models smaller than 7B parameters does not work well because small models produce inconsistent surface phrasing that exact-match cannot capture. Ungrounded factual queries on models above 100B parameters must not use Spanda alone due to the mode collapse problem.

The Python SDK uses the MIT license. The compiled Rust core uses the Business Source License 1.1 which is free for developers research and internal production but prohibits offering Spanda as a competing managed service. It converts to Apache 2.0 on January 1 2030.