Formal specification fills gap in deterministic LLM inference
LLM inference systems make optimization choices that can change batch composition, prompt chunking, prefill and decode execution, and KV-cache reuse, eviction, or recomputation. These decisions should not alter system outputs, but production engines lack a formal system-level specification to guarantee it.
Production modes fall short of bitwise identity
Researchers formalized deterministic LLM inference as the property that under a fixed model and deployment configuration, requests with the same prompt and initial sampler state produce bitwise-identical logits at corresponding output positions across executions. Tests of existing production modes found that vLLM's batch-invariant mode and SGLang's deterministic mode can produce different logits under certain execution variations, revealing the gap the paper addresses.
Vosti: an inference engine built to a specification
The authors present Vosti, an inference engine designed and verified against a formal specification of deterministic inference. Vosti chooses GPU kernels independently of runtime engine state and ties cached KV values to their logical token prefixes. This design ensures that cached cache entries are logically grounded rather than positionally arbitrary.
Proof splits at the engine/GPU kernel boundary
The verification proof decomposes at the boundary between the inference engine and GPU kernels. A Verus inductive proof establishes that scheduling decisions and the paged, prefix-sharing KV-cache preserve output logits. A separate Triton analyzer proves that selected kernel outputs are bitwise-equal across batches, query lengths, and various paged KV-cache layouts. Both components together form the guarantee.
Determinism verified across every execution variation
Vosti produces bitwise-identical logits across every tested execution variation. On decode-heavy workloads its performance matches vLLM's batch-invariant mode while providing a stronger determinism guarantee that is formally verified rather than empirically observed alone.
Why deterministic inference matters
When LLM serving systems vary batch composition or cache management, non-deterministic output complicates reproducibility testing, debugging, and security scanning. Applications that rely on consistent logits—for example, certain reward models or verification pipelines—can produce unstable results if the underlying engine is not deterministic. A formally verified engine gives developers confidence that system behavior does not depend on hidden runtime state.
Technical contribution and practical impact
The paper contributes a formal definition of deterministic LLM inference, evidence that current production modes do not fully satisfy it, and Vosti as a verified alternative. For teams building or operating LLM serving infrastructure, the work offers a concrete reference specification and an engine that meets it without sacrificing throughput on typical workloads. The formal methods used—inductive proof over scheduling and kernel-level bitwise analysis—may also inform future inference system design.