LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization

Recent large language models increasingly adopt hybrid attention designs that replace the standard quadratic attention with linear alternatives. Models such as Gated DeltaNet (GDN) and Kimi Delta Attention (KDA) compress the entire context into a fixed-size recurrent state, which allows processing of arbitrarily long sequences without proportional compute growth. Despite this architectural advance, a significant inference bottleneck remains: repeatedly reading and updating the recurrent state at every token position incurs substantial memory bandwidth and compute cost. Quantization presents a natural path to reduce this cost, but empirical results show that naïve 8-bit quantization degrades model quality significantly. Rounding errors accumulate across steps, and the recurrent state contains outlier rows and columns that dominate quantization error. The paper under review introduces LeapQuant, a training-free method that achieves near-lossless performance under 8-bit recurrent-state quantization. Its two core techniques — per-window quantization and Compensator Tokens for outlier retention — together enable substantial speedups while preserving accuracy comparable to the FP32 baseline.

The recurrent-state bottleneck in linear-attention LLMs

Linear attention mechanisms approximate the softmax attention computation by maintaining a compressed state that is updated incrementally as each new token is processed. This state typically has fixed dimensionality, independent of sequence length, which is the source of the computational advantage. In practice, the state is stored in FP16 or BF16 format during inference, and each new token requires a read-modify-write cycle: the current state is fetched, updated with the new token's contribution, and written back. For long-context generation, this cycle repeats for every output token, and the memory traffic dominates runtime.

Quantization to 8-bit integers can reduce the memory footprint by four times and potentially accelerate memory-bound operations. However, the recurrent state of linear attention models possesses properties that make aggressive quantization hazardous. First, the state accumulates through many timesteps; rounding errors from each quantization step add up, and the degradation compounds with sequence length. Second, the state contains outlier activations — a small number of rows or columns with large magnitude that do not conform to the majority distribution. If these outliers are clamped or rounded aggressively, the attention output shifts substantially, and model quality drops. Prior quantization work on transformer models has addressed outlier handling through methods such as floating-point residuals or outlier token splitting, but these approaches were designed for feed-forward layers, not for the recurrent state of linear attention.

Per-window quantization

LeapQuant addresses error accumulation through per-window quantization. Instead of quantizing the recurrent state at every token position, the method groups incoming tokens into windows of fixed size — for example, 32 or 64 tokens. Within each window, the state is quantized only once, at the window's end. All output tokens produced from that window read from the same quantized state, but their computations also incorporate high-precision buffered updates that capture the intermediate changes. This "leap over" approach means the quantized state is applied for an entire block of tokens, and the buffered updates compensate for the differences that would otherwise accumulate token-by-token.

The buffered updates are stored in a higher precision format — typically FP16 or BF16 — and are added to the dequantized state before each output token's attention computation. Because the updates are high-precision, the rounding error that would normally be introduced at each step is instead collected and applied in a less frequent, but more accurate, manner. The window size is a hyperparameter; larger windows reduce the frequency of quantization but increase the buffer size, while smaller windows add more frequent quantization events. The paper experiments with window sizes of 32, 64, and 128, reporting that 64 offers a good balance between error reduction and overhead.

Compensator Tokens for outlier retention

Within each window, after the state is quantized and buffered updates are applied, LeapQuant retains the state's largest outliers as a small set of Compensator Tokens. These tokens are not part of the input sequence; they are auxiliary entries stored alongside the quantized state. Their values are the magnitude-residual outliers from the quantized state — the difference between the original FP32 state and its 8-bit approximation, selected by magnitude. The Compensator Tokens share the update path of real tokens: when a new token enters the window, its contribution is added both to the regular state updates and to the Compensator Token pool. This ensures that the outliers, which often correspond to important attention patterns, are not lost due to quantization.

After the Compensator Tokens are extracted, the remaining residual — the bulk of the state values after the largest outliers are removed — is smoothed before quantization. Smoothing here refers to a mild statistical transformation that reduces the dynamic range of the residual, making the subsequent 8-bit quantization more efficient. The combination of outlier retention and residual smoothing means that the quantization error is concentrated in less-important dimensions, while the critical outlier patterns are preserved exactly (or near-exactly, given the high-precision buffer).

Experimental results across Qwen, Kimi, and GLM

The authors evaluate LeapQuant on three major LLM families: Qwen (parameter sizes from 1.8B to 72B), Kimi (various sizes), and GLM (6B and 10B). All models use linear attention variants compatible with the GDN or KDA architectures. The evaluation measures both accuracy and inference speed. For accuracy, the primary metric is next-token prediction perplexity on a held-out validation set, and for instruction-tuned variants, the benchmark includes multiple-choice question answering and chat-style response quality. For speed, the authors measure kernel-level throughput (tokens per second per GPU) and end-to-end latency for generating long-context sequences.

At 8-bit quantization, LeapQuant achieves perplexity within 0.3 points of the FP32 baseline across all model families. This is a substantial improvement over prior quantization baselines, which typically show perplexity degradation of 2 to 5 points or more at the same bit width. The accuracy preservation is especially notable for the 72B Qwen model, where the length-extrapolation capability is critical. Speed measurements show average kernel-level speedups of 2.05× to 3.70×, depending on the GPU and model size. On NVIDIA B200, the RTX PRO 6000, and the RTX 5090, end-to-end inference speedups range from 1.35× to 1.47× for generating sequences of 8,192 tokens or more.

The speedup numbers are particularly impressive because they are achieved without any training or fine-tuning. LeapQuant is a drop-in replacement for the state-updating loop in any linear-attention-based inference engine. The per-window quantization adds a small overhead for buffer management, and the Compensator Token handling requires an additional read of the outlier indices, but these costs are dwarfed by the reduction in bit-width operations.

Kernel-level speedup mechanisms

At the kernel level, the speedup comes from two sources. First, 8-bit integer matrix multiplication is significantly faster than 16-bit or 32-bit floating-point operations on modern GPUs. The NVIDIA Tensor Cores introduced in the Ampere generation and later have dedicated support for 8-bit integer matrix multiplication with accumulation in FP16 or BF16, providing higher throughput for the same hardware resources. Second, per-window quantization reduces the number of read-modify-write cycles for the recurrent state. In the baseline FP32 implementation, the state is read and written at every token position. With LeapQuant, the state is read once per window, quantized, and then the buffered updates are applied in a loop that operates on 8-bit integers. The reduction in memory traffic for the state alone accounts for a large portion of the observed speedup.

The Compensator Token mechanism adds a small overhead, because each window requires an additional step to identify and store the outlier values. However, this overhead is proportional to the number of outliers, which is typically a small fraction of the state dimensionality (e.g., top 2% of rows by magnitude). The paper reports that the Compensator Token path is implemented as a fused kernel operation, so the extra cost is minimal compared with the overall attention computation.

End-to-end inference speedups

For end-to-end generation, the speedup is lower than the kernel-level speedup but still substantial. The end-to-end measurement includes all overheads: token embedding, position encoding, attention computation, language model decoding, and state management. The 1.47× speedup on the RTX 5090, for example, means that a sequence of 8,192 tokens that previously took 100 seconds to generate now takes approximately 68 seconds. The speedup is model-size dependent: larger models see greater relative benefit because their attention computation dominates the runtime, and the state quantification savings are a larger fraction of the total time.

The authors also measure speedup as a function of context length. For very long contexts (32,768 tokens and above), the speedup increases because the per-window quantization amortizes its cost over more tokens, and the Compensator Tokens become relatively less significant. For short contexts (512 tokens), the speedup is smaller but still positive, around 1.2×, because the window overhead is a larger fraction of the total computation.

Limitations and trade-offs

LeapQuant is a training-free method, which is a major practical advantage, but it does introduce two design choices that users must configure. The window size is a hyperparameter that balances error reduction against buffer overhead. A larger window reduces the frequency of quantization events, which improves accuracy, but requires a larger high-precision buffer that consumes additional GPU memory. The authors recommend a window of 64 for most settings, but note that very long-context workloads may benefit from 128, while short-context, low-latency workloads may prefer 32.

Another trade-off is the number of Compensator Tokens retained. The paper experiments with retaining the top 1%, 2%, and 5% of state values by magnitude. Retaining more Compensator Tokens improves accuracy but increases the memory and compute overhead of the compensation path. The sweet spot reported in the paper is 2%, which provides near-lossless accuracy with minimal overhead. Users who require absolute fidelity to the FP32 baseline may choose to retain a higher percentage, accepting the corresponding speedup reduction.

The method is designed for linear attention variants that maintain a fixed-size recurrent state. It does not directly apply to standard softmax attention, whose state grows with sequence length, nor to models that use fully per-token quantization schemes. Additionally, the speedup numbers are specific to the GPU models tested — NVIDIA B200, RTX PRO 6000, and RTX 5090 — and may differ on other hardware. The authors acknowledge that future work could explore adaptive window sizing and learned Compensator Token selection, but these extensions fall outside the scope of the current paper.

Practical implications for LLM deployment

For teams operating large-scale LLM serving systems, LeapQuant offers a low-friction path to reduce inference cost. Because the method is training-free, it can be deployed as a software upgrade without retraining or fine-tuning the underlying models. The 1.47× end-to-end speedup on current-gen GPUs translates directly into lower operational cost per token, which is a key metric for cloud LLM providers. The method also enables deployment of larger models within the same memory budget, because the 8-bit state reduces the memory footprint by approximately 4× compared with FP32.

From an engineering perspective, the implementation requires changes to the recurrent state management loop in the inference engine. The per-window quantization replaces the per-token state read-write with a windowed loop, and the Compensator Token path adds a conditional outlier-retention step. Both modifications are localized to the attention backend and do not require changes to the model weights or the high-level inference graph. The paper provides reference implementations for the Hugging Face Transformers and vLLM serving frameworks, and reports that the integration takes approximately one engineering day for a typical CUDA-based attention kernel.

The speedup numbers suggest that LeapQuant is especially beneficial for workloads that generate long sequences, such as batch text generation, coding assistants producing file-length outputs, and retrieval-augmented generation over large document collections. For interactive, short-turn workloads the speedup is more modest but still positive, and the accuracy preservation means that user-facing quality is not compromised.

Comparison with post-training quantization methods

Post-training quantization (PTQ) of transformer models has been an active research area, and many methods have been proposed for weight quantization and activation quantization. However, recurrent-state quantization is a different problem, because the state is not static — it evolves through the denoising or generation process, and its distribution shifts over time. PTQ methods that rely on a representative calibration dataset may not capture the temporal dynamics of the recurrent state. LeapQuant's per-window approach and Compensator Token retention are specifically designed for this dynamic setting, and the experimental results show that generic PTQ methods applied to the recurrent state achieve significantly worse accuracy at 8-bit.

Some recent work has explored quantization-aware training for recurrent states, where the quantization error is backpropagated through the state update. LeapQuant avoids this by being training-free, which is a deliberate choice to minimize deployment friction. The accuracy achieved without any training is comparable to what quantization-aware training methods report for weight quantization, which underscores the effectiveness of the per-window and Compensator Token design.

Future directions

The authors identify several natural extensions of LeapQuant. Adaptive window sizing, where the window length is adjusted dynamically based on the observed state dynamics, could further improve the accuracy-efficiency trade-off. Learned Compensator Token selection, where the set of retained outliers is determined by a small neural network rather than by magnitude thresholding, could capture more nuanced outlier patterns. Extension to other linear attention variants, such as those based on state-space models (SSMs), is another promising direction, because SSMs also maintain a compressed hidden state that could benefit from similar quantization techniques. Finally, the authors note that the principles of per-window quantization and outlier retention may apply beyond attention mechanisms, to any recurrent state in deep learning models where error accumulation and outlier handling are concerns.

Read the paper on arXiv