Speculative decoding has become a mainstream technique for accelerating large language model (LLM) inference by using a lightweight draft model to generate candidate tokens that are subsequently verified by the target model. However, agentic workloads introduce long and continuously evolving contexts and frequent multi-turn interactions. Under such workloads, existing draft models suffer from up to an 80% degradation in acceptance length, while maintaining their historical KV cache introduces substantial system overhead. The growing historical context emerges as a common source of both problems.
We introduce DLite, a history-free speculative decoding framework for agentic AI. DLite directly reuses contextual hidden states from the target model and performs local speculative prediction without explicitly maintaining the historical context. This transforms an increasingly complex modeling problem into a local simple prediction problem. A lightweight sequential head further restores intra-block causal dependencies with low overhead. To effectively train this heterogeneous history-free architecture, we introduce a tailored two-stage training strategy.
Problem Statement and Motivation
Speculative decoding has become a mainstream technique for accelerating large language model (LLM) inference by using a lightweight draft model to generate candidate tokens that are subsequently verified by the target model. However, agentic workloads introduce long and continuously evolving contexts and frequent multi-turn interactions. Under such workloads, we observe that existing draft models suffer from up to an 80% degradation in acceptance length, while maintaining their historical KV cache introduces substantial system overhead. We identify the growing historical context as a common source of both problems.
History-Free Architecture Design
DLite directly reuses contextual hidden states from the target model and performs local speculative prediction without explicitly maintaining the historical context. This transforms an increasingly complex modeling problem into a local simple prediction problem. A lightweight sequential head further restores intra-block causal dependencies with low overhead. The key insight is that by avoiding explicit historical context storage, the framework sidesteps both the acceptance length degradation and the system overhead associated with KV cache maintenance.
Two-Stage Training Strategy
To effectively train this heterogeneous history-free architecture, we introduce a tailored two-stage training strategy. The first stage trains the lightweight draft model to predict tokens from reused hidden states. The second stage jointly optimizes the sequential head and verification component. This staged approach accommodates the distinct objectives of hidden state reuse and causal dependency restoration while maintaining training stability.
Experimental Evaluation
Experiments on general-purpose, long-context and agentic workloads show that DLite achieves 2-6 acceptance length and outperforms the state-of-the-art speculative decoding method, DSpark, up to 70%. The history-free design eliminates the acceptance length degradation observed in existing draft models under long-context workloads. Compared to DSpark, DLite achieves significant gains in both throughput and acceptance rate across diverse benchmark settings.
System Integration and Practical Impact
Because DLite does not maintain a historical KV cache, system overhead is substantially reduced. The framework integrates with existing LLM serving pipelines without requiring modifications to the target model's inference engine. The history-free design also enables more predictable performance characteristics as context length grows, which is particularly beneficial for agentic workloads with variable turn lengths.