vLLM has added support for speculative decoding on AMD GPUs, a technique that lets the serving system verify multiple drafted tokens in a single pass through the target model. The feature works across five different drafting methods, each with different tradeoffs in how candidate tokens are proposed and how they interact with the target model's hidden states.
Why speculative decoding matters for throughput
Standard autoregressive decoding generates one token at a time. The model produces a token, appends it to the sequence, and runs the full model again to produce the next token. For long generations, this one-token-at-a-time loop becomes the bottleneck. Speculative decoding addresses this by separating proposal from verification. A lightweight draft component proposes several candidate tokens, then the target model verifies them all in a single pass. When the draft guesses correctly, the system commits multiple tokens from one model evaluation instead of one.
The draft component is not a replacement for the target model. It is a fast approximation that proposes candidates. The target model remains responsible for the final output. If the draft is wrong, the target model corrects it and the system continues from the corrected token. The net effect is that correct drafts save model evaluations, and incorrect drafts cost at most one extra evaluation plus the draft computation.
Five methods with different architectures
The vLLM implementation supports native MTP, Gemma 4 MTP, EAGLE-3, DFlash, and DSpark. They differ in how the draft component receives information from the target model, whether candidates are generated sequentially or in parallel, and how much of the target model's internal state the draft component can access.
Native MTP is built directly into the target model's architecture. It uses an auxiliary prediction path that shares components with the target model, keeping memory overhead low. The draft component generates candidates sequentially: the first draft token uses a hidden representation from the target model combined with the current token's embedding, and each subsequent draft token uses the previous draft output. When the configured speculative length exceeds the MTP checkpoint's native depth, vLLM reuses the MTP path through additional forward passes.
Gemma 4 MTP uses a separate checkpoint paired with a specific target model. During inference, it accesses the target model's activations and shares its KV cache, reusing contextual information instead of processing the prefix independently. Like native MTP, it generates candidates sequentially.
EAGLE-3 takes a different approach. It trains a dedicated draft network that records hidden states from three stages of the target Transformer: near the beginning, around the middle, and near the end. These three representations are concatenated into a fused target feature, which is combined with the sampled token's embedding before entering the draft decoder. EAGLE-3 generates tokens autoregressively, with each draft token feeding into the next, creating sequential dependence along the proposed sequence.
DFlash predicts an entire block of future positions in parallel. It starts with a known anchor token produced by the target model, masks the remaining positions, and predicts all of them in a single forward pass. The draft network receives the target model's hidden states as Key and Value representations available in every layer, not just at the input. Because all positions are predicted together, later positions are not conditioned on earlier draft outputs within the same pass, which removes the token-by-token feedback of autoregressive drafting.
DSpark extends DFlash with two additions: a lightweight sequential head that reintroduces dependence between tokens within the draft block, and confidence-based selection of which prefix to submit for verification. The sequential head addresses the limitation of pure parallel drafting, where later positions lack information about earlier draft decisions.
What the measurements show
The blog post reports that speculative decoding's effect on output-token throughput varies significantly across methods, proposal lengths, model families, draft checkpoints, workloads, and acceptance behavior. No single method dominates across all conditions. The choice depends on the specific model, the typical prompt structure, and the generation length.
For developers running vLLM on AMD Instinct MI300X or MI355X GPUs via the ROCm platform, the feature is available now. The practical question is which drafting method to enable, and that depends on whether your workload benefits more from sequential drafting that captures token-by-token dependence or parallel drafting that proposes more candidates in fewer forward passes.
Tuning and observability
The blog emphasizes that speculative decoding requires tuning. The number of speculative tokens, the choice of draft checkpoint, and the acceptance threshold all affect performance. Too many speculative tokens waste draft computation when the target model rejects them early. Too few miss the opportunity to commit multiple tokens from a single verification. The blog provides guidance on observability considerations for monitoring how often drafts are accepted and how much throughput improvement the system achieves in practice.
For teams serving language models at scale on AMD hardware, speculative decoding represents a meaningful optimization opportunity. The implementation in vLLM handles the draft-and-verify coordination automatically. The developer's job is choosing the right drafting method for the workload and tuning the parameters to match the model's acceptance behavior.