vLLM, the inference engine with over 91,000 GitHub stars, has a pull request that adds structured generation support for DiffusionGemma, a diffusion-based language model from Google. The PR also fixes a concurrency bug that crashes parallel requests, and it includes performance work that keeps the overhead off the hot path for ordinary generation.

DiffusionGemma works differently from standard autoregressive language models. Instead of producing tokens one at a time from left to right, it runs an iterative denoising process across a fixed canvas of token positions. Each step refines the canvas, gradually replacing noise with coherent text. The approach is similar to how image diffusion models generate pictures, but applied to sequences of tokens.

The pull request, opened by contributor mmastrac on 16 September 2026, gives clients control over that denoising process and fixes a bug that makes concurrent DiffusionGemma generations unreliable.

A logprobs bug breaks concurrent requests

The diffusion sampler stores a request's top-k logprobs on the step where it converges, then attaches them to the output on its commit step. The code that popped those stashed logprobs ran for every decoding request in the batch, not just the ones committing at that step. When two requests had schedules offset by one step, the first request's logprobs got emitted under the wrong request. The second request's commit found no stash, the API returned fewer logprobs than tokens, and chat completions failed with an IndexError in the logprobs creation function.

The fix restricts the pop to slots that are actually committing in that step. Any two concurrent DiffusionGemma requests whose schedules aligned one step apart would trigger the original bug.

Three new parameters for structured reads

The PR adds three optional fields to SamplingParams.extra_args (exposed as vllm_xargs on the OpenAI-compatible server):

  • diffusion_seed_canvas takes a list of integers matching the canvas length. It replaces the all-noise initial canvas after prefill. The caller places answer template tokens at known positions and fills the remaining slots with random IDs, telling the model which positions to decide on.
  • diffusion_max_steps caps the number of denoise steps before the request commits. A value of 1 returns logprobs from a single forward pass at every canvas position.
  • diffusion_read_only is a boolean. When true, the request commits the argmax canvas as soon as the step cap is reached, skipping the normal convergence wait. The logprobs come from the model's logits at temperature 1, not the schedule-tempered logits used during denoising.

Validation was moved to SamplingParams.verify so bad inputs return HTTP 400 instead of crashing the engine. A seed ID outside the vocabulary used to trigger a device-side assert during embedding gather, and wrong types or lengths crashed the engine loop. Both failures took down the entire engine rather than just the offending request.

Read-only requests also enforce a contract: verify caps max_tokens at the served canvas length and sets ignore_eos. Without this, a read-only request that emitted its argmax canvas would roll into a second block from fresh noise, losing the seed and producing garbage output.

Keeping the hot path clean

Every denoise step previously ran a GPU synchronization for the read-only mask, calling .any() and .tolist() regardless of whether any read-only request existed in the batch. Each finished prefill uploaded its seed canvas one slot at a time in a Python loop, producing one host-to-device copy per slot.

The PR introduces host-side sets of seeded and read-only slots alongside the GPU flags. The sampler checks the CPU sets first, so ordinary generation pays no overhead. Seeds live in a tensor shaped [max_num_reqs, canvas], uploaded once at request creation and applied to an entire prefill batch with a single masked copy. The step cap tensor was also changed from float to int32 to match the step counter's dtype inside the compiled denoise step.

A minor cleanup removes a redundant commit term from the read-only mask. Read-only slots never enter the encoder phase, so the ~is_committing condition was always true for them.

What the model can do with structured canvas control

The PR description frames this as a "Jev-like" mode. The term references a structured output technique where fixed token positions constrain the model's choices to bounded options. The caller pre-seeds the canvas with an answer template and noise in the unfixed slots. The model diffuses single-token answers into those positions and provides logprobs from which a client can derive entropy, a measure of the model's uncertainty.

If entropy exceeds a threshold, the client can run additional samples to measure agreement between draws. The PR includes a sample interposer server that translates a JSON schema into a seeded canvas, so clients never construct canvases by hand.

The model can handle yes/no questions, scale ratings, and multiple-choice classification. One test classifies a support ticket by urgency, category, and tone. When the model is uncertain, the auto policy re-reads until entropy drops below threshold.

Test results on a single DGX Spark

All tests ran on one DGX Spark with a 64-row canvas and three decisions per request. Single reads processed at 4.9 requests per second (0.20 seconds each). At 32-way concurrency, throughput reached 21.8 requests per second (1.47 seconds each), about 65 decisions per second.

On a programming language classification task, the model scored 10 out of 10. Human language classification got 9 out of 10 (confusing Portuguese with French). Unit comparison tests scored 10 out of 12. The model also solved an ASCII maze using only cardinal direction options.

The PR remains open, with code owner reviews pending from njhill and NickLucche. All commits are co-authored with Claude Fable 5.1. The example structured-decision server lives in the PR's docs commit, along with the concurrency bugfix and the performance improvements to the diffusion hot path.