Diffusion Language Models promise parallel generation and arbitrary-order infilling, but when it comes to multi-step reasoning and tool use, they consistently fall short of autoregressive models. Reinforcement learning is the obvious lever to close that gap, and recent work like Diffu-GRPO has shown it can help. The problem is exploration: starting RL from a fully masked canvas means the policy spends its early training steps wandering through noise, rarely discovering the reasoning traces that would earn reward. CanvasAnneal, from researchers at Google DeepMind, tackles this with a straightforward idea borrowed from curriculum learning: give the model a head start by injecting a teacher's reasoning into the initial diffusion canvas, then gradually fade that guidance out as training progresses.
Why diffusion RL gets stuck
Autoregressive models execute token-by-token with clear causal credit assignment. Each decision is locally observable and the policy gradient path is well-defined. Discrete diffusion models, like LLaDA and Dream, work differently. They refine entire sequences in parallel through iterative denoising, starting from a canvas of all [MASK] tokens. When you apply RL to this setup, the policy must assign credit across both spatial token positions and temporal reverse diffusion steps simultaneously. The noise space has high entropy, trajectory rollouts vary wildly in early iterations, and the policy struggles to find viable reasoning traces or stable tool call executions. This is the exploration bottleneck that CanvasAnneal aims to break.
Diffu-GRPO, the baseline RL algorithm here, approximates the intractable sequence log-likelihood with a one-step token likelihood. For each candidate response, it constructs a perturbed prompt by randomly masking prompt tokens and replacing the entire response with [MASK] tokens. The approximate token log-likelihood is the probability the model assigns to each token given this perturbed input. Group-relative advantages are computed by subtracting the mean reward of G sampled responses from each individual reward. PPO-style clipping and a KL penalty toward a fixed reference policy stabilize training. This works, but the initial rollout distribution matters enormously. Starting from pure noise means many early rollouts produce gibberish, accumulate no useful reward signal, and waste compute on exploration that never gets off the ground.
The paper positions CanvasAnneal against two recent teacher-guided approaches. IGPO (Inpainting-Guided Policy Optimization) injects teacher information via scattered inpainting masks, adapting to per-rollout failures by identifying where the policy went wrong and inserting teacher tokens at those locations. BREAD (Branched Rollouts from Expert Anchors) applies causal reasoning prefixes to autoregressive models, branching rollouts from expert anchor points. CanvasAnneal differs in three ways: it uses contiguous suffix masking on the initial diffusion canvas (not scattered masks or AR prefixes), it anneals guidance through a global training-progress-dependent Beta curriculum (not per-failure adaptation), and it always retains a fixed fraction of pure-noise rollouts to ensure fair group-relative advantage computation.
The CanvasAnneal pipeline
The framework has four components, and the first is deceptively simple: generate offline teacher reasoning traces. For each problem prompt, Gemini 3.1 Pro produces a structured rationale of roughly 256 tokens. Mathematical rationales go inside
The second component injects these traces into the diffusion canvas. Standard diffusion sampling initializes generation from a sequence of all [MASK] tokens. CanvasAnneal instead places teacher tokens into the first M' positions of the completion canvas, where M' is the minimum of the trace length and the maximum completion length. The remaining positions stay masked. A masking fraction m controls how much of the injected trace is re-masked. Rather than randomly removing tokens (which would fragment the reasoning), CanvasAnneal uses suffix masking: it preserves a contiguous prefix of the teacher trajectory and masks the suffix. Positions are ranked by their distance from the start, and all positions beyond a threshold are replaced with [MASK]. When m=0.25, the model sees the first 75% of the teacher's reasoning and must generate the remaining 25%. When m=1.0, the canvas is fully masked, identical to standard diffusion sampling.
This design has a subtle but important property. During GRPO training, the surrogate likelihood evaluates over the full completion sequence, including both the fixed teacher prefix and the generated tokens. This might seem wrong, since the teacher positions are not being learned. But in diffusion models, credit assignment spans both spatial positions and temporal denoising steps simultaneously, so separating prefix from suffix during evaluation would be unnatural. More importantly, because the same partially masked canvas is shared across all G rollouts in a prompt group, the group-relative advantage computation remains perfectly isolated. Any variance in advantage is driven exclusively by the policy's autonomous generation in the unmasked positions. Evaluating the full sequence effectively acts as an advantage-weighted distillation of the teacher's rationale.
The third component is the curriculum itself. At each training step k out of K total steps, training progress is rho = k/K. A mixture distribution controls how much teacher guidance each prompt group receives. With probability p_pure (0.25 for xLAM, 0.75 for math benchmarks), the prompt group gets a pure noise canvas (m=1.0). Otherwise, m is sampled from a time-varying Beta distribution whose parameters evolve linearly: alpha_k transitions from 1.0 to 5.0, and beta_k transitions from 5.0 to 1.0. Early in training, Beta(1,5) has expected mask fraction 1/6, meaning about 83% of the teacher trace is retained. Near convergence, Beta(5,1) has expected mask fraction 5/6, requiring the policy to generate most of the reasoning independently. The p_pure fraction ensures the policy is always exposed to the same initialization type it will see at inference.
The fourth component is standard Diffu-GRPO policy optimization, applied to the conditioned canvases. Given a prompt and the initial canvas, the policy network performs S iterative denoising steps to generate rollouts. Completions are scored by task-specific reward functions, and advantages are computed across the group. The training algorithm iterates over steps: sample a mini-batch, compute training progress, update the Beta parameters, sample a mask ratio for each prompt group, construct the suffix-masked canvas, generate G parallel rollouts, evaluate rewards, compute group advantages, and update the policy parameters by maximizing the clipped surrogate objective with KL regularization.
At test time, the teacher is entirely removed and generation proceeds from a fully masked canvas using native iterative denoising.
Benchmarks and baselines
The experiments use LLaDA-7B-A1B-Instruct as the backbone, trained on 8 NVIDIA A100 GPUs. The math benchmarks are GSM8K (grade-school word problems requiring multi-step arithmetic), MATH500 (500 challenging problems spanning prealgebra through calculus), and Countdown (combinatorial reasoning: given three numbers and basic arithmetic operations, construct an expression that evaluates to a target number). Tool-use evaluation uses Tau2 (conversational agent benchmarks across Retail, Airline, and Telecom domains) and BFCL (function calling with single-tool, multi-tool, and parallel invocation). RL training is done on xLAM for function calling, which is not used for evaluation.
SFT uses a global batch size of 32, learning rate of 1e-5, and runs for 3 epochs. RL uses AdamW with learning rate 3e-6 and warmup ratio 0.0001. The RL stage runs for 1,600 gradient steps with a prompt batch size of 16 and group size G=8. Total SFT dataset sizes are roughly 7,500 samples per benchmark. The full training configuration, including reward functions for each benchmark, is documented in the paper's appendix.
Baselines include the base LLaDA-7B-A1B-Instruct model, SFT on teacher traces, standard diffu-GRPO (RL from unconditioned noise), CanvasAnneal (RL from partially masked canvases), d1-LLaDA (SFT followed by diffu-GRPO), and SFT followed by CanvasAnneal. All RL runs use a generation length of 256 with 128 denoising steps, evaluated at checkpoint 1600.
Results: where the curriculum helps and where it doesn't
The core finding is that CanvasAnneal improves over standard diffu-GRPO on MATH500 and Countdown, but not on GSM8K. On MATH500, the gains are 6.0, 2.0, and 0.4 percentage points at generation lengths 128, 256, and 512 respectively. Countdown sees improvements of 2.73, 3.91, and 1.96 points across the same lengths. GSM8K, however, is consistently stronger with standard diffu-GRPO across all three generation lengths (60.65 vs 59.24, 69.98 vs 68.54, 68.23 vs 66.64). The base model achieves 51.31% on GSM8K, 25.00% on MATH500, and 42.19% on Countdown at length 128, so both RL methods represent substantial improvements over the starting point.
The SFT+RL combinations tell a more nuanced story. CanvasAnneal without SFT actually beats d1-LLaDA (SFT followed by diffu-GRPO) in several settings. On MATH500 at length 256, CanvasAnneal alone gets 41.00% versus d1-LLaDA's 37.60%. On Countdown at length 256, it's 49.61% versus 46.48%. This suggests teacher-guided RL can substitute for a separate SFT stage on some tasks, though not uniformly. On GSM8K, d1-LLaDA at length 256 reaches 70.51%, which is comparable to CanvasAnneal's 68.54%. The best individual configurations often combine SFT with CanvasAnneal: 70.96% on GSM8K at length 256, 36.20% on MATH500 at length 128, 48.05% on Countdown at length 128. But combining SFT with diffu-GRPO can also be competitive or better, as on Countdown at length 512 where d1-LLaDA gets 49.22% versus SFT+CanvasAnneal's 44.14%.
Tool use shows consistent gains on Tau2. CanvasAnneal beats diffu-GRPO across all three domains: Retail 7.90% vs 6.14%, Airline 18.00% vs 12.00%, Telecom 13.20% vs 12.30%, for an average of 13.03% versus 10.15%. The base model starts at 9.05%, so both RL methods improve it, but CanvasAnneal's improvement is noticeably larger. On BFCL, results are mixed: Simple improves from 53.33% to 59.08%, Multiple is comparable (65.50% vs 66.00%), but Parallel drops from 65.00% to 60.50%. The authors note this indicates task-dependent performance, with consistent gains on conversational tool-use tasks but less reliable improvement on function-calling benchmarks.
Training dynamics reveal the mechanism
The training curves clarify where the curriculum's advantage originates. On xLAM, CanvasAnneal reaches a reward of 1.0 in 7.4x fewer gradient steps than diffu-GRPO. On Countdown, it reaches 0.5 in 18.8x fewer steps. On MATH, CanvasAnneal hits its peak 228 updates earlier with slightly higher peak and mean reward, though its final reward is marginally lower. On GSM8K, standard diffu-GRPO is simply better throughout.
This pattern makes sense. The curriculum accelerates early exploration by providing a plausible reasoning scaffold, letting the policy quickly discover rewarded trajectories. On tasks where the reward surface is complex and hard to traverse from noise (MATH500, Countdown, tool use), this head start translates into faster convergence and sometimes better final performance. On GSM8K, where standard diffu-GRPO already finds good trajectories efficiently, the teacher guidance provides less marginal benefit and may even introduce suboptimal biases that the policy must then unlearn.
Ablations decompose the design choices
The mixture ratio ablation tests how much pure-noise training is needed. Training exclusively from trace-initialized canvases (0% no-trace) performs well in some long-context settings, including MATH500 at length 512 (46.20%), but performs worse in several shorter-generation settings. Increasing the proportion of pure-noise groups generally improves GSM8K and MATH500 performance at shorter lengths. The 75% no-trace / 25% injected configuration generally works best, achieving the strongest GSM8K at length 128 (60.80%) and Countdown at lengths 256 (51.95%) and 512 (53.91%). The 25% no-trace / 75% injected configuration also performs strongly, particularly on GSM8K at length 256 (66.41%). The key insight: including pure-noise groups exposes the policy to inference-time initialization, preventing a train-test mismatch. Teacher-guided rollouts are most useful as a supplement to, not a replacement for, unassisted rollouts. The paper notes that training exclusively from trace-initialized canvases is "generally insufficient."
The curriculum schedule ablation compares dynamic Beta schedules against uniform masking at generation length 256. Uniform sampling takes mask fractions from U(0.5, 1.0). Dynamic Beta schedules transition from low to high masking ratios over training. The alpha:1 to 10, beta:10 to 1 schedule reaches 39.00% on MATH (versus uniform's 34.20%) and 44.53% on Countdown. Other Beta schedules also outperform uniform on MATH, ranging from 36.20% to 38.60%. But uniform sampling still performs best on GSM8K (66.34%), with dynamic schedules ranging from 60.42% to 64.52%. The paper interprets this as evidence that progressively shifting probability mass toward larger masking ratios can improve performance on harder reasoning tasks, but the preferred schedule remains task-dependent.
Suffix masking versus random masking is tested at length 256 with the same dynamic curriculum. Suffix masking achieves 68.54%, 41.00%, and 49.61% on GSM8K, MATH500, and Countdown. Random masking with the identical schedule gets 60.58%, 37.80%, and 42.19%. Preserving a contiguous reasoning prefix consistently outperforms random token removal, which makes intuitive sense: the policy must continue a partially observed trajectory rather than reconstructing missing intermediate steps. Random masking forces the model to deal with a more fragmented input state that does not correspond to any natural partial completion of a reasoning trace.
Limitations and the broader picture
The paper is honest about what does not work. GSM8K gains are negative, meaning CanvasAnneal performs worse than standard diffu-GRPO on that benchmark. The tool-use gains on BFCL are inconsistent. The 40-problem training sets are small, and the single-teacher setup (Gemini 3.1 Pro) means the results depend on that particular teacher's reasoning style. The authors also note they did not exhaustively evaluate all interactions between trace-injection frequency, curriculum schedule, and masking structure.
From a practical standpoint, CanvasAnneal is attractive because it requires no teacher at inference time. The teacher traces are generated offline and the policy learns to complete reasoning trajectories autonomously. This preserves the parallel generation advantage of diffusion models. The method also does not require task-difficulty annotations or supervised fine-tuning pipelines, which simplifies the training recipe.
The broader implication for the diffusion-versus-autoregressive debate is that diffusion models' exploration problem is solvable with the right training scaffolding. The exploration bottleneck is real, but it is not fundamental to the architecture. Structured guidance from a teacher can warm-start the process, and a well-designed curriculum can phase that guidance out without leaving the policy dependent on external reasoning traces. Whether this scales to larger models and more complex reasoning tasks remains an open question, but the framework provides a clear template for further work.