Why GRPO's Importance-Sampling Ratio Is the Wrong Weight
Reinforcement learning with verifiable rewards (RLVR) has become the standard recipe for teaching language models to reason. The pattern is straightforward: generate multiple responses to a math problem, use a verifier to score them, and push the model to produce more correct answers. Group-Relative Policy Optimization (GRPO) popularized this approach by computing advantages from group-normalized rewards and applying PPO-style clipped surrogates with per-token importance-sampling ratios. But the importance-sampling ratio, which measures how the current policy's token probability compares to the rollout policy's, is a blunt instrument for credit assignment. It does not distinguish between tokens that matter for the final outcome and tokens that are just along for the ride. Every token in a response receives the same group-normalized advantage, and the importance ratio scales each gradient equally. Song, Xu, Zhang, and Bing (Apodex US and Princeton) propose Bellman Policy Optimization (BPO), a critic-free method that replaces this ratio with a weight derived from the Bellman equations, achieving substantially stronger results on mathematical reasoning benchmarks.
The Problem with Token-Level Importance Ratios in Autoregressive RL
In GRPO, the per-token loss is defined as the negative minimum of the importance ratio times the advantage and the clipped version of the same product. The importance ratio for token t in response i is rti = pi(yti | sti) / mu(yti | sti), where pi is the current policy and mu is the rollout policy. This ratio measures how much the current policy has shifted its probability on the sampled token relative to the rollout. The clipping mask zeros out gradients when the ratio drifts too far in the wrong direction relative to the advantage sign.
The issue is that this ratio treats every token uniformly. In autoregressive generation with terminal rewards (a verifier scores the final response), all tokens in a correct response share the same advantage, and all tokens in an incorrect response share the same disadvantage. But the tokens are not equally important. Some tokens are critical branching points that determine whether the reasoning goes down a correct path. Others are fillers that would be generated correctly under almost any reasonable policy. The importance ratio does not account for this heterogeneity. It can suppress learning on important tokens if the current policy has shifted probability mass away from them, or amplify noise on unimportant tokens where the policy shift is coincidental.
The value function in standard RL assigns credit by decomposing the return into advantages at each state. But training a value model for autoregressive generation is expensive, requires additional memory, and can produce inaccurate estimates on long-horizon reasoning tasks. GRPO sidesteps the value model entirely, but pays the price with its flat token-level weighting. BPO aims to recover the benefit of value-aware credit assignment without the cost of a learned critic.
How Bellman Equations Eliminate the Critic
The key insight is that for terminal-reward MDPs, the Bellman equations create a telescoping relationship. The value at any state is the expected reward from that state onward. For autoregressive generation, the value at the initial state (the prompt) is the expected reward under the rollout policy. The value at each intermediate state is the expected reward given the prefix generated so far. The advantage at step t equals the difference between the value at the next state and the value at the current state: Amu(st, yt) = Vmu(st+1) - Vmu(st).
Summing these advantages over the entire trajectory, they telescope. The intermediate values cancel out, leaving only the terminal reward minus the initial value: sum of Amu = R(x,y) - Vmu(x). This is the crucial identity. It means that the total advantage signal for an entire response can be expressed using only two quantities: the verifier's score and the expected reward for the prompt under the rollout policy. No intermediate state values are needed.
The initial value Vmu(x) is just the mean reward of the sampled responses for that prompt, which is directly estimated from the rollout data. The terminal reward is provided by the verifier. So the entire advantage structure can be reconstructed without training a value model at all.
From PMD to the BPO Loss
BPO is derived from Policy Mirror Descent (PMD), which optimizes a policy by maximizing expected advantage minus a KL divergence penalty from the rollout policy. The PMD optimality condition gives a closed-form update for the optimal policy at each state. The authors show that this update can be reformulated as a trajectory-level objective that equals zero at the PMD optimum. Minimizing this objective recovers the same unique optimal solution as the original PMD problem, and they prove this equivalence rigorously using a martingale argument.
The practical BPO loss is obtained by linearizing this objective around the rollout policy and approximating the gradient. Three approximations produce the final form. First, the initial value is estimated as the mean reward within each group of sampled responses. Second, the full reverse KL divergence is replaced with a binary KL divergence, which partitions the vocabulary into the sampled token and its complement. This binary approximation has a clean gradient identity: the gradient of the log-probability plus the binary KL equals the ratio (1 - mu(y|s)) / (1 - pi(y|s)) times the gradient of the log-probability. Third, additive smoothing is applied to this ratio for numerical stability, producing the mismatch-correction weight:
omegati = (1 + epsilon - mu(yti | sti)) / (1 + epsilon - pi(yti | sti))
This weight replaces the importance-sampling ratio in GRPO. When the current policy's probability on the sampled token is close to the rollout probability, the weight is close to one, and the update looks like standard policy gradient. When the current policy assigns much higher probability to the token than the rollout did, the weight shrinks below one, damping the gradient. When the current policy assigns much lower probability, the weight grows above one, amplifying the gradient. The weight is capped at a constant C and clipped using the same asymmetric masking rule as GRPO-ClipHigher.
The key difference from GRPO's importance ratio is that this weight is asymmetric. It operates on the complement probability (1 - p) rather than the token probability p itself. This means it responds to how the policy has shifted on all the tokens that were not sampled, not just on the sampled token. The weight effectively asks: given that this token was generated under the rollout policy, how different is the current policy's belief about the space of alternative tokens? This is a more informative signal about whether the policy update is moving in a useful direction.
Experimental Results on Mathematical Reasoning
BPO is evaluated on Qwen3-30B-A3B-Base trained on the English subset of DAPO-Math-17k, a dataset of mathematical reasoning prompts. The comparison is against GRPO-ClipHigher (with the asymmetric clipping interval from DAPO), GSPO, CISPO, and DPPO. All methods share identical experimental settings except for the policy loss. Each rollout batch contains 256 prompts with 16 responses per prompt. One training step consists of one rollout batch followed by eight minibatch updates. Training runs for 400 steps (3200 optimizer updates). Evaluation uses AIME 2024, 2025, and 2026, with 32 responses sampled per question at temperature 1.0 to estimate Pass@1 via Avg@32.
The main results show BPO achieving a peak average accuracy of 50.5% across the three AIME benchmarks. The baselines score 39.5% (GRPO-ClipHigher), 43.5% (GSPO), 47.4% (CISPO), and 46.4% (DPPO). The gains over the strongest baseline (CISPO) are 3.1 percentage points, and over GRPO-ClipHigher they are 11.0 points. BPO achieves the highest accuracy on all three individual benchmarks: 57.4% on AIME24, 41.0% on AIME25, and 53.0% on AIME26.
Training curves show that BPO also reaches higher final accuracy. After 400 steps, BPO's average accuracy is 49.4% compared to 45.5% for DPPO, the strongest baseline at that point. The gap widens in BPO's favor over the course of training, suggesting that the mismatch-correction weight provides a more stable and effective optimization signal than the importance-sampling ratio.
Hyperparameter Sensitivity and Robustness
Ablation studies on Qwen3-4B-Base test sensitivity to the two BPO hyperparameters: the smoothing parameter epsilon and the truncation constant C. For epsilon in {0.05, 0.1, 0.2, 0.3} with C fixed at 3.0, BPO achieves average accuracies of 25.8%, 25.4%, 25.5%, and 24.1% respectively. The performance is stable across epsilon values from 0.05 to 0.2, with a slight drop at 0.3. All settings outperform the GRPO-ClipHigher baseline at 20.5%. For C in {2.0, 3.0, 4.0} with epsilon fixed at 0.1, BPO shows similar stability. The default setting of epsilon=0.1 and C=3.0 works well across both model sizes.
The robustness to hyperparameters is notable because GRPO's performance can be sensitive to the clipping interval. The asymmetric clipping used in DAPO (with separate epsilon-low and epsilon-high values) was itself an improvement over symmetric PPO clipping, but it added two hyperparameters to tune. BPO's mismatch-correction weight and its single truncation constant appear to be less sensitive choices.
What the Mismatch-Correction Weight Actually Does
The weight omegati = (1 + epsilon - mu) / (1 + epsilon - pi) deserves careful interpretation. Consider what happens at different operating points. When pi(y|s) = mu(y|s), the weight is exactly 1, identical to an unweighted policy gradient. When pi(y|s) > mu(y|s), meaning the current policy has become more confident in the sampled token, the denominator is smaller than the numerator, so the weight exceeds 1. This amplifies the gradient, pushing the policy further toward tokens it already prefers. When pi(y|s) < mu(y|s), meaning the policy has shifted probability away from the sampled token, the weight falls below 1, dampening the gradient.
This behavior is the opposite of what you might expect from a "correction" weight. It seems like it would amplify tokens the policy already likes and suppress tokens it has moved away from. But the interaction with the advantage sign matters. For tokens in correct responses (positive advantage), amplifying tokens the policy prefers pushes it toward solutions it has learned to generate. For tokens in incorrect responses (negative advantage), the clipping mask zeros out gradients when the weight pushes too far, preventing destructive updates. The net effect is that the weight acts as a dynamic trust region that adapts per-token based on how the policy distribution has shifted, rather than applying a fixed clip like PPO.
The connection to binary KL divergence is also important. The weight emerges from approximating the full reverse KL with a binary partition. This approximation is exact in the sense that the gradient identity holds without error. The binary KL is a lower bound on the full KL, which means the BPO update is more conservative than a full PMD update. This conservatism may contribute to the training stability observed in the experiments.
Practical Implications for RLVR Training
For practitioners training reasoning models, BPO offers a concrete improvement over GRPO with minimal additional complexity. The loss function has the same structure: per-token gradient, advantage, clipping mask, and a scalar weight. The only change is replacing the importance-sampling ratio with the mismatch-correction weight, which requires computing the rollout policy's token probability (already available) and applying additive smoothing. No additional model, no additional forward passes, no additional memory.
The theoretical grounding is also appealing. GRPO's clipped surrogate is justified primarily by its empirical success and loose connections to trust-region methods. BPO's loss is derived from a provably equivalent reformulation of Policy Mirror Descent, with a clear mathematical identity showing that the trajectory-level objective recovers the PMD optimum. The martingale argument that proves necessity (every optimal solution of the reformulated objective must match PMD on all reachable states) is a strong theoretical guarantee that the approximation does not lose information.
The results also suggest that the choice of per-token weighting matters more than previously appreciated. GRPO assigns uniform advantage to all tokens in a response and lets the importance ratio handle the rest. BPO shows that a more principled weighting, derived from the Bellman structure of the problem, produces measurably better outcomes. This opens a direction for further work on token-level credit assignment that goes beyond the uniform-advantage assumption.
Limitations and Open Questions
The paper does not address several important aspects. The evaluation is limited to mathematical reasoning on AIME benchmarks. Whether BPO's advantages hold for other domains (code generation, scientific reasoning, general instruction following) is unknown. The model is a 30B-parameter MoE (Qwen3-30B-A3B-Base), and it is unclear how the method scales to larger or smaller models. The experiments use a fixed training budget of 400 steps, and while the curves suggest BPO continues to improve, the comparison at different training durations could reveal different dynamics.
The connection between the mismatch-correction weight and practical optimization behavior deserves more analysis. The weight is always positive (for probabilities in [0,1] with epsilon > 0), so it never flips the gradient direction. But its magnitude can vary significantly across tokens. Understanding when and why this variability helps optimization, and when it might hurt, would strengthen the practical guidance for using BPO.
Finally, the paper does not compare against value-function-based methods that train a critic. DPPO (which uses a learned value model) scores 46.4% on the benchmarks, lower than BPO's 50.5%. But DPPO is known to have training instability issues that affect its final performance. A fairer comparison would involve tuning the value model more carefully. The theoretical promise of critic-free methods is appealing, but the empirical question of whether a well-tuned critic could match or exceed BPO's performance remains open.
Read the paper on arXiv