Two Clocks: Why Larger Rollout Batches Learn Slower but Fail on Schedule in Reinforcement Learning with Verifiable Rewards

Reinforcement learning with verifiable rewards RLVR relies on costly model-generated responses, motivating larger rollout batches and square-root learning-rate scaling. Reported to approximately preserve progress per sample with one optimizer step per rollout, this rule is less understood in practical multi-step training. This work develops a two-clock framework that separates the samples needed to make progress from those consumed before collapse.

The Two-Clock Framework for RLVR

Under a local quadratic model, the square-root rule preserves first-order drift per sample but not progress. Larger batches take fewer, longer updates and lose more to curvature, yielding a learning-rate-dependent critical batch beyond which sample efficiency degrades. The framework defines two distinct clocks: one governing the samples needed for meaningful progress, and another tracking samples consumed before the model's performance collapses. The drift-threshold hypothesis predicts that the second clock's sample count remains approximately invariant along the square-root rule, even as batch size varies.

Square-Root Batch Scaling and First-Order Drift

The reported square-root scaling rule suggests that increasing rollout batch size by a factor of k should allow learning-rate scaling by √k to preserve per-sample progress. However, the analysis reveals this preservation is limited to first-order drift only. When batch size grows, the optimizer takes fewer but larger steps, each subject to greater curvature effects. The quadratic approximation shows that the loss surface's curvature causes larger updates to overshoot or undershoot the true gradient direction, reducing effective progress per sample. This creates a critical batch size beyond which additional samples yield diminishing returns, and ultimately degrade sample efficiency.

The Drift-Threshold Failure Clock

A complementary failure clock emerges from the drift-threshold hypothesis. This clock tracks the number of samples consumed before the model's parameters drift beyond a threshold that guarantees meaningful improvement. Crucially, the hypothesis predicts this sample count remains approximately invariant along the square-root rule. Experiments support this: across a 16× batch range, the number of samples before performance collapse stays roughly constant, even though the optimizer configuration changes. This invariance explains why larger batches appear to consume samples faster—they simply reach the drift threshold more quickly in terms of optimizer steps, even though the sample count is preserved.

GRPO Experiments Across Batch Schedules

Controlled GRPO experiments on Qwen2.5-Math-1.5B across a 16× batch range support both predictions. The results show that larger batches indeed learn slower in terms of progress per sample, consistent with the curvature argument. Ratio clipping delays failure, especially for larger batches, but the fundamental progress gap persists without it. Widening the clip trades stability for a transient speed gain, temporarily improving performance but not eliminating the underlying efficiency problem. The progress gap reappears on another dataset, a general-purpose checkpoint, and a 7B model, confirming the framework's broad applicability.

Deriving a Multi-Step RLVR Schedule

Using the two-clock framework, the authors derive a simple multi-step RLVR schedule: start below the critical batch and switch to a larger batch before failure. This approach exploits the invariance of the drift-threshold clock while avoiding the efficiency loss that comes from staying at large batch sizes too long. The schedule begins with a conservative batch size that ensures good progress per sample, then transitions to a larger batch once the model has accumulated sufficient useful updates. The transition point is determined by monitoring the drift threshold, triggering the batch switch before the failure clock runs out.

This simple schedule addresses the core tension in RLVR: the need for large batches to amortize the cost of model-generated responses, versus the efficiency losses that come from excessively large batches. By operating below the critical batch initially and switching strategically, the approach achieves better overall sample efficiency than either extreme. Code released with the paper provides implementation details and the schedule logic for practitioners.