Reinforcement learning post-training has become a standard approach for improving the capabilities of large models, but its application to diffusion transformers has been costly and poorly understood. The prevailing assumption has been that enhancing visual quality, text rendering, and compositional accuracy requires learning complex, input-dependent transformations prompted by the diverse conditioning signals these models receive. A team of researchers challenged this assumption by investigating whether the substantial gains from RL post-training could instead be captured by a much simpler mechanism: a fixed set of activation steering vectors applied to the outputs of a frozen base model.

The paper begins by noting that capabilities like rendering requested text or composing multiple objects in diffusion generation are inherently prompt-dependent. From this, a natural hypothesis emerges: improving these capabilities requires the model to learn transformations that vary with the input. To test this, the authors probe each RL-tuned model with a deliberately simple parameterization—not learned per-prompt, but a fixed set of steering vectors added to the block outputs of the frozen base model. On SD3.5-M, these vectors hold 65x fewer parameters than the RL-tuned LoRA (0.29M against 18.8M), representing a dramatic reduction in parameter count.

The authors fit these steering vectors to match the velocity predictions of public RL-tuned checkpoints, with no reward in the objective. The vectors are identical for every prompt, token, and sampling seed—yet they recover 90.0–96.3% of the reward gains of six RL-tuned image models. On two RL-tuned video models, the recovery rates are 89.8% and 111.9%, respectively. The fact that the second video model exceeds 100% suggests the steering vectors not only capture the RL gains but potentially uncover additional improvement directions that the original RL objective did not fully exploit.

An especially compelling finding involves vectors initialized at zero and trained with online RL. These vectors, acting alone, exceed the reward gains of weight-based RL on text rendering (OCR) and human preference (PickScore) on SD3.5-M. This result is striking: a constant shift, with no input-dependent transformations, can steer the pretrained generator toward capabilities it already computes in latent space. The authors interpret this as evidence that policy-gradient RL post-training does not primarily introduce new computational pathways but rather amplifies or realigns what the frozen generator is already capable of producing.

The problem of RL post-training in diffusion transformers

Diffusion transformers have achieved remarkable results in image and video generation, but their default behavior often falls short on tasks that require precise text rendering or complex compositional reasoning. Reinforcement learning post-training addresses these gaps by optimizing a reward model that evaluates generated outputs against human preferences or automatic metrics. The resulting improvements can be substantial: users report better text fidelity, more accurate object counting, and more coherent multi-element compositions.

However, RL post-training is computationally expensive. It requires multiple rounds of policy optimization, reward model evaluation, and checkpoint selection. The parameter overhead is significant as well: LoRA adapters applied during RL training can introduce millions of additional parameters. On SD3.5-M, the RL-tuned LoRA contains 18.8M parameters, while the steering vectors proposed in this work contain only 0.29M—roughly 65x fewer. This parameter disparity raises a fundamental question: if such a small, input-independent perturbation can capture most of the RL benefit, what exactly is the RL objective achieving?

The authors' decision to fit steering vectors to match velocity predictions rather than directly optimizing reward is worth pausing on. Velocity predictions refer to the direction in latent space along which the model should move to improve the generated output. By aligning steering vectors to these velocity signals without any reward signal, the authors demonstrate that the RL-tuned checkpoints encode a consistent latent direction that can be approximated by a fixed vector. This is non-trivial because one might expect the optimal direction to depend on the conditioning prompt, the current timestep, or the noise configuration. The fact that a single vector works across all these variables suggests that the RL process converges to a relatively low-dimensional subspace of improvements.

Parameter efficiency and the scale of the discovery

The 65x parameter reduction is not merely an engineering detail; it fundamentally changes the economics of RL post-training. A diffusion modeler working with limited compute resources could apply these steering vectors without needing to replicate the full RL training pipeline. The vectors can be stored, shared, and applied post-hoc, much like LoRA weights but with an order of magnitude less storage and computation.

This efficiency also has implications for model auditing and transparency. When RL post-training introduces 18.8M parameters, it becomes difficult to isolate which changes contribute to which capabilities. Steering vectors, by contrast, are transparent: they are simple additive perturbations that can be inspected, ablated, or modified individually. If a particular steering vector improves text rendering but harms compositional accuracy, its effect can be measured and adjusted without retraining the entire model.

The paper's recovery rates—90.0–96.3% on image models and 89.8% and 111.9% on video models—suggest that the steering vectors capture the dominant direction of improvement. The video model exceeding 100% recovery is particularly noteworthy. It could mean that the RL objective for video was slightly misaligned with the true preference signal, and the steering vectors correct this misalignment. Alternatively, it could indicate that the video diffusion transformer has more "headroom" for improvement through simple latent shifts than the image models do.

Zero-initialized vectors trained with online RL

The most provocative experiment in the paper involves steering vectors that are initialized at zero and then trained with online RL. Online RL means the vectors are updated continuously during generation, rather than being pre-computed and applied offline. These vectors, in isolation, surpass the reward gains of weight-based RL on two key metrics: OCR text rendering and PickScore human preference, both on SD3.5-M.

This finding directly addresses the paper's central thesis. If randomly initialized, zero-magnitude vectors can, through online RL, outperform the full weight-based RL objective, then the weight updates during RL post-training are not introducing fundamentally new functions. Instead, they are fine-tuning the existing parameters toward a direction that the already-trained generator can already navigate. The authors' characterization—that a constant shift suffices for most of the gain of policy-gradient RL post-training without new input-dependent transformations—is supported by this experiment.

One might ask whether the online RL training of these vectors is itself a form of "new transformation." The authors would likely argue that the transformation is still input-independent: the same vector is applied regardless of prompt, and the learning signal comes from a reward model that evaluates the final output. What changes is the magnitude and direction of the shift, but the functional form remains a fixed additive perturbation.

Limitations and future directions

The paper acknowledges that steering vectors cannot capture everything that RL post-training achieves. The 4–10% of reward gains not recovered by the vectors likely correspond to benefits that do depend on input conditioning—perhaps changes to attention patterns, adapter modules, or timestep-dependent modulations that vary across prompts. The authors also note that their experiments were conducted on SD3.5-M; whether the same phenomenon holds on other diffusion architectures (e.g., Flux, Stable Diffusion v3) remains an open question.

Another limitation is the single-vector-per-model approach. The paper uses a fixed set of steering vectors, but the dimensionality of this set and how it scales with model size are not fully detailed in the abstract. Practitioners will want to know how many vectors are needed, whether they should be prompt-specific, and how the vectors interact with other optimization techniques like classifier-free guidance or adaptive sampling schedules.

Finally, the paper's interpretation that RL post-training primarily realigns rather than introduces new capabilities is compelling but not exhaustive. It could be that RL both realigns existing pathways and modestly expands the model's representational capacity through weight updates that enable previously inaccessible latent trajectories. The zero-initialized vector experiment strongly suggests the realignment component is dominant, but a complete picture would require ablation studies that isolate the contribution of weight changes versus activation shifts.

What it means for working developers

For developers building or fine-tuning diffusion transformers, this paper offers a practical takeaway: the heavy computational cost of RL post-training may be partially avoidable. If the primary goal is to improve text rendering or human preference alignment, a modest set of steering vectors applied to a frozen model can capture the bulk of the benefit. This does not mean RL post-training is obsolete—its remaining gains may be precisely the ones that matter for edge cases, domain-specific adaptations, or pushing performance to the frontier—but it does mean that the field should reconsider how much resources are allocated to weight-based RL versus activation-based interventions.

The steering vector approach also opens a pathway for rapid experimentation. Developers can fit vectors to match velocity predictions from a small set of RL checkpoints, test different vector initializations, and explore online versus offline training schemes without the overhead of maintaining and updating LoRA adapters. The 65x parameter reduction translates to faster inference, lower memory pressure, and simpler model deployment pipelines.

On the research side, the finding that a constant shift can steer a pretrained generator toward what it already computes invites new questions about the geometry of diffusion latent space. Why should a single direction in block output space capture the aggregate improvement of an RL objective? What structure in the model's parameterization makes this possible? Future work could probe the singular value decomposition of the RL weight updates, examine the alignment between velocity predictions and steering vectors across timesteps, and explore whether the same phenomenon holds for other generative model families.

Ultimately, this paper reframes our understanding of RL post-training in diffusion models. The breakthrough is not merely that steering vectors are efficient—they are—but that they reveal the RL objective's primary mechanism: not the introduction of novel computations, but the amplification of latent capacities already present in the pretrained model. This perspective may prove as valuable as the practical efficiency gains, guiding future work toward more targeted, interpretable, and computationally economical methods for improving diffusion model performance.

Read the paper on arXiv