Reinforcement learning post-training has become the backbone of modern foundation model alignment. Whether it is making a language model helpful and harmless or tuning a diffusion model to produce aesthetically pleasing images, RL fine-tuning is expensive, sometimes unstable, and must be rerun from scratch every time the reward function changes. A team at MIT CSAIL asks a deceptively simple question: if you already have a set of models trained on different rewards, can you predict the outcome of training on a new reward without actually running the training? The answer, they show, is yes. The paper, titled "PoEM: Predicting RL Outcomes from Existing Policies," was submitted September 24, 2026.
The Core Insight: Log-Policies Live in a Low-Dimensional Space
The theoretical foundation of the approach rests on an elegant observation about how RL modifies a model's behavior. When a model is trained with KL-regularized RL on a reward function, the optimal policy takes a specific mathematical form: it is the reference model multiplied by an exponential tilt of the reward. This means that if a new reward happens to be a linear combination of existing rewards, the corresponding optimal policy can be expressed as a product of the existing policies, each raised to a power equal to the weight of that reward in the linear combination.
This is the "exact composition" result. In the idealized world where every policy perfectly maximizes its reward and the rewards are truly linear combinations of one another, predicting the outcome of RL on a new reward requires nothing more than computing the right weighted mixture of already-trained models.
The authors then make a more surprising empirical observation. Even when the rewards are not linearly connected, the log-ratios between the post-trained policies and the reference model span a far lower-dimensional space than the reward geometry would suggest. Twenty experts trained with GRPO on twenty different programmatic rewards have near-orthogonal parameter updates in weight space, yet their log-ratios use only about half as many effective directions. This means that a new reward's optimal policy, even when it cannot be expressed as a literal linear combination of basis rewards, often lies inside the span of the existing log-policies.
This distinction between the parameter space and the policy log-ratio space is central. The authors observe that the weight updates of near-orthogonal adapters are close to full rank, while the log-ratios collapse to far fewer dimensions. A visualization of the log-ratio matrix, sorted by its first principal component, shows that nearly all experts move together along one shared direction. The policy space is far more constrained than the reward space.
How PoEM Works in Practice
The PoEM framework operationalizes these insights through a four-step pipeline. Given a basis of single-reward post-trained adapters and a new target reward, the system first scores a small calibration set of prompts using both the new reward and the basis rewards. It then estimates composition weights by regression — either in reward space or in policy log-ratio space — and checks whether the target reward falls within the span of the existing policies using a coverage score. Finally, it composes the basis policies at inference time, with no parameter updates and no additional RL run.
For autoregressive language models, the composition is implemented as a token-level log-ratio mixture. At each generation step, the log probability of the next token under the PoEM policy equals the reference model's log probability plus the weighted sum of the basis policies' log-ratios. This is computationally efficient because it requires only the next-token logits of the reference and each basis model, and it performs no training. The authors note that this local token-level composition is an approximation to the globally normalized sequence distribution, but the two are similar when continuation normalizers do not depend on the generated path.
For diffusion models, the composition takes a different form. The denoiser for the target reward can be expressed as a linear combination of the existing denoisers plus a correction term. When the basis policies are well-aligned, this correction term is small, and the linear combination serves as a good approximation to the target denoiser. The key result is that this allows skipping RL training for the new reward entirely.
Recovering the composition weights is the crux of the method. When the basis rewards are directly available, a ridge regression on the calibration set estimates the reward-space coefficients. When only the basis policies are available, the implicit reward of each trained policy can be recovered from its log-ratio against the reference model, and the weights are estimated through policy-space regression. In both cases, the regression requires only a small calibration pool of prompts and responses, not the expensive RL training run that would normally be needed.
Empirical Validation Across Text and Image
The experiments span multiple bases, reward types, and modalities. On the programmatic reward basis, the authors train 2,020 LoRA adapters on Qwen3-0.6B using either GRPO or DPO, each adapted to one of 20 programmatic rewards that test properties like vocabulary diversity, formality, and readability. For the combined reward experiments, the target rewards are exact linear combinations of two to sixteen basis rewards, with weights ranging from one dominant reward to uniformly mixed.
The results show that PoEM recovers most of the reward gain that a directly trained RL policy achieves on composite rewards, with an error close to the difference between two independent RL runs. In other words, composing existing models nearly matches what you would get from training a fresh model on the combined reward — without the training cost.
The more challenging setting involves held-out rewards that are not linear combinations of the basis. Using a basis of ten diverse public reward models adapted with PPO, PoEM's composed policy is closer to the directly trained RL policy than the leading single expert on nine out of ten held-out rewards. The coverage score, which measures the explained variance of the target reward against the policy log-ratio features, successfully ranks in advance which held-out rewards PoEM can reach well and which it cannot.
For image generation, the authors work with thirteen DDPO adapters on Stable Diffusion v1.4 trained on image statistics and image reward models including aesthetic score, PickScore, and CLIP score. PoEM successfully approximates a held-out RL adapter from the remaining basis, demonstrating that the approach transfers beyond text to generative image models.
The Geometry of Policy Space
A particularly striking finding is the structural gap between reward geometry and policy geometry. The cumulative variance analysis reveals that while the weight updates of twenty different experts are close to full rank — meaning each expert has moved the model in a distinct direction in parameter space — the log-ratios span far fewer effective directions. The rewards themselves vary along many more dimensions than the policies that optimize them.
This has a practical implication that the authors term the "policy space rank gap." A new reward that is not a linear combination of the basis rewards may still have its optimal policy inside the span of the existing log-policies. This is because the policies, despite being trained on different objectives, have moved the model along largely overlapping behavioral directions. The low-rank structure of policy space is the enabling condition that makes PoEM work beyond the strict linear reward case.
The coverage score formalizes this: it measures the R-squared of the target reward against the policy log-ratio features. High coverage means the target reward lies close to the span of the existing policies, and PoEM can be safely used. Low coverage means the new reward requires genuinely new training. This gives practitioners a diagnostic tool to decide whether to use PoEM or invest in a fresh RL run.
Implications and Future Directions
The practical implications of PoEM are significant. Training foundation models with RL is expensive — sometimes running into thousands of GPU hours per reward function. The ability to predict RL outcomes by composing existing policies could dramatically reduce the cost of aligning models with multiple, evolving, or user-specific reward functions. A company that has already fine-tuned a model for safety, helpfulness, and formatting could quickly compose a new policy for a different use case without launching a new training run.
The composition strength adjustment adds another layer of practicality. Because different RL recipes rescale rewards differently — GRPO normalizes by within-prompt standard deviation, DPO only keeps ranking information, PPO preserves the reward scale — the composed log-ratios can be systematically too weak. The geometric scaling factor automatically compensates for this without requiring access to the target reward or any training. This makes PoEM robust to the varied training recipes that different practitioners use.
The paper also connects to broader questions about policy space structure and successor feature transfer. The reusable features here are given by previous post-training runs rather than being pre-specified, which is a different paradigm from earlier work on transfer between tasks. The coverage score provides a principled way to know when the existing basis is sufficient and when it is not.
Limitations remain. The approach assumes that the basis policies were trained from a shared reference model with the same KL coefficient, which may not hold in all practical settings. The calibration set must be representative of the distribution on which the composed policy will be evaluated. For tasks where the target reward falls outside the span of existing policies — when coverage is low — PoEM cannot substitute for training, and the authors acknowledge that abstention or new basis creation is necessary in those cases.
Overall, PoEM introduces a compelling framework for what the authors call "predicting RL outcomes from existing policies." By exploiting the low-dimensional structure of policy space, it turns the question of "can I avoid retraining" from a theoretical curiosity into a practical tool validated across text and image modalities, with a principled coverage diagnostic to guide its application.