A Zeroth-Order Paradigm for LLM Preference Alignment

The Likelihood Displacement Problem in Preference Alignment

Training large language models to follow human preferences has become a core bottleneck in shipping reliable AI systems. Reinforcement learning from human feedback (RLHF) was the original solution, but it requires learning a reward model and then running reinforcement learning on top of it, an expensive multi-stage pipeline. Direct alignment methods like DPO and SimPO cut out the reward model entirely, optimizing the policy directly on preference pairs where one response is preferred over another. These methods are simpler, more stable, and cheaper to run.

But they have a well-documented flaw. When you train a model with DPO on a preference pair, you are maximizing the log-likelihood margin between the preferred and dispreferred responses. This is a relative signal. It says nothing about absolute likelihood. As a result, the preferred response's absolute probability can actually decrease during training. The model learns that preferred is better than dispreferred, but shifts probability mass to other, potentially unsafe responses. This is called likelihood displacement, and it can produce genuinely dangerous behavior. When Gemma-2B-it is fine-tuned with DPO on a prompt asking for steps to infiltrate a government agency, it initially refuses. After training, the model complies with the unsafe request because the likelihood of refusal responses drops even as the margin between preferred and dispreferred responses increases.

Recent analysis from Razin et al. (2025) shows that the worst offenders for likelihood displacement are preference pairs where the preferred and dispreferred responses are similar in some model-dependent sense. They call these "low-margin pairs." The current workaround is filtering them out entirely. But filtering discards potentially useful information. A pair where the two responses are similar in embedding space might still provide a meaningful directional signal: a good policy perturbation should increase the preferred likelihood and decrease the dispreferred likelihood simultaneously, even if the margin is small.

This is the core intuition behind ComPO (Comparison-based Preference Optimization), proposed by Peter Chen, Xi Chen, Wotao Yin, and Tianyi Lin from UC Berkeley, NYU, Alibaba, and Columbia University. ComPO reframes preference alignment from loss optimization to comparison: instead of optimizing a differentiable loss function on noisy pairs, it uses those pairs as comparison oracles to estimate update directions. The approach operates without gradients of a preference loss on the low-margin subset, making it a genuine zeroth-order method.

How ComPO Uses Comparison Oracles Instead of Loss Functions

The standard comparison oracle in optimization takes two parameter vectors and returns +1 or -1 depending on which one achieves a lower function value. For LLM alignment, the function is implicit. There is no single scalar objective for "alignment" that you can evaluate. Instead, ComPO defines a preference comparison oracle over a set S of preference pairs. Given current parameters theta and a perturbed copy theta', the oracle checks two quantities: the average change in log-likelihood of preferred responses across S, and the average change in log-likelihood of dispreferred responses. The oracle returns -1 (theta' is better) only if the preferred likelihood increased and the dispreferred likelihood decreased. Otherwise it returns +1.

This is a strict condition. A perturbation that improves the margin but decreases the preferred response's absolute likelihood will not be accepted. The oracle enforces that both sides of the preference pair move in the right direction.

To estimate an update direction from these one-bit signals, ComPO draws m random perturbation vectors from the unit sphere, queries the oracle on each, and solves a sparse 1-bit compressed sensing problem. The key insight is that LLM parameter gradients are approximately sparse: only a small fraction of the output-layer parameters need updating at any given step. By constraining the solution to be sparse, the number of required perturbations grows only logarithmically in the ambient dimension, not linearly. This is what makes the method tractable even for models with billions of parameters.

The Practical Pipeline: Partitioning Clean and Noisy Pairs

In practice, ComPO does not replace DPO. It complements it. The pipeline works in two stages. First, the dataset is partitioned using the reference model. Any preference pair where the absolute log-likelihood margin falls below a threshold delta_margin is classified as noisy. The rest are clean. DPO (or SimPO) is trained on the clean subset as usual. Then ComPO runs on the noisy subset, using comparison oracles to extract directional information that DPO's margin-based loss cannot safely use.

The practical implementation has several engineering decisions that distinguish it from the basic scheme. First, ComPO only perturbs the output-layer weights. The rest of the model is frozen. This is a significant computational savings: for a 7B model like Mistral-7B, the output layer is about 0.13B parameters. Second, instead of solving the full sparse recovery problem, ComPO normalizes the sum of signed perturbation vectors and applies entry-wise clipping, zeroing out gradient entries whose magnitude falls below a threshold lambda_g. Third, an update is only applied if the fraction of perturbations that returned -1 exceeds a threshold lambda. This acts as a gate: if fewer than lambda of the m perturbations indicate improvement, the model stays put.

The threshold lambda_g turns out to be the most important hyperparameter. At lambda_g = 2.2e-4, only about 1% of output-layer entries are updated per iteration. For Mistral-7B, that means roughly 1.5 million parameters change per step, about 0.02% of the full 7 billion. Performance degrades if too many entries (lambda_g = 0, all 100%) or too few (lambda_g = 2.5e-4, 0.15%) are retained. The sweet spot is sparse but not too sparse.

Hardware requirements are modest. ComPO on Llama-3-8B peaks at about 23 GB of GPU memory on a single A40, compared to 77 GB on an H100 for DPO. Runtime scales linearly with the number of perturbed parameters. For Mistral-7B with 1600 perturbations, the ComPO stage runs in about 50 seconds per iteration on 30 A40 GPUs.

Convergence Guarantees and Oracle Compatibility

The paper provides a best-iterate convergence guarantee for the offline scheme under three assumptions: the implicit objective is smooth, its gradient is approximately sparse, and the preference comparison oracle is compatible with this objective. Compatibility means the oracle returns -1 whenever the perturbed parameters achieve a lower value of the latent objective. Under these conditions, the gradient norm of the iterates drops below epsilon with high probability after T iterations, where T scales as O(Delta / epsilon squared) times a logarithmic factor in the dimension.

The number of comparison queries scales as O((1 + l*Delta/epsilon^2) * (s * log(2d/s) + log(2 + l*Delta*epsilon^{-2}/Lambda))), where l is the smoothness constant, Delta is the initial suboptimality, s is the sparsity level, d is the dimension, and Lambda is the confidence parameter. For fixed sparsity, this is polylogarithmic in the parameter count, which is the key practical result: you do not need to query the oracle a number of times proportional to the number of parameters.

The compatibility assumption is the most delicate part. The paper acknowledges that the objective f is latent and not explicitly defined. The convergence result is a theoretical benchmark, not a practical stopping criterion. But it establishes that the comparison-oracle approach is mathematically well-founded, not just heuristic.

Online ComPO: Reverse-KL Control Without New Labels

The offline method has a limitation: it uses a fixed set of noisy preference pairs and has no mechanism to prevent the policy from drifting too far from the reference. Online ComPO addresses this by adding reverse-KL regularization estimated from unlabeled generations of the current policy. The comparison oracle still operates on the fixed offline noisy pairs. No new preference labels are acquired. Instead, the model generates responses from its current policy on a set of online prompts, and the average log ratio of the policy to the reference policy is used to damp the step size.

The basic scheme uses an accept-or-reject rule: if a candidate update would push the reverse KL beyond a threshold tau, the update is rejected and the model stays at its current parameters. This preserves feasibility by construction. The practical scheme is softer: it estimates the reverse KL from online generations using a length-normalized statistic and uses it to scale down the step size when the policy drifts too far. The scaling factor is gamma / (1 + rho * max(d_hat - tau_p, 0)), where d_hat is the length-normalized reverse-KL estimate and rho controls the damping strength.

The theoretical analysis establishes that every iterate satisfies the reverse-KL constraint exactly (in the basic scheme), and that under local coverage, the performance gap between the current policy and the best policy in the reverse-KL neighborhood is bounded by C_tau times the square root of the in-distribution pairwise reward error. Local coverage is a standard assumption in offline RL: it requires that the reference policy assigns enough probability mass to all responses that the learned policy might assign non-negligible probability to. The coverage constant C_tau depends on the density ratio bound within the reverse-KL neighborhood, which is guaranteed by the KL constraint itself.

The practical scheme also introduces a replay mechanism. Training is divided into blocks of n iterations. At the start of each block, a buffer of previously successful preference mini-batches (those that passed the comparison gate) is used for replay with probability alpha. Each replayed mini-batch gets fresh perturbations around the current parameters. This recycles useful training signals while preventing overfitting to any single noisy pair.

Experimental Results Across Five Model Families

ComPO is evaluated on eight models from five families: Mistral-7B, Llama-3-8B, Gemma-2-9B, Qwen3-4B, Llama-3.2-3B, and Gemma-3-4B. Benchmarks include AlpacaEval 2 (with length-controlled win rates), Arena-Hard, and MT-Bench. All ComPO runs use 30 NVIDIA A40 GPUs with 46 GB memory each. The preference data is UltraFeedback throughout.

On Mistral-7B-Instruct, DPO_clean+ComPO achieves 26.17% length-controlled win rate on AlpacaEval 2, versus 23.89% for DPO_clean and 24.14% for DPO. On Llama-3-8B-Instruct, the improvement is from 32.92% (DPO_clean) to 35.79% (DPO_clean+ComPO). The gains are consistent across both base and instruction-tuned variants.

ComPO also stacks on top of SimPO. SimPO+ComPO on Gemma-2-9B-it reaches 62.42% LC win rate versus 60.36% for SimPO alone, with Arena-Hard staying at 61.1. On Llama-3-8B-Instruct, SimPO+ComPO achieves 49.53% versus 48.71%. The compatibility with both DPO and SimPO suggests ComPO is a general post-processing step, not tied to one specific alignment method.

Pair-level diagnostics provide direct evidence of likelihood displacement mitigation. The paper measures the pairwise log-likelihood of preferred and dispreferred responses before and after ComPO training. On Llama-3-Instruct-8B, starting from (-46.761, -47.410), ComPO with step size 1 produces values around (-46.744, -47.411) to (-46.753, -47.517) across three trials. The preferred response's log-likelihood stays roughly constant or increases slightly, while the dispreferred response's likelihood decreases. This is the opposite of what happens with DPO alone, where preferred likelihood can drop even as the margin increases.

Online Training Improvements

Online ComPO with reverse-KL damping and replay is evaluated on Qwen3-4B-Base, Llama-3.2-3B-Instruct, and Gemma-3-4B-it. Adding reverse-KL damping improves AlpacaEval 2 LC by 1.23 percentage points for Qwen3-4B and 2.07 points for Gemma-3-4B-it relative to offline ComPO. Adding replay (with a window of 50 iterations and replay probability alpha) provides additional gains across all three models. The full pipeline (DPO + ComPO + RKL + replay) on Gemma-3-4B-it reaches 42.55% LC win rate and 63.7 Arena-Hard win rate, versus 38.30% and 56.9 for DPO alone.

The combination of damping and replay is additive, not redundant. Damping prevents policy drift during the online generation step. Replay reuses successful noisy-pair mini-batches with fresh perturbations, allowing the model to extract more information from the same data without overfitting.

Ablations: Where the Gains Come From

Several ablations clarify the design choices. Increasing the number of perturbations from 800 to 5400 improves mean AlpacaEval 2 win rate from 17.32% to 19.69%, with diminishing returns. Memory is unchanged because ComPO accumulates a running average rather than storing all perturbation results. Perturbing three layers instead of one (adding MLPs in layers 30-31 of Mistral-7B) improves all metrics but adds about 0.4 GB of peak memory and 10 seconds per iteration. For most practical setups, single-layer perturbation is sufficient.

The gradient threshold ablation reveals the sparsity tradeoff. At 100% entries retained, win rate is 15.72%. At 1% retention, it peaks at 19.21%. At 0.15% retention, it drops back to 16.10%. The comparison oracle produces noisy one-bit signals, and the sparse recovery step filters out noise by concentrating on the strongest gradient entries. Too many entries lets noise through. Too few discards useful signal.

The number of noisy preference pairs also matters. Going from 100 to 300 pairs improves win rate from 19.21% to 20.07% and Arena-Hard from 11.02 to 11.76. The gains suggest that the noisy subset contains genuine information that ComPO can extract, not just noise.

A compatibility check shows that ComPO works even when applied directly to DPO checkpoints trained on the full dataset (including noisy pairs), without first retraining DPO on clean pairs only. DPO+ComPO achieves 27.03% LC on Mistral-7B-Instruct, comparable to DPO_clean+ComPO at 27.14%. This means ComPO can be used as a drop-in improvement on existing DPO models without retraining.

Limitations and Open Questions

ComPO has several limitations the authors acknowledge. The convergence theory assumes the latent objective is smooth and compatible with the oracle, but this objective is never explicitly defined. The result is a benchmark, not a practical guarantee. The practical scheme approximates the basic estimator with normalized sums and thresholding, and the connection between these approximations and the theoretical conditions is not formalized.

The output-layer-only perturbation is a practical choice, not a theoretical requirement. Multi-layer perturbation improves results but the paper does not analyze the tradeoff between perturbation depth and the sparsity assumption. For very large models, the output layer may be too narrow to capture all relevant gradient directions.

The online scheme's coverage assumption is not implied by the reverse-KL constraint. Coverage is a separate property of the reference policy and prompt distribution. If the reference policy does not cover the responses that the learned policy might assign probability to, the performance bound degrades. The paper does not provide methods for verifying or improving coverage.

ComPO is not designed to control verbosity, though the paper evaluates length-controlled win rates as a proxy for quality after length adjustment. Whether the comparison oracle mechanism could be extended to penalize verbosity directly is an open question.

What This Means for Practitioners

For developers already using DPO or SimPO, ComPO offers a concrete, low-cost improvement. The pipeline is straightforward: partition your preference dataset by likelihood margin, train your alignment method on the clean subset, then run ComPO on the noisy subset. The hyperparameters are modest (perturbation radius, number of perturbations, gradient threshold), and the method is compatible with existing training infrastructure. The memory footprint is smaller than DPO, and the code changes are minimal.

The deeper contribution is conceptual. Preference pairs where the two responses are similar are not noise to be discarded. They are comparison signals that a zeroth-order method can extract. This opens a path for other alignment methods that could benefit from non-gradient-based optimization on specific data subsets, particularly when the loss function's geometry makes gradient descent unreliable.

Future directions include extending ComPO to other settings like multi-turn dialogue and diffusion model alignment, applying it to reasoning tasks where preference signals are harder to formalize, and developing tighter theoretical bounds that connect the practical approximations to the convergence guarantees. The method's generality suggests it could become a standard post-processing step in the alignment pipeline, alongside techniques like filtering, regularization, and data curation.

Read the paper on arXiv