Vision-language models are being adopted as reward functions for robotic learning at an accelerating pace. The idea is straightforward: feed a robot trajectory and a text description of the goal into a VLM, and the model outputs a scalar score estimating how far the robot has progressed toward completing the task. This removes the need to hand-engineer reward functions or collect dense human feedback. But there is a property that these models must satisfy to work reliably in reinforcement learning, and most of them fail it.

Why Paraphrase Invariance Matters for Robot Rewards

In reinforcement learning, reward predictions determine which trajectories get reinforced during exploration. If a reward model assigns different scores to the same robot behavior depending on how the goal is worded, the learner gets conflicting signals about what counts as progress. A user might write "pick up the radish and place it in the pink bowl." Another might write "place the radish in the pink bowl after picking it up." These are semantically identical instructions, but the reward model could score the exact same trajectory as a failure under one phrasing and a success under the other. This isn't a theoretical concern. The authors demonstrate it concretely, showing cases where two meaning-preserving rewrites of a goal instruction produce opposite categorical judgments for identical robot behavior.

The practical consequence is severe. When reward predictions vary with instruction wording rather than actual robot performance, the optimization process chases linguistic artifacts instead of task completion. This can slow policy improvement, select suboptimal trajectories, and ultimately produce robots that behave poorly because the training signal was driven by phrasing rather than progress.

RoboRMBench: A Benchmark Built on Real Robot Trajectories

The paper introduces RoboRMBench to measure this failure mode systematically. The benchmark starts with 2,390 real-robot trajectories drawn from Open X-Embodiment and RoboArena, spanning 14 robot embodiments, both exocentric and egocentric camera views, and diverse manipulation tasks. Each trajectory has a ground-truth progress label verified by humans.

From these trajectories, the authors generate paraphrases using three strategies with increasing linguistic divergence: lexical substitution (replacing content words with synonyms), syntactic restructuring (changing sentence structure while preserving meaning), and action-goal perspective shift (rewriting action-centric descriptions as goal-state descriptions, or vice versa). The key insight is that paraphrases closer to the original phrasing are easier for models to handle, while structural and perspective-level rewrites are where things break down.

Before any paraphrase enters the benchmark, it passes through a semantic-equivalence filter. An ensemble of three independent models (gemini-3-flash, claude-sonnet-4-6, and deepseek-v3.2) checks each candidate, and a paraphrase is discarded if any model flags a meaning change. The authors validate this filter against human annotations on 210 sampled paraphrases, achieving 96% recall on human-flagged inequivalent paraphrases while being intentionally conservative. The final benchmark contains 21,673 validated paraphrased instructions across 2,390 trajectories.

Two Metrics That Capture Different Failure Modes

The evaluation protocol uses three metrics. Score Crossing Rate (SCR) maps predicted scores to failure-level (scores 1-2) and success-level (scores 4-5), treating score 3 as intermediate. A trajectory exhibits a score crossing if at least one paraphrase gets a failure-level prediction and at least one gets a success-level prediction. SCR is the fraction of trajectories where this happens. It's reference-free: it doesn't depend on which paraphrase is treated as canonical, and it directly measures whether equivalent goal descriptions can produce contradictory reward judgments.

Flip Rate (FR) uses the original RoboRewardBench instruction as a reference and measures how often individual paraphrases flip the failure-success judgment relative to that reference. Mean Error (ME) averages predicted scores across paraphrases and compares them to ground-truth progress labels, measuring prediction accuracy separately from robustness.

The distinction between SCR and ME matters. A model can have low prediction error while still producing contradictory judgments across paraphrases, and a model can be robust to paraphrasing while remaining systematically miscalibrated.

The Numbers: Instability Is Widespread and Severe

Across proprietary and open-source VLMs, paraphrase-induced instability is substantial. Under action-goal perspective shifts, Gemini2.5-flash-lite and Llama4-scout cross the failure-success boundary on more than half of all trajectories. That means the same robot behavior gets labeled as success or failure depending solely on instruction wording.

The instability grows with linguistic divergence. For most models, SCR and FR increase from lexical substitution to syntactic restructuring to action-goal perspective shift. GPT-5.1, for example, shows SCR of 0.153 under lexical substitution, 0.204 under syntactic restructuring, and 0.300 under action-goal perspective shift.

Claude-sonnet-4.6 is the most robust general-purpose VLM under lexical substitution with SCR of 0.051, but its ME stays near 1.0, indicating stable but biased predictions. Gemini3-flash achieves the lowest ME among general-purpose VLMs under lexical substitution at 0.681, but has substantially higher SCR at 0.271, showing better average accuracy but weaker invariance to paraphrasing.

These results demonstrate that accuracy and robustness are distinct properties. A model can be accurate on average while still changing its categorical judgment under equivalent instructions, or robust to paraphrases while remaining systematically miscalibrated.

Scale and Reasoning Don't Help

Two findings challenge common assumptions. First, increasing model scale does not improve paraphrase robustness. Within the Qwen3-vl and Gemma3 families, larger models tend to exhibit higher SCR across paraphrasing strategies. Second, enabling explicit reasoning does not reliably reduce instability. Comparing reasoning-enabled and non-reasoning variants of Qwen3-vl-235B-A22B and Gemini3-flash, the authors find that enabling reasoning actually increases SCR for both models. For GPT-5.1, reasoning helps, but for Claude-sonnet-4.6, it reduces robustness. The effect is inconsistent.

This is counterintuitive. You might expect that a model capable of chain-of-thought reasoning would be better at recognizing that two semantically equivalent instructions describe the same task. The data says otherwise. Paraphrase fragility appears to be a structural property of how these models process language, not something that can be patched with more parameters or reasoning steps.

Dedicated Reward Models Succeed Where General-Purpose VLMs Fail

The dedicated reward models RR-4B and RR-8B, trained with trajectory-grounded reward supervision, are substantially more stable. RR-4B achieves SCR of 0.034 under lexical substitution, below all general-purpose VLMs, and both models maintain SCR below 0.12 even under the most divergent action-goal perspective shifts. They also achieve the lowest ME across all paraphrasing strategies.

The RR models are smaller than most proprietary VLMs tested, yet they are more robust. This suggests that task-specific reward supervision matters more than generic model capability for paraphrase robustness. General-purpose VLMs are trained on broad multimodal objectives and may rely heavily on textual cues in the instruction, whereas the RR models are trained directly on trajectory and reward data, learning to ground reward predictions in visual behavior rather than linguistic patterns.

Why This Affects Real Policy Improvement

The paper goes beyond diagnostic metrics to show that paraphrase instability has downstream consequences. In offline best-of-NN trajectory selection with N=20, models with lower SCR select trajectories with higher ground-truth progress and lower regret, even when average prediction error is similar. Across 11 models, SCR is strongly correlated with regret (Pearson r=0.882, Spearman rho=0.856).

In pairwise trajectory preference evaluation, where the model must rank two trajectories from the same task by their ground-truth progress, lower SCR correlates with higher pair accuracy (Pearson r=-0.893, Spearman rho=-0.855). Claude-sonnet-4.6, with SCR of 0.164 under perspective-shift paraphrases, achieves pair accuracy of 0.713 and selected GT of 3.410. RR-8B, with SCR of 0.114, achieves pair accuracy of 0.855 and selected GT of 3.639. The gap between a model that picks the right trajectory 71% of the time versus 86% of the time is the difference between a robot that completes its task and one that doesn't.

Mitigation: Aggregation and Training-Time Penalties

The authors explore two mitigation strategies. Paraphrase aggregation averages predictions across multiple semantically equivalent variants of the same instruction. For each trajectory, the model generates five additional variants from each seed prompt and averages the predicted scores. This consistently reduces both SCR and prediction error across all evaluated models and paraphrasing strategies. The downside is that it requires multiple reward-model queries per prediction.

Variance reduction training adds a penalty term to the loss function that encourages the model to produce consistent rewards for semantically equivalent descriptions. The loss combines standard prediction loss with a variance term over reward predictions across paraphrase sets. Applied to Qwen3-VL-4B, this approach reduces SCR from 0.076 to 0.042 under lexical substitution, from 0.168 to 0.048 under syntactic restructuring, and from 0.200 to 0.057 under action-goal perspective shift. It also improves ME dramatically, from over 1.0 to around 0.24-0.37. Unlike aggregation, this approach incurs no additional inference-time cost.

Limitations and What Remains Unknown

The benchmark is limited to English instructions and end-of-episode reward predictions. Multilingual or code-switched settings may exhibit different robustness properties. Step-level reward shaping or trajectory comparison setups may also behave differently than the end-of-episode evaluation used here. The semantic-equivalence filter, while validated, is conservative by design and may have excluded paraphrases that humans would consider equivalent.

What This Means for Developers

If you are building robotic systems that use VLMs as reward functions, this paper establishes a hard requirement: your reward model must be paraphrase-robust. Testing with a single canonical instruction is insufficient. The authors recommend using multiple paraphrases and checking whether the model's categorical judgments remain consistent across semantically equivalent descriptions.

The strong performance of the RR-4B and RR-8B models suggests a practical path forward: train dedicated reward models on trajectory-grounded data rather than relying on general-purpose VLMs. The variance reduction training technique provides a concrete method for improving robustness during training without inference-time overhead.

RoboRMBench is available as a framework for measuring this property. The benchmark's 2,390 trajectories and 21,673 validated paraphrases provide a standardized testbed for evaluating whether new reward models maintain consistent judgments across equivalent goal descriptions.

The core lesson is that paraphrase invariance is not a nice-to-have property for VLM-based reward modeling. It is a structural requirement that determines whether the reward signal reflects actual robot progress or linguistic accidents. Until reward models satisfy this property reliably, their use in robotic reinforcement learning will remain fragile.