The Default Training Recipe for Language Agents Has a Hidden Blind Spot

When you fine-tune a language model on agent trajectories, the training loss is applied only to the tokens the agent wrote. The environment responses, terminal outputs, error messages, and tool results that appear in every trajectory are included in the context window but excluded from the prediction targets. This convention seems natural: the deployed model never generates observations, so why train it to predict them?

A team at AWS AI Labs and the University of Maryland now shows that this convention, while harmless for immediate benchmark performance, creates a worse starting point for reinforcement learning. Their method, ActObs, simply unmasks the observation tokens already present in each trajectory and applies the language-modeling loss to both actions and observations. The data, model, parameters, and training procedure remain identical. Only the loss mask changes.

After supervised fine-tuning, ActObs and standard action-only SFT perform almost identically. After the same GRPO reinforcement learning procedure, they diverge sharply. On Qwen3-4B, GRPO from ActObs achieves higher pass@k at every evaluated sampling budget on Terminal-Bench 2.0. On Qwen3-8B, it trades some pass@1 reliability for higher pass@k and solves more distinct tasks. The advantage even transfers to cross-domain code editing on aider-polyglot, despite those tasks being unseen during both SFT and RL.

What ActObs Actually Changes

The formal setup is clean. A trajectory is a sequence of task prompts, assistant actions, and environment observations: (x, a1, o1, a2, o2, ..., aT, oT). Standard agent SFT computes loss only on the action-token indices. ActObs adds the observation-token indices to the loss, weighted by a parameter lambda. At lambda equals zero, it recovers action-only SFT. At the default lambda equals one, actions and observations receive equal weight. Prompts remain masked in both objectives.

The effect is that each trajectory becomes both an imitation example and a transition example. Predicting what observation follows an action requires the model to represent what that action does to the environment. This is a form of consequence modeling, and it happens without adding any new data, parameters, sequence tokens, forward passes, or changes to the RL algorithm. It is purely a label-mask change.

There is also a timing control called Obs then Act, which applies one epoch of observation-only SFT followed by one epoch of action-only SFT. This receives the same total observation supervision as ActObs over two epochs but in a sequential schedule. It tests whether the benefit comes from observation exposure at all, or from joint action-observation learning.

How They Set Up the Experiments

All methods share one corpus: 50k multi-turn terminal trajectories totaling 0.71B tokens, generated by DeepSeek-V3.2 using the Nemotron-Terminal-Corpus. Observation tokens make up about 45% of the corpus. The team fine-tunes Qwen3-4B and Qwen3-8B for one epoch of 781 steps with batch size 64, cosine schedule, and peak learning rate of 10 to the negative fifth.

They then apply GRPO for 135 steps on 2,392 containerized tasks from Endless Terminals, which are disjoint from the SFT corpus. Binary verifiers score success, with 16 rollouts per task, batch size 32, and learning rate of 10 to the negative sixth. The observation supervision is used only during SFT. Standard GRPO contains no observation-prediction loss.

Evaluation uses Terminal-Bench 2.0 (89 tasks, 16 attempts each) and aider-polyglot (225 multilingual code-editing tasks, 4 attempts each). All evaluation tasks are disjoint from both the SFT corpus and the RL set. The pass@k metric uses the unbiased estimator from Chen et al. 2021, with bootstrap standard error from 20,000 resamples.

The Numbers After SFT and After RL

After SFT, the three initializations (ActionSFT, ActObs, Obs then Act) remain closely matched. On Terminal-Bench 2.0 at 4B, ActionSFT scores 4.5 pass@1 and 16.9 pass@16. ActObs scores 4.4 and 18.0. On aider-polyglot at 4B, ActionSFT reaches 1.4 pass@1 and 5.8 pass@4, while ActObs reaches 1.0 and 3.6. At 8B after SFT, ActionSFT actually leads ActObs by 1.3 points at pass@1 and 2.2 points at pass@16.

After GRPO, the picture changes. At 4B, ActObs produces the strongest post-RL policy across the entire sampling curve: 7.2 pass@1, 12.5 pass@4, 15.5 pass@8, 19.1 pass@16, versus ActionSFT's 5.6, 11.1, 14.0, 18.0. That is a 29% relative advantage at pass@1 and 6% at pass@16.

At 8B, ActionSFT still leads at pass@1 (12.3 versus 11.0), but ActObs overtakes it at pass@4 and its advantage grows to 3.4 points at pass@16, a 14% relative margin. ActObs solves 24 of 89 tasks, three more than ActionSFT's 21. Critically, three of those tasks are ones that no SFT policy or ActionSFT-then-GRPO can solve, showing that RL from the ActObs initialization reaches beyond the SFT-solvable set.

The sequential Obs then Act control does not achieve this benefit despite receiving the same observation supervision. At pass@16, ActObs leads it by 2.2 points at 4B and 3.4 points at 8B. Joint action-observation learning, not observation exposure alone, creates the stronger RL initialization.

Cross-Domain Transfer to Code Editing

Aider-polyglot tests whether the benefit extends beyond terminal tasks. None of its 225 code-editing tasks, spanning six programming languages, appears in either the terminal SFT corpus or the Endless Terminals RL set. At 4B, ActObs-then-GRPO outperforms ActionSFT-then-GRPO by 4.2 points at pass@1 and 4.9 points at pass@4, relative advantages of 43% and 24%.

This advantage cannot be attributed to a stronger code-editing policy before RL. The 4B ActObs SFT checkpoint actually trails both alternatives on aider-polyglot before RL. GRPO raises ActObs by 12.9 points at pass@1 and 21.7 points at pass@4, far more than it raises ActionSFT. At 8B, ActObs leads ActionSFT-then-GRPO by 1.5 points at pass@1, a 12% relative advantage. The SFT objective shapes downstream learning even when the tasks change completely.

Entropy Retention and Policy Movement

The performance difference traces to how ActObs changes the RL dynamics. During GRPO at 4B, both methods begin with similar reward and training entropy, but their trajectories separate. ActObs reward rises more smoothly and finishes higher, while its training entropy continues to rise late in RL as ActionSFT entropy falls and only partially recovers. ActObs rollouts also become progressively shorter, so the higher late-stage entropy does not come from generating longer trajectories.

At the final checkpoints, ActObs retains more self-entropy (0.779 versus 0.716 for ActionSFT) while requiring less policy movement from its SFT initialization (smaller endpoint KL divergence). This combination is key. The ActObs endpoint stays closer to its own initialization, so GRPO reallocates less probability mass and leaves more of the starting distribution accessible to repeated sampling.

Raising the sampling temperature of ActionSFT-then-GRPO until its self-entropy matches ActObs changes pass@1 through pass@16 by at most 0.7 points and leaves the pass@16 gap intact. The high-k advantage comes from the distribution learned by the policy, not from injecting more randomness at inference time. Similarly, ECHO produces the highest endpoint entropy but does not match ActObs-then-GRPO at pass@16, confirming that entropy alone is insufficient.

The Gradient Orthogonality Finding

The deepest insight comes from analyzing how the SFT objectives shape the initial policies. At each saved SFT checkpoint, the team separately computes action-token and observation-token gradients on the same 256 held-out trajectories. At the pretrained checkpoint, the two token streams initially favor a largely shared update, with cosine similarity of 0.83. Within 10 to 20 SFT steps, this similarity falls to the noise floor and stays there.

After the shared direction disappears, action-only training cannot supply the orthogonal observation gradient component. Under ActionSFT, the norm ratio of observation gradient to action gradient grows to about 41 because fitting actions makes the action gradient small while the unfitted observation gradient remains large. Under ActObs, both signals are optimized and the ratio stays near 0.5 through the final checkpoint.

The consequence is measurable. ActionSFT degrades observation prediction below the base model's ability. ActObs preserves or improves it. When you train a model to predict only what the agent does, it forgets how to predict what the environment does next. This forgotten capacity is exactly what RL needs for effective exploration, because predicting action consequences is the foundation for reasoning about which actions to try.

Where Observation Supervision Helps and Where It Does Not

The token-level analysis reveals that ActObs retains more entropy not at the beginning of commands, but later in the command where tokens encode arguments, flags, and paths. After GRPO, ActObs and ActionSFT are similarly sharp at the first few command tokens, but ActObs preserves more variation in how a selected command is instantiated. A representative success uses this freedom to sweep QEMU launch parameters one at a time, systematically varying memory, CPU, and other flags until finding a working configuration.

The lambda sweep shows a clean tradeoff. As observation supervision weight increases from 0 to 1, pass@8 after GRPO increases monotonically from 21.0 to 23.6, while pass@1 decreases from 12.3 to 11.0. Greater observation supervision progressively trades single-attempt reliability for success under repeated sampling. At lambda equals 0.25, the pass@1 drop is minimal (12.0 versus 12.3) while pass@8 gains 1.5 points.

Ablations confirm the design choices matter. Observation-only SFT (no action loss) scores zero at all budgets. Shuffled observations, where the pairing between action and observation is scrambled, drop pass@1 from 7.9 to 3.7 at 8B. Wild logs (format-matched but unrelated observations) retain 7.9 pass@1, better than shuffled but worse than correctly paired observations. Salient-filtered observations, keeping only important lines, score 7.2 pass@1 and 13.5 pass@4, below the full-observation variant's 7.9 and 16.1.

The Limitations the Authors Acknowledge

The method is confined to terminal agents. The aider-polyglot evaluation adds cross-domain code editing, but the results do not establish benefit for web navigation, GUI interaction, or embodied robotic environments. All experiments use Qwen3 at 4B and 8B with a single teacher (DeepSeek-V3.2) and a single data pipeline. Whether the findings generalize to other model families, larger scales, or different teacher models remains open.

The SFT, RL, and evaluation pipelines differ in observation formatting, context construction, and shell environment. Within each stage, all methods use the same scaffold so comparisons are controlled, but absolute scores may be affected by these infrastructure differences. The method also requires that observations be available as tokens in the trajectory, which holds for terminal and tool-use agents but may not apply to all agent architectures.

What This Means for Agent Training Pipelines

For practitioners, the takeaway is immediate. If your agent training pipeline uses SFT followed by RL, and your trajectories contain environment observations, unmasking those observations in the SFT loss is a zero-cost change that can meaningfully improve downstream RL. The modification requires no new data collection, no architectural changes, no additional parameters, and no changes to your RL algorithm. You change one line in your loss mask and retrain.

The deeper lesson is about what SFT is supposed to accomplish. The conventional view treats SFT as pure behavior cloning: learn to reproduce the teacher's actions. ActObs reveals that SFT also shapes the representational substrate on which RL operates. When action-only SFT erodes the model's ability to predict action consequences, it degrades the foundation that exploration requires. Joint supervision preserves that foundation, letting RL do its job more effectively.

The pass@1-to-pass@k tradeoff controlled by lambda adds a practical tuning knob. If your deployment relies on a single attempt, lower observation weight preserves single-shot reliability. If you can afford repeated sampling, higher weight improves the probability that at least one attempt succeeds. The choice depends on your inference budget and failure tolerance.

Whether this finding transfers to larger models, different architectures, and non-terminal agent domains remains the most important open question. The gradient orthogonality analysis suggests the mechanism is general: any SFT objective that fits one subset of tokens while ignoring another will leave unfitted gradients in orthogonal directions. If those gradients matter for downstream RL, the same one-sided specialization problem will appear. But the magnitude of the effect and whether it matters at 70B or 405B scale is unknown.