Large language model agents and multi-agent systems increasingly tackle complex tasks through long-horizon tool use and inter-agent collaboration. As execution unfolds, these systems are prone to deviating from the overall goal, a failure termed goal drift. The drift wastes execution resources and leads to undesired outcomes. A single agent may persist along an unproductive chain of actions. An agent in a multi-agent system may prioritize its local objective over the collective goal. This paper proposes a steering agent that supervises execution on the fly and steers the system back on track when drift occurs.
Goal drift in single- and multi-agent execution
An LLM-based agent executes a task with an overall goal by interacting with an environment. At each step the agent generates an action conditioned on the goal and the interaction history, then receives environment feedback. A multi-agent system distributes this process across specialized agents. Each agent handles a temporary subgoal derived from the overall goal, such as retrieving evidence or verifying an answer, and hands its output to subsequent agents. In both settings execution unfolds as a sequence of temporary subgoals in service of the overall goal.
Goal drift appears when a temporary subgoal dominates the overall task goal, leading to unproductive actions that might be locally reasonable. The drift reduces execution efficiency by consuming resources on actions that contribute little to task completion. If the deviation remains uncorrected, the agent system may never finish the task or finish it with an undesired outcome.
In multi-hop search tasks the agent needs to retrieve several distinct pieces of evidence and combine them to answer a question. A common failure mode is that the agent keeps searching over the same topic when key information has already been obtained. The first search already identifies the actress required to answer the question, yet the agent spends three more searches on the same film instead of moving on to the next step. This pattern of redundant and unproductive search wastes execution budget.
In a multi-agent system goal drift arises when an agent prioritizes its role objective over the overall task goal. A commonly observed failure mode is over-rejection, where a verifier falsely rejects results that already satisfy the task goal. The verifier repeatedly rejects the answers the searchers have delivered, leading execution to repeated revision rounds that can turn a successful execution into a failure. This reflects the verifier overweighting its own goal of admitting only answers beyond doubt.
How earlier systems attempt on-the-fly supervision
To keep agents on track during long-horizon execution, prevalent agent systems employ planning and verification. Planning decomposes a goal into manageable subgoals that guide subsequent actions. Verification checks whether goals have been met before an agent declares completion, prompting further execution or revision when needed. However, a plan does not ensure reliable execution, while verification may uncover deviations after substantial effort has already been spent.
These limitations motivate a natural on-the-fly supervision design, where an external supervisor monitors intermediate execution and intervenes when goal drift occurs. Recent work explores this idea with a primary focus on detection, identifying when and where an agent goes wrong. SupervisorAgent takes a step toward correction by prompting an external LLM, but it targets easily fixable issues such as errors and repetitive actions. Goal drift is more challenging to detect and correct in real time, as it emerges gradually from locally plausible actions. This paper addresses the challenge with a steering agent that jointly learns drift detection and correction through reinforcement learning.
The steering agent's observation space and action space
The steering agent observes the execution context through a defined observation space. A function exposes the goal and available execution context, producing an observation at each decision point. The action space contains two alternatives: pass, which lets execution continue, and interventions supported by the system. At each steering decision the agent reads the observation and returns a steering action drawn from a policy realized by an LLM with parameters theta.
The steering process is formalized as a partially observable Markov decision process. The agent system being supervised acts as the environment. At decision step t the steering agent observes the observation rather than the full execution state, takes an action, and the agent system transitions to the next state until the next observation update. The task reward is received when execution terminates and measures goal completion. This steering interface achieves different steering granularities depending on the observation and action spaces. Exposing every execution step of a single agent enables fine-grained supervision, while exposing only inter-agent handoffs enables coarse-grained supervision over a multi-agent system.
The fast path: single-token steering priority judgment
To process each observation update at low cost the fast path judges the need for intervention within a single LLM forward pass. The fast path estimates the steering priority of the current state, which measures how much a steering action at this state can change the final outcome. This allows the steering agent to achieve a larger impact with a fixed intervention budget.
The fast path measures steering priority by a yes/no judgment. The probability of two tokens is read from the shared LLM backbone. The log-odds between these tokens produces a raw score, which the logistic function normalizes to steering priority in the range (0,1). A soft threshold on the priority controls activation of the slow path. The trigger probability exceeds one half once the priority passes the set value and approaches one as the gap grows. This soft threshold is a smoothed version of a hard rule that triggers when priority exceeds the threshold. The fast path thus provides a low-cost judgment about whether intervention is warranted.
The slow path: chain-of-thought reasoning and action generation
Once triggered the slow path conducts chain-of-thought reasoning and generates a steering action. The LLM first generates reasoning and then commits to the action conditioned on that reasoning. The generated sequence contains both the reasoning and the resulting action. Through the transition in the steering interface, the steering action changes the execution state and thereby the actions the executor takes next, steering the system back on track. Execution of the agent system being supervised then resumes until the next observation update.
The slow path can be represented by an auto-regressive generation process. Each token is generated auto-regressively conditioned on the preceding tokens. The slow path thus produces deliberative interventions when the fast path indicates they are needed. Both paths share the same LLM backbone parameterized by theta, enabling joint training.
Pairwise ranking trains the fast path to rank states by impact
To train the fast path the paper uses a pairwise ranking objective. The training target measures the priority of a state by the mean absolute outcome change that steering causes relative to the no-intervention baseline. For states from the same task the predicted steering priority scores are compared against the measured priorities. The loss pushes the predicted scores above the others whenever a state shows higher measured priority. Tied pairs contribute no signal. This trains the fast path to rank states by their potential impact, enabling it to identify the most important states for intervention.
The pairwise ranking loss operates on state pairs with different measured priorities. For each pair the loss term encourages the predicted priority of the higher-priority state to exceed the other, aligning the predicted ordering with the measured one. This objective teaches the fast path which states carry high steering priority through comparison of final outcomes.
GRPO trains the slow path to produce rewarding actions
The slow path is trained with group-relative policy optimization. For each state multiple steering attempts are sampled. The advantage of each action is calculated by subtracting the mean of the pooled group (which includes steered outcomes and the unsteered baseline) and dividing by the group standard deviation. An action receives a positive advantage only when its mean reward exceeds the pooled average. The advantage is assigned to every generated reasoning and action token, and the standard PPO loss is applied with a clipping range on the token probability ratio.
Combining the objectives of the fast and slow paths yields a joint training loss that updates the shared parameters of the steering agent. The executor remains frozen during training, and only the steering agent parameters are updated. The training pipeline samples intermediate states that previous rollouts have reached, lets the steering agent attempt interventions there, generates several steering actions at the sampled states, and compares their final rewards together with those of the unsteered continuations.
In-domain search-task results: single-agent gains
Experiments are conducted on information-seeking search tasks following the experimental setting of ASearcher. The seven benchmarks cover multi-hop reasoning and single-hop retrieval with a local Wikipedia retrieval server. The executor is a Qwen2.5-7B-Instruct search agent, which shows basic search ability but is prone to goal drift. The steering agent backbone is a stronger Qwen3-8B model. Results report F1 following the same protocol, with average denoting the mean over the seven benchmarks.
Results on in-domain benchmarks show that zero-shot steering yields a performance gain of 1.84 points over no intervention, with 1.11 steering actions and 6.5 executor turns per task. Among three supervision forms (planning, verification, and steering), zero-shot steering brings the largest gain and leaves executor turns nearly unchanged. Planning and verification help less or even degrade performance and lengthen execution.
Reinforcement learning substantially improves steering intelligence. Applying the same learning algorithm to the steering agent raises the gain from 1.84 to 6.66 points and reaches the strongest results consistently on all seven benchmarks, with the fewest steering actions (0.93 per task) and executor turns. The RL-optimized steering agent also outperforms filtered behavior cloning, further validating the necessity of online reinforcement learning.
Filtered behavior cloning imitates the sampled steering actions that improve the task outcome. It falls behind the online algorithm, showing that offline imitation of successful interventions is insufficient. The online algorithm learns both which states merit intervention and which actions improve outcomes through controlled comparisons among steering actions and the unsteered baseline.
Search-task results for multi-agent systems
In the multi-agent setting the steering agent is introduced into a five-role pipeline for search tasks following TacoMAS, where agents with specialized roles collaborate in a fixed order. The observation space of the steering agent is the task goal and inter-agent handoffs. The action space includes appending a guidance note or rejecting the message for retry. The steering agent, the training procedure, and the benchmarks stay the same as in the single-agent experiments.
With the threshold tuned so that steering actions cover 25% of agent turns, zero-shot steering lowers the average F1 by 1.04 points. Forcing steering at every handoff increases the performance drop to 7.14 points, with agent turns inflated from 9.2 to 24.1 per task. Supervising a multi-agent system with lower granularity is more challenging for the steering agent. In comparison, the RL algorithm effectively improves the steering intelligence, achieving gains of 3.90 and 5.34 points under the two frequency settings respectively. These results show that the same steering interface and learning algorithm apply to both fine-grained supervision of a single agent and coarse-grained supervision of a multi-agent system.
Generalization to out-of-domain tasks
To study whether the learned steering intelligence generalizes to out-of-domain tasks the paper evaluates on tasks that require an online search API, exposing more complex real-world information. The same steering model is tested without additional training on three more challenging benchmarks requiring deeper multi-hop search: Frames, GAIA, and xBench-DeepSearch. Steering improves both average and best-of-n accuracy on Frames and GAIA, and raises the best-of-n accuracy on xBench-DeepSearch. These results show that the learned steering capability generalizes beyond the training distribution.
The steering agent mainly learns to recognize and correct goal drift, while task-specific ability stays within the frozen executor. This separation mitigates overfitting to the training tasks. The out-of-domain tasks require deeper multi-hop search and expose more complex real-world information, yet the steering agent trained on local Wikipedia retrieval transfers effectively to these online search environments.
Transfer to unseen executors without retraining
Transferability of the learned steering intelligence to different model families is tested by using the same steering model to supervise three unseen executors from different model families: Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, and Phi-4. Zero-shot steering brings small gains consistently across the four model families. However, the steering agent trained with one executor shows strong transferability, improving the three unseen executors by 15.78, 12.01, and 5.92 points respectively while effectively reducing executor turns.
These results show that the learned steering intelligence transfers to executors unseen during training, allowing the executor to be freely switched without retraining. The steering agent learns general principles of goal drift detection and correction that apply across different model architectures and capabilities. This transferability is practical for developers who may want to upgrade or change their executor models without retraining the steering component.
Controllable intervention frequency at deployment without retraining
The paper examines whether intervention frequency can be controlled at deployment without retraining. By adjusting the fast path threshold at deployment the intervention frequency is varied. RL steering consistently improves task performance across all intervention budgets, surpassing zero-shot steering and other baselines. At the highest intervention frequency the total gain reaches 17.11 points. These results show that the fast path design turns intervention frequency into a deployment-time control, allowing the same model to trade steering cost for task performance.
At low intervention frequency the RL-optimized agent achieves its largest gain per intervention, while zero-shot steering and filtered BC remain low across frequencies, lacking the ability to judge where intervention matters most. This demonstrates that the fast path learns to prioritize steering at the states that influence the final outcome most, yielding the largest gain under a fixed intervention budget. The gain per intervention metric shows that RL steering extracts maximum value from each intervention, making it practical for resource-constrained deployments.
Practical implications for developers building agent systems
The steering agent provides an effective on-the-fly supervision tool for LLM-based agent systems. Its dual-path design separates low-cost judgment from deliberative action generation, balancing monitoring efficiency and steering effectiveness. The joint reinforcement learning with pairwise ranking and GRPO unlocks steering intelligence that improves both single-agent and multi-agent systems. The learned intelligence generalizes to out-of-domain tasks, transfers to unseen executors, and supports controllable intervention frequency without retraining.
For developers building long-horizon agent systems, the steering agent offers a practical method to reduce goal drift and improve task performance and efficiency. The steering agent can be deployed as a drop-in supervision layer without modifying the executor. The controllable intervention frequency allows developers to tune the trade-off between supervision overhead and task performance at deployment time. The transferability across executor models means the steering component remains valuable even as underlying models evolve.