AutoAgent0: An Agentic Runtime for Safe Closed-Loop Driving

The closed-loop problem in autonomous driving has grown alongside improvements in end-to-end driving policies. These policies map raw sensor inputs to steering and control commands, and their performance on individual frames has improved considerably. However, a single frame decision does not guarantee safe progress when the decision is executed in sequence with prior and subsequent actions. Errors compound across dozens of steps, and in unseen scenarios the policy can drive the vehicle into states from which the same policy struggles to recover. The system does not fail at one moment; it fails across a trajectory. This makes closed-loop safety a problem of execution reliability rather than per-frame accuracy. AutoAgent0 addresses this by shifting the problem from improved planning to reliable test-time execution. The runtime sits between the driving expert and the vehicle, adding complementary assessment, verification, and recovery without changing the core policy. This division of labor lets the expert focus on motion primitives while the runtime handles the closed-loop guarantees.

The Detection Module and Complementary Perception

To address inaccurate scene perception, an independent detection module constructs scene context using complementary perception skills. The driving expert, whether an end-to-end policy or a modular planning system, relies on its own front-end perception stack. That stack is optimized for the task at hand but may miss structures or dynamics that are rare or operate at a different scale. The detection module in AutoAgent0 operates alongside the expert, using different sensor interpretations, network architectures, or reasoning patterns. It does not attempt to replace the expert's perception; instead, it materializes a secondary view of the drivable world, agent positions, and surrounding dynamics. This secondary view becomes the context against which proposed trajectories are verified. Because the module employs complementary skills—such as different feature hierarchies or alternative temporal modeling—it can surface events or objects that the primary stack attenuates. The combined context, originating from two independent assessments, forms the basis for the rule-based verifier that follows.

Trajectory Verification against Constructed Context

A rule-based verifier checks proposed trajectories against the constructed scene context before execution. The verifier does not execute the trajectory; it examines whether the planned sequence of states is feasible given the world as described by the detection module. Rules encode constraints such as "no forward movement into a detected obstacle," "maintain a safe distance from tracked agents," or "execute a lane change only when lane markings are visible in the context." When a proposed trajectory violates a rule, the verifier flags it as unreliable and prevents downstream execution. This check happens at test time without any gradient updates or policy modifications. The verifier's output is a binary verdict: acceptable or unacceptable. If unacceptable, the runtime does not simply discard the plan; it forwards the plan to the orchestrator, which activates Recovery Planning. The verification step thus serves as a gate, ensuring that only trajectories consistent with the observed scene proceed toward execution or recovery.

Recovery Planning as an Agentic Loop

When the nominal trajectories are unreliable, the orchestrator of the runtime activates Recovery Planning, where an agentic loop constructs alternative action proposals from action primitives and iteratively refines them using verification feedback. The loop does not rely on a single monolithic planner; instead, it decomposes the recovery problem into primitives such as "reduce speed," "steer left," "change lane," or "come to a complete stop." Each primitive is evaluated against the constructed context, and a selection is made that is likely to steer the vehicle toward a safer state. After a primitive is applied, the detection module updates the scene context, and the verifier reassesses the new trajectory. If the revised trajectory is still flagged as unacceptable, another primitive is selected and applied. This cycle repeats until either a trajectory passes verification or a termination condition is met, such as a maximum number of iterations or the vehicle reaching a stop state. The agentic nature of the loop means it can explore multiple recovery paths within a single timestep, something a static fallback policy cannot do. The loop also respects a time budget; if too many iterations elapse without success, the orchestrator may alert the higher-level planning stack or initiate an emergency stop. This design addresses the limited recovery capabilities of individual experts by providing a dynamic, context-aware mechanism that adapts as the scene evolves.

Evaluation on Fail2Drive and NavSafe-∞

Evaluations on Fail2Drive and NavSafe-∞ demonstrate improved closed-loop robustness over single-expert execution in long-tail and safety-critical scenarios. Fail2Drive is a benchmark that exposes driving policies to failure-inducing situations, such as sudden cut-ins, pedestrian appearances, and traffic signal changes. NavSafe-∞ extends the evaluation to an infinite horizon of interleaved maneuvers, testing the policy's ability to maintain safety over dozens of consecutive steps. In both datasets, AutoAgent0 outperforms the baseline of nominal expert execution without runtime supervision. The improvement is especially pronounced in long-tail scenarios, where the specific combination of events that trigger failure is rare but collectively significant. Safety-critical scenarios, which involve imminent collisions or near-miss states, also show reduced incident rates when the runtime is active. The results indicate that the combination of independent scene assessment, rule-based verification, and agentic recovery produces measurable robustness gains without requiring retraining of the underlying driving policy. Across multiple driving experts—including both end-to-end policies and modular planning approaches—the runtime consistently raises the threshold of scenarios that can be handled safely.

Constraints and Trade-offs Acknowledged by the Authors

The authors acknowledge several constraints. First, the quality of the detection module directly influences the verifier's effectiveness; if the module fails to surface a relevant dynamic, the verifier cannot enforce a rule against it, and the recovery loop may not select the appropriate primitive. The paper reports ablations showing that degraded detection quality reduces the runtime's benefits, though it still offers some improvement over no runtime at all. Second, the recovery loop introduces a small computational overhead per timestep; the agentic loop and repeated verification checks add latency compared to immediate execution of a nominal plan. This overhead is amortized across the reduction in safety incidents, but in real-time systems with strict per-timestep budgets, the trade-off must be weighed carefully. Third, the rule set is not exhaustive; new types of obstacles or interaction patterns that are not captured by the existing rules will not be caught by the verifier. The authors suggest that expanding the rule set requires domain expertise and systematic evaluation of failure modes, and they leave fully automatic rule generation as future work. Fourth, the runtime's benefits are most evident in scenarios where the expert's policy has already failed or is failing; in nominal, easy driving conditions the runtime may not add meaningful value, and the overhead may not be justified. These acknowledged limits shape the paper's conclusion that runtime supervision is a practical supplement rather than a complete solution to closed-loop safety.

Practical Takeaways for Developers Integrating Runtime Supervision

Developers who wish to integrate runtime supervision into a driving system can follow a modular path. Begin by inserting an independent detection module that uses complementary perception skills. The module does not need to be state-of-the-art; even a lightweight network with different feature hierarchies can provide sufficient context for verification. Next, encode a set of rules that check proposed trajectories against the constructed context. Start with a minimal set of high-impact constraints—such as collision avoidance with dynamically tracked agents—and expand based on observed failure modes. Then, implement an orchestrator that activates a recovery loop when the verifier flags a trajectory as unacceptable. The recovery loop should operate on action primitives and iterate with verification feedback, as described in the paper. The loop can be given a time budget and a maximum iteration count to prevent excessive latency. Finally, evaluate the integrated system on relevant benchmarks, observing both the reduction in safety incidents and the added per-timestep latency. Developers can expect the most benefit in long-tail and safety-critical scenarios, with modest improvements in nominal conditions. The key practice is to treat the runtime as a supplement to the existing expert, not a replacement; the expert continues to generate motion primitives, and the runtime adds the closed-loop guarantees that make those primitives reliable in unseen situations. The AutoAgent0 release provides reference code for the detection module, verifier, and recovery loop, serving as a starting point for developers who wish to experiment with runtime supervision on their own driving stacks.