Post-Training Driving Policies Without Rendering Every Step

End-to-end driving policies are pre-trained with behavior cloning on human demonstrations, mapping raw sensor data to actions or waypoint trajectories. This works in open loop, where the policy is supervised only on states the human demonstrator visited. But during closed-loop deployment, the policy's own actions determine future states. Small errors compound, the policy drifts outside the training distribution, and the result is collisions, off-road excursions, and safety-critical failures that the training data never covered.

Reinforcement learning can fix this by directly optimizing for safety and progress in closed loop. Vectorized RL policies trained on billions of simulated interactions have achieved infraction rates below those of human drivers. But every RL exploration step for a camera-based policy requires 3D rendering of the scene, which is computationally expensive. Running RL at the scale needed for superhuman driving on sensor inputs has not been demonstrated. The two costs of closed-loop post-training, sample-inefficient exploration and sample-efficient supervision, are currently coupled: both happen in the same expensive simulator.

Damiano Da Col, Maximilian Igl, Peter Karkus, and colleagues at KE:SAI, NVIDIA Research, ETH Zürich, Stanford, and the ELLIS Institute Tübingen propose OPTED (On-Policy fine-Tuning for End-to-end Driving), which decouples these two costs by splitting them across different simulators.

Two Simulators, One Teacher, One Student

The core insight is that RL exploration does not require sensor input. An RL policy trained on vectorized inputs, HD-map polylines and actor bounding boxes, can serve as the expert for on-policy supervision, provided the two simulators are synchronized to the same state. The expensive rendered interaction is limited to the end-to-end model, while RL exploration, which is computationally infeasible with rendering, is confined to a render-free simulator.

The pipeline has three components. First, a privileged teacher is trained with PPO in PufferDrive, a vectorized render-free simulator. The teacher observes ego kinematics, surrounding agents' poses and velocities, and the local road graph, and emits continuous control actions. It is first pre-trained on a large corpus of vectorized driving scenes, then adapted to the target scene distribution. After this adaptation, the teacher is frozen.

Second, a pre-trained end-to-end student, which processes RGB images, is rolled out in closed loop in a sensor-based simulator. In AlpaSim, this means rendering 3D Gaussian splatting reconstructions of real driving logs. At each visited state, the scene is synchronized into the teacher's vectorized simulator. The teacher is stepped K times from that state to produce a trajectory in the student's planning frame. This trajectory becomes the distillation target.

Third, the student is updated by minimizing an imitation loss against the teacher's target. For waypoint-predicting students, this is a weighted L1 regression. For action-predicting students, it is a forward KL divergence. Only the planning decoder is trained; the backbone is frozen to avoid overfitting to rendering artifacts and to allow caching backbone latents in a FIFO buffer of recent rollouts.

Bridging the Output Representation Gap

A practical challenge is that state-of-the-art RL policies output actions, while end-to-end policies predominantly predict waypoint trajectories. OPTED bridges this mismatch by rolling out the teacher from the student's state and converting the resulting motion into a target trajectory in the student's frame. No constraint is placed on the student's architecture or output representation. The distillation loss is adapted to whatever head the student uses.

This is important because it means OPTED can be applied to any pre-trained driving model without architectural modification. The student retains its human-data prior and its original output format. The teacher's role is purely to provide supervision at states the student actually reaches, not to impose a different output structure.

The teacher's reward is designed for smooth, human-like driving rather than the fast, goal-reaching behavior that the default PufferDrive reward encourages. A dense route-progress reward gated on lane alignment is added to the sparse goal bonus and safety penalties. The progress term is spatial rather than temporal, rewarding advancement along the route independently of speed. Safety penalties do not terminate the episode, so the teacher can accumulate penalties while driving in infraction states and learns to recover from them.

Results on AlpaSim

OPTED is applied to two architecturally diverse public models: TransFuser (LTFv6) and VaVAM. Both are fine-tuned for two epochs on the PAI-AV NuRec dataset in AlpaSim, which renders 3D Gaussian splatting reconstructions of real-world driving scenes including highway, urban, and rural settings.

TransFuser's scene score increases from 26.5% to 41.8%, a 1.6× improvement. VaVAM's scene score jumps from 3.9% to 37.1%, a 9.5× improvement. The km/infr. metric (distance driven per infraction) improves alongside. These gains come after only two epochs of fine-tuning the planning decoder, with the backbone frozen.

The comparison with baselines isolates the sources of improvement. Open-loop BC fine-tuning on the same rendered scenes (but at logged states only) gives modest gains, highlighting domain adaptation. Closed-loop SFT with log-anchored targets (RoaD-recovery) improves further but is biased toward logged trajectories and constructs targets with manual heuristics that often exhibit sub-optimal behavior. OPTED outperforms both because its rollouts are unconstrained by the log and its targets come from an RL-trained teacher rather than heuristic recovery.

As a reference, OPTED post-trained policies outperform Qwen-Drive 1.0 (a 5.1B parameter VLA) on scene score, though they still lag behind Alpamayo 1.5 (a 10B parameter VLA). Both VLAs are orders of magnitude larger and have inference times of 940-1181 ms, making them unsuitable for real-time execution. OPTED students run at 23 ms (TransFuser) or 112 ms (VaVAM) per planning step.

Three Orders of Magnitude Fewer Simulator Interactions

The most striking result comes from a controlled study on vectorized policies in PufferDrive on the Waymo Open Motion Dataset. Here both teacher and student receive vectorized inputs, so the rendering cost is eliminated and RL fine-tuning can be run to convergence.

OPTED matches the closed-loop performance of direct RL fine-tuning of the same student with approximately three orders of magnitude fewer closed-loop simulator interactions: 6,800 episodes for OPTED versus 6.2 million for RL fine-tuning to reach 95% of the teacher's scene score. The teacher explores cheaply in the render-free simulator, and the student inherits the result through distillation at the states it actually visits.

OPTED also stays closer to the human prior than RL fine-tuning. RL pushes the policy further from the demonstrated distribution in pursuit of reward, while OPTED's distillation anchors the student to the teacher's behavior at visited states, which is itself close to human-like because the teacher was trained with a human-route progress reward.

Open-Loop Agreement and the Controller Problem

A nuanced finding is that open-loop and closed-loop performance are decoupled. BC fine-tuning minimizes average displacement error (ADE) against logged trajectories and has the lowest scene score of the fine-tuning methods. Qwen-Drive has the lowest nuPlan ADE in the table yet a lower scene score than both OPTED students. Better agreement with human trajectories in open loop does not translate to better closed-loop driving.

OPTED does increase ADE compared to BC and RoaD-recovery, because the teacher's RL-optimized targets differ from human trajectories. But the drift is bounded. Fine-tuned students stay reasonably close to human driving while gaining closed-loop competence.

The controller analysis reveals an important practical detail. VaVAM does not observe ego speed and is pre-trained on slower scenes (fewer than 1% of nuScenes and nuPlan scenes exceed 50 km/h, versus 22% of NuRec scenes). At highway speed, it predicts plans slower than the current state. The default MPC controller, which bounds longitudinal but not lateral acceleration, converts the implied braking into lateral swerving and off-road events. Adding a friction circle constraint to the controller at evaluation time raises VaVAM's scene score from 37.1 to 50.6. The failure originates in the plan, not the controller, but the controller can mask or amplify it.

Computational Cost and Practical Deployment

On an 8-GPU NVIDIA H100 node, one OPTED iteration with VaVAM spends 524 seconds collecting 32 scenes of 20 seconds each in AlpaSim, 13.5 seconds on teacher labeling, and 6 seconds on gradient updates over 512 buffered scenes. Rendering and student inference dominate runtime. Two epochs of fine-tuning take 17 hours. The teacher is trained once on a single NVIDIA RTX 4090 in about 24 hours, plus 6 hours of adaptation to the target dataset, and is reused for every student.

This is substantially cheaper than direct RL post-training on sensor inputs, which requires rendering at every exploration step. The teacher's training is a one-time cost amortized across all students. The student's fine-tuning cost is dominated by its own closed-loop rollouts, which are necessary regardless of the supervision method.

What This Means for Autonomous Driving Post-Training

OPTED addresses a practical bottleneck in the autonomous driving pipeline. Pre-training on human data gets you a policy that works in the states humans visit. Post-training with RL gets you a policy that works in states the policy itself visits. But RL on sensor inputs is too expensive to run at scale. OPTED makes post-training practical by moving the expensive exploration to a cheap simulator and transferring the result through distillation.

The method is general. It works with any pre-trained driving model regardless of architecture or output representation. It works with any vectorized RL teacher. The teacher and student can be pre-trained on independent, unpaired datasets. The only requirement is that the two simulators can be synchronized to the same state, which the authors achieve by projecting the student's state into the vectorized representation.

The project page is at https://01dami23.github.io/opted/. The results suggest that the future of driving post-training is not bigger models or more rendering, but smarter decoupling of exploration and supervision across different levels of simulation fidelity.