MILER: Semantic Mid-Level Representation for Sim-to-Real Reinforcement Learning in Unstructured Autonomous Driving

Getting RL Policies Off the Simulator and Onto Real Dirt

Reinforcement learning promises superhuman autonomous driving, but nearly every RL policy that works in simulation collapses when transferred to a real vehicle. The gap is especially severe in unstructured environments like off-road terrain, where the visual and physical diversity far exceeds what structured urban roads demand. Thomas Steinecker, Denis Trescher, and colleagues at the University of the Bundeswehr Munich tackle this with MILER, a framework that trains policies in a deliberately abstract simulator and deploys them on real vehicles without any per-vehicle retraining. The key moves are a semantic mid-level representation that bridges perception between simulation and reality, and a trajectory-alignment trick that lets the real vehicle shadow a virtual one. They drove 17.3 kilometers across two different vehicles on a 3.0-kilometer test track at speeds up to 33.6 km/h without a single human intervention.

Why High-Fidelity Simulators Are the Wrong Answer

The conventional wisdom for sim-to-real transfer is to build a simulator as close to reality as possible: model the vehicle dynamics in detail, render realistic textures and lighting, simulate sensor physics including LiDAR ray casting and camera optics. This approach works for structured urban driving where the visual domain is relatively constrained. For unstructured environments, it breaks down. Off-road terrain varies wildly in appearance, texture, and physical properties. A simulator that faithfully models one grassy field does not generalize to another. The effort required to build and maintain such a simulator scales poorly, and the domain gap remains large.

MILER takes the opposite approach. Instead of making simulation match reality, it creates an abstract intermediate representation that both sides can agree on. The simulator does not render textures or simulate sensor physics at all. It works directly with semantic grid maps that classify terrain into a handful of categories. The real vehicle, meanwhile, processes raw camera and LiDAR data through BEVFusion to produce the same kind of semantic grid. If both sides produce the same representation, the domain gap shrinks to the difference in how that representation is generated, not the full visual and physical complexity of the real world.

The Mid-Level Representation Simulator

The MLR simulator defines four semantic classes: Drivable (paved roads and gravel), Semi-drivable (grass and tall grass), Non-drivable (obstacles and stationary vehicles), and None (unknown cells used during deployment). Map generation uses Perlin noise, the same technique Minecraft uses for terrain, overlaying multiple layers with different grid sizes and assigning value ranges to the semantic classes. The result is a diverse set of maps with different terrain distributions and obstacle patterns.

For training, the system generates 3,800 maps, each 500 meters by 500 meters at 0.4-meter resolution. Each map gets a connected graph of random nodes placed in drivable or semi-drivable areas, linked by straight connections through traversable space. This graph provides the target waypoints that the policy must follow. In each episode, a random map is selected, two random nodes are chosen and connected via breadth-first search, and the resulting path is smoothed using Curvature Corrected Moving Average to produce realistic-looking trajectories. The agent is spawned with Gaussian noise on position, heading, velocity, acceleration, steering angle, and steering rate, forcing it to recover from imperfect starts.

The BEV grid cells are 25 centimeters by 25 centimeters, covering a 60-meter-by-60-meter area centered on the vehicle. The target waypoints are rasterized into an additional channel, placing the route in the same spatial frame as the obstacles. A single BEV frame is used rather than stacking temporal frames, which avoids temporal inconsistencies that would complicate sim-to-real transfer.

Training: PPO with Discrete Actions and Undiscounted Rewards

The policy network processes the BEV through a CNN (five strided convolutions producing a 256-by-8-by-8 feature map, flattened and projected to 512 dimensions) and the remaining state components through an MLP. The fused 640-dimensional feature feeds separate actor and critic heads. The action space is factorized into two independent categorical distributions with five bins each, one for jerk and one for steering-rate change. Using higher-order derivatives (jerk and steering-rate change rather than acceleration and steering rate) increases the RL problem's complexity but bridges the sim-to-real gap more effectively, because inertia and delay effects in the real vehicle are easier to accommodate when the control commands are smoother.

The authors train with PPO using an undiscounted objective (gamma = 1.0). This is unusual and makes training harder, requiring longer bootstrap sequences, but it achieves global optimization and discourages hasty behavior. The agent has infinite time to accumulate rewards, so it learns to drive patiently rather than cutting corners or rushing. The reward is built around progress along the target waypoints, gated by lateral deviation. Additional reward terms peak at the target velocity and when the vehicle is aligned with the path. Penalties are applied for smoothness violations, unsafe velocities (which ramp up near obstacles and decrease on semi-drivable terrain), collisions, and going out of bounds.

Curriculum learning ramps up penalty coefficients over the first 2% of training, allowing the agent to explore freely before being constrained by safety requirements. Without this, the agent discovers that remaining stationary is the safest policy and never learns to drive. State normalization uses statistics estimated from 50,000 samples collected with a random policy, embedded in the network so training and deployment cannot drift apart.

The full training runs for 10^9 samples, more than 2 million episodes, taking about 68 hours on one NVIDIA A100.

Deployment: Trajectory Alignment with a Virtual Vehicle

Applying the policy's actions directly to the real vehicle would fail. The bicycle model used in simulation does not account for unmodeled time delays, friction variations, vehicle-model inaccuracies, and other real-world effects. At low speeds this might be tolerable, but at 30 km/h the discrepancy causes oscillations and unsafe behavior.

MILER's solution is trajectory alignment. During deployment, a virtual vehicle runs inside the MLR simulator in real time. The policy outputs actions to this virtual vehicle, which advances its state through the bicycle model. Meanwhile, the real vehicle's longitudinal and lateral controllers (including a Stanley controller) minimize the pose difference between the real and virtual vehicles. The real vehicle shadows the virtual one rather than executing the policy's commands directly.

The perception pipeline generates the BEV for the real vehicle using BEVFusion processing camera and LiDAR data. This BEV must be transformed into the virtual vehicle's coordinate frame. Direct transformation would create undefined cells near the BEV boundaries under rotational offsets, so the system applies a circular crop to eliminate rotational artifacts, then reduces the radius to account for translational offsets between the real and virtual vehicles. The same crop is applied in simulation during training to ensure identical input dimensions and structure.

The entire software stack runs on a Jetson AGX Orin, with the policy network exported to ONNX and compiled into a TensorRT engine achieving a mean inference time of 1.9 milliseconds.

Real-World Evaluation: 17.3 Kilometers Without Intervention

The test track is 3.0 kilometers long and includes straight roads for maximum speed, large obstacles (trees, containers), smaller obstacles (bushes, poles, construction barriers), hairpin curves, off-road sections with tall grass or gravel, and narrow corridors. Two different vehicles are used: MuCAR-4 (1.8 tons, 2.60-meter wheelbase) and MuCAR-3 (2.5 tons, 2.86-meter wheelbase). The only vehicle-specific parameter in training is the wheelbase, and the authors train using only MuCAR-4's wheelbase while still getting comparable results on MuCAR-3.

The vehicles differ in mass, steering limits, low-level control implementations, camera mounts, and calibration. Vehicle A drives in daylight and dry conditions; Vehicle B drives in evening rain. Both vehicles complete three laps of the track without human intervention. Vehicle B stops during the third lap due to overly conservative obstacle prediction caused by darker, rainy conditions, yielding a total of 17.3 kilometers.

Mean velocity is 4.36 m/s for Vehicle A and 4.19 m/s for Vehicle B. Maximum velocity reaches 9.33 m/s (33.6 km/h) for Vehicle A. Mean cross-track deviation is under 1.1 meters for both vehicles. Maximum centripetal acceleration stays at 2.19 m/s^2, below the hard limit of 3.0 m/s^2.

The driving behavior is not always human-like. The agent does not swing out before hairpin curves or dodge obstacles at reduced velocity the way a human driver would. It drives conservatively, staying close to the target waypoints rather than taking optimal racing lines. This reflects a tradeoff: the target waypoints encode prior knowledge of drivable areas, and deviating too far from them risks entering areas that appear drivable but are later reclassified as semi-drivable or non-drivable.

Ablations: What Matters and What Breaks

The ablation studies isolate the contributions of each component. When both vehicles receive identical BEVs from an HD map rather than their own sensor-derived BEVs, the mean velocity difference between them drops from 0.317 m/s to 0.150 m/s, and the mean trajectory distance drops from 0.124 meters to 0.078 meters. This confirms that BEV differences are a significant source of the residual gap between vehicles, and that the policy itself generalizes well when the perception input is consistent.

When the trajectory-alignment mechanism is removed and the policy's actions are applied directly to the real vehicle, the results are poor. The vehicle oscillates, makes delayed maneuvers, and fails to navigate curves. This confirms that the bicycle model discrepancy is too large to ignore at the speeds tested.

Limitations and the Speed Ceiling

The 60-meter BEV range imposes a speed ceiling. At 30 km/h, the minimum braking distance with a jerk of -2.0 m/s^3 and acceleration of -2.0 m/s^2 is 21.4 meters. At 35 km/h it is 28.4 meters. With only 30 meters of visibility on each side, the agent cannot safely drive faster than about 35 km/h because it cannot guarantee it can stop in time for an obstacle at the BEV boundary. Higher speeds require either a larger BEV or longer planning horizons.

The agent also fails to discover sophisticated maneuvers even after a billion training samples. No swing-out before hairpin curves, no anticipatory braking for sequential obstacles. The authors suggest that improved exploration techniques or incorporating recorded human-driving data might address this, and note that incorporating dynamic agents is future work.

What It Means in Practice

MILER demonstrates that you do not need a high-fidelity simulator to train driving policies that work on real vehicles. The semantic mid-level representation sidesteps the hardest part of sim-to-real transfer, the visual domain gap, by making both sides speak the same abstract language. The trajectory-alignment strategy sidesteps the dynamics gap by letting a virtual vehicle absorb the policy's commands while the real vehicle follows along.

For teams working on off-road or unstructured autonomous driving, this is a practical blueprint. The MLR simulator is cheap to build and fast to run. The BEVFusion adaptation produces compatible inputs from real sensors. The trajectory-alignment mechanism requires only a standard longitudinal and lateral controller on the real vehicle. And the whole stack fits on a Jetson AGX Orin, which is the hardware many field robots already carry.

The speed record of 33.6 km/h for an RL-based vehicle-sized platform in an unstructured environment is a meaningful milestone, but the real contribution is the architecture. By decoupling what the policy sees (a semantic grid) from what the vehicle does (follow a virtual trajectory), MILER makes zero-shot transfer feasible across vehicles with different mass, wheelbase, and control characteristics. That generalization, more than any single speed number, is what makes this work worth building on.

Read the paper on arXiv