Why Humanoid Robots Keep Falling (and a Learning-Based Fix)
A humanoid robot performing balance tasks under unexpected loads is a recipe for collapse. The robot's controller must simultaneously track a desired motion, maintain balance against disturbances, and avoid joint torques or postures that lead to falls. Reinforcement learning has made humanoid locomotion and motion tracking remarkably capable, but learned policies still produce unsafe actions that destabilize the robot, especially when conditions deviate from training. The core difficulty is that safety, task performance, and robustness pull in opposite directions within a single reward function, and gradient-based RL converges to locally optimal solutions on the Pareto frontier rather than finding a well-balanced operating point.
The paper from Qu, Zhang, Zhang, Wang, Sreenath, Tomlin, and Choi (UC Berkeley and UCLA) addresses this with a deceptively simple idea: stop asking one policy to do everything. Instead of training a single controller that must juggle motion tracking, safety, and robustness through carefully weighted reward terms, they split the problem in two. A nominal reference policy handles task performance. A separate residual policy learns safety corrections that modify the nominal action only when necessary. This decoupled architecture, called ResSafe, turns out to produce a meaningfully better performance-safety trade-off than single-policy training, both in simulation and on a physical Unitree G1 humanoid.
The Single-Policy Problem and the Safety Filter Perspective
Classical safe control theory defines safety as constraint satisfaction: keep the system state inside a safe set at all times. Hamilton-Jacobi reachability analysis computes a value function that characterizes the maximal control-invariant set within a constraint region, and a safety filter then projects any reference action onto the closest safe action. This min-norm correction modifies the reference input only as much as required to maintain safety.
The problem with applying this directly to humanoids is scalability. HJ reachability and control barrier functions (CBFs) both depend on explicit dynamics models and suffer from exponential complexity with state dimension. A Unitree G1 has 58 state dimensions and 29 control dimensions. Computing exact safety certificates for a system this size is intractable.
Constrained MDP formulations from the safe RL literature (like constrained policy optimization) treat safety as a cumulative cost to be kept below a threshold. This works for long-horizon cost accumulation but does not capture instantaneous failures like falls or collisions. And the Lagrangian formulation that enforces the constraint couples safety and performance into a single optimization, which is exactly the coupling the authors want to break.
Residual RL offers a path forward. Prior work has used residual policies to compensate for model errors in manipulation, quadcopter flight, and humanoid whole-body control. The key insight in ResSafe is that the residual correction term plays the same role as a safety filter correction. When the reference action is safe, the residual is small or zero. When it would violate safety, the residual modifies it. The correction is learned rather than computed from a model, which sidesteps the scalability problem.
How the Residual Safety Filter Works
The architecture is straightforward. Given a state s, the nominal reference policy produces an action uref. The residual policy takes the state and the reference action as input and outputs a correction Delta u. The final action is u = uref + Delta u.
The residual policy is trained with PPO to maximize a reward that combines two objectives. First, an alive bonus (the constant 1 at each timestep) encourages the robot to survive as long as possible before falling. This reformulates the safety problem as maximum survival time, which can be expressed as a sum-over-reward rather than the min-over-time objective required by standard HJ reachability. The sum-over-reward formulation is directly compatible with standard RL algorithms like PPO without requiring specialized Bellman equation modifications. Second, a min-norm penalty (lambda times the L2 norm of the correction squared) discourages the residual from making large modifications when they are not needed, mimicking the min-norm objective of classical safety filters.
The paper demonstrates this connection with a cart-pole running example. A reference policy learns to move the cart to a target position but violates a position constraint. The learned residual RL safety filter produces corrections that closely match the min-norm solution from a learned discriminating hyperplane safety filter, with slightly more conservatism. That extra conservatism, the authors argue, acts as a robustness margin that helps in practice.
Scaling to Humanoid Balance
The jump from cart-pole to a 58-dimensional humanoid requires careful engineering. The reference policy is trained on a dataset of 10,000 diverse balance motions generated programmatically, covering a wide range of humanoid configurations. Training on this diverse set rather than a single motion pushes the residual policy to learn general balance priors rather than overfitting to one trajectory.
The reward for the residual policy includes two additional terms beyond the alive bonus and min-norm penalty: a base velocity stabilization term that penalizes excessive horizontal drift, and an action smoothness penalty that reduces abrupt control changes. These are additional safety-relevant objectives that the residual must satisfy alongside the core survival objective.
Two training strategies are critical for generalization. First, during training, the residual policy encounters varying payloads at different body locations and random pushes that expose it to failure and near-failure states. Without this excitation, the policy learns an overly conservative filter that refuses to make any correction. Second, the residual policy is trained to handle multiple reference policy checkpoints, not just one. This forces it to learn a general correction that works across different reference behaviors rather than a correction specific to one policy's failure modes.
The training uses an asymmetric actor-critic architecture. The actor (the deployable residual policy) sees only observations available at deployment time: proprioceptive states and the reference policy action. The critic, used only during training, has access to privileged simulation information like ground-truth body states. This asymmetric setup is standard in sim-to-real humanoid RL but important to note because it means the deployed policy does not cheat with simulator-only information.
Simulation and Hardware Results
The experiments test three hypotheses: whether decoupled training beats single-policy training on the performance-safety-robustness trade-off, whether the residual generalizes across reference policies, and whether it transfers to real hardware. All experiments use the Unitree G1 humanoid.
Performance-robustness trade-off. The baseline policy (trained without payload randomization) achieves 24.95 mm tracking error without payload but 38.6 mm with payload, and falls 7.29% of the time under payloads. Training directly with payload randomization (Baseline*) drops the fall rate to 1.20% but tracking error jumps to 43-44 mm. ResSafe achieves the lowest fall rate at 1.11% while keeping tracking error at 37.66 mm with payload and 34.04 mm without. The decoupled approach gets closer to the nominal policy's tracking accuracy while matching the robustness of the payload-augmented policy.
Performance-safety trade-off. When safety cost weights are tuned in a single-policy framework, moderate weights reduce falls but larger weights catastrophically degrade tracking. Baseline* with a safety weight of 50 achieves 0% falls but tracking error balloons to 101.68 mm. At weight 100, tracking error reaches 353.55 mm and falls actually increase to 1.63%. ResSafe achieves 0.5% falls with 68.65 mm tracking error, a much more favorable operating point on the Pareto frontier.
Generalization across reference policies. When the residual is trained on a single reference policy checkpoint and tested on out-of-distribution checkpoints, the fall rate spikes sharply. But when trained across multiple checkpoints (using half the dataset for training), it generalizes well to unseen reference policies. The discriminating hyperplane baseline performs worse in both safety and robustness under the same conditions.
Hardware validation. The policy trained entirely in simulation is deployed on a physical Unitree G1 and tested across four reference motions with and without payloads. For motion #9350 with payload, the baseline falls every trial (10/10). Baseline* achieves 0/10 falls. ResSafe achieves 2/10 falls with comparable tracking error (49.81 mm vs 47.73 mm). For motion #1665 without payload, the baseline falls 8/10 times, Baseline* falls 2/10, and ResSafe falls just 1/10. Across multiple motions, ResSafe consistently achieves among the lowest fall rates while maintaining reasonable tracking error, and it generalizes better than the discriminating hyperplane filter which shows high variance in performance.
What the Numbers Mean and What They Don't
The results are compelling but come with important caveats that the authors acknowledge. The learned safety filter does not provide hard safety guarantees. Unlike a CBF or HJ reachability-based filter that provably maintains the system within a safe set, ResSafe is a learned policy that can fail. The alive bonus reward formulation maximizes expected survival time, not worst-case safety.
The residual policy is consistently more conservative than the discriminating hyperplane filter. The authors note that understanding this conservatism theoretically is an open question. The extra conservatism helps in practice (it provides the robustness margin mentioned earlier) but it also means the residual sometimes modifies safe actions unnecessarily, which slightly degrades tracking performance compared to the nominal policy.
Sim-to-real transfer relies on standard techniques (system identification, domain randomization) and does not address the gap between simulation and hardware explicitly. The gap is visible in the results: fall rates on hardware are generally higher than in simulation for the same motions.
The generalization capability is limited by the diversity of the training motion dataset. Fine-tuning can adapt to new motions, but the current framework does not learn on the fly. And the system operates without exteroceptive perception. Human balance relies heavily on visual feedback and environmental awareness. ResSafe is a blind safety filter that reacts only to proprioceptive states, which limits its applicability in unstructured environments.
Practical Implications for Humanoid Robotics
For engineers working on humanoid control, ResSafe suggests a practical architectural pattern. If you have a reference policy that tracks motions well but occasionally fails under disturbance, you can train a residual safety policy on top of it without retraining the reference. The residual can be trained across multiple reference policy versions, which means you can iterate on your nominal controller without retraining the safety layer from scratch each time.
The training recipe is also instructive. The combination of maximum survival time (alive bonus) with min-norm penalty and auxiliary safety rewards (velocity stabilization, action smoothness) produces a filter that is both effective and generalizable. The domain randomization strategy of varying payloads, locations, and random pushes during training is essential for avoiding over-conservative behavior. Training across multiple reference policy checkpoints is essential for generalization to unseen controllers.
The broader lesson is that decoupling safety from performance is not just a theoretical nicety. It produces measurably better results than trying to optimize both through a single reward function. This is consistent with the observation that classical safety filters work by modifying reference actions rather than replacing them, and ResSafe brings this principle into the learned-policy domain without requiring explicit dynamics models or safety certificates.
The limitations point toward future work: incorporating perception for environment-aware safety, developing theoretical guarantees for learned safety filters, and reducing the conservatism of the residual correction. The gap between simulation and hardware performance also suggests that more targeted sim-to-real techniques for the safety layer, rather than just the nominal policy, could yield further improvements.
Read the paper on arXiv