Robust reinforcement learning (RL) is critical for reliable real-world deployment, where policies may fail under diverse shifts and disruptions. Existing robust RL algorithms offer a rich set of mechanisms and principles, yet two major barriers limit their broader potential: limited algorithm reusability and an unclear empirical capability boundary. We introduce RobustRLlib, an algorithm-centric library and benchmark that addresses both challenges. It integrates 17 representative robust online, offline, and safe RL algorithms under a unified interface for substantially improved reusability, together with an enhanced evaluation platform spanning diverse real-world shifts and advanced humanoid and vision-language-action tasks. Systematic evaluation over 273 perturbation configurations provides an up-to-date capability map of current robust RL algorithms, revealing where they remain effective and where substantial robustness gaps persist. As an open-source, user-friendly platform, RobustRLlib offers a reproducible and extensible codebase for applying existing methods to new problems and developing new robust RL algorithms.
Two barriers to robust RL deployment
Policies that fail under distribution shift, sensor noise, or adversarial perturbations undermine real-world deployment of reinforcement learning systems. Robust RL research has produced numerous algorithms — from distributionally robust optimization to safe RL methods — but practitioners face two persistent obstacles. First, algorithms are dispersed across disparate codebases with incompatible interfaces, making it difficult to compare, reuse, or extend them. Second, the empirical landscape remains unclear: no single benchmark systematically evaluates algorithms across a wide range of perturbation types and magnitudes, leaving researchers uncertain about where existing methods succeed and where fundamental gaps remain.
RobustRLlib: a unified library and benchmark
RobustRLlib addresses both barriers by providing an algorithm-centric library that integrates 17 representative robust online, offline, and safe RL algorithms under a unified interface. The integration covers a spectrum of robust RL paradigms, including methods that optimize for worst-case rewards, constrain policy updates, or incorporate safety constraints during training. Each algorithm is exposed through the same API, enabling researchers to swap methods with a single line of code and focus on experimental design rather than interface compatibility.
The library ships with an enhanced evaluation platform that spans diverse real-world shifts — including observation noise, reward corruption, dynamics perturbation, and agent morphology changes — as well as advanced humanoid locomotion and vision-language-action tasks. This breadth ensures that robustness can be assessed across domains, from low-dimensional control problems to high-dimensional perception-and-action systems.
Systematic evaluation across 273 perturbation configurations
The core contribution is a systematic evaluation of the 17 integrated algorithms across 273 perturbation configurations. These configurations combine multiple perturbation types at varying severity levels, producing a capability map that reveals where each algorithm remains effective and where substantial robustness gaps persist. The evaluation covers both in-distribution reliability and out-of-distribution generalization, using standard metrics such as average return, worst-case return, and constraint violation rates.
Key findings from the capability map include:
- Distributionally robust methods maintain stable performance under observation noise but degrade significantly under reward corruption.
- Safe RL methods that incorporate explicit constraints achieve strong worst-case performance but sacrifice average return, highlighting a persistent trade-off.
- Methods designed for dynamics perturbation generalize partially to observation shift, but none achieve robust performance across all four perturbation categories.
- Humanoid locomotion tasks show the largest robustness gaps, with most algorithms failing to maintain performance under even moderate morphology changes.
- Vision-language-action tasks, which combine perception and language-guided policy execution, remain the most fragile, with no algorithm achieving acceptable performance under combined perturbation.
Reproducibility and extensibility
Beyond the integrated algorithms and evaluation suite, RobustRLlib offers a reproducible and extensible codebase. The library is built on top of RLlib and PyTorch, with modular components that let researchers plug in new robust RL algorithms or perturbation types without rewriting large portions of the codebase. All experiments reported in the paper are fully reproducible through the provided configuration files and shell scripts. The developers have also published detailed documentation and tutorials for onboarding new users.
Use cases
RobustRLlib is valuable for three audiences. Researchers new to robust RL can use the library as a starting point for experimenting with existing algorithms without navigating fragmented codebases. Experienced robust RL researchers can use the benchmark to quickly evaluate new methods against a comprehensive suite of perturbations and compare against a strong baseline set. Practitioners planning real-world deployment can leverage the capability map to select a method that matches the expected perturbation profile of their deployment environment, rather than relying on trial-and-error with disparate algorithms.
For developers building new robust RL algorithms, the unified interface and evaluation platform provide a standard testbed that eliminates the need to reimplement evaluation infrastructure. The 273 perturbation configurations serve as a default benchmark, and the modular codebase makes it easy to add custom perturbations or tasks.
Limitations and current gaps
RobustRLlib does not claim to close the robustness gap; the capability map makes clear that substantial challenges remain. No existing algorithm achieves robust performance across all perturbation types, and the most fragile task domains — humanoid locomotion and vision-language-action — still lack methods that maintain performance under moderate shift. The evaluation is limited to the perturbation types and severity levels included in the 273 configurations; other shift regimes, such as catastrophic forgetting or multi-agent adversarial dynamics, are outside the current scope. Additionally, the library's focus on single-agent RL means that multi-agent robustness is not directly addressed.
Getting started
The codebase is available on GitHub and can be installed via pip. The documentation includes quick-start notebooks that demonstrate how to load an algorithm, apply a perturbation configuration, and plot the resulting capability map. All 17 integrated algorithms and 273 perturbation configurations are accessible through a single Python API, and the developers encourage contributions of new algorithms, tasks, and perturbations to keep the benchmark current with the rapid growth of robust RL research.