Reinforcement learning is one of those fields where the gap between "I read the textbook" and "I can implement this" is unusually wide. The core loop, agent observes state, chooses action, receives reward, updates policy, sounds simple in prose. In practice, it involves a web of interacting objects: value functions, policies, experience buffers, target networks, advantage estimators, each cycling through updates that depend on the others. Students can read Sutton and Barto cover to cover and still not know where to start when they open a blank file. The gap is not about intelligence. It is about the distance between equations and working code.
RLLBC-Lib, built by researchers at RWTH Aachen University's Institute for Data Science in Mechanical Engineering, is designed to close that gap. It is not another deep RL library with production-grade abstractions. It is a teaching tool: a collection of Jupyter notebooks that walk students from tabular Q-learning to deep policy gradients, with consistent structure, detailed explanations, and built-in support for automated grading. The library also situates RL within the broader field of learning-based control, giving students the context to know when RL is the right tool and when a classical controller would do better.
Why tabular methods come first
The library's most deliberate design decision is starting with tabular methods before touching neural networks. Most deep RL libraries (CleanRL, Stable-Baselines3, Tianshou) begin with DQN or PPO and treat tabular methods as a historical footnote. RLLBC-Lib reverses this. The tabular section includes dynamic programming (policy iteration, value iteration), Monte Carlo methods (first-visit and every-visit), and temporal difference learning (Q-learning, SARSA, Dyna-Q). Each is implemented in a self-contained single-file notebook with markdown explanations that connect the code directly to the relevant equations in Sutton and Barto.
The reason is pedagogical, not technical. Tabular methods make the feedback loops visible. In Q-learning, you can see exactly how the Q-table updates after each step, how the greedy maximization over the next state's values pulls the policy, and how epsilon-greedy exploration introduces stochasticity. In Dyna-Q, you can trace how simulated transitions from a learned model supplement real experience. These loops are the conceptual core of RL, and they disappear behind abstraction layers in deep implementations. By implementing them from scratch in a notebook, students build the intuition they need before the function approximation and replay buffers of deep methods obscure the mechanism.
The tabular section also includes visualizations that make the learning dynamics concrete. A grid world environment displays the full state-action value function as colored triangles at each state, one per action, so students can watch the Q-values shift across training. A recycling robot implementation, drawn from Example 3.3 in Sutton and Barto, introduces MDP dynamics and value function learning on a problem small enough to reason about by hand.
From tabular to deep, with the same skeleton
The deep RL section mirrors the tabular section's structure. Every notebook follows the same four-part template: agent and environment setup, hyperparameter configuration, training loop, and evaluation. This is not cosmetic. It lets students draw direct parallels between tabular Q-learning and DQN, between Monte Carlo policy evaluation and REINFORCE, between SARSA and actor-critic methods. The deep notebooks are based on CleanRL's single-file implementations, using PyTorch for neural network training, with the gymnasium interface for environment interaction.
The deep section covers value-based methods (DQN), policy gradient methods (REINFORCE), and actor-critic methods (A2C, TRPO, PPO, DDPG, TD3, SAC). Each notebook includes markdown blocks explaining the implementation details, referencing the original papers and connecting pseudocode to code. For instance, the PPO notebook discusses how the clipped surrogate objective is implemented, how generalized advantage estimation is integrated, and how the trust region constraint is enforced. The DQN notebook explains experience replay, target networks, and the epsilon-greedy exploration schedule.
A practical addition is the integration with Weights and Biases for experiment tracking. Students can log return curves, temporal difference losses, and other training metrics across multiple random seeds, then compare hyperparameter settings in real time. Videos of agent performance can be logged to W&B projects, allowing students to connect a particular return value to the corresponding behavior. This introduces a workflow that is standard in research but rarely taught explicitly in courses.
The library also handles agent evaluation at configurable frequency over a variable number of episodes, helping students understand how exploration introduces stochasticity in performance. Checkpointing saves the best-performing version of the agent, so students can load a trained model after an expensive training run and compare it to other agents or investigate failure scenarios.
Learning-based control: the broader context
The "LBC" in RLLBC-Lib stands for learning-based control, and this is where the library distinguishes itself from pure RL teaching tools. RL is one approach to sequential decision-making, but it is not the only one. Control theory offers model-based methods like linear quadratic regulators, model predictive control, and system identification that often work better when a dynamics model is available or can be learned. Bayesian optimization provides a sample-efficient alternative for controller tuning.
The LBC section includes notebooks on each of these: linear quadratic regulation, model predictive control, system identification and dynamics learning, and Bayesian optimization for controller tuning. These notebooks contrast RL with control-theoretic approaches, discussing when each is appropriate. A student who has only seen RL might treat every control problem as a Markov decision process. The LBC notebooks show that sometimes a simple PID controller or an MPC formulation is more appropriate, more sample-efficient, and more interpretable.
This perspective is especially valuable for computer science students, who typically have limited or no control background. Understanding the control-theoretic alternatives prevents the "if all you have is a hammer" problem and supports interdisciplinary collaboration. The boundary between RL and other learning-based methods is fluid: works like information-theoretic MPC and deep reinforcement learning in a handful of trials using probabilistic dynamics models sit at the intersection, and the LBC notebooks help students understand these connections.
Programming assignments with automated grading
The library includes three programming assignments designed for self-study or course use, all with automated grading via nbgrader.
The first assignment covers tabular methods. Students implement an MDP for a robot playing basketball as a gymnasium environment from a text and graph description, then implement double Q-learning and an off-policy Monte Carlo algorithm. The second assignment covers deep RL, starting from DQN, building to double DQN with prioritized experience replay, and integrating generalized advantage estimation into PPO. Both assignments mix provided code with empty cells for students to fill in.
The third assignment is an open racing challenge. Students pick their algorithm of choice, adapt the state formulation and action scaling, and submit their agent to a leaderboard that ranks the best performances across all teams. The only grading criterion is racing performance. This assignment mirrors real research and industry problem settings, where the problem is specified but the solution approach is open.
The automated grading system includes an upload page that continuously grades new submissions, so students can iteratively improve their scores. The assignments and grading page are kept unpublished for didactic reasons, but the infrastructure is designed to scale to large courses.
Design choices that matter for teaching
Several design decisions reflect experience teaching RL to hundreds of students in both classroom and online formats. The single-file implementation style, following CleanRL's approach, means each algorithm is self-contained. Students do not need to navigate a package hierarchy to understand how the pieces fit together. The common virtual environment based on pixi-env handles dependency management, avoiding the "it works on my machine" problem that plagues course assignments.
The consistent notebook structure between tabular and deep methods is not just organizational. It makes the parallels explicit. When a student implements Q-learning in the tabular section and then sees DQN in the deep section, the same four-part template (setup, hyperparameters, training loop, evaluation) makes it clear that DQN is Q-learning with a neural network replacing the Q-table, experience replay replacing single-transition updates, and a target network stabilizing bootstrapping. The markdown explanations in each notebook make these connections explicitly.
The library is available at github.com/Data-Science-in-Mechanical-Engineering/RLLBC under a CC BY 4.0 license, and is used in the authors' MOOCs on edX. The repository includes a comparison table of RL teaching libraries (Appendix A.1) that maps which features each provides: Jupyter notebooks, dynamic programming, tabular RL, deep RL, examples and explanations. RLLBC-Lib is the only library in the table that checks all five boxes.
Limitations and what it does not cover
The library focuses on foundational algorithms. It does not include recent advances like offline RL, multi-agent RL, or reward learning from human feedback (RLHF). The deep RL section covers the standard set of algorithms but does not extend to transformer-based architectures or world models. The LBC section covers classical control but does not include recent work on differentiable simulation or learned physics engines.
The programming assignments are kept unpublished, which limits adoption by instructors who might want to use them directly. The automated grading infrastructure, while designed for scale, requires setup that may be non-trivial for instructors unfamiliar with nbgrader.
The library targets students and learners, not practitioners looking for production-grade implementations. The single-file style prioritizes readability over modularity, and the gymnasium interface, while standard, adds overhead that production systems would avoid. These are deliberate trade-offs in service of the pedagogical goal.
What this means for RL education
RL education has a infrastructure problem. The textbooks are good. The theory is well-developed. But the gap between reading about Q-learning and implementing it, between understanding PPO's clipped objective and debugging a training run that diverges, is where most students get stuck. RLLBC-Lib addresses this gap directly, not with more theory, but with accessible, explained, runnable code.
For instructors, the library provides a ready-made course resource that scales from introductory lectures (the Tic-Tac-Toe self-play notebook where a lecturer can play against the agent in real time) to advanced assignments (the open racing challenge with a leaderboard). The automated grading via nbgrader means the assignments can be deployed in courses with hundreds of students without proportional increases in grading effort.
For self-learners, the library provides a structured path from "I know what a Markov decision process is" to "I can implement PPO and compare it to SAC on a continuous control task." The consistent structure and detailed explanations mean that each notebook builds on the previous ones, rather than requiring students to piece together implementations from disparate sources.
The library also makes a subtle but important point: RL is not a standalone technique. It exists within the broader field of learning-based control, alongside model predictive control, system identification, and Bayesian optimization. Teaching RL in this context, as RLLBC-Lib does, produces students who know not just how to use RL but when to use it and when to reach for a different tool. That is a more useful education than mastery of any single algorithm.
Read the paper on arXiv