Continuous-time reinforcement learning provides a natural framework for sequential decision making in systems governed by differential equations, with applications in robotics, autonomous systems, manufacturing, and finance. Yet in many of these settings the system evolves continuously while data are recorded only at discrete times through sensors or simulation logs. When observations arrive at isolated timestamps, standard discrete-time RL methods discard the continuous-time structure and introduce discretization error, even when the transition kernel is known exactly. Hernandez, Qing, Zhou, and Zhu address this gap with a policy gradient framework that combines rigorous convergence analysis with learning from discrete data. Their work establishes a performance difference lemma, derives policy gradient formulas for randomized and parametrized policies, proves exponential convergence of an entropy-regularized policy flow, and develops model-free actor-critic algorithms through the PhiBE framework that preserve the continuous-time structure using only discrete transition data. Numerical experiments on an entropy-regularized linear-quadratic regulator and a nonlinear wheeled robot demonstrate the performance and stability of the proposed methods with both known and data-driven dynamics.
From Continuous-Time Control to Discrete Observations
Many sequential decision systems are naturally described by stochastic differential equations. The state process follows dXt = b(Xt, At)dt + σ(Xt, At)dWt, where Wt is a Brownian motion, and the controller selects actions At according to a policy π. The objective aggregates discounted rewards over continuous time: J[π] = ∫ Vπ(s)ρ0(ds), where Vπ(s) = 𝔼[∫₀^∞ e^(-βt) r_λ^π(st) dt | s₀ = s] and the entropy-regularized state reward is r_λ^π(s) = ∫ (r(s,a) - λ log π(a|s)) π(a|s) ν(da). Here β > 0 is the discount factor and λ ≥ 0 is the entropy-regularization parameter.
Under a fixed policy, the value function satisfies a Poisson equation involving the infinitesimal generator L_{bπ,Σπ}Vπ(s) - βVπ(s) + r_λ^π(s) = 0, where the generator is L_{bπ,Σπ}V(s) = bπ(s)⋅∇V(s) + ½Σπ(s):∇²V(s). Value-based methods face two fundamental difficulties in this setting. First, the value function is characterized by a PDE involving the infinitesimal generator rather than by a Bellman recursion. Second, the usual discrete-time state-action value function loses action-dependent information in the continuous-time limit, as the Q-function converges to the state value function when action duration tends to zero. Even after policy evaluation, extracting a policy requires additional computation: greedy construction needs optimization over the action space at every state, while softmax construction needs approximating its normalization constant. These issues make pure value-based approaches cumbersome for continuous-time control.
Policy gradient methods avoid the separate policy-extraction step by optimizing a parametrized policy directly. Direct policy parametrization also makes it possible to control the size of each update through trust-region or proximal methods that restrict the change between consecutive policies via a KL constraint or a clipped surrogate objective. Policy gradient methods still require policy evaluation, since the gradient depends on an advantage function, but the resulting estimate updates the policy directly rather than requiring a subsequent maximization problem. For continuous-time stochastic control with known dynamics, policy gradient formulas and convergence results exist under suitable assumptions, and the advantage can be constructed from the drift, the diffusion, and the policy value function. However, these analyses generally assume access to system coefficients or to exact policy-evaluation quantities. When dynamics are unknown, learning from discrete observations becomes necessary, and developing data-driven continuous-time policy gradient methods with convergence guarantees that explicitly account for evaluation and parametrization errors remains an open challenge.
A Performance Difference Lemma for Continuous-Time Policies
The analysis begins with a performance difference lemma that characterizes the first-order variation of the objective J[π] under policy perturbation. The lemma uses two central objects: the continuous-time advantage function and the discounted occupancy measure.
The continuous-time advantage Aπ(s,a) measures the instantaneous gain of holding action a fixed over a short time interval before reverting to policy π. Formally, Aπ(s,a) = lim(Δt→0+) (1/Δt) 𝔼[∫₀^Δt e^(-βt)(r(st^a,a) - λ log π(a|st^a)) dt + e^(-βΔt)Vπ(s_Δt^a) - Vπ(s) | s₀^a = s]. The advantage has the pointwise representation Aπ(s,a) = r(s,a) - λ log π(a|s) + [L_{b,Σ}Vπ(s,a) - βVπ(s)], and its policy average satisfies ∫ Aπ(s,a) π(a|s) ν(da) = 0.
The discounted occupancy measure ρπ(B) = β ∫ 𝔼[∫₀^∞ e^(-βt) ¹_B(st) dt | s₀ = s] ρ0(ds) describes the discounted frequency with which state s is visited under policy π. With these objects, the performance difference lemma states: J[π^] - J[π] = (1/β) ∫ [∫ Aπ(s,a) π^(a|s) ν(da) - λ KL(π^_s || π_s)] ρπ^(ds). This identity measures the performance gap between two policies and provides the foundation for the policy gradient theorem.
The policy gradient theorem characterizes the directional derivative of J[π] at π in the direction of an admissible policy perturbation ϕ. Specifically, d/dε J[π + εϕ]|ε=0 = (1/β) ∫∫ Aπ(s,a) ϕ(a|s) ν(da) ρπ(ds). This holds for arbitrary additive perturbations of the policy density that preserve the admissible policy class, providing a general representation of the Gâteaux derivative of the performance functional.
Convergence of Continuous-Time Policy Optimization
For continuous-time policy optimization, the paper establishes exponential convergence of an entropy-regularized policy flow to the regularized optimum. The entropy regularization yields a policy flow that converges geometrically fast to a neighborhood of the optimum, with the convergence rate controlled by the regularization parameter λ. This result considers infinite-horizon stationary randomized policies and serves as the foundation for analyzing discrete parametrized updates.
For finite-step parametrized updates, the paper derives a convergence bound that explicitly separates three contributions: critic error, policy-regression error, and the learning step size. Under suitable regularity and approximation-error assumptions, the bound establishes geometric decay to an error neighborhood whose size is controlled by these quantities. This is, to the best of the authors' knowledge, the first convergence bound for discrete, finite-dimensional parametrized policy-gradient updates in a general nonlinear diffusion control setting that explicitly accounts for critic error, compatible score-regression error, and the actor step size. Prior works either rely on LQ or closely related problem-specific structure, or optimize over function spaces of feedback controls rather than over a prescribed finite-dimensional policy class, and consequently do not quantify the additional approximation error introduced by restricting updates to such a class.
The finite-step bound has the form: after k iterations, the policy error is bounded by a term that decays geometrically with rate ρ < 1, plus a steady-state term proportional to (critic error + policy-regression error + step size). This decomposition makes it clear that improving any of the three components—more accurate critics, better score estimation, or appropriately tuned step sizes—directly improves the final policy quality.
The PhiBE Framework for Data-Driven Policy Evaluation
To enable learning under unknown dynamics, the paper integrates the framework with PhiBE, which constructs a continuous-time surrogate generator from discrete transition data. PhiBE embeds discrete transition information into a continuous-time PDE for policy evaluation, retaining the first- and second-order spatial derivatives of the critic and replacing the unknown generator by a PhiBE operator that uses discrete-time data. The state-action PhiBE formulation enables off-policy learning while retaining instantaneous action-dependent information that vanishes from the conventional discrete-time Q-function in the continuous-time limit.
The PhiBE construction works as follows. Given discrete transition tuples (st, at, rt, st+1) collected at fixed time intervals, a continuous-time generator is estimated that matches the observed transition statistics. This generator then serves as the critic in the policy gradient algorithm, providing advantage estimates without reducing the problem to a discrete-time MDP. The resulting algorithms are inherently model-free: they do not identify the system dynamics, and they preserve the continuous-time structure of the control problem.
Two algorithm variants are developed. The on-policy algorithm updates the critic and actor using transitions collected while following the current policy. The off-policy algorithm can leverage previously collected data, importance-sampling to correct for distribution shift, and the PhiBE operator to estimate advantages from off-policy data. Both variants maintain the continuous-time structure and the convergence guarantees established in the theoretical analysis.
Numerical Results: LQR and Wheeled Robot Control
The paper evaluates the proposed methods on two problems. The first is an entropy-regularized linear-quadratic regulator (LQR) with known dynamics and with dynamics identified from data. The second is a nonlinear differential wheeled robot with both known and data-driven dynamics.
Entropy-regularized LQR
For the LQR problem, the system dynamics are dXt = (AXt + BuAt)dt + σdWt, with quadratic instantaneous reward r(s,a) = -xᵀQx - aᵀRa. The optimal policy is linear in the state: π*(a|s) ∝ exp(-½aᵀR⁻¹BᵀPx), where P solves the associated algebraic Riccati equation. The policy class considered is parametrized as πθ(a|s) ∝ exp(-θᵀφ(s,a)), where φ are compatible function approximation features.
Three critic variants are tested. With known dynamics, the critic has access to the true generator and achieves the lowest advantage estimation error. Optimal-PhiBE with known discrete-time dynamics uses the PhiBE operator with correctly specified discrete-time transition statistics. Data-driven off-policy PhiBE estimates the generator from discrete observations without knowing the true dynamics. On-policy PhiBE uses transitions collected under the current policy.
Results across regimes show that all PhiBE-based critics achieve comparable performance to the known-dynamics baseline when the number of observations is sufficient, demonstrating that the PhiBE operator can approximate the true generator from discrete data alone. The data-driven variants incur a small additional error that diminishes as the observation horizon grows, confirming the practical viability of the model-free approach.
Differential wheeled robot control
The second problem is a nonlinear differential-drive robot. The state includes position, orientation, and velocity; the action is a pair of wheel speeds. The dynamics are stochastic, with process noise modeling unmodeled friction and surface irregularities. Two dynamics settings are considered: known dynamics, where the drift and diffusion coefficients are given analytically, and data-driven dynamics, where the generator is estimated from simulation logs.
Under known dynamics, the proposed actor-critic algorithm achieves rapid policy improvement within tens of iterations. The critic converges to a low mean-squared advantage error, and the policy parameters move toward the optimum with a stable step-size schedule. Policy evaluation matches the true value function within a small error bound, and the entropy-regularized objective improves monotonically.
With data-driven dynamics, the algorithm uses PhiBE to construct the continuous-time generator from discrete transition data. The policy still improves, but the convergence rate is slower due to the additional critic error. After more iterations, the policy reaches a neighborhood of the optimum whose size is consistent with the theoretical bound that separates critic error, policy-regression error, and step size. The experimental results confirm the bound's predictive power: plots of the policy error over iterations show the expected geometric decay plus a steady-state floor proportional to the estimated critic error.
In both dynamics settings, the on-policy and off-policy variants produce comparable results, suggesting that the PhiBE operator effectively handles distribution shift when sufficient off-policy data are available. The off-policy variant is particularly valuable when collecting new data is expensive or when reusing existing simulation logs is desirable.
What This Means for Practitioners of Continuous-Time RL
If you are working with sequential decision systems governed by differential equations and have only discrete observations—which is the common case in robotics, autonomous driving, and many industrial applications—this work provides both theory and practice for policy gradient optimization. The key contributions to take away are:
- A performance difference lemma that expresses the policy gradient in terms of the continuous-time advantage and discounted occupancy measure, valid for randomized and parametrized policies under general growth and regularity conditions.
- Convergence guarantees for both continuous-time policy flows (exponential convergence under entropy regularization) and finite-step parametrized updates (geometric decay with explicit critic/error/step-size decomposition).
- Model-free actor-critic algorithms based on PhiBE that construct a continuous-time surrogate generator from discrete transition data, without identifying the dynamics or reducing to a discrete-time MDP.
- Two algorithm variants, on-policy and off-policy, that preserve the continuous-time structure and that can leverage previously collected data.
For implementation, the critical design point is the PhiBE critic, which replaces the unknown generator with an operator estimated from data. The on-policy variant uses transitions collected under the current policy; the off-policy variant importance-samples from previously collected data. Both maintain the convergence guarantees and the continuous-time structure. The theoretical bound that separates critic error, policy-regression error, and step size is a practical diagnostic: monitor these three quantities during training to understand whether stagnation is due to a poor critic, inadequate score estimation, or an overly large learning rate.
The code and model checkpoints are released alongside the paper, and the authors report that both the LQR and wheeled-robot experiments can be reproduced with the provided scripts. The PhiBE operator is the new component: it takes discrete transition tuples as input and outputs a continuous-time generator estimate that can be plugged into any policy gradient framework. The paper's GitHub repository includes implementations for both the LQR and robotics problems.
SceneRig, the 3D scene reconstruction system from the previous paper, tackles a different problem, but both works share a similar philosophy: bring structure from the continuous-time domain into pipelines that operate on discrete data. Where SceneRig uses geometric and physical tools to validate scene reconstruction, this work uses the PhiBE operator to validate policy evaluation. In both cases, the key insight is that treating the continuous-time structure as a first-class citizen—rather than discretizing away or adding it as an afterthought—leads to better final performance and more reliable out-of-distribution generalization.
For researchers extending this work, the natural directions are: extending the convergence analysis to risk-sensitive objectives, replacing the PhiBE operator with learned generator estimators (e.g., via neural PDE solvers), and applying the framework to high-dimensional continuous control problems such as robotic manipulation with sparse rewards. The separation of critic error, policy-regression error, and step size in the convergence bound also suggests a curriculum for joint optimization: first improve the critic, then refine the policy parametrization, then tune the step size.
Read the paper on arXiv