Reinforcement learning for dynamic treatment regimes requires policies that adapt to patient-specific conditions while remaining reliable despite uncertainties in measurement and model structure. This paper develops a robust reinforcement learning framework that addresses both generalization across patients and robustness under errors in measured biomarkers and shifts in disease dynamics. The core innovation models the patient as a hidden context, enabling a unified treatment of generalization and robustness through robust contextual Markov decision processes. This formulation allows the learning algorithm to maintain performance when biomarker measurements are accurate and disease evolution follows anticipated patterns, while also providing guarantees under perturbed conditions.
The framework is applied to adaptive therapy for metastatic prostate cancer using a simulator calibrated to clinical data. The resulting policies demonstrate substantial improvement over existing adaptive therapy methods. A key finding is that the robust policies attain near-original performance when the biomarker is accurate and the disease evolves as anticipated, showing that robustness does not come at significant nominal cost. Across a diverse array of scenario perturbations—including biomarker noise and model misspecification—the proposed policies Pareto-dominate the existing method, meaning they achieve at least as good outcomes across all measured dimensions and strictly better outcomes in many cases.
The robust contextual MDP formulation works by maintaining a distribution over possible patient contexts rather than assuming a single known state. When a biomarker measurement arrives, the algorithm updates this distribution using robust update rules that account for possible measurement error. Policy evaluation and optimization then proceed with respect to this posterior context distribution, ensuring that the resulting treatment policy performs well across the plausible range of patient states rather than overfitting to a single observed context. This approach differs from standard contextual RL, which conditions on the observed biomarker value directly and can be fragile when that value is noisy or misleading.
The paper provides extensive evaluation across multiple uncertainty scenarios. Performance is assessed under varying levels of biomarker measurement error, different rates of disease dynamics shift, and combinations of both perturbations. In all cases, the robust policies maintain meaningful performance relative to the nominal case, while existing methods degrade more substantially. The authors also analyze the trade-off between robustness and average performance, showing that the proposed method achieves a favorable point on this trade-off frontier.
From a methodological perspective, the contribution extends robust RL beyond the setting of known transition dynamics and reward functions. By incorporating context uncertainty into the MDP framework, the approach provides a principled way to learn policies that are both generalizable—performing well across diverse patient populations—and robust—maintaining performance when the world does not match the assumptions under which the policy was learned. The robust contextual MDP formulation can be adapted to other domains where patient heterogeneity and measurement uncertainty are critical factors.
For clinical applications, the framework offers a way to generate adaptive treatment policies that physicians can trust even when diagnostic measurements are imperfect or patient conditions deviate from expected trajectories. The prostate cancer therapy setting demonstrates that such policies can substantially improve upon current standard-of-care adaptive methods, which typically do not account for measurement uncertainty or dynamics shifts in a systematic way. The authors note that the simulator used for evaluation, while calibrated to clinical data, is a simplified representation of real-world disease progression, and that further validation in clinical trials would be necessary before direct deployment.