Several state-of-the-art methods for online reinforcement learning in continuous control improve policies using action gradients of a learned critic. However, critics are typically trained to predict returns, and accurate value predictions do not necessarily yield accurate action derivatives, potentially leading to unreliable policy updates. We propose Forward Entropy-Regularized Policy Optimization (FERPO), an on-policy maximum entropy reinforcement learning algorithm that performs policy improvement using critic values without differentiating the critic with respect to actions.
Algorithm Design
FERPO derives an optimal target action distribution from a policy-improvement objective regularized by entropy and Kullback-Leibler (KL) divergence. The actor is fitted to this target by minimizing a forward-KL objective, estimated using self-normalized importance sampling (SNIS) with actions drawn from the rollout policy. By limiting the target distribution's deviation from the rollout policy, the KL regularization helps keep these importance weights well behaved.
Exploration Properties
In contrast to reverse-KL objectives, which can favor a subset of the target distribution's modes, the forward-KL objective encourages coverage of multiple high-value modes and thereby promotes exploration. This property distinguishes FERPO from existing maximum entropy RL algorithms and contributes to its sample-efficiency gains.
Experimental Results
Experiments and ablations on MuJoCo Playground and ManiSkill show competitive performance and sample-efficiency gains. Computational benchmarks also demonstrate faster actor updates than Relative Entropy Pathwise Policy Optimization (REPPO). These results indicate that the forward-KL objective provides practical benefits beyond theoretical considerations.
Key Advantages
FERPO performs policy improvement using critic values without differentiating the critic with respect to actions, avoiding the issue where accurate value predictions do not necessarily yield accurate action derivatives. The forward-KL objective's encouragement of multiple high-value mode coverage promotes exploration while maintaining stable importance weight behavior through KL regularization.