Intrinsic rewards are designed to guide exploration in reinforcement learning by assigning value to an agent's experience, for example through prediction error or learning progress. However, maximizing these rewards need not produce the most informative experience available. A formal criterion for exploration is proposed that compares policies by the counterfactual information they acquire: how well their histories can substitute for experience under alternative policies. The authors construct a single, simple environment in which specified count-based, prediction-error, empowerment, and information-gain objectives have maximizing policies that are Pareto-suboptimal at acquiring counterfactual information. Failures are explained and conditions established under which existing intrinsic rewards successfully encourage optimal exploration. An objective is also constructed that assigns a higher value whenever exploration strictly improves under the counterfactual criterion.
Counterfactual Information Criterion for Exploration
The core contribution is a formal criterion that evaluates policies by how well their histories can substitute for experience under alternative policies. This shifts the evaluation from maximizing intrinsic reward signals to assessing the informational value of the experiences collected. A policy that maximizes an intrinsic reward may nonetheless acquire less counterfactual information than another policy, meaning the reward signal alone is a incomplete guide to useful exploration.
Pareto-Suboptimal Maximizers in a Single Environment
In a carefully constructed environment, count-based, prediction-error, empowerment, and information-gain objectives all produce maximizing policies that are Pareto-suboptimal with respect to acquiring counterfactual information. This means each objective, when maximized in isolation, can lead the agent toward experiences that are rewarding according to that objective but informative in a different, counterfactual sense. The paper uses this single environment to isolate and diagnose the failure modes of each objective.
Explaining the Failures
Count-based methods can be misled by spurious frequency signals; prediction-error objectives may settle on predictable but uninformative patterns; empowerment maximization can concentrate on trivial action repetitions; and information-gain objectives may overfit to the agent's current feature representations. The paper details how each failure arises from the mismatch between the objective's myopic optimization and the broader counterfactual information criterion.
Conditions for Successful Exploration
Despite the failures, the authors establish conditions under which existing intrinsic rewards do successfully encourage optimal exploration. These conditions relate to the structure of the environment, the richness of the agent's representations, and the alignment between the intrinsic reward's optimization target and the counterfactual information gain. When these conditions hold, the intrinsic reward becomes a reliable proxy for useful exploration.
A New Objective for Counterfactual Exploration
The paper concludes by constructing an objective that assigns a higher value whenever exploration strictly improves under the counterfactual information criterion. This objective provides a direct optimization target that addresses the identified gaps in prior approaches. In the constructed environment, this new objective produces policies that are not Pareto-suboptimal for counterfactual information acquisition, demonstrating that the proposed criterion can guide more effective exploration.
By formalizing exploration as counterfactual information acquisition and exposing the limitations of standard intrinsic rewards, the work provides both diagnostic insight and a concrete path toward more reliable exploration in reinforcement learning.