Deep search agents solve knowledge-intensive tasks through long-horizon multi-turn interactions. Assigning credit to individual search decisions is fundamentally difficult. Outcome-based reinforcement learning provides only a single terminal reward shared across an entire trajectory.
Recent process-supervision methods aim to assign credit at finer granularity. But they typically anchor credit to either surface-level trajectory states or model-internal signals. Both become unreliable in deep search where observations are heterogeneous and retrieved contexts are long and noisy.
Knowledge-grouped alignment design
We propose KnowGPO, a reinforcement learning framework that aligns trajectories by the task-relevant knowledge they have acquired rather than by their surface observations. For each query, KnowGPO decomposes the information required to derive the answer into atomic knowledge conditions. The state of a trajectory at each step is represented by the subset of conditions it has satisfied. Trajectories sharing the same knowledge state are grouped for within-group comparison. The incremental knowledge gain between consecutive steps serves as an environment-grounded process reward.
This grouping shifts the comparison surface. Instead of pairing trajectories that differ in surface tokens or hidden activations, KnowGPO pairs trajectories that have acquired the same set of knowledge conditions. The reward signal then measures only what changed — which new condition was satisfied — rather than judging the entire path from scratch.
Benchmarks and observed gains
Experiments on Xbench, BrowseComp-ZH, and GAIA show KnowGPO consistently outperforms outcome-only (GRPO) and process-supervision baselines (GiGPO, IGPO) by 3.8 to 4.2 points. An 8B model trained with KnowGPO rivals or exceeds most 32B open-source systems, suggesting that knowledge-grouped optimization can extract stronger learning signals from smaller models.
Inferred limitations and trade-offs
The quality of the knowledge decomposition step determines the usefulness of the grouping. If atomic conditions are too fine-grained, trajectories cluster sparsely and the reward signal becomes high-variance. If conditions are too coarse, the discrimination between meaningful progress and stagnation diminishes. The framework also depends on accurately identifying the knowledge conditions required for each query, which may require task-specific engineering. Incremental knowledge gain assumes a well-defined ground-truth set of conditions per query, and errors in that set propagate directly into the process reward.
Practical takeaways
A working developer can integrate KnowGPO into a search agent pipeline without requiring dense surface annotations. The knowledge-grouped comparison provides a more stable learning signal for long-horizon tasks, and the 8B-versus-32B result indicates that cost-effective deployment becomes feasible when the training signal is well targeted. Builders interested in process supervision for retrieval-augmented or multi-step reasoning systems can start by defining atomic knowledge conditions per task and grouping trajectories by the conditions they satisfy at each step.
Read the paper on arXiv