Belief-shift branching is a technique for improving Tree-structured Reinforcement Learning (RLVR) by dynamically placing forks within the rollout chains. The method identifies pivots in the expected outcome curve of the model, focusing on points where consecutive answers diverge most significantly. This placement minimizes the cost of additional sampling, a key constraint in realistic RL budgets. The technique utilizes a black-box probe, a logit-lens depth profile, or a learned activation direction to determine these divergence points.
The probe costs approximately 1% of step compute on mathematics and under 5% on code when integrated into the rollout engine. Validation against Monte-Carlo value curves demonstrates belief-shift branching's effectiveness, ranking first across eight model-benchmark panels. Specifically, the method achieves a +2.6 aggregate and +2.9 on AIME 2026 over the strongest baseline using the OLMo-3-7B model, and sweeps every OLMo code column with a +6.5 gain on LiveCodeBench-medium.
These results highlight the importance of strategic fork placement in RLVR. By focusing on areas of high belief divergence, the method maximizes the information gained from each branching point, leading to improved reward signals and ultimately, better performance. The approach offers a practical and computationally efficient way to enhance the capabilities of tree-structured reinforcement learning agents.
Source: https://arxiv.org/abs/2609.11061