Looped Transformers achieve parameter efficiency by repeatedly executing a shared block across recurrent loops. Each loop yields an intermediate representation decodable for the same next token, yet standard decoding discards earlier states. Because earlier loops embody less computation, recurrence inherently supplies aligned weak-and-strong prediction pairs without auxiliary models or external training.
LoopCD-Logits: Contrasting Final Predictions with an Earlier Logit Pass
LoopCD-Logits guides token selection by contrasting the final prediction with an earlier recurrent pass in logit space. The framework requires one extra output pass through the looped transformer, producing a comparison signal between the strong prediction from the last loop and the weak prediction from an earlier loop. This contrastive signal directs the decoder toward more reliable token choices without any training.
LoopCD-Hidden: Zero-Overhead Guidance in Hidden‑State Space
LoopCD-Hidden operates entirely in hidden‑state space, adding zero output overhead. Instead of comparing logits, it contrasts hidden representations from the final loop against those from an earlier recurrent pass. This approach preserves the full decoding pipeline while still supplying the benefit of a weak-strong prediction pair, making it suitable for deployments where even a single extra forward pass is undesirable.
Substantial Gains at Full Recurrent Depth
Across four looped Transformer families, LoopCD delivers consistent improvements at full depth. LoopCD-Logits raises Ouro‑2.6B‑Thinking's AIME 2024 pass@1 from 61.88% to 73.33%, while LoopCD-Hidden lifts Huginn's HumanEval pass@1 from 22.56% to 31.71%. These results demonstrate that the contrastive signal from earlier loops reliably improves token selection even when the full number of recurrent steps is executed.
Loop Halving: Matching Baselines with Half the Compute
Perhaps the most striking effect of LoopCD is that the performance gains it enables allow the number of recurrent loops to be halved while still matching or exceeding full‑depth unguided baselines. This reduction cuts forward FLOPs by 22.5% to 48.2% across the evaluated families, showing that intermediate recurrent states carry useful decoding information even after only a few loops.
From Intermediate States to Effective Decoding Signals
By transforming intermediate recurrent states into effective guidance signals, LoopCD achieves superior decoding quality while substantially reducing inference compute. The framework does not require auxiliary models, additional training data, or modified architecture—it works with any looped Transformer by simply contrasting the final prediction with an earlier recurrent pass.
LoopCD demonstrates that recurrence itself provides the weak-and-strong prediction pairing needed for contrastive decoding, eliminating the need for separate reward models or value networks. The approach is training‑free and can be applied at inference time with minimal disruption to existing pipelines.