Looped Transformers execute a shared block across recurrent loops to achieve parameter efficiency. Each loop produces an intermediate representation from which the next token can be decoded, yet standard decoding discards earlier loop states. Because earlier loops perform less computation, recurrence naturally provides aligned weak-and-strong prediction pairs without auxiliary models or external training.
How Looped Transformers Achieve Parameter Efficiency
Looped Transformers reduce parameter count by repeatedly executing a shared computational block across recurrent loops. At each loop, the transformer processes the current hidden state and produces an output suitable for the next token prediction. This recurrence means the same parameters are reused many times, yielding a compact model that nevertheless processes long sequences through iterative refinement.
The Recurrence-Driven Weak-and-Strong Prediction Pair
Earlier loops embody less computation, so their predictions are weaker. The final loop produces a strong prediction after full refinement. This alignment arises naturally from the architecture: the same computation path is traversed increasingly deeply, creating an inherent weak-strong pair that contrastive decoding can exploit without adding separate networks or requiring additional training.
LoopCD-Logits: Contrasting Final and Earlier Logits
LoopCD-Logits guides token selection by contrasting the final prediction with an earlier recurrent pass in logit space. The framework requires one extra output pass through the looped transformer, producing a comparison signal between the strong prediction from the last loop and the weak prediction from an earlier loop. This contrastive signal directs the decoder toward more reliable token choices without any training.
LoopCD-Hidden: Zero-Overhead Hidden-State Guidance
LoopCD-Hidden operates entirely in hidden‑state space, adding zero output overhead. Instead of comparing logits, it contrasts hidden representations from the final loop against those from an earlier recurrent pass. This approach preserves the full decoding pipeline while still supplying the benefit of a weak-strong prediction pair, making it suitable for deployments where even a single extra forward pass is undesirable.
Substantial Gains at Full Recurrent Depth
Across four looped Transformer families, LoopCD delivers consistent improvements at full depth. LoopCD-Logits raises Ouro‑2.6B‑Thinking's AIME 2024 pass@1 from 61.88% to 73.33%, while LoopCD-Hidden lifts Huginn's HumanEval pass@1 from 22.56% to 31.71%. These results demonstrate that the contrastive signal from earlier loops reliably improves token selection even when the full number of recurrent steps is executed.
Loop Halving: Matching Baselines with Half the Compute
Perhaps the most striking effect of LoopCD is that the performance gains it enables allow the number of recurrent loops to be halved while still matching or exceeding full‑depth unguided baselines. This reduction cuts forward FLOPs by 22.5% to 48.2% across the evaluated families, showing that intermediate recurrent states carry useful decoding information even after only a few loops.
From Intermediate States to Effective Decoding Signals
By transforming intermediate recurrent states into effective guidance signals, LoopCD achieves superior decoding quality while substantially reducing inference compute. The framework does not require auxiliary models, additional training data, or modified architecture—it works with any looped Transformer by simply contrasting the final prediction with an earlier recurrent pass.
LoopCD demonstrates that recurrence itself provides the weak-and-strong prediction pairing needed for contrastive decoding, eliminating the need for separate reward models or value networks. The approach is training‑free and can be applied at inference time with minimal disruption to existing pipelines.
Read the paper on arXiv