Every recurrence of a looped language model adds cost in training, decoding, prefill, and reinforcement learning RL. The closer recurrent states get to fixed points, the less the path to them matters. This enables truncated backpropagation in training; terminal key-value KV sharing for decoding with almost no loss in accuracy; a distilled student that prefills up to 1.79x faster; and RL updates that compute gradients from saved rollout states 2x faster than backpropagating through the replayed trajectory. We therefore improve the two components of training that shape these fixed points: the depth prior and input injection. Fixed-depth training breaks KV sharing, and Huginns broad depth prior supports sharing but dilutes supervision at the target depth more than sharing requires; we learn the prior from prediction feedback, with an entropy term that keeps it broad. Existing injection schemes let the states component along the input amplify or cancel the injection; we remove this component with orthogonal injection. From 100M to 1.6B parameters, the learned prior and orthogonal injection lower perplexity at every scale relative to Huginns prior and existing injection schemes, respectively. At 1.6B, the learned prior with a 3x smaller KV cache matches the downstream average of fixed-depth training with the full cache.
Shaping fixed points in looped language models
Looped language models repeatedly apply a shared block of layers, but whether they should learn fixed points remains contested. One camp forgoes fixed points and keeps the loop count small and fixed. The other camp shapes fixed points and loops much deeper. In typical implementations, every loop adds cost: activations to store during training, keys and values KV to cache during decoding, compute to prefill each prompt, and a replay to score each rollout during reinforcement learning RL. In this work, we show that fixed points hold the potential to cut each of these costs, which makes deep looped models cheaper at every stage: pretraining, serving, and post-training.
The fixed-point equation z⋆=Fθ(z⋆,x) involves only the endpoint, not the path to it, so once a models states are near a fixed point, the endpoint can stand in for the full trajectory. Reusing terminal KV caches across loops tests this claim: it barely changes the zero-shot GSM8K accuracy of Huginn but collapses that of Ouro at four loops. The split tracks convergence: Huginns states approach fixed points; Ouros do not.
Similarly, the same fixed point gives each remaining cost a shortcut. In pretraining, backpropagating through only the last few loops approximates the equilibrium gradient without the warm-up that implicit gradients require. In decoding, terminal KV sharing keeps only each prefix tokens terminal KV, with nearly unchanged downstream accuracy. In prefilling, a student distilled to predict the fixed point directly replaces the loops. In RL, scoring rollouts from saved fixed-point states avoids replaying the loops. Each shortcut removes a depth-dependent cost, so its savings grow with every added loop. Looped models already take the first two steps; fixed points explain why they work and yield the other two.
Improving the depth prior from prediction feedback
The benefits depend on how training shapes the fixed points. Two common techniques do this shaping: depth sampling sets where supervision lands, and input injection sets what a fixed point retains about the input. We improve both.
Because tokens converge at different depths, the sampled depth decides whether training sees the converged prefix context that terminal KV sharing supplies at inference. Huginn samples the training depth from a shifted Poisson–log-normal PLN distribution: ξ∼N(μ,σξ²), K∣ξ∼Poisson(exp(ξ)), R=K+1, where σξ=1/2 and μ=log(R¯−1)−σξ²/2, so that 𝔼[R] equals the target mean R¯>1. Under a fixed training budget, the choice of depth prior trades prediction quality at the target depth against fixed-point shaping.
Fixed-depth training concentrates supervision at one depth and predicts better there. PLN sampling spreads supervision across depths and yields the fixed-point behavior that supports KV sharing. We therefore learn the depth prior: it spreads supervision across depths but adapts its allocation to training feedback, balancing prediction quality against fixed-point shaping at equal mean depth.
We replace the PLN prior with a categorical prior pϕ=softmax(ϕ) over depths R={1,…,Rmax}, with learnable logits ϕ. At each optimizer step, every microbatch draws a depth r∼pϕ and receives supervision at that depth. We learn pϕ from this feedback by minimizing Jprior(ϕ)=−𝔼^n[stopgrad(An)log pϕ(r)]−λH H(pϕ)+λm(𝔼pϕ[R]−R¯)², where the advantage An scores the sampled depth. We map the mean next-token CE ℓ to a reward u=ω(stopgrad(ℓ)) with a decreasing function ω; we use inverse perplexity, ω(ℓ)=exp(−ℓ). The advantage is then An=sn−1(u−un−1): an exponential moving average (EMA) baseline un−1 centers the reward, and sn−1=σℓ,n−1/σu,n−1 rescales it to the scale of CE, following PopArt. A positive An thus raises the probability of the sampled depth.
Prediction feedback under a budget constraint favors depths that already predict well. Left alone, it concentrates supervision on a narrow range of depths and can weaken fixed-point shaping. The entropy term H(pϕ)=−∑r∈Rpϕ(r)log pϕ(r) counters this tendency. Increasing λH spreads the depth allocation. Since depths start at 1, the wider allocation extends mainly above R¯, toward Rmax. This trades finite-budget predictive performance for supervision at later depths, where settled prefix context supports KV sharing. The coefficient λH thus gives a direct knob for balancing performance and memory efficiency.
The third term penalizes deviations of the expected depth from the target R¯, keeping the allocation near the prescribed forward recurrence budget. This budget control keeps the allocation near the prescribed forward recurrence budget.
Orthogonal input injection removes amplification and cancellation
Input injection feeds the same embedding qt into every recurrence, keeping the fixed points input conditioned. A common additive form is ζtr=Mhtr+qt, Hr+1=Γθ(ζ1r,…,ζNr), where htr is the recurrent state at token tt and recurrence rr, M controls state carryover, and Γθ is the recurrent transformer core. Parcae uses a diagonal carryover M=diag(exp(−Δ⊙α)) with learned positive vectors Δ and α, so Λ has entries in (0,1) and ‖Λ‖2<1. It injects the prelude output et=Pθ(x)t as qt=Δ⊙We t. In Parcae, the carryover Λhtr can have a component along qt that amplifies or cancels the input, so the input a fixed point retains depends on the state; this can hurt performance. Our orthogonal input injection OrthoInj removes this component:
ζtr=Qqt Λ htr+qt, Qqt=I−qt qt⊤‖qt‖²+εIn this formula, Qt=I−qt qt⊤‖qt‖²+ε removes the component of Λhtr along qt, so every recurrence receives the input at the same strength regardless of the states component along the input direction.
From 100M to 1.6B parameters, OrthoInj trains to lower loss, lowers validation perplexity by 0.3–1.4%, and raises the downstream average. The learned prior with a 3x smaller KV cache at 1.6B matches the downstream average of fixed-depth training with the full cache.
What fixed points enable
Fixed-point shaping trains a looped model with supervision near the fixed points of its recurrence. At a fixed point, the endpoint can stand in for the trajectory. Conventional looped language models pay for every recurrence in both training and inference; fixed-point shaping decouples both costs from depth.
In training, truncated backpropagation differentiates through only the last b recurrences, so its memory and compute scale with b rather than R. Its gradient D~b evaluates the Jacobians Jr=∂zFθ(zr,x) and Br=∂θFθ(zr,x) at visited states zr rather than at z⋆. This frees TBPTT from needing a fixed point: from any state, D~b is the exact gradient of the loss through the last b recurrences with the earlier state held fixed, so every update descends a well-defined objective. As the visited states converge to z⋆, D~b approaches Db regardless of R. TBPTT is therefore as stable as full BPTT early in training and recovers the fixed-point gradient once fixed points form, using a single estimator with no warm-up phase or switch.
In decoding, terminal KV sharing removes the KV cache growth: each prefix token keeps only its terminal KV, and every recurrence of later tokens attends to it. This requires a model that predicts well from terminal context. Sampled training depths push the model toward this regime: deep draws let supervised recurrences read nearly settled prefix banks, and supervising many depths penalizes states that drift as recurrence continues.
In prefilling, a student distilled to predict the fixed point directly replaces the loops. The student is a lightweight network Sψ that replaces the iterated recurrent core during prefill; the prelude and coda are reused from the teacher. From the prelude output e=Pθ(x), the student predicts ẑ=Sψ(e)≈zθR(x); prefill then applies one teacher recurrence and the coda to ẑ and retains each physical attention layers KV bank. Because fixed-point shaping makes the endpoint an approximate fixed point, one recurrence from a nearby ẑ yields banks close to the teachers terminal banks.
In RL, scoring rollouts from saved fixed-point states avoids replaying the loops. Full BPTT computes D~R of equation (1); when every visited state is close to z⋆, its Jacobians are close to J⋆ and B⋆, so D~R approaches D⋆, whose truncated Neumann series can be evaluated at the last state zR. The update cost and memory of reuse are thus independent of R, though rollout generation still takes most of the training time.
Experimental results across model scales
We evaluate from 100M to 1.6B parameters. The learned prior and orthogonal injection lower perplexity at every scale relative to Huginns prior and existing injection schemes, respectively. At 1.6B, the learned prior with a 3x smaller KV cache matches the downstream average of fixed-depth training with the full cache. The learned prior keeps prediction quality under terminal KV sharing: from 100M to 1.6B, it lowers perplexity and raises the downstream average over Huginns fixed prior, and at 1.6B, with a 3x smaller KV cache, it even matches the full-cache downstream average of fixed-depth training.
OrthoInj improves on the strongest injection baseline, Parcae. From 100M to 1.6B, it trains to a lower loss, lowers validation perplexity by 0.3–1.4%, and raises the downstream average. Fixed-depth training breaks KV sharing, and Huginns broad depth prior supports sharing but dilutes supervision at the target depth more than sharing requires; we learn the prior from prediction feedback, with an entropy term that keeps it broad.
RL updates from saved rollout states
RL usually runs the recurrent forward pass twice: once without gradients to sample rollouts, and again with gradients to score them. Near a fixed point, the second pass need not replay the trajectory. Full BPTT computes D~R of equation (1); when every visited state is close to z⋆, its Jacobians are close to J⋆ and B⋆, so D~R approaches D⋆, whose truncated Neumann series can be evaluated at the last state zR: D~R=∑k=0R−1(∏j=1k JR−j) BR−1−k≈∑k=0b−1 JRk BR. The update cost and memory of reuse are thus independent of R, though rollout generation still takes most of the training time. Reusing rollout states makes the update 2x faster than full BPTT, and stays within 1.6 points of that baselines GSM8K pass@1.
Distilled prefill faster than recurrence
Because decoding reads only terminal KV banks, prefill can be distilled into a student that predicts only the recurrent endpoint; without terminal sharing, the student would have to reproduce the KV banks of every recurrence. The student is a lightweight network Sψ that replaces the iterated recurrent core during prefill; the prelude and coda are reused from the teacher. We train the student against depth-R targets computed online from the frozen teacher, using normalized hidden-state regression plus a next-token KL term. With a suitable loss, the student may also be able to serve as a standalone language model or a drafter for speculative decoding. Distilled prefill trades downstream accuracy for 1.5–1.8× faster prefill. At matched latency, it scores 0.5–0.9 points above the teacher prefilling with two recurrences.
Related work and conclusion
Looped model efficiency has been explored in prior work, and shaping fixed points explains why existing shortcuts work and yields new ones. Limitations include that the LLM-based invasion extraction relies on prompt design and may miss subtle or equivocal mentions in reports, and the framework depends on organ segmentation quality from a pre-trained model, which may vary across scanners or patient populations. The dataset, while large for this task, still represents a subset of real-world clinical diversity, and performance on completely unseen populations may differ. The graph structure encodes anatomical prior knowledge that may not generalize to cancer types with different spread patterns.
In practice, the framework can be integrated into radiology workflows as a post-processing step. Given a CT volume and its paired report, the system generates an organ-level graph, enriches it with textual evidence, and outputs a T-stage probability. Developers can replace the organ segmentation model, the language model, or the graph neural network architecture without redesigning the entire pipeline, making the approach adaptable to other anatomical regions or cancer types. The code and pretrained models are released to facilitate reproducibility and further research.
This work demonstrates that explicit anatomy-structured modeling and fine-grained cross-modal alignment improve T-staging accuracy when CT and radiology reports are combined. By moving away from global feature fusion and toward organ-level graph reasoning, the framework provides a practical pathway toward non-invasive, computationally aided cancer staging.