When a Video Model Generates the Wrong Motion, the Right Motion Is Still in There
A video model trained on spring-mass oscillations learns that red masses move slowly and blue masses move quickly. When shown a red mass with fast observed motion at test time, it generates slow motion. The obvious question is whether the model learned the wrong rule (red means slow, always) or learned the correct rule (follow observed motion) but failed to use it. Wang, Zheng, Yuan, Yang, and Liu from Tsinghua and Peking show it is the latter: the correct motion remains available inside the model's representations and can be made to control the generated video through a small internal edit. They call this ability causal writability. It implies that the model knows more than it uses, and that the gap between what a model knows and what it produces can be bridged by understanding where and when internal signals lose their ability to reach the output.
Predictive Underspecification in Oscillatory Video
The setup is a controlled spring-mass task. During training, red masses oscillate slowly (2.2 to 3.0 rad/s) and blue masses oscillate quickly (5.2 to 6.4 rad/s). No conflicting color-frequency combinations appear during training. At test time, the authors create unseen combinations by holding observed motion fixed and changing color. A red mass with fast observed motion, or a blue mass with slow observed motion. These are strict conflict cases: the color cue says one thing, the observed motion says another.
The model is Spring, a 488M-parameter latent flow-matching Transformer with 30 bidirectional DiT blocks and a frozen Wan2.1 VAE. Each video is 128x128 at 20fps, with 65 observed frames followed by 64 generated frames. Two history regimes are tested: Long (all 65 observed frames visible) and Short (only the final 8 motion frames, with the first 57 replaced by a fixed background). The model can follow color and generate slow motion, a predictive shortcut. Or it can follow observed motion and generate fast motion, the physically correct rule.
The competition between these cues is sharp. Appearance cues mainly select between the two training-supported frequency bands rather than continuously tuning speed. Around ambiguous hues (neither clearly red nor clearly blue), the generated future is more likely to follow observed motion than at the endpoints. With strong color cues, the model favors the frequency paired with the observed color. Under ambiguous appearance, fast motion favors a blue future and slow motion a red one. The model has learned both directions of the motion-appearance association and can complete it in either direction.
More visible motion shifts control toward the physical solution. In separately trained Short and Long models, motion-consistent futures persist farther toward the conflicting red or blue endpoint under Long in both conflict directions. The generated future depends on the relative evidence supplied by appearance and observed motion. This is the underspecification: training data supports both rules, and only conflict cases reveal which one controls generation.
A Compact Physical Edit Restores the Correct Future
The central experiment tests whether the correct motion remains causally accessible when the conflicting color cue controls generation. For each strict aligned-conflict pair (same observed motion, same boundary state, same generation seed, different color), the authors compute the paired difference d = hA - hC between aligned and conflict observed-frame token states after a given DiT block. Adding this difference to the conflict run's activations redirects its decoded future.
The full edit works, but it is high-dimensional. PCA on the paired differences reveals that four coordinates retain nearly the full effect. Across three independently trained Short models after 50K training steps, 87.8% of held-out top-4 writes satisfy a recovery threshold (0.75 < Romega < 1.25), compared with 92.4% for the full paired-difference edit. The edits are not random: boundary position and velocity organize the write. Held-out edits form smooth phase-ordered loops in both rewrite directions (slow-to-fast and fast-to-slow). For harmonic motion, the boundary phase theta* = atan2(-v*/omega_true, x*) compactly encodes normalized position and velocity.
A simple controller fitted on separate trajectories predicts edits from boundary state and target direction. The model is a first-harmonic map: z-hat = beta_0 + beta_c cos(theta*) + beta_s sin(theta*). This raises held-out coordinate R-squared to 0.83-0.88, up from 0.51-0.60 for a direction-only model. For a held-out input, the frozen controller predicts four coordinates from target direction and boundary state, then reconstructs the activation edit through the frozen PCA basis. Across six run-direction groups, 85.9% of these synthesized held-out writes satisfy the recovery threshold, with held-out decoded recovery R = 0.94-0.99.
The controller is donor-free. It does not need a matched aligned activation as input. It predicts the edit entirely from the boundary state and the desired target direction. This means a new trajectory can receive a corrective edit without ever seeing its aligned counterpart. The edits also rewrite associated appearance: slow-to-fast shifts future color toward blue. The same low-order organization recurs in Pendulum (another oscillatory system) and Free Fall (non-oscillatory dynamics), where a donor-free controller reaches median held-out recovery R-g = 0.967 across 64 decoded rollouts.
The Sharp Depth Boundary: When Does the Future Become Hard to Change?
At fixed edit strength, there is a sharp depth boundary over a narrow range of layers: the same edit changes the video before the boundary but not after it. The authors define writability W-omega(l) as the fraction of strict failures that can still be redirected at site l, and total writability D-omega as the sum across all 31 network sites (one before block 0 and one after each of the 30 blocks). A larger D-omega means the same write remains effective deeper into the network.
Individual trajectories close at different sites, so averaging broadens the population profile. But the boundary is sharp for each trajectory. The edit either changes the decoded video or it does not, and the transition happens over a narrow depth range. This closure marks commitment for that write: the point at which the observed-frame information has been written into future-frame states and can no longer be redirected by editing the observed-frame activations.
The alternative-motion signal survives closure. After an early write that redirects the decoded future, the authors project the difference between edited and natural-conflict future-frame activations onto the aligned-minus-conflict direction. This projection remains measurable from the first non-writable block through the final block, even after observed-frame writes stop changing the generated frequency. The signal is present but has lost its ability to reach the output.
What Stops the Signal: Attention as the Causal Bottleneck
The authors localize the pathway through which the physical direction causally affects generation. They replace either the post-attention future-frame state or the subsequent MLP output with its natural-conflict counterpart. Restoring the conflict post-attention future-frame state removes frequency recovery. Restoring the following MLP output preserves frequency recovery. This localizes the write to self-attention.
Within self-attention, restoring Q (future-frame queries) has little effect. Jointly restoring observed-frame K+V removes recovery in all six run-direction groups. Future-frame queries obtain the motion-controlling information from observed-frame K/V. This is the mechanism: self-attention carries motion information from observed frames into future-frame states, and strengthening the corresponding write restores the correct motion in decoded video.
Amplification confirms this. Head-level responsibility differs across runs: the effect is concentrated in one K head, distributed across K/V, or localized to a selective V head. In one run, amplifying only the observed-frame V of one of nine heads (V_C + lambda * delta-V where delta-V = V_M - V_C) restores physics-following. At lambda=8, the rate peaks at 56.3% across 48 held-out strict failures. At lambda=16, all outputs remain valid but most are off-family. Excessive gain overshoots. The signal is present but its unamplified attention write is too weak to control the video.
Early Writability Predicts Which Errors Training Later Corrects
Across 15 Short runs, physics-following behavior rises from 0.52 at 5K to 0.60 at 100K steps, while mean writability among remaining shortcut errors falls from 10.88 to 9.55 sites. The key finding: at an early checkpoint (5K), errors that training later corrects are writable at more network depths than errors that persist. Future-rescued trajectories (those that follow observed motion at both 80K and 100K) are writable at 3.79 more sites on average (95% CI [1.57, 6.53]) than persistent failures at 5K. Early writability separates the two outcomes before their generated behavior diverges.
This connects writability to learning dynamics. A classifier can improve held-out accuracy while becoming more confidently wrong on some remaining examples, increasing their cross-entropy loss. The writability measure provides a complementary signal: errors that are writable at many depths are those the model has the internal representation to fix but has not yet organized its downstream computation to use. Errors that are writable at few depths are those where the signal has already been committed to a different future.
Two Ways to Delay Commitment
More observed motion delays commitment. In all three matched-seed comparisons at 50K, Long (65 observed frames) remains writable to a later block than Short (8 observed frames). The physical alternative stays accessible deeper into the network when more evidence supports it.
Training order moves the commitment boundary. The authors adapt a pretrained 1.3B Wan video DiT to the same spring-mass task via two paths. Direct adaptation learns the biased red-slow/blue-fast task immediately. Neutral-first adaptation first learns motion from achromatic videos, where appearance carries no frequency information, and then receives the biased data. Conflict physics-follow at 10K is 80.5% after Neutral-first adaptation and 9.4% after Direct adaptation, on the same 128 held-out histories. Both adaptations close sharply, but Neutral-first closes between blocks 14-15 while Direct closes between blocks 12-13. Learning cue-independent motion first preserves access to the physical future deeper into the network.
The pretrained Wan 1.3B results also show that the compact edit remains effective. A controller fitted on separate trajectories approaches the Top-4 reference in both write directions. This supports generality across model scale and training regime: causal writability and its sharp closure are not confined to models trained from scratch on controlled data.
Cross-Model Transfer: Different Behavior, Shared Physical Solution
Independently trained models differ in how often their rollouts follow observed motion, yet corrective edits transfer between them. The three Short 50K runs have physics-follow rates spanning about 22 percentage points. A scale and four-dimensional orthogonal map fitted on separate trajectories transfers controller-predicted edits across all six directed run pairs, with near-full held-out decoded recovery. The behavioral differences may arise from how downstream computation uses a shared physical solution. This shifts the question from whether the physical solution exists to when it loses its ability to affect generation.
What This Means for Understanding and Controlling Video Models
The distinction between predictive underspecification and causal writability is control-theoretic. Underspecification asks which solutions fit the training data. Writability asks which can still reach the output as inference proceeds. This parallels the distinction between observability and controllability: a physical solution may remain detectable after it loses control over generation. Commitment marks this loss of causal access, not the loss of physical information.
For practitioners, the findings suggest several directions. The sharp depth boundary implies that there is a window during generation where corrective edits are effective, and a point of no return beyond which they are not. The boundary depends on evidence strength (more observed motion delays it) and learning history (learning motion before introducing appearance shortcuts delays it). This has implications for training curricula and for inference-time interventions.
The attention mechanism as the causal bottleneck is also actionable. Self-attention K/V mediates the motion write from observed to future frames. Understanding which heads carry which information could enable more targeted interventions. The amplification experiment shows that strengthening the right attention write can restore control, but excessive gain overshoots. This suggests a control-theoretic framing where the gain is a tuning parameter, not a binary switch.
The donor-free controller, which predicts edits from boundary state and target direction without needing a matched aligned activation, is particularly interesting for practical applications. It means a corrective edit can be computed from observable quantities (position, velocity, desired direction) without requiring a counterfactual reference. The first-harmonic model that predicts the edit is simple enough to be analyzable and compact enough to be generalizable across training seeds.
The limitations are real. The analysis uses controlled synthetic dynamics where exact counterfactual matching is possible. The learned edits follow the coupled appearance-dynamics modes in training, and closure is specific to the observed-frame edit, sites, and checkpoints tested. Independent runs distribute the attention write across different heads. Whether naturally occurring shortcuts in broadly pretrained world models exhibit the same point of no return remains open. The pretrained Wan 1.3B results are encouraging but do not fully address this question.
The paper opens a direction for diagnosing model behavior beyond what the generated output shows. When a video model produces incorrect physics, the failure may not be in what it learned but in how it uses what it learned. Causal writability provides a framework for distinguishing these two failures and for intervening when the correct solution is present but unused.
Read the paper on arXiv