Why Video Diffusion Hits a Wall at Long Sequences

Video diffusion transformers process spatiotemporal token sequences that grow with both spatial resolution and temporal duration. A 14.3-second, 768p video produces thousands of latent frames, each containing hundreds of spatial tokens. Every denoising step computes full pairwise attention across this entire sequence. In MiniMax H3, one of the highest-quality open video models, Softmax attention accounts for more than 85% of denoiser runtime. The quadratic cost makes long-form video generation impractical for real-time applications like livestreaming, where latency must stay under a second.

Linear attention offers a solution. Instead of computing all pairwise interactions, it compresses context into a fixed-size recurrent state, reducing cost from quadratic to linear in sequence length. Language models have adopted this successfully. But directly applying linear attention to video diffusion fails. The compressed state cannot preserve the fine-grained spatial interactions that determine texture quality, object boundaries, and motion coherence. A video frame's patches are not independent tokens the way words are; they share spatial structure that simple sequential processing destroys.

Haocheng Xi, Yiming Xie, and colleagues from UC Berkeley and Impossible Inc. present Video DeltaNet (VDN), a hybrid architecture that keeps exact Softmax attention for nearby frames and short-range interactions, while using a new form of linear attention called Video Delta Attention (VDA) for distant context. The result is a 14.5x speedup over the 50-step Dense H3 baseline when generating a 14.3-second 768p video, with quality that matches or exceeds the original across standard benchmarks.

Splitting Attention by Temporal Role

VDN divides video-to-video attention by temporal role. Nearby frames contain local correspondences that determine texture, object boundaries, and short-term motion. These interactions require direct token-to-token matching, so VDN retains exact Softmax attention within a bidirectional temporal window. H3's VAE decodes five consecutive latent frames as one temporal chunk, so each query chunk attends to itself and its immediately preceding and following chunks. This produces a 15-frame window except at sequence boundaries.

VDN also adds two boundary anchors with four-way connectivity. Every video frame attends to all tokens in the first and last latent frames, and the first and last frames attend to the complete sequence. These anchors provide explicit global references from opposite ends of the clip, which is particularly natural for full-clip video diffusion and for first-and-last-frame conditioning tasks where provided visual conditions constrain the entire generated sequence.

The remaining distant context goes to bidirectional linear attention. For a query frame t, the forward state summarizes all frames before its Softmax window, and the reverse state summarizes all frames after the window. Boundary anchors are excluded from both states because they are already available through Softmax. The two temporal regions are disjoint, so their query readouts can be added without double-counting. The linear memory is also text-aware: text tokens are summarized into a state that initializes both directional scans, providing global text conditioning through the linear branch while text remains directly visible to the Softmax branch.

Video Delta Attention: Frame-Wise Delta Rule

The core innovation is Video Delta Attention, which extends the delta rule from individual tokens to entire video frames. Standard delta-rule linear attention updates its recurrent state one token at a time, processing keys and values sequentially. This makes sense for autoregressive language decoding where one new token arrives per step. Video diffusion is different: all spatial tokens in a latent frame are available simultaneously. Imposing an arbitrary patch order is unnecessary and harmful, because correlated writes from overlapping patches can interfere.

VDA defines the frame-level state as the solution to a joint optimization problem. The new memory should stay close to the decayed inherited state, but also fit all key-value associations in the frame simultaneously. The solution has a closed form: a matrix inverse in key-channel space that couples the writes so that overlapping key directions are resolved inside the update rather than interfering. Each frame produces a compact d_k-by-d_k statistics matrix and a value-key write matrix, and the state update is computed from these in a single step.

The authors prove that VDA has a non-expansive inherited-state transition: the contribution carried from earlier frames cannot be amplified by a frame update. This is important because additive frame-wise writes can amplify inherited state without proper normalization. VDA obtains this stability directly from the matrix inverse structure, without the key scaling that other approaches like SANA-WM require.

The bidirectional scans compose these per-frame transitions into chunk-wise transitions. Since attention reads memory only at VAE-chunk boundaries, each chunk's frame transitions are composed into a single affine transition, and the scan processes the shorter chunk sequence rather than individual frames. This reduces scan depth and kernel launch overhead by roughly the chunk size.

Combining Two Branches with Learned Gates

The two branches have fundamentally different output scales. Softmax, restricted to local windows and boundary anchors, concentrates its probability mass over fewer keys, producing different magnitude outputs than the linear branch. VDN addresses this with branch-specific calibration.

The Softmax readout gets a content-dependent sigmoid gate. The linear readout is RMS-normalized and passed through its own sigmoid output gate. Each branch has its own output projection matrix, allowing them to contribute in different residual-stream directions. The gated branch outputs are projected independently and then added. The combined output feeds into the pretrained residual stream, while the original feed-forward sublayer is left unchanged.

Separate output projections are critical. Without them, the two branches compete for the same representational capacity in the residual stream. With them, each branch can specialize: Softmax handles fine-grained local interactions, while the linear branch provides global context and long-range motion information.

Staged Adaptation: Adding Linear Attention to a Pretrained Model

The adaptation recipe has three stages. Stage A1 trains each linear branch independently for 200 steps, aligning it with frozen pretrained activations. The backbone is frozen, the Softmax gate is fixed at 0.99, and gradients are clipped per-layer. This local alignment avoids sending initialization noise through a deep stack of simultaneously changing blocks.

Stage A2 installs the calibrated branches together and optimizes end-to-end for 500 steps. This corrects composition errors that are invisible when blocks are trained in isolation. The pretrained weights and Softmax gates remain frozen.

Stage B adds LoRA adapters to the Q, K, V, and output projections, training them jointly with the linear pathway and Softmax gates for 2,000 steps. This co-adapts the hybrid layers while preserving the pretrained backbone. The staged approach matters because adding a randomly initialized linear pathway to a Softmax-pretrained model changes both information flow and residual-stream activation statistics. Without careful adaptation, it disrupts learned capabilities before the new branch becomes useful.

After architecture adaptation, the 50-step model is distilled to eight steps using a DMD2-style objective without the GAN term. The student is initialized from the community MiniMax-H3-Turbo-LoRA and trained against VDN-H3's own 50-step sampler, isolating step reduction from architecture conversion.

Results: Quality Preserved, Speed Dramatically Improved

The evaluation uses 103 prompts from a fixed third-party set, rendered at 1344x768 resolution with 345 frames at 24 fps (14.375 seconds). The baselines are Dense H3 with 50 neural function evaluations and FastH3 with four NFEs.

Across five no-reference quality metrics (EvalCrafter VQAA and VQAT, Q-Align, FAST-VQA, DOVER++), eight-step VDN-H3 matches or exceeds 50-step Dense H3, with differences ranging from +0.06 to +1.00. FastH3 is 2.70 to 12.74 points lower. VDN-H3 preserves similar RAFT motion magnitude (11.71 vs. 11.55 pixels), while FastH3 falls to 9.19. On FIRM-Video, VDN-H3 matches Dense H3 in Instruction Following (2.25), is slightly higher in Perceptual Quality (4.45 vs. 4.40), and remains comparable in World Coherence (1.77 vs. 1.84).

For first-and-last-frame conditioning, VDN-H3 shows small gaps: 0.18 dB in PSNR (28.67 vs. 28.85), 0.007 in SSIM (0.826 vs. 0.833), and 0.011 in LPIPS (0.1156 vs. 0.1044). FastH3 shows much larger degradation.

On efficiency, the optimized VDN-H3 backbone reduces latency from 16.0 to 6.2 seconds on one B200 (2.6x) and from 35.35 to 11.16 seconds on one H200 (3.2x). With eight-step distillation and eight-GPU inference, DiT denoising takes 6.70 seconds on B200s and 12.5 seconds on H200s. This corresponds to a 14.5x speedup over the 50-step Dense H3 baseline on the same GPU count.

The speedup scales with video length. As latent frames grow from 42 to 102, the Softmax attention density falls from 42.1% to 20.0%, and the whole-backbone speedup increases from 1.8x to 3.2x on H200 and from 1.7x to 2.6x on B200. This is because VDN keeps only a fixed-width local window and boundary anchors in Softmax, while VDA handles the growing distant context with linear scaling.

Kernel Optimizations: Where the Real Speedup Comes From

The paper profiles VDN's core operators at 102 latent frames and isolates the contribution of each optimization. Fused VDA kernels are the biggest win. VDA-Prep (temporal convolution, SiLU, L2 normalization, layout conversion) drops from 18.0 to 1.6 ms on H200 and from 17.1 to 3.4 ms on B200. VDA-Gather (collecting directional states at window boundaries with decay bridge) provides 7.2x and 7.5x speedups. VDA-Epilogue (RMS normalization, output gating, layout conversion) provides 7.3x and 9.1x.

The small-matrix inverse is computed with a custom CUDA kernel that performs blocked Gauss-Jordan elimination in registers and directly emits the transition and injection terms, replacing a multi-kernel Cholesky path. This reduces latency from 7.8 to 1.7 ms on H200 and from 6.3 to 1.3 ms on B200.

Other optimizations include varlen FlashAttention for window Softmax, head-sharded VDA executed on a side stream alongside window Softmax, MXFP8 for wide GEMMs while keeping recurrent states in FP32, and precomputed AdaLN modulation parameters.

Limitations and Open Questions

VDN's quality ceiling depends on the Softmax window size. A 15-frame window captures short-range interactions but misses longer-range texture and motion dependencies that would require larger windows. Increasing the window would improve quality but reduce the speedup, and the tradeoff has not been systematically explored.

The staged adaptation recipe requires careful hyperparameter tuning at each stage. The paper provides details for H3 but does not test generalization to other video diffusion architectures. Whether the same three-stage approach works for DiT variants with different normalization or positional encoding schemes is unclear.

The evaluation uses a fixed third-party prompt set, which avoids cherry-picking but may not cover all use cases. The quality metrics are automated and do not capture subjective preferences that matter for livestreaming applications. A human evaluation or a more comprehensive benchmark would strengthen the claims.

The linear branch processes text through the same mechanism as video, summarizing all text tokens into a state. For long prompts with many tokens, this summary may lose important conditioning information. The paper does not ablate the effect of prompt length on generation quality.

What It Means in Practice

For teams building video generation systems, VDN provides a concrete path to reducing inference cost without retraining from scratch. The hybrid architecture is applicable to any video diffusion transformer that uses Softmax attention, and the staged adaptation recipe requires only that the pretrained model can be evaluated on the target task. The linear branch adds minimal parameters and can be trained in a few thousand steps.

The 6.7-second latency for a 14.3-second 768p video on eight B200 GPUs brings video generation into the range where real-time or near-real-time applications become feasible. Livestreaming, interactive video editing, and other latency-sensitive use cases that previously required expensive dense attention can now run with linear-scaling context.

The kernel-level optimizations are reusable. The fused VDA kernels, the chunk-wise scan, and the small-matrix inverse CUDA kernel are all general-purpose components that could accelerate other linear-attention implementations. The SGLang serving stack integration means the optimized model can be deployed without custom infrastructure.

The open-source release, including weights on HuggingFace and code on GitHub, makes this immediately reproducible. The blog post at openvdn.github.io provides additional context. For practitioners, the most important number is not the 14.5x speedup headline but the fact that quality is preserved across standard benchmarks. Speed without quality is useless for video generation, and VDN demonstrates that the two are not mutually exclusive when the hybrid architecture is designed around the structure of video data.

Read the paper on arXiv