Video diffusion models have achieved remarkable fidelity in text-to-video synthesis, yet scaling these models to capture complex spatiotemporal dynamics remains computationally expensive. Diffusion Transformers (DiTs) have become the backbone of state-of-the-art systems, but parameter counts grow quickly as motion diversity and video length increase. Mixture-of-Experts (MoE) offers a compelling path forward: by activating only a subset of parameters per token, MoE can increase total model capacity while keeping active FLOPs within reasonable bounds. The paradigm, popularized by large language models, has been extended to visual generative models. However, applying language‑centric MoE designs to video data introduces structural mismatches that limit the benefits of sparsity.

Conventional token‑wise MoE routes each token independently within a homogeneous expert pool. A load‑balancing objective enforces uniform usage across all experts, preventing any single expert from receiving too many tokens. While this approach maximizes capacity utilization in text, it misaligns with the inherent properties of video tokens. Visual tokens are densely correlated and spatially continuous, unlike the discrete syntactic units found in language. When uniform allocation is applied, coherent patches from the same object may be scattered across different experts, and redundant background patches may dominate multiple experts simultaneously. The result is routing fragmentation: neighboring tokens lose their association, and the model struggles to maintain spatiotemporal consistency across frames.

The paper identifies this as a "uniformity trap." Existing visual MoEs fall into this trap because semantically under‑organized routing, compounded by uniform expert‑usage regularization, scatters coherent patches across disparate experts, causing routing fragmentation and structural distortion. The authors argue that the strict uniformity constraint is poorly matched to video data that is spatiotemporally redundant and semantically long‑tailed. In other words, the most frequent visual patterns—such as uniform skies or plain walls—are overrepresented, while rare semantic categories appear infrequently. A routing scheme that forces uniform distribution across experts cannot adequately serve both the frequent and the rare, and the overall generation quality suffers as a result.

How the Uniformity Trap Forms in Video MoE

The authors investigate the uniformity trap through empirical analysis of self‑similarity patterns in language versus visual tokens. Language tokens exhibit relatively high entropy and low cohesion, meaning that tokens from the same semantic class are naturally separated by syntactic boundaries. Visual tokens, by contrast, form dense, continuous clusters across both space and time. When a standard MoE router assigns these visual tokens without semantic guidance, the resulting expert assignments lack discriminative power: tokens from the same semantic region are dispersed across multiple experts, while visually similar but semantically distinct regions may land in the same expert.

To make the issue concrete, the authors grouped video tokens with off‑the‑shelf semantic segmentation masks and analyzed the routed experts. They measured two properties: semantic distinctiveness, which evaluates whether different semantic classes are assigned to more category‑specific experts, and intra‑sample routing dominance, which evaluates whether tokens from the same semantic region are consistently routed to the same expert. High values in both metrics indicate routing that is discriminative across semantic classes and coherent within each semantic region. Standard MoE achieved consistently low scores on both metrics across most semantic classes, confirming that the uniformity regularization disperses coherent patches and prevents specialization.

This fragmentation manifests in concrete generation failures. In qualitative examples, baselines using standard MoE show local distortions, unnatural deformation of object geometry, and temporal jitter where similar regions switch experts across frames. Homogeneous areas are artificially split to satisfy load constraints, producing a "striping" effect. Adjacent tokens from the same object scatter across unrelated experts, causing semantic fragmentation that degrades physical plausibility and temporal coherence.

Splitting Semantic from Generic: The SplitMoE Partition

To break the shackles of uniformity, the paper proposes SplitMoE, a split‑role sparse architecture that explicitly decouples the expert pool into two disjoint subsets: semantic experts and generic experts. The total expert count M is partitioned into Ms semantic experts and Mg generic experts, so that M = Ms + Mg. This partitioning mirrors the idea that an artist first sketches a structural outline before rendering generic visual elements; semantic experts model high‑level structures, while generic experts capture local appearance and generative residuals.

The decoupled partitioning alleviates capacity interference that arises when a single isotropic expert pool must jointly optimize low‑frequency semantics and high‑frequency generic features. By separating these roles, each branch can specialize without competing for the same representational space. The semantic branch learns to assign tokens from the same semantic region to consistent expert groups, while the generic branch preserves flexible capacity for residual variations that are not well captured by discrete semantic grouping.

Given a patchified video token x ∈ ℝC, the router initially projects the token to compute semantic‑generic affinities. The token is projected via a GELU‑activated linear transformation, and the resulting hidden state is compared to each expert weight using a clipped‑scale sigmoid. Unlike a global softmax that forces competition among all experts, the use of independent sigmoid scores allows a single token to be highly compatible with both a semantic expert and a generic expert simultaneously. The scores are divided into their respective groups, and the top Ks semantic experts and Kg generic experts are selected via Top‑K operations. The final mixture weights are computed dynamically from the clean raw scores, ensuring routing fidelity.

A fixed active expert budget is maintained by ensuring that K = Ks + Kg equals the Top‑K value of the standard MoE baseline. For example, in a Top‑2 routing setup, Ks = 1 and Kg = 1, so two experts are activated per token total. This design ensures that performance gains stem from role‑aware capacity allocation rather than an inflated computational budget.

Prototype Alignment and Pull‑Push Prototype Optimization

A core challenge in splitting the expert pool is ensuring that the semantic branch routes tokens meaningfully, rather than falling back to arbitrary balancing. The paper introduces Prototype‑Guided Semantic Routing, which uses learnable prototypes as a bridge between visual model features and discrete semantic experts. A set of prototypes P = {p1, …, pMs} serves as a semantic anchor for one semantic expert in the VAE feature space. These prototypes are not hand‑crafted; they are learned end‑to‑end during training.

Two complementary objectives optimize the routing and prototype topology. The first is router alignment (ℒalign), which trains the router to assign visually similar tokens to consistent semantic prototypes. For each visual token, a teacher assignment is computed by comparing its corresponding clean VAE feature with the semantic prototypes using temperature‑scaled cosine similarity. The teacher assignment is treated as a fixed target (stop‑gradient), and the router prediction—derived from the DiT visual token and the expert logits—is aligned with this target via KL divergence. This objective encourages the router to assign tokens that share visual characteristics to the same expert group, without allowing the router objective to directly distort the prototype topology.

The second objective is prototype optimization (ℒproto), which learns a stable and diverse prototype topology through coupled pull and push forces. The pull term attracts prototypes toward both current mini‑batch features and recent historical features stored in a circular token bank, keeping prototypes close to meaningful regions of the VAE feature manifold and reducing dead modes. The push term imposes inter‑prototype repulsion with a margin, preventing multiple prototypes from collapsing onto redundant background‑dominated regions. Together, these two terms encourage prototypes to form separated and well‑covered centers over the visual feature distribution. Removing either term degrades the prototype layout: without push, pull attracts prototypes toward dominant visual regions and limits coverage; without pull, push drives many prototypes away from the data manifold, producing underused or dead experts.

The prototypes operate on clean VAE features, which are available without external representation models. This design choice leverages the rich visual priors already present in the diffusion pipeline, avoiding the need for separate semantic segmentation or captioning pipelines during training. The prototype-guided routing thus provides a lightweight yet effective mechanism for semantic‑aware expert assignment.

Emergent Coarse‑to‑Fine Denoising Logic

One of the most striking findings of the paper is that SplitMoE reveals an emergent coarse‑to‑fine denoising logic, without any explicit timestep‑conditioned routing constraints. The authors tracked the routing weights assigned to semantic experts over a 50‑step denoising trajectory. The semantic weight peaks in the early high‑noise stage, around steps 10–15, when the model establishes global layout, object placement, and coarse semantic structure. As denoising progresses toward later stages, the semantic weight gradually decreases, and the generic experts take over for local appearance and detail refinement.

This coarse‑to‑fine pattern aligns with the intuition behind the diffusion process: early steps must capture broad structural information, while later steps refine fine‑grained details. The fact that SplitMoE naturally develops this behavior demonstrates that the semantic‑generic partition is not arbitrary; it aligns with the inherent generation order of the diffusion process. The semantic branch learns a stage‑dependent role, allocating its capacity to the phases where high‑level structure formation is most needed, while the generic branch absorbs the residual variations throughout the trajectory.

The emergent logic provides a modality‑aware scaling path: for video data, semantic capacity is Front‑loaded during structural phases and tapered during detail phases, while generic capacity provides the flexible residual modeling that persists across all timesteps. This behavior validates the intuition behind the semantic‑generic partition and establishes a principled approach for modeling heterogeneous video tokens.

Benchmark Results: Faster Convergence and Higher Fidelity

Extensive experiments demonstrate that, under an equivalent activated‑parameter budget, SplitMoE outperforms traditional load‑balanced MoEs in convergence speed, routing coherence, and video generation quality across standard benchmarks. The authors upcycled the low‑noise checkpoint of Wan 2.2 by replacing the dense feed‑forward networks at even‑numbered layers from 15 to 35 with sparse MoE layers. Each MoE layer contains one continuously active shared expert and 100 routed experts, partitioned into 20 semantic and 80 generic experts. For each token, a Top‑8 router activates Ks = 2 semantic experts and Kg = 6 generic experts, maintaining 14B of activated parameters out of 27B total, matching the dense baseline.

Quantitative comparisons on VBench‑2 and T2V‑CompBench show that SplitMoE achieves competitive or superior scores across multiple evaluation dimensions. On VBench‑2, SplitMoE improves Creativity and Human Fidelity over the base Wan2.2 model, suggesting that the routing design benefits diverse content generation and human‑centric video synthesis. On T2V‑CompBench, the method obtains competitive compositional‑alignment results, indicating that the introduced MoE structure preserves text‑video consistency under compositional prompts. Compared with a dense fine‑tuning baseline under the same active‑parameter budget, SplitMoE outperforms across all evaluation dimensions, confirming that MoE upcycling raises the capacity and performance ceiling without increasing per‑token activated parameters.

The validation diffusion loss curves further support the advantage: SplitMoE converges faster, reaches comparable loss with roughly 70% of the training steps, and maintains lower loss throughout training. This accelerated convergence translates to practical training time savings. Ablation studies confirm the contributions of each component. Removing prototype guidance (SplitMoE w/o PG) yields similar performance to the Wan2.2‑MoE baseline, showing that expert partitioning alone brings limited gains without explicit semantic direction. Removing the pull or push term of the prototype objective each leads to degradation, confirming that the coupled mechanism is essential for a balanced prototype layout. Full SplitMoE, with both pull and push active, achieves the lowest validation loss and the highest routing coherence.

Speed comparisons reveal that SplitMoE adds negligible overhead over the baseline MoE under an iso‑activated‑parameter budget. Both methods show a slight per‑iteration latency increase over the dense model, primarily due to routing operations and memory bandwidth bottlenecks, but the architecture‑induced overhead of SplitMoE is minimal compared with the gains in generation quality.

Modality‑Aware Scaling Path for Video World Models

Beyond the immediate experimental results, the paper positions SplitMoE as a critical reference for building large‑scale video world models. The core insight—that video data demands a routing scheme that respects its spatiotemporal redundancy and semantic long‑tail—generalizes beyond the specific architecture examined here. The split‑role design, prototype‑guided routing, and pull‑push regularization together provide a template for modality‑aware sparse modeling.

The authors argue that the uniformity trap is not unique to video diffusion; any sparse routing scheme applied to data with strong internal coherence and imbalanced semantics will face similar challenges. By explicitly bifurcating the expert pool and guiding routing through learned prototypes, SplitMoE avoids forcing all tokens into a single uniform distribution. Instead, it allows tokens to cluster naturally by semantic attributes, which in turn improves routing coherence and generation quality.

The emergent coarse‑to‑fine denoising logic further suggests that modality‑aware sparse routing can be layered with the diffusion process itself. For video world models that must generate long‑horizon interactions and physically plausible motions, having a dedicated semantic capacity that is active primarily during structural phases and a generic capacity that handles residual details across all phases offers a principled way to allocate compute where it matters most. This insight may inform the design of future MoE‑based systems for not only video generation but also multimodal modeling, where different modalities impose different distributional constraints on the routing mechanism.

Limitations acknowledged by the authors include reliance on frozen visual features (VAE features used as teacher signals are fixed during training) and the sparse‑routing overhead, though they note that the added latency is minor compared with the quality gains. Future work directions include adaptive partitioning that adjusts the semantic‑generic split based on input content, efficient distributed routing schemes for very large expert counts, and extending the framework to broader video generation tasks such as long‑horizon interaction synthesis and multimodal conditioning.

In summary, SplitMoE addresses a fundamental mismatch between language‑centric MoE designs and the structural properties of video data. By splitting the expert pool into semantic and generic branches, guiding routing with learnable prototypes, and regularizing prototypes through coupled pull‑push forces, the architecture breaks the uniformity trap that plagues conventional visual MoEs. The result is faster convergence, more coherent routing, and higher‑quality video generation under matched activated‑parameter budgets. The emergent coarse‑to‑fine routing pattern reveals a modality‑aware scaling path that will be relevant for researchers and practitioners building the next generation of large‑scale video world models. Read the paper on arXiv.