Sparse Routing in Multimodal Mixture-of-Experts Architectures

Mixture-of-Experts (MoE) architectures have become a cornerstone for scaling modern multimodal models. The standard Transformer feed-forward layer replaces a single dense network with dozens or hundreds of expert subnetworks. A learned routing function, often implemented as a small feed-forward network itself, directs each incoming token to a subset of experts — typically the top-1 or top-2 experts with the highest gating scores. This design decouples model capacity from per-token computation: adding more experts increases the model's representational ceiling while keeping the cost per token roughly constant. Sparsity was introduced primarily for efficiency, not to create modular organization. Experts are learned jointly through end-to-end training, and the router adapts dynamically per token. Yet consistent routing patterns over many tokens could induce implicit modularity, with experts specializing to different kinds of information. Prior work has studied emergent expert structure in text-only MoEs, but only recently in multimodal models, where the same expert layers process both visual and textual representations jointly. The question motivating this work is whether sparse multimodal MoEs are semantically organized: do experts specialize by modality, domain, and visual concept, and if so, can this emergent structure be used for efficient adaptation?

The three models studied in this work — Qwen3-VL-30B-A3B, Kimi-VL-A3B, and Gemma-4 26B-A4B — span different MoE configurations and training objectives. Qwen3-VL-30B-A3B routes image patches and text tokens through a shared expert layer stack, with the router taking combined visual-linguistic features as input. Kimi-VL-A3B employs a similar architecture but with a smaller expert pool per layer, which influences the granularity of specialization that can emerge. Gemma-4 26B-A4B uses a slightly different expert interleaving pattern, inserting expert layers within the visual backbone rather than only in the multimodal connector. Despite these architectural variations, all three models display the same fundamental pattern: experts develop modality preferences, domain structure, and fine-grained semantic specializations without any explicit pressure toward modularity. The emergence of this structure appears to be a byproduct of training on diverse multimodal data through a sparse computation budget.

Emergent Semantic Organization Without Explicit Modularity Training

The first diagnostic the paper presents is modality specialization. In a multimodal MoE, each expert receives activation from either image patches or text tokens, not both equally. The fraction of each expert's top-K activations arising from image versus text is measured across layers and models. The results are striking: most experts strongly favor one modality, and activations are rarely balanced between image and text. For Qwen3-VL-30B-A3B, the distribution of text-token activation shares is clearly bimodal. Hartigan's dip test rejects unimodality (dip = 0.0136, p < 0.001), with an image-leaning mode at 0–5% text share (462 experts) and a text-leaning mode at 60–65% (433 experts). Image specialization is more extreme, while text specialization spreads across 60–80%. Gemma-4 26B-A4B shows a similar bimodal pattern, while Kimi-VL-A3B is more nearly unimodal, likely due to its smaller expert pool. The practical consequence is that experts are not interchangeable — an expert that has learned to process visual features cannot easily substitute for one that processes linguistic features, even within the same layer.

Beyond modality, visual representations and router weights organize strongly by semantic domain. Medical imaging, mathematical reasoning, and remote sensing satellite imagery form distinct clusters in the activation space of image patches, even though the models were trained on broadly curated web-scale data without domain-specific fine-tuning. t-SNE projections of layer-18 image-patch activations from Qwen3-VL-30B-A3B show datasets from the same domain overlapping substantially. For example, breast histopathology and MedMnist cluster together, as do RsVqaLr and RsVqaHr remote sensing subsets. These clusters are not explained by low-level properties such as color histograms or spatial resolution — the domain structure persists even when controlling for these factors. Text-token activations, by contrast, align most strongly with linguistic structure such as part-of-speech tags, and are more diffuse across the projected space. Router weights concentrate near image activations, while text activations spread more widely. The takeaway is that visual domain organization is not an artifact of low-level statistics; it reflects higher-level semantic grouping that the router has learned to exploit.

Fine-Grained Visual Semantics and Expert Specialization

The paper then asks whether specialization extends to finer-grained object-level semantics. Within TinyImageNet's 200 classes, the authors form 19,900 object-class pairs spanning a range of semantic similarity and sample 100 images per class. For each image, they track which experts activate on its patch tokens, counting an expert as active only if it fires on at least m patches (m ∈ {3, 5, 8, 12, 16}). Per class and layer, they retain experts active in at least 60% of images. They then measure Jaccard overlap between each class pair's expert sets and standardize the score with a Z-score relative to all class pairs, where larger values indicate stronger shared specialization.

The results show that related class pairs share more experts than unrelated pairs across layers. Semantic distance, measured by WordNet Wu-Palmer distance, is inversely correlated with expert-set overlap (Pearson r = −0.380, p < 0.001). For 10 TinyImageNet classes forming five semantic pairs — such as trains and freight cars, jellyfish and sea slugs, tabbies and golden retrievers — related classes show greater expert overlap across layers despite visual variation. Importantly, high expert overlap is not limited to visually similar classes. While some related pairs share visual structure (trains and freight cars), others differ substantially in appearance and background yet recruit similar expert sets. Jellyfish and sea slugs, for instance, have different shapes, colors, and marine backgrounds, but their expert overlap Z-scores remain above the unrelated-class baseline, often exceeding 3 standard deviations in intermediate layers. The same object-level specialization is observed across other MoE-VLMs. These results suggest that visual expert specialization reflects conceptual knowledge rather than dataset identity or low-level similarity.

ExpertLens: Decoding Router Weights into Semantic Vocabulary Tokens

The paper's central methodological contribution is ExpertLens, a data-free method for identifying domain-specialized experts directly from pretrained model weights. The method exploits a simple but powerful observation: in an MoE, an expert is selected when its router weight strongly aligns with a token representation. If those token representations encode semantics — and the paper's analyses confirm they do — then a domain-specialized expert's router weight should align with semantically related directions. ExpertLens therefore maps each router weight through the model's LM head to obtain vocabulary-token logits, a technique analogous to logit-lens decoding that has become a staple of mechanistic interpretability work.

Formally, for expert i in layer l, let w_i^(l) denote its router weight vector and U the model's LM head matrix. The logits are computed as z_i^(l) = U w_i^(l). The top-M tokens are extracted: V_M^(l,i) = TopM(z_i^(l)). In all experiments M = 10. The ExpertLens score for domain d is then S_EL^(l,i)(d) = (1/M) ∑_{v ∈ V_M^(l,i)} Rel(v,d), where Rel(v,d) ∈ {0,1} indicates whether token v is semantically related to domain d, as judged by the model. An expert is classified as domain-specialized when S_EL^(l,i)(d) ≥ 0.9. The Rel function is determined by checking whether the token's semantics match the target domain label — for instance, whether a text token pertains to medical concepts, mathematical notation, or remote sensing terminology. No forward passes on domain-specific data are required; the method uses only the pretrained weights.

The decoded preferences closely track the inputs that activate each expert. Representative medical expert E121, math expert E76, and remote sensing expert E93 (all from layer 18 of Qwen3-VL-30B-A3B) show decoded logits whose highest-activating tokens align semantically with their domains. E121's top tokens include medical terminology and anatomical terms; E76's include mathematical symbols and function names; E93's include satellite imagery vocabulary and geographic names. This correspondence holds across layers, not just a single layer, which means the semantic preferences are stable features of the expert's weight vector rather than transient routing artifacts. ExpertLens thus provides a way to recover expert semantics from weights alone, making it possible to target expert adaptation for new domains without collecting any domain-specific training data.

Selective Expert Fine-Tuning Across Math, Medical, and Remote Sensing Domains

The paper explores whether the emergent specialization can enable more efficient multimodal adaptation. Using ExpertLens, domain experts are selected with S_EL ≥ 0.9. For Qwen3-VL-30B-A3B, this yields 2,768 experts for math, 1,103 for medical, and 1,290 for remote sensing. The expert sets are largely distinct, with Jaccard overlap of 0.13% between math and medical, 6.48% between math and remote sensing, and 13.20% between medical and remote sensing. This low overlap confirms that the model has allocated largely non-overlapping expert subsets to each domain.

Selected expert MLPs are fine-tuned while freezing all other experts and the vision encoder. Shared decoder parameters — including QKV projections, router weights, and the LM head — are also trained, since these parameters shape how inputs are routed to and represented within the trained experts. Since expert MLPs contain most model parameters (roughly 53.0–78.3% of the total depending on the model), freezing the remaining experts and the vision encoder means that only a sparse subset of the model receives gradient updates. The fine-tuning protocol sequentially adapts a single model to math, medical, and remote sensing tasks, without resetting to the pretrained checkpoint between domains. Training uses 4,000 examples per domain and evaluates on multiple benchmarks spanning the three domains. The sequential protocol tests whether targeted expert adaptation remains effective as the same model is repeatedly adapted, which is the more realistic scenario for continuous learning or domain expansion scenarios.

Matching Full Fine-Tuning With 21.7–47.0% Parameter Updates

ExpertLens matches or surpasses full fine-tuning while updating only 21.7–47.0% of model parameters. At the end of each training stage, ExpertLens achieves 64.6% accuracy on math versus 63.8% for full fine-tuning, 53.7% versus 53.6% on medical, and 51.9% versus 51.6% on remote sensing. The fact that sparse updates can match or exceed full-model adaptation is the paper's most surprising finding. It indicates that the pretrained MoE already contains sufficient domain-relevant capacity; the challenge has been identifying and activating the right experts. Full fine-tuning updates all model parameters, while ExpertLens updates a sparse subset and yet achieves comparable or better results.

The speedup comes from two compounding factors. First, fewer updated parameters mean fewer gradient computations per training step. Second, the selective update pattern reduces the computational overhead of the training loop, since the framework can skip zero-gradient operations for the frozen experts. Relative to full fine-tuning, ExpertLens achieves speedups of 3.2× on math, 4.0× on medical, and 4.8× on remote sensing, averaging 4.0× across the entire training sequence. This means that for the same wall-clock time, ExpertLens can either achieve comparable results or achieve better results in less time. The 4.0× average training speedup is a substantial practical improvement, especially for organizations that need to iterate quickly across multiple domains.

Outperforming LoRA in Both Performance and Speed

ExpertLens also outperforms LoRA in both adaptation performance and training efficiency. LoRA (Low-Rank Adaptation) is the most widely used parameter-efficient fine-tuning method. It reduces adaptation cost by introducing a small number of trainable rank-decomposition matrices while keeping the pretrained model frozen. LoRA ranks R ∈ {32, 64} with scaling factor α = 2R following prior work, and the best configuration reported uses R = 32.

In direct comparison, ExpertLens achieves higher adaptation performance than LoRA while also reducing training time more substantially. LoRA reaches 55.8% on math, 52.0% on medical, and 49.4% on remote sensing — below both full fine-tuning and ExpertLens. Compared with LoRA, ExpertLens trains 2.9× faster and achieves higher performance across all domains. The performance gap exists because LoRA adds new parameters on top of a frozen model, while ExpertLens uses already-specialized expert subnetworks that are part of the pretrained architecture. LoRA's low-rank updates cannot fully exploit the modular structure that has emerged spontaneously in the MoE. ExpertLens, by contrast, selects and fine-tunes the exact experts that the pretrained router has already routed to particular domains, which means the adaptation starts from a more semantically aligned initialization. The expert-weighted initialization provides a head start that LoRA's random-rank additions cannot match.

Robustness, Design Choices, and Broader Implications

Ablation studies reveal that shared decoder parameters matter for effective adaptation. The paper systematically ablates which parameter groups are trained alongside the selected experts. Training only the selected expert MLPs yields weaker adaptation, especially on math. Adding the router improves math accuracy by 11.2 points, with QKV projections and the language-model head providing further gains across domains. These results suggest that selective adaptation benefits from updating the shared parameters that route inputs to the trained experts and shape their representations. The full ExpertLens configuration — selected experts plus router plus QKV plus LM head — performs best across all domains, indicating that the method works most effectively when the entire routing and representation pathway is finely tuned rather than just the expert feed-forward subnetworks.

ExpertLens is robust to domain ordering. The paper repeats adaptation under three training orders: math → medical → remote sensing, medical → math → remote sensing, and remote sensing → math → medical. ExpertLens outperforms full fine-tuning in mean final accuracy under all three orderings, by 1.7, 2.0, and 0.2 percentage points respectively. It is also less sensitive to ordering, with the standard deviation across orders decreasing from 1.37 percentage points for full fine-tuning to 0.78 percentage points for ExpertLens — a 43% reduction in variability. Although gains vary across orders, the overall benefit of selective adaptation persists across different training sequences. This robustness to domain ordering is important for real-world deployment scenarios where the order of domain adaptation is not under the system designer's control.

ExpertLens approaches privileged expert selection without using domain data. Frequency Selection ranks experts by current-domain activation counts, while the Privileged Oracle ranks by current-domain counts minus counts from other domains. The oracle requires data from all domains, including future ones, and depends on the specific datasets used. Despite using no domain examples, ExpertLens achieves comparable final accuracy to both baselines. Thus, ExpertLens matches count-based selection while remaining usable without target examples or future-domain knowledge. This is a key practical advantage: in many adaptation scenarios, labeled data from the target domain may be scarce or expensive to obtain, and ExpertLens sidesteps this requirement entirely.

Emergent Modularity as a Resource for Efficient Adaptation

The findings confirm that sparsity introduced for efficiency can give rise to semantic modularity that is directly useful for efficient adaptation. The router's weighting patterns, learned purely for computational efficiency through a sparsity constraint, encode information about which experts handle which modalities and domains. This emergent structure is not a side effect engineered for modularity; it arises naturally from the diversity of training data passing through a sparse computation bottleneck. The practical consequence is that a pretrained MoE already contains a degree of domain-specialized capacity that can be used without additional training data or architectural changes. The sparsity that makes MoEs computationally tractable also creates a form of implicit organization that can be exploited for efficient adaptation.

A working developer can use ExpertLens to identify which experts to fine-tune for a target domain without collecting domain-specific examples. The method runs purely on model weights, taking just a few minutes on a single GPU for a model the size of Qwen3-VL-30B-A3B. The process is: (1) compute router weight logits through the LM head, (2) extract top-10 tokens per expert, (3) compute the ExpertLens score for the target domain, (4) select experts with score ≥ 0.9, and (5) fine-tune only those experts while freezing the rest. The selected experts can then be fine-tuned with standard procedures, and the remaining experts and vision encoder are frozen. The result is adaptation that matches or surpasses full fine-tuning while updating a fraction of the parameters and achieving 4× speedup on average. For organizations deploying multimodal models, this means lower compute costs, faster iteration cycles, and the ability to adapt models to new domains without representative training data. The method is model-agnostic in the sense that it works with any MoE architecture that exposes router weights, and the 0.9 threshold can be adjusted based on how strict the domain specialization needs to be.

Future work could explore whether the same approach generalizes to other MoE architectures, whether the ExpertLens decoding strategy can be extended to other internal model representations (such as attention pattern weights or layer-norm gradients), and whether the observed modularity holds across different scaling regimes and training data distributions. The paper's code and pretrained checkpoints are publicly available, encouraging further investigation. Potential extensions include applying ExpertLens to text-only MoEs, combining it with other parameter-efficient fine-tuning methods, and investigating whether the emergent modularity is robust to pruning or quantization of the pretrained model. The fundamental insight — that sparsity for efficiency creates modularity that can be used for adaptation — is a general principle that may apply beyond the multimodal MoE case studied here.