When researchers look inside multimodal large language models, they often rely on a simple diagnostic: measuring how closely visual and textual representations align as they pass through the model's layers. The conventional reading is straightforward — as alignment scores rise with depth, the model is progressively integrating visual content into a shared representation space. A team from the University of Science and Technology of China shows that this interpretation is dangerously misleading. The paper, "The Alignment Illusion in Multimodal Large Language Models," was submitted September 24, 2026.
The Assumption Behind the Assumption
Tools like CKA, SVCCA, and principal-angle analysis have become standard instruments for studying how MLLMs process visual information. They compare the hidden states of visual and text tokens at each layer of the language model. The resulting similarity scores typically increase with depth, and this curve is widely interpreted as evidence that the model is learning a shared cross-modal representation — that visual content is being "integrated" into the language understanding pathway.
The problem is that this interpretation rests on an assumption that has never been directly tested in MLLMs: that geometric similarity between hidden states reflects content-level cross-modal interaction. In the original settings where these tools were developed, the compared representations came from independently trained networks, where high similarity genuinely indicated convergence to similar features. In MLLMs, visual and text tokens enter the same sequence of Transformer blocks and are processed by the identical attention and MLP weights. The independence condition that licenses the standard interpretation simply does not hold.
Once both streams share the same weights, an increase in measured alignment can arise from two fundamentally different sources. The modalities may genuinely interact at the content level, drawing their representations closer. Or the shared MLP and attention weights may impose similar geometry on the two streams even when the visual tokens carry no task-relevant content whatsoever. The paper calls this second source weight-induced alignment.
The Core Experiment: Noise Instead of Content
The authors test the assumption directly through a controlled intervention. They take 13 MLLMs from five families — LLaVA-OV, LLaVA-OV-1.5, Qwen2-VL, Qwen2.5-VL, and InternVL3 — spanning 0.5 billion to 72 billion parameters, and replace the projector-output visual tokens with Gaussian noise vectors that preserve the original per-token norm but carry no visual content whatsoever. If alignment scores reflect content-level interaction, they should collapse when the content is removed.
The task accuracy collapses as expected, dropping by 38 to 50 percentage points across all 13 models. But the leading principal-angle cosine σ1 barely changes. In many layers, it closely matches or even exceeds the original curve, and in deep layers both settings approach unity. The model looks internally aligned even when the visual content needed to answer the question is gone.
This pattern holds across all four standard scalar measures tested. CKA, SVCCA, MIR, and the leading principal-angle cosine all fail to reliably distinguish the original visual stream from the corrupted one. The separation scores — which quantify how well each measure ranks the original above the noise condition — are negative or close to zero across the model set. The conventional diagnostic simply cannot tell the difference between a model that uses visual content and a model that does not.
Where the Illusion Comes From
The paper traces the illusion through a four-step mechanistic analysis. First, computing the leading principal-angle cosine between the fixed projector output and layer-wise text representations yields only moderate values that remain close to the random reference over nearly all layers. The near-unity values observed for layer-wise visual-text states are reached only after both streams have passed through the shared language-model pathway. The alignment is amplified inside the model, not captured at the input.
Second, the amplification localizes primarily to the MLP sublayer. Bypassing the MLP produces a 3.5 times larger change in the leading principal-angle cosine than bypassing attention across all 13 models. A random-initialization control confirms that the inflated alignment under noise is not a generic architectural artifact but depends on trained weight structure.
Third, the MLP down-projection matrix W_out has dominant output directions. The ratio of the top singular value to the second-largest lies between 1.2 and 1.6 across models, exceeding the matched random reference in every single model. For a Gaussian random matrix of the same shape, this ratio is close to one. The trained MLP is directionally anisotropic — it pushes both visual and text tokens toward a small set of common output directions.
Fourth, the principal-angle directions themselves concentrate in those dominant output directions. When the visual-side principal-angle basis is projected onto the top-10 left singular subspace of W_out, it captures 2.8 to 13.6 times more energy than random directions. The principal-angle basis is a structured subset of the MLP's dominant output directions, not a random match.
The mathematical argument is clean. A shared down-projection that pulls both visual and text tokens toward a common output direction inflates downstream alignment scores without any content-level coupling between the two inputs. The bound contains no term that couples the visual input to the text input — for a fixed common direction, the alignment depends only on each modality's relationship to the weight structure, not on whether the two modalities actually share semantic content.
The Principal-Angle Gap
If the leading principal-angle cosine σ1 is dominated by the single strongest output direction of W_out, then the secondary components σ2, σ3, and beyond should capture input-specific structure. When the visual input carries no structure, alignment collapses onto a single direction. When it carries structured content, alignment spans multiple shared directions. This motivates the paper's key diagnostic contribution: the principal-angle gap, defined as the difference between the first two principal-angle cosines.
A small PA gap indicates that the alignment spans multiple shared directions and reflects structured visual content. A large PA gap indicates that the alignment is limited mainly to the direction set by the MLP's weight structure. At the model level, the layer-averaged σ1 provides small and inconsistent separation between original and noise conditions — the difference remains close to zero and can even reverse direction. The PA gap, by contrast, is larger under noise across the full model set and is the only measure that ranks the original condition above the noise condition in all 13 models.
Tracking Graded Visual Degradation
The authors further validate the PA gap by interpolating between original visual tokens and Gaussian noise at ten levels. As the visual-token stream contains more original content, task accuracy rises before saturating. The PA gap achieves a Pearson correlation of 0.894 with task accuracy across the 13 models, with a median correlation of 0.917 and a minimum of 0.655. It exceeds 0.80 in 12 of 13 models and has the expected sign in all 13 models.
The competing scalar measures perform substantially worse. The leading principal-angle cosine achieves a mean |r| of only 0.730 with sign consistency in just 4 of 13 models. CKA and SVCCA perform similarly. MIR, which is already known to be more robust than CKA and SVCCA in some settings, achieves a mean |r| of 0.760 with sign consistency in 11 models but still falls well short of the PA gap.
When Structure and Relevance Come Apart
The graded-degradation experiment does not separate visual structure from task relevance. The original images carry both, and the noise removes both. To disentangle them, the authors introduce a fourth condition: irrelevant images, which preserve natural visual structure but break its relevance to the question by using the projector output of a randomly sampled image from a different question.
The results are revealing. If MLLMs ignored irrelevant visual input, the irrelevant condition would match the text-only baseline. Instead, it falls below the text-only baseline across all 13 models, by a median of 5.0 percentage points. Meanwhile, the noise condition is closer to the text baseline, with a median difference of only 2.1 percentage points. Irrelevant visual input is actively processed rather than ignored, and processing it costs more accuracy than removing visual content altogether.
The PA gap exposes this regime where internal geometry and task accuracy diverge. Under the irrelevant-image condition, the PA gap reveals that alignment spans multiple directions — the model is genuinely processing the visual structure — but that structure is not task-relevant. The scalar scores, by contrast, remain high and would misleadingly suggest that visual integration is intact and beneficial.
Implications for Model Evaluation
The paper's implications extend beyond the specific diagnostic it introduces. The central message is methodological: a high internal alignment score is not, by itself, evidence that an MLLM uses the image to answer the question. Standard alignment measures conflate weight-induced geometry with content-level cross-modal interaction, and they will systematically overstate visual integration whenever the MLP's anisotropic structure pulls representations toward common directions.
The PA gap offers a more reliable diagnostic by separating the one-directional weight-induced component from multi-directional visual structure. But even this measure is most informative when calibrated by controlled task evidence. The paper recommends pairing internal alignment analysis with interventions such as visual-token corruption or irrelevant-image substitution, which can reveal whether the alignment actually supports task performance or merely reflects the geometry of the shared language-model pathway.
The results also carry implications for model design and training diagnostics. If alignment scores are inflated by weight-induced effects, then training procedures that aim to increase visual-text alignment may be improving the geometry of the shared pathway rather than the genuine integration of visual content. The paper does not resolve this question, but it establishes that the current diagnostic toolkit needs to be recalibrated before alignment scores can be used with confidence as indicators of cross-modal understanding.