Multimodal sentiment analysis has made enormous strides in recent years, but a persistent vulnerability remains: real-world data is messy. Cameras freeze, microphones clip, text gets truncated. When one or more of the language, vision, or acoustic modalities goes missing, most existing models crumble. A team of researchers from Shandong University — Wenhao Li, Zhibin Wu, Chong Xiao, and Qiangchang Wang — introduces SemMSA, a framework accepted at NeurIPS 2026 that tackles this problem head-on by leaning on large language models to supply the semantic glue that holds multimodal understanding together when evidence is incomplete.
The Problem with Patching Missing Modalities
Current approaches to multimodal sentiment analysis with missing data fall into two camps, and neither is fully satisfactory. Reconstruction-based methods attempt to fill in the gaps: given a partially observed video segment, they train a model to regenerate the missing frames or audio. The trouble is that these methods tend to recover low-level statistical patterns — the texture of a skin tone, the pitch contour of a syllable — rather than the high-level semantic content that actually determines sentiment. Worse, the same sentence can be delivered with a smirk or a frown, meaning reconstruction can hallucinate facial expressions or vocal prosodies that directly contradict the sentiment the text conveys.
Fusion-based methods take a different route. Instead of reconstructing missing data, they try to extract robust representations by designing increasingly complex cross-modal interaction modules — graph neural networks, recurrent translation modules, dominance-guided architectures that rely on a primary modality as an anchor. These approaches work reasonably well when all three modalities are present, but under severe missingness they overfit to co-occurrence patterns learned during training and produce unstable, noisy guidance. When the anchor modality itself is compromised, the entire alignment collapses.
The core gap, as the authors identify it, is that neither paradigm operates at the level of meaning. They compensate at the feature level or the representation level, but not at the semantic level — the level where humans actually make sense of partial evidence by invoking contextual knowledge and prior understanding of emotional expression.
How the Cross-modal Semantic Refinement Works
SemMSA addresses this gap with two tightly integrated components. The first, Cross-modal Semantic Refinement (CSR), serves as the bridge between heterogeneous non-linguistic modalities and the internal representation space of a frozen large language model. The intuition is that an LLM already encodes rich semantic knowledge about how language, facial expressions, and vocal tones relate to sentiment — the challenge is getting the visual and acoustic data into that space without expensive fine-tuning or autoregressive text generation.
The process begins with modality-specific encoders that extract feature sequences from the incomplete visual and acoustic inputs. Each modality encoder is frozen, meaning its parameters are not updated during training. To map these features into the LLM's embedding space, lightweight adapters are introduced. Each adapter consists of a small set of learnable prompt embeddings — eight in the default configuration — processed through a two-block transformer. These prompt embeddings act as queries that attend over the raw modality features through cross-attention, selectively aggregating reliable evidence. They also interact with each other through self-attention to capture intra-modality dependencies. Learnable positional embeddings are injected to preserve temporal ordering in the otherwise permutation-invariant feature sequences.
After the adapter transformer, the output vectors are linearly projected into the LLM's hidden dimension, producing compact visual and acoustic prefix tokens. These are concatenated with the language tokens directly embedded by the frozen LLM, forming an initial multimodal prefix that contains partially observed evidence from all three modalities in a unified embedding space.
The refinement process is where things get interesting. Rather than generating explicit text descriptions of the sentiment — which would be computationally expensive and introduce additional noise — CSR performs iterative hidden-state refinement in the continuous LLM space. The final hidden state of the LLM, given the current multimodal prefix, is extracted as a contextual summary. This summary is then appended as a continuous latent token to the prefix, and the entire process repeats. With four refinement steps, this produces a sequence of four latent semantic states, each a dense vector in the LLM's embedding space. These states form a compact, sentiment-relevant semantic representation that captures progressively deeper understanding of the available multimodal evidence, all without a single token of decoded text.
The resulting semantic representation is obtained by pooling the sequence of latent states and projecting it through a linear layer. This yields a compact semantic vector that will later serve as one of four components in the spectral alignment.
Aligning Everything Without an Anchor
The second component, Cross-modal Spectral Alignment (CSA), addresses the question of how to integrate the refined semantics with the visual, acoustic, and language representations. Traditional approaches align modality pairs through a predefined anchor modality — for instance, aligning vision and audio to language as the reference. But when the anchor modality is degraded or missing, this strategy falters.
Instead, CSA builds a kernel Gram matrix over the normalized representations of all four modalities — semantic, visual, acoustic, and language — for each instance. The Gram matrix captures pairwise similarities among the representations. To move beyond linear relationships, the matrix is computed in a high-dimensional Reproducing Kernel Hilbert Space using a radial basis function (RBF) kernel, which measures the exponential of the squared Euclidean distance between pairs of representations.
The key insight is that if all four representations are well-aligned — pointing toward the same underlying sentiment direction in the kernel space — the Gram matrix will exhibit a strong low-rank structure, approaching a rank-one matrix in the ideal case. Eigendecomposition of this kernel Gram matrix yields a set of eigenvalues, with the largest eigenvalue representing the principal nonlinear semantic direction shared by all representations.
CSA then enhances the dominance of this leading eigenvalue through a softmax-based spectral objective, treating the eigenvalues as logits and encouraging the model to concentrate alignment along the top spectral direction. This is supplemented by an instance-level spectral separation regularizer that penalizes high similarity between the dominant eigenvectors of different instances, preserving cross-sample discriminability and preventing the catastrophic representation collapse where all inputs map to the same point.
The final fused representation is simply the element-wise sum of the four modality representations, which is then fed into a linear classifier for sentiment prediction. The overall training objective combines this spectral alignment loss and separation loss with a standard mean squared error between predicted and ground-truth sentiment scores.
What the Numbers Show
The experiments span three standard benchmarks: MOSI and MOSEI, both English datasets with sentiment scores ranging from −3 to +3, and SIMS, a Chinese dataset annotated on a −1 to +1 scale. Missingness is introduced through instance-wise Bernoulli masking during training, with 50% of samples kept complete, and testing proceeds across missing rates from 0.0 to 0.9 in increments of 0.1.
Under intra-modal missingness, where individual tokens, frames, or acoustic segments are corrupted within each modality, SemMSA consistently outperforms prior methods. On MOSI, it improves Acc-5 and Acc-7 over the best baseline, TF-Mamba, by 2.43 and 2.23 percentage points respectively, while achieving the lowest MAE of 1.011. On MOSEI, SemMSA pushes Acc-2 to 79.61% and F1 to 80.62%. On the Chinese SIMS dataset, it surpasses the previous best P-RMF baseline by 0.087 in Pearson correlation.
Under inter-modal missingness, where entire modalities are removed from test samples, the results are equally compelling. On MOSEI, SemMSA achieves an average F1 of 77.89 across all missing-modality configurations, outperforming the second-best method, CorrKD, by 2.71 percentage points on average. Notably, SemMSA maintains strong performance even when language is entirely absent — the {a,v} configuration achieves 85.17% F1 — demonstrating that the semantic refinement process can effectively extract sentiment-relevant information from nonverbal cues alone.
The ablation studies confirm that each component contributes meaningfully. Introducing CSR alone improves Acc-2 and F1 by 5.68 and 11.11 percentage points on average over the base model, underscoring the power of LLM-derived semantic compensation. Adding the spectral alignment loss provides further gains, and the full model with all three components — CSR, spectral alignment loss, and separation loss — achieves the best results on both MOSI and SIMS.
Efficiency and Practical Trade-offs
A noteworthy finding from the efficiency comparison is that SemMSA achieves its strong performance without prohibitive computational cost. The method uses a frozen Qwen3-1.7B model for semantic feature generation, but the trainable parameters are limited to the lightweight adapters and alignment modules. SemMSA reduces task GFLOPs by 50.5% compared to LNLN and cuts inference latency by 54.3% compared to P-RMF, while using only 4.7 million trainable parameters for the adapter components. The token-efficient refinement design — appending just four continuous latent states rather than generating full text descriptions — is the primary source of this efficiency.
The paper also includes an ablation of different LLM choices, showing consistent performance whether using Qwen3-1.7B, Llama3.1-8B, Qwen3-8B, or even Qwen2.5-Omni-7B as the semantic backbone. This suggests the approach is not tightly coupled to a specific model architecture and could flexibly incorporate future, more capable LLMs.
Limitations and Open Questions
The paper acknowledges that the reliance on a frozen LLM introduces a dependency on the quality of the base model's internal representations. If the LLM lacks specific cultural or contextual knowledge relevant to the sentiment task, the semantic refinement may inherit those gaps. The method also assumes that modality-specific encoders can produce meaningful features from partially observed inputs — if the missingness is so severe that a single frame or a single audio segment carries almost no information, the adapters have little to work with.
The theoretical analysis in the appendix establishes the connection between dominant spectral components and nonlinear alignment, providing formal grounding for the approach, but the practical sensitivity of the kernel bandwidth parameter σ and temperature τ to different datasets and missingness patterns is not extensively explored in the main text.
What This Means in Practice
For a working developer, SemMSA offers a compelling blueprint for building robust multimodal systems that degrade gracefully rather than catastrophically. The architecture is modular: frozen encoders for each modality, lightweight adapters for projection, and a spectral alignment loss that can be added on top of existing training pipelines. The key design principle — exploiting LLMs for semantic-level compensation in the hidden space rather than through costly text generation — is particularly relevant as LLMs become more accessible and efficient.
The approach also points toward a broader shift in how we think about multimodal AI: rather than trying to reconstruct what is missing, it may be more effective to reason about what remains using a model that already understands the semantics of emotional expression. SemMSA is a concrete, experimentally validated step in that direction.