Self-supervised multimodal representation learning has achieved strong results across domains, yet capturing synergistic information remains difficult due to the complexity of cross-modal interactions. Unlike shared information that exists independently in each modality, synergy emerges when task-relevant signals appear only from the joint configuration of multiple modalities and cannot be recovered from any single modality in isolation. This paper addresses how to preserve information capacity for such synergistic signals in multimodal representations. The key insight is that synergistic information manifests as higher-order statistical dependence among modalities, providing a principled target for explicitly modeling joint interactions. The authors propose Higher-order Representation and Information Learning (HRIL), which constructs an empirical cross-moment tensor over modality embeddings to represent multi-way interactions, then applies Tucker decomposition to obtain a core tensor, complemented by a synergy-aware regularizer that prevents energy concentration and preserves higher-order coupling capacity.
The Problem of Synergistic Information in Multimodal Learning
Multimodal models typically optimize for shared or redundant information across modalities—signals that appear in multiple inputs simultaneously. This approach misses synergistic information: patterns that only become meaningful when modalities are combined. For example, a visual scene and an audio track might each be ambiguous alone, but together they uniquely identify an event. Standard contrastive objectives that align modality embeddings tend to capture only the intersection of information, discarding the higher-order dependencies that constitute synergy.
The paper formalizes this gap by distinguishing between shared information, which is recoverable from individual modalities, and synergistic information, which requires the joint configuration. Existing methods lack explicit mechanisms to preserve the capacity for synergistic signals in the learned representations, leading to systematic underperformance on tasks where synergy dominates.
Higher-Order Tensor Modeling via Cross-Moment Construction
HRIL models multi-way interactions directly by constructing an empirical cross-moment tensor from modality embeddings. Given embeddings from M modalities, the method computes the outer product to form an M-th order tensor whose entries capture joint statistical moments. This tensor representation preserves the full higher-order dependence structure that lower-order approximations (such as pairwise dot products or concatenation) inevitably lose.
The cross-moment tensor is large and sparse. To make it tractable, HRIL applies Tucker decomposition, which factorizes the tensor into a core tensor and factor matrices for each modality. The core tensor captures the multi-way interactions among latent factors, while the factor matrices map modality embeddings into the shared latent space. This decomposition compresses the interaction structure while retaining the higher-order coupling information.
Synergy-Aware Regularization to Prevent Energy Concentration
A core challenge in tensor decomposition is energy concentration: the optimization tends to collapse the core tensor into a few dominant components, effectively reducing the representation to lower-order interactions. This defeats the purpose of modeling synergy, which requires distributed higher-order coupling.
HRIL introduces a synergy-aware regularizer that penalizes energy concentration in the core tensor. The regularizer encourages the singular values of the core tensor's unfoldings to remain balanced, preserving capacity across all interaction modes. This constraint ensures the learned representation maintains the ability to encode synergistic signals rather than defaulting to shared information pathways.
Experiments on Controlled and Real-World Benchmarks
The paper evaluates HRIL on a controlled synergy task designed to isolate synergistic interactions, as well as standard multimodal benchmarks. On the controlled task, HRIL achieves consistent improvements over multimodal contrastive baselines, demonstrating that the explicit higher-order modeling and regularization directly translate to better synergy capture.
On real-world benchmarks spanning vision-language, audio-visual, and other multimodal domains, HRIL shows steady gains, with the largest improvements on tasks where synergistic interactions dominate the decision boundary. The results confirm that the theoretical advantage of higher-order tensor modeling materializes in practice.
Limitations and Trade-offs
The Tucker decomposition introduces additional parameters (the core tensor and factor matrices), increasing model size and compute relative to simple contrastive baselines. The cross-moment tensor construction scales with the product of embedding dimensions, which can become expensive for high-dimensional embeddings or many modalities. The synergy-aware regularizer adds a hyperparameter controlling the balance between energy dispersion and task performance, which may require tuning. The paper notes that the controlled synergy task, while valuable for isolation, may not reflect the full complexity of real-world synergistic interactions.
Practical Implications for Multimodal System Design
For engineers building multimodal systems, HRIL offers a concrete architectural pattern: replace pairwise alignment objectives with explicit higher-order interaction modeling when the task demands synergistic reasoning. The cross-moment tensor and Tucker decomposition can be implemented as a differentiable module inserted after modality encoders. The synergy-aware regularizer is a lightweight addition to the loss function. Code release enables direct integration and experimentation.
Conclusion
HRIL establishes that synergistic information in multimodal learning corresponds to higher-order statistical dependence, and that this dependence can be explicitly modeled through cross-moment tensor construction and Tucker decomposition with a synergy-aware regularizer. The approach consistently outperforms contrastive baselines on both controlled and real-world tasks, with the largest gains where synergy is the dominant signal. The work provides a principled framework for preserving synergistic capacity in multimodal representations.