Multimodal Diffusion Transformers process visual and textual representations jointly throughout generation. These architectures repeatedly update text tokens through multimodal attention, forming dynamic contextual tokens whose function has remained opaque. This paper introduces a framework for reading this contextual space through natural-language interrogation. The approach trains a lightweight bottleneck network that maps intermediate contextual tokens into the input space of a frozen Large Language Model, enabling the LLM to answer questions about the emerging image directly from these hidden representations.

Decoding the Contextual Token Space

The trained reader reveals that contextual tokens encode a rich, global representation of the emerging scene. Generation-specific semantics, including attributes left underspecified by the prompt, become accessible surprisingly early in the denoising process. As generation progresses, increasingly fine-grained details enter the decodable range. A particularly striking finding: this information remains readable even when the multimodal diffusion transformer receives an empty prompt. The contextual tokens accumulate substantial image-specific information from the evolving visual representation itself, suggesting that the model builds a persistent scene description regardless of textual conditioning.

The study further finds that generations with more readable contextual representations tend to receive higher human-preference scores. This correlation positions the readability of hidden tokens as a proxy for perceived generation quality, offering a potential automatic evaluation signal that aligns with human judgment.

Contextual Alignment: Reinforcing Visual-Semantic Information

Building on the observation that contextual token readability predicts human preference, the authors introduce Contextual Alignment, a training technique that explicitly reinforces the visual-semantic information encoded in the contextual tokens. The technique operates as a regularizer added to the standard diffusion objective. It encourages the model to preserve and amplify visual-semantic features in the token space, rather than allowing them to be overwritten by generic prior distributions.

Contextual Alignment improves both generation quality and distributional coverage. Generations show higher fidelity to the intended concept, and the model produces a wider variety of valid outputs across the same prompt distribution. The authors ablate the regularizer's placement and strength, finding that intermediate-layer application yields the most consistent gains without collapsing diversity.

Interpretability as a Path to Better Generative Models

The paper frames its contributions as dual: contextual tokens serve as both an interpretable view into the internal dynamics of multimodal diffusion transformers and an effective target for improving generative models. The reader framework opens a window into what the model has "seen" and "represented" at each timestep, while Contextual Alignment provides a concrete method to shape those representations toward more desirable outputs.

This interpretability-driven approach differs from lines of work that treat hidden activations as purely diagnostic. Here, the analytical insight directly informs a training recipe. The success of Contextual Alignment suggests that making implicit model behavior explicit can yield practical improvements, a pattern that may extend to other generative architectures.

Relation to Existing Lines of Work

Multimodal Diffusion Transformers build on earlier diffusion models and on transformer-based approaches for cross-modal processing. Prior work has examined attention maps, cross-modal alignment, and feature visualization, but these have typically operated at the level of individual attention heads or token embeddings. This paper operates at the level of the dynamically updated contextual token, which aggregates information across multiple attention layers and timesteps. The bottleneck reader maps this aggregated representation into a space where natural-language questioning becomes possible, a step that generalizes beyond earlier probe-based analyses.

The idea that diffusion models accumulate image-specific information in hidden states is not entirely new; prior work has shown that intermediate latent representations retain details about the conditioning input. What this paper adds is the demonstration that this accumulation persists even with an empty prompt, and that the accumulated information quantitatively predicts human preference. The Contextual Alignment regularizer is likewise a novel contribution, framing the improvement objective directly in terms of the contextual token space.

Experimental Setup and Evaluation

The authors evaluate on standard multimodal diffusion benchmark suites. Human preference comparisons are collected against baseline models trained without Contextual Alignment. The reader probe is evaluated on a set of question-answer pairs about the generated image, covering both coarse attributes (object presence, color, layout) and fine-grained details (texture, material, spatial arrangement). Readability is measured as the probability that the LLM correctly answers each question, averaged across the test set. Human preference is collected on a pairwise comparison format, with wins/losses/ties recorded against baselines.

Results show that Contextual Alignment consistently improves reader accuracy across all question categories, with the largest gains on coarse semantics that are earliest in the denoising trajectory. Human preference wins increase by a statistically significant margin across all evaluated prompts. Distributional coverage, measured as the ratio of successful captions to total test prompts, also improves, indicating that the model covers more of the plausible output space.

Limitations and Directions for Future Work

The authors acknowledge that the reader bottleneck network, while lightweight, adds a small computational overhead at inference time if deployed at scale. The technique has so far been evaluated on a fixed set of diffusion models and datasets; transfer to other modalities or architectural variants remains to be explored. The relationship between contextual token readability and other evaluation metrics—such as CLIP score or prompt adherence—is established empirically but not theoretically grounded in this work. Future research may explore whether the regularizer's effect is specific to the tokenizer vocabulary or whether it shapes a more abstract semantic space.

Conclusion

This work establishes that contextual tokens in multimodal diffusion transformers carry a rich, decodable representation of the emerging scene. A lightweight reader network can map these tokens into an LLM's input space, revealing generation-specific semantics early in denoising and fine-grained details over time. Critically, this information persists even with an empty prompt, demonstrating that the model builds a visual representation independent of textual conditioning. Generations with more readable contextual tokens receive higher human-preference scores. The authors introduce Contextual Alignment, a training technique that reinforces visual-semantic information in the contextual tokens, improving generation quality and distributional coverage. Together, the results position contextual tokens as both an interpretable window into model dynamics and a effective target for improving generative models.

Read the paper on arXiv