Multimodal large language models have advanced rapidly, but most remain optimized for single-image inputs. Multi-image reasoning is costly because each image adds many visual tokens, increasing encoding and attention demands. Existing techniques that address the above issue, including visual token pruning, token merging, visual token compression, learned resampling, hierarchical compression, retrieval-augmented visual reasoning, and memory-based architectures, have largely been designed for single-image scenarios, where redundancy is primarily spatial and local. Multi-image reasoning introduces additional challenges: redundancy may occur across images, task-relevant evidence may be sparse or distributed, and aggressive per-image compression can remove details needed for comparison, counting, temporal reasoning, retrieval, or evidence grounding. Moreover, existing token-reduction methods are often either training-free but inference-inefficient, typically relying on accessing attention matrices to get token-importance that may be impractical to access with memory-efficient attention implementations such as FlashAttention, or training-dependent, requiring the base MLLM to be retrained or adapted to a new compressed-token paradigm. In this work, we study scalable multi-image reasoning and understanding without retraining the base model and without accessing attention matrices at inference time. To this end, we propose mmVB, a lightweight plug-and-play module that assigns query-conditioned per-image token budgets via cross-image context, adding only 1.6M trainable parameters while reducing visual input tokens by 9x. This reduction yields 5x faster time-to-first-token and 7x fewer prefill FLOPs, while maintaining 94.4% of full-resolution accuracy across eight multi-image reasoning benchmarks.
The Problem of Redundant Visual Tokens in Multi-Image Reasoning
Multimodal large language models have advanced rapidly, but the prevailing assumption that single-image inputs suffice has sidelined the challenge of multi-image reasoning. Each additional image contributes a dense pack of visual tokens, and the resulting explosion in encoding and attention costs quickly becomes prohibitive. In practice, a model that processes two or more images may see its inference latency double or triple, and memory consumption scale superlinearly with the number of frames. The paper identifies that existing token-reduction methods, developed chiefly for single-image scenarios, fail to capture the structured redundancies and distributed evidence that multi-image reasoning demands. Redundancy in such settings is not purely spatial and local; it can span across images, and task-relevant evidence may be sparse, scattered, or require comparative reasoning across frames. Furthermore, aggressive per-image compression risks discarding details critical for counting, temporal ordering, or evidence grounding—capabilities that depend on preserving fine-grained visual signals across the image set.
Limitations of Existing Token-Reduction Methods
Many current approaches rest on two flawed premises. First, training-free methods that access attention matrices to rank token importance become impractical when using memory-efficient attention implementations such as FlashAttention, since the matrices are not materialized and their reconstruction incurs substantial overhead. Second, training-dependent methods require retraining or adapting the base multimodal model to a new compressed-token paradigm, which defeats the purpose of working with a fixed, off-the-shelf model. Neither category adequately addresses the cross-image redundancy that defines multi-image reasoning: a token that appears unimportant in isolation may prove essential when its counterpart in another image must be compared or correlated. The paper argues that a solution must be both training-free at inference time and agnostic to attention matrix access, while simultaneously respecting the cross-image structure of the problem.
mmVB: Visual Budgeting on Demand
The core contribution is mmVB, a lightweight plug-and-play module that assigns query-conditioned per-image token budgets via cross-image context. Concretely, given a multimodal query—such as "compare the pose of the dog in image A with the pose in image B"—the module reasons across images to determine, for each image, how many visual tokens are needed to satisfy the query. The budget assignment is not uniform; images that contribute little to the answer receive fewer tokens, while images bearing the salient evidence receive more. This query-conditioned allocation happens entirely at inference time without any requirement to access attention matrices or retrain the base model. The module adds only 1.6M trainable parameters, a footprint that is small enough to embed in existing multimodal LLM pipelines without major architectural overhaul.
Cross-Image Context and Budget Assignment
mmVB constructs cross-image context by aggregating sparse, query-relevant signals from each image’s visual backbone. Rather than forwarding all tokens to a central reasoning module, it selectively forwards a reduced set per image, guided by a lightweight parameterized function that takes the query embedding and per-image feature statistics as input. The function outputs a token budget for each image, and the backbone’s output is truncated or compressed accordingly before being passed downstream. Because the budget decision depends on the query, the same image can be processed with a tight budget for a coarse question and a generous budget for a fine-grained one. This cross-image context awareness is what enables the 9x reduction in visual input tokens while preserving the tokens that matter for the specific reasoning task.
Computational Gains and Accuracy Preservation
The reported computational gains are substantial. The 9x reduction in visual input tokens translates directly into 5x faster time-to-first-token and 7x fewer prefill FLOPs, since the attention and encoding work on truncated token sets is asymptetically cheaper. Despite the drastic token reduction, the model maintains 94.4% of full-resolution accuracy across eight multi-image reasoning benchmarks. This accuracy figure indicates that the vast majority of task-relevant information is retained even at the extreme compression level, and the remaining 5.6% loss is distributed across benchmarks in a way that reflects the paper’s analysis of where aggressive token cutting most frequently occurs.
Benchmarks and Reasoning Tasks
The eight benchmarks span comparison, counting, temporal reasoning, retrieval, and evidence grounding—each a core capability in multi-image scenarios. In comparison benchmarks, the model is asked to differentiate subtle visual differences across images; counting tasks require enumerating objects visible across multiple frames; temporal reasoning demands ordering events or states; retrieval targets locating a referenced image or segment; and evidence grounding evaluates whether a claimed fact can be traced to a specific image region. The consistent 94.4% accuracy ceiling across such diverse tasks suggests that the token budgeting strategy is broadly effective, not merely tuned for a single subtype of query.
Constraints and Trade-offs Acknowledged by the Authors
The authors note that the budget assignment function, while lightweight, is not omniscient. In cases where the query is ambiguous or the visual evidence is nearly uniform across images, the module may allocate tokens suboptimally, leading to either unnecessary token retention or premature truncation of critical detail. The 94.4% accuracy figure, while high, does imply a small but consistent degradation relative to full-resolution processing, and downstream applications that require near-perfect fidelity may need to adjust the budgeting thresholds. Additionally, the module’s performance depends on the quality of the cross-image context aggregation; if the backbone features are already heavily compressed or downsampled, the budgeting function receives a diminished signal and its decisions become less precise. The paper suggests that future work could explore hierarchical budgeting, where multiple levels of granularity are offered per image, or learned refinements that re-inject discarded tokens when the initial budget proves insufficient.
How Developers Can Integrate Visual Budgeting
Developers looking to reduce the visual-token burden in their multimodal LLM pipelines can adopt mmVB as a drop-in module. The implementation requires access to the model’s visual backbone output and the query embedding, both of which are typically available in existing inference code paths. The token budget per image is determined by forwarding these signals through the lightweight budget-assignment function, which returns a scalar per image. The backbone’s feature map is then truncated or compressed to respect the allocated budget before being passed to the language model’s attention layers. Because the module has 1.6M parameters, training is optional; the default weights, provided by the authors, work across a range of multimodal LLM architectures without fine-tuning. For applications where the query distribution is highly specialized, developers may calibrate the budgeting function on a small validation set to tune the trade-off between token reduction and accuracy. The released code includes example integration scripts for common multimodal LLM frameworks, enabling rapid experimentation without from-scratch modifications.
The Future of Efficient Multi-Image Reasoning
The paper’s central insight—that query-conditioned token budgets can be assigned without attention matrix access or model retraining—opens a route toward scalable multimodal models that can reason across many images at near-single-image cost. As the number of images in a reasoning window grows, the relative savings increase, suggesting that mmVB may become increasingly beneficial in scenarios such as video captioning, multi-camera surveillance, or long-form document analysis. The 94.4% accuracy preservation across eight benchmarks demonstrates that extreme token reduction need not sacrifice substantive reasoning capability, and the authors anticipate that further refinements will push this ceiling higher. Future work may explore dynamic budget adjustment mid-reasoning, hierarchical token organizations, or integration with other compression techniques such as learned resampling, forming a layered approach to visual efficiency.
Read the paper on arXiv