How multimodal language models should encode visual input remains up for debate. Most models use a pretrained vision encoder, while some recent recipes train the encoder from scratch with the language model or remove it entirely. Vision encoders bundle together three design choices: a visual prior from upstream pretraining, dedicated vision capacity, and pre-fusion placement of that capacity (i.e., before the multimodal backbone). Because published models also differ in data, architecture, and compute, comparing them cannot isolate the effect of each choice. We disentangle these choices via a controlled framework where we pretrain all models from scratch on common data and compare them on visual question answering and language benchmarks at matched lifetime compute. We find that the visual prior is not necessary; an encoder trained jointly from scratch is on par with a pretrained one, without a separate training stage. What matters instead is dedicated vision capacity: an encoder-free model trails on visual understanding even when its backbone is enlarged by the encoder's parameter count. Yet that capacity need not precede multimodal fusion, as vision-only modules added in-fusion to early backbone layers perform comparably to a pre-fusion encoder and can lower time to first token. These results also hold across backbone sizes, when encoder pretraining is treated as free, and when starting from a pretrained language model. Overall, our findings favor dedicated vision capacity trained jointly with the backbone and placed early within it.
Why multimodal vision encoding splits into three choices
Recent multimodal language models rely on a vision component to convert images into tokens that the language model can process. The prevailing approach stacks a pretrained vision encoder before the language model, often freezing or lightly finetuning the encoder weights. However, a growing body of work explores training the vision component from scratch alongside the language model, or dispensing with a dedicated vision tower entirely. Each approach carries implications for model quality, training cost, and inference latency.
The difficulty in comparing these approaches stems from the fact that published multimodal models differ along many dimensions. A model that uses a CLIP vision encoder may have been trained on 400 million image-text pairs, while a from-scratch model may use a different dataset, different architecture hyperparameters, and a different compute budget. These confounds make it hard to attribute observed performance differences to a single design choice such as "pretrained versus from-scratch."
This paper argues that vision encoders bundle three interrelated design decisions that are typically assessed together: (1) the visual prior obtained from upstream pretraining on large-scale image-text data, (2) the amount of dedicated vision capacity (parameters and compute allocated to the vision tower), and (3) the placement of that capacity relative to the multimodal backbone—specifically, whether the vision tower appears before the backbone (pre-fusion) or is integrated within it. Because these choices are entangled, the authors construct a controlled experimental framework that isolates each factor.
A common-data framework for isolating encoder design
The authors pretrain all vision encoders from scratch on a shared image-text corpus, ensuring that data, preprocessing, and curriculum are identical across configurations. They then attach each encoder to a common multimodal backbone and evaluate on visual question answering (VQA) and language modeling benchmarks. Critically, they compare models at matched lifetime compute, meaning that the total amount of training FLOPs is held constant across variants. This eliminates the confounded variable of "the pretrained model saw more data" and allows a cleaner assessment of architecture and initialization choices.
Within this framework, the authors systematically vary three factors: whether the vision tower receives an external visual prior (simulating ImageNet or LAION pretraining), the number of parameters allocated to the vision tower, and the position of the vision tower relative to the multimodal fusion layers. By crossing these factors, they produce a set of ablations that reveal the individual contribution of each decision.
From-scratch training matches pretrained vision encoders
One of the central findings is that a vision encoder trained from scratch on the common data achieves performance comparable to a vision encoder pretrained on external data, when both are given the same lifetime compute budget. The from-scratch encoder eliminates the need for a separate upstream training stage, which reduces engineering complexity and total training time. The authors attribute this result to the shared data and compute budget: when the vision tower receives sufficient dedicated capacity and is trained jointly with the language model, it learns a functional visual representation without relying on a pretrained prior.
This finding has practical consequences. Teams that lack access to large-scale image-text pretraining pipelines can still produce capable multimodal models by allocating enough compute to the vision tower during joint training. The "visual prior" advantage that many published models advertise disappears when the comparison is fair.
Dedicated vision capacity matters more than pre-fusion placement
Perhaps the most striking result concerns the importance of dedicated vision capacity. The authors find that a model lacking a vision encoder but with its language backbone expanded by the same number of parameters trails on visual understanding tasks. Simply enlarging the language model does not compensate for the absence of a vision tower with dedicated capacity. Vision understanding appears to require a component that is specifically designed and trained to process visual input, rather than repurposing purely linguistic capacity.
This result underscores that the "vision encoder" is not merely a tokenizer for images; it allocates specialized capacity—often convolutional or attention-based modules—that learn visual features such as edges, textures, and object parts. When that capacity is removed or merged into the language model, visual performance drops even if the total parameter count is preserved.
In-fusion placement matches pre-fusion encoders and lowers time to first token
The authors also examine where the vision tower is placed relative to the multimodal backbone. The conventional design puts the vision encoder before the fusion layers, i.e., the vision tokens are generated and then fed into the multimodal transformer. The authors explore an alternative: inserting vision-only modules directly into early layers of the multimodal backbone, so that vision and language features are mixed from the first computational steps. They term this "in-fusion" placement.
Vision-only modules added in-fusion to early backbone layers perform comparably to a pre-fusion encoder on VQA and language benchmarks. Moreover, this placement can reduce the time to first token, because the model does not need to wait for a separate vision encoder to produce its output before multimodal processing can begin. In a pipeline where the vision encoder and the backbone run sequentially, in-fusion modules eliminate that serialization bottleneck.
The authors note that this benefit is orthogonal to the question of whether the vision tower is pretrained or trained from scratch. An in-fusion, from-scratch encoder matches a pre-fusion, pretrained encoder in quality while offering lower latency.
Results across backbone sizes, pretraining budgets, and language model initialization
The authors verify that their findings persist across several conditioning variables. First, they experiment with different backbone sizes, from a modest 200-million-parameter vision tower to a large 1-billion-parameter tower. The pattern remains consistent: dedicated vision capacity is the dominant factor, and in-fusion placement is viable across the size spectrum.
Second, they treat encoder pretraining as a free resource. Even when the cost of producing a pretrained vision encoder is ignored (as might be the case for teams that reuse an existing checkpoint), the from-scratch encoder still competes effectively, provided it receives sufficient dedicated capacity during joint training.
Third, they initialize the language model from a pretrained checkpoint (e.g., GPT-2 or LLaMA) and then add the vision tower. The results hold: the visual prior is not necessary, and dedicated vision capacity placed early in the backbone yields strong visual understanding.
These cross-conditioning experiments strengthen the claim that the three bundled choices—visual prior, dedicated capacity, and pre-fusion placement—can be decoupled and assessed independently.
Design recommendations for multimodal architecture
Taken together, the experiments suggest a set of practical guidelines for building multimodal language models. First, if a team has limited compute for vision-specific training, allocating that compute as dedicated vision capacity within a from-scratch joint-training regime is sufficient; a separate upstream pretraining stage is not required. Second, the vision tower should be given enough parameters and FLOPs to extract visual features; simply enlarging the language model does not substitute for this dedicated capacity. Third, placing vision modules early within the multimodal backbone—rather than as a separate pre-fusion stage—can maintain or improve accuracy while reducing inference latency, especially the time to first token.
These recommendations are agnostic to the choice of base language model and can be applied when starting from scratch or when initializing from an existing checkpoint. The authors make their code and pretrained checkpoints publicly available to enable follow-up work.
In summary, the paper disentangles three design choices that have been entangled in prior multimodal work. The visual prior from upstream pretraining is shown to be unnecessary when the vision tower is trained jointly from scratch with adequate dedicated capacity. Dedicated vision capacity is the key driver of visual understanding; a language-only model enlarged to match the encoder's parameter count falls short. And vision-only modules placed in-fusion within early backbone layers can match the quality of a pre-fusion encoder while offering lower latency. These findings point toward a more modular and compute-efficient paradigm for multimodal model design.
# Example: Configuring a vision tower for from-scratch joint training
vision_config = {
"type": "conv_attention",
"parameters": "200M",
"pretrained": false, # train from scratch
"placement": "in-fusion", # early backbone layers
"joint_training": true,
}
Read the paper on arXiv: https://arxiv.org/abs/2611.17334