The dominant paradigm for text-to-image (T2I) generation has long relied on internet-scale language pretraining. Models such as those in the LlamaGen and JiT-T2I families incorporate large language model components pretrained on vast corpora of text, enabling them to condition image generation on free-form captions. But this raises a fundamental question: is that massive textual pretraining actually necessary, or is it simply the path of least resistance?
Chandrasegaran et al. ask precisely this and introduce Halcyon, a family of early-fusion multimodal autoregressive models pretrained entirely from scratch on GPIC, a dataset of 100M image-text pairs. By eliminating the need for separate language pretraining, the authors demonstrate that strong T2I performance is achievable with substantially less compute.
The before: language pretraining as default
Most current T2I systems follow a two-stage approach. First, a language model is trained on billions of text documents, learning to represent semantics, syntax, and world knowledge. Second, this pretrained text encoder is fused with an image generation component, typically a diffusion or autoregressive model, and finetuned on image-caption pairs. The rationale is that a high-quality text representation is a prerequisite for guiding image synthesis. This approach has produced impressive results, but it comes at a cost: the language stage consumes the majority of total pretraining compute and carbon budget.
The gap: untested necessity
Whether language pretraining is truly indispensable has received little empirical attention. Prior work has assumed that a capable text encoder is a bottleneck, but no one had systematically ablated the text-pretraining stage while keeping the image generator constant. The question is not merely academic: if language pretraining can be dispensed with, the compute and energy savings could be substantial, and the research landscape would shift toward multimodal foundations that are more efficient to train from scratch.
Halcyon: early-fusion multimodal autoregressive modeling
Halcyon departs from the two-stage recipe. Instead of pretraining a language model separately and then attaching it to an image generator, Halcyon models image and text tokens in a single autoregressive sequence from the outset. The architecture uses an early-fusion strategy: at each timestep, both visual and textual tokens are processed jointly by the same transformer stack. This means there is no separate "text encoder" stage; the model learns to attend to language and vision signals simultaneously.
The pretraining data, GPIC, consists of 100M image-text pairs. Each pair contains a single caption and its associated image. No external text corpora are used. The model is optimized with a standard next-token prediction objective on the interleaved sequence, and the same objective also enables image captioning by treating the task as token generation in the opposite direction (image-to-text).
Halcyon-0.52B versus LlamaGen-XL/8
The smallest Halcyon variant, with 520M parameters, is evaluated against LlamaGen-XL/8, a 4B-parameter T2I model that incorporates language pretraining. On the gFD (generative few-shot diversity) metric, Halcyon-0.52B achieves a score of 48.0, while LlamaGen-XL/8 scores 63.3. At first glance, the absolute numbers differ, but the more striking finding is the compute ratio: Halcyon reaches its result with 18.2× less pretraining compute. This means that, per unit of compute, the compute-efficient model not only competes but surpasses the language-pretrained baseline.
Halcyon-1.6B versus JiT-T2I-XXL/8++
The larger Halcyon-1.6B variant, with 1.6B parameters, is compared against JiT-T2I-XXL/8++, a 20B-parameter T2I system that also relies on extensive language pretraining. Halcyon-1.6B achieves a gFD of 35.3, compared to 41.0 for the larger model. The compute advantage is even more dramatic: 275× less pretraining compute. Even though JiT-T2I-XXL/8++ is nearly 13× larger in parameter count, the compute-efficient Halcyon variant still achieves a competitive score while using a fraction of the training budget.
Text-only pretraining does not help
To test whether any amount of text-only pretraining could further improve T2I quality, the authors ran controlled ablations. Models initialized from text-only checkpoints, or models that received an additional pretraining phase on caption-only data, showed no consistent gain in generation quality over models trained from scratch on GPIC alone. The conclusion is clear: once the multimodal autoregressive objective is in place, additional text-only signal does not transfer benefit to the image generation task.
Image captioning from the same objective
One of the notable side effects of Halcyon’s design is that it naturally supports image captioning. By inverting the generation direction—conditioning on image tokens and producing text tokens—the model can produce captions for unseen images. The authors evaluate this by replacing the standard caption-dropout masking during training with image-to-text (I2T) supervision. The resulting model maintains competitive T2I generation quality under the same pretraining budget, meaning the captioning capability does not come at the expense of the primary task.
What this means for practitioners
For engineers and product teams building T2I systems, the results suggest that the research community has been operating under an unnecessary compute burden. A multimodal autoregressive approach that fuses vision and language from the first token can eliminate the language-pretraining stage entirely, reducing compute by one to two orders of magnitude while matching or exceeding the generation quality of much larger models. The GPIC-scale dataset is the only prerequisite; no curated text corpora, no massive language modeling runs, and no cross-modal distillation steps are needed.
Additionally, the same model architecture can be repurposed for captioning by simply switching the conditioning direction, offering a unified backbone for both generation and description tasks. This dual capability could simplify pipeline infrastructure for applications that need both image synthesis and textual commentary.
Limitations and remaining questions
The authors acknowledge that GPIC, while large at 100M pairs, is still modest compared to the billions of text documents used in typical language pretraining. Whether the same compute savings would scale to even larger multimodal models remains an open question. Also, the evaluation focuses on a single metric (gFD) and a specific set of baselines; other generation quality metrics or downstream tasks might reveal different trade-offs. Finally, the image captioning experiment, while promising, is evaluated on standard captioning benchmarks without extensive ablations on caption quality versus generation quality.
Nevertheless, the central finding stands: strong T2I generation does not require internet-scale language pretraining. The Halcyon family provides a concrete, compute-efficient alternative that opens the door to more sustainable multimodal model development.