A developer on Hacker News is asking a question that cuts to the heart of why the AI boom has produced advances across so many domains simultaneously: if large language models process text, why did their breakthrough also unlock progress in images, audio, and video?
The architecture is the connection
The transformer architecture, the mechanism that made large language models possible, was designed to process sequences. Text is a sequence of tokens. Images can be represented as sequences of patches. Audio is a sequence of waveform samples or spectral frames. Video is a sequence of images with temporal ordering. The transformer does not care what the elements of the sequence represent. It learns patterns in the relationships between elements, regardless of whether those elements are words, pixels, or audio frames.
When researchers applied the transformer architecture to images, they divided images into patches, flattened those patches into sequences, and trained the model to predict relationships between them. The same attention mechanism that learns which words follow other words in a sentence learns which patches follow other patches in an image. The architecture transfers because the problem structure, learning patterns in sequences, is the same across modalities.
This is why the progress is not merely correlated with LLM investment. It is directly built on the same architectural insight. The transformer is the breakthrough. Language models were the first application that proved it worked at scale. The same architecture, adapted to different sequence types, produced progress in images, audio, video, and other domains.
Why the timing looks coincidental
The investment boom in language models created the compute infrastructure, the research talent, and the engineering tooling that made applying transformers to other modalities practical. Training a large image model requires the same kind of GPU clusters that train large language models. The software frameworks, distributed training techniques, and optimization methods developed for language transfer directly to other domains. The excitement around language models funded the infrastructure that enabled progress everywhere else.
But the technical connection is not just about money. The scaling laws that showed language models improve predictably with more compute and data also apply to other modalities. The discovery that larger models trained on more data produce better results is a general property of the architecture, not a property specific to language. When researchers applied the same scaling approach to images, audio, and video, they found similar improvements. The pattern holds because the architecture is the same.
What the non-text models actually do differently
The differences between language models and other modalities are in the input representation and the output generation, not in the core architecture. Language models take token sequences and output token sequences. Image models take patch sequences and output patch sequences, which are then reconstructed into images. Audio models take waveform or spectral sequences and output them.
The generation step differs because the output formats differ. Language models produce discrete tokens from a fixed vocabulary. Image models produce continuous pixel values that need to be decoded into images. Audio models produce waveform samples that need to be assembled into sound. These are engineering challenges, not architectural differences. The core learning mechanism, the transformer's attention over sequences, remains the same.
For a computer science major trying to understand the connection, the simplest framing is this: the transformer learned to find patterns in sequences. Text was the first sequence type to be explored at scale. Once the architecture proved itself, researchers applied it to every other sequence type they could represent. The progress across modalities is the same architectural insight applied to different data formats. The investment and excitement around language models provided the resources, but the technical transfer is direct and fundamental.