Imagine teaching a machine to draw a red bicycle leaning against a brick wall. One familiar route is to first give it a vast education in written language, then connect that language-trained system to an image-maker. But does the machine need to read the internet before it can learn what a caption has to do with a picture?
A new study tests that assumption directly. Its answer is strikingly practical: for the image-generation task they examined, a model could learn the connection from paired captions and pictures, without first inheriting an internet-scale language model. That does not make it language-free. It changes where, and how, the language learning happens.
The researchers call their model family Halcyon. Its design puts words and images into one sequence, rather than relying on a separate, pretrained text model to interpret the prompt for an image generator. The image itself is first translated into a grid of discrete visual tokens: at 256-by-256-pixel resolution, Halcyon represents it with 1,024 tokens arranged in a 32-by-32 grid. The model learns to predict the next item in a sequence, whether that item is part of a caption or part of an image.
Think of the model learning from a picture-caption pair as one joined lesson: the words arrive, then the visual pattern that goes with them. The researchers trained Halcyon’s image tokenizer, text tokenizer, and multimodal model from scratch using the GPIC dataset, which contains about 100 million image-text pairs. There is an important qualification: OpenWebText was used to help build the text tokenizer, the component that breaks writing into units. But it was not used to pretrain the generative model itself; that model learned from paired image-and-caption data.
To see whether skipping language pretraining was more than a clever shortcut, the team compared Halcyon with selected text-to-image systems that use pretrained language models. They counted the computing work needed to build each system—including the language model, image tokenizer, and image generator—rather than tallying only the final image-making stage. On the GPIC evaluation, they measured generation Fréchet distance, or gFD, a score for which lower is better.
Halcyon’s 0.52-billion-parameter model scored 48.02 gFD, compared with 63.29 for a tested LlamaGen system. The 1.6-billion-parameter Halcyon scored 35.28, compared with 40.97 for a tested JiT-T2I system. On the researchers’ accounting, those Halcyon models used 18.2 times and 275 times less total pretraining compute, respectively, than those comparison systems. These are results on a particular dataset and evaluation setup—not a guarantee that the same ranking will hold for every image task.
The researchers also asked a more revealing question: what happens if they give Halcyon the text-only education it supposedly needs? Under a fixed training budget, reserving 10%, 20%, or 50% of that budget for text-only pretraining made image-generation results worse. They ran another test that added text-only pretraining while still allowing a full round of image-caption training. That cost 1.5 times as much compute, yet did not improve the score. And when they started with a language model pretrained on 36 trillion text tokens, its image-generation result was similar to that of the corresponding model started from random initialization.
The finding is not that words do not matter. Captions are essential to the task: they give the model examples of how written descriptions correspond to visual details. The narrower lesson is that learning to continue text well and learning to turn a particular prompt into an image may not require the same preparation. In these experiments, a broad text-only warm-up did not give the image-making system a useful head start.
Halcyon could also describe images, not just create them. The team used 10% of its training budget for image-to-text examples, in which the model saw an image and learned to produce a caption. They did this by replacing examples where the prompt had been deliberately dropped, keeping the overall training budget and image-token exposure comparable. At the 0.52-billion-parameter size, the modified model’s image-generation scores remained broadly comparable to the original, while its image-captioning ability improved. The trade-off illustrates how one training stream can teach a model to move in both directions between pictures and words.
There are boundaries to what this result establishes. The experiments centered on GPIC and images at 256-by-256 resolution. The paper notes that testing other datasets and higher resolutions is important, and its controlled tests of training recipes used a model around 0.52 billion parameters. The comparisons also do not settle whether language pretraining helps other kinds of multimodal systems, or whether the same approach works for video.
For now, Halcyon offers a useful change of perspective. A text-to-image model may not need to first become a general-purpose reader of the internet. It may be able to learn the specific bridge it needs—between a phrase and the scene it describes—by studying those two things together. The broader question is whether other AI systems can skip similarly expensive detours when the task already supplies examples of the connection they need to learn.
