Imagine opening a box of 16 tiny Hubble images. Most show ordinary galaxies. One might hold a curved streak of light, a lopsided tail, or a shape that hints at galaxies tugging on each other. Which image deserves an astronomer’s attention? Now imagine repeating that choice across a sky survey containing millions of pictures. The challenge is not just finding what looks unusual. It is finding what might be scientifically worth investigating.
That distinction is the starting point for a new approach from Yixuan Cheng, Nolan Koblischke, and Jo Bovy. Instead of asking a computer to flag images that look most different from the crowd, the researchers tested whether a vision-language model could rank galaxy images by scientific interest. The model could interpret pictures alongside written instructions, and it had not been trained specifically on the Hubble images in the test. The idea: give the model a task closer to the one a human screener faces, then see whether its rankings help researchers decide where to look first.
The stakes are rising with the size of astronomical archives. A previous search had examined 99.6 million Hubble image cutouts for unusual objects. Researchers can use machine learning to sort such pictures by visual rarity, but rare does not always mean valuable. An instrumental artifact can be unusual, too. A lens-shaped blur or a jagged edge may stand out to an algorithm without revealing a new kind of cosmic object. The new study asks whether language-guided visual judgment can help separate promising oddities from distractions.
To test the idea, the team assembled a benchmark of 20,000 Hubble images. It included 167 objects previously identified as anomalies through a separate search and expert inspection, plus 19,833 randomly selected images. Each image was a small cutout of the sky, scaled to preserve visible shapes and features. The benchmark was deliberately designed to test whether the model could recover known anomalies. It was not a blind survey of the entire Hubble archive.
One strategy asked Gemini 3.1 Flash-Lite to inspect a grid of 16 images at a time. The prompt told it to select images that seemed scientifically interesting, skip blank or uninformative frames, and rate selected images from 1 to 5. Each image appeared in ten randomly assembled grids; the scores were added to produce a ranking. In effect, the model repeatedly scanned a crowded contact sheet, with different neighbors around each image, and made a fresh judgment each time.
The researchers also tested a tournament-style approach. Here, the model reviewed four images at a time and kept those it considered worth human inspection. Images that survived moved into later rounds, while discarded ones dropped out. Both approaches turned a mountain of pictures into a shorter list. Neither asked the model to identify exactly what an image showed or to establish that an object was new.
On the enriched benchmark, Gemini’s grid-scoring approach placed 164 of the 167 reference anomalies among 2,095 selected images: a recovery rate of 98.2 percent. It found about half of the known anomalies in its top 300 images. A second vision-language model, Qwen, recovered 95.8 percent in a similarly sized selected set. For comparison, conventional anomaly rankings based on visual features recovered 85.0 percent with one method and 70.1 percent with another among their top 2,000 images.
The visual examples help explain what the model was sorting for. Its high scores included a known gravitational lens, where the gravity of a foreground object bends light from something farther away, and images with disturbed or tidal-looking structures. The test also surfaced objects already recognized as possible mergers, ring galaxies, or jellyfish galaxies. The team’s point was not that the model had diagnosed these objects. It was that a natural-language prompt could steer a general-purpose model toward images astronomers might reasonably want to inspect.
There was a practical bonus: the model could screen at modest estimated cost. In the paper’s test, the grid-scoring run cost about $5.05 per 10,000 images using the listed Gemini service prices. A tournament version cost less, though it recovered slightly fewer reference anomalies. These are estimates tied to the models and prices used in the study, not a guarantee about the cost of future surveys or services.
Then comes the catch—and it changes how to read the headline result. The benchmark contains a much higher share of known anomalies than a typical archive. The authors estimate that a more realistic search might have only about one anomaly per 1,200 images. When they adjusted their results to reflect that scarcity, the score threshold that recovered 98.2 percent of the reference anomalies would also send 19,474 images to human review rather than about 2,000. At the same inspection budget of 2,000 images, estimated recovery fell to 45.6 percent.
That is not a reason to dismiss the approach. It is a reminder that rare objects are hard to find precisely because they are rare. A result that looks strong when unusual examples are packed into a test set can translate into a much longer screening list in the wild. The researchers also note that their reference labels are incomplete: some images treated as ordinary background may themselves contain interesting objects. So both apparent misses and apparent false alarms need careful interpretation.
To look beyond the known examples, the team checked high-scoring images against catalogs and existing research. After removing matches and objects found in a manual literature review, they kept 102 candidates for further investigation. “Candidate” matters here. A high score is a reason to take a closer look, not proof that an image contains a discovery. Astronomers still need to examine the sources, compare observations, and establish what the objects are.
The shift in the method is subtle but important: instead of defining “anomaly” only as distance from the crowd, it asks a model to make a first-pass judgment about scientific promise. That judgment is imperfect, and the benchmark shows why human review remains essential. But with future sky surveys expected to produce enormous image collections, even a rougher first pass could help direct limited expert attention toward more intriguing corners of the data. The next test is whether the approach holds up on different archives, including survey data it has never seen.
