Imagine remembering a plot but not the paper it came from: a curve that sinks as a star grows hotter, perhaps, or a map showing where a distant galaxy is brightest. You can describe what you saw, but the caption may speak a different language—packed with symbols, survey names, and technical shorthand. Astronomy has search engines for papers. Finding that one figure has been another matter.
That mismatch matters because a figure can carry the central result of a study. Yet familiar scholarly searches mostly index words in titles, abstracts, and full text. Ask for a plot by describing what it shows, and your everyday phrasing may not match the caption at all. A paper-level search might bring back the right article—but leave you to hunt through its figures yourself.
Hurum Maksora Tohfa and Francisco Villaescusa-Navarro built a different kind of search tool, called FOTO. Instead of treating the paper as the smallest searchable unit, FOTO indexes individual figures from astronomy papers. A researcher can describe a plot in ordinary language; the system searches for the figure whose title and caption best match that description, then checks a shortlist of candidates.
The scale is substantial: FOTO’s index contains 482,750 figure-caption entries from 53,839 peer-reviewed astronomy papers posted through June 2023. The researchers start with each figure’s caption and prepend the paper’s title. That small design choice proved surprisingly important. In an initial test on 20,000 figures, recall at 20—the share of queries for which the target appeared among the first 20 results—rose from 0.491 with captions alone to 0.914 when the title was added.
Why should a title help so much? A caption may describe axes, colors, and plotted quantities while leaving the broader subject implicit. The title can supply that missing context: the particular object, survey, or scientific question that makes the plot meaningful. The researchers also tested having a model rewrite captions into fuller descriptions. Once the title was included, rewriting added almost nothing, so FOTO keeps the original text rather than generating a new description for every figure.
For the search itself, the team chose an openly available 110M-parameter text model that runs locally. It converts each title-and-caption pair—and, later, a user’s query—into a numerical representation that captures meaning. The system compares the query with the indexed figures and returns the closest matches. The index is built in advance; at search time, the more demanding checking step is limited to the shortlist, rather than sending every figure through a large model.
That second step helps address a particular risk: a fluent answer is not necessarily a real one. FOTO does not ask a language model to invent a citation or draw a plausible-looking plot. It retrieves figures that already exist in the indexed papers, then asks a model to judge whether the candidates actually fit the query. The distinction is important. The goal is not to generate scientific evidence, but to help someone locate a published result.
The researchers tested the tool against 500 sampled figures, writing queries in five styles: terse, casual, vague, detailed, and using notation different from the caption. On the strict task of finding the exact figure within the first 20 results, FOTO succeeded for 29 to 79% of queries, depending on the style. The comparison systems, Pathfinder and Semantic Scholar, were evaluated on the easier task of finding the paper containing the figure, not the figure itself; they reached 6 to 20% and 8 to 16%, respectively. Even with that difference in scoring, FOTO performed better across every query style.
Retrieval is not the whole story. A candidate can be somewhere in the results without appearing near the top, especially when the query is vague. The team therefore tested a verifier that reranks the shortlist. In a separate test of 60 queries—30 detailed and 30 vague—reranking raised the chance of placing the target first from 0.667 to 0.875 for detailed queries, and from 0.071 to 0.571 for vague ones. The biggest improvement came where the initial search struggled most.
There is a limit: no verifier can recover a figure that the search never retrieves. In the study, vague queries were the hardest case, and finding more of their targets could mean checking a deeper shortlist. That brings a tradeoff: more candidates give the verifier more chances, but also require more model calls. And the test results come from a benchmark, not evidence that researchers have already transformed how they work with the literature.
FOTO’s index also stops at papers posted through June 2023, so it does not yet cover the newest work. The authors describe plans to extend it and to offer a free public version; at the time of the paper, the hosted version required a user-paid API. Those are practical constraints, not small footnotes: a useful search tool depends on fresh coverage as well as clever retrieval.
Still, the underlying shift is simple and revealing. Instead of asking only, “Which paper mentions this topic?”, FOTO asks, “Which figure shows what I mean?” That reframing turns the visual evidence inside astronomy papers into something searchable in its own right. It won’t replace reading the paper, checking the axes, or judging the science. But it may make the first step—finding the plot worth examining—feel less like searching a library blindfolded.
