Imagine asking a question in a library so vast that searching every book would take too long. The usual shortcut is to match your words to likely pages, then ask a more capable reader to sort through the shortlist. But what if the shortcut misses the one page that matters? A brilliant answer-writing model cannot use evidence its search system never found.
A new arXiv study proposes a different route: let an AI search by navigating a collection’s structure, moving from broad groups of documents toward individual passages. The system, called DecisionRAG, treats retrieval less like a race through matching keywords and more like a series of choices about which shelves deserve a closer look.
The distinction matters because retrieval is often the first step in retrieval-augmented generation, or RAG. A language model can consult outside documents before answering, rather than relying only on what it learned during training. But that approach depends on a search stage to locate useful evidence. If that stage overlooks the right passage, the answer-generating model may never see it.
DecisionRAG tries to give that first stage more judgment. At each level of a document hierarchy, a decision model evaluates multiple candidates together. It can assess, for instance, which topic groups seem relevant, then which documents inside those groups, and finally which passages. Rather than writing out a sequence of search instructions, the model makes choices in a single pass without generating answer tokens.
That shared comparison is central to the idea. Consider a question about the main character in Christopher Nolan’s 2000 film. A page about another movie from that year might share tempting keywords. Judged alone, it could look like a reasonable match. But when candidate pages appear together, the model can weigh one against the others, follow the promising branch, and eventually reach the page about Memento and its protagonist, Leonard Shelby.
There is a catch: the route needs a map. Some collections already have useful structure, such as help centers organized by topic, page, and passage. For Wikipedia, the researchers grouped 108,626 pages into 1,000 clusters and created short descriptions for those groups. That preparation gave the system a hierarchy to explore. DecisionRAG does not need a conventional retrieval index at query time, but it does rely on a collection whose structure and descriptions help the model choose where to go.
The team tested the approach on four benchmarks, including conversational questions about government help documents, technical documentation, and questions grounded in Wikipedia. The largest collection contained 2.68 million Wikipedia paragraphs. On that benchmark, DecisionRAG improved its top-result hit rate by 17.4 percentage points over dense retrieval followed by an 8-billion-parameter reranker. On a government-document dialogue benchmark, it improved recall at rank one by 13.8 points over the same kind of comparison system. These are results on particular test sets, not a guarantee that the method will win on every collection or question.
Speed was another striking part of the experiment. Across the four benchmarks, median retrieval time ranged from 0.68 to 1.76 seconds per query. The researchers reported 1.73 seconds on the Natural Questions benchmark, despite searching across millions of paragraphs. Their explanation is that navigation’s number of sequential decisions depends more on the depth of the hierarchy than on the total number of documents. Many candidates can be assessed in parallel at each step.
The researchers also checked whether better retrieval led to better answers. They gave the same answer-generating model the top five passages from each retrieval system. On technical-support dialogues, answers judged correct rose from 39.4% with a dense retriever and reranker to 44.1% with DecisionRAG. Adding passages from dense retrieval raised that figure to 47.3%. That combination is revealing: the new route and conventional search can find complementary evidence rather than one making the other obsolete.
Navigation has a sharp failure mode, though. Once the system drops a branch, it does not go back. If the relevant page sits inside a discarded cluster, the answer is lost before passage-level reading begins. On Wikipedia, keeping just 1% of the top-level clusters pushed recall at rank ten down to 63.7%. Keeping 15% brought recall to 86.2%, slightly above the 84.5% achieved by scoring all page titles directly—but cost more and took longer than a narrower search.
Organization can also mislead. Poorly arranged documents or vague group descriptions may point the model down the wrong corridor. The researchers found that increasing the number of pages explored improved the chance of including useful evidence, but with diminishing returns for placing the best passage first. More exploration means higher cost and latency, so the system has to balance thoroughness against efficiency.
That trade-off gives the study its larger promise and its price. If decision models can judge document options quickly and reliably, search systems may become better at finding evidence that does not share obvious words with a question. That could help the language models downstream answer from more relevant material. But the method shifts attention from building a retrieval index to building a good map—and choosing how much of it the AI is allowed to explore.
For now, DecisionRAG is a research prototype tested on benchmark collections, not proof that every archive can be searched this way. Still, it suggests a useful change in perspective: perhaps the future of AI search is not simply matching a question to pages, but teaching a system to decide which parts of a vast library are worth entering.
