Adaptive RAG: Match retrieval effort to query complexity

Adaptive RAG is a retrieval-augmented generation (RAG) framework that chooses a retrieval strategy for each query based on how complex the query is. A small classifier sends simple questions straight to the large language model (LLM), moderate questions to one round of retrieval, and multi-hop questions to iterative retrieval. In the original 2024 study, this approach matched the accuracy of multi-step RAG while taking less than half its time per query.

Key takeaways

• Fixed RAG pipelines either spend compute on simple questions that never needed retrieval or give weak answers to complex questions that needed several rounds of it.
• With GPT-3.5, Adaptive RAG matched the accuracy of multi-step retrieval at 44% of its time per query (Jeong et al., 2024).
• Routing accuracy is where the remaining gains sit, because a perfect classifier lifted F1 to 62.80 in the same test.

What problem does adaptive RAG solve?

Adaptive RAG solves the mismatch between one fixed retrieval pipeline and queries that vary widely in difficulty. Single-step retrieval handles questions whose answer sits in one document, but it misses multi-hop questions that need facts from several documents. Multi-step retrieval handles those, yet it adds LLM calls and latency to questions the model could answer from its own training.

The authors of the Adaptive-RAG study (Jeong et al., 2024) argue that real users ask simple questions far more often than complex ones, so a pipeline tuned for the hardest questions slows down every easy one, while a pipeline tuned for speed produces wrong answers on the hard ones. The gap between paths is wide, and in the study's setup a query answered without retrieval took 0.35 seconds, a single-step query took 3.08 seconds, and a multi-step query took 27.18 seconds.

How does adaptive RAG work?

Adaptive RAG adds one decision before retrieval starts, in which a classifier reads the incoming query, predicts its complexity, and sends it down one of three paths.

1. Level A, simple. The LLM answers from its own knowledge with no retrieval.
2. Level B, moderate. The system retrieves documents once, and the LLM answers from them.
3. Level C, complex. The system alternates retrieval and reasoning over several rounds until it can answer.

The Adaptive-RAG study (Jeong et al., 2024) used T5-Large, a 770-million-parameter model, as the classifier, and built the multi-step path on IRCoT, a method that interleaves retrieval with chain-of-thought reasoning. Because the LLM and the retriever stay the same on every path, the system moves between paths without any change to model architecture or parameters.

How is the query-complexity classifier trained?

In the Adaptive-RAG study (Jeong et al., 2024), the classifier trains on labels collected automatically, because no dataset of query-complexity labels existed. The authors combined two labelling methods.

• Model predictions. Each training question runs through all three strategies and receives the label of the simplest strategy that answers it correctly, so the classifier learns to prefer the cheaper path.
• Dataset patterns. When all three strategies fail, the question takes its label from its source dataset, which means Level B for single-hop datasets and Level C for multi-hop datasets.

Classifier size made little difference to answer quality. With a 60-million-parameter classifier, the system's F1 was 45.83, against 46.94 with the 770-million-parameter version, which keeps the routing step cheap to run.

How does adaptive RAG compare with single-step and multi-step RAG?

Adaptive RAG delivers multi-step accuracy at less than half the multi-step time. The Adaptive-RAG study (Jeong et al., 2024) tested it on six open-domain question-answering datasets, with SQuAD, Natural Questions, and TriviaQA covering single-hop questions and MuSiQue, HotpotQA, and 2WikiMultiHopQA covering multi-hop questions.

Source: Table 1, Jeong et al., 2024. Results are averaged across all six datasets with GPT-3.5 Turbo as the LLM.

Single-step retrieval scores below no retrieval here because GPT-3.5 already answers many single-hop questions from memory. The authors note that GPT-3.5 gains most from Adaptive RAG on complex multi-hop questions.

Adaptive retrieval falls behind because it only decides whether to retrieve, which leaves multi-hop questions without the iterative path they need.

What are the limits of adaptive RAG?

Adaptive RAG depends on the accuracy of its classifier, which reached 54.5% overall in the Adaptive-RAG study (Jeong et al., 2024) and correctly identified only about 30% of the queries that needed no retrieval. When the classifier routes a query too low, the answer is wrong or incomplete, and when it routes one too high, the system spends time and compute it does not need. With GPT-3.5, a perfect classifier raised F1 from 50.91 to 62.80 and also cut time per query.

Two further limits matter for enterprise use:

• Three complexity levels are coarse, and the authors leave finer-grained levels to future work.
• The study used BM25 keyword retrieval over Wikipedia, so results may differ with the retrievers and private content an enterprise uses.

How can enterprise teams apply adaptive RAG?

Enterprise teams can apply adaptive RAG by training the router on their own queries instead of public benchmarks.

1. Label past queries and train a small classifier. Apply the labelling method from the Adaptive-RAG study (Jeong et al., 2024) to a few hundred real queries from company content, then train a small classifier on the labels.
2. Keep fixed rules for risky queries. LLMs know nothing about events after their training and little about rare or company-specific facts, as the time-aware adaptive retrieval (TA-ARE) blog explains, so route those questions to retrieval whatever the classifier predicts.
3. Track misroutes in both directions. In the study, the classifier sent 47% of no-retrieval queries to single-step retrieval, which only wastes time, and 31% of multi-hop queries to single-step retrieval, which costs accuracy. The second error does more damage, so review it first.
4. Check what retrieval returns. A lightweight evaluator, as used in Corrective RAG, scores the retrieved documents and triggers a fallback such as web search when they are weak.

Teams that cannot label queries yet have other ways to decide when to retrieve. TA-ARE uses prompting and needs no training, DRAGIN retrieves during generation when the model's information needs rise, and Self-RAG trains the LLM to signal when it needs retrieval.

How does Kore.ai apply query-aware retrieval in enterprise search?

Adaptive RAG decides how much to retrieve, while Kore.ai's agentic RAG for enterprise search decides how to retrieve. Before retrieval starts, LLM agents read each query and set the retrieval parameters and filters to match its intent.

The Kore.ai agentic RAG documentation describes four agents:

• Query rephrasing. Rewrites follow-up questions using the conversation so far.
• Result type classification. Decides whether the user wants a direct answer or a list of results.
• Query transformation. Extracts key terms and boosts documents that contain them.
• Metadata extraction. Maps a query to structured sources and fields, such as the assignee and status of a Jira ticket.

The same documentation lists two trade-offs, and both mirror the limits of adaptive RAG:

• Latency and cost. Each agent adds an LLM call, which raises response time and token use.
• Source accuracy. A wrong source identification returns irrelevant results.

To keep these decisions visible, the debug logs show which agents ran, what each one received and returned, and how long each call took. This gives teams the data they need to track misroutes.

Source and related reading

• Jeong, S., Baek, J., Cho, S., Hwang, S. J., and Park, J. C. (2024). Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity. NAACL 2024. Read the Adaptive-RAG paper on arXiv
• Why agentic RAG is the next step for enterprise AI
• MultiHop-RAG: why multi-hop queries challenge standard RAG
• What is retrieval-augmented generation (RAG)?
• How RAG breaks knowledge silos in enterprise search

Next step: See how enterprise search applies retrieval to company knowledge

What is adaptive RAG? 

Adaptive RAG is a retrieval-augmented generation framework that chooses a retrieval strategy for each query based on its complexity. A small classifier routes simple questions to the LLM with no retrieval, moderate questions to single-step retrieval, and complex multi-hop questions to iterative retrieval. The approach comes from a 2024 study by Soyeong Jeong and colleagues at KAIST, presented at NAACL 2024.

How is adaptive RAG different from standard RAG? 

Standard RAG runs the same retrieval process for every query. Adaptive RAG first predicts how complex a query is and then picks no retrieval, one retrieval round, or several rounds. Simple questions return quickly while complex questions still get multi-step reasoning, and in the original study it matched multi-step accuracy at 44% of the multi-step time per query.

When should a team use adaptive RAG instead of single-step RAG? 

Adaptive RAG pays off when query traffic mixes simple lookups with multi-hop questions that need facts from several documents. If almost every query is a single-document lookup, single-step retrieval is simpler and gives similar accuracy. If many queries are multi-hop, adaptive RAG keeps their accuracy while sparing simple questions the cost of several retrieval rounds, which is where the Adaptive-RAG study found its gains.

When does a query need multi-step retrieval? 

A query needs multi-step retrieval when the answer depends on facts spread across several documents, such as comparing two policies or tracing a chain of events. Multi-step retrieval gathers one fact, reasons over it, and retrieves again. Questions answered by a single passage gain little from extra rounds, and each round adds latency and LLM cost.

Does adaptive RAG lower cost? 

Adaptive RAG lowers the number of retrieval and LLM steps for simple and moderate queries. In the original study with GPT-3.5, it averaged 1.03 retrieval steps per query against 2.81 for multi-step retrieval, at similar accuracy. The study measured steps and time rather than currency, so the actual saving depends on a team's query mix and on how accurately the classifier routes queries.

How is adaptive RAG different from Self-RAG and corrective RAG? 

Adaptive RAG decides on a retrieval strategy before retrieval starts, using a separate classifier. Self-RAG trains the LLM itself to decide when to retrieve and to critique its own output. Corrective RAG evaluates retrieved documents and corrects course when their quality is low. Teams can combine these methods because each one acts at a different point in the pipeline.

‍