Large language models are increasingly applied to tasks grounded in long, heterogeneous information sources. Conventional Retrieval-Augmented Generation (RAG) relies on fixed similarity-based retrieval, while agentic variants adapt queries and tool use but remain largely retrieval-centric. However, in many tasks, the evidence required for a solution is not explicitly present in any single source item. Instead, it must be derived through filtering, aggregation, or computation across multiple source items. In this work, we introduce RECAST (Routing Evidence through Computation, Access, and Synthesized Tools), a learned framework that formulates evidence construction as a sequential decision process over heterogeneous retrieval and computation operations, allowing evidence to be actively derived rather than merely retrieved. A lightweight RouterLM iteratively selects and formulates primitive operations or specifies customized operations for a frozen CompilerLM to translate into executable code. Once it judges the evidence sufficient, RouterLM passes the accepted evidence to a frozen AnswerLM to produce the final solution. We train RouterLM with supervised fine-tuning (SFT) followed by group relative policy optimization (GRPO). Across six heterogeneous benchmark families, RECAST achieves a mean success rate of 75.6%, outperforming the strongest large-model baseline by 15.9%. Moreover, training enables the Qwen3.5-9B RouterLM to outperform a training-free Gemini 3.5 Flash RouterLM by 5.0%. On three held-out benchmarks, RECAST improves over the strongest baseline by 15.0% on average, demonstrating strong zero-shot generalization across tasks and heterogeneous source representations.
Why Fixed Retrieval Falls Short for Derived Evidence
Large language models (LLMs) are increasingly applied to diverse tasks such as question answering, data analysis, and personalized assistance, many of which require reasoning over long contexts drawn from heterogeneous external sources such as document collections, databases, tables, semi-structured records, and user histories. Although modern LLMs can process increasingly long inputs, supplying the complete source incurs substantial computational and token costs while still offering no guarantee that the model will identify and correctly use the information required for a particular task. Retrieval-augmented generation (RAG) offers a practical solution by retrieving a small amount of relevant content and conditioning generation on the retrieved evidence. Conventional RAG typically performs a single retrieval step before generation, whereas iterative and agentic variants allow LLMs to reformulate queries, invoke retrieval operations over multiple rounds, and assess whether additional evidence is needed.
These retrieval-based formulations generally assume that useful evidence can be located directly within the source. However, LLM applications increasingly rely on heterogeneous sources, from which the required evidence may need to be derived rather than simply retrieved. For example, suppose the source provides monthly revenue and operating income, while the task asks which three-month period had the largest increase in operating margin. Answering the question requires computing the margins and comparing their changes across periods. More broadly, the required operations vary across tasks, from lexical or semantic retrieval to relational filtering and aggregation, and some tasks require customized computation that must be synthesized for the task at hand. Context construction must therefore determine not only where relevant information appears, but also how it should be transformed into useful evidence. Although iterative and agentic methods make retrieval more adaptive, they primarily change how the system searches for source content and offer limited support for constructing such derived evidence. This gap motivates RECAST, which formulates evidence construction as a learned sequential decision process over both retrieval and computation operations.
RECAST: Adaptive Evidence Construction Through Sequential Decision
To address the limitations of fixed retrieval, we introduce RECAST (Routing Evidence through Computation, Access, and Synthesized Tools), a multi-round framework in which a lightweight RouterLM determines what evidence is needed and formulates a task-specific operation for obtaining it. It may invoke a parameterized lexical, semantic, or relational primitive, or request synthesis of a customized operation that a frozen CompilerLM translates into executable code. The resulting evidence and execution outcome inform subsequent decisions, allowing RouterLM to refine the constructed evidence over multiple rounds. Once the evidence is sufficient, RouterLM accepts it and passes it as context to a frozen AnswerLM for final solution generation. Only RouterLM is trained; the CompilerLM and AnswerLM remain frozen throughout.
The framework proceeds in four steps. Step 1: Source preparation. A deterministic, rule-based preprocessor converts the source context into a uniformly formatted list of records and constructs a compact source profile summarizing the source scale, structure, formats, and observed fields. This profile provides the RouterLM with a high-level view of the source without requiring it to receive the complete source in its prompt; the preparation uses no language model and is independent of the task. As an initialization step, both the evidence set and interaction history are empty. Step 2: Evidence-operation formulation. At round t, the RouterLM observes the task, source profile, retained evidence, and interaction history. It selects an action according to a learned policy that either invokes a primitive operation from {Lexical, Semantic, Relational}, requests a synthesized evidence operation, or accepts the current evidence. The action must formulate the task-specific request, including the arguments or output requirements that specify how the evidence should be obtained and what should be returned. Step 3: Operation execution and feedback. For a primitive action, the selected operation is executed using task-specific arguments formulated by the RouterLM. A synthesis request is translated by the CompilerLM into a Python program and then executed over the source. Either path returns an operation outcome describing execution status, result counts, and any errors. The evidence and interaction states are updated by appending the latest action and outcome to the interaction history and adding any new evidence to the retained set. Step 4: Evidence assessment and solution generation. Steps 2–3 are repeated until the RouterLM considers the accumulated evidence sufficient. The RouterLM then accepts the constructed evidence and terminates the evidence-construction loop. Finally, the AnswerLM is invoked once with the original task and the accepted evidence to produce the solution.
RouterLM: The Lightweight Evidence Router
The RouterLM is the only trainable component in RECAST. It is a compact language model that, at each round, selects and formulates evidence operations and judges whether sufficient evidence has been gathered. The RouterLM observes the task query, the compact source profile, the retained evidence from earlier rounds, and the interaction history consisting of action–outcome pairs from previous rounds. Based on this context, it selects one of three action types: CALL_PRIMITIVE, which selects and formulates a lexical, semantic, or relational operation; SYNTHESIZE, which specifies a customized operation for the CompilerLM; or ACCEPT_CONTEXT, which terminates evidence construction and passes the current evidence to the AnswerLM. The RouterLM also generates any task-specific arguments needed to carry out the chosen action, such as a search query for a semantic operation, a SQL query for a relational operation, or a natural-language specification for the CompilerLM.
The action space is structured so that each round produces either additional evidence or a decision to stop. CALL_PRIMITIVE and SYNTHESIZE construct additional evidence, whereas ACCEPT_CONTEXT terminates evidence construction and proceeds to final solution generation. The RouterLM is trained with supervised fine-tuning (SFT) to establish valid multi-round evidence-construction behavior, including operation selection, task-specific formulation, and evidence assessment. After SFT, group relative policy optimization (GRPO) improves the routing policy using downstream task outcomes, reinforcing evidence-construction behaviors that produce better task outcomes while penalizing those that fail. The complete dataset and training details of SFT and GRPO are provided in the paper's appendices.
Evidence Operations: Primitives, Synthesis, and Acceptance
RECAST supports three types of evidence operations under CALL_PRIMITIVE: Lexical, Semantic, and Relational. Lexical ranks source records using BM25 term matching, which performs exact keyword-based retrieval. Semantic ranks source records by cosine similarity using BGE-M3 embeddings, which performs vector-based semantic retrieval. Relational executes a read-only SQL query over the normalized records, supporting filtering, joins, grouping, aggregation, sorting, and arithmetic. When the required transformation cannot be expressed through these primitives, the RouterLM requests a synthesized operation under SYNTHESIZE. The RouterLM specifies the transformation to perform, the evidence and source identifiers to return, and why the primitive operations are insufficient. The specification must preserve the task-specific constraints required to implement the operation correctly, since the CompilerLM does not receive the original task. The CompilerLM receives the synthesis specification, source profile, source samples, and prior operation outcomes, and translates the specification into a Python program in one call. The program can process the complete source, use previously obtained evidence, and invoke primitive operations when useful. It returns either the requested evidence or an execution error to the RouterLM.
After each evidence-producing action, the RouterLM receives returned evidence and structured feedback generated by the operation executor. The feedback reports the execution status, result counts, and any errors. The evidence is deduplicated and added to the current evidence set, while the action and feedback are recorded in the interaction history. The RouterLM then assesses whether another operation is needed. If so, it may reformulate the request or select a different operation in the next round. Once the RouterLM determines that the constructed evidence is sufficient for final inference, it selects ACCEPT_CONTEXT. This terminates evidence construction and passes the constructed evidence to the AnswerLM for solution generation.
Training RouterLM with SFT and GRPO
RECAST trains the RouterLM in two stages. The first stage, supervised fine-tuning (SFT), establishes valid multi-round evidence-construction behavior. Using the pretrained RouterLM, the authors generate multiple trajectories for each training question. Each trajectory executes its evidence operations and invokes the AnswerLM with the constructed evidence. A frozen LLM judge then evaluates the solution against the reference. For each question with at least one correct, executable, and structurally valid trajectory, one trajectory is selected as the SFT expert, preferring more accurate and efficient candidates. Capped upsampling repeats examples from underrepresented benchmark families and empirical difficulty levels. The complete expert trajectory is used for SFT, supervising each RouterLM decision conditioned on the evidence and feedback available at that round.
The second stage, group relative policy optimization (GRPO), improves the routing policy using downstream task outcomes. GRPO data is constructed from questions that were not consistently solved during SFT trajectory collection; each selected question must have at least one valid expert trajectory, indicating that it is solvable, and at least one failed trajectory, indicating room for policy improvement. Starting from the SFT checkpoint, a new group of four complete trajectories is sampled for each selected question. Mixed-outcome filtering retains only groups containing at least one successful and one unsuccessful trajectory, allowing GRPO to compare alternative evidence-construction behaviors for the same question and reinforce those that produce better task outcomes.
For a trajectory τ, the reward is defined as R(τ) = λ_C C(τ) + λ_F F(τ) + λ_V V(τ), where C(τ) denotes final-answer correctness evaluated by the frozen LLM judge, F(τ) is token-level F1 between the generated and reference answers, and V(τ) indicates whether the trajectory follows the required action structure. The coefficients are set to λ_C = 0.90, λ_F = 0.08, and λ_V = 0.02, prioritizing task correctness while retaining smaller signals for partial answer quality and structural validity.
Experimental Results: In-Domain and Held-Out Benchmarks
RECAST is evaluated across six in-domain benchmark families—DataBench, FinQA, HiTab, HotpotQA, LaMP, and MultiHiertt—and three held-out benchmarks—2WikiMultiHopQA, TAT-QA, and WikiTableQuestions. The in-domain benchmarks span heterogeneous source representations, including dataframes, financial reports, hierarchical tables, Wikipedia passage collections, and user profiles. The held-out benchmarks exclude 2WikiMultiHopQA, TAT-QA, and WikiTableQuestions from SFT, GRPO, and checkpoint selection; 100 examples per benchmark are evaluated.
Comparing RECAST against five baseline families spanning increasingly expressive forms of context construction, the results show that RECAST achieves the strongest overall performance. Direct provides only the task to the AnswerLM without accessing the source; Fixed Retrieval uses the task as the query for a single dense retrieval and returns the top three source chunks; Direct Code generates and executes a single Python program over the source without iterative evidence construction; IRCoT alternates sentence-level reasoning with BGE-M3 retrieval; and Interact-RAG uses a planner–reasoner–executor workflow to formulate retrieval actions and decide when to stop. IRCoT and Interact-RAG are adapted to the per-instance sources and shared model and retrieval components used in the evaluation; both are evaluated with Qwen3.5-9B and Gemini 3.5 Flash as their respective LMs.
Table 1 compares success rates of RECAST and baselines across six benchmarks. RECAST with Qwen3.5-9B (SFT+GRPO) obtains an average success rate of 75.6%, a 15.9% improvement over the strongest large-model baseline, Gemini-based Interact-RAG at 59.7%. The trained Qwen3.5-9B RouterLM also outperforms a training-free Gemini 3.5 Flash RouterLM by 5.0%, achieving 75.6% versus 70.6%. On three held-out benchmarks, RECAST obtains 79.3% compared with 64.3% for the strongest baseline, demonstrating strong zero-shot generalization across tasks and heterogeneous source representations. Re-evaluating predictions with two alternative LLM judges and token-level F1 preserves the overall performance advantage.
The ablation studies reveal three main takeaways. First, the evidence-operation space matters: RECAST train-free with primitive-only operations achieves 56.8% average success, while synthesis-only achieves 59.9%; the full multi-round routing with both primitives and synthesis achieves 64.1%. Second, the GRPO reward and data selection improve over SFT alone: SFT+GRPO achieves 75.6% versus SFT at 67.3% and GRPO alone at 72.6%. Third, the two training stages are complementary: SFT provides the foundation for valid multi-round behavior, while GRPO refines the policy using outcome-based rewards, and their combination yields the strongest results. The trained Qwen3.5-9B RouterLM also generalizes across alternative CompilerLMs, maintaining strong performance when the CompilerLM is replaced with other models.
Ablation Studies: What Drives the Gains
The ablation studies provide granular insight into RECAST's design choices. Removing multi-round routing and reducing the evidence-operation space to primitives only drops the average success rate from 75.6% to 56.8%, a reduction of 18.8 percentage points. Removing synthesis entirely drops it to 59.9%, a reduction of 15.7 points. This indicates that both primitive retrieval operations and the ability to synthesize customized computations are necessary for strong performance, and that their interaction—adaptively choosing between them across rounds—is what enables RECAST to tackle diverse task demands.
Replacing the GRPO reward with answer correctness alone (SFT+GRPO, Accuracy Only) reduces the average success rate to 68.9%, a reduction of 6.7 points. Removing mixed-outcome filtering (SFT+GRPO, Unfiltered) further reduces it to 73.0%, a reduction of 2.6 points. These results confirm that the shaped reward, which combines final-answer correctness, token-level F1, and structural validity, together with the mixed-outcome filtering that compares alternative behaviors for the same question, is essential for guiding the RouterLM toward robust evidence-construction policies.
Evaluating the two training stages separately shows that SFT alone achieves 67.3% and GRPO alone achieves 72.6%. Their combination, SFT+GRPO, achieves 75.6%, confirming that the two stages are complementary: SFT establishes valid multi-round evidence-construction behavior, while GRPO improves the routing policy using downstream task outcomes. The gap between the individual stages and their combination quantifies the additional value of training the RouterLM with both imitation and reinforcement signals.
Limitations and Trade-Offs
The method assumes access to a frozen CompilerLM that can translate synthesized operation specifications into executable Python programs. If the CompilerLM cannot successfully compile or execute the specified program—for example, due to unsupported language features or runtime errors—the evidence-construction loop may fail or produce incomplete evidence. The paper does not ablate the choice of CompilerLM beyond using Gemini 3.5 Flash as the default; alternative CompilerLMs may have different capabilities and failure modes. The evidence-construction loop is limited to six rounds and 4,000 characters of retained evidence, which may be insufficient for very long or very complex source collections. Very complex tasks may require more rounds or a larger character budget, which would increase computational cost. The RouterLM is trained on six in-domain benchmark families and three held-out benchmarks; generalization to entirely new task categories or source types beyond this scope has not been evaluated. The method's performance depends on the quality of the frozen AnswerLM and judge; if these models have weaknesses or biases, the RouterLM may learn to route evidence toward solutions that are correct according to the judge but not necessarily according to human evaluation. The computational cost of multi-round evidence construction is higher than single-step RAG, as each round requires a RouterLM inference, a potential CompilerLM translation and execution, and an AnswerLM invocation at the end; the paper's cost analysis in Appendix D shows that RECAST's additional cost is moderate relative to the performance gains achieved.
What This Means for Retrieval-Augmented Generation Practitioners
For practitioners building retrieval-augmented generation systems that must reason over heterogeneous sources, RECAST demonstrates that fixed similarity-based retrieval is insufficient when the required evidence must be derived through filtering, aggregation, or computation across multiple source items. The RECAST framework provides a practical architecture: a lightweight RouterLM that adaptively selects and formulates evidence operations, a frozen CompilerLM that translates synthesized operations into executable code, and a frozen AnswerLM that produces the final solution. The key insight is that evidence construction can be treated as a learned sequential decision process, where the RouterLM learns when to stop gathering evidence and pass it to the AnswerLM.
The practical workflow for adopting RECAST is as follows. First, ensure the source context can be converted into a uniformly formatted list of records and that a compact source profile can be constructed; the paper's rule-based preprocessor handles this deterministically without requiring a language model. Second, select a RouterLM—Qwen3.5-9B is a strong default—and train it with the two-stage SFT+GRPO pipeline on task-specific data. The training data consists of question–source pairs with reference answers; the SFT stage supervises each RouterLM decision conditioned on the evidence and feedback available at that round, and the GRPO stage reinforces policies that produce better task outcomes. Third, after training, deploy the RECAST system: at inference time, the RouterLM iteratively selects and formulates evidence operations, the CompilerLM translates synthesized operations into Python programs, and the AnswerLM produces the final solution from the accepted evidence. Fourth, evaluate the deployed system not only on in-domain benchmarks but also with held-out tasks to assess zero-shot generalization. The paper shows that training on six in-domain benchmarks yields strong performance on three held-out benchmarks, but practitioners should verify generalization for their specific task distribution.
Developers should also consider the trade-offs. RECAST's computational cost is higher than single-step RAG, as each round requires RouterLM inference and potential CompilerLM execution; however, the paper's cost analysis shows that the additional cost is moderate relative to the performance gains achieved. The six-round limit and 4,000-character evidence budget may constrain very long or very complex source collections; if these limits are reached, the system may need to fall back to alternative strategies. The method's performance depends on the quality of the frozen AnswerLM and judge; if these models have weaknesses or biases, the RouterLM may learn to route evidence toward solutions that are correct according to the judge but not necessarily according to human evaluation. For tasks where the required evidence can be located through fixed similarity-based retrieval, RECAST may provide no benefit over simpler approaches, and the added complexity may not be justified.
For researchers, RECAST opens several directions for future work. The analysis focuses on retrieval and computation operations; extending the framework to other evidence operations such as database queries, API calls, or model-based reasoning would broaden its applicability. The current framework trains the RouterLM on a fixed set of benchmark families; curricula that gradually increase task difficulty or expose the RouterLM to more diverse source types could further improve its generalization. The structural analysis of RouterLM decisions—such as which operation types are selected in which rounds and how evidence accumulation correlates with final correctness—could reveal principles for more efficient evidence construction. Finally, the framework's separation ofRouterLM, CompilerLM, and AnswerLM is modular; replacing any component while preserving the training paradigm should be explored.
Read the paper on arXiv