Memory has become integral to the LLM agent ecosystem, supporting information retention and reuse across interactions. However, most existing agent memory systems construct memory in a query-agnostic manner, which can incur unnecessary preprocessing cost and discard details that later prove essential. Recent studies have begun shifting memory processing toward runtime adaptation, but typically specialize in particular operations or fixed processing schemes, leaving flexible control over performance, cost, and latency largely underexplored. To address this challenge, we present MemPilot, a flexible framework that orchestrates on-demand memory curation under different performance--cost--latency preferences. Specifically, we optimize a multi-step LLM policy via reinforcement learning to iteratively choose between retrieving from query-agnostic memory and delegating query-specific curation of raw multimodal history to heterogeneous LLMs and VLMs. The policy jointly controls evidence amount, curation instructions, model selection, and visual access, enabling fine-grained allocation of runtime computation. To optimize this policy under competing objectives, we adapt objective-wise advantage decoupling by separately estimating each objective's advantage before aggregation. Moreover, we introduce prefix-based marginal utility estimation for fine-grained credit assignment across multi-step rollouts. Experiments on five multimodal agent-memory benchmarks demonstrate favorable performance--cost--latency trade-offs across optimization preferences, with preference sweeps yielding broader frontiers than existing trade-off-aware baselines.
Where prior memory systems fall short
LLM agents increasingly rely on memory mechanisms to preserve and reuse information from long-term interaction histories. Most existing text-centric systems construct memory before future queries are known, using either hand-designed mechanisms for extraction and organization or learned policies for memory construction, updating, and retrieval. Beyond text-centric memory, recent work has extended agent memory to multimodal interactions, maintaining structured representations over textual, visual, and other sensory experiences. Despite increasingly adaptive and multimodal memory designs, they still construct memory in a largely query-agnostic manner.
A growing line of work instead shifts memory processing toward runtime adaptation. GAM performs just-in-time research over stored memory and raw history, while BudgetMem learns query-aware runtime extraction under different computation budgets. MemGen and AdaMEM dynamically generate or refresh memory guidance according to evolving reasoning states, whereas MemRL adapts the utility of stored experiences from interaction feedback. More recent work further explores query-time evidence distillation and deferred memory construction. However, these approaches typically specialize in a particular form of runtime adaptation or follow a predefined processing pattern. MemPilot instead learns to orchestrate heterogeneous multimodal memory curation at runtime under controllable performance--cost--latency preferences.
Dual-view memory representation
To support flexible runtime adaptation while retaining efficient access to query-agnostic memory, MemPilot maintains two complementary views of the interaction history: a query-agnostic memory bank and the raw multimodal history. In this work, the query-agnostic memory bank can be initialized with memories constructed by any off-the-shelf agent memory system or a simpler memory construction scheme, keeping memory construction orthogonal to MemPilot's runtime orchestration and ensuring broad compatibility with existing memory systems. Meanwhile, the raw multimodal history is segmented into fixed-length token chunks while preserving the original multimodal context within each chunk. These two views serve complementary roles: the former enables efficient memory access, whereas the latter retains fine-grained information that may be absent from the query-agnostic memory representation.
Sequential decision formulation
Given a query, MemPilot formulates on-demand memory curation as a multi-step decision process governed by an orchestrator policy. At step t, the orchestrator conditions on the query and previously acquired evidence, and produces a structured memory action. Specifically, the action type z_t denotes whether to Retrieve from the query-agnostic memory bank or to Curate the raw history. The Retrieve action directly accesses compressed memory, while Curate performs query-specific runtime curation over the raw history, including evidence retrieval and subsequent processing. After a variable number of steps, the policy terminates memory curation and produces the final answer. This process yields a query-adaptive curation trajectory with at most T_max steps, whose behavior can vary substantially across queries and optimization preferences.
Factorized memory curation and delegation
We factorize the action arguments to expose fine-grained control over runtime memory curation. Rather than relying on a fixed processing pipeline or predefined budget tiers, MemPilot factorizes each action into several controllable dimensions, allowing the orchestrator to adapt what evidence to retrieve, how much to retrieve, how it should be curated, which model to invoke, and whether visual evidence is needed. These decisions jointly provide flexible control over runtime computation and its resulting quality--cost--latency trade-offs.
For the Retrieve action, the action arguments specify the retrieval query and the evidence count, allowing the orchestrator to adapt retrieval depth across steps rather than relying on a fixed top-k. For the Curate action, it further specifies a query-specific curation instruction, selects the processing backend, and determines whether original visual evidence is included. Importantly, we decouple what to retrieve from how to curate it, avoiding conflating two semantically distinct operations in a single textual field while keeping retrieval evidence-focused and curation task-directed.
Beyond deciding what evidence to retrieve and how to curate it, MemPilot also adapts which model performs the runtime curation, since heterogeneous models exhibit distinct capabilities, monetary costs, and latencies, making model choice a natural and effective control dimension for runtime trade-offs. MemPilot therefore delegates runtime curation to a configurable pool of LLMs and VLMs, allowing the orchestrator to adaptively select a model for each curation step under different optimization preferences. For vision-capable routes, v_t further determines whether raw images should be included in the curation request. Together, these factorized decisions allow each query to induce its own curation trajectory over evidence depth, processing strategy, model capacity, and modality access, providing flexible trade-off control without requiring a rigid processing pipeline or predefined budget tiers.
Resource-aware reward modeling
Having exposed multiple controls over runtime curation, we train the orchestrator to use them according to different optimization preferences. For a preference vector with quality weight set to 1 throughout this work, we optimize the policy with Group Relative Policy Optimization over answer quality, monetary cost, and latency.
For each query, the policy samples a group of curation trajectories and evaluates each trajectory with three objective-specific rewards: the task-specific answer reward, the negative monetary cost, and the negative latency. Monetary cost accounts for token-based inference costs incurred by both the orchestrator and delegated external models. Latency similarly considers both sources of runtime delay, using a proxy estimator to ensure training stability.
Since directly combining multiple rewards before group normalization can obscure their individual learning signals, we instead normalize each objective independently within the rollout group. We then aggregate the normalized advantages according to the preference vector and further normalize the combined signal at the batch level. The resulting advantage is used in the clipped GRPO loss, yielding policies with different quality--cost--latency trade-offs while preserving objective-specific learning signals.
Trajectory-level advantages provide the same learning signal across curation stages, obscuring their individual contributions. We therefore probe the rollout after each stage by truncating subsequent curation and directly generating an answer. This estimates the marginal answer-quality improvement contributed by that stage, and we use this signal to refine credit assignment for the policy decisions associated with that stage. Other tokens retain the trajectory-level advantage, and the stage-refined advantage replaces it in the loss function for tokens belonging to the curation response, incurring little overhead due to short answer completions and largely shared prefix KV-cache reuse.
Experimental setup and benchmarks
We mainly evaluate MemPilot in multimodal settings involving both textual and visual interaction histories. Specifically, we use five multimodal agent-memory benchmarks: Mem-Gallery, WorldMemArena, H2HMem, MemEye, and MemLens, with MemEye and MemLens serving as out-of-distribution evaluation sets. We evaluate answer quality using F1 score and an LLM-judge score, with GPT-4o-mini as the judge model. Cost measures the total token-based monetary cost incurred by both the orchestrator and delegated external models. Latency measures their total inference latency in seconds. The same data protocol is applied to all applicable baselines, with detailed dataset statistics provided in the appendix. For the main comparison, we evaluate three representative MemPilot preferences: Perf, Bal, and Cost, prioritizing answer quality, balanced quality--resource trade-offs, and inference cost, respectively.
We merge the training portions of three of the benchmarks into a joint training set and evaluate in-domain performance on their held-out test sets. We further use the two out-of-distribution benchmarks exclusively for out-of-domain evaluation to assess generalization to unseen memory distributions. The same data protocol is applied to all applicable baselines. For training, we adopt a 7/1/2 train/validation/test split. Unless otherwise specified, the query-agnostic memory bank is constructed with LLMLingua-2 at a 0.4 compression ratio for policy training and main evaluation. We retrieve at most five evidence items per step for both MemPilot and applicable baselines. Since memory items from different systems can vary substantially in length, we directly compare monetary cost and latency rather than memory-item counts.
Main results across quality-cost regimes
MemPilot spans distinct quality--cost regimes across both models and benchmarks. The results reveal several consistent trends across methods and settings. Text-centric systems achieve low cost largely by avoiding additional multimodal processing and runtime curation, but this often comes with lower answer quality. In comparison, MemPilot-Bal generally achieves stronger overall quality at moderate cost, while MemPilot-Cost reaches similarly low-cost regimes on most datasets. Runtime adaptation does not necessarily translate into controllable trade-offs. Although some prior methods adapt memory processing at inference time, they largely expose a fixed operating regime rather than explicitly controlling the quality--cost balance. In contrast, MemPilot spans multiple operating points. MemPilot-Perf achieves the best LLM-judge score across all five benchmarks under both models, while MemPilot-Bal substantially lowers cost and provides a distinct intermediate quality--cost trade-off. Multimodal memory processing can be costly without adaptive allocation. Some multimodal systems incur substantially higher inference cost without consistently translating that expense into better answer quality. In contrast, MemPilot selectively invokes external curation and achieves stronger performance at much lower cost in most settings, with the same trend extending to the out-of-distribution evaluation sets.
Overall, MemPilot provides controllable quality--cost trade-offs rather than a single performance-oriented operating point. MemPilot-Perf prioritizes answer quality, MemPilot-Bal offers a balanced operating point, and MemPilot-Cost further reduces resource usage with greater quality sacrifice. We analyze latency and finer-grained performance--cost--latency frontiers in the subsequent section.
Performance--cost--latency frontiers
We further examine fine-grained trade-off frontiers by sweeping MemPilot's optimization preferences. In addition to the main baselines, we include methods that explicitly support performance--efficiency trade-offs. For latency, we report representative fixed-inference baselines together with runtime-adaptive methods, since most query-agnostic systems exhibit nearly unchanged inference behavior and collapse to similar single operating points.
MemPilot forms a stronger frontier across a broad cost range and largely envelopes the competing methods. Compared with trade-off-aware baselines, MemPilot reaches higher quality as additional computation is allocated, while also covering competitive low-cost operating points. Most conventional memory systems expose only a fixed quality--cost point, highlighting the additional flexibility provided by learned runtime orchestration. Multimodal systems such as some prior methods generally incur much higher cost due to visual processing, whereas MemPilot reduces unnecessary multimodal computation through on-demand curation and selective model delegation.
For latency, MemPilot spans lightweight to performance-oriented configurations, whereas most baselines expose only a narrow region or a single operating point. Even among runtime-adaptive methods, the achievable latency range remains limited. This further highlights that cost and latency capture complementary resource constraints. MemPilot provides a broader, more controllable performance--cost--latency frontier through adaptive runtime orchestration, enabling flexible computation allocation under resource preferences.
How MemPilot allocates runtime computation
To better understand how MemPilot realizes different trade-offs, we analyze its learned runtime curation behavior. As the optimization preference shifts toward lower cost or lower latency, the policy adjusts multiple control dimensions in a coordinated manner rather than relying on a single knob. Moving from left to right, MemPilot generally reduces the number of policy steps, the frequency of Curate actions, image access, and policy output tokens. At the same time, evidence count also changes adaptively, indicating that the policy does not simply retrieve less information, but learns to reconfigure how memory is accessed and processed under different resource preferences.
Model selection is a key mechanism behind this adaptation. Under higher-cost settings, MemPilot relies more heavily on stronger VLMs and LLMs, while lower-cost settings shift computation toward cheaper model groups. Together, these results show that MemPilot learns a coherent allocation strategy over curation depth, modality access, and routed model capacity, which explains its ability to realize controllable performance--cost--latency trade-offs.
Ablation study
An ablation study evaluates the contribution of each component. Removing LLM/VLM delegation degrades both answer quality and cost efficiency. Omitting the marginal utility estimation similarly reduces quality and increases cost. Without the preference-optimized objective, performance drops significantly. Removing the retrieval-instruction decoupling also hurts both objectives. These results confirm that each design choice in MemPilot contributes meaningfully to the overall system effectiveness.
Limitations and future work
The framework relies on the availability of heterogeneous model pools through external APIs, which may introduce latency variability depending on server load and pricing tiers. The prefix-based marginal utility estimation focuses only on answer quality to avoid double-counting resource penalties, which means cost and latency contributions at individual stages are approximated rather than precisely measured. The current implementation assumes that the query-agnostic memory bank is provided by an external system, and end-to-end learning of memory construction alongside curation orchestration remains an open direction. Additional benchmarks spanning other modalities beyond text and imagery, such as audio or sensor data, would help assess the generality of the orchestration framework. Finally, the analysis primarily considers offline evaluation with fixed model pools; online model discovery and dynamic pool expansion are natural extensions for real-world deployment.
Broader impact
MemPilot's on-demand memory curation framework enables more efficient deployment of LLM agents in resource-constrained settings, such as mobile devices or edge environments where computation budget and latency are at a premium. By allowing flexible trade-off control, the framework empower developers to tune agent behavior according to specific application requirements, whether prioritizing answer accuracy for critical decision-support tasks or minimizing operational cost for high-volume query workloads. The release of code and model checkpoints fosters further research into runtime adaptation for agent memory systems. Potential misuse concerns are mitigated by the framework's inherent cost-awareness, which makes aggressive computation allocation easily observable and controllable by system operators.
Code and model checkpoints are released to facilitate reproducibility and further research. The framework is implemented based on the RL infrastructure, with the heterogeneous model pool specified through anonymized model IDs and lightweight profiles. Additional implementation details, hyperparameter configurations, and per-benchmark results are provided in the appendices.