Long-context reasoning in language models faces a fundamental trade-off. You can process everything at once and pay quadratic attention costs, or you can process text sequentially in chunks, maintaining a fixed-size memory that gets updated as you go. The sequential approach, exemplified by MemAgent, achieves linear complexity but introduces two problems: it cannot parallelize (each step depends on the previous one), and it requires reinforcement learning to train the memory policy, which tends to overfit to the specific dataset it was trained on. ConvMem from the Chinese Academy of Sciences proposes a third option: treat the problem like a convolutional neural network, scanning text segments in parallel through a hierarchical tree structure that compresses the reasoning path from linear to logarithmic depth.
The sequential bottleneck and RL overfitting
MemAgent and similar sequential memory agents read text as a stream, maintaining a fixed-size memory state that evolves step by step. At each step, the agent decides what to keep, overwrite, or discard. This works, but the strict temporal dependency means step t cannot start until step t-1 finishes. For a document with N tokens, this means O(N) sequential LLM calls. The latency scales linearly with document length.
To train these memory policies, researchers use reinforcement learning, optimizing the agent's overwrite decisions through reward signals derived from downstream task accuracy. The paper reveals a critical flaw in this approach: RL-trained agents overfit to their training distribution. On RULER-HotpotQA, the dataset MemAgent was trained on, MemAgent achieves 75.55 F1 at 28k tokens and maintains 68.80 at 896k. But on RULER-2WikiMultiHopQA, an out-of-distribution dataset constructed with the same protocol, MemAgent's F1 drops to 60.92 at 28k and 58.41 at 896k. The gap between in-distribution and OOD performance reveals that the RL training has taught the agent to rely on parametric memorization rather than genuine in-context reasoning.
The paper demonstrates this with a concrete case. A query asks which filmmaker was known for animation: Lev Yilmaz or Pamela B. Green. The context explicitly states "Lev Yilmaz." The dataset ground truth contains a typo: "Levni Yilmaz." MemAgent outputs "Levni Yilmaz," matching the memorized label but contradicting the provided context. ConvMem outputs "Lev Yilmaz," faithfully extracting from the context. The RL agent has learned to reproduce training labels rather than reason over evidence.
Reframing reasoning as convolution
ConvMem draws an analogy between long-context reasoning and convolutional neural networks. In a CNN, a kernel slides over an image, performing local operations that are combined hierarchically. ConvMem treats a frozen LLM prompted with a specific query as a semantic convolutional kernel. This kernel takes a text segment and produces a condensed summary plus a relevance score (0 for irrelevant, 1 for relevant, 2 for critical). The kernel is not a neural network layer in the traditional sense. It is an LLM call with a specific prompt that extracts and compresses information relevant to the query.
The document is sliced into overlapping segments of W = 8,000 tokens with a stride of S = 1,600 tokens, meaning each token is scanned five times from different window positions. All segments at a given layer are processed in parallel, producing M summaries. These summaries are concatenated and fed as input to the next layer, where the same kernel operation repeats. The process continues recursively until the total summary length fits within the kernel size, typically requiring 2-4 layers for contexts up to 128k tokens and 3-4 layers for contexts up to 1 million tokens.
The key structural difference from sequential agents is the information flow. Sequential agents process tokens in a linear chain: O(N) depth. ConvMem processes them in a binary tree: O(log N) depth. Each layer halves the effective sequence length through semantic compression, with an average compression factor alpha. This means the reasoning path from any token to the final answer passes through only log(N) kernel calls rather than N.
Three mechanisms for faithful evidence capture
Configurable strides address boundary truncation. When text is sliced into non-overlapping chunks, semantic dependencies that span chunk boundaries get disrupted. With stride S < W, tokens at the edge of one segment appear near the center of the next, preserving their contextual integrity. The ablation study confirms this: single-pass scanning (S = W) produces a noticeable performance drop, while 5x over-scanning (S = 1600, W = 8000) achieves the optimal balance between coverage and noise.
Skip connections prevent detail loss during hierarchical compression. When the kernel assigns a relevance score of 2 (critical) to a segment, that raw segment bypasses all summarization layers and is stored in a residual buffer. The final answer generation step receives both the high-level global summary and these unmodified raw segments, allowing it to reason over both abstract patterns and precise details. The ablation shows that removing skip connections degrades fine-grained entity retrieval, confirming that hierarchical summarization alone loses too much detail.
Multi-kernel convolution handles multi-hop reasoning. A query like "What government position was held by the woman who portrayed Corliss Archer in the film Kiss and Tell?" requires tracking two distinct entities (Shirley Temple and the film Kiss and Tell) across different parts of the document. ConvMem decomposes the query into C sub-questions (typically 2-4, capped at 5), each driving its own kernel that scans the document in parallel. The sub-questions are disentangled into separate semantic channels, preventing interference between reasoning paths. The ablation confirms that single-kernel models conflate distinct threads, while multi-kernel convolution significantly boosts multi-hop accuracy.
Results: outperforming RL without any training
On RULER-HotpotQA (in-distribution for MemAgent), MemAgent still leads with 75.55 F1 at 28k tokens versus ConvMem's 67.44. But at 896k tokens, ConvMem closes the gap to 63.09 versus MemAgent's 68.80. The RL-trained agent maintains its advantage on its training distribution, as expected.
On RULER-2WikiMultiHopQA (OOD), the story reverses. ConvMem achieves 72.30 F1 at 28k tokens versus MemAgent's 60.92, and 59.06 at 896k versus MemAgent's 58.41. ConvMem outperforms MemAgent at every context length on this OOD benchmark. The Sub-EM scores are even more striking: ConvMem reaches 82.81 at 28k versus MemAgent's 70.31, and maintains 70.62 at 896k versus MemAgent's 69.31. The training-free method generalizes better than the RL-trained specialist when the data distribution shifts.
Compared to other training-free baselines, ConvMem dominates. MemAgent-W/O-RL (the sequential agent without RL training) achieves 63.95 F1 at 28k on HotpotQA but degrades to 58.73 at 896k, and on 2WikiMultiHopQA ranges from 60.51 to 49.65. RAG-BM25 performs poorly on multi-hop tasks (34.54 to 17.34 on 2WikiMultiHopQA) because it retrieves isolated chunks that lack global connectivity. The vanilla Qwen2.5-32B-Instruct drops from 61.90 to 16.78 on HotpotQA across context lengths, demonstrating the lost-in-the-middle phenomenon that ConvMem is designed to address.
Mem-alpha (both trained and untrained variants) shows extremely low F1 scores (single digits) across all settings, though its Sub-EM scores are more reasonable. The paper attributes this to the model producing excessively verbose answers that hurt precision metrics while containing partially correct content.
Model-agnostic gains across scales
ConvMem is training-free, so it can be applied to any backbone LLM without adaptation. The paper tests Qwen2.5-7B-Instruct, Qwen2.5-32B-Instruct, and Qwen2.5-72B-Instruct, and finds that ConvMem yields consistent performance improvements across all three sizes. The framework effectively scales the long-context reasoning of diverse LLMs without requiring any parameter updates or task-specific fine-tuning.
This model-agnostic property is a practical advantage. A developer can swap in a larger backbone and immediately get better long-context performance without retraining. The computational cost is higher than sequential scanning (multiple parallel passes over the text), but the latency is logarithmic rather than linear, and the reasoning fidelity improves because errors do not accumulate across a long chain of sequential steps.
Limitations and the token cost trade-off
The paper acknowledges two limitations. First, total computational cost is higher than sequential methods. Each layer processes the full set of overlapping segments in parallel, and there are multiple layers. A document scanned with 5x over-scanning through 3-4 layers requires substantially more total tokens than a single sequential pass. The payoff is lower latency (logarithmic scaling) and better accuracy (shorter reasoning paths), but the total FLOPs increase.
Second, the system depends on the quality of query decomposition. If the initial decomposition produces flawed or incomplete sub-questions, the downstream kernels operate on incomplete premises. The upper limit of 5 channels prevents explosion, but a bad decomposition at the top propagates through the entire hierarchy. This is a structural dependency that could be mitigated by ensemble decomposition or iterative refinement, but the current implementation relies on a single decomposition pass.
For practitioners, the choice between ConvMem and sequential agents comes down to the deployment constraints. If latency matters and the context is very long, ConvMem's parallel tree structure wins. If the training distribution is well-matched and RL training is acceptable, MemAgent still performs well in-domain. If generalization across distributions matters, ConvMem's training-free approach avoids the overfitting trap that RL introduces. The framework provides a new architectural direction for long-context reasoning that trades raw compute for parallelism and fidelity, without requiring any training data or reward engineering.