Why Troubleshooting Agents Need Stateful Retrieval, Not Static Document Lookup

Enterprise customer support tickets do not resolve in a single shot. They unfold across stages: a symptom report, hypothesis formation, iterative log inspection, root cause identification, and final remediation. When an AI agent assists a support engineer, its ability to find relevant historical cases depends on understanding where the investigation currently stands, not just what the original error message said. Existing retrieval-augmented generation (RAG) systems treat each closed case as a flat, static document. They ignore the multi-stage nature of troubleshooting entirely, returning incoherent chunks or irrelevant matches that waste the agent's context budget and degrade reasoning.

RAFT, introduced by researchers at Microsoft and accepted to EMNLP 2026's Industry Track, addresses this gap with a stateful retrieval architecture designed specifically for troubleshooting histories. The core idea is straightforward but powerful: decompose each historical case into a directed chain of timeline entries representing distinct investigative states, then retrieve at the entry level so that cases whose intermediate states match the active case are surfaced with their full trajectory anchored at the matched point.

The Gap Between Document RAG and Troubleshooting RAG

Standard RAG pipelines chunk documents, embed them, and retrieve the nearest matches to a query. GraphRAG variants add entity-relation graphs extracted by LLMs to guide retrieval. Both approaches work reasonably well for general question answering over static corpora. But troubleshooting cases violate every assumption these systems make.

The problems are concrete. Raw ticket data interleaves substantial non-technical noise (customer pleasantries, administrative notes, auto-generated system messages) with diagnostically relevant details. The signal is scattered across many turns. When vanilla RAG returns chunks from a similar case, the agent receives isolated fragments without context, making it impossible to judge whether the case is truly relevant or to learn from its resolution trajectory. Many closed tickets end without any meaningful investigation, closed due to customer unresponsiveness or administrative reasons, yet RAG treats them identically to cases with complete diagnostic evidence. Enterprise data also carries privacy constraints that require abstraction or redaction before indexing.

GraphRAG methods partially address coherence by returning chunks with entity relationships, but their entity extraction is unstable and adds significant indexing cost. More fundamentally, entity-centric graphs do not represent the progression of individual investigations. A case where a DNS misconfiguration was diagnosed through three rounds of hypothesis testing and a case where the same root cause was identified through log analysis share entities but differ in the investigative states that matter for an agent currently mid-investigation.

How RAFT Works: Two-Level Case Representation

RAFT operates at two levels: per-case extracted resolution trajectories and an optional case-level graph.

Indexing: From Raw Tickets to Timeline Chains

The indexing pipeline processes each closed case independently. An agent-based extraction workflow takes the raw, time-ordered turns (emails, notes, logs) and produces a structured representation containing: the reviewer assessment, a chronological timeline of entries, the documented root cause, resolution steps, and troubleshooting-relevant entities.

Each timeline entry distills a contiguous segment of the case history that captures a meaningful stage of the investigation. A new entry begins at each meaningful state transition: the opening symptom report, a hypothesis being added or discarded, the root cause being confirmed, a resolution being proposed and verified. Minor updates and acknowledgments that add no new insight are absorbed into the current entry. This segmentation principle keeps the timeline compact while ensuring every entry carries actionable information.

The extraction workflow handles long histories through bounded, incremental processing. Workers process the ordered artifacts in batches, carrying the evolving case state into each subsequent pass. Each worker receives the next batch alongside accumulated state, adding new findings or revising earlier interpretations as evidence develops. A reviewer then checks and refines the completed state, consulting source evidence and revision history to resolve inconsistencies. This design accommodates case histories that exceed a single context window.

An actionability filter based on the reviewer assessment excludes non-actionable cases from the index, solving the problem of tickets that closed without meaningful investigation.

Entry-Level Retrieval

The retrieval algorithm is where RAFT departs most sharply from standard approaches. Instead of embedding and retrieving whole cases or fixed-size chunks, RAFT embeds every timeline entry across all indexed cases. Given a query reflecting the current state of an active case, the system computes a hybrid score combining semantic similarity and BM25 lexical similarity via Reciprocal Rank Fusion, then ranks entries from the remaining cases.

For each high-scoring entry, the system promotes it to its parent case, greedily selecting up to n distinct cases within a token budget. Each returned result includes both the full case representation and the specific timeline entry that triggered the match. This lets the agent see exactly which investigation state was matched, not just which case was returned.

The key insight is that the match point shifts as the investigation progresses. Early queries carrying only surface symptoms match the opening entries of past cases. Later queries, enriched with diagnostic context, align with corresponding intermediate states rather than re-matching symptoms. This is precisely the behavior the authors validate in their experiments.

Optional Case-Level Graph

An optional undirected graph links stored cases through configurable similarity representations. In the authors' experiments, root-cause and resolution texts are concatenated and scored using the same hybrid semantic-plus-lexical approach, with each case connected to its top-k neighbors. Shared-nearest-neighbor weights provide principled edge weighting. This graph supports expansion beyond initial entry-level matches, surfacing cases that share an underlying cause or remediation strategy despite exhibiting different symptoms or intermediate states.

Evaluation: A Synthetic Benchmark and Real Jira Issues

Evaluating troubleshooting retrieval requires multi-stage histories with labels identifying similar cases, and suitable public data is extremely rare. The authors build two evaluation datasets.

Synthetic Benchmark from Microsoft Learn

The synthetic corpus is grounded in Microsoft Learn Windows Server troubleshooting documentation. The synthesis pipeline organizes source articles into a structured wiki, then generates 2 to 4 support cases per documented root cause, injecting context from related articles to produce realistic diagnostic ambiguity. The final benchmark contains 826 cases across seven categories: Active Directory, Windows Security, Remote Desktop, Group Policy, Licensing and Activation, Networking, and Backup and Storage. Cases average 10.4 messages and 2,767 tokens.

For evaluation, one case per root-cause group is held out as the test query, with the remaining cases forming the indexed corpus. Queries are constructed from prefixes of each test case at three progress points: 0% (initial symptom only), 30%, and 60% of turns.

Apache Jira Transfer Evaluation

Real Apache Jira issues provide a transfer test. The evaluation set contains 30 audited duplicate groups where engineers linked issues in their normal workflow, evaluated against 570 additional resolved issues as distractors. The authors apply RAFT with no modifications from the synthetic experiments: same extraction prompt, schema, models, and retrieval procedure.

Results on the Synthetic Benchmark

RAFT achieves the best scores on every metric at every progress level. The primary metric, Case Hit, measures whether at least one retrieved passage belongs to a ground-truth similar case sharing the same root cause and resolution.

At 0% progress (initial symptom only), RAFT achieves 84.2% Case Hit compared to 67.3% for vanilla RAG and 65.0% for HippoRAG2. At 30% progress, RAFT reaches 87.1% versus 71.9% for vanilla RAG. At 60%, RAFT hits 88.8% versus 76.9%. These gains are statistically significant at all three progress points, validated via bootstrap resampling clustered by root-cause group.

Root Cause Coverage and Resolution Steps Coverage, measuring how much of the gold diagnostic and remediation evidence the retrieved context supports, also favor RAFT, though the margins are smaller. RAFT reaches 0.649 Root Cause Coverage at 0% versus 0.597 for vanilla RAG, and 0.563 Resolution Steps Coverage at 0% versus 0.528.

GraphRAG baselines underperform vanilla RAG. Fast-GraphRAG achieves only 0.421 Case Hit at 0%, and HippoRAG2 reaches 0.650. This aligns with findings that complex entity extraction and graph reasoning offer little benefit when the task requires surfacing similar cases rather than hierarchical knowledge retrieval or deep contextual reasoning across documents.

Why Entry-Level Retrieval Works

The authors analyze where within matched cases the retrieval hit occurs. At 0% progress, the matched entry sits at 9.1% depth (near the beginning of the extracted case). At 30% progress, it moves to 20.0%. At 60%, it reaches 54.0%. This confirms that the retrieval is state-aware: early queries match opening entries, later queries match deeper investigative states.

RAFT also shows greater robustness to noisy queries. When test queries are perturbed with off-topic content, typos, and dropout, RAFT degrades less than vanilla RAG.

Transfer to Real Jira Issues

On the Apache Jira evaluation set, RAFT improves Case Hit by +16.7, +17.3, and +10.5 percentage points at the 0%, 30%, and 60% progress points respectively, using the identical pipeline from the synthetic experiments. This provides directional evidence that the stateful retrieval advantage transfers to real case histories, though the 30-group evaluation set carries no confidence intervals and should be treated as preliminary.

Limitations and Trade-offs

The authors acknowledge three limitations. The main evaluation uses a synthetic dataset of moderate scale, while production corpora typically contain far more cases with substantially higher token counts per case. The Apache Jira evaluation comprises only 30 audited duplicate groups, limiting statistical power. And this work scopes to the retrieval layer only, not evaluating final diagnosis, resolution success, engineer productivity, or other end-to-end troubleshooting outcomes.

The extraction pipeline introduces its own cost. Each historical case requires agent-based processing with bounded incremental passes, followed by reviewer assessment. For large corpora, this indexing step is non-trivial, though it runs once per case rather than at query time. The hybrid scoring adds latency compared to pure semantic search, though the BM25 component is computationally inexpensive.

The case-level graph adds an optional second retrieval path, but the authors do not evaluate it in isolation. Graph construction requires pairwise similarity computation over all indexed cases, which scales quadratically without approximate nearest neighbor techniques. The configurable linking representation (root cause, summary, or other fields) introduces a design decision that may require tuning per deployment.

What It Means in Practice

For teams building troubleshooting agents, RAFT suggests a concrete architectural pattern. Instead of dumping historical cases into a vector store as flat documents, extract structured timelines that capture the progression of each investigation. Retrieve at the entry level to match the active case's current state, and return the full trajectory anchored at the matched point so the agent sees both tactical next steps and the broader resolution path.

The actionability filter is a practical necessity that most generic RAG systems overlook. In any real support corpus, a meaningful fraction of closed tickets contain no useful diagnostic evidence. Filtering these at index time improves retrieval quality without any model changes.

The evaluation approach is itself useful: measuring retrieval independently of a full agent system removes the need for production deployment and makes cross-system comparison reproducible. Teams can benchmark their retrieval layer against the released synthetic dataset and Apache Jira evaluation set without access to proprietary support data.

The released artifacts, including the benchmark, implementation, and Apache Jira evaluation set at github.com/microsoft/RAFT, provide a starting point for teams working on enterprise support agents. The EMNLP 2026 Industry Track acceptance signals that the approach has been vetted by reviewers who understand practical deployment constraints.

RAFT does not claim to solve troubleshooting end-to-end. It addresses a specific, measurable bottleneck: getting the right historical evidence to the agent at the right stage of an investigation. That narrow focus, combined with concrete gains over established baselines, makes it a useful contribution for anyone building retrieval systems over stateful, multi-stage domain data.

Read the paper on arXiv