Why the Wrapper Around the Model Matters More Than You Think
When we talk about autonomous coding agents, the conversation usually centers on the model itself: which foundation model, how many parameters, what training data. But the model is only half the story. The other half is the coding harness, the software layer that decides what tools the model can call, how much of its conversation history it gets to keep, and whether it maintains an explicit plan as it works through a problem.
Change the harness while holding the model fixed and you can substantially change performance. That observation is not new, but existing evaluations typically treat harnesses as monolithic systems. A comparison between SWE-Agent and OpenHands, for example, tells you which one performed better on a benchmark. It does not tell you whether the win came from better context management, a different action interface, the planning scaffold, or some interaction among these components.
This paper from researchers at UMass Amherst, Zoom Video Communications, Emory University, and UNC Charlotte takes a different approach. Instead of comparing complete harnesses, they break the harness into three independently configurable components, planning, action space, and context management, and run a 176-setting ablation across four models and two benchmarks. The result is a component-level map of what each piece of the harness actually does, when it helps, and when it wastes money.
Harness Components Are Not Created Equal
Prior work on coding agents has established that long-horizon software engineering tasks require three capabilities that the raw model does not provide on its own. Planning maintains task structure across many turns, preventing the agent from losing track of what it has tried and what remains. The action space translates the model's intentions into executable workspace operations, whether that means calling specialized file-editing tools or shelling out to bash. Context management decides what part of the growing interaction history survives when the context window fills up.
Most coding agent papers acknowledge these components exist but evaluate them as part of the whole system. Cao et al. (2026) found that Claude-Opus-4.5 works best with OpenHands while Claude-Sonnet-4.5 works best with SWE-Agent, suggesting harness preferences vary across models. But the comparison between complete harnesses conflates multiple mechanisms. A performance gap between two agents could come from any of their many differences.
The gap this paper fills is the absence of controlled, component-level comparisons across both model capability and context-window budget. Prior studies have examined individual components in isolation, but none have jointly characterized planning, workspace action interfaces, and context-management policies on long-horizon coding tasks while sweeping context-window size and model scale.
A Minimal Harness With Three Swappable Parts
The authors built a lightweight coding harness from scratch on LangGraph, with each task running in its own container managed by the Harbor framework. The execution loop follows a standard ReAct pattern: the model thinks, acts, and observes, turn after turn. The key design decision is that this loop stays fixed while three components get swapped in and out.
Planning maintains an explicit, persistent task plan. When enabled, the system prompt instructs the model to create a plan before acting, and the plan gets injected into the model's input on every subsequent turn (without being stored in conversation history). The model can update the plan via an update_plan tool. When planning is disabled, all planning instructions, reminders, plan injections, and the tool itself are removed, while everything else stays the same.
Action space comes in two flavors. The predefined-tool setting exposes eight structured tools: read_file, write_file, edit_file, list_files, glob_files, grep_text, web_fetch, and bash. Each has typed argument schemas and protocol descriptions. The bash-only setting removes all predefined file and search tools, leaving only bash for workspace interaction. Auxiliary tools required by planning and context management (like update_plan and recall_event) remain available in both settings. The predefined tools also enforce read-before-write checks, track file state, and trigger automatic diagnostics (ruff, pyflakes) after edits, so the comparison captures the full effect of the action interface, not just tool count.
Context management uses three composable mechanisms. Elision (M1) replaces the body of stale tool observations with short stubs, reclaiming tokens cheaply. Recall (M2) stores elided observations externally and exposes a recall_event tool for on-demand retrieval, making elision reversible. Summarization (M3) folds older messages into a running natural-language summary via a separate LLM call. These combine into five tiers: T0 (no management, trajectories that overflow the window simply terminate), T1 (elision only), T2 (elision plus recall), T3 (summarization only), and T4 (all three mechanisms, staged so elision happens first at a soft threshold and summarization only fires if the context still exceeds a hard threshold).
The staged ordering in T4 is the key design choice. Elision is cheap (it is a string replacement, no LLM call required) and handles many cases before the more expensive summarization step is needed. The preamble and a budgeted recent window of at least two turns always stay verbatim; only the middle region gets compacted.
Supporting components like workspace guards, post-edit diagnostics, and stuck detection are held fixed across all ablations. Stuck detection monitors for streaks of identical tool calls and injects a reminder after five consecutive identical calls, terminating after eight consecutive identical failing calls.
176 Settings Across Two Benchmarks
The evaluation uses four models: three sizes of Nemotron-3 (30B, 120B, 550B) as a within-family capability axis, and Mistral-Medium-3.5-128B as a cross-family comparison. These are served locally with SGLang in BF16 precision at temperature 0, with output capped at 16,384 tokens per turn. The models function as probes of capability and interaction style, not permanent optimization targets.
Two complementary benchmarks test different aspects of long-horizon coding. SWE-Bench Verified comprises 500 human-verified real GitHub issues requiring repository-level changes. Terminal-Bench 2.1 comprises 89 end-to-end tasks in a command-line environment. Both report task success rate and mean cost per task.
The experimental design covers 20 context-management settings per model-benchmark pair (five tiers at four context-window budgets: 32k, 64k, 96k, 128k), plus two component ablations at T4/128k (planning off, bash-only). That gives 22 settings per model-benchmark pair, 176 total across four models and two benchmarks. Statistical comparisons use two-sided exact McNemar tests with Benjamini-Hochberg correction at q < 0.05.
Context Management Matters Most When the Window Is Small
The headline finding is about when context management pays off. The value of context management, measured as the success-rate gap between managed tiers (T1-T4) and no management (T0), shrinks dramatically as the context-window budget increases.
On SWE-Bench, averaged across all models, the managed-vs-T0 gap goes from 35.7 percentage points at 32k down to 15.9 at 64k, 5.5 at 96k, and 2.7 at 128k. On Terminal-Bench the pattern is similar: 9.5, 7.5, 4.8, and 2.8 points respectively.
The explanation is straightforward and visible in the trajectory data. T0's failure mode at tight budgets is context overflow: the agent runs out of window before it can finish, and the trajectory terminates with an error. At 32k, 78.7% of T0 trajectories on SWE-Bench overflow; at 128k, only 8.7% do. All managed tiers produce zero overflow failures at every budget. Context management does not change what the agent does; it prevents the agent from being cut off mid-execution.
The practical implication is clear. If you are running a model with a large context window (128k or more) and your tasks do not produce extremely long histories, context management provides a small accuracy boost. If your budget is tighter, it becomes essential.
T4: Cheap Elision Before Expensive Summarization
Among the five context-management tiers, T4 delivers the best accuracy-cost trade-off. It achieves success rates comparable to T1, T2, and T3 across all budgets, while producing the lowest mean cost per task in seven of eight model-benchmark panels.
The mechanism is visible in the compression and invocation data. At tight budgets (32k, 64k), T1 and T2 trajectories still reach nearly the full context window because elision replaces content with stubs but does not reduce token count as aggressively. T3 and T4 keep peak context substantially below the nominal budget because summarization compresses the middle region into shorter text. T4 has the lowest average peak-context ratio at all four window budgets.
T4 also invokes summarization less often than T3 at every budget. The early elision step handles many cases before summarization becomes necessary. Since summarization requires an LLM call (the same model under evaluation, called without tool access), avoiding those calls directly reduces cost.
The recall mechanism (M2) tells a different story. T2 differs from T1 only in the availability of recall, and across 32 model-benchmark-window comparisons, T2 outperforms T1 in 15 settings, underperforms in 14, and ties in three. The equal-weight mean difference is -0.36 percentage points. Across 64 T2 and T4 settings, 56.3% never call recall_event at all. The median invocation rate is zero. Recall use is concentrated under the greatest context pressure and almost entirely in Nemotron-3 30B. The strongest models rarely invoke it. Even the heaviest-use configuration (Nemotron-3 30B on Terminal-Bench at 32k under T2) averages 4.3 calls per task and scores 3.4 points below T1.
Lossless retrieval sounds theoretically superior to lossy elision. In practice, models almost never bother to retrieve what was elided. The machinery adds complexity without accuracy.
Planning: Accuracy Scaffold for Weak Models, Cost Saver for Strong Ones
Planning produces opposite effects depending on model capability. For Nemotron-3 30B, the weakest model tested, planning increases success rate by 11.6 points on SWE-Bench and 4.5 points on Terminal-Bench, at higher cost on both. For Nemotron-3 550B and Mistral-Medium-3.5-128B, planning reduces cost by approximately 30% on SWE-Bench, with success-rate changes of -2.0 and -0.4 points respectively.
Trajectory analysis explains the divergence. Planning extends the execution trajectories of Nemotron-3 30B, which would otherwise terminate before attempting a code edit. The planning scaffold keeps the agent organized enough to reach the modification stage. This additional work costs more tokens and turns. For the stronger models, planning removes turns that would have been spent on redundant post-edit verification. The agent checks its work less often, reducing cost with minimal accuracy impact.
On Terminal-Bench, the pattern is similar but less clean. Planning costs more for the 30B model, has mixed effects on the 120B, and saves cost for the 550B. The intermediate model (120B) shows that planning's value at mid-capability is task-type-dependent and not uniformly positive or negative.
The execution statistics quantify these changes. With planning enabled, Nemotron-3 30B on SWE-Bench sees turns increase by 293%, tool calls by 474%, and average input tokens per call by 104%. Mistral-Medium-3.5-128B sees the opposite: turns decrease by 24%, tool calls by 25%, and input tokens by 12%.
Predefined Tools Help Weak Models; Bash-Only Helps Strong Ones
The action-space comparison reveals a similar model-dependent pattern. For Nemotron-3 30B on SWE-Bench, switching from predefined tools to bash-only drops success rate from 25.2% to 10.2%, a 15-point loss. For Nemotron-3 550B, bash-only actually improves success rate from 65.8% to 69.4%, a 3.6-point gain, at roughly half the cost ($1.11 vs $2.33).
On Terminal-Bench, the strongest model (Nemotron-3 550B) also benefits from bash-only, jumping from 44.9% to 50.6%. Mistral-Medium-3.5-128B, a different architecture, shows the opposite: bash-only drops its SWE-Bench score from 68.6% to 45.4%, suggesting that Mistral relies more heavily on structured tool interfaces.
The trajectory-level explanation is about action granularity. Bash-only enables the agent to combine multiple operations in a single shell command. A model with strong bash proficiency can, for example, read a file, search for a pattern, and apply a sed substitution all in one call. The predefined tool set forces smaller, more granular actions, each requiring its own turn. For models that handle bash well, this granularity is overhead. For models that struggle with bash, the structured tools provide scaffolding that compensates for weaker shell skills.
The cost difference is substantial. On SWE-Bench, bash-only cuts Nemotron-3 550B's cost from $2.33 to $1.11 per task, and Mistral's from $3.14 to $1.72. The savings come from fewer interactions and shorter trajectories overall.
Trajectory Analysis: What Each Component Actually Changes
The paper's trajectory-level analysis is what ties the four main findings together into a coherent explanation. Each harness component changes a different property of the execution trajectory:
- Context management extends trajectories without substantially altering agent behavior. The agent does the same things; it just gets to do them for longer before being cut off.
- Planning changes where trajectories stop. For weak models, it keeps them alive long enough to reach the edit phase. For strong models, it removes unnecessary verification turns near the end.
- The action space changes the granularity at which code is written. Predefined tools produce many small actions; bash-only produces fewer, larger actions.
This decomposition explains why the same component can have opposite effects on different models. Context management does not care about model strength; it either prevents overflow or it does not. Planning and the action space interact with model capabilities because they change the structure of the trajectory, and what counts as a good trajectory structure depends on how capable the model is.
What This Means for Harness Builders
The practical guidance is model-aware and budget-aware. If you are running a strong model on a large context window, you can strip the harness down: disable planning (or use it only as a cost optimizer), offer bash-only, and rely on lightweight elision for context management. If you are running a weaker model or operating under tight context budgets, invest in planning, structured tools, and aggressive context management.
The staged approach to context management (T4) is the general recommendation. Elision first, summarization second. The recall mechanism, despite its theoretical appeal, is not worth the implementation complexity. Models do not use it.
The modular evaluation framework the authors provide is also useful beyond these specific findings. The ability to independently vary planning, action space, and context management while holding the execution loop fixed means future harness components can be tested in the same controlled setting. The 176-setting ablation methodology is a template for evaluating any new component without needing to rebuild the entire system.
These findings arrive at a time when coding agent harnesses are proliferating, each making different trade-offs across these exact dimensions. The component-level clarity this paper provides should help harness builders make those trade-offs deliberately rather than by accident.