Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation

The Silent Bias: Why Your Local Agent Benchmarks May Be Measuring the Server, Not the Model

When a coding agent attempts to call a tool, the request must first pass through a serving layer before the model's weights are ever consulted. That layer decides whether the request is accepted, rejected, or silently transformed. A new study by Lijuan Tang and Yuemeng Zheng of Northeastern University demonstrates that this serving layer can systematically contaminate tool-use evaluation, producing measurements that reflect server configuration rather than model capability. The paper, accepted at the 2nd Workshop for Research on Agent Language Models (REALM) at EMNLP 2026, exposes a measurement validity threat that is easy to miss precisely because the contaminated numbers look like ordinary results.

The Protocol Step That Precedes the Semantic Step

Before a coding agent can execute a tool call, it must emit a valid parseable invocation of that tool in the provided schema. This is a protocol step, entirely separate from the harder semantic problem of choosing the right tool. The researchers set out to measure how often this protocol step succeeds on small local models. They used an off-the-shelf ReAct coding-agent harness, an extension of LOCA-bench, on a fixed aggregation task requiring several tool calls. Every assistant turn was labeled as a valid in-schema call, a hallucinated call (parseable but referencing a tool not in the schema), unparseable text, no-call prose, or a non-response.

What they found was not a measurement of model behavior. It was a measurement of the serving stack.

How Ollama Silently Rejects Entire Models

The core mechanism is deceptively simple. In the OpenAI-compatible protocol that local servers expose, a tool-call request carries a tools array of function specifications alongside the messages. The model never receives this array directly. The serving layer must render the specifications into the prompt using the model's chat template, then parse the generated text back into a structured tool_calls object. Both steps happen outside the model weights. Whether a server performs them is determined by the model's template and the server's launch configuration.

In Ollama, this is exposed per model as a capabilities list through /api/show, and the tools parameter is documented as usable only "if supported." On a fixed stack running Ollama 0.30.8, the researchers observed three distinct behaviors for the identical request:

• Qwen2.5-Coder (all sizes) accepted the request and returned calls as text.
• Llama-3.2 accepted the request and returned native tool_calls.
• Phi-3 and Gemma-3 were rejected with an HTTP 400 error reading "does not support tools." The model never ran.

The harness retries, exhausts its retries, and writes a generic error string into the trajectory as an assistant turn. The error type is not preserved as structured metadata. A downstream per-turn analysis that reads only the message stream sees an ordinary non-call turn and attributes it to the model. A capable model is then reported at 0% fidelity.

What a Control Arm Reveals About Prompt Versus Channel

To separate the serving channel from the prompt content, the researchers designed a control arm they call native+hint. The tools parameter is still sent to the server, but a plain-text tool list and an explicit JSON call format with an allowed-name list are also injected into the prompt. This isolates prompt guidance from the serving channel. A third condition, text-tools, drops tools entirely and relies on the same text guidance in the prompt, with calls parsed from text.

For every model the server accepts, adding the text hint while keeping the native channel substantially raised per-seed fidelity. Qwen2.5-Coder 0.5B through 14B went from 0%, 38%, 23%, 59%, and 60% respectively under native mode to 35%, 58%, 82%, 89%, and 80% under native+hint. The low native numbers reflected a default call path without explicit format guidance, not model inability. The native+hint configuration was close to the uniform text-tools rate for most models.

Llama-3.2 was the informative exception. Its native+hint fidelity reached 82%, but the uniform text-tools protocol dropped it to 44%. Llama-3.2 has real native tool-call support that the text protocol discards. The best-performing interface is model-dependent, meaning no single serving configuration maximizes measured fidelity across all models.

Four Stacks, Four Different Answers

The most striking result came from cross-stack probes. The same GGUF weights that Ollama rejected with HTTP 400 ran when served by llama.cpp: Phi-3 and Gemma-3 both returned a text response under an identical tools request, while Qwen-0.5B's text call and Llama-3.2's native call were reproduced on both stacks.

Three other stacks behaved differently still:

• vLLM refused the tools request outright with HTTP 400 by default, regardless of model, until launched with --enable-auto-tool-choice and a --tool-call-parser. Even then, for the two small models probed, neither produced a native tool_call.
• SGLang accepted the request by default but returned the call as text rather than a native tool_call unless launched with --tool-call-parser.

Whether a request is refused and whether a call is recognized once accepted are governed by the serving stack and its launch configuration, not by the model. The same identity produces different measurement outcomes depending on which server software is running it.

The Fidelity Number Is Not Robust

The researchers also documented a statistical fragility that compounds the serving-layer confound. Turn-pooled and per-seed rates diverge sharply when a model produces one long looping episode. Qwen2.5-Coder 0.5B under text-tools was 85% pooled but only 34% per-seed: seven of eight episodes failed in one or two turns while a single 41-turn episode of repeated valid calls dominated the pool. Denominators were as small as 8 to 16 turns over 8 seeds for the weaker models, making seed-level bootstrap confidence intervals correspondingly wide.

The gap between pooled and per-instance estimates reached about 55 points. This means a single configuration choice in how results are aggregated can flip the ranking of models, and the reported fidelity number is not robust to that choice.

Constrained Decoding Fixes the Protocol but Creates a New Failure

As a baseline, the researchers constrained decoding to a JSON schema whose name field is an enum of the available tools, using Ollama's structured outputs. On a single tool-call step, all nine local models emitted a valid in-schema call on 8 of 8 trials. Parseability and in-schema names held by construction.

But they did not run this as a full agentic condition, and for good reason: forced to emit a tool call on every turn, the weaker models never terminated, producing 600 to 900-turn loops within a single episode. Qwen2.5-Coder 0.5B reached 869 assistant turns in one seed before being cut off. Constrained decoding fixed the protocol layer being measured but traded an unparseable-call failure for a non-termination failure on weak models, further evidence that the configuration governs the observed failure rather than the model.

Replication Across Tasks Confirms the Mechanism

Because Ollama refuses the tools request before the prompt is processed, the rejection is a property of the model-server pair and should be task-independent. The researchers confirmed this on a structurally different dependency-chain task and on HumanEval. The native tool-gating replicated exactly in both cases: Phi-3 and Gemma-3 were rejected with HTTP 400, Llama-3.2 returned a native call, Qwen was accepted as text. Under text-tools, the same qualitative pattern recurred across tasks, though magnitudes were task-dependent and noisy at small denominators.

Why This Will Get Worse

The confound documented here is not a one-off quirk of one library. It is a structural consequence of how local agent stacks are assembled. A modern agent request passes through a harness, a tool-protocol adapter, a serving engine, and a per-model chat template, each of which can accept, rewrite, or reject a tool call independently of the model's ability. The Model Context Protocol and similar tool-calling standards plausibly add yet another translation step. As more practitioners evaluate small or local agents for cost and privacy reasons, and as the stacks they use grow more complex, the gap between "what the model can do" and "what the measurement records" widens.

A Checklist for Honest Measurement

The paper concludes with a protocol for measuring local-agent tool use. For standardized model comparisons, the serving interface should be held fixed across models, with the stack and its version pinned and reported. Transport-level failures, including rejected requests, timeouts, and empty responses, should be logged as a distinct outcome category, never as a model non-call. Per-seed rates with confidence intervals should replace single turn-pooled numbers. When a 0% tool-call rate appears, the diagnosis should proceed in order: first determine whether the serving layer refused or emptied the request; second check whether a tool-call parser is configured for that stack and model; only after both are ruled out should the failure be attributed to the model.

The remedy is cheap but requires deliberate design. Future local-agent benchmarks should report the serving configuration alongside the model identity, just as they already report decoding parameters and hardware. Treating the serving layer as part of the evaluation protocol rather than a transparent intermediary is becoming as important to agent evaluation as the benchmarks themselves.

The authors are explicit about what they do not claim. The per-seed rates show some variation with model size within the Qwen family and across families, but the intervals overlap and the estimates depend on the pooling choice. No scale law, family effect, or reasoning dissociation is asserted. The qualitative findings are deterministic and version-pinned: the per-model rejection, the cross-stack contrast, and the prompt-recovers-fidelity result reproduce exactly given the same Ollama release. The per-seed magnitudes do not reproduce to the digit because decoding is sampled at temperature 1.0 and seeds index task instances rather than the sampling RNG.