Every language model request carries a token budget, the maximum number of tokens the model may generate. For agents that call tools, this budget creates a failure mode that most developers have encountered but few have a good solution for. The model starts writing a JSON tool call, the budget expires mid-structure, and the engine returns something unparseable. The agent cannot execute the call, so it retries, burning the tokens and time it was already short on.

For local models, this is not a marginal annoyance. It is a fundamental constraint. Local inference is slow, context windows are small, and token budgets are set tight on purpose. A broken tool call that forces a retry consumes a meaningful fraction of the available budget. When the retry also breaks, the entire request fails.

A project called Runner addresses this by guaranteeing that tool calls survive truncation. When the token budget expires mid-call, Runner does not return broken JSON. It closes the document legally, producing a parseable result that the agent can execute, and marks the response so the caller knows the budget was hit.

How Most Engines Fail

The failure modes vary across engines, but none of them produce an executable result. Some return a finish reason indicating length exhaustion with an empty tool call. Others leak the model's tool-call framing into the assistant's prose, mixing structured output with free text in a way that breaks parsing. Some return a tool_calls object whose arguments fail to parse because the JSON is incomplete or malformed.

Runner's documentation ranks these failure modes from most to least useful for a caller. An executable call, even one completed by the engine rather than the model, is the best outcome. A detectable empty response or error is second, because at least the agent knows something went wrong. Protocol rendered as prose is worse, because the agent might attempt to parse it and get a confusing error. A tool_calls object with silently failing arguments is the worst, because the agent proceeds as if the call succeeded and fails downstream.

The Schema-Aware Closer

Runner's approach depends on a capability that standard constrained decoding does not provide. Ordinary JSON-Schema constrained decoding restricts which token may come next, ensuring that output is valid if it finishes. But it says nothing about what happens when the output cannot finish. If the budget runs out, the partial output is still broken.

Runner's schema compiler produces a streaming validator that knows, at every byte, what the smallest legal completion of the document is. When the budget expires, the validator closes the string the pattern still accepts, completes the number inside its declared range, supplies required properties, and closes the object. The result is a well-formed JSON document that satisfies the tool's schema.

The validator also enforces a structural constraint: it refuses to nest a document deeper than the parser can read back. This prevents a class of bugs where the closer produces output that looks valid but cannot be parsed by the consuming code. The Runner team discovered this class of bug through fuzzing and fixed it by making the validator refuse early rather than widening the parser, because refusing is the direction that cannot ship a broken document.

Benchmark Results Across Engines

Runner's benchmarks test the same schema, the same prompt, and the same model across six engines with seven token budgets: 1, 2, 3, 5, 8, 16, and 64 tokens. The 64-token control proves that the failure is truncation, not model misconfiguration. Every engine completes successfully at 64 tokens. Below that threshold, only Runner returns a parseable tool call.

The engines tested include vLLM, llama.cpp, Ollama, TensorRT-LLM, and SGLang, in addition to Runner. At 16 tokens, the differences are stark. vLLM and llama.cpp leak the tool-call framing into the content field. llama.cpp emits unparseable arguments. Ollama hides the framing but returns an HTTP 500 error. TensorRT-LLM and SGLang return an empty message with no tool call and an HTTP 200 status code, giving the agent no indication of what happened beyond the missing call.

None of these engines close the document. They all leave the agent holding something it cannot use, forcing a retry that consumes the remaining budget.

An Engine Guarantee, Not a Model Property

The critical distinction is that truncation-safe tool calling is an engine guarantee, not a model quality property. The same model produces broken output in other engines and parseable output in Runner. The grammar and closer do the work, not the model's ability to predict when to stop.

Runner validates this property on every release using a random two-layer CI fixture and granite-4.1-3b, both running on CPU without a GPU. The test runs without competitor engines installed, confirming that the property is self-contained. An agent-torture test applies the same failure scenario inside multi-turn agent loops, verifying that truncation recovery works in the context where it matters most.

Quantization and Argument Fidelity

The benchmarks also test constrained decoding across a full quantization ladder, from the original weights down to Q4_0. Schema conformance and tool selection remain at 100% across all quantization levels. The closer guarantees that the call is well-formed and names the right tool regardless of how aggressively the model is compressed.

Argument agreement with the Q8_0 reference does not hold at 100%. It decays to roughly 50% at the lowest quantization levels. This is expected: the closer ensures structural validity, but the content of the arguments still comes from the model, and quantization degrades the model's ability to produce the exact arguments it would have produced at higher precision.

For most practical applications, this tradeoff is acceptable. A well-formed call that names the right tool with approximate arguments is far more useful than a broken call that cannot be parsed at all. The agent can validate arguments after execution, retry individual parameters, or accept approximate values for non-critical fields.

Schema Support and Compile-Time Safety

Runner's schema compiler supports the subset of JSON Schema relevant to tool calling: objects, arrays, enums, const values, type unions, numeric bounds, string lengths and anchored patterns, array counts, and the tool-discriminated union that agent clients use to route calls to specific functions.

Unsupported or ambiguous constraints fail at compile time rather than being silently weakened. This is a deliberate design choice. A constraint that cannot be enforced at runtime should not be accepted silently, because the caller assumes it is being enforced. Failing early surfaces the problem during development rather than producing incorrect output in production.

Choice Logprobs for Routing

Because the validator knows the legal branches at each decision point in the schema, Runner can record choice logprobs: each schema branch as a legal alternative, a posterior probability renormalized over the legal set, and the probed probability mass. This is useful for routing and calibrated classification, where the system needs not just the model's choice but a measure of confidence in that choice.

An included calibration tool converts labeled decisions into accuracy, Brier score, and expected calibration error gates. For teams building systems that route tool calls based on model output, this provides a quantitative measure of how well the model's confidence aligns with its accuracy.

Practical Implications

The immediate benefit is reduced retry overhead in agent loops. Every broken tool call that does not need to be retried saves tokens, time, and context window space. For local models running on consumer hardware, where a single request might take seconds or minutes, this compounds quickly.

The secondary benefit is reliability. Agent loops that depend on tool calls fail when the calls fail. Hardening the tool call layer makes the entire agent more robust, particularly under the tight token budgets that local inference demands.

The broader implication is that constrained decoding is evolving from a output quality feature into an output availability feature. The ability to produce parseable output under adverse conditions, not just under ideal ones, is becoming a requirement for production agent systems. Runner's approach demonstrates that this is achievable without model changes, using only engine-level guarantees.