When an AI agent can either call tools directly or write a Python program to call those same tools inside a sandbox, the assumption is that it will pick whichever route is more efficient. A new technical benchmark from researcher kukushko challenges that assumption, and the results suggest that making both options available does not mean the agent will choose the better one.
The Experiment: Two Ways to Execute a Task
The benchmark tested an open-source terminal agent against ten practical tasks using two distinct execution modes. In the hybrid mode, the agent had access to ordinary direct tools for files, web search, page fetching, and Python execution, plus a special orchestration entrypoint called jobs.run that could execute model-written Python code inside a Docker sandbox. In the jobs-only mode, the direct tools were removed from the top-level catalog entirely, leaving only the orchestration entrypoint, which could in turn call delegated tools inside the sandbox.
The model used was Qwen3.6-35B-A3B-UD-Q4_K_XL, a quantized model running behind an OpenAI-compatible local endpoint with thinking enabled and a 131,072-token context window. Each task was run once per mode in fresh agent sessions, with the modes alternated between tasks to reduce systematic effects from endpoint load and changing web data. The sandbox used Python 3.12.
The Central Finding: Zero Voluntary Choices
The most striking result was not a failure but an omission. When both the direct tools and the orchestration entrypoint were available, the model chose to write a program and route it through the sandbox zero times across all ten tasks. Every single task was completed using direct tool calls. The model never voluntarily selected the orchestration path, even though the tool documentation described its intended use and the entrypoint was present in its catalog throughout.
Only when the direct tools were removed from the catalog did the model begin writing programs. In jobs-only mode, it generated workflows that included fallback searches, URL selection, batch fetching, file reading, semantic extraction, and compact output generation. This reveals two distinct questions: whether a model can construct a workflow when asked to, and whether it selects that approach when given the choice between it and simpler direct calls.
The Numbers: Token Savings With a Catch
On the nine task pairs that did not hit a semantic-search timeout, the jobs-only mode showed meaningful token savings when considering only the main orchestration calls. The mean tokens per task dropped from 29,060 in hybrid mode to 12,161 for the main orchestration layer. However, when nested LLM calls inside the sandbox were added back in, the combined token count rose to 28,697, only 1.3 percent below the hybrid total.
The gap was entirely explained by one outlier task. When asked about the Qwen developer documentation and license, the model produced nested LLM calls totaling 92,456 tokens inside the sandbox, pushing the combined jobs total to 107,388, roughly 2.7 times the hybrid count of 39,347. Excluding that single task, the remaining eight non-timeout pairs showed that jobs mode used approximately 32 percent fewer tokens than the hybrid approach.
The wall time told a different story. The hybrid mode averaged about 130.7 seconds across the nine tasks. Jobs mode took roughly 267.2 seconds, approximately twice as long, even before accounting for the token outlier. The sandbox execution, nested LLM calls, and job repair cycles all added overhead.
Where Orchestration Actually Helped
There were specific cases where the jobs-only mode produced noticeably better results. A task requiring the three longest non-empty lines in a result.py file failed in hybrid mode because the model exhausted its four-call outer limit using unsuitable direct calls. In jobs-only mode, it succeeded with 5,615 tokens. Fetching and summarizing a webpage took 8,750 combined tokens in jobs mode versus 20,992 in hybrid mode. Calculating the mass of two hollow lead spheres used 9,976 tokens versus 15,762.
The mechanism appears to be that a program can pass intermediate tool results directly between operations, so the main model does not need to repeatedly receive and reason over every page, file, or partial calculation. In the hybrid mode, the model must reason over the output of each individual tool call before deciding what to do next, which accumulates token cost.
The Reliability Problem
The jobs-only mode introduced its own failure modes. Generated programs sometimes lacked the required global result, referenced undefined variables, passed the wrong data type to nested LLM calls, failed to handle empty search results, or assumed incorrect result fields. Repair often meant generating an entirely new job. The Bitcoin price task required two jobs to succeed, the Taylor approximation required two, the EUR/USD task required four before failing, and the Qwen task required three.
Both modes also struggled with the same semantic-search task, which hit a 240-second harness timeout due to recursive directory searching across roughly 202 files. This points to an underlying indexing problem rather than a catalog design issue.
What This Means for Agent Builders
The benchmark suggests that simply making an orchestration tool available and documenting its purpose is not enough to get an agent to use it. The model needs either stronger guidance, examples that demonstrate when orchestration is preferable, or a constraint on the action space that forces the choice. The researchers explicitly recommend comparing four different policies: hybrid with default guidance, hybrid with stronger orchestration instructions, hybrid with examples of when to choose a job, and jobs-only.
The deeper question the benchmark raises is whether a model can learn to recognize when turning a sequence of tool calls into an executable workflow is the right move. That requires not just knowing how to write a program, but understanding when the cost-benefit calculus favors it. With the current model and configuration, that recognition did not happen naturally.
The benchmark is a case study in tool selection and cost distribution rather than a verdict on either approach. Both modes solved eight of ten tasks successfully. Direct calls remain the sensible default for this model. But the cases where jobs mode used roughly a third of the tokens suggest that for deterministic transformations, aggregation, and predictable multi-step workflows, the program-based route has clear advantages that agents are not yet reliably choosing on their own.