At Funnel, a company experimenting with language models powering Reef, their internal data agent platform, ran a head-to-head test between two prominent models. The experiment pitted GLM-5.2, an open-weight model, against Claude Opus 4.8. Each new user task received a model assignment that stayed consistent across subsequent interactions, including deeper investigations and the creation of interactive dashboards the team calls explorations.
The early numbers looked favorable for GLM. During the observation window, GLM averaged 7.8 seconds per model call, compared to 18.1 seconds for Opus — more than twice as fast per individual request. However, when the team measured complete task execution from start to finish, the picture shifted. GLM issued 66% more tool calls on average, hitting 19 calls per task versus Opus's 11. Even though each GLM response came quicker, overall task completion took 14% longer than with Opus.
In two separate test runs, GLM executed the same successful SQL query 113 and 254 times. One instance involved a model trying to pull additional data from a tool that only displayed a 20-row preview. Instead of recognizing the preview limit, the model kept running the identical query. Every individual call succeeded, but the agent produced no real progress. As the team noted, a successful tool call is not necessarily a useful tool call.
The team also observed similar repetitive behavior with Kimi K2.6 and Qwen3.8 27B, with Qwen showing twice the tool-aggression level of GLM. Blaming the model alone misses the root cause. Some saved agent instructions pointed to Athena tables and fields that did not actually exist. Other issues arose from ambiguous tool contracts, missing guards against repeated calls, and data-retrieval instructions that lacked sufficient clarity. Opus handled most of these friction points better, but the underlying problems remained.
Why model swaps expose hidden architectural assumptions
Different models process the same instructions, tool definitions, and stopping conditions in distinct ways. When teams replace one model for another, previously invisible assumptions about how the system should behave can surface. A model that is more aggressive with tool calls may reveal that the surrounding framework lacks sufficient guards or clear expectations about what constitutes a completed action.
The Funnel team identified three practical areas where agent platforms can be strengthened, regardless of which model drives them.
Make tool contracts explicit
Tool definitions should clearly distinguish between preview data and complete results. Pagination needs to be visible in the interface, and the system should communicate what a successful response actually means. If a tool only returns a subset of rows, the model should receive a signal that tells it whether more data exists or if it has reached the final set.
Bound agent behavior
Platforms should detect when a model issues repeated equivalent calls and stop the loop. Execution budgets can set a sensible limit on how many tool attempts a single task receives before the system steps in. Saved instructions should validate against real schemas to catch references to non-existent tables or fields before they reach the model.
Benchmark the whole system
Metrics need to cover more than individual model latency. Useful task completion, correctness, latency, tool efficiency, billed costs, and the effect of input caching all matter. Measuring these together gives a clearer picture of whether a model change actually improves the user experience or just shifts where the bottlenecks appear.
Open-weight models present exciting paths toward lower cost, better performance, and more control. But realizing those benefits typically requires more than pointing an API endpoint at a different provider. The surrounding framework often needs adjustment to support the model's tendencies rather than fighting them.