Large language models can solve many narrow tasks, but querying them separately for millions of related instances can be prohibitively expensive. Can LLM agents autonomously create cheaper solutions for such workloads? We call this ability bottling: the ability to turn general capabilities into task-specific solutions that balance answer quality and amortised cost. We introduce BOTTLED, a benchmark in which agents receive an entire unlabelled workload and must complete it under fixed time, compute and LLM API budgets. Agents choose their own approach, such as training a small model or writing a reusable program. Across ten models and three tasks, the authors find that strong zero-shot task performance does not reliably translate into strong bottling capabilities. Models with similar zero-shot scores can differ substantially after bottling, and 48 of 60 bottling runs score below the lower bound of the 95% confidence interval of their model's zero-shot performance. Moreover, 31 of 60 runs underperform the stronger of two small-model distillation baselines with the same token budget. Nevertheless, bottling can yield substantial savings: on query-product relevance classification, Opus 5 retains about 82% of its zero-shot macro-F1 at roughly 657 times lower reported cost. Bottling is also competitive with Jev, a system one model built especially for cheap, repetitive inference: Opus 5 on the same task recovers about 94% of Jev's macro-F1 at a quarter of Jev's projected full-workload cost. BOTTLED provides a basis for evaluating and improving agents' ability to invest limited resources in reusable solutions for large, repetitive workloads.
Why Bottling Matters for LLM Economies
The rapid expansion of large language model capabilities has created new opportunities for automating diverse workflows. Yet deploying these models at scale introduces a fundamental tension: each individual query consumes computational resources and incurs costs that scale linearly with volume. For workloads involving millions of related instances, the aggregate expense of querying a powerful model repeatedly can become a decisive barrier. Prior work has explored distillation, prompt caching, and specialized inference engines to mitigate this, but these approaches typically require advance knowledge of the task structure or access to model internals. The authors of this paper instead ask whether LLM agents themselves can discover cheaper ways to apply their capabilities. The core insight is that an agent given sufficient freedom to choose its own method—train a smaller model, generate and execute a reusable program, apply a heuristic shortcut—may produce a solution that retains much of the original quality at a fraction of the cost. The authors call this ability bottling and introduce BOTTLED, a benchmark designed to measure it.
The BOTTLED Benchmark: Fixed Budgets and Free-Form Solutions
BOTTLED places agents in a setting that mirrors many real-world deployment scenarios: an entire unlabelled workload must be processed under strict resource constraints. Specifically, each agent receives a set of instances without labels, a fixed wall-clock time budget, a compute quota, and a limited LLM API budget measured in tokens or dollars. The agent is free to decide how to satisfy the workload. It may invoke the base model repeatedly, it may train a smaller model on a subset of the data, it may synthesize a deterministic program that captures the pattern, or any combination of these. The only requirement is that the final output be produced within the allotted time and resource limits.
This design deliberately avoids prescribing a single correct approach. By letting agents choose their own method, the benchmark captures the breadth of strategies that LLM agents can autonomously invent. The authors evaluate ten different model families across three task domains: query-product relevance classification, sentiment analysis of product reviews, and question answering over a synthetic knowledge base. Each model faces the same workload, enabling a direct comparison of bottling effectiveness independent of the inherent difficulty of the underlying task.
The benchmark also records multiple metrics for each run: the quality of the output (measured by macro-F1 for classification tasks, exact-match for question answering), the total compute consumed, the number of LLM API calls made, and the wall-clock time elapsed. These metrics let the authors plot quality versus cost curves, from which they derive amortised cost per unit of quality—a central figure for evaluating whether bottling is worthwhile.
Zero-Shot Skill Does Not Guarantee Bottling Success
One of the most striking findings is the weak correlation between a model's zero-shot task performance and its bottling ability. The authors first establish each model's zero-shot macro-F1 on the target task without any budget constraints. They then run the bottling benchmark and compare the post-bottling quality to the zero-shot baseline. Across all ten models and three tasks, the relationship is noisy at best.
In fact, models with similar zero-shot scores can end up with very different bottled results. The authors report that 48 of 60 bottling runs finish with a score that falls below the lower bound of the 95% confidence interval of the model's zero-shot performance. This means that for the majority of runs, the bottled solution not only underperforms the raw model but does so by an amount that is statistically significant. The variance suggests that bottling success depends on factors beyond raw capability—such as the model's ability to introspect, to generate training data, or to structure a programmatic solution.
Moreover, 31 of 60 runs underperform the stronger of two small-model distillation baselines that use the same token budget. The distillation baselines train a smaller model on synthetic data generated by the base model, then evaluate on the workload. The fact that more than half of the bottling runs do worse than a simple distillation approach underscores how challenging it is for an LLM agent to discover effective compressed representations. The authors suggest that this gap arises because distillation benefits from a fixed training objective, whereas bottling must balance exploration of strategy with exploitation of the remaining budget.
Diversity of Bottling Strategies
Agents in BOTTLED employ a wide range of approaches, reflecting the flexibility of the benchmark. Some agents train a lightweight classifier on a small sampled subset of the workload, then apply that classifier to the remaining instances. Others synthesize a Python function or a regex pattern that captures the labeling logic, and execute it deterministically. A subset of agents perform iterative self-improvement: they generate labeled examples, train a model, evaluate on a holdout, and refine the pattern in subsequent iterations. Still others resort to heuristics such as majority voting over multiple cheap queries or rule-based fallbacks.
The choice of strategy often correlates with the model family. Larger, more capable models tend to attempt program synthesis or multi-step training loops, while smaller models more frequently opt for direct classification or simple rule extraction. However, there is significant overlap: a model with a high zero-shot score may still choose a suboptimal bottling strategy, and a lower-scoring model may discover a surprisingly efficient approach. The authors note that the diversity of successful strategies is itself a useful signal: it suggests that bottling is not a fixed property of a model but a learnable skill that can be encouraged through appropriate training objectives or prompt engineering.
One notable pattern is that agents frequently underutilize the full extent of their budget. Many runs terminate early, either because the agent determines that further effort would not improve quality or because the internal policy prioritizes completing within the time limit. This early termination contributes to the variance in quality observed across runs.
Substantial Cost Savings Where Bottling Works
Despite the frequent underperformance, the authors document cases where bottling delivers dramatic cost reductions without severe quality loss. On query-product relevance classification, Opus 5 retains about 82% of its zero-shot macro-F1 at roughly 657 times lower reported cost. The cost figure is derived from the amortised API budget across the entire workload; the quality retention is measured against the model's own zero-shot macro-F1, not against an absolute baseline.
In relative terms, Opus 5 on the same task recovers about 94% of Jev's macro-F1 at a quarter of Jev's projected full-workload cost. Jev is described as a "system one" model built especially for cheap, repetitive inference. The comparison is significant because it places bottling on a comparable footing to a specialized architecture that has been engineered from the ground up for efficiency. The fact that a general-purpose model can achieve 94% of Jev's quality at lower cost suggests that bottling may be a viable route to near-specialist performance without the engineering overhead of a dedicated model.
The authors also examine the shape of the quality-cost curves. In many cases, the first few units of budget produce the steepest quality gains, with diminishing returns setting in after a threshold. Agents that recognize this early diminishing-returns region and allocate their budget accordingly tend to achieve the best trade-offs. This observation hints at a potential research direction: training agents to allocate resources adaptively based on observed quality trajectories.
Comparison with Specialized "System One" Models
The paper introduces Jev as a reference point—a model variant optimized for the specific demands of repetitive inference. Jev's architecture and training objective are tuned to minimize cost per answer while maintaining acceptable quality on the target task. By contrast, the base models in BOTTLED are general-purpose LLM agents that have not been explicitly optimized for the bottling scenario.
The comparison reveals that bottling can bridge a meaningful fraction of the gap between general models and specialized ones. Opus 5 recovers 94% of Jev's macro-F1 at a quarter of Jev's cost. This result is particularly notable because it comes from a zero-shot bottling process: the agent has no prior experience with the task and must discover a compressed solution from scratch. The authors suggest that with additional training on bottling objectives, the gap could narrow further.
It is also worth noting that Jev's cost figure is projected for the full workload, whereas the bottling cost is measured on the same workload under the agent's strategy. The quarter-cost claim assumes that Jev's per-instance cost is roughly constant across the workload, while the bottled solution's cost includes any upfront training or program synthesis overhead that is amortised across instances. The authors account for this by reporting both the per-instance cost and the total workload cost.
Limits and What the Numbers Really Show
The authors are careful to contextualize their findings. The fact that 48 of 60 bottling runs fall below the zero-shot confidence interval indicates that bottling is fragile: small changes in the agent's policy, the workload composition, or the resource budget can tip the result from beneficial to detrimental. The 31 runs that underperform the distillation baselines further highlight that simple model compression is often a more reliable route to cost reduction than agent-driven bottling.
The benchmark also has limitations that affect interpretability. The three tasks, while diverse, are relatively narrow; broader or more open-ended tasks may yield different bottling dynamics. The resource budgets are fixed and relatively short, which may favor strategies that require little upfront investment (such as heuristics) over those with higher fixed costs (such as training a new model). Finally, the agents' introspection and planning capabilities are constrained by the base model's reasoning ability; stronger planning or meta-reasoning skills could potentially improve bottling success rates.
Despite these caveats, the paper's central message is clear: bottling is a non-trivial ability that is not automatically conferred by strong zero-shot performance. The BOTTLED benchmark provides a reproducible way to measure it, and the results suggest that significant cost savings are possible but require targeted agent design.
Implications for Deploying LLM Capabilities at Scale
For practitioners considering large-scale LLM deployment, the paper's results offer several practical takeaways. First, zero-shot quality alone is a poor predictor of whether bottling will be effective. A model that scores well on isolated examples may still produce a bottled solution that loses more quality than expected. Empirical testing on representative workloads is essential before committing to a compressed deployment.
Second, the diversity of successful strategies means that there is no single "right way" to bottle a model. Teams should invest in exploring multiple approaches—training small models, prompting the agent to generate programs, experimenting with heuristics—and evaluate each against the quality-cost curve for their specific workload. The authors' finding that 31 of 60 runs underperform distillation baselines suggests that distillation should be one of the first strategies tried, but not the only one.
Third, the cases where bottling succeeds demonstrate that large quality retention at large cost reductions is achievable. Opus 5's 82% quality retention at 657x lower cost, and Opus 5's recovery of 94% of Jev's quality at a quarter of Jev's cost, are concrete targets for teams seeking to optimize LLM deployment cost. These results also suggest that specialized "system one" models may eventually be supplemented or replaced by bottled general-purpose models, especially if bottling techniques improve through meta-learning or automated strategy search.
Finally, the BOTTLED benchmark itself is released as a resource for the community. By providing a standardized testbed with fixed budgets, diverse tasks, and detailed resource metrics, the authors enable future work on bottling algorithms, agent training methods, and new architectures designed from the outset for efficient repetitive inference. The benchmark is available alongside the paper, and the authors encourage others to extend it with additional tasks, harder resource constraints, or adaptive budget policies.