Autonomous Research Reaches 90% of Human State-of-the-Art in Telecom Ticket Retrieval
The promise of autonomous ML research is straightforward: let an LLM agent iterate through experiments faster and cheaper than a human researcher, discovering better models without human intuition or creativity. Previous demonstrations have been confined to narrow search spaces, language modeling benchmarks, or biomedical tasks where the training data is fixed and the degrees of freedom are limited. A team from Georgetown and Nokia Bell Labs, led by Junghyun Min with Huseyin Uzunalioglu and Mohamed Trabelsi, asks what happens when you throw autonomous research at an open-ended industrial problem with degrees of freedom in representation, architecture, and training data generation.
The case study is telecom ticket retrieval: given a query incident ticket, retrieve the relevant resolution document from a corpus of 250,000 incident reports, 204,000 fault analyses, and 89,000 technical analyses. The search space spans not just hyperparameters but also document preprocessing, negative sampling strategies, pair type definitions, and encoder choices. The authors run systematic campaigns across framework structure (single-agent vs. multi-agent), LLM agent choice (Claude Sonnet 5, Cursor Composer 2.5, GPT-OSS 120B), and informedness (whether the agent is told about human-designed state-of-the-art techniques). The results are honest and instructive.
Why Telecom Ticket Retrieval Is a Good Testbed
Telecom ticket retrieval has three properties that make it a demanding test for autonomous research. First, it is difficult: the tickets contain highly contextual language, domain-specific jargon, and varying levels of detail depending on which engineer submitted them. Tickets often do not contain the full details relevant to their causal incidents. Second, it is open: the degrees of freedom go beyond architecture and hyperparameter search. Training data generation (negative sample mining), loss function design, and document representation are all part of the search space. Third, the human-designed state-of-the-art system is complex, incorporating data augmentation, model ensembling, and re-ranking, none of which are standard hyperparameter tweaks.
The task is defined as finding the optimal preprocessing function, sampling configuration, and encoder weights that maximize top-K recall over a held-out evaluation split of 7,600 queries and 1,200 gold resolution documents. A correct retrieval means the query and retrieved document belong to the same cluster, addressing the same incident or resolution.
The comparison baselines are all human-designed: BM25 (Recall@1 of 0.072), a fine-tuned MPNet model (0.200), an ensemble of fine-tuned models (0.251), and an unpublished internal state-of-the-art system that uses LLM-based data augmentation and re-ranking (0.380). The gap between the simple fine-tuned model and the internal SOTA illustrates how much engineering goes into a production retrieval system.
The Operational Harness
Adapting autonomous research to an open-ended retrieval problem required building an operational harness. The authors discovered through trial and error that three components were necessary for reliable campaigns.
Task and environment documentation must be explicit. Unlike language modeling, where data is unstructured and single-stream, telecom ticket retrieval involves complex, heterogeneous document structures. The harness documents the data structure, task scope, agent role, and the primary optimization metric (Recall@1). It also documents available hardware and instructs agents to maximize GPU utilization. Despite explicit instructions, agents frequently underutilized compute resources.
Search space definition prevents agents from defaulting to standard hyperparameter optimization. In the multi-agent implementation, three agent teams are assigned separate search spaces: representation, training data generation, and architecture and hyperparameters. In the single-agent implementation, a set of variables maps to decisions in representation and dataset generation within the codebase. This forces broader exploration but introduces a trade-off: agents primarily toggle pre-defined candidate options rather than implementing novel code modifications.
The scripted loop replaces agentic self-looping with a deterministic outer bash script. Prompting agents to manage their own execution loops proved unreliable, frequently resulting in stalled agents or redundant experiments. The harness initializes an agent session, retrieves a session resume key, and runs a bash while loop that resumes the session with a prompt to continue experimenting. This is the most reliable implementation of a never-stopping loop that ensures continuity until the human researcher stops it.
The Best Discovered Model
The best system discovered across all campaigns is a single fine-tuned MPNet retriever. It achieves Recall@1 of 0.343, Recall@10 of 0.630, and Recall@50 of 0.768. This substantially outperforms BM25, the standard fine-tuned model, and the human-designed ensemble. It does not surpass the internal state-of-the-art (Recall@1 of 0.380), which incorporates structural components the agent never attempted: re-ranking, data augmentation, and model ensembling.
The discovered system's key innovations are applying 2x oversampling to peer ticket (T-T) pairs and expanding the encoder sequence length to 384 tokens. These are concrete, well-motivated changes that a human researcher might also discover, but the agent found them through systematic exploration rather than intuition.
At 90% of state-of-the-art performance, the discovered model reaches a level that would be useful in production for many applications, even if it does not match the most complex human-engineered system. The gap is real but bounded: it comes from architectural decisions (ensembling, re-ranking) that the agent consistently failed to consider.
Design Choices Barely Matter
The most surprising finding is that framework structure, LLM agent choice, and informedness have negligible impact on peak downstream performance. Half of the campaigns converged to a tight performance band, with Recall@1 between 0.30 and 0.34.
Across framework structures, the multi-agent implementation distributed search spaces but their discussions rarely introduced effective experimental proposals beyond parameter tuning. The single-agent loop produced the best model. The hypothesis that multi-agent discussion would enable wider searches and stronger experiment proposals did not hold for this open-ended task.
Across LLM agents, reasoning ability did not correlate with downstream performance. Cursor Composer 2.5 generated the top-performing single model, but no single agent produced a particularly strong or weak result. A locally hosted open-weight GPT-OSS with no prior knowledge outperformed a team of Claude Sonnet 5 agents with prior knowledge in one comparison.
Across informedness, providing documentation about human-designed state-of-the-art techniques had little effect. Regardless of explicit documentation, no agent attempted to implement re-ranking, data augmentation, or model ensembling. They consistently defaulted to single-model fine-tuning. Providing human insight in the form of documentation is insufficient to induce architectural changes in current agents.
Cost and Time
A single 10-20 experiment campaign using Cursor Composer 2.5 costs an estimated $150 to $200 in API charges, excluding repository setup and harness development. Claude Sonnet 5 campaigns cost approximately 2 to 3 times more ($300 to $600). Open-weight models hosted locally incur no API cost given available GPU resources.
In terms of time, the 17-run autonomous campaign that produced the best model completed within 1 GPU-week of compute time and 10 weeks of end-to-end execution. The human-engineered comparison systems required approximately 10 months of development. The authors note this is not a controlled comparison since the autonomous campaigns start from a human-designed codebase, documentation, and evaluation pipeline, and the human researcher remains responsible for monitoring and stopping campaigns.
Where Agents Fall Short
The primary limitation is that agents lack intuition-driven hypothesis generation. Rather than considering training dynamics (loss trajectory analysis, early stopping) or training data distribution (average or median document length), agents arbitrarily changed parameters. They proposed extending training from 1 to 2 epochs instead of implementing early stopping. They extended maximum document length from 256 to an arbitrary 384 tokens instead of analyzing the length distribution of documents in the dataset.
Agents are conservative in exploring alternate representation encoders. Single-agent campaigns started with MPNet and rarely proposed moving to a more modern or larger architecture. Multi-agent runs started with MiniLM and showed the same conservatism. The only other proposed encoder was the BAAI General Embedding model. This pattern reflects documented limits in LLM reasoning and the structural constraints imposed by next-token pre-training objectives.
Agents also exhibited distinct failure patterns. Cursor Composer often re-executed duplicate experiments. GPT-OSS failed to reliably spawn sub-agents, frequently explaining how to start the run instead of acting as the orchestrator. Claude Sonnet sometimes opted to debug environment scripts rather than run experiments. All agents underutilized GPU memory, often proposing smaller batch sizes than what could fit.
What This Means for ML Research
The paper recommends three takeaways. First, autonomous research and human intuition complement each other. Agents excel at deep, narrow hyperparameter sweeps but lack the creativity to propose systemic architectural changes. The optimal workflow combines agent-optimized single models with human-proposed solutions like ensembling and re-ranking. Second, autonomous research saves time for a modest cost. The 10x speedup over human development is significant even accounting for the non-controlled comparison. Third, agents need human babysitting. Operational struggles are real and require human intervention to resolve.
The broader implication is that autonomous research is not a replacement for human researchers but a tool that extends their capabilities. An agent can systematically explore a search space and find good configurations much faster than a human, but it cannot step back and ask whether the search space itself is the right one. The human researcher defines the problem, sets the search space, and selects implementation specifications. The agent fills in the details. This division of labor appears to be the practical sweet spot given current tools.
The case study also raises questions about the limits of current LLM-based agents for scientific research. If agents cannot reliably implement architectural innovations even when explicitly told about them, the gap between autonomous research and fully automated science remains wide. The 90% result is encouraging for narrow optimization tasks, but the remaining 10% may require the kind of creative problem-solving that current agents cannot perform.