The Skill Router Hiding Inside Your LLM
When an LLM agent encounters a task that requires a skill, picking the right one from a library of thousands is the bottleneck. Current systems handle this in one of two ways, and both have serious problems. Progressive disclosure, used by Claude Code and Codex, preloads every skill's name and description into the system prompt so the agent can decide which to read. This crowds the context window, degrades the agent's reasoning on the actual task, and forces a hard cap on library size. Retrieval pipelines move selection outside the context by using an embedding model and reranker to pick skills before the agent sees them. But the retriever does not understand the task as well as the agent does, and it does not improve as the agent improves.
Chen, Wang, Chen, Li, and Huang from Tsinghua University show that the frozen agent LLM already contains the routing signal in its own forward passes, and that two linear maps with 7.9 million parameters are enough to extract it. Their system, Gavel (Glance And Verdict from a frozen LLM), routes without putting any skill text into the context until a skill is actually chosen. It reads the model's intermediate hidden states to score the entire library, then uses the model's own full attention on shortlisted candidates to make the final decision. On Qwen3-32B, Gavel outperforms progressive disclosure and retrieve-and-rerank pipelines that add 1.2B to 16B external parameters, and in a bash-agent harness the same 32B model triggers the correct skill more often than far larger frontier models running in Codex.
Why Progressive Disclosure and Retrieval Both Fail at Scale
Progressive disclosure works by listing every installed skill's name and description in the system prompt. The agent LLM reads this menu and decides which skill to load. The problem is mechanical: the metadata crowds the context in proportion to the library size, and public skill libraries now reach tens of thousands of entries. Codex caps skill metadata at 2% of the context window and drops the rest, which confines a deployment to a small library where routing accuracy still decays logarithmically with the number of skills. The fundamental cost is intrinsic to the approach: routing inside the context spreads the agent's attention thin over every skill it considers and weighs only the summaries that fit, which omit much of what selection depends on.
Retrieval pipelines sidestep the context problem by handing selection to external models. An embedding model computes a similarity score between the task and each skill, a reranker scores the top candidates, and the selected skill is injected into the context. The context is relieved, but the selection is cut off from the agent's capability. A standalone retriever may be good at matching text, but it judges what a task needs less reliably than the agent LLM, especially when the need arises mid-rollout amid noisy multi-turn context. And a retrieval pipeline does not improve when the agent backbone is upgraded. You need a new retriever or reranker, or at minimum a new embedding, to benefit from a stronger model.
The tension is clear: selection inside the context taxes the agent, yet an external model does not understand tasks and skills as well as the agent does. Gavel resolves this by drawing every routing signal from the frozen agent LLM's own forward passes, with nothing beside the backbone and no skill text in the context until one is chosen.
The Glance: A Lightweight Scan of the Full Library
Gavel's first stage, the glance, projects the task's and each skill's mid-layer hidden states through two trained linear maps and scores the full library. The maps are a query map Wq and a key map Ws, both reading from the same intermediate layer of the frozen LLM. The query map reads each task token's state at a layer chosen at the floor of the model's compression valley, where the matrix entropy of the hidden states bottoms out, roughly 70% through the network (layer 45 of 64 for Qwen3-32B). The key map reads each skill token's state at the same layer, computed once at installation time.
The key insight is that the model's intermediate layers compress the input's semantics but entangle them with information that serves only next-token prediction. No native read-out, such as attention keys or initialized heads, ranks a library of skills. So the glance grafts a new attention head onto the frozen model. The head performs hard attention: each task token keeps its strongest inner product against every skill's token keys, as in late-interaction retrieval. This means each task token votes for the skills it matches best, so a few decisive tokens are not averaged away by the rest of the span.
The maps are trained once with a multi-positive contrastive loss on the SkillRet training corpus (51,104 query-skill pairs over 9,084 skills). Gradients stop at the hidden states, so the LLM stays frozen. At routing time, the glance is one sweep of the head over the bank, which costs no additional forward passes since the task token states are a byproduct of decoding.
A new skill costs one forward pass and no training. The key map stores a key per token in a fixed prompt that renders the skill's metadata and body. But a skill that dwells on one capability leaves a cluster of keys pointing nearly the same way, and since the glance takes the max over keys, a single representative serves the whole cluster. Gavel compresses each skill's bank to an epsilon-cover by farthest-first traversal, retaining only keys such that every discarded key lies within distance epsilon of a retained one. This shrinks banks by roughly 8.5 times with a bounded distortion: the max similarity drops by at most epsilon and inflates none. The cover radius is fixed at 0.83 during training.
The Verdict: The Model Examines Its Own Candidates
The glance is cheap but limited. Its factorized head misses subtler inferences that routing can turn on. Full attention over skill and task together does not, but it costs one forward pass per candidate, which no library affords. The verdict spends that pass on the glance's shortlist alone. For each shortlisted skill, it resumes the installation render with the task appended (the order used in classical query-likelihood scoring), and reads two signals from one forward pass.
The first is the mean log-likelihood of the task tokens given the skill as the prefix: L(s|x) = (1/|x|) * sum of log p(xi | r(s), x<i). This measures how well the skill primes the model to generate the task. The second is the model's log-odds judgment of whether the skill serves the task, taken at the final position after a yes/no question: V(s|x) = log sum of yes-probabilities minus log sum of no-probabilities. Since the question follows the task, the causal mask leaves every task position untouched, so one pass yields both read-outs.
The three scores (glance, likelihood, judgment) can each be read as an estimate of the same log posterior log p(s|x), on a scale of its own. The glance reads it contrastively, the likelihood reads it generatively, and the judgment reads it discriminatively. So they fuse as a product of experts: S(s|x) = g(s|x) + alpha * L(s|x) + gamma * V(s|x), where eS multiplies the three experts. Any one of them can veto a candidate the other two merely tolerate. The exchange rates are calibrated on SkillRet's validation split: alpha=1.0, gamma=0.025. Only candidates the glance scores within a margin of its best (Delta=0.133, roughly nine on average) receive a verdict.
Results: Beating Pipelines with 1000x Fewer Parameters
Gavel is evaluated on three public benchmarks and a new one the authors build. SkillRet pairs 4,997 queries with 6,660 held-out skills. SRA-Bench changes the genre on both sides, with tasks from reasoning and coding benchmarks and 26,262 skills collected from the web. Eval-Core poses 75 real task.md queries against 78K documents in two pools. All scores use adjudicated Hit@1: a judge compares the committed skill against every gold label and credits it when it wins at least as often as it loses.
On SkillRet, Gavel beats the strongest retrieve-and-rerank pipeline (Qwen3-Embedding-8B + Qwen3-Reranker-8B) by 3.8 points. On SRA-Bench, where the skills shift from SKILL.md files to procedural web pages, the gap widens to 13.4 points. On Eval-Core, Gavel leads by 1.3 to 2.7 points across the easy and hard pools. The trained embedding models (SkillRouter-Emb-0.6B and SKILLRET-Emb-0.6B) match the glance on SkillRet, whose queries resemble their training data, but collapse below BM25 on SRA-Bench when the domain shifts. Qwen3-Embedding-8B, an untrained general-purpose embedder, fares better on SRA-Bench but trails even the small trained embedders on SkillRet.
The ablations isolate what each design choice contributes. Replacing the glance with Jina-ColBERT-v2, a retriever trained for exactly the same late-interaction scoring, loses badly on every benchmark. The glance profits from the far better compression of the agent LLM it reads, not from how tokens are compared. Dropping the product of experts in favor of the model's judgment alone scrambles ambiguous queries on SkillRet. Replacing the verdict's likelihood signal with skill-body likelihood discards it outright (alpha goes to zero) and falls behind on the remaining two signals.
Mid-Rollout Routing: Where Pipelines Break Down
The three public benchmarks hand the router a single written task. A live agent instead works from a long multi-turn context, and the need for a skill may surface only as it works through the task. Gavel builds SkillTraj, a benchmark of 372 simulated agent trajectories, each a multi-turn dialogue with tool calls, marking the point where a skill becomes needed. The trajectories cover four scenarios: user request (the need is stated outright), tool evidence (a tool result reveals it), agent plan (the agent's own written plan creates the need), and wrong-skill recovery (the agent has loaded the wrong skill and must reroute with the misleading document still in context).
On SkillTraj, Gavel leads every scenario by 8.6 to 21.9 points over the strongest other system. The retrieve-and-rerank pipelines face a dilemma: embed the full noisy trajectory, which mixes every topic the agent has touched, or embed only the latest turn, which may lose the evidence that matters. The larger untrained Qwen3 models are better served by the clean last message than by the full context that holds all the usable evidence. Progressive disclosure holds its own on wrong-skill recovery, where the agent has just watched a skill fail and sums up the missing capability in a sentence that a name and description match fine, but collapses on agent plan, where the freshly drafted plan carries details only the skill body can confirm.
Gavel reads the context where it already sits, inside the agent's own forward pass. The glance applies a vote decay that weights recent tokens most heavily, so the routing point receives the most influence. The verdict continues from the most recent message, putting the backbone's reasoning to work on a need only partly spelled out.
End-to-End Deployment: Triggering Skills in a Live Agent
The benchmarks above assume the router knows when a skill is needed. A live rollout requires a gate that decides when to attempt a skill call. Gavel trains one, taking SkillTraj decision points as positives and trajectories that never warrant a skill as negatives. It reads the glance scores the preceding tokens have accumulated and the final-layer state from which the model predicts its next token. Once the gate fires on two consecutive tokens, Gavel attempts a selection, and if the verdict judges even the winner unfit (V < 0), loads nothing.
On Skill-Use, a benchmark of 177 executable tasks over 79 skills, the results are striking. Under progressive disclosure, Qwen3-32B almost never reads a skill at all (1.1% trigger rate). With Gavel integrated, the same frozen model triggers the correct skill in 90.9% of tasks. That puts it ahead of every frontier model running in Codex under progressive disclosure (GLM-5.1 at 70.6%, MiniMax-M3 at 86.4%, Qwen3.6-Max at 68.4%, DeepSeek-V4-Pro at 65.0%) and ahead of retrieve-and-rerank pipelines that load a skill on every trajectory whether one is needed or not.
The gate is the only part that requires training on domain-specific data, and it is tiny: a linear classifier over glance scores and the final-layer state. The core projections Wq and Ws remain frozen from their single SkillRet training run.
Scaling: The Router Improves as the Backbone Does
A key advantage of Gavel is that routing accuracy improves as the LLM backbone improves, because the routing signal comes from the model's own forward passes. On SRA-Bench, both the glance stage and the full pipeline climb as the backbone grows in size and steps up in generation. Gavel on a 0.6B model, one that all but fails at picking from a metadata menu, already beats the 32B under progressive disclosure (which has the same backbone pick from a menu of twenty candidates retrieved by Qwen3-Emb-8B). Even on Qwen3.8-27B, whose agentic ability draws level with far larger frontier models on many benchmarks, picking from the metadata menu still trails Gavel on the same backbone by 12.2 points.
This scaling property means Gavel gets stronger for free when the agent is upgraded, unlike retrieval pipelines where the embedder and reranker must be retrained or replaced. The 7.9M parameters of the two projections are the only additional cost, and they are trained once on SkillRet and deployed zero-shot everywhere else.
Limitations and What Comes Next
Gavel requires access to the LLM's intermediate hidden states, which means it works with open-weight models or models behind an API that exposes layer activations. It does not work with a black-box API that returns only text. The authors acknowledge this in their discussion and note that routing behind a cloud API would require a different interface, potentially a probing head exposed by the provider.
The gate that decides when to attempt a skill call is trained on SkillTraj decision points, which are simulated trajectories. Whether it generalizes to truly live deployments with noisy, unpredictable user interactions is an open question. The authors test on Skill-Use, which is a controlled benchmark, and report qualitative results from deployment walkthroughs, but large-scale live evaluation is not provided.
The skill libraries tested are large (up to 78K documents) but curated. Real-world skill libraries may contain poorly written, duplicate, or conflicting skills. How Gavel's contrastive glance and product-of-experts verdict handle such noise is not systematically evaluated. The epsilon-cover compression assumes skills have a low intrinsic dimensionality in the hidden-state space, which may not hold for all skill types.
The paper does not compare against fine-tuning the backbone to incorporate routing natively. Fine-tuning a large LLM on skill selection could potentially match or exceed Gavel's performance, at the cost of modifying the base model. Gavel's selling point is that it works with a frozen model, which preserves the model's general capabilities and avoids the expense of fine-tuning. But the comparison is worth making to understand the ceiling of what frozen-route-read approaches can achieve.
What This Means for Agent Architecture
Gavel suggests a shift in how LLM agents interact with skill libraries. Instead of either crowding the context with metadata or delegating selection to an external model, agents can route using their own internal representations. The glance provides a fast, library-scale scan without any forward passes over the task. The verdict provides a careful, attention-based evaluation of the top candidates. And the product of experts fuses them in a principled way that allows any one signal to veto the others.
For developers building agent systems, the practical implication is that you can have a large, growing skill library without paying the context tax. The glance scores the full library in one sweep, and only the shortlisted candidates receive forward passes. New skills cost a single forward pass at installation and no training. The router improves when you upgrade the backbone. And the entire additional parameter cost is 7.9 million, compared to the 1.2B to 16B parameters of the retrieve-and-rerank pipelines it outperforms.
Read the paper on arXiv