When Skill Libraries Grow: Why Agents "Can't Find, Can't Tell Apart, Can't Let Go" (In-Depth Review)

1. Background: from "progressive disclosure" to "skill-library governance"

When Anthropic introduced Agent Skills in October 2025, it summarized the core design principle as progressive disclosure: at startup, only each skill's name and description go into the system prompt; when the agent decides a skill is relevant, it reads the full SKILL.md; finer-grained attachments are opened as needed [1]. This means a single skill can hold almost unlimited content without hitting context limits, and it makes the "name and description" the key basis on which the agent decides whether to use a skill [1].

Anthropic also cautions in the same post that skills can introduce security risks: malicious skills may introduce vulnerabilities or induce data exfiltration, so it recommends installing skills only from trusted sources and reviewing their code dependencies and external network connections before use [1]. In other words, skill-library governance is not only an efficiency issue from day one; it's also a security and compliance issue.

This is exactly where the problem lies. Progressive disclosure solves "a single skill is too big," but leaves "too many skills" to the model: every additional skill adds another description to the system prompt and another candidate the model has to rule out. October's research breaks the scaling problem of skill libraries into four stages:

1. Find: locate relevant skills in a large library [2][3][4].
2. Distinguish: pick the truly suitable one among similar skills, and handle conflicts between skills and memory [5][9].
3. Keep: judge whether a skill is still useful, has been internalized by the model, or helps no one, and decide whether to keep, retire or delete it [6][7][8].
4. Evolve: where skills come from, how they evolve with a codebase, and whether they can be "compiled" into more stable programs [10][11][12].

2. Find: routing collapse and retrieval representations

2.1 Implicit routing "collapses"

Lost in Implicit Routing (2026-10-13) [2] runs a clean experiment: keep SkillsBench's tasks and their "gold skills" fixed and only add irrelevant skills to the library. Per the paper, after adding 1,000 irrelevant skills, skill-loading recall drops from 73.6% to 27.4%, and the share of runs that load no skill at all rises from 9.3% to 51.6%. When the model instead makes an explicit selection from the same library, recall is still 73.9% [2].

The authors rule out the explanation that "the model loaded other plausible-looking skills" — that happens too rarely to explain the drop. Two pieces of evidence are key: masking only the descriptions of irrelevant skills largely restores loading behavior; and with complete metadata, even explicitly asking the agent to plan its skill use first still leaves recall below preselection. The authors therefore argue there are two kinds of interference: semantic interference from descriptions, and instruction interference when "choosing skills" competes with "doing the task" [2]. The paper also evaluates the coverage and latency trade-offs of explicit preselection as a mitigation.

Analogy. A new hire given a 10-page operations manual will flip through it when a problem comes up. Given a 1,000-page manual with a dense table of contents, they may well just wing it from experience. It's not that the answer isn't in the manual; it's that "finding the answer" itself crowds out attention for doing the work.

2.2 Browsing on demand: SkillBrowser

SkillBrowser (2026-10-16) [3] proposes "selective skill exposure": provide relevant procedural content when needed while avoiding unnecessary exposure that interferes with reasoning. Progressive disclosure keeps stuffing more metadata into context, and static retrieval only searches on the initial question, which can miss capability needs that surface during reasoning. SkillBrowser uses a "meta-skill" to manage the process: the agent identifies current capability needs, constructs targeted queries and loads on demand. Per the paper, across six SRA-Bench subsets and five models, its overall accuracy beats static retrieval and LLM-based selection, and it mitigates accuracy drops on "instances that were already solvable." The authors also acknowledge that gains depend on the model and task, and that insufficient calls to useful skills and extra latency are the main limitations [3].

2.3 The design space of retrieval representations: SkillRep

If you retrieve, you have to answer "how do you compare a request with a skill?" User requests and skill documents are written for different purposes in different words. SkillRep (2026-10-23) [4] breaks the retriever's representation design into five choices — encoder fine-tuning, query-side expansion, document-side expansion, background projection and a lexical channel — and evaluates them together as modules of the same retriever. Per the paper, with a 0.6B encoder plus LLM expansion, SkillRep's MRR@10 on SRA-Bench is 7.02 points above the strongest same-size retriever on the official skill pool and 8.79 points above it on UniPool (a 111,247-skill pool the authors built from public registries), and it also performs better on SkillsBench [4].

More interesting is how it measures "the gap." The authors characterize the gap between requests and skills with three quantities: background (embedding components that can't distinguish skills), unspoken vocabulary (content in the gold skill that the request doesn't mention) and dilution (information lost when a skill is encoded as a single vector). The results show that each module's value depends on the others; for example, fine-tuning reverses the ranking of four training-free modules, turning them from complementary to overlapping. Modules that shrink the same part of the gap compete with each other, while those that shrink different parts complement each other [4]. The practical takeaway: don't just stack retrieval tricks; first measure which part of the gap you're dealing with.

3. Distinguish: similar skills and conflicting sources

3.1 Algorithm breakdown ①: SKALOS's "weakest witness" scoring

Intuition (illustrative). A library has three skills all called "PDF processing": one extracts tables, one fills forms and one merges files. The user wants to "fill form data into a PDF and merge it into one document." Ranked by text similarity, all three look alike; what's actually needed is an implementation that has both "fill forms" and "merge." SKALOS (2026-10-26) [5] compares skills with capability graphs rather than just description text.

Analogy. A job posting requires "Python and SQL." One candidate is an expert in Python and has only heard of SQL; another is proficient in both. By "average score" the first may rank higher, but by "weakest area" only the second meets the requirement. SKALOS scores by "weakest area."

Steps [5]:

1. Build the graph. Construct a multi-level capability graph for each skill, aligning equivalent expressions across skills while keeping skill-specific capability evidence.
2. Extract requirements. Extract the list of required capabilities from the query.
3. Weakest-witness scoring. Use a Max–Min semiring: a ⊕ b = max(a, b), a ⊗ b = min(a, b). A support path scores as its weakest node or relation (⊗ = min); among alternative paths, take the strongest (⊕ = max). Also check that all requirements are jointly supported within the same implementation.
4. Calibrate. Use ranking feedback to adjust unreliable extracted evidence and reduce the impact of extraction errors.

def witness_score(skill, requirements):
per_req = []
for r in requirements:
branches = skill.graph.paths_supporting(r)            # support paths within one implementation
per_req.append(max(min(edge.conf for edge in b) for b in branches) if branches else 0)
return min(per_req)                                      # every requirement must hold: take the weakest
rerank(candidates, key=witness_score)

Results (per the paper). On SkillRet, SKALOS's NDCG@10 is 2.68 points above a matched text-only reranker; using witness paths to guide skill selection raises the ALFWorld task success rate from 22% to 29% [5].

When it fails. Capability graphs depend on extracting capability elements from skill documents. When documents are vague or vary widely in format, extraction errors feed directly into the min operation, and one underestimated link can drag down an entire skill's score. The calibration step mitigates this but doesn't eliminate it.

3.2 When skills and memory disagree, who wins?

Skills aren't the only thing injected into context. Production agents often run episodic memory and procedural skills at the same time, placing both in the same context with no defined priority. Navigating Epistemic Parity (2026-10-23) [9] calls this state "epistemic parity" and builds 52 scenarios (four conflict types: fact, format, procedure and permission), running 1,383 times on three open-weight models. Per the paper: no source always wins, and the outcome depends on conflict type, model and injection order. Moving the skill block closer to the task raised the skill's win rate in all six cells of fact and procedure conflicts but reversed it for format conflicts, showing that the arbitration follows token position rather than source semantics. Explicitly stating "skills take absolute priority" raised compliance on the two stronger models, but 3%–10% still didn't comply, and it pushed the smallest model in the wrong direction on fact and procedure conflicts. Especially worrying: in runs where the agent acted via tool calls, 98.8% included no text at all, and 89% of runs never disclosed that a contradiction existed — silent arbitration is the default behavior [9].

4. Keep: internalization, retirement and lifecycle

4.1 Algorithm breakdown ②: internalization judgment and the validity gate

The problem. A public skill library may hold around a hundred thousand skills serving many models. As models get stronger, they "learn" some skills, and continuing to load them just wastes context. Existing retirement rules compare the gain between a run "with the skill" and one "without," and retire the skill when the gain is near zero. Beyond Gain (2026-10-23) [6] points out a blind spot: near-zero gain has two causes — the model has internalized the skill (it doesn't need it), or neither the model nor the skill can do the task (which the paper calls "uncovered"). The former should be retired; the latter should not.

Analogy. A tutoring class makes no difference to a student's grades. Maybe the student already knew the material, or maybe the class just can't teach it. Looking only at "the difference before and after the class," you can't tell the two apart; you have to know the student's level "without the class."

Steps (per the paper's definitions [6]):

1. Look only at capability items. For tasks synthesized from a skill, pick the rubric items C that "can only be passed using information in the skill" (not generic items like "file exists" or "code compiles").
2. Compute two scores. Run k times without the skill to get the unassisted capability score U; run with the skill to get the assisted capability score A. Both are average pass rates on capability items.
3. Validity gate. A high U might only mean the task is too easy. So accept a task only if at least one "witness model" benefits from the skill on the same capability items: max_w (A_w − U_w) > γ, with γ = 0.1.
4. Level rule. After passing the gate, judge the skill internalized and retirable only if U ≥ τ = 0.8.

def retire(skill, target, witnesses, task):
C = capability_items(task, skill)
if max(A(w, C) - U(w, C) for w in witnesses) <= 0.1:
return "undetermined"   # no model benefits from the skill: a high score may just mean the task is too easy
return "retire" if U(target, C) >= 0.8 else "keep"

Small example (illustrative). Skill S teaches "the configuration migration steps for an open-source library." Without S, the target model's capability pass rate is U = 0.85, so it looks like it already knows. But if three witness models score the same with or without S, the task may not need S's information at all; the gate doesn't pass and the skill can't be retired on that basis. Conversely, if a weaker witness model rises from 0.3 to 0.7 with S, the gate passes, and the target model's 0.85 is real evidence of internalization.

Results (per the paper). The gate passed 844 of 1,165 scored tasks (72.4%), but only 52.6% of tasks with U ≥ 0.8. In other words, nearly half of the high scores that "look internalized" came from tasks where no model could benefit from the skill. Under four models, applying the published gain rule with one task per skill deletes 58%–65% of "uncovered" skills, while the level rule deletes 0%. On SkillsBench, 28 of 31 retirements were based on probes that no run fully passed, while retiring by unassisted score caused no detectable reward loss. Unassisted judgments replicate across tasks more consistently than the gain rule (Cohen's kappa 0.42 vs. 0.23), but they differ across models, so skill libraries must read judgments separately per model. The authors released 1,251 skills with their tasks and rubric items under four models [6].

When it fails. The judgment depends on the quality of rubric items. In the paper, the target model Qwen3.6-27B also served as the judge, and the authors discuss this judge dependence in their limitations [6]. The choice of witness models also affects the gate: the authors found that two witness models recover 85%–90% of the information from three, and including a weaker model keeps the gate informative as models get stronger [6].

4.2 Algorithm breakdown ③: TrackRL — randomized controlled trials during training

Intuition. Beyond Gain addresses "how to retire skills in an inference-time library"; TrackRL (2026-10-18) [7] addresses "how to manage a skill library during reinforcement-learning training." During training, some skills become redundant as the policy improves, some stay useful, and some become harmful. The difficulty: skills retrieved together "free-ride" on each other, task difficulty confounds the effect, and there's no counterfactual evidence for individual skills.

Analogy. To find out whether a fertilizer works, a more reliable way is to randomly mark off fertilized and unfertilized plots in the same field. TrackRL turns what GRPO already does — "sampling multiple trajectories for the same task" — into a free randomized controlled trial.

Steps [7]:

1. Randomized injection. Within each GRPO group, decide independently for each selected skill whether to inject it with Bernoulli(p), producing "with skill / without skill" arms for the same task.
2. Compare within groups, accumulate across groups. Skill utility u_s is defined as the change in success rate from injecting vs. not injecting; evidence is accumulated across groups with reliability weighting. A "mastery" score m_s is tracked at the same time, reflecting how far the model has internalized the capability.
3. Evidence-gated state machine. A new skill is admitted only if it is dissimilar both to existing skills and to deleted harmful skills. Once admitted, it enters a protected incubation period and cannot exit while evidence is insufficient. Once evidence is sufficient:

• Delete: the upper confidence bound of utility is below zero (u_s + λ·SE < 0); permanently excluded.
• Retire: utility ≤ 0 and the lower confidence bound of mastery is above both an absolute threshold and the baseline for similar tasks.
• Keep: all other skills stay in the library as long-term "scaffolding."
• A soft capacity cap controls library growth. If forgetting makes a retired skill necessary again, later failures trigger its regeneration and readmission [7].

for group in grpo_groups:
for s in selected_skills(group.task):
z[s, e] ~ Bernoulli(p)  for e in group.episodes       # independent randomized injection
rollout(group, inject=z)
for s: update_utility(s, mean(R | z=1) - mean(R | z=0), weight=reliability(group))
lifecycle_step(library)   # incubate → keep / retire / delete, under a soft capacity cap

Results. The paper reports that TrackRL outperforms recent skill-augmented RL baselines in both "with-library evaluation" and "no-skill evaluation," and analyzes how skill value changes over the course of policy learning [7].

When it fails. Randomized injection means some trajectories deliberately go without useful skills, sacrificing success on part of the training samples. When groups are small, single-group comparisons are noisy, and more groups are needed to accumulate enough evidence.

4.3 Selective injection: SkillScaffold

SkillScaffold (2026-10-13) [8] approaches the same problem from the other side: estimating skill utility by sampling is expensive and unstable when samples are limited. It reuses outcome-verified training trajectories to generate candidate skills, estimates utility through "outcome-aware likelihood comparison" without sampling separately for each candidate, and falls back to skill-free execution when no candidate is estimated to help. The paper reports gains over strong baselines on ALFWorld, WebShop, ScienceWorld and τ²-Bench, and stresses the importance of "utility-based selection and abstention" [8].

5. Evolve: where skills come from, and what they become

• Hierarchical distillation from coding trajectories. HSE (2026-10-26) [11] first extracts task-level evidence from individual trajectories, then aggregates it into repository-level candidates within each repository, and finally consolidates patterns consistent across repositories into transferable skills. It's driven by an external "writer–evaluator" loop while the coding model itself stays frozen. Per the paper, just 10 skills evolved from SWE-Gym improved coding performance in every evaluation setting on SWE-bench Verified and DeepSWE, across multiple models and harnesses [11].
• Continuous evolution with pull requests. Skill Dreaming (2026-10-26) [12] extracts actionable knowledge from verified pull requests to build a repository-specific skill library and keeps updating it with information from new pull requests. The skills guide not only feature implementation but also maintenance of tests, documentation, compatibility and repository conventions [12].
• "Compiling" skills into programs. FlashScript (2026-10-24) [10] argues that for recurring workflows, skills still make the agent rerun the full process every time, with inconsistent results. It compiles a successful run into a neuro-symbolic program: actions and data transformations become code, instance-specific values become parameters, and places that need model judgment become LLM calls. Per the paper, on 120 WorkBench analysis tasks (Claude Haiku 4.5), these programs were correct on every run with fully consistent tool calls, used about 1/70 of the input tokens of skill-guided agents, and took about a third of the time. On compositional variants, skill-guided agents' accuracy fell to 23% while the programs stayed fully correct, and the programs generalized to all 108 unseen instances of the same templates [10]. Compilation doesn't eliminate uncertainty; it localizes it, so deterministic programs and neural calls can be reviewed and tested separately [10].

6. Side-by-side comparison

Note: tasks, models and metrics differ across studies, so the numbers are not directly comparable.

Practice: a skill-library governance table

To put this research into engineering practice, we recommend keeping one row per skill with at least these fields:

The value of this table is that it turns "is this skill any good" from personal impressions into queryable, reviewable evidence. The table itself isn't complicated; the hard part is keeping it updated at every model upgrade and every new skill. Teams that have already accumulated hundreds of skills can start with the twenty most frequently used.

7. Our view

First, the bottleneck of skill libraries is shifting from "writing skills" to "managing skills." Writing a skill keeps getting cheaper, and agents can even generate them automatically [11][12]. But the larger the library, the more easily routing collapses [2] and conflicts get arbitrated silently [9]. Skill libraries need a catalog, tests and a retirement process, just like codebases.

Second, don't make the model pick skills while it's doing the work. One root cause of routing collapse is that "choosing skills" competes with "doing the task" [2]. When there are many skills, explicit preselection or on-demand browsing [3] is often more reliable than stuffing every description into the system prompt.

Third, retirement should look at "level without the skill," not just "gain with the skill." Zero gain may mean the model already learned it, or that no one can do it [6]. We recommend that after a model upgrade, enterprises run a per-model "unassisted capability test" on their skills and use a weaker model as a witness.

Fourth, frequent, stable workflows should move from skills to programs. FlashScript's results show that once a process is fixed, compiling it into a testable program brings clear advantages in consistency and cost [10]. Skills suit the parts that "need judgment"; programs suit the parts that "must be consistent." This matches the approach of Great Unicorn Intelligence's rule engine: hand deterministic business logic to rules, and leave the parts that need judgment to the model.

8. Outlook

6–12 months:

• Skill platforms will commonly provide explicit preselection or a retrieval layer, putting only a few high-confidence candidates in the system prompt instead of every description [2][3][4].
• Skill metadata will gain fields such as "applicable models," "internalization status" and "last verified date," with retirement judgments stored per model [6].
• Priority between memory and skills will be written into protocols, and agents will be required to disclose conflicts explicitly when they find them [9].

1–3 years:

• Skill libraries will get "continuous integration": each skill carries tasks and rubric items, is retested automatically on model upgrades, and the results drive keep, retire or delete decisions [6][7].
• "Skill → program" compilation will become a routine capability, with agents proactively suggesting compilation when they recognize a recurring workflow [10].
• Repository-level skills will evolve together with code, with pull requests serving as both the source and the test of skills [12].

9. Implications for financial risk control and enterprise deployment

1. Limit how many skills are exposed at once. As risk, compliance and operations teams each build up skills, the total grows fast. Preselect by business domain instead of attaching everything to one agent [2].
2. Distinguish similar skills by "weakest capability." For example, multiple "report parsing" skills should each be labeled with the report types and fields they support, and selected by capability matching rather than description similarity [5].
3. Set priority between memory and skills, and require disclosure. When a definition in historical memory conflicts with the definition in a current skill, the agent must state which one it used and why [9].
4. Run a skill health check after model upgrades. Use unassisted capability tests to decide which skills can be retired, and keep "witness model" evidence to avoid wrongful deletions [6].
5. Compile frequent fixed processes into rules or programs. Once processes like daily reconciliation and fixed-format reports are compiled into testable programs, consistency and cost are easier to control [10]; the parts that need judgment go to the model.

10. Conclusion

Skills turn experience into reusable assets, and once there are many assets, they need governance. October's research offers four levers: retrieval and preselection for "can't find," capability evidence for "can't tell apart," unassisted tests and randomized trials for "should it stay," and compilation and continuous evolution for "how does it get more stable." The larger the skill library, the more these levers matter.

To learn more about Great Unicorn Intelligence's work on rule engines and intelligent decision systems, visit www.hxallai.com.

References

1. [1]Anthropic. Equipping agents for the real world with Agent Skills. Anthropic Engineering Blog, 2025-10-16. (background) https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills
2. [2]Jinlin Liu, Wei Chen, Xiaojin Zhang. Lost in Implicit Routing: How LLM Agents Fail to Load Relevant Skills at Scale. arXiv, 2026-10-13. https://arxiv.org/abs/2610.18522
3. [3]Jin Ye, Yifan Jiang, Lan-zhe Guo. SkillBrowser: Selective Skill Exposure for Scalable Skill-Using Agents. arXiv, 2026-10-16. https://arxiv.org/abs/2610.23554
4. [4]Tianyi Sun, Pengfei Yao, Yanze Wang, Hongda Chen et al. SkillRep: Characterizing and Understanding the Representation Design Space of Agent Skill Retrieval. arXiv, 2026-10-23. https://arxiv.org/abs/2610.32336
5. [5]Wenhan Xiao, Ziwei Zhang, Xingcheng Fu, Chuanyue Yu et al. SKALOS: Learning to Distinguish Similar Skills with Capability Graphs. arXiv, 2026-10-26. https://arxiv.org/abs/2610.34728
6. [6]Yanze Wang, Pengfei Yao, Tianyi Sun, Hongda Chen et al. Beyond Gain: Measuring Skill Internalization for Agent Skill Retirement. arXiv, 2026-10-23. https://arxiv.org/abs/2610.32322
7. [7]Xi Zhu, Yuanlin Duan, Changyi Yang, Kai Mei et al. Trackable Skill Library Lifecycle Management for Agentic Reinforcement Learning. arXiv, 2026-10-18. https://arxiv.org/abs/2610.25126
8. [8]Wenlin Zhong, Hao Wang, Kun Yuan, Fan Yang et al. SkillScaffold: Scaffolding Agentic Reinforcement Learning with Policy-Adaptive Skill Credit Assignment and Selective Injection. arXiv, 2026-10-13. https://arxiv.org/abs/2610.18807
9. [9]Prakhar Pandey. Navigating Epistemic Parity in LLM Agents: A Benchmark for Cross-Source Conflict Resolution Between Memory and Procedural Skills. arXiv, 2026-10-23. https://arxiv.org/abs/2610.33128
10. [10]Sadra Sabouri, Hosein Hasanbeig, Adam Sage, Shreya Sinha et al. FlashScript: Compiling Recurring Agent Workflows into Reusable Neuro-Symbolic Programs. arXiv, 2026-10-24. https://arxiv.org/abs/2610.33876
11. [11]Jun Gao, Yun Peng, Changhai Zhou, Yuhan Zhu et al. Hierarchical Skill Evolution for Coding Agent. arXiv, 2026-10-26. https://arxiv.org/abs/2610.35616
12. [12]Xinyuan Song, Hasibul Haque, Jay Somasundaram, Ryan Turner et al. Skill Dreaming: Evolving Repository Skills through Pull Request Streams. arXiv, 2026-10-26. https://arxiv.org/abs/2610.36188