Agent Skill Evolution
Definition
Skill evolution is an LLM agent improving without touching model weights, by turning its own execution history into reusable procedural knowledge. A skill is a filesystem directory holding a SKILL.md with frontmatter (name, description) plus full instructions and applicability conditions — the same open format Claude Code uses.
WikiSkill (arXiv 2608.27454) is the framework that adds the missing middle layer. Earlier methods (Trace2Skill, EvoSkill, SkillOpt) all run the same loop — roll out, analyse traces, propose a skill edit, gate on validation — but keep no separate record of what was learned. Insight stays scattered across optimisation history and is re-derived every iteration. WikiSkill inserts a persistent knowledge base between raw experience and executable skill, and never rolls it back.
Core Ideas
The three-layer architecture
| Layer | Holds | Mutability |
|---|---|---|
Raw (raw/) | complete execution traces — reasoning, tool calls, outputs, answers | immutable |
Wiki (wiki/) | patterns/ documenting failure modes and successful strategies, plus index.md, logs.md, skill-impact.md | accumulates, never reset |
Skills (skills/) | the active skill set the inference agent reads — SKILL.md plus PURPOSE.md mapping each skill back to the patterns that motivated it | gated, can roll back |
The separation is the whole design. Raw experience is too long to reason over; skills are too compressed to carry the why. The wiki holds the middle: consolidated patterns with enough history that the next iteration does not repeat a rejected idea.
The evolutionary loop
- Inference Agent runs the training tasks with the current skill set, producing traces.
- Wiki Maintainer reads a stratified sample of passing and failing traces, does root-cause analysis, and patches the pattern pages — appending, replacing, inserting spans rather than rewriting. It updates
index.mdand appends tologs.md. - Skill Proposer works ReAct-style: it starts with only the wiki index, the skill-impact tracker, and a pass/fail summary, then calls
read_fileto pull the specific patterns and traces it needs. It emits one atomic proposal targeting a single skill. - Gating and rollback: the candidate is scored on a validation split. Better than the running best → accepted. Worse → the skills revert, but the wiki never rolls back. Either way the harness appends the proposal diff, validation score, and accept/reject verdict to
skill-impact.md.
That audit trail is the mechanism. In the paper’s ALFWorld case study, iteration 0 proposes goal-directed-action and it is rejected; because the rejection and its diff persist, iteration 1 proposes the sharper break-repetition-loop with a concrete rule (Never Return an Item to Its Origin Location), which is accepted and later refined again from new pattern evidence.
The results
Across five benchmarks (LiveMath, SealQA, SpreadsheetBench, OfficeQA, ALFWorld) and five models, WikiSkill has the highest average of every method tested, beating the strongest competitor by 3.3 to 12.0 points. Gemini-3.5-Flash goes 33.0% → 72.6% on LiveMath and 50.5% → 76.6% on SpreadSheet.
The ablation is the load-bearing result: giving the Skill Proposer wiki access lifts the average from 48.7% to 63.7% (+15.0). Persistent knowledge, not the loop itself, is what works.
Skills complement scale, they do not substitute for it
Within the Qwen family the gain grows with model size: +12.3 (4B), +17.5 (9B), +23.9 (27B). Yet skills also compensate for scale — Qwen-3.5-9B with WikiSkill (47.4%) beats Qwen-3.6-27B without skills (39.4%). Model capability and evolved procedural knowledge are complementary axes, not competing ones.
Skills transfer across models — and can transfer badly
Skills evolved by one model frequently improve another, sometimes beating self-evolved skills. Qwen-3.6-27B’s SpreadSheet skills take Qwen-3.5-9B to 50.5%, against 24.3% no-skill and 33.6% self-evolved. Transfer also runs upward: Qwen-3.5-4B skills improve the much larger Gemma-4-31B.
Negative transfer has a diagnosable cause. Qwen-3.5-4B’s SpreadSheet skills drop Gemini-3.5-Flash from 50.5% to 18.1%, because they encode low-level workarounds — single-line Python commands, string-conversion rules — that rescue a weak model but stop a strong one writing end-to-end scripts. General procedure transfers; model-specific crutches do not.
The paper separates two things self-evolution normally conflates: discovering useful procedural knowledge, and executing it at inference time. A model can write skills more useful to another model than to itself — Qwen-3.5-4B’s OfficeQA skills lower its own score (30.2% → 28.5%) while raising Qwen-3.6-27B’s (42.1% → 52.9%).
The counter-intuitive ablation
Giving the Inference Agent wiki access during training rollouts hurts (63.7% → 60.9%, LiveMath 72.6% → 64.8%). The hypothesis: when the agent can solve tasks from the wiki directly, its traces stop revealing what the skills fail to cover, so the traces become less informative for skill development. Keep the source of learning out of the run you are learning from.
Shape of the output
Skills stay compact while the wiki grows. Gemma-4-31B produces 45-line skills, Qwen models 119–129 lines; wiki patterns accumulate 6.3–8.9 creations and 7.0–18.4 edits per run. Refinement never stops early — only 39–52% of accepted updates land in the first two iterations.
Relationships
- Context Engineering — designing what the agent sees; skill evolution designs what it keeps between runs
- Loop Engineering — the outer loop that stop conditions and gating sit inside
- Knowledge Management — the human analogue: capture, consolidate, revisit so it compounds
- AI-Native SDLC — skills as versioned institutional knowledge in a real engineering process
- Coding Agents and AI Tools — parent topic
References
- WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution — arXiv 2608.27454