AI Agent Skill Self-Evolution: Automated Learning Without GPU

An engineering practice record on how to make the AI Agent's skill system "come alive". Based on cutting-edge research like SkillRL (arXiv 2602.08234), a lightweight solution deployed on a GPU-less VPS.

TL;DR

We built a complete Agent Skill lifecycle management system:

• 56 skills, each with structured principle / when_to_apply / common_mistakes
• Semantic retrieval: embedding vector matching, not keyword guessing
• Failure closed-loop: wrong skill used → auto-record → auto-downgrade → auto-trigger evolution
• Reflection system: task ends → auto-decide if reflection is needed → extract patterns → avoid pitfalls next time
• SkillGraph: DAG path search + SkillBank field linking, supporting composition recommendations

Core code ~4500 lines Python + 160 lines Bash, fully TDD, 50+ tests all green.

1. Problem: Agent Skills Are "Dead"

Most Agent frameworks treat skills as a read-only library:

user request → match skill → load into prompt → execute → end

No feedback loop. When correct, you don't know why. When wrong, you don't know why. Skill quality depends entirely on manual maintenance.

Our goal is to close this loop:

user request → match skill → execute → success/failure → record outcome → update skill quality → better matching next time

2. System Architecture

┌─────────────────────────────────────────────────────┐
│                  Hermes Agent                        │
│                                                     │
│  ┌──────────┐  ┌──────────┐  ┌──────────────────┐  │
│  │ Skill     │  │ Skill    │  │ Unified          │  │
│  │ Index     │  │ Graph    │  │ Reflection       │  │
│  │ (Search)  │  │ (Compose)│  │ (Reflection)     │  │
│  └─────┬────┘  └────┬─────┘  └────────┬─────────┘  │
│        │            │                  │             │
│        ▼            ▼                  ▼             │
│  ┌─────────────────────────────────────────────┐    │
│  │           skill-index.json                   │    │
│  │  (56 skills, 384d embeddings, quality scores) │    │
│  └─────────────────────────────────────────────┘    │
│        │            │                  │             │
│        ▼            ▼                  ▼             │
│  ┌──────────┐  ┌──────────┐  ┌──────────────────┐  │
│  │ .usage.json│  │failure_  │  │ compound.sh      │  │
│  │ (Usage Stats)│  │log.jsonl │  │(Post-task Reflect)│  │
│  └──────────┘  └──────────┘  └──────────────────┘  │
└─────────────────────────────────────────────────────┘

Core Components

3. SkillBank Structuring: From Free Text to Structured Fields

Inspired by the SkillRL paper, we added three structured fields to each skill:

{
  "name": "code-review",
  "description": "Code review skill",
  "principle": "First understand the intent, then review the implementation; focus on correctness and maintainability",
  "when_to_apply": "When reviewing code, writing PRs, or checking quality",
  "common_mistakes": [
    "Only check syntax, not logic",
    "Ignore edge cases",
    "Don't verify test coverage"
  ]
}

These three fields are automatically extracted from the SKILL.md body:

def _extract_principle(body: str) -> str:
    """Extract core methodology from Steps / How to / first paragraph"""
    # Prioritize the first item under Steps
    # Fall back to How to section
    # Finally take the first paragraph
 
def _extract_when_to_apply(body: str) -> str:
    """Extract from When to use / Triggers / When to load"""
 
def _extract_common_mistakes(body: str) -> list[str]:
    """Extract from Notes/Pitfalls/Caveats/Warnings"""

Extraction rate: Among 56 skills, 44 have principle (79%), 31 have common_mistakes (55%), 23 have when_to_apply (41%).

Why This Matters

1. SkillGraph linking: suggest_composition() now matches not only provides but also when_to_apply
2. Enhanced recommendation: find_paths() results include principle and common_mistakes as warnings
3. RL training foundation: These structured fields are the reward signal foundation for future GRPO training

4. Semantic Retrieval: Embeddings Don't Guess

skill_index.py uses paraphrase-multilingual-MiniLM-L12-v2 (384-dim) for semantic search:

# During index construction: generate embedding for each skill
text = f"{name} {description} {tags} {triggers} {body_preview}"
embedding = model.encode(text, normalize_embeddings=True)
 
# At search time: query embedding vs all skill embeddings
scores = cosine_similarity(query_embedding, all_embeddings)
top_k = sorted(scores, reverse=True)[:limit]

Degradation strategy: When sentence-transformers is unavailable, automatically fall back to hash-based embedding (MD5 bucketing) to ensure functionality.

Actual results:

• "deploy" → matches server-operations (score 0.82)
• "debug" → matches debugging-toolkit (score 0.79)
• "review" → matches code-review (score 0.85)

5. Failure Closed-Loop: Record When Wrong, Fix When Recorded

5.1 Usage Tracking

Each time a skill is used, _update_skill_usage() updates .usage.json:

{
  "code-review": {
    "use_count": 12,
    "total_outcomes": 10,
    "success_rate": 0.7
  }
}

5.2 Failure-Driven Evolution

def _trigger_skill_evolution(event):
    """Consistently low success rate → mark for review"""
    if success_rate < 0.3 and total > 5:
        _flag_for_review(skill_name, entry)
    if "not found" in error_msg:
        _log_discovery_trigger(event)  # Trigger skill discovery

5.3 Auto-Evolve (New)

compound.sh reflect now automatically detects skill-related tasks:

# Extract skill names from files parameter
skills_from_path=$(echo "$files" | grep -oP 'skills/\K[^/]+' | sort -u)
skills_from_md=$(echo "$files" | grep -oP '([^/]+)/SKILL\.md' | ...)
 
# Auto-call evolve
for skill_name in $all_skills; do
    python3 unified_reflection.py evolve "$skill_name" "$outcome"
done

Effect: compound.sh reflect error_recovered high 0 0 "skills/code-review/SKILL.md" automatically marks code-review as a successful usage.