Testing an authored Claude Code Skill before you ship it to a team

TL;DR. Test two things separately: (a) does the description fire on the right prompts and stay quiet on the wrong ones, and (b) does the body produce the right behavior once loaded [1]. The 2026 de-facto harness is Anthropic’s skill-creator eval loop (evals/evals.json → run with-skill vs without-skill → grade assertions → blind A/B), which got a dedicated description-tuning mode on March 3 2026 that scores should-trigger/should-not-trigger prompts and rewrites the description to cut false positives and false negatives [2][3]. For repeatable CI, the most concrete pattern is MLflow’s headless-Claude-Code-plus-judges harness [4]. Honest state of the art: trigger-matching is still ~50% reliable on autonomous activation, collision detection is manual, and a purpose-built “skill linter in CI” is mostly vaporware. Nobody has a clean push-button answer yet.

The core split: trigger vs. behavior

Anthropic’s own docs state the trap directly: “Seeing a skill trigger tells you Claude found it, not that it did what you intended. To know a skill is working, measure two things separately: whether Claude invokes it on the prompts it should, and whether the output matches what you expect when it does.” [1] Both are checked by the same primitive — a baseline comparison: run realistic prompts in a fresh session with the skill available and again with it disabled, then diff [1]. A fresh session is load-bearing — leftover authoring context masks gaps in the written instructions [1].

(a) Testing that the description triggers — and doesn’t over-trigger

The description field is the trigger; Claude fuzzy-matches against it to decide whether to load the skill, so a vague description silently never fires [6]. Autonomous activation is unreliable by default — practitioners report “roughly a 50% success rate” because Claude prioritises task completion over checking available tools [5].

skill-creator’s description-tuning mode is the only purpose-built trigger harness in 2026. It “generates should-trigger and should-not-trigger prompts, measures the hit rate, and proposes description edits when the skill activates on the wrong requests” [1]. The March 2026 upgrade frames this as a false-positive/false-negative classifier: it “analyzes your current description against sample prompts and suggests edits that cut both false positives and false negatives”; Anthropic ran it across their document-creation skills and “saw improved triggering on 5 out of 6 public skills” [3]. The wrapping tooling is branded the “Skill Description Improver” / grader / blind-comparator / analyzer set [10].

Manual/diagnostic checks that catch trigger failures before ship:

Pre-write the trigger evals. The practitioner consensus is to write the test cases before the SKILL.md body — “write three test cases for your skill before writing the SKILL.md body” (a positive, a negative, an edge case) so triggering has a measurable baseline [5].

(b) Testing the body behaves — snapshot & regression

This is the evals/evals.json loop, formalised in the Agent Skills open standard [2]. A test case is {prompt, expected_output, files, assertions}; the runner executes each prompt twice (with-skill / without-skill) in isolated subagent contexts, writing grading.json (per-assertion PASS/FAIL + evidence), timing.json (tokens, duration), and an aggregated benchmark.json with a delta block — what the skill costs (time, tokens) vs. what it buys (pass-rate lift) [2].

// evals/evals.json — the one file you author by hand
{ "skill_name": "csv-analyzer",
  "evals": [{
    "id": 1,
    "prompt": "find the top 3 months by revenue in data/sales_2025.csv and make a bar chart",
    "expected_output": "A bar chart of the top 3 months by revenue, labeled axes.",
    "files": ["evals/files/sales_2025.csv"],
    "assertions": ["The chart shows exactly 3 months", "Both axes are labeled"] }]}

Snapshot / regression patterns that actually map to Skills:

• Versioned baseline (the snapshot). Before editing, cp -r <skill> <workspace>/skill-snapshot/ and run the new version against the snapshot as baseline — old_skill/ vs the edit — instead of against no-skill [2]. This is the closest thing to a behavioral snapshot test.
• Blind A/B version comparison. skill-creator runs a blind comparison between two skill versions — the judge scores holistic quality without knowing which is which, “so you can confirm an edit is an improvement before committing it” and avoid confirmation bias [1][2].
• Iteration directories. Each pass writes to iteration-N/, so regressions are diffable across runs; flaky evals show up as high stddev in the benchmark → signal of an ambiguous instruction, not just model noise [2].
• Rule-based assertions over LLM judgment for mechanical checks (valid JSON, row counts, file exists) — “scripts are more reliable than LLM judgment for mechanical checks and reusable across iterations” [2].

Skill-specific CI

There is no first-party “test-this-skill” GitHub Action yet. The two real patterns:

MLflow’s headless harness is the most concrete CI-shaped approach: a YAML config (project_dir, skills, prompt, judges, timeout) drives mlflow autolog claude to trace every tool call, then LLM + rule-based judges grade the trace; a refinement loop feeds failing judge rationales back to claude -p "...Fix SKILL.md." [4]. Crucially for regression: multiple test configs cover the same skill so “a fix that addresses one failing test config must not cause another to regress” — overfit protection [4]. This runs anywhere a shell runs (claude -p headless), which is the generic CI hook [11].

Static linting in CI is where the hype outruns reality. A dev.to post describes a pulser zero-dependency CLI / GitHub Action that lints skill frontmatter (YAML parse, required fields, description-quality score, broken cross-refs) and fails the PR on exit code 1 [9]. ⚠ Flag: I could not verify pulser exists — no pulserin/pulser repo and no pulser npm package resolve as of 2026-06-24. Treat it as unverified/aspirational, not a tool you can adopt.

Detecting description collisions between skills

No tooling does this automatically — it’s manual and behavioral. The failure mode: “if two skills have overlapping descriptions you can get the wrong one… the choice is non-deterministic enough to be a debugging nightmare” [6]. Detection and mitigation in practice:

• Run skill-creator’s should-trigger/should-not-trigger set with both skills installed and watch which one fires — a should-not-trigger hit on skill B for skill A’s prompt is the collision signal [3].
• Add explicit disambiguation lines — “use this for X, not Y” / “Do not use for X” — after observed misfires; narrow each skill’s domain [6].
• Keep the active set small. Practitioner heuristic: “5–8 active skills per project”; beyond that, descriptions truncate and collisions multiply, and /doctor shows what’s being dropped [7][1].
• For workflows that must fire, bypass the trigger lottery entirely: a UserPromptSubmit hook emitting an explicit Use Skill(name) is deterministic where description matching is ~50% [5].

Where to get the harness

Talk thesis, answered

“How do you know the skill you authored actually works before you push it to your team?” — In 2026 the honest answer: run a with-skill/without-skill baseline on a handful of real prompts in a fresh session, let skill-creator tune the description against should/should-not-trigger sets, snapshot the prior version for a blind A/B before each edit, and wire claude -p evals into CI if you want regression gates. But say the quiet part out loud in the talk: autonomous triggering is still a coin-flip-ish ~50% [5], collision detection is eyeballing, and a real CI-native skill linter doesn’t exist yet [9]. The maturity gap is the story.