The per-PR floor,
$ rendered as a pipeline
What a 3-person team can run on every .claude/** change without standing up an eval platform — deterministic gates first, one cheap headless model check last. Each stage below is a real command and the tool that runs it.
For a small team the realistic floor is deterministic, no-model-call checks gated on .claude/** paths: plugin validate --strict, JSON-schema validation against the official SchemaStore schemas, one sub-second frontmatter linter, and a "does it load" smoke test. Add at most one headless claude -p review — only after the deterministic layer is green. A full eval platform is overkill here, and honestly little is standardized yet. [2][1]
The pipeline
Left to right: each gate must pass before the next runs. The first three lanes never call a model — they finish in seconds, cost nothing, and can't flake. The fourth lane is optional, spends tokens, and is advisory only.
on: pull_request — paths: ['.claude/**', '.claude-plugin/**', '.mcp.json']
— the cost-control lever: gates run only when extension files change [6]
Deterministic lint
Manifest structure + SKILL.md frontmatter
No model callclaude plugin validate . --strict # then one frontmatter linter: npx pulser eval
runs: claude plugin validate (built-in) + pulser ⭐ 17
Catches: misspelled / leftover manifest fields (--strict = warnings-as-errors), unparseable YAML, missing description, the 1,536-char listing cap. [2][3]
Schema validate
Manifests + MCP defs vs SchemaStore
No model callnpx ajv-cli validate \ -s https://json.schemastore.org/\ claude-code-plugin-manifest.json \ -d ".claude-plugin/plugin.json"
runs: ajv-cli / check-jsonschema → official SchemaStore schema
Catches: structurally invalid plugin.json/marketplace.json/.mcp.json. The community schema repo is now archived because Anthropic's schemas shipped to SchemaStore — use the official path. [2][8]
Load smoke test
"Does it actually load?"
One trivial runclaude --bare -p "ok" --plugin-dir ./plug \ --output-format stream-json --verbose \ | jq -e '.plugin_errors | length == 0'
runs: claude -p headless → parse system/init
Catches: plugins that fail to load. Docs say outright: "use the plugin fields to fail CI when a plugin did not load." --bare = same result on every machine; for MCP add --strict-mcp-config (⚠ doesn't override disabledMcpServers — pin a clean HOME). [1][4]
Headless model check
Optional · advisory · spends tokens
Optional · 1 model callgit diff origin/main | claude --bare -p \ --append-system-prompt "review CC skills…" \ --output-format json \ --json-schema '{…"issues":[…]}' \ | jq -e '.structured_output.issues|length==0'
runs: claude -p · or claude-code-action ⭐ 8.1k
Catches: descriptions that won't trigger, references to missing files. Schema-conforming output lands in structured_output; payload carries total_cost_usd to budget. Keep it a single advisory check, not a hard gate. [1]
Any PR-triggered model run is a prompt-injection surface. A June 2026 disclosure showed a flaw where one malicious issue could hijack repositories through the Claude Code GitHub Action. Pin action versions, scope --allowedTools tightly (or --permission-mode dontAsk), and never grant write/secrets to fork-triggered runs. [14]
Stage-01 linters — pick exactly one
They overlap heavily; running more than one just duplicates findings. Stars are a triage signal — most are weeks-to-months old and single-maintainer. skill-validator is the most feature-complete (and the only one with built-in CI annotations); pulser is the fastest zero-dep option.
Validates plugin.json / marketplace.json structure. --strict = warnings-as-errors; wrong-typed fields fail even without it.
Spec compliance, token counts (o200k_base), link resolution (HEAD), file hygiene, unclosed code fences. Optional LLM-judge score evaluate mode.
5 checks: YAML parse, required fields, description quality, file structure, cross-refs. ~200ms / 40 skills, zero deps.
Cross-agent skill schema (Claude / Codex / Cursor / …). name + description core plus per-flavor rules — keeps a skill portable.
Validates against the Agent Skills open standard (free). Pro adds anti-slop, WCAG, and security scanning.
Pioneered JSON-schema for manifests — now archived, superseded by the official SchemaStore schemas. Use stage 02 instead.
Why the cheap layer is where the value is
.claude/** — paths filter keeps it off every other PR. [6]claude plugin validate); it's intentionally lenient, so deterministic community checks have a high catch rate. [2]Marketplace-scale precedent & test corpora
Large marketplaces already gate manifests exactly this way — and these repos double as fixtures to test your own tooling against.
Runs claude plugin validate over marketplace.json + every plugin.json in CI so invalid manifests can't merge. [17]
The reference marketplace.json to diff your schema against. [18]
A clean small skill collection — a starting corpus to test tooling against. [19]
The minimum bar, in priority order
Stop wherever the budget for maintaining checks runs out. Steps 1–5 are the floor and they're enough.
claude -p --output-format stream-json → assert empty plugin_errors. [1]one trivial runAnything beyond this — golden datasets, LLM-as-judge rubrics, statistical delta gates — is the eval-platform territory this brief deliberately leaves out. A local claude doctor belongs in the contributor checklist, not the gate. [13]
Sources
// stages 01–03 are the floor — deterministic, free, non-flaky. stage 04 is the only place you spend a token.