skills.rest
david-driscoll
david-driscollCOMMUNITY
Explore all skills by this author →
Website𝕏@david_dotnetGitHub

instruction-eval

Evaluate Claude skill and CLAUDE.md updates with PASS/PARTIAL/FAIL scoring.

1|Updated Apr 6, 2025
Search Tags:#regression testing#skill routing#anthropic api#instruction evaluation#claude posture#git ab testing#ci reporting
One-click install
npx skills add https://github.com/david-driscoll/stargate-command-cluster --skill instruction-eval
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: instruction-eval
Source: https://github.com/david-driscoll/stargate-command-cluster/tree/main/.agents/skills/instruction-eval
Command: npx skills add https://github.com/david-driscoll/stargate-command-cluster --skill instruction-eval

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires anthropic, pyyaml, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill helps you detect whether updates to skills or CLAUDE.md improve or degrade Claude’s operational behavior, preventing silent regressions before they reach users.

Core Features & Use Cases

  • Spot-Check Validation: Run interactive probes using the test prompts in the skill’s references to quickly judge routing and constraint compliance.
  • Automated Regression Testing: Execute the evaluation runner to systematically score probes and produce CI-friendly results.
  • Actionable PASS/PARTIAL/FAIL Interpretation: Translate scoring outcomes into concrete next steps, including when manual review is required.
  • Git-Aware A/B Checks: Compare behavior across git refs to pinpoint whether a change set caused improvements or regressions.

Quick Start

Run python .claude/skills/instruction-eval/scripts/run-eval.py --json to execute all probes and generate a machine-readable CI report.

Frequently Asked Questions about instruction-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I prevent Claude skill changes from breaking routing behavior?▼

Run automated evaluation probes against test prompts in your skill's references to score routing and constraint compliance using PASS, PARTIAL, or FAIL outcomes. This detects whether updates to CLAUDE.md or skills degrade operational posture before changes reach production.

How do I set up regression testing for Claude instructions and CLAUDE.md updates?▼

Define test prompts in tests.yaml files under .claude/tests.yaml and .claude/skills/*/tests.yaml, then execute the evaluation runner to systematically score probes and generate CI-friendly reports validating your instruction updates.

Can I compare Claude's operational behavior across git branches?▼

Compare Claude's operational behavior across git refs using Git-Aware A/B checks to pinpoint whether a specific change set caused improvements or regressions in routing behavior and operational posture.

Does the instruction evaluation runner support CI reporting?▼

The instruction evaluation runner supports CI reporting by executing python .claude/skills/instruction-eval/scripts/run-eval.py --json to run all probes and generate a machine-readable report for automated pipelines.

Do I need the Anthropic API to run Claude skill evaluations?▼

You need the Anthropic API and pyyaml dependencies to run Claude skill evaluations, enabling the runner to interact with Claude and parse the required tests.yaml configuration files for probe discovery.

What does a PARTIAL score mean when evaluating Claude skill updates?▼

A PARTIAL score indicates partial compliance with routing and constraint expectations during evaluation, signaling that manual review is required to determine if the Claude skill update adequately supports the intended operational behavior.

Related Skills

Explore More Software Engineering Skills→
ktg-onektg-one

skill-lookup

Search, retrieve, and install Claude skills from a directory.

Community
Intermediate
BerryKuipersBerryKuipers
7

validate-coverage-threshold

Validate JavaScript/TypeScript test coverage against configured thresholds from coverage reports.

Community
Intermediate
zenbuappszenbuapps

doc-sync-playbook

Synchronize CLAUDE.md with .claude/rules/*.md using git-diff analysis.

Official
Advanced
AllBeingsFutureAllBeingsFuture
9

skill-creator

Create and iterate AI skill definitions with evaluation workflows.

Community
Advanced