skill-comply

Measures whether agents follow skills and rules by running scenarios and classifying tool call traces.

1|Updated Oct 11, 2025
One-click install
npx skills add https://github.com/ibytechaos/claude --skill skill-comply-ibytechaos
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: skill-comply
Source: https://github.com/ibytechaos/claude/tree/main/plugins/everything-claude-code/skills/skill-comply
Command: npx skills add https://github.com/ibytechaos/claude --skill skill-comply-ibytechaos

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pyyaml, pytest, and includes scripts (resource) components.

What problem does it solve? Teams write skills, rules, and agent definitions for Claude Code but have no way to verify whether agents actually follow them in practice. This Skill quantifies real behavioral compliance instead of assuming instructions are obeyed. ## Core Features & Use Cases - Automatic Spec Generation: Extracts an expected behavioral sequence (ordered steps with required flags) from any SKILL.md, rule, or agent definition file using an LLM. - Three-Level Scenario Testing: Generates supportive, neutral, and competing prompts for the same task to measure prompt independence — whether a rule is followed even when the prompt does not ask for it. - LLM-Based Trace Classification: Runs scenarios via claude -p with stream-json output, classifies each tool call against spec steps by meaning (not regex), and checks temporal ordering deterministically. - Use Case: After adding a new TDD rule to your project, run a dry run to inspect the generated spec, then execute full scenarios to get a compliance report showing which steps agents skip and whether hook promotion is recommended. ## Quick Start Run the skill-comply analysis on my testing.md rule file to measure whether agents actually follow it.

Frequently Asked Questions about skill-comply

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I measure whether Claude Code follows my custom rules?

Run skill-comply against the rule file with uv run python -m scripts.run followed by the file path. It generates a behavioral spec, executes three scenarios at decreasing prompt strictness, and reports a compliance rate per step.

How to test a Claude Code skill without spending API credits?

Use the --dry-run flag, which generates the compliance spec and the three test scenarios without executing any agent runs. This lets you review the extracted steps and prompts at zero cost before a full run.

What types of files can skill-comply analyze?

It supports skills (SKILL.md files), rules (Markdown files such as testing.md or security.md), and agent definitions. For agent definitions it verifies whether the agent gets invoked, though internal workflow verification is not yet supported.

Does skill-comply use regex to detect rule compliance?

No. Tool calls are classified against spec steps by an LLM based on the meaning of each call, not keyword patterns. Only the temporal ordering checks (before_step/after_step) are deterministic.

What are the limitations of automated skill compliance testing?

Each scenario run invokes claude -p, so full runs consume API usage and take time. Classification depends on LLM judgment, and agent definition testing only covers invocation, not internal behavior.