skill-comply

Measures whether coding agents follow skills and rules by running scenarios and grading tool call traces.

Updated Mar 25, 2026
One-click install
npx skills add https://github.com/Femad-6/my-skills --skill skill-comply-femad-6
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: skill-comply
Source: https://github.com/Femad-6/my-skills/tree/main/.github/skills/skill-comply
Command: npx skills add https://github.com/Femad-6/my-skills --skill skill-comply-femad-6

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pyyaml, pytest, and includes scripts (resource) components.

What problem does it solve? Teams write skills, rules, and agent definitions for coding agents but have no way to verify the agent actually follows them. This Skill quantifies behavioral compliance by auto-generating test scenarios, executing them, and reporting compliance rates with full tool call timelines. ## Core Features & Use Cases - Automatic Spec Generation: Extracts an expected behavioral sequence (ordered steps with required/optional flags) from any skill, rule, or agent definition Markdown file using an LLM. - Three Strictness Levels: Generates supportive, neutral, and competing prompt scenarios to test whether a rule is followed even when the prompt does not explicitly support it. - LLM-Based Classification with Deterministic Ordering: Classifies tool calls against spec steps by meaning (not regex), then checks before/after temporal constraints deterministically. - Self-Contained Reports: Produces Markdown reports with the spec, scenario prompts, per-scenario compliance scores, and annotated tool call timelines, plus optional hook promotion recommendations. - Use Case: After adding a new TDD rule to your agent configuration, run a dry run to inspect the generated spec, then execute full scenarios to see whether the agent writes tests before implementation across all three prompt strictness levels. ## Quick Start Ask the agent to run skill-comply against a specific rule or skill file, for example to measure whether your testing.md rule is actually being followed.

Frequently Asked Questions about skill-comply

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I measure whether an AI coding agent follows my rules?

Run skill-comply against the rule file with uv run python -m scripts.run followed by the file path. It generates a behavioral spec, executes three scenarios at decreasing prompt strictness, and reports a compliance rate per scenario.

How to test a skill file without spending money on model calls?

Use the --dry-run flag, which generates the compliance spec and the three test scenarios without executing them. This lets you review the extracted steps and prompts at no cost before committing to a full run.

What types of agent configuration files can be tested for compliance?

It supports skill files (SKILL.md), rule files such as testing.md or security.md, and agent definition files. For agent definitions it verifies whether the agent gets invoked when expected, though internal workflow verification is not yet supported.

Does skill compliance checking use regex to match tool calls?

No, tool calls are classified by an LLM based on the meaning of each call, not regex patterns. Only the temporal ordering checks (before_step and after_step constraints) are enforced deterministically after classification.

What are the limitations of automated skill compliance measurement?

It requires the claude CLI to be installed and runs scenarios in a sandbox under /tmp, so it is Linux-oriented. Classification depends on LLM output quality, and agent definition testing only covers invocation, not internal behavior.