skill-comply

Measures whether coding agents follow skills and rules by running scenarios and classifying tool call traces.

Updated Mar 18, 2026
One-click install
npx skills add https://github.com/freedom909/real-estate-saas --skill skill-comply-freedom909
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: skill-comply
Source: https://github.com/freedom909/real-estate-saas/tree/main/.trae/skills/skill-comply
Command: npx skills add https://github.com/freedom909/real-estate-saas --skill skill-comply-freedom909

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pyyaml, pytest, and includes scripts (resource) components.

What problem does it solve? Teams write skills, rules, and agent definitions for coding agents but have no way to verify the agent actually follows them. This Skill quantifies behavioral compliance by auto-generating test scenarios, executing agents, and reporting compliance rates with full tool call timelines. ## Core Features & Use Cases - Automated Spec Generation: Extracts expected behavioral sequences from any skill or rule Markdown file using an LLM, producing a YAML compliance spec with ordering constraints. - Three-Level Scenario Testing: Generates supportive, neutral, and competing prompts for the same task to measure whether a rule is followed even without prompt support. - LLM-Based Trace Classification: Classifies captured tool calls against spec steps by meaning rather than regex, then checks temporal ordering deterministically. - Use Case: After adding a new TDD rule to your agent configuration, run the tool against the rule file to see the compliance rate across three prompt strictness levels and identify which steps need hook promotion. ## Quick Start Run the compliance measurement on a rule file with the command: uv run python -m scripts.run path/to/rule.md, or add --dry-run to preview the generated spec and scenarios without execution cost.

Frequently Asked Questions about skill-comply

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I measure if a coding agent follows my rules?▼

Run the CLI with the path to your rule or skill file: uv run python -m scripts.run <path>. The tool generates a behavioral spec, creates three test scenarios at decreasing prompt strictness, executes the agent, and reports compliance rates per step.

How to test skill compliance without spending API credits?▼

Use the --dry-run flag, which generates only the compliance spec and scenario prompts without executing any agent runs. This lets you review the extracted behavioral steps and test prompts at zero cost before committing to a full run.

What types of agent definitions can be tested for compliance?▼

The tool supports skills defined in SKILL.md files, mandatory rules in Markdown files such as testing or security guidelines, and agent definition files. For agent definitions it verifies invocation timing, though internal workflow verification is not yet supported.

Does the compliance classifier use regex pattern matching?▼

No, tool call classification uses an LLM that matches calls to spec steps by meaning rather than keywords. Only the temporal ordering checks between steps are performed deterministically after classification.

Why does a compliance run fail with a subprocess error?▼

Failures occur when the claude CLI is not installed, returns a non-zero exit code, or exceeds the timeout limit. Check that claude is available in your PATH and that the specified model is one of the allowed values: haiku, sonnet, or opus.

What are the limitations of automated compliance measurement?▼

Results depend on LLM classification accuracy and the quality of auto-generated specs, which may need review via dry runs. Scenarios are limited to tasks completable in under 30 tool calls, and agent definition testing only covers invocation, not internal behavior.