skill-testing-framework

What problem does it solve?

Ensuring AI skills function correctly and consistently across updates is challenging. This framework provides structured testing to prevent regressions, validate behavior, and maintain high-quality, reliable AI tools.

Core Features & Use Cases

• Multi-Level Testing: Supports unit tests for individual components, integration tests for complete workflows, and regression tests against known baselines.
• Test Case Generation: Automatically generates test templates based on skill structure, simplifying the initial setup and creation of test suites.
• Output Validation: Compares actual outputs against expected results using various methods like exact match, substring containment, or regex patterns.
• Baseline Management: Helps create, validate, and update baselines for robust regression testing, ensuring unintended changes are caught early.
• Use Case: After updating your 'pdf-processor' skill, you can run its test suite to verify that all PDF extraction, merging, and form-filling functions still work as expected, and that new features integrate seamlessly without breaking existing functionality.

Quick Start

Generate a test template for the skill located at '/path/to/my-new-skill' and save it as 'my-skill-tests.json'.