skill-tester

To create an evaluation rubric, define structural criteria, pedagogical goals, and anti-patterns for the AI skill. This establishes clear quality standards and specific checks to measure skill performance against defined objectives.

Skill testing is the process of assessing AI performance against defined quality standards using rubrics. You need it to ensure your AI skills are effective, reliable, and consistently meet pedagogical objectives before deployment.

Design test scenarios by crafting specific user interactions that probe various situations. This helps assess actual skill usage against the defined rubric and provides detailed feedback on performance.

Yes, you can support both automated and human evaluation of skill performance. The rubric defines quality criteria and anti-patterns, enabling comprehensive conversation evaluation against consistent standards.

Generating comprehensive rubrics requires an understanding of skill mechanics and pedagogical objectives. This knowledge allows you to define accurate structural criteria, anti-patterns, and specific test scenarios for evaluation.

The best way to prevent anti-patterns is to explicitly define them within the evaluation rubric. This allows you to assess actual skill usage and ensure the AI consistently avoids undesirable behaviors like giving unauthorized advice.