skill-evals

Automate skill quality measurement by building evaluation suites that consistently test trigger accuracy and output quality. These suites support both deterministic code-based graders and LLM-as-judge rubrics, enabling pre-deploy validation and continuous monitoring.

The best way to validate a skill before deployment is to run automated evaluation suites that verify response accuracy and adherence to predefined metrics. This framework provides pre-deploy validation to ensure trigger reliability and output quality before release.

Yes, you can use LLM-as-judge graders with defined rubrics for subjective evaluation of open-ended skill responses. This approach complements code-based deterministic graders, which are better suited for skills with predictable outputs.

Code-based graders work by implementing deterministic checks that test skills with predictable outputs. They provide consistent, metric-driven assessment of trigger accuracy, ensuring automated evaluation suites return reliable pass or fail results over time.

Yes, predefined quality metrics are required to set up continuous monitoring and pre-deploy validation. The evaluation framework uses these metrics alongside defined rubrics to consistently measure trigger accuracy and output quality over time.