skill-evaluator

Name: skill-evaluator
Availability: InStock
Author: chenhom

Evaluate skill performance against baselines and analyze failure patterns across test cases.

Updated Dec 26, 2022

Search Tags:#evaluation #benchmark #baseline #rubric #transcript-analysis #blind-comparison

Ready to run

npx skills add https://github.com/ChenHom/note --skill skill-evaluator-chenhom

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: skill-evaluator
Source: https://github.com/ChenHom/note/tree/main/ai/skills/skill-evaluator/skill-evaluator
Command: npx skills add https://github.com/ChenHom/note --skill skill-evaluator-chenhom

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Use this Skill to assess whether another Skill is performing correctly, compare it against a baseline, and analyze failure patterns across test cases. It focuses on eval prompts, assertions / rubrics, benchmark summaries, blind comparisons, and regression tracking.

Core Features & Use Cases

Build and curate eval prompts and rubrics to measure skill performance.
Compare outputs against baselines and track regression across versions.
Analyze fail clusters, variance, and edge cases to guide improvements.

Quick Start

Provide an evaluation plan by creating eval prompts, rubrics, and a benchmark; then run a blind comparison and summarize results.

Frequently Asked Questions about skill-evaluator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I evaluate skill performance against a baseline?▼

You can evaluate skill performance against a baseline by creating eval prompts and rubrics, then running a blind comparison to track regression across versions. This identifies how well a skill performs and highlights failure patterns across test cases.

What is rubric-based grading for benchmark regression analysis?▼

Rubric-based grading for benchmark regression analysis is a process of scoring outputs against predefined criteria to track performance changes across versions. It helps analyze fail clusters, variance, and edge cases to guide improvements.

How do I build eval prompts and rubrics to measure skill performance?▼

You build eval prompts and rubrics by defining specific test cases and grading criteria within an evaluation plan. This process measures skill performance by applying deterministic checks and summarizing benchmark results.

Can I run a blind comparison to compare outputs across skills?▼

Yes, you can run a blind comparison to compare outputs across skills by reading and comparing SKILL.md frontmatter and related evaluation data. This supports validating eval prompts and highlighting variance.

Does this approach support deterministic checks and benchmark summaries?▼

Yes, this approach supports deterministic checks and benchmark summaries to validate skills. It requires reading and comparing SKILL.md frontmatter and related evaluation data to accurately assess performance against a baseline.

When do I need transcript analysis for failure patterns in test cases?▼

You need transcript analysis for failure patterns in test cases when you want to identify how well a skill performs against a baseline. It helps analyze fail clusters and edge cases to guide improvements and track regression.