skill-evaluator

Execute eval suites combining structural checks and LLM rubric scoring.

1|Updated Jan 13, 2023

Search Tags:#llm #benchmarking #rubric #eval #variance #skill-evaluator

Ready to run

npx skills add https://github.com/theodrosyimer/.dotfiles --skill skill-evaluator-theodrosyimer

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: skill-evaluator
Source: https://github.com/theodrosyimer/.dotfiles/tree/main/claude/skills/meta__skill-evaluator
Command: npx skills add https://github.com/theodrosyimer/.dotfiles --skill skill-evaluator-theodrosyimer

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides a framework to validate the quality and reliability of other skills by running structured evaluation suites that combine deterministic structural checks (Layer 1) with LLM-based quality scoring (Layer 2), and by analyzing variance across multiple runs to surface stability issues and regression risks.

Core Features & Use Cases

Determines whether a skill consistently produces the correct outputs by applying cross-run variance analysis.
Supports building eval suites that pair deterministic assertions with rubric-based judgments to quantify quality, completeness, and adherence to conventions.
Provides tooling to aggregate results, compare with baselines, and generate human-readable reports to guide self-improvement of skill descriptions and behavior.

Quick Start

Quickly benchmark a skill by running an eval suite and reviewing the resulting benchmark to identify stability bottlenecks and improvement opportunities.

Frequently Asked Questions about skill-evaluator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I benchmark LLM skill quality and detect regressions across multiple runs?▼

Cross-run variance analysis surfaces stability issues by executing structured eval suites repeatedly, combining deterministic structural checks with LLM-judged quality rubrics to measure output consistency and identify regression risks across runs.

What is LLM rubric evaluation and how does it work for skill benchmarking?▼

LLM rubric evaluation pairs deterministic assertions with rubric-based judgments to quantify quality, completeness, and adherence to conventions, aggregating results to compare against baselines and generate human-readable reports for skill self-improvement.

How do I set up and run an eval suite to measure variance in skill outputs?▼

Running an eval suite to measure variance requires an isolated workspace with bundled scripts, loading eval prompts and rubrics, executing the suite across multiple runs, and reviewing the benchmark report to identify stability bottlenecks and improvement opportunities.

Does skill-evaluator require an isolated workspace or specific dependencies to run?▼

Skill-evaluator expects to be run in an isolated workspace with bundled scripts and carries no external dependencies, loading its own eval prompts, rubrics, and outputs to execute evaluation suites without additional setup.

What is the best way to combine deterministic checks with LLM-judged scoring for benchmarking?▼

Building eval suites that pair Layer 1 deterministic structural checks with Layer 2 LLM-based quality rubric scoring aggregates both mean performance and variance, quantifying quality and detecting stability issues across runs for benchmarking.

Why does my skill evaluation show inconsistent results across multiple runs?▼

Cross-run variance analysis surfaces inconsistent skill evaluation results by measuring stability against baselines, combining deterministic structural checks with LLM-judged rubric scoring to identify regression risks and stability bottlenecks across multiple runs.