skills.rest
sumithr
sumithrCOMMUNITY
Explore all skills by this author →
GitHub

regen-eval-baseline

Run promptfoo eval baselines for a named skill and store JSON snapshots.

5|1|Updated May 12, 2026
Search Tags:#quality assurance#TDD#test planning#promptfoo#snapshot testing#delta analysis#eval baseline
One-click install
npx skills add https://github.com/sumithr/sumo-qa --skill regen-eval-baseline
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: regen-eval-baseline
Source: https://github.com/sumithr/sumo-qa/tree/main/.claude/skills/regen-eval-baseline
Command: npx skills add https://github.com/sumithr/sumo-qa --skill regen-eval-baseline

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

Capturing and comparing how a single skill’s behavior changes over time—especially after editing its SKILL.md—without losing historical evidence.

Core Features & Use Cases

  • Per-skill eval baselines: Runs promptfoo for one named skill and saves the judge output JSON to a local, gitignored snapshot directory.
  • Labelled snapshots with delta: Supports a kebab-case label (default baseline) and automatically reports pass/fail deltas versus the most recent prior snapshot for the same skill.
  • Guardrails for reproducibility: Enforces valid kebab-case inputs, checks that the corresponding eval YAML exists, requires OPENAI_API_KEY to be set, and blocks clobbering existing snapshots unless forced.

Quick Start

Run the baseline script with the skill name you want to measure and an optional label by executing: python3 .claude/skills/regen-eval-baseline/scripts/run_baseline.py --skill <skill-name> --label <label>.

Frequently Asked Questions about regen-eval-baseline

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I snapshot promptfoo eval results for a single skill?▼

Run the baseline script targeting a specific skill to execute promptfoo and store the resulting JSON snapshot locally. This captures a fresh evaluation baseline for delta analysis.

How do I compare eval results before and after editing a SKILL.md?▼

Generate a new snapshot after editing SKILL.md to automatically receive pass and fail delta reporting against the most recent prior snapshot for that skill.

Do I need an OPENAI_API_KEY to run promptfoo eval baselines?▼

Yes, the OPENAI_API_KEY environment variable must be set. The script validates this requirement before executing promptfoo evaluations and writing JSON snapshots.

Can I overwrite an existing promptfoo eval baseline snapshot?▼

Existing snapshots are protected from being clobbered unless explicitly forced. The script enforces this guardrail to preserve historical evaluation evidence.

What file format does the promptfoo eval config need to be in?▼

The script requires a matching YAML configuration file located at tests/evals/promptfoo/skill-<name>.yaml. It validates that this file exists before running the evaluation.

Related Skills

Explore More Software Engineering Skills→
gerald-icagerald-ica

promptfoo-evals

Automate creation and maintenance of promptfoo evaluation suites, prompts, tests, assertions, and provider configurations.

Community
Advanced
KamiJeongKamiJeong

speckit-token-analyzer-compare

Compare token consumption between spec-kit analysis runs and compute metric deltas.

Community
Intermediate
WrongStackWrongStack
224

skill-creator

Create frontmatter-driven SKILL.md files for WrongStack skills.

Official
Intermediate
haabehaabe

prompt-optimizer

Compare CLAUDE.md instruction variants against baselines using evaluation reports.

Community
Advanced