Verified:GitHub

sdlc-eval

Runs golden-set evaluations for LLM prompt versions and compares quality metrics against baselines.

Updated Jun 1, 2026

Search Tags:#llm evaluation #quality gates #golden dataset #baseline comparison #eval harness #prompt regression #judge rubric

Ready to run

npx skills add https://github.com/juartaurus98/codebase --skill sdlc-eval-juartaurus98

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: sdlc-eval
Source: https://github.com/juartaurus98/codebase/tree/main/.claude/skills/sdlc-eval
Command: npx skills add https://github.com/juartaurus98/codebase --skill sdlc-eval-juartaurus98

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Prompt changes can silently degrade output quality, and without a recorded baseline there is no way to detect regression. This Skill runs a deliberate, cost-aware evaluation harness for a specific prompt version, records quality metrics and cost, and compares them against the previous version so regressions are caught before they ship. ## Core Features & Use Cases - Tiered oracle evaluation: Runs structural assertions (schema, token caps, banned strings), reference-based similarity checks, and rubric-scored LLM judge evaluations with n=3 median sampling and variance flagging. - Baseline comparison and verdicts: Aggregates pass rates, judge scores, latency, and cost, then issues a pass, regression, or below_threshold verdict against the recorded baseline, with regression-origin cases always treated as blockers. - Persistent run records: Writes results to .agent/state/evals/, updates the eval spec's baselines table, the prompt spec's version history, and the registry metadata. - Use Case: After publishing prompt version [email protected], invoke /eval summarize to run its golden-set cases, confirm the pass rate and judge scores did not regress versus 1.1.0, and record the run as the new baseline. ## Quick Start Ask the AI to run /eval for a prompt name such as "summarize" to execute its golden-set evaluation suite and report the verdict versus the previous baseline.

Frequently Asked Questions about sdlc-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I evaluate an LLM prompt for regression before release?▼

Run /eval with the prompt name and optional version, for example /eval [email protected]. It executes the golden-set cases, aggregates pass rate and judge scores, and compares them against the previous version's recorded baseline to issue a pass, regression, or below_threshold verdict.

What is a golden-set evaluation for LLM prompts?▼

A golden-set evaluation runs a curated cases.jsonl dataset against a prompt version using three oracle tiers: structural assertions, reference-based similarity, and rubric-scored judge evaluations. Judge and similarity cases run n=3 with the median recorded to control variance.

What happens when no baseline exists for a prompt evaluation?▼

The first run becomes the baseline and this is stated explicitly in the report. A first run cannot detect regression, so it is never reported as "no regression" since that would be misleading.

Can I fix a failing prompt evaluation by editing the prompt?▼

No. The eval command never edits prompts, cases, or thresholds to force a pass. A prompt change is a new version decided in /techdoc and written by /gen-code, and eval cases are authored only by /gen-testcase.

Why does the eval command require explicit invocation instead of running automatically?▼

Running the suite calls paid LLM providers, so the cost must always be a deliberate choice. The gate checkpoint states the case count and estimated spend before execution, and the command is never triggered as part of another command.