Design, score, and run evals for LLM features with golden datasets and rubrics.
npx skills add https://github.com/f4rkh4d/forge-skill --skill forge-evals
Please help me install this Agent Skill. Skill: forge-evals Source: https://github.com/f4rkh4d/forge-skill/tree/main/skills/llm/forge-evals Command: npx skills add https://github.com/f4rkh4d/forge-skill --skill forge-evals
This Skill helps teams stop relying on “looks good” prompts and instead measure whether an LLM feature improves or regresses after changes.
Ask an AI agent to help you design a rubric and create a 50-example held-out eval dataset for your LLM feature, then wire an eval runner into CI with ship/block thresholds and per-criterion reporting.
High-intent search queries and answers about installing and using this skill.
LLM regression testing in CI requires wiring an eval runner into your pipeline that re-runs held-out datasets on prompt or model changes and fails builds when quality drops below defined thresholds. You need a golden dataset and multi-criterion scoring to track per-example outcomes.
LLM-as-judge calibration works by using structured judge outputs to score LLM behavior against a rubric, periodically validating against human reviews to prevent judge drift. This ensures automated scoring remains accurate and aligned with expected behavior criteria over time.
The best way to design a golden eval dataset is to build and version 50 or more held-out examples with clear intent and expected behavior criteria. This dataset serves as the baseline for multi-criterion rubric scoring and regression detection during CI re-runs.
You can track cost and latency budgets by including p50 and p95 latency along with per-call cost as evaluation dimensions alongside quality scores. This allows you to monitor production behavior constraints and ensure features stay within acceptable performance budgets.
Validate offline LLM evals by using them to predict quality changes, then running online A/B tests to measure real user outcomes. This offline-to-online approach confirms that rubric scoring and held-out dataset results translate to actual production improvements.
vecearAutomate LLM evaluation with golden datasets, rubrics, thresholds, and regression reports.
Veloxia-agencyVersion, evaluate, and promote production prompt assets with deployment gates.
DevMHOneMonitor provider health, circuit-breaker states, and latency metrics in real time.
yanochka11Calibrate LLM judges against human labels using TPR/TNR analysis.
