forge-evals

What problem does it solve?

This Skill helps teams stop relying on “looks good” prompts and instead measure whether an LLM feature improves or regresses after changes.

Core Features & Use Cases

• Golden eval datasets: build and version 50+ held-out examples with clear intent and criteria.
• Rubric-based scoring: score multiple criteria (not a single aggregate) for clearer signal and safer thresholds.
• LLM-as-judge with calibration: use structured judge outputs and periodically calibrate against humans to avoid judge drift.
• Regression + CI integration: re-run evals on prompt/model/retrieval changes and fail builds when quality drops.
• Offline-to-online validation: use offline evals for prediction and validate with online A/B tests for real user outcomes.
• Cost and latency budgets: include p50/p95 latency and per-call cost as dimensions alongside quality.

Quick Start

Ask an AI agent to help you design a rubric and create a 50-example held-out eval dataset for your LLM feature, then wire an eval runner into CI with ship/block thresholds and per-criterion reporting.