internal-eval-score

Score per-case results against oracle baselines and generate regression diffs as JSON and Markdown.

1|Updated Mar 15, 2026

Search Tags:#oracle #regression #diff #baseline #score #harness

Ready to run

npx skills add https://github.com/paulingham/.claude --skill internal-eval-score

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: internal-eval-score
Source: https://github.com/paulingham/.claude/tree/main/skills/internal-eval/score
Command: npx skills add https://github.com/paulingham/.claude --skill internal-eval-score

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Scores per-case results against oracle baselines, captures baselines, and diffs the run against the baseline across four quadrants.

Core Features & Use Cases

Per-case scoring against oracles and safety proxies.
Baseline capture, regression analysis, and four-quadrant diff reporting.
A/B-style comparison utilities to evaluate changes across runs and gates.

Quick Start

Run the internal-eval score workflow to compute per-case scores, baseline snapshots, and regression diffs across quadrants.

Frequently Asked Questions about internal-eval-score

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I score model evaluation results against oracle baselines?▼

The tool computes per-case scores against oracle baselines and compares the current run against the baseline to identify regressions in your evaluation pipeline.

What is a four-quadrant diff in regression analysis?▼

Four-quadrant diff reporting categorizes baseline changes by comparing current and previous run outcomes across four distinct quadrants, highlighting newly passed, newly failed, and persistent test cases.

How do I capture a baseline snapshot for end-to-end evaluation pipelines?▼

Run the internal-eval score workflow to capture per-case oracle results and safety scores as a baseline snapshot, storing them as JSON artifacts for future cross-run regression analysis.

Can I run A/B safety retention assessments across multiple model runs?▼

Yes, built-in A/B-style comparison utilities evaluate changes and verify safety gates across different evaluation runs to assess safety retention in model workflows.

What output formats are generated for regression artifacts?▼

The workflow generates regression artifacts in both JSON and Markdown formats, providing programmatic access to safety scores, baseline diffs, and structured reporting for pipelines.

Does this regression scoring tool require external dependencies?▼

No, this regression scoring tool operates without external dependencies, providing a self-contained programmatic interface to compute safety scores and baseline diffs within model evaluation workflows.