skills.rest
paulingham
paulinghamCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

internal-eval-validate

Validate internal-eval flows with a stubbed pipeline and managed phases.

1|Updated Mar 15, 2026
Search Tags:#automation#testing#validation#end-to-end#ci#harness#qa-engineering
One-click install
npx skills add https://github.com/paulingham/.claude --skill internal-eval-validate
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: internal-eval-validate
Source: https://github.com/paulingham/.claude/tree/main/skills/internal-eval/validate
Command: npx skills add https://github.com/paulingham/.claude --skill internal-eval-validate

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Delivery validation for /harness:internal-eval: proves the baseline → inject → diff → restore → confirm-clean flow end-to-end with a stubbed inner pipeline. Runs locally and in CI; does NOT execute real /harness:pipeline cases.

Core Features & Use Cases

  • Phase A — Baseline: seed 3 deterministic cases, run-suite with the stub manifest empty (all cases pass), capture-baseline, assert pass_rate == 1.00.
  • Phase B — Inject: flip ≥2 cases in the stub manifest to fail, re-run at a new run-id, rewrite failed_build → failed_diff to match oracle-rejection semantics, re-aggregate.
  • Phase C — Diff: run diff-vs-baseline.sh, assert verdict == "EVAL_FAILED", regression_count ≥ 2, and that both flipped case-ids appear in the regressions quadrant.
  • Phase D — Restore: revert the stub manifest, re-run at a third run-id. Byte-equivalence check: shasum agents/code-reviewer.md before and after MUST match (the live harness is never touched).

Quick Start

bash skills/internal-eval/validate/run-validation-sequence.sh [tmp-dir]

Frequently Asked Questions about internal-eval-validate

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I validate end-to-end evaluation flows in CI without running real pipeline cases?▼

End-to-end validation uses a stubbed inner pipeline to verify baseline, injection, differential evaluation, restoration, and clean verdict across multiple runs without executing real pipeline cases.

What is the baseline inject diff restore validation sequence for QA engineering?▼

The validation sequence seeds deterministic baseline cases, injects failures into a stub manifest, asserts differential regressions, and restores the manifest while verifying byte-equivalence via shasum.

How do I automate differential evaluation testing to detect regressions locally?▼

Automate differential evaluation by running a validation sequence script that captures a baseline, flips cases to fail, re-aggregates, and asserts the verdict and regression count against the baseline.

Does the internal-eval validation flow require a specific manifest file to run?▼

Yes, validation requires a manifest file alongside a run-validation-sequence driver and supporting library scripts to seed cases, orchestrate phases, and assert outcomes locally or in CI.

How does the restoration phase ensure the live harness is never modified during testing?▼

The restoration phase reverts the stub manifest and performs a byte-equivalence check using shasum on agents before and after runs, ensuring the live harness remains untouched.

What is the best way to assert regression counts during differential evaluation in a CI harness?▼

Run a diff-vs-baseline script to assert the verdict equals EVAL_FAILED, verify the regression count is two or more, and confirm flipped case-ids appear in the regressions quadrant.

Related Skills

Explore More Software Engineering Skills→
sovr610sovr610

Seven-Phase Training Orchestrator + Reproducible Manifests + Automatic Ablations

Coordinate seven-phase training pipelines with reproducible manifests and automatic ablations.

Community
Advanced
kali20gakkikali20gakki
7

mindstudio_profiler_data_check

Validate MindStudio profiler data integrity and collection status before analysis.

Community
Advanced
yongwoonyongwoon
8

ywc-refactor-clean

Identify and remove dead code with tiered safety verification.

Community
Advanced
AZANIRAZANIR
2

qa-orchestrator

Coordinate multi-skill QA workflows through pipeline modes and handoff chains.

Community
Advanced