loop-evals

What problem does it solve?

This Skill helps you design an evaluation harness that proves whether a long-running agent loop is actually done, instead of merely confident or self-reported as successful.

Core Features & Use Cases

• Layered Evaluation Design: Defines a seven-layer suite that orders deterministic checks before rubric-based judgment.
• False-Completion Detection: Makes false-completion-rate a first-class metric so loops cannot claim success without evidence.
• Judge Calibration: Sets up human-labeled calibration so model judges stay trustworthy over time.
• Regression and Anti-Cheat Coverage: Builds repo-native regression cases, trace-based checks, and security probes to catch shortcut behavior and verifier gaming.
• Use Case: Use this Skill when an agent keeps saying it is finished but the outputs are wrong, unverified, or underspecified.

Quick Start

Ask the skill to design the evaluation harness for your loop, identify the required verification layers, and draft the repo-native metrics and regression structure.