origin-output-eval

Runs a bounded evaluation loop that re-scores artifacts against machine-checked thresholds until they pass.

Updated Dec 9, 2025
One-click install
npx skills add https://github.com/Aki2022/skills --skill origin-output-eval-aki2022
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: origin-output-eval
Source: https://github.com/Aki2022/skills/tree/main/origin-output-eval
Command: npx skills add https://github.com/Aki2022/skills --skill origin-output-eval-aki2022

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve? Evaluating deliverables (code, designs, documents, slides, sites) by hand is inconsistent: the author grades their own work, reviewers get stale context, and pass/fail counts are eyeballed. This Skill provides a shared evaluation loop that picks the right method (tests, review skills, or independent judges), mechanically judges each round, and iterates fixes until the artifact reaches the pass line. ## Core Features & Use Cases - Three evaluation methods with selection rules: deterministic tests (A) first, existing review skills (B) second, and independent judge/reader subagents (C) only for subjective quality, always in A→B→C order. - Machine-enforced independence: a required provenance.json declares each evaluator's freshness, allowed inputs, and cache state; mismatches, missing evaluators, or side information stop judgment with exit 2. - Bounded unattended loop: run_round.sh chains judge_round.py (schema validation and threshold checks) and eval_state.py (cross-round state, convergence via blocking_count, max-rounds cap), producing a final report with a human gate outside the loop. - Use Case: After generating a landing page, declare the deterministic gates passed, spin up fresh judge and first-time-reader subagents with a rubric, and run rounds until scores clear the thresholds—then hand the final report to a human for the last decision. ## Quick Start Ask the AI to review your artifact with this skill and keep fixing and re-evaluating it until it passes the quality gate.

Frequently Asked Questions about origin-output-eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run an automated review loop until my artifact passes?

Place evaluator outputs in round_K directories, write a provenance.json declaring each evaluator, then run run_round.sh with your thresholds.json. The script judges the round, records state, checks convergence, and returns exit codes telling you to fix and re-run or stop.

How do I choose between tests, review skills, and AI judges for evaluation?

Use tests (method A) for anything deterministically checkable, existing review skills (method B) for what remains, and independent judge subagents (method C) only for subjective quality. When several apply, run them in A→B→C order and record the reasons.

Why does judge_round.py exit with code 2 instead of pass or fail?

Exit 2 means the input is invalid, not that the artifact failed. Common causes are a missing provenance.json, evaluator declarations not matching actual output files, fresh set to false, readers receiving side information, round mismatches, or out-of-vocabulary severity values.

Can the same AI agent re-evaluate its own work in the loop?

No. Every round requires fresh evaluator subagents, and provenance.json enforces this mechanically: fresh must be true, readers may only receive the artifact, and cache_cleared must be true. Violations stop judgment with exit 2 before any scoring happens.

What is the difference between findings_count and blocking_count?

findings_count totals all evaluation units including comments and minor reader confusion, so it breaks when the artifact is restructured. blocking_count only counts items that must be fixed to pass, and it is the value used for convergence decisions between rounds.

Does passing this evaluation loop mean the deliverable is approved?

No. A pass from the loop is not a substitute for human confirmation. The skill always ends with a final report, and exactly one human gate sits outside the loop to make the final decision.