skills.rest
pedrohcgs
pedrohcgsCOMMUNITY
Explore all skills by this author →
Website𝕏@pedrohcgsGitHub

vaccinate

Qualifies automated checks by seeding known defects and measuring recall and false-positive rates.

1.5k|3.0k|Updated Feb 6, 2026
Search Tags:#quality gate#reproducibility#Recall#false positives#seeded defects#check qualification#reviewer testing
One-click install
npx skills add https://github.com/pedrohcgs/claude-code-my-workflow --skill vaccinate
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: vaccinate
Source: https://github.com/pedrohcgs/claude-code-my-workflow/tree/main/.claude/skills/vaccinate
Command: npx skills add https://github.com/pedrohcgs/claude-code-my-workflow --skill vaccinate

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Automated checks, review agents, and referee simulations are often trusted without evidence that they can actually detect the failures they claim to catch. This Skill grades the grader: it seeds known defects into a copy of a real artifact, runs the checker blind, and records recall and false-positive rates in a qualification ledger so a green light means something.

Core Features & Use Cases

  • Seeded-defect qualification: Plants defects from a categorized defect library (analysis code, manuscripts, replication packages, slides, infrastructure) into copies of artifacts, plus a mandatory clean control.
  • Blind scoring protocol: Runs the checker in a fresh context per variant and scores recall, false-positive rate, localization, and a baseline delta against a simpler alternative like grep.
  • Qualification ledger: Appends PASS, FAIL, or BLOCKED verdicts to quality_reports/qualification/LEDGER.md so every trusted check has a recorded measurement.
  • Use Case: Before relying on a referee simulation for a paper submission, run the skill to prove the reviewer detects planted defects such as a text coefficient that no longer matches its table.

Quick Start

Ask the AI to vaccinate a specific checker against a specific artifact, for example: qualify whether scripts/check-model-versions.sh can detect a superseded model claim seeded into README.md.

Frequently Asked Questions about vaccinate

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I test whether my automated code reviewer actually works?▼

Seed known defects into a copy of a real artifact, run the reviewer blind on each variant plus a clean control, and measure recall and false-positive rate. This skill automates that protocol and records the result in a qualification ledger.

What is seeded defect testing for quality gates?▼

Seeded defect testing plants controlled failures into an artifact to verify a checker can detect them. It measures recall on seeded variants and false positives on an unmodified control, proving the gate detects what it claims to catch.

Why do I need a clean control when qualifying a checker?▼

Without a clean control you measure only recall and call it accuracy. The control reveals false positives: findings on an unmodified artifact that are factually wrong, which indicate an over-reporting checker.

When should a check be requalified after changes?▼

Requalify after any material change to the checker, the artifact's interface or schema, the model, prompt, or rubric. A check qualified against an old target may silently stop testing anything, so its prior PASS no longer applies.

What are the limitations of AI reviewer agreement as evidence?▼

Agreement between models is not independent evidence because they share failure modes and converge on the same wrong answers. Qualified checks need defensible reference answers, and scientific judgments cannot be automated at all.

Related Skills

Explore More Education & Research Skills→
hellofrommorganhellofrommorgan

seed

Seed new source files into the processing queue with duplicate detection and archival.

Community
Intermediate
OilProductsOilProducts

auditor-gate

Read handoff and verdict JSON to emit audit.json with a gate status.

Community
Intermediate
pure-golangpure-golang
2

skill-creator

Draft SKILL.md files and run trigger/evaluation loops to improve Claude Code skills.

Official
Advanced
ilove323ilove323

audit-support

Guide SOX 404 internal control testing with sampling and documentation standards.

Community
Intermediate