experiment-audit

Audits experiment integrity using cross-model review to detect fake ground truth and phantom results.

Updated Jul 9, 2026
One-click install
npx skills add https://github.com/Lingjie-wang/autoRL --skill experiment-audit-lingjie-wang
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: experiment-audit
Source: https://github.com/Lingjie-wang/autoRL/tree/main/Auto-claude-code-research-in-sleep/skills/experiment-audit
Command: npx skills add https://github.com/Lingjie-wang/autoRL --skill experiment-audit-lingjie-wang

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? LLM agents can unintentionally produce fraudulent experimental results, such as fabricating ground truth from model outputs, self-normalizing scores to near 1.0, citing nonexistent result files, or overstating evaluation scope. This Skill adds an independent integrity check before any claims are written. ## Core Features & Use Cases - Cross-Model Independent Review: The executor only collects file paths; an external reviewer backend (Codex MCP or Manual Review MCP) reads the code and judges integrity, preventing self-acquittal. - Six-Point Fraud Checklist: Checks ground truth provenance, score normalization, result file existence, dead code detection, scope assessment, and evaluation type classification. - Structured Audit Reports: Writes EXPERIMENT_AUDIT.md and EXPERIMENT_AUDIT.json with PASS/WARN/FAIL verdicts that downstream skills like /result-to-claim and /paper-write can consume. - Use Case: After an experiment pipeline finishes and before writing paper claims, run the audit to verify that reported metrics actually exist in result files and that ground truth comes from the dataset rather than model outputs. ## Quick Start Ask the agent to audit the experiment results in your project directory for integrity before writing any claims, for example by saying "audit results in ./experiments/run-03".

Frequently Asked Questions about experiment-audit

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I audit experiment results for integrity before writing paper claims?

Run the audit after experiments complete and before writing claims. The executor collects paths to eval scripts, result files, trackers, and paper drafts, then an external reviewer reads them against a six-point checklist and returns a PASS, WARN, or FAIL verdict.

What fraud patterns does an experiment integrity audit detect?

It detects fake ground truth derived from model outputs, score normalization using the model's own statistics, phantom results referencing nonexistent files or mismatched numbers, dead metric code never called, and scope language exceeding actual evidence.

Why does the audit use a cross-model reviewer instead of self-review?

Reviewer independence prevents self-acquittal: the executor only collects file paths and never judges integrity. A different model family (Codex or Manual Review MCP) reads the code directly, so the agent that produced the results cannot grade its own work.

Can the experiment audit block my research pipeline on failure?

No, the audit is advisory and never blocks the pipeline. On WARN or FAIL it prints alerts and tags downstream claims with integrity labels like [INTEGRITY CONCERN], so /result-to-claim and /paper-write can downgrade or footnote affected claims.

What happens if the manual review MCP backend is unavailable?

If the manual-review MCP is unavailable, the skill stops and prints the install command rather than silently falling back to Codex. This preserves the configured reviewer routing policy and keeps review tracing consistent.