experiment-audit

Name: experiment-audit
Availability: InStock
Author: duypham2801

Audit experimental artifacts for untrustworthy reporting and generate PASS/WARN/FAIL findings.

Updated May 25, 2026

Search Tags:#fraud detection #metric validation #experiment integrity #audit results #evaluation scripts #research transparency #ML benchmarking

Ready to run

npx skills add https://github.com/duypham2801/ThS_LLM --skill experiment-audit-duypham2801

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: experiment-audit
Source: https://github.com/duypham2801/ThS_LLM/tree/main/.claude/skills/experiment-audit
Command: npx skills add https://github.com/duypham2801/ThS_LLM --skill experiment-audit-duypham2801

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

It reduces the risk of reporting untrustworthy experimental outcomes by checking whether ground truth, metrics, and claimed numbers genuinely come from the intended evaluation pipeline.

Core Features & Use Cases

Integrity-first audit across artifacts: Scans evaluation code, result files, configs, and paper claims for common failure modes such as fake ground truth and phantom metrics.
Cross-model verification workflow: Uses an independent reviewer model to read files and judge integrity while keeping the executor role restricted to artifact collection.
Machine-readable audit outputs: Writes both EXPERIMENT_AUDIT.md and EXPERIMENT_AUDIT.json so downstream steps can tag verdicts and claims consistently.

Quick Start

Run the audit on your experiment outputs by passing the experiment directory path you want to verify, and then review EXPERIMENT_AUDIT.md for PASS/WARN/FAIL findings.

Frequently Asked Questions about experiment-audit

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I audit ML benchmark results for fake ground truth or phantom metrics?▼

An experiment integrity audit examines evaluation scripts, result files, and configurations to detect fake ground truth, self-normalized score fraud, and phantom metrics by cross-verifying artifacts with an independent reviewer model, generating PASS/WARN/FAIL findings.

How do I validate evaluation scripts after an ML training run to ensure research transparency?▼

Validating evaluation scripts after an ML training run requires restricting the executor to artifact path collection while an independent reviewer model fully inspects files to verify evaluation scope, detect dead code paths, and confirm research transparency.

What is self-normalized score fraud in machine learning evaluation?▼

Self-normalized score fraud in machine learning evaluation occurs when reported metrics are manipulated through improper normalization; auditing experimental artifacts detects this by cross-verifying evaluation scripts and result files against intended pipeline outputs.

Can I check for missing result-file claims and dead evaluation code paths before publication?▼

Yes, you can check for missing result-file claims and dead evaluation code paths before publication by running an experimental audit that scans artifacts and outputs evidence-linked PASS/WARN/FAIL verdicts to machine-readable JSON and Markdown formats.

Does experiment integrity auditing require reading the contents of all training output files?▼

Experiment integrity auditing uses a split workflow where the executor side only collects artifact paths without reading contents, while the reviewer side performs full file inspection to judge research credibility across ML training and evaluation workflows.

What format does an experimental audit output for downstream integrity tagging?▼

An experimental audit outputs findings to EXPERIMENT_AUDIT.md and EXPERIMENT_AUDIT.json, providing both human-readable reports and machine-readable files for downstream integrity tagging of PASS/WARN/FAIL verdicts and paper-narrative claims.