paper-claim-audit

Verifies every numeric claim in a research paper against raw result files using a zero-context cross-model reviewer.

Updated Jul 9, 2026
One-click install
npx skills add https://github.com/Lingjie-wang/autoRL --skill paper-claim-audit-lingjie-wang
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: paper-claim-audit
Source: https://github.com/Lingjie-wang/autoRL/tree/main/Auto-claude-code-research-in-sleep/skills/paper-claim-audit
Command: npx skills add https://github.com/Lingjie-wang/autoRL --skill paper-claim-audit-lingjie-wang

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? When the same agent runs experiments and writes the paper, confirmation bias creeps in: numbers get rounded up, best seeds get reported as averages, and deltas get miscalculated. This Skill audits the paper with a fresh reviewer that has zero prior context, so every number, comparison, and scope claim is checked against raw evidence without expectations. ## Core Features & Use Cases - Claim-by-claim evidence tracing: Extracts every quantitative claim from .tex files and matches it to exact values in raw JSON/CSV/YAML result files, classifying each as exact_match, rounding_ok, mismatch, or unsupported. - Zero-context cross-model review: Sends only paper sources and raw result files to a fresh GPT-5.5 thread (never codex-reply), excluding all executor summaries, logs, and prior audits to prevent confirmation bias. - Structured verdict artifacts: Always writes PAPER_CLAIM_AUDIT.json (with audited input SHA256 hashes, verdict, and reason codes) plus a human-readable PAPER_CLAIM_AUDIT.md, even for NOT_APPLICABLE or BLOCKED runs. - Use Case: Before submitting a paper, run the audit to catch that the abstract claims a 15% improvement while the raw results show only 12.8%, or that a reported average actually reflects the best of five seeds. ## Quick Start Run the paper-claim-audit skill on my paper directory to verify that every number in the .tex files matches the raw experiment result files.

Frequently Asked Questions about paper-claim-audit

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I verify that numbers in my research paper match the experiment results?

Run this audit on your paper directory. It extracts every quantitative claim from the .tex files, traces each one to raw JSON/CSV/YAML result files, and reports per-claim match status plus an overall PASS, WARN, or FAIL verdict.

What is a zero-context reviewer for paper auditing?

A zero-context reviewer is a fresh model thread that receives only the paper source files and raw result files, with no experiment logs, summaries, or conversation history. Because it has no expectations about the results, it catches inflated numbers and cherry-picked seeds that the original author overlooks.

How is paper-claim-audit different from experiment-audit or result-to-claim?

experiment-audit checks whether the experiment code is honest, and result-to-claim checks whether data scientifically supports a claim. paper-claim-audit checks whether the paper reports the data truthfully and precisely, focusing on number fidelity, rounding, aggregation, and scope.

What failure modes does a paper claim audit detect?

It detects number inflation beyond standard rounding, best-seed cherry-picking reported as averages, config mismatches between compared methods, aggregation count mismatches, incorrect delta arithmetic, caption-table mismatches, and scope overclaims such as 'consistently outperforms' on limited evaluations.

What happens if the paper has no numeric claims or no raw result files?

The skill still writes PAPER_CLAIM_AUDIT.json in both cases. A paper with no numeric claims gets a NOT_APPLICABLE verdict, while numeric claims without raw evidence produce a BLOCKED verdict, so downstream verifiers always find the artifact.

Does the audit block paper submission on a FAIL verdict?

No, the skill is advisory and never blocks. A FAIL prints an alert and prevents the draft from being marked submission-ready, but the parent workflow decides whether the verdict blocks finalization.