paper-experiment-audit

Audits LaTeX paper experiments for dataset version consistency, ablation reproducibility, and citation integrity.

6|1|Updated May 11, 2026
One-click install
npx skills add https://github.com/yakeworld/Synthos --skill paper-experiment-audit-yakeworld
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: paper-experiment-audit
Source: https://github.com/yakeworld/Synthos/tree/main/skills/private/paper-experiment-audit
Command: npx skills add https://github.com/yakeworld/Synthos --skill paper-experiment-audit-yakeworld

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Research papers often contain hidden experimental flaws: metrics compared across different dataset versions (e.g., WDBC 699 vs 569 samples), ablation results that cannot be reproduced, bibliography entries out of sync between thebibliography and references.bib, and figures cited redundantly. This Skill systematically audits a paper's experiment directory to detect and flag these integrity issues before publication or peer review. ## Core Features & Use Cases - Dataset Version Locking: Records and fixes dataset versions before auditing, blocking cross-version metric comparisons that exceed the 0.5% difference threshold with a MISMATCH flag. - Ablation Reproducibility Checks: Re-runs each ablation configuration's independent script and compares output JSON/CSV against claimed values, marking differences above 0.5% as inconsistent. - Citation & Figure Validation: Verifies one-to-one correspondence between thebibliography and references.bib entries, and detects figures (like fig6) cited more than twice in the text for deduplication. - Multi-Source Cross-Validation: Requires at least two independently generated JSON/CSV outputs per metric; single-source metrics are marked UNVERIFIED. - Use Case: Given a LaTeX paper directory with WDBC experiments, ablation scripts, and output JSON files, the Skill produces a structured audit log where each of six verification checks is marked pass/fail with traceable evidence, and any MISMATCH blocks downstream conclusions. ## Quick Start Audit the experiments in my LaTeX paper directory, checking dataset version consistency, ablation reproducibility, bibliography synchronization, and figure citation redundancy.

Frequently Asked Questions about paper-experiment-audit

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I audit ablation experiments in a research paper for reproducibility?

Run each ablation configuration's independent script and compare its output JSON or CSV against the claimed metric. A reproduction difference of 0.5% or less is judged consistent; larger differences are flagged as MISMATCH with the source of the discrepancy recorded.

How to check thebibliography and references.bib consistency in LaTeX?

Compare entries one-to-one between the thebibliography environment and references.bib. Missing or extra entries on either side are marked MISMATCH and must be fixed so both sources stay synchronized before submission.

Why is comparing metrics across dataset versions a problem?

Different dataset versions, such as WDBC with 699 versus 569 samples, have different sample compositions, so metric differences may reflect data changes rather than model improvements. The audit locks one version and blocks cross-version comparisons exceeding the 0.5% threshold.

What happens when a metric has only one source during an audit?

A metric backed by a single JSON or CSV output is marked UNVERIFIED. The audit requires at least two independently generated outputs from different scripts to cross-validate each reported metric before accepting it.

When should figure citations be merged in a paper?

When the same figure, such as fig6, is cited more than twice in the body text, the redundant citations are flagged and merged into a single reference. This keeps the manuscript clean without losing the pointer to the figure.