reproducibility-audit

Audits research paper reproducibility by verifying dataset version locks, bibliography sync, and golden tests.

6|1|Updated May 11, 2026
One-click install
npx skills add https://github.com/yakeworld/Synthos --skill reproducibility-audit-yakeworld
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: reproducibility-audit
Source: https://github.com/yakeworld/Synthos/tree/main/skills/private/reproducibility-audit
Command: npx skills add https://github.com/yakeworld/Synthos --skill reproducibility-audit-yakeworld

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Reproducing scientific papers often fails silently because dataset versions are unpinned, inline thebibliography entries drift out of sync with .bib files, and verification steps are skipped without records. This Skill audits a paper's working directory against a golden test set to determine whether its results are actually reproducible. ## Core Features & Use Cases - Dataset Version Verification: Checks that dataset versions are locked (e.g., via a lock file) before accepting any reproduction claim, blocking unpinned data as "reproduction without evidence". - Dual Bibliography Sync Check: Detects drift between inline thebibliography environments and .bib files, requiring both sources to be updated together. - Golden Set Testing: Runs weighted pass/fail checks (critical/high/medium) against golden cases, requiring a weighted score of at least 0.80 with all critical checks passing. - Failure Traceability: Requires every verification step to be executable, recorded, and reproducible, with failures logged alongside concrete recovery actions. - Use Case: Given a paper directory with a pinned dataset, 32 bibliography entries in both sources, and a runnable pipeline, produce a reproducibility report confirming version consistency, bibliography sync, and golden test passage. ## Quick Start Audit the paper directory 'papers/repro-target' for reproducibility and generate a report covering dataset version locking, bibliography synchronization, and golden test results.

Frequently Asked Questions about reproducibility-audit

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I audit a research paper for reproducibility?

Point the audit at the paper's working directory containing its dataset, bibliography files, and code pipeline. The audit checks dataset version locking, bibliography synchronization, and golden test results, then produces a reproducibility report with a weighted score.

How to check if thebibliography and .bib files are in sync?

Compare entry counts and keys between the inline thebibliography environment and the .bib file. The audit flags drift such as entries missing from the .bib file or orphaned entries, and requires both sources to be updated together.

What happens when a dataset version is not locked?

An unpinned dataset version is treated as a critical failure: the audit rules the reproduction lacks evidence and blocks acceptance until the version is locked and the golden test is rerun. Silent skipping of this check is not allowed.

What is the passing criteria for the golden test set?

Passing requires a weighted score of at least 0.80 with all critical-weight checks passing. Critical checks carry weight 1.0, high checks 0.7, and medium checks 0.4.

How are verification failures handled during an audit?

Every verification failure must be recorded with its specific cause and a corresponding recovery action. The audit forbids silently skipping failed steps and requires each step to be executable, recorded, and reproducible.