eval-design-forensics

What problem does it solve?

This Skill audits the evaluation design and reporting validity of research papers, ensuring that the reported results are reliable and accurately reflect the research methods.

Core Features & Use Cases

• Evaluation Design Audit: Checks if the evaluation design in a paper measures what it claims and if the reporting is complete.
• Leakage Detection: Identifies leakage between training and testing data.
• Judge Validity Check: Verifies if the evaluation judge is unbiased and validated.
• Selective Reporting Analysis: Inspects for selective reporting in the paper's evaluation.
• Use Case: If you are reviewing a paper and want to ensure that the evaluation metrics are reliable, use this Skill to analyze the paper's evaluation design and reporting.

Quick Start

Use the eval-design-forensics skill to audit the evaluation design and reporting of the paper in the 'paper-dir'.