eval-design-forensics

To detect data leakage in a research paper, you can audit its evaluation design to identify potential overlap or leakage between training and testing data, ensuring reported results are reliable.

Evaluation validity ensures a paper's reported metrics accurately reflect its research methods. You can check it by auditing the evaluation design, verifying judge validity, and inspecting for selective reporting.

You can verify if an evaluation judge is unbiased by auditing the paper's evaluation design to check whether the judge is validated and free from systematic bias affecting the reported results.

To inspect a research paper for selective reporting, audit its evaluation design and reporting validity to identify if evaluation metrics are selectively chosen or incomplete, affecting result reliability.

Yes, you can audit the evaluation design and reporting validity of papers without writing custom scripts by using an automated skill to analyze the paper's claims, methodology, and results directly.

Automated evaluation audits rely on analyzing the paper's claims, methodology, and results, meaning they cannot detect issues omitted from the text or evaluate experimental execution beyond the reported design.