ava-serious-research

Split train, validation, and test sets before any preprocessing and seal the test set. Declare the conclusion type (performance estimation, model selection, or algorithm comparison), pick baselines that isolate one axis at a time, and pre-register the hypothesis, metric, and procedure.

Run the 8-category leakage audit: check for missing independent test sets, preprocessing before splitting, feature or target leakage, tuning on the test set, group overlap, augmentation before splitting, cross-validation contamination, and implementation leakage like shared global state.

Use McNemar's test (or exact binomial when discordant pairs are under 50) to compare two fixed model instances on one test set. For algorithm-level claims across data splits, use the combined 5x2cv F-test; never use the resampled paired t-test.

Write it in three-part form (studying X to find out Y so the audience understands Z) and pass the So-what test. Derive falsifiable hypotheses with a minimal decisive experiment and identify the live tension in the literature the answer would resolve.

Use nested CV on small or medium datasets whenever hyperparameter tuning occurs, so the inner loop tunes and the outer loop estimates generalization. Reporting the best single-level validation score as generalization performance introduces optimistic bias.

No. It explicitly excludes paper writing and publication, which are treated as a separate craft. It focuses on doing the research well and making the process legible and verifiable for a human collaborator.