What problem does it solve?
Structured pre-flight checklist that verifies the entire experiment pipeline produces correct results before committing to a full sweep. Catches data bugs, model loading errors, measurement mistakes, ablation failures, and baseline mismatches — all cheaper to fix now than after 200 GPU-hours.
Core Features & Use Cases
- Data Integrity Checks: shape checks, distribution checks, label verification, reproducibility checks, and leakage prevention.
- Model Loading Verification: ensure non-trivial outputs, correct shapes, hooks, device alignment, and dtype conformity.
- Measurement Correctness: validate identity/consistency, known-answer checks, numerical stability, and scaling invariants.
- Ablation Verification: confirm zeroed components, measurable impact, restoration, and component specificity.
- Baseline Replication: compare against published baselines or sensible trivial baselines with defined tolerance.
- Compute Environment & Reproducibility: verify GPU availability, device consistency, CUDA version, and deterministic seeds.
- ML-Specific Validation: gradient check, edge-case tests, minimum test coverage, and determinism checks.
- End-to-End Smoke Test: run a complete pipeline configuration and produce a validated report plus timing data.
Quick Start
Run the full pre-flight validation checklist on the current project to generate a validation-report.md.
