setup-validation

What problem does it solve?

Structured pre-flight checklist that verifies the entire experiment pipeline produces correct results before committing to a full sweep. Catches data bugs, model loading errors, measurement mistakes, ablation failures, and baseline mismatches — all cheaper to fix now than after 200 GPU-hours.

Core Features & Use Cases

• Data Integrity Checks: shape checks, distribution checks, label verification, reproducibility checks, and leakage prevention.
• Model Loading Verification: ensure non-trivial outputs, correct shapes, hooks, device alignment, and dtype conformity.
• Measurement Correctness: validate identity/consistency, known-answer checks, numerical stability, and scaling invariants.
• Ablation Verification: confirm zeroed components, measurable impact, restoration, and component specificity.
• Baseline Replication: compare against published baselines or sensible trivial baselines with defined tolerance.
• Compute Environment & Reproducibility: verify GPU availability, device consistency, CUDA version, and deterministic seeds.
• ML-Specific Validation: gradient check, edge-case tests, minimum test coverage, and determinism checks.
• End-to-End Smoke Test: run a complete pipeline configuration and produce a validated report plus timing data.

Quick Start

Run the full pre-flight validation checklist on the current project to generate a validation-report.md.