ml-training-pipeline

Set up distributed training in PyTorch by starting with single-node multi-GPU DDP verification using torchrun, enabling bf16 mixed precision, and saving model, optimizer, scheduler, step, and RNG states to ensure training continuity.

Switch from DDP to FSDP only when DDP provably triggers out-of-memory errors. This Skill provides explicit guardrails and decision rules for adopting FSDP in your distributed training pipeline.

The best way to verify a distributed training pipeline is to run an overfit-one-batch test locally using single-node DDP before scaling, catching silent failures related to logging, checkpoint rank, effective batch size, and seed handling early.

Mixed precision checkpointing in PyTorch enables bf16 from day one and saves model, optimizer, scheduler, step, and RNG states with verified reloads, preserving training continuity and preventing silent failures during distributed model training.

PyTorch DDP training fails silently due to incorrect logging, improper checkpoint rank handling, mismatched effective batch sizes, or poor seed handling. Use a dedicated silent-failure checklist to verify these components during setup.

Yes, you can use torchrun for multi-node DDP, but the recommended approach is to first verify overfit-one-batch succeeds on single-node multi-GPU DDP to catch distributed bugs locally before scaling to multi-node training.