distributed-training-advisor

Use data parallelism when dataset volume makes epochs too slow on one device; each worker holds a full model replica. Use model parallelism only when parameters and activations exceed single-device memory, preferring PyTorch FSDP over hand-partitioning.

Use Ray Train for multi-GPU model training and Ray Tune for hyperparameter search. Use Dask for pandas-scale preprocessing, batch inference, and sklearn scale-out. Use Spark only when the organization already operates a big-data estate, not for greenfield ML.

No. Kueue admission alone does not gang-schedule arbitrary pods. Pair it with a supported integration such as JobSet, LeaderWorkerSet, or Kubeflow Training Operator, or use Volcano with its PodGroup integration, and prove all-or-nothing admission with a contention test.

Low GPU utilization is often storage starvation, not insufficient compute. Check data-loader wait time, shard object keys across workers instead of broadcasting, use many similar-sized part-files, stream large datasets, and cache across epochs on local NVMe.

Yes, but only with mandatory checkpoint-resume to object storage proven in a drill. Synchronous allreduce training restarts the entire gang when one worker is preempted, so budget wall-clock accordingly; async or elastic training is the alternative only when the framework supports it.

Avoid distribution when single-node levers suffice: a bigger GPU, mixed precision, gradient accumulation or checkpointing, QLoRA quantized fine-tuning, or better data loading. Single-node multi-GPU jobs also avoid gang scheduling entirely since they run as one pod.