ray-train

Wrap your training loop in a function, call train.torch.prepare_model on the model, and pass it to TorchTrainer with a ScalingConfig specifying num_workers and use_gpu. The same code runs on 1 GPU or 1000 GPUs without further changes.

Ray Train is best for multi-node orchestration, hyperparameter tuning via Ray Tune, and fault tolerance. Use Accelerate for simpler single-node multi-GPU, Lightning for high-level abstractions and callbacks, or DeepSpeed for maximum performance with complex setup.

Yes, Ray Train provides TransformersTrainer for HuggingFace integration. You define a training function using the standard transformers Trainer API, and Ray handles distributed execution across workers and nodes automatically.

Define a search space with tune.loguniform or tune.choice, then pass your TorchTrainer to tune.Tuner with a TuneConfig specifying num_samples and a scheduler like ASHAScheduler. Ray runs trials in parallel across the cluster with early stopping.

Check cluster status with ray status to verify nodes and GPUs are registered. If workers are missing, restart the head node with ray start --head --port=6379, then reconnect workers using ray start --address=<head-ip>:6379.

Yes, Ray Train supports fault tolerance through checkpointing and FailureConfig with max_failures to retry on worker failure. Save checkpoints periodically with train.report and resume state via train.get_checkpoint when training restarts.