ray-train

Wrap your training loop in a function, call train.torch.prepare_model on your model, and pass the function to TorchTrainer with a ScalingConfig specifying num_workers and use_gpu=True. Ray handles device placement, coordination, and metric aggregation automatically.

Ray Train is best for multi-node orchestration, hyperparameter tuning with Ray Tune, and fault tolerance. Use Accelerate for simpler single-node multi-GPU, Lightning for high-level abstractions and callbacks, and DeepSpeed for maximum performance with complex setup.

Yes, Ray Train provides TransformersTrainer for HuggingFace integration. Define your training function with standard Transformers Trainer and TrainingArguments, then scale it across nodes using ScalingConfig with the desired number of GPU workers.

Define a search space using tune.loguniform or tune.choice, wrap your TorchTrainer in a tune.Tuner, and configure num_samples with a scheduler like ASHAScheduler for early stopping. Ray distributes trials across the cluster and returns the best configuration.

Install the KubeRay operator via Helm, then apply a RayCluster manifest defining head and worker GPU groups. Submit training jobs by port-forwarding the Ray client port and setting RAY_ADDRESS to ray://localhost:10001.

Check cluster status with ray status to verify nodes and GPUs are registered. If workers are missing, restart the head node with ray start --head --port=6379, then reconnect workers using ray start --address=<head-ip>:6379, ensuring firewall rules allow ports 6379 and 8000-9000.