huggingface-accelerate

Distributed training across multiple GPUs is simplified by using a unified API that automates device placement and mixed-precision setup with minimal code changes. You can scale seamlessly from a single GPU to multi-node clusters.

Mixed-precision training with DeepSpeed or FSDP is enabled through a single high-level API that supports FP16, BF16, and FP8. Extensible plugins automatically apply these configurations without requiring manual precision setup.

Yes, you can use HuggingFace Accelerate for Megatron-LM on multi-node setups. It provides extensible plugins and an interactive configuration workflow to apply Megatron-LM scaling across distributed clusters.

Yes, the unified API supports automatic device placement for single-GPU training. It applies the same high-level interface and accelerator preparation to scale smoothly from a single device to multi-node clusters.

To launch your training script using the accelerate workflow, apply accelerator.prepare to your objects and run the single-launch command. This interactive configuration automates distributed setup and accelerates prototyping.