huggingface-accelerate

To convert a PyTorch script for distributed training, wrap models, optimizers, and dataloaders with an Accelerator object using accelerator.prepare(). This allows the same training loop to scale across single GPU, multi-GPU, and multi-node environments with minimal code changes.

Yes, mixed precision training with FP16, BF16, or FP8 is fully supported. You can enable these precision options alongside gradient accumulation and checkpointing to optimize performance without rewriting your core training logic.

The best way to switch between DDP, FSDP, and DeepSpeed ZeRO is by using plugin-based settings during configuration. This unified API manages the underlying parallelism patterns so you can change strategies without altering the core training loop.

Yes, it supports scaling across multi-node environments and integrates with Megatron-LM parallelism patterns. You can configure multi-node launches and apply advanced distributed strategies using the accelerate launch workflow.

You need accelerator.backward() because it properly handles gradient synchronization across distributed processes. Replacing standard loss.backward() ensures gradients are correctly managed when scaling across multiple GPUs or nodes.

You should not use a unified distributed API if your training scenario requires highly customized CUDA device logic or specialized hardware configurations that fall outside standard DDP, FSDP, or DeepSpeed ZeRO patterns.