deepspeed

Create a DeepSpeed JSON config specifying the ZeRO stage (1, 2, or 3) under zero_optimization, then pass it to deepspeed.initialize in your training script. Stage 3 partitions parameters, gradients, and optimizer states across all GPUs for maximum memory savings.

ZeRO stage 2 partitions optimizer states and gradients across data-parallel ranks, while stage 3 additionally partitions model parameters themselves. Stage 3 enables training much larger models but adds parameter-gathering communication overhead.

Yes, DeepSpeed integrates with Hugging Face Transformers and Accelerate by passing a DeepSpeed config file to the Trainer or accelerate launch command. The Skill lists transformers and accelerate as supported dependencies.

Use ZeRO-Offload to offload optimizer states and parameters to CPU memory, or ZeRO-Infinity to also leverage NVMe storage. This enables training models up to 13 billion parameters on a single GPU with competitive throughput.

Out-of-memory errors usually come from an insufficient ZeRO stage, missing activation checkpointing, or no CPU offloading. Increase the ZeRO stage, enable gradient checkpointing, and configure offload_optimizer or offload_param to CPU or NVMe.

Use 1-bit Adam on bandwidth-limited clusters where communication is the bottleneck, as it reduces communication volume by up to 5x with the same convergence as Adam. It requires a warmup phase of roughly 15-20% of training steps before compression starts.