Guide DeepSpeed distributed training with ZeRO, pipeline parallelism, and mixed precision.
npx skills add https://github.com/ovachiever/droid-tings --skill deepspeed
Please help me install this Agent Skill. Skill: deepspeed Source: https://github.com/ovachiever/droid-tings/tree/main/skills/deepspeed Command: npx skills add https://github.com/ovachiever/droid-tings --skill deepspeed
Expert guidance for distributed training with DeepSpeed - ZeRO optimization stages, pipeline parallelism, FP16/BF16/FP8, 1-bit Adam, sparse attention.
Install and configure DeepSpeed with a simple training script, then iterate on ZeRO stages and precision modes to achieve scalable training with reduced memory overhead.
High-intent search queries and answers about installing and using this skill.
DeepSpeed enables distributed training by implementing ZeRO optimization stages that partition model states, activations, and gradients across devices. Configure ZeRO stages (1–3) in your training script to reduce per-GPU memory and scale from millions to billions of parameters efficiently.
ZeRO Stage 1 partitions gradients, Stage 2 adds optimizer states, and Stage 3 partitions model parameters across GPUs. Start with Stage 1 for minimal overhead; use Stage 2 or 3 when memory is constrained or when training massive models that don't fit on single devices.
Yes. DeepSpeed supports FP16, BF16, and FP8 mixed precision training natively. Lower precision reduces memory overhead and increases throughput while maintaining convergence; configure precision mode in DeepSpeed config alongside ZeRO for optimal memory efficiency.
Pipeline parallelism splits model layers across GPUs and stages the forward and backward passes to maximize device utilization. DeepSpeed implements pipelined execution to reduce idle time, enabling training of models too large for single-GPU memory or all-reduce communication.
Enable DeepSpeed's profiling and logging to identify bottlenecks in communication and compute. Increase ZeRO stage, reduce batch size or model size, enable sparse attention or 1-bit Adam, and tune activation checkpointing. Start with Stage 1 or 2 before advancing to Stage 3.
Yes. DeepSpeed integrates with transformers models through the accelerate library and direct DeepSpeed API calls. Wrap your training loop with DeepSpeed's engine, pass your transformer model, and apply ZeRO and mixed precision automatically without modifying model code.
cxnaiveRun distributed PyTorch training across multi-GPU and multi-node clusters.
TheBushidoCollectiveExport TensorFlow models to SavedModel and convert to TFLite for deployment.
eddieluengAutomate PyTorch training loops with LightningModule and Trainer abstractions.
ovachieverApply PyTorch FSDP for sharded, mixed-precision distributed training of large models.
