Apply PyTorch FSDP for sharded, mixed-precision distributed training of large models.
npx skills add https://github.com/ovachiever/droid-tings --skill pytorch-fsdp
Please help me install this Agent Skill. Skill: pytorch-fsdp Source: https://github.com/ovachiever/droid-tings/tree/main/skills/pytorch-fsdp Command: npx skills add https://github.com/ovachiever/droid-tings --skill pytorch-fsdp
This Skill provides expert guidance on Fully Sharded Data Parallel (FSDP) training in PyTorch, covering parameter sharding, mixed precision, CPU offloading, and advanced FSDP configurations for large-scale models.
Initialize a small-scale FSDP example with a 2-GPU setup and progressively scale.
High-intent search queries and answers about installing and using this skill.
PyTorch FSDP (Fully Sharded Data Parallel) distributes model parameters across GPUs, reducing per-GPU memory footprint. FSDP automatically shards weights, activations, and gradients during training, enabling larger batch sizes and faster convergence on multi-GPU or multi-node setups with torch>=2.0.
Combine FSDP with mixed-precision training to compute in lower precision while maintaining accuracy, reducing memory and bandwidth. Enable CPU offloading to move inactive parameters to CPU during forward/backward passes, freeing GPU memory for larger models or batches in memory-constrained environments.
Yes, FSDP works with transformer models (compatible with transformers library and torch>=2.0). Configure parameter bucketing, gradient allreduce orchestration, and DDP initialization synchronization; scale deterministically from 2–8+ GPUs by adjusting shard strategy and synchronization settings per setup size.
FSDP2 provides enhanced multi-node distributed training with improved gradient allreduce orchestration and deterministic playbooks. It refines parameter bucketing and synchronization mechanisms, making large-scale training (8+ GPUs across multiple nodes) more stable and predictable than prior FSDP versions.
CPU offloading reduces peak GPU memory by moving inactive parameters to host RAM, enabling larger model training. Trade-off: increased CPU-GPU communication latency and bandwidth overhead; effective for parameter-heavy models where GPU memory is the bottleneck, not compute speed.
overviewlabsApply Fully Sharded Data Parallel to large PyTorch models for distributed training.
Orchestra-ResearchIntegrate PyTorch FSDP2 with DTensor sharding and DCP checkpointing into training scripts.
ovachieverApply PyTorch FSDP for sharded, mixed-precision distributed training of large models.
ovachieverTrain large language models with Megatron-Core using configurable 3D and 4D parallelism.
