Configure PyTorch FSDP training with parameter sharding and CPU offloading.
npx skills add https://github.com/Signmanal/VIGIL --skill pytorch-fsdp-signmanal
Please help me install this Agent Skill. Skill: pytorch-fsdp Source: https://github.com/Signmanal/VIGIL/tree/main/optional-skills/mlops/pytorch-fsdp Command: npx skills add https://github.com/Signmanal/VIGIL --skill pytorch-fsdp-signmanal
Training large PyTorch models across multiple GPUs or nodes requires complex configuration of parameter sharding, communication backends, and memory optimization to avoid out-of-memory errors and training hangs, which is error-prone even for experienced machine learning engineers.
Use the pytorch-fsdp skill to configure FSDP2 with per-parameter sharding and CPU offloading for your 13B parameter transformer model to fit training on 8 GPUs.
High-intent search queries and answers about installing and using this skill.
Configure FSDP for large-scale PyTorch distributed training by applying parameter sharding across GPU clusters, enabling mixed precision optimization, and setting up CPU offloading to reduce memory consumption and avoid out-of-memory errors.
Migrate from FSDP1 to FSDP2 by transitioning from legacy flat-parameter sharding to modern per-parameter sharding, utilizing step-by-step migration guidance to update your distributed training configuration and sharding logic.
Set up multi-node distributed training by initializing process groups and selecting appropriate communication backends like NCCL or Gloo, ensuring stable collective communication and proper uneven input handling across GPU clusters.
Yes, you can train a 70B parameter language model on a 16-GPU cluster by correctly configuring FSDP2 sharding, enabling CPU offloading, and applying mixed precision to fit the model within available GPU memory.
PyTorch FSDP training hangs or fails due to incorrect communication backend selection, improper process group initialization, or unhandled uneven inputs, which disrupt collective communication across distributed GPU clusters.
pmcdowallOptimize PyTorch model training with fully sharded data parallel techniques.
Clay-HHKImplement and train large language models with LitGPT single-file recipes.
xiaoquqiApply PyTorch FSDP memory sharding across data-parallel workers for distributed training.
tianhao909Configure DeepSpeed distributed training with ZeRO optimization and mixed precision.
