Integrate PyTorch FSDP2 fully_shard into training scripts with DCP checkpointing.
npx skills add https://github.com/uv-xiao/pkbllm --skill uv-pytorch-fsdp2
Please help me install this Agent Skill. Skill: uv-pytorch-fsdp2 Source: https://github.com/uv-xiao/pkbllm/tree/main/knowledge/ML/distributed/pytorch-fsdp2 Command: npx skills add https://github.com/uv-xiao/pkbllm --skill uv-pytorch-fsdp2
This Skill enables coding agents to correctly integrate PyTorch FSDP2 (Fully Sharded Data Parallel 2.0) into training scripts, addressing the complexities of distributed training for large models that exceed single-GPU memory.
fully_shard bottom-up for optimal memory and performance.Integrate PyTorch FSDP2 into your existing training script by following the step-by-step procedure outlined in the skill's documentation.
High-intent search queries and answers about installing and using this skill.
PyTorch FSDP2 enables training large models exceeding single-GPU memory by applying the `fully_shard` API bottom-up, distributing parameters, gradients, and optimizer states across multiple GPUs and nodes.
Bottom-up sharding with `fully_shard` in PyTorch involves applying the API sequentially from the innermost model layers to the outermost, ensuring optimal memory management and performance for distributed training.
Distributed checkpointing with DCP for FSDP2 training scripts involves integrating PyTorch Distributed Checkpoint to robustly save and load sharded model states across multiple devices during training.
Mixed precision and CPU offloading can be configured for memory-bound distributed training scenarios in PyTorch FSDP2, allowing you to manage GPU memory limits when training massive models.
You need PyTorch FSDP2 when training massive language models that require more GPU memory than available on a single device, requiring the distribution of parameters, gradients, and optimizer states across multiple GPUs.
handsomelong922Integrate PyTorch FSDP2 into training scripts for per-parameter sharding.
RheasilviaConfigure fully sharded data parallelism for distributed PyTorch training.
ricableAdd PyTorch FSDP2 sharding and distributed checkpointing to training scripts.
unclehowellConfigure PyTorch FSDP training with sharding, mixed precision, and CPU offloading.