skills.rest
cryptopafi
cryptopafiCOMMUNITY
Explore all skills by this author →
GitHub

pytorch-fsdp

Troubleshoot PyTorch FSDP training with mixed precision and CPU offloading.

1|Updated Apr 12, 2026
Search Tags:#sharding#pytorch#distributed-training#mixed-precision#fsdp#cpu-offloading#fsdp2
One-click install
npx skills add https://github.com/cryptopafi/nexusos-skills --skill pytorch-fsdp-cryptopafi
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: pytorch-fsdp
Source: https://github.com/cryptopafi/nexusos-skills/tree/main/mlops/training/pytorch-fsdp
Command: npx skills add https://github.com/cryptopafi/nexusos-skills --skill pytorch-fsdp-cryptopafi

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

PyTorch FSDP (Fully Sharded Data Parallel) addresses the memory and scalability limits of large-model training by sharding parameters, gradients, and optimizer states across devices and workers to enable training models that would otherwise not fit in GPU memory. It also helps engineers migrate from legacy FSDP1 patterns to the newer FSDP2/DTensor-based workflows, and provides guidance for mixed precision, CPU offloading, and collective configuration to avoid hangs and performance regressions.

Core Features & Use Cases

  • Parameter and optimizer sharding: Advice on per-parameter sharding, grouping strategies, and when to apply fully_shard bottom-up for best communication efficiency.
  • Mixed precision & memory optimizations: Recommendations for AMP/autocast, gradient scaling, and resharding strategies to maximize throughput while conserving memory.
  • CPU offloading and cluster integration: Guidance for using CPU offload, device meshes, NCCL/Gloo backend selection, and environment variables for multi-node setups.
  • Debugging and migration: Tips to debug init_process_group issues, common NCCL/Gloo pitfalls, and steps to migrate from FSDP1 to FSDP2 with DTensor and distributed checkpointing.
  • Use case: Training large transformer models across 4+ GPU nodes with reduced memory footprint and overlapped communication.

Quick Start

Show me how to configure PyTorch FSDP2 with mixed precision, CPU offloading, and a DTensor device mesh for a multi-node GPU cluster.

Frequently Asked Questions about pytorch-fsdp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I configure PyTorch FSDP2 with mixed precision and CPU offloading on a multi-node GPU cluster?▼

Configure PyTorch FSDP2 by initializing a DTensor DeviceMesh across nodes, applying fully_shard for parameter sharding, enabling autocast for mixed precision, and activating CPU offload to reduce GPU memory usage during distributed training.

What is the difference between FSDP1 and FSDP2 for distributed training?▼

FSDP2 migrates from legacy FSDP1 patterns by using DTensor and DeviceMesh for distributed training, offering improved collective operation ordering and distributed checkpointing while maintaining parameter sharding and mixed precision capabilities.

Why does my PyTorch FSDP training hang during collective operations on multi-node GPU clusters?▼

PyTorch FSDP training hangs often stem from incorrect process group initialization, NCCL/Gloo backend configuration issues, or mismatched collective operation ordering and timeouts across workers in the distributed training setup.

When should I use CPU offloading with PyTorch FSDP?▼

Use CPU offloading with PyTorch FSDP when training large models that exceed GPU memory limits, allowing parameters and optimizer states to offload to CPU RAM to prevent out-of-memory errors across multi-node GPU clusters.

Can I use PyTorch FSDP with Gloo backend instead of NCCL for distributed training?▼

PyTorch FSDP supports both NCCL and Gloo backends for distributed training, with NCCL preferred for GPU-to-GPU communication efficiency and Gloo available as a fallback for specific cluster configurations and CPU offloading scenarios.

What is the best way to shard parameters in PyTorch FSDP for transformer models?▼

The best way to shard parameters in PyTorch FSDP is applying fully_shard bottom-up with appropriate per-parameter grouping strategies, maximizing communication efficiency and enabling effective gradient resharding for large transformer models.

Related Skills

Explore More Software Engineering Skills→
xiaoquqixiaoquqi

pytorch-fsdp

Apply PyTorch FSDP memory sharding across data-parallel workers for distributed training.

Community
Advanced
zerwizzerwiz

huggingface-accelerate

Convert single-GPU PyTorch training scripts into distributed mixed-precision workflows.

Community
Advanced
gagan114662gagan114662

pytorch-fsdp2

Integrate PyTorch FSDP2 with parameter sharding and distributed checkpointing.

Community
Advanced
gagan114662gagan114662

pytorch-lightning

Organize PyTorch training loops and automate distributed training with DDP, FSDP, and DeepSpeed.

Community
Advanced