skills.rest
eddielueng
eddieluengCOMMUNITY
Explore all skills by this author →
GitHub

pytorch-fsdp

Configure and troubleshoot PyTorch FSDP2 fully_shard distributed training.

1|Updated Apr 14, 2026
Search Tags:#sharding#pytorch#distributed training#fsdp#mixed precision#fsdp2#cpu offloading
One-click install
npx skills add https://github.com/eddielueng/hermes-agent-zh --skill pytorch-fsdp-eddielueng
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: pytorch-fsdp
Source: https://github.com/eddielueng/hermes-agent-zh/tree/main/optional-skills/mlops/pytorch-fsdp
Command: npx skills add https://github.com/eddielueng/hermes-agent-zh --skill pytorch-fsdp-eddielueng

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

It resolves confusion and performance pitfalls when setting up and operating Fully Sharded Data Parallel (FSDP) training in PyTorch, especially around parameter sharding, mixed precision, CPU offloading, and the FSDP2 (fully_shard) workflow.

Core Features & Use Cases

  • FSDP configuration guidance: Helps you reason about how parameter sharding, precision settings, and CPU offloading affect memory and throughput.
  • FSDP2 (fully_shard) operational model: Explains the pre-forward/backward all-gather and post-forward/backward free/reshard steps so you can implement training correctly.
  • Distributed setup and debugging context: Provides supporting knowledge for distributed collectives (e.g., torch.distributed backends and initialization) that commonly impacts FSDP stability.

Use case: You’re training a large Transformer and need to reduce GPU memory usage while keeping communication correct; you want guidance for using FSDP2 (fully_shard) and understanding when parameters are all-gathered and freed during forward/backward.

Quick Start

Use the pytorch-fsdp skill to get an end-to-end explanation of how to apply PyTorch FSDP2 (fully_shard) to your model and what to expect during pre-forward/backward and post-forward/backward hooks.

Frequently Asked Questions about pytorch-fsdp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How does parameter sharding work in PyTorch FSDP2?▼

Parameter sharding in PyTorch FSDP2 works by all-gathering parameters before the forward and backward passes, then freeing or resharding them afterward to reduce GPU memory usage during distributed training.

How do I configure mixed precision and CPU offloading in PyTorch FSDP?▼

To configure mixed precision and CPU offloading in PyTorch FSDP, you must reason about how these settings affect memory and throughput, applying them to manage parameter precision and offload parameters to CPU during sharded training.

When do I need fully sharded data parallel for large-scale distributed training?▼

You need fully sharded data parallel for large-scale distributed training when you must manage parameter sharding across GPUs to reduce memory usage while maintaining correct collective communication ordering for large models like Transformers.

Why does my PyTorch FSDP training fail with collective communication errors?▼

PyTorch FSDP training fails with collective communication errors when there is misalignment with torch.distributed initialization and backend constraints, requiring correct collective communication ordering to maintain stability.

Can I use the fully_shard API for my Transformer model training?▼

You can use the fully_shard API for Transformer model training to apply FSDP2 sharding, which handles the pre-forward and post-backward hook timing to all-gather and free parameters correctly during your training workflow.

What are the limitations of using FSDP2 for distributed training?▼

Limitations of using FSDP2 for distributed training include the complexity of managing the sharding lifecycle, ensuring correct hook timing for parameter all-gather and resharding, and aligning with torch.distributed backend constraints.

Related Skills

Explore More Software Engineering Skills→
ovachieverovachiever
52

pytorch-fsdp

Apply PyTorch FSDP for sharded, mixed-precision distributed training of large models.

Community
Advanced
NVIDIA-NeMoNVIDIA-NeMo
852

perf-moe-optimization-workflow

Optimize MoE model training throughput by addressing memory, communication, and compute bottlenecks.

Official
Advanced
MesferAliMesferAli

pytorch-fsdp2

Implement PyTorch FSDP2 distributed training with DTensor sharding and checkpointing.

Community
Advanced
LynxLabVNLynxLabVN

pytorch-fsdp

Optimize PyTorch FSDP training with parameter sharding, mixed precision, and CPU offloading.

Official
Advanced