skills.rest
zerwiz
zerwizCOMMUNITY
Explore all skills by this author →
GitHub

pytorch-fsdp

Configure and troubleshoot PyTorch Fully Sharded Data Parallel training.

1|Updated Apr 10, 2026
Search Tags:#sharding#pytorch#distributed training#fsdp#mixed precision#dtensor
One-click install
npx skills add https://github.com/zerwiz/wayofpi --skill pytorch-fsdp-zerwiz
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: pytorch-fsdp
Source: https://github.com/zerwiz/wayofpi/tree/main/.hermes/hermes-agent/optional-skills/mlops/pytorch-fsdp
Command: npx skills add https://github.com/zerwiz/wayofpi --skill pytorch-fsdp-zerwiz

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps engineers design, tune, and debug large-scale PyTorch training with Fully Sharded Data Parallel, reducing memory pressure while preserving strong throughput and operational clarity.

Core Features & Use Cases

  • Parameter Sharding Strategy: Understand how parameters, gradients, and optimizer states are split across workers for efficient distributed training.
  • Performance and Memory Tuning: Apply mixed precision, CPU offloading, bucket sizing, and communication overlap to balance speed and footprint.
  • FSDP2 Migration Guidance: Work with DTensor-based sharding, device meshes, and the newer fully_shard workflow for modern eager-mode training.
  • Use Case: A team training a transformer across multiple GPUs can use this Skill to choose the right sharding setup, avoid communication bottlenecks, and keep memory usage stable during forward and backward passes.

Quick Start

Ask for an FSDP setup plan for your model, specifying your hardware, desired precision, offloading needs, and whether you are using FSDP1 or FSDP2.

Frequently Asked Questions about pytorch-fsdp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I configure PyTorch FSDP for multi-node distributed training?▼

Tune FSDP memory and speed by applying mixed precision, enabling CPU offloading, and adjusting bucket sizing to overlap communication. This balances memory footprint and throughput during forward and backward passes.

How does parameter sharding work in PyTorch distributed training?▼

Migrate to FSDP2 by adopting DTensor-based sharding and the newer fully_shard workflow for modern eager-mode training. This involves utilizing device meshes and DTensor-aware optimization instead of older FSDP1 implementations.

What is the best way to tune PyTorch FSDP memory usage and throughput?▼

Tune FSDP memory and speed by applying mixed precision, enabling CPU offloading, and adjusting bucket sizing to overlap communication. This balances memory footprint and throughput during forward and backward passes.

How do I migrate from FSDP1 to FSDP2 using DTensor?▼

Migrate to FSDP2 by adopting DTensor-based sharding and the newer fully_shard workflow for modern eager-mode training. This involves utilizing device meshes and DTensor-aware optimization instead of older FSDP1 implementations.

Why does my PyTorch FSDP training hang during backward passes?▼

PyTorch FSDP training hangs during backward passes often due to incorrect process-group initialization or communication-unsafe state transitions. Ensure proper bottom-up module sharding and correct checkpoint state handling to resolve bottlenecks.

Related Skills

Explore More Software Engineering Skills→
gqf2008gqf2008

pytorch-fsdp

Apply FSDP2 per-parameter sharding to PyTorch models with device meshes.

Community
Advanced
NVIDIANVIDIA
2.8k

memory-tuning

Mitigate GPU memory pressure and OOM during Megatron training with fragmentation fixes.

Official
Advanced
ovachieverovachiever
52

pytorch-fsdp

Apply PyTorch FSDP for sharded, mixed-precision distributed training of large models.

Community
Advanced
RheasilviaRheasilvia

pytorch-fsdp

Configure fully sharded data parallelism for distributed PyTorch training.

Community
Advanced