pytorch-fsdp

Name: pytorch-fsdp
Availability: InStock
Author: attentiondotnet

Shard model parameters across data-parallel workers with PyTorch FSDP2 device meshes.

Updated Mar 30, 2026

Search Tags:#pytorch #distributed-training #fsdp #memory-optimization #dtensor #per-parameter-sharding #device-mesh

Ready to run

npx skills add https://github.com/attentiondotnet/hermes-agent --skill pytorch-fsdp-attentiondotnet

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: pytorch-fsdp
Source: https://github.com/attentiondotnet/hermes-agent/tree/main/skills/mlops/training/pytorch-fsdp
Command: npx skills add https://github.com/attentiondotnet/hermes-agent --skill pytorch-fsdp-attentiondotnet

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Memory constraints limit training large models; this skill guides implementing Fully Sharded Data Parallel (FSDP) in PyTorch to reduce memory footprint and enable scalable distributed training.

Core Features & Use Cases

Per-parameter sharding across a data-parallel mesh to maximize memory savings.
All-gather before forward and after backward to obtain unsharded parameters for computation.
Guidance for migrating from FSDP1 to FSDP2 and optimizing with device meshes and mixed precision.

Quick Start

Apply fully_shard to your PyTorch model and begin training using a device mesh to enable per-parameter sharding.

Frequently Asked Questions about pytorch-fsdp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How does PyTorch FSDP2 reduce memory pressure during distributed training?▼

PyTorch FSDP2 reduces memory pressure by disaggregating model parameters across data-parallel workers. It uses a device mesh to enable per-parameter sharding, ensuring each worker only holds a fraction of the parameters during large-model training.

How do I migrate my model from FSDP1 to FSDP2?▼

To migrate from FSDP1, apply the fully_shard API to your model components instead of wrapping entire modules. This enables per-parameter sharding and integrates with device meshes for scalable distributed training.

What's the best way to shard transformer parameters across a device mesh?▼

The best way to shard transformer parameters is using PyTorch FSDP2's per-parameter sharding. It performs all-gather before forward and after backward to reconstruct unsharded parameters, then resharding after computation to maximize memory savings.

Can I use mixed precision workflows with per-parameter sharding in PyTorch?▼

Yes, you can use mixed precision workflows with per-parameter sharding. PyTorch FSDP2 supports mixed precision alongside device mesh configurations to optimize memory usage and accelerate training for large transformer-like models.

When do I need fully sharded data parallel for my PyTorch model?▼

You need fully sharded data parallel when memory constraints limit training large models. FSDP allows you to disaggregate parameters across distributed workers, bypassing single-device memory limits and enabling scalable transformer-like model training.