pytorch-fsdp

Name: pytorch-fsdp
Availability: InStock
Author: Jason

Apply PyTorch FSDP to shard parameters across distributed GPU workers.

Updated May 11, 2026

Search Tags:#memory #sharding #pytorch #distributed-training #transformers #fsdp

Ready to run

npx skills add https://github.com/jason660519/Project-Manager --skill pytorch-fsdp-jason660519

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: pytorch-fsdp
Source: https://github.com/jason660519/Project-Manager/tree/main/hermes-agent/optional-skills/mlops/pytorch-fsdp
Command: npx skills add https://github.com/jason660519/Project-Manager --skill pytorch-fsdp-jason660519

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

PyTorch FSDP enables memory-efficient training of very large models by sharding parameters across data-parallel workers, reducing peak memory consumption while preserving accuracy.

Core Features & Use Cases

Fully Sharded Data Parallel (FSDP) orchestration across modules and layers.
Support for FSDP2 with per-parameter sharding, mixed precision, and CPU offloading to scale large models on constrained hardware.
Use cases include training transformers and other memory-intensive architectures where GPU memory limits require smaller per-device footprints or larger effective batch sizes.

Quick Start

Provide a minimal PyTorch script that applies fully_shard to a simple model and demonstrates all-gather of parameters before forward and reduce-scatter of gradients after backward.

Frequently Asked Questions about pytorch-fsdp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How does PyTorch Fully Sharded Data Parallel reduce GPU memory for large transformer training?▼

PyTorch Fully Sharded Data Parallel reduces GPU memory by sharding parameters across data-parallel workers. It performs all-gather of parameters before forward and reduce-scatter of gradients after backward, lowering peak memory consumption while preserving model accuracy.

How do I apply fully_shard to a PyTorch model for distributed training?▼

To apply fully_shard for distributed training, you wrap your PyTorch model or specific modules with the FSDP orchestration API. This shards parameters across GPUs, executing all-gather before forward passes and reduce-scatter after backward passes.

Can I use PyTorch FSDP with mixed precision and CPU offloading on constrained hardware?▼

Yes, PyTorch FSDP supports mixed precision and CPU offloading to scale large models on constrained hardware. These features allow you to train memory-intensive architectures by reducing the per-device memory footprint.

What is the difference between FSDP and FSDP2 per-parameter sharding?▼

FSDP2 introduces per-parameter sharding, advancing the original FSDP module-level orchestration. This per-parameter approach aligns with modern PyTorch distributed APIs, offering finer memory control for memory-intensive transformer architectures.

Do I need a specific PyTorch version to use FSDP for memory-efficient training?▼

Yes, you need PyTorch version 2.0 or higher to use FSDP for memory-efficient training. The implementation requires PyTorch distributed APIs and supports Linux and macOS environments alongside the transformers library.

When should I use FSDP instead of standard data parallel for training memory-intensive architectures?▼

Use FSDP instead of standard data parallel when GPU memory limits constrain your batch size or model depth. FSDP shards parameters across workers to reduce peak memory, enabling larger effective batch sizes for transformer-based architectures.