pytorch-fsdp

Name: pytorch-fsdp
Availability: InStock
Author: plaidmustache

Apply PyTorch FSDP for memory-efficient distributed training across GPUs.

Updated May 4, 2026

Search Tags:#mlops #pytorch #distributed-training #fsdp #gpu-memory #model-sharding

Ready to run

npx skills add https://github.com/Plaidmustache/hermes-nulab --skill pytorch-fsdp-plaidmustache

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: pytorch-fsdp
Source: https://github.com/Plaidmustache/hermes-nulab/tree/main/optional-skills/mlops/pytorch-fsdp
Command: npx skills add https://github.com/Plaidmustache/hermes-nulab --skill pytorch-fsdp-plaidmustache

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Enables scalable, memory-efficient distributed training of large models by guiding the correct application of PyTorch Fully Sharded Data Parallel (FSDP) and FSDP2, including parameter sharding, mixed precision, and CPU offloading.

Core Features & Use Cases

Distributed parameter sharding with PyTorch FSDP2 and integration with FSDP1 paradigms.
Support for mixed precision, CPU offloading, and advanced memory-management strategies.
Use cases include training large transformers and other memory-intensive models across multi-GPU/multi-node environments.

Quick Start

Initialize an FSDP-enabled model, define a mesh, and run a short training loop to observe memory savings and proper gradient synchronization.

Frequently Asked Questions about pytorch-fsdp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

What is PyTorch FSDP and when do I need it for distributed training?▼

PyTorch Fully Sharded Data Parallel (FSDP) is a technique for memory-efficient distributed training that shards parameters across GPUs. You need it when training large models that exceed single-GPU memory limits in multi-GPU or multi-node environments.

How do I set up FSDP2 for a large transformer model?▼

To set up FSDP2, initialize an FSDP-enabled model, define a device mesh configuration, and configure pre-forward hooks. Run a training loop to observe memory savings and ensure proper gradient synchronization across the distributed environment.

Does PyTorch FSDP support mixed precision and CPU offloading?▼

Yes, PyTorch FSDP supports mixed precision and CPU offloading as advanced memory-management strategies. These features help reduce GPU memory consumption during distributed training of memory-intensive models across multiple GPUs or nodes.

What's the best way to reduce GPU memory usage when training large models across multiple nodes?▼

Using PyTorch FSDP for parameter sharding is an effective way to reduce GPU memory usage. It distributes model parameters across GPUs, enabling scalable training of large transformers without overwhelming individual device memory.

Do I need a specific PyTorch version to use Fully Sharded Data Parallel?▼

Yes, you need PyTorch version 2.0 or higher to use Fully Sharded Data Parallel. The implementation requires careful setup of device mesh configuration, pre-forward hooks, and gradient synchronization strategies for proper execution.

Why does gradient synchronization fail in FSDP after setting up the device mesh?▼

Gradient synchronization in FSDP can fail if pre-forward hooks are not properly configured or if the device mesh setup is incorrect. Careful configuration of these elements is required to ensure proper gradient synchronization during distributed training.