pytorch-fsdp

Name: pytorch-fsdp
Availability: InStock
Author: Khyro

Optimize PyTorch FSDP training workflows with sharded, memory-efficient distributed learning.

Updated Apr 2, 2026

Search Tags:#sharding #pytorch #distributed-training #mixed-precision #fsdp #cpu-offloading #dtensor

Ready to run

npx skills add https://github.com/JKhyro/HERMES-AGENT --skill pytorch-fsdp-jkhyro

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: pytorch-fsdp
Source: https://github.com/JKhyro/HERMES-AGENT/tree/main/skills/mlops/training/pytorch-fsdp
Command: npx skills add https://github.com/JKhyro/HERMES-AGENT --skill pytorch-fsdp-jkhyro

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps engineers train large PyTorch models more efficiently by reducing memory pressure, coordinating distributed communication, and simplifying the use of Fully Sharded Data Parallel strategies.

Core Features & Use Cases

Memory-Efficient Training: Shard parameters, gradients, and optimizer state across workers to fit larger models on available hardware.
Modern Distributed Workflows: Apply FSDP2 patterns, device meshes, mixed precision, CPU offloading, and hook-driven collectives for production training.
Practical Scenarios: Use it when migrating from older FSDP setups, tuning multi-GPU or multi-node training, or debugging sharding and synchronization behavior in transformer-scale workloads.

Quick Start

Ask for a PyTorch FSDP training plan that configures sharding, mixed precision, and optimizer setup for your model and hardware.

Frequently Asked Questions about pytorch-fsdp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I shard parameters and optimizer state for PyTorch distributed training?▼

PyTorch distributed training shards parameters, gradients, and optimizer state across workers using FSDP to fit larger models on available hardware. This reduces memory pressure by keeping only shards in GPU memory during transformer-scale workloads.

What is the best way to migrate older PyTorch FSDP setups to FSDP2?▼

Migrating to FSDP2 requires applying modern distributed workflows including device meshes, DTensor-aware optimizer handling, and hook-driven collectives. You must configure bottom-up module sharding to replace older FSDP patterns for production training.

Does PyTorch FSDP support mixed precision and CPU offloading for large models?▼

PyTorch FSDP supports mixed precision and CPU offloading to optimize memory-efficient distributed learning. These features allow you to offload parameters to CPU and use lower precision computations to fit larger models on limited GPU memory.

How do I configure device-mesh setup for bottom-up module sharding in PyTorch?▼

Configuring device-mesh setup for bottom-up module sharding requires correct initialization of the distributed process group and mesh topology in PyTorch. This setup coordinates hook-based all-gather and reduce-scatter operations across workers for FSDP2 migration scenarios.

Why does my PyTorch FSDP training fail during sharding and synchronization?▼

PyTorch FSDP training fails during sharding and synchronization when device-mesh setup is incorrect or modules are not sharded bottom-up. Debugging requires verifying hook-based all-gather and reduce-scatter coordination to ensure proper distributed communication.

When should I not use FSDP for distributed training?▼

You should not use FSDP for distributed training when models are small enough to fit entirely on a single GPU without memory constraints. FSDP's overhead from hook-based all-gather and reduce-scatter coordination is unnecessary for transformer workloads that do not require parameter sharding.