pytorch-fsdp

Configure PyTorch FSDP training with parameter sharding and mixed precision.

3|Updated Apr 4, 2026

Search Tags:#sharding #pytorch #distributed-training #mixed-precision #fsdp #cpu-offloading #dtensor

Ready to run

npx skills add https://github.com/twjarviszyra-web/hermes-unbound --skill pytorch-fsdp-twjarviszyra-web

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: pytorch-fsdp
Source: https://github.com/twjarviszyra-web/hermes-unbound/tree/main/skills/mlops/training/pytorch-fsdp
Command: npx skills add https://github.com/twjarviszyra-web/hermes-unbound --skill pytorch-fsdp-twjarviszyra-web

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps you design, debug, and optimize PyTorch FSDP workflows for training large models with lower memory use and efficient distributed communication.

Core Features & Use Cases

Parameter sharding, mixed precision, and CPU offloading guidance for memory-constrained training.
FSDP2 and fully_shard migration help, including DTensor-based parameter handling and hook-driven execution.
Practical advice for transformer-scale models, optimizer setup, state dict management, and performance tuning across multi-GPU or multi-node jobs.

Quick Start

Ask for step-by-step help applying PyTorch FSDP to your model, including sharding strategy, device mesh setup, and the right training and checkpointing workflow.

Frequently Asked Questions about pytorch-fsdp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I set up PyTorch FSDP for large model distributed training?▼

PyTorch FSDP sets up distributed training by applying parameter sharding, mixed precision, and communication-aware memory management to reduce memory use across multi-GPU or multi-node jobs.

What is the difference between FSDP and FSDP2 with fully_shard migration?▼

FSDP2 migration introduces DTensor-based parameter handling and hook-driven execution, replacing older sharding workflows with a more granular fully_shard approach for transformer-scale workloads.

Can I use CPU offloading with PyTorch FSDP to handle memory-constrained training?▼

CPU offloading is supported within PyTorch FSDP to manage memory-constrained training by offloading parameters, alongside mixed precision and coordinated all-gather and reduce-scatter operations.

How do I apply bottom-up module sharding and device mesh setup in PyTorch FSDP?▼

Bottom-up module sharding in PyTorch FSDP requires correct device mesh setup and DTensor-aware parameter handling to properly coordinate sharding across transformer-scale model architectures.

What are common limitations when configuring optimizer integration and state dict workflows in FSDP?▼

FSDP optimizer integration and state dict workflows require careful coordination of all-gather, reduce-scatter, and pre-forward or backward hook behavior to avoid sharding inconsistencies during checkpointing.