pytorch-fsdp

Name: pytorch-fsdp
Availability: InStock
Author: aissenliu

Implement and debug PyTorch FSDP training workflows with sharding and mixed precision.

Updated Apr 27, 2026

Search Tags:#sharding #pytorch #distributed-training #transformers #mixed-precision #fsdp #cpu-offloading

Ready to run

npx skills add https://github.com/AissenLiu/EasyHermes --skill pytorch-fsdp-aissenliu

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: pytorch-fsdp
Source: https://github.com/AissenLiu/EasyHermes/tree/main/hermes-agent/optional-skills/mlops/pytorch-fsdp
Command: npx skills add https://github.com/AissenLiu/EasyHermes --skill pytorch-fsdp-aissenliu

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Provides expert guidance for building and debugging PyTorch Fully Sharded Data Parallel (FSDP) workflows, enabling memory-efficient training of large models.

Core Features & Use Cases

In-depth explanations of FSDP concepts, APIs, and best practices for distributed training.
Step-by-step patterns for common setups (FSDP2, parameter sharding, mixed precision, CPU offloading).
Practical debugging tips and real-world use cases for Transformers and other large models.

Quick Start

Apply fully_shard to a transformer model and start distributed training on multiple GPUs.

Frequently Asked Questions about pytorch-fsdp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I set up PyTorch FSDP for large-model distributed training?▼

To set up PyTorch FSDP for distributed training, apply the fully_shard API to your transformer model and configure multi-GPU clusters. This enables parameter sharding for memory-efficient training of large models.

What is PyTorch FSDP and when do I need it for training transformers?▼

PyTorch FSDP is a Fully Sharded Data Parallel technique needed when training large transformer models that exceed single-GPU memory. It shards parameters across multiple GPUs to enable memory-efficient distributed training workflows.

Can I use mixed precision and CPU offloading with PyTorch FSDP?▼

Yes, you can use mixed precision and CPU offloading with PyTorch FSDP to optimize memory usage during distributed training. These features are supported as common setup patterns within the FSDP workflow.

Does PyTorch FSDP work with FSDP2 and multi-node clusters?▼

Yes, PyTorch FSDP supports FSDP2 and scales across multi-node clusters for large-model distributed training. It requires torch version 2.0 or higher and the transformers library to implement workflows.

Why does my PyTorch FSDP training workflow fail on large transformer models?▼

PyTorch FSDP training workflows often fail on large transformer models due to incorrect parameter sharding or mixed precision configurations. Debugging these distributed pipelines requires applying specific best practices and dependency checks.