skills.rest
zhuangbiaowei
zhuangbiaoweiCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

pytorch-fsdp2

Integrate PyTorch FSDP2 with fully_shard for distributed model training.

2|Updated Feb 4, 2026
Search Tags:#deep learning#checkpointing#pytorch#distributed training#large models#fsdp2#dtensor
One-click install
npx skills add https://github.com/zhuangbiaowei/smart_bot --skill pytorch-fsdp2-zhuangbiaowei
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: pytorch-fsdp2
Source: https://github.com/zhuangbiaowei/smart_bot/tree/main/skills/pytorch-fsdp2
Command: npx skills add https://github.com/zhuangbiaowei/smart_bot --skill pytorch-fsdp2-zhuangbiaowei

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill enables the correct and efficient use of PyTorch's Fully Sharded Data Parallelism (FSDP2) for training large models that exceed single-GPU memory, by guiding the integration of its advanced features like DTensor-based sharding and distributed checkpointing.

Core Features & Use Cases

  • FSDP2 Integration: Correctly initializes and applies torch.distributed.fsdp.fully_shard bottom-up.
  • Memory Optimization: Configures reshard_after_forward, mixed precision, and CPU offload for memory-bound scenarios.
  • Distributed Checkpointing: Implements robust saving and loading using Distributed Checkpoint (DCP) or state dict helpers.
  • Use Case: Integrate FSDP2 into a large language model training script to shard parameters, gradients, and optimizer states across multiple GPUs, allowing models that would otherwise OOM to be trained effectively.

Quick Start

Integrate PyTorch FSDP2 into your existing training script by following the step-by-step procedure outlined in the skill's documentation.

Frequently Asked Questions about pytorch-fsdp2

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I use PyTorch FSDP2 to train large models that exceed single-GPU memory?▼

PyTorch FSDP2 allows training large models exceeding single-GPU memory by integrating `fully_shard` into training scripts, applying bottom-up sharding to distribute parameters, gradients, and optimizer states across multiple GPUs via DeviceMesh.

What is the correct way to initialize DTensor parameters for distributed training with PyTorch FSDP2?▼

Initializing DTensor parameters for PyTorch FSDP2 involves correctly applying `torch.distributed.fsdp.fully_shard` bottom-up, ensuring proper handling of DTensor parameter distribution and DeviceMesh-based sharding across the training environment.

How do I configure PyTorch FSDP2 mixed precision and CPU offload for memory optimization?▼

PyTorch FSDP2 memory optimization is configured by enabling `reshard_after_forward`, mixed precision, and CPU offload settings during initialization, directly addressing memory-bound scenarios in large model training.

Does PyTorch FSDP2 support distributed checkpointing for saving and loading training states?▼

PyTorch FSDP2 supports distributed checkpointing by implementing robust saving and loading operations using Distributed Checkpoint (DCP) and state dict helpers to manage sharded training states effectively.

Can I use PyTorch FSDP2 to shard models that would otherwise cause out-of-memory errors during training?▼

PyTorch FSDP2 can shard models that would otherwise OOM by distributing parameters, gradients, and optimizer states across multiple GPUs, enabling effective large language model training beyond single-GPU memory limits.

Related Skills

Explore More Software Engineering Skills→
handsomelong922handsomelong922

pytorch-fsdp2

Integrate PyTorch FSDP2 into training scripts for per-parameter sharding.

Community
Advanced
gagan114662gagan114662

pytorch-fsdp2

Integrate PyTorch FSDP2 with parameter sharding and distributed checkpointing.

Community
Advanced
ricablericable

pytorch-fsdp2

Add PyTorch FSDP2 sharding and distributed checkpointing to training scripts.

Community
Advanced
arsityarsity

deepspeed

Configure DeepSpeed ZeRO and 3D parallelism for distributed training of large models.

Community
Advanced