skills.rest
quiznat
quiznatCOMMUNITY
Explore all skills by this author →
Website𝕏@QuiznatGitHub

pytorch-fsdp

Implement per-parameter sharding for PyTorch models with FSDP2 and device meshes.

Updated Mar 31, 2026
Search Tags:#pytorch#distributed-training#fsdp#memory-optimization#dtensor#parameter-sharding#device-mesh
One-click install
npx skills add https://github.com/quiznat/Hermes_Sapho --skill pytorch-fsdp-quiznat
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: pytorch-fsdp
Source: https://github.com/quiznat/Hermes_Sapho/tree/main/.hermes/skills/mlops/training/pytorch-fsdp
Command: npx skills add https://github.com/quiznat/Hermes_Sapho --skill pytorch-fsdp-quiznat

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

PyTorch FSDP enables memory-efficient distributed training for large models; this Skill provides expert guidance, best practices, and hands-on patterns to implement FSDP-based training.

Core Features & Use Cases

  • Per-parameter sharding using Fully Sharded Data Parallel (FSDP) for large models.
  • Support for FSDP2 with DTensor-based sharding, device meshes, and mixed precision.
  • Robust initialization, pre-forward and post-forward hooks, and gradient communication strategies to maximize throughput and memory savings.
  • Use Case: Scaling model training across multiple GPUs and nodes with memory constraints.

Quick Start

Apply fully_shard to your model and train with a mesh to enable per-parameter sharding.

Frequently Asked Questions about pytorch-fsdp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train large PyTorch models when GPU memory is insufficient?▼

You can use Fully Sharded Data Parallel (FSDP) to enable memory-efficient distributed training for large PyTorch models. FSDP applies per-parameter sharding across multi-GPU and multi-node setups to bypass memory constraints.

What is the best way to apply per-parameter sharding using PyTorch FSDP2?▼

The best way to apply per-parameter sharding with FSDP2 is by using the fully_shard API. You apply fully_shard to your model and train with a device mesh to enable DTensor-based sharding.

Does PyTorch FSDP support mixed precision and CPU offloading for distributed training?▼

Yes, PyTorch FSDP supports mixed precision and CPU offloading for distributed training. These features are integrated alongside DTensor-based sharding and device meshes to maximize memory savings and throughput.

How do pre-forward and post-forward hooks work in FSDP training workflows?▼

Pre-forward and post-forward hooks in FSDP training workflows manage parameter initialization and gradient communication. They ensure robust parameter management and efficient gradient synchronization across the distributed setup during training.

When should I choose DTensor-based sharding over standard FSDP for multi-GPU scaling?▼

Choose DTensor-based sharding over standard FSDP when upgrading to FSDP2 for multi-GPU scaling. DTensor integrates natively with device meshes, providing robust parameter management and advanced sharding strategies for large models.

Related Skills

Explore More Software Engineering Skills→
InverterNetworkInverterNetwork

pytorch-lightning

Standardize PyTorch training workflows with Lightning for distributed strategies.

Official
Advanced
jacardljacardl

pytorch-fsdp

Apply Fully Sharded Data Parallel to shard parameters across multi-GPU workers.

Community
Advanced
gqf2008gqf2008

pytorch-fsdp

Apply FSDP2 per-parameter sharding to PyTorch models with device meshes.

Community
Advanced
Rawgrowth-ConsultingRawgrowth-Consulting

pytorch-fsdp

Implement distributed PyTorch FSDP training with per-parameter sharding and CPU offloading.

Official
Advanced