Apply FSDP2 per-parameter sharding to PyTorch models with device meshes.
npx skills add https://github.com/gqf2008/hermez-ai --skill pytorch-fsdp-gqf2008
Please help me install this Agent Skill. Skill: pytorch-fsdp Source: https://github.com/gqf2008/hermez-ai/tree/main/skills/mlops/training/pytorch-fsdp Command: npx skills add https://github.com/gqf2008/hermez-ai --skill pytorch-fsdp-gqf2008
Fully Sharded Data Parallelism (FSDP) in PyTorch enables memory-efficient, scalable training for very large models by sharding parameters and managing gradients across processes, including FSDP2 with per-parameter sharding, mixed precision, and CPU offloading.
Apply fully_shard to your PyTorch model with a defined device mesh and begin distributed training to leverage per-parameter sharding.
High-intent search queries and answers about installing and using this skill.
PyTorch FSDP minimizes peak memory by sharding model parameters across distributed processes. It performs pre-forward all-gather to reconstruct layers locally, then reshards them post-forward to maximize throughput while keeping memory consumption low.
You apply per-parameter sharding by defining a device mesh and applying the fully_shard function directly to your PyTorch model. This leverages DTensor-based memory layout orchestration for scalable multi-node training of transformer-scale workloads.
FSDP2 introduces per-parameter sharding using a DTensor-based memory layout, moving away from the flat parameter management of FSDP1. This provides finer memory optimization and better device mesh integration for transformer-scale workloads.
Yes, FSDP supports mixed precision and CPU offloading to balance compute and memory resources in heterogeneous environments. This enables training large transformer models that exceed single-device memory budgets.
You need a compatible PyTorch environment, a defined device mesh, and proper distributed initialization. These prerequisites enable parameter sharding and gradient synchronization across your training processes.
VYRE-StudiosConfigure PyTorch FSDP training with parameter sharding, mixed precision, and CPU offloading.
attentiondotnetShard model parameters across data-parallel workers with PyTorch FSDP2 device meshes.
ovachieverApply PyTorch FSDP for sharded, mixed-precision distributed training of large models.
tianhao909Configure DeepSpeed distributed training with ZeRO optimization and mixed precision.
