skills.rest
yo-steven
yo-stevenCOMMUNITY
Explore all skills by this author →
GitHub

perf-megatron-fsdp

Configure Megatron FSDP in Megatron-Bridge with fsdp_dtensor checkpoints and validate via 2-GPU pytest.

Updated May 23, 2026
Search Tags:#checkpointing#fsdp#megatron#training performance#debugging oom#distributed data parallel
One-click install
npx skills add https://github.com/yo-steven/skills-exploration-20260522 --skill perf-megatron-fsdp-yo-steven
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: perf-megatron-fsdp
Source: https://github.com/yo-steven/skills-exploration-20260522/tree/main/skills/Megatron-Bridge/perf-megatron-fsdp
Command: npx skills add https://github.com/yo-steven/skills-exploration-20260522 --skill perf-megatron-fsdp-yo-steven

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill helps you reliably enable and verify Megatron FSDP in Megatron-Bridge when you’re aiming for performance improvements or investigating an OOM/regression caused by a FSDP-related config change.

Core Features & Use Cases

  • Enable Megatron FSDP correctly: Switch Bridge and DDP wrappers to use_megatron_fsdp and set the sharding strategy to optim_grads_params.
  • Avoid checkpoint and validation pitfalls: Ensure checkpoint.ckpt_format is fsdp_dtensor and respect mutual exclusivity with use_torch_fsdp2.
  • Verify with a functional smoke test: Run the existing 2‑GPU pytest test and confirm the training iteration completes without checkpoint-format assertions.

Quick Start

Configure Bridge with use_megatron_fsdp, set data_parallel_sharding_strategy to optim_grads_params, disable average_in_collective, set checkpoint.ckpt_format to fsdp_dtensor, then run the provided 2‑GPU functional pytest smoke test to confirm a stable FSDP pretrain start.

Frequently Asked Questions about perf-megatron-fsdp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I enable Megatron FSDP in Megatron-Bridge without hitting OOM errors?▼

To enable Megatron FSDP and prevent OOM errors, configure Bridge with use_megatron_fsdp, set data_parallel_sharding_strategy to optim_grads_params, and disable average_in_collective to ensure stable distributed training.

What is the correct checkpoint format for Megatron FSDP training runs?▼

The correct checkpoint format for Megatron FSDP training runs is fsdp_dtensor. You must set checkpoint.ckpt_format to fsdp_dtensor and ensure mutual exclusivity with use_torch_fsdp2 to avoid checkpoint format validation failures.

Do I need multiple GPUs to verify a stable FSDP pretrain start in Megatron?▼

Yes, verifying a stable FSDP pretrain start requires running a functional 2-GPU pytest smoke test. This test confirms that the training iteration completes successfully without triggering checkpoint-format assertions.

Why does my Megatron FSDP configuration fail checkpoint validation assertions?▼

Checkpoint validation assertions fail when checkpoint.ckpt_format is not set to fsdp_dtensor or when use_torch_fsdp2 is incorrectly enabled alongside use_megatron_fsdp, violating their mutual exclusivity requirement during FSDP training.

Can I use optim_grads_params sharding strategy to debug training performance regressions?▼

Yes, setting data_parallel_sharding_strategy to optim_grads_params is required for Megatron FSDP and directly aids in debugging training performance regressions or OOM issues caused by FSDP-related parallelism configuration changes.

Related Skills

Explore More Software Engineering Skills→
vikrant-projectvikrant-project
2

huggingface-accelerate

Simplify distributed training of PyTorch models with a unified API.

Community
Advanced
castform-aicastform-ai
49

launch-run

Launch Castform training runs after validation, upload, and argument checks.

Official
Intermediate
sayalinvidiasayalinvidia

nemo-mbridge-perf-megatron-fsdp

Enable Megatron FSDP overrides in Megatron-Bridge for distributed training.

Community
Advanced
yo-stevenyo-steven

mlm-bridge-training

Correlate Megatron-LM and Megatron Bridge training runs by matching configs and diagnosing loss divergence.

Community
Advanced