megatron-fsdp

To enable Megatron FSDP for distributed training, set both cfg.dist.use_megatron_fsdp and cfg.ddp.use_megatron_fsdp to True, then run the Megatron-Bridge performance harness to verify the setup.

The difference between Megatron FSDP and Torch FSDP2 involves a runtime path choice provided during setup, allowing you to select and validate the specific Fully Sharded Data Parallel implementation for robust experimentation.

You need Fully Sharded Data Parallel for large model training when optimizing memory usage and improving training scalability across multiple GPUs for very large Megatron models.

To ensure checkpoint format compatibility with Megatron FSDP, the system enforces compatible checkpoint formats during the FSDP configuration process to maintain memory efficiency and proper validation checks.

Yes, Megatron-Bridge supports FSDP for large-scale pretraining and fine-tuning workflows that demand distributed memory management and performance gains across multiple GPUs.

Common pitfalls when configuring Megatron FSDP are mitigated by using provided code anchors, validation steps, and pitfalls guidance to ensure correct runtime selection and robust experimentation.