nemo-mbridge-perf-megatron-fsdp

To mitigate OOM errors during distributed training, enable the Megatron FSDP override in Megatron-Bridge to apply memory-efficient data parallelism. This provides configuration snippets and code anchors to optimize large model scalability.

The correct checkpoint format for Megatron FSDP is fsdp_dtensor. Verifying this checkpoint format is a required step to ensure compatibility and prevent memory regressions during FSDP-based training.

No, you cannot use Megatron FSDP and Torch FSDP2 simultaneously because they are mutually exclusive. The override verifies compatibility rules to ensure only one data parallelism strategy is active.

Debug memory regressions when tuning data parallelism strategies by applying the Megatron FSDP override and verifying checkpoint formats. This process diagnoses memory constraints and provides pitfall notes for FSDP-based training.

Limitations of using Megatron FSDP overrides include strict mutual exclusion with Torch FSDP2 and a hard requirement for the checkpoint format to be fsdp_dtensor, which constrains compatible training configurations.