nemo-automodel-distributed-training

Use strategy fsdp2, which is the recommended default and the only strategy supporting TP plus PP. Set tp_size and pp_size, add a pipeline sub-config with pp_schedule and pp_microbatch_size, and let dp_size be inferred as world_size divided by tp_size times pp_size times cp_size.

Set strategy fsdp2 with ep_size greater than 1, which creates a separate moe_mesh alongside the main device mesh. Optionally add a moe sub-config mapping to MoEParallelizerConfig. Note that ep_size must evenly divide dp_size times cp_size.

No. In NeMo AutoModel, megatron_fsdp does not support pipeline parallelism, expert parallelism, or sequence_parallel, and MeshContext validation raises on those combinations. Use fsdp2 when you need PP, EP, or sequence_parallel; DDP is simple data parallelism only.

Context parallelism requires SDPA with Flash Attention or Efficient Attention backend, or Transformer Engine attention; SDPBackend.MATH is incompatible with DTensor. Set cp_size greater than 1, and when combining CP with sequence packing, packed_sequence_size must be divisible by cp_size.

TP is communication-intensive and must stay within a single NVLink domain, typically one node of up to 8 GPUs. Running TP across InfiniBand links degrades throughput severely, so use pipeline parallelism or data parallelism for cross-node scaling instead.