parallel-strategy-analyzer

To choose a distributed training strategy, you analyze model architecture and hardware constraints to recommend optimal configurations across DP, FSDP, TP, PP, EP, and CP. This provides ranked candidates with memory, communication, compute, and pipeline bubble estimates.

Estimating memory for parallel training involves evaluating per-layer activation and parameter estimates against your memory budget. The analysis outputs per-device memory estimates alongside global baseline and candidate configurations to ensure constraints are satisfied.

Yes, you can analyze MoE architectures like Mixtral on 64 devices. The tool evaluates diverse model scales and hardware topologies, auto-synthesizing device mesh recommendations and ranked strategy configurations tailored to your specific setup.

Configuring device mesh topologies for GQA or MHA models requires evaluating strategy candidates across diverse parallel approaches. The tool auto-synthesizes device mesh recommendations by analyzing your specific model architecture and hardware constraints.

When evaluating pipeline parallelism, consider the pipeline bubble estimates and communication overhead generated by the analysis. These constraints are evaluated against your memory budget and compute optimization goals to rank viable candidate configurations.