perf-parallelism-strategies

To choose a parallelism strategy, evaluate your model size, sequence length, and hardware topology. Guide the selection by combining data, tensor, pipeline, expert, and sequence parallelism to ensure optimal resource utilization and performance scalability.

Tensor parallel partitions individual model layers across GPUs, while pipeline parallel splits sequential model layers across devices. Aligning these parallelism methods with system constraints ensures efficient resource utilization and performance scalability during large-scale training.

Configure parallelism by setting tensor_model_parallel_size, pipeline_model_parallel_size, and sequence_parallel in the model provider. Base these settings on your specific hardware topology and model size to optimize training performance.

Minimum GPU counts for parallelism configurations depend on heuristic guidelines based on your model size and hardware topology. Profiling and sizing advice helps plan resources efficiently by providing specific hardware constraints for different setups.

Use sequence parallelism in distributed training when working with long sequence lengths. Aligning this method with your hardware topology and model size constraints ensures optimal resource utilization and performance scalability.

Yes, the strategy selection guides configuring expert parallelism alongside data, tensor, pipeline, and sequence parallelism. This ensures optimal resource utilization and performance scalability by aligning parallelism methods with system constraints.