torch-pipeline-parallelism

Pipeline parallelism partitions transformer layers across multiple GPUs to train models that exceed single-device memory. It coordinates forward and backward passes across ranks, allowing large distributed training without running out of memory.

AFAB (All-Forward-All-Backward) scheduling processes microbatches by iterating them sequentially through forward passes before executing backward passes. This method caches activations during the forward phase and applies correct loss scaling for each microbatch to ensure accurate gradient flow.

Transformer layer partitioning involves slicing the model into per-rank stages and assigning embeddings and output heads to appropriate ranks. This distributes the computational workload across available GPUs and balances memory usage for distributed training.

Inter-rank communication deadlocks and tensor mismatches occur when dtype and device consistency are not preserved. Using shape-first send and recv communication patterns ensures tensor dimensions match across ranks, preventing communication bugs during distributed training.

Pipeline parallelism supports distributing LLaMA-style transformers across multiple GPUs. By partitioning model layers and applying AFAB microbatch training, you can train larger models while maintaining correct gradient flow and minimizing communication bugs.

Activation caching is required during AFAB scheduling to store intermediate tensors during the forward pass. These cached activations are then retrieved during the backward pass to compute gradients correctly for each microbatch in distributed training.