torch-pipeline-parallelism

Pipeline parallelism splits transformer layers across multiple GPUs and coordinates forward and backward passes using scheduling strategies like AFAB (All-Forward-All-Backward) and 1F1B. This approach reduces per-GPU memory requirements and enables training of larger models by distributing computation across ranks while maintaining gradient flow and synchronization.

Model partitioning assigns consecutive transformer layers to different ranks based on your pipeline depth and GPU count. Define clear boundaries between stages, handle activation caching at partition points, and use explicit shape tracking across devices to ensure tensors maintain correct dimensions during inter-rank communication.

Maintain gradient flow by coordinating backward passes across ranks using AFAB or 1F1B scheduling, applying loss scaling to prevent gradient underflow, and synchronizing gradient updates between pipeline stages. Use torch.distributed P2P operations (send/recv) to exchange activations and gradients while preserving computation order.

Yes, pipeline parallelism uses torch.distributed P2P operations like send and recv for direct communication between ranks. This enables fine-grained control over activation and gradient exchange without requiring collective operations, making it suitable for non-homogeneous pipeline topologies and custom scheduling strategies.

Common issues include incorrect activation caching leading to stale gradients, shape mismatches during inter-rank communication, loss of gradient synchronization between stages, and inefficient scheduling causing pipeline bubbles. Explicit shape handling, careful verification of gradient flow, and rigorous testing across all ranks prevent these failures.