torch-tensor-parallelism

Tensor parallelism in PyTorch splits individual neural network layers across multiple devices to overcome single-device memory bottlenecks. You need it when training models too large for one GPU, enabling distributed training by partitioning weights and inputs across ranks.

Implement column parallel and row parallel linear layers by creating ColumnParallelLinear and RowParallelLinear classes. Partition weights and biases across ranks, then use all_gather for column outputs and all_reduce for row outputs to assemble final results in distributed training.

torch.distributed handles model parallelism communication using all_gather and all_reduce patterns. These collectives assemble sharded outputs from column parallel layers and synchronize gradients across ranks, ensuring correct tensor shape alignment and gradient flow during distributed training.

Yes, you can shard weights and biases across ranks using PyTorch tensor parallelism. ColumnParallelLinear shards output dimensions by splitting weights vertically, while RowParallelLinear shards input dimensions by splitting weights horizontally across distributed workers.

Test tensor parallel shapes and gradients by running distributed tests in PyTorch across multiple ranks. Verify that ColumnParallelLinear and RowParallelLinear outputs maintain expected shapes and that all_reduce operations correctly aggregate gradients back to each rank.

Column parallel splits the output dimension of a layer across workers, requiring all_gather to concatenate results. Row parallel splits the input dimension, requiring all_reduce to sum partial outputs. Both are tensor parallelism techniques for distributing large layers in PyTorch.