torch-tensor-parallelism

Tensor parallelism in PyTorch requires sharding weight matrices and managing biases across distributed ranks. You create column-parallel and row-parallel linear layers to split weights along appropriate dimensions for multi-GPU training.

Column-parallel and row-parallel linear layers differ by the weight dimension they split. Column-parallel shards the output dimension requiring all-gather, while row-parallel shards the input dimension using all-reduce to maintain output shape consistency.

Bias handling in tensor parallelism requires specific strategies for column and row configurations to avoid duplication or missing gradients. Proper bias management maintains gradient flow and output shape consistency across distributed ranks during multi-GPU training.

Yes, tensor-parallel linear layers require a torch.distributed environment to function. This environment handles the all-gather and all-reduce communication needed to coordinate weight splitting and gradient flow across multi-GPU or multi-node ranks.

Tensor parallelism output shapes break when dimension chunking is incorrect or communication operations are misapplied. Ensuring proper weight splitting and correct all-gather or all-reduce operations maintains output shape consistency across distributed ranks.