torch-tensor-parallelism

Name: torch-tensor-parallelism
Availability: InStock
Author: bianhaifeng789-hue

Implement ColumnParallelLinear and RowParallelLinear with all_gather and all_reduce in PyTorch.

Updated Apr 14, 2026

Search Tags:#pytorch #distributed-training #tensor-parallelism #torch.distributed #model-parallelism #column-parallel #row-parallel

Ready to run

npx skills add https://github.com/bianhaifeng789-hue/openclaw-config --skill torch-tensor-parallelism-bianhaifeng789-hue

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: torch-tensor-parallelism
Source: https://github.com/bianhaifeng789-hue/openclaw-config/tree/main/skills/tb2/torch-tensor-parallelism
Command: npx skills add https://github.com/bianhaifeng789-hue/openclaw-config --skill torch-tensor-parallelism-bianhaifeng789-hue

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Tensor parallelism enables training of models larger than a single device by splitting layers across ranks, reducing memory bottlenecks and increasing achievable model sizes.

Core Features & Use Cases

Implement ColumnParallelLinear and RowParallelLinear to partition weight and input across workers
Use all_gather and all_reduce communication patterns to assemble outputs and gradients
Validate shapes and gradients across ranks with distributed testing in PyTorch

Quick Start

Set up a minimal distributed example with two ranks implementing ColumnParallelLinear and RowParallelLinear and verify outputs across ranks.

Frequently Asked Questions about torch-tensor-parallelism

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

What is tensor parallelism in PyTorch and when do I need it for distributed training?▼

Tensor parallelism in PyTorch splits individual neural network layers across multiple devices to overcome single-device memory bottlenecks. You need it when training models too large for one GPU, enabling distributed training by partitioning weights and inputs across ranks.

How do I implement column parallel and row parallel linear layers in PyTorch?▼

Implement column parallel and row parallel linear layers by creating ColumnParallelLinear and RowParallelLinear classes. Partition weights and biases across ranks, then use all_gather for column outputs and all_reduce for row outputs to assemble final results in distributed training.

How does torch.distributed handle communication for model parallelism?▼

torch.distributed handles model parallelism communication using all_gather and all_reduce patterns. These collectives assemble sharded outputs from column parallel layers and synchronize gradients across ranks, ensuring correct tensor shape alignment and gradient flow during distributed training.

Can I shard weights and biases across ranks using PyTorch tensor parallelism?▼

Yes, you can shard weights and biases across ranks using PyTorch tensor parallelism. ColumnParallelLinear shards output dimensions by splitting weights vertically, while RowParallelLinear shards input dimensions by splitting weights horizontally across distributed workers.

How do I test tensor parallel shapes and gradients across distributed ranks?▼

Test tensor parallel shapes and gradients by running distributed tests in PyTorch across multiple ranks. Verify that ColumnParallelLinear and RowParallelLinear outputs maintain expected shapes and that all_reduce operations correctly aggregate gradients back to each rank.

What is the difference between column parallel and row parallel in model parallelism?▼

Column parallel splits the output dimension of a layer across workers, requiring all_gather to concatenate results. Row parallel splits the input dimension, requiring all_reduce to sum partial outputs. Both are tensor parallelism techniques for distributing large layers in PyTorch.