skills.rest
Zurybr
ZurybrCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

torch-tensor-parallelism

Implement column-parallel and row-parallel linear layers with torch.distributed.

Updated Feb 24, 2026
Search Tags:#pytorch#distributed training#tensor parallelism#model parallelism#linear layers#torch.distributed
One-click install
npx skills add https://github.com/Zurybr/lefarma-skills --skill torch-tensor-parallelism
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: torch-tensor-parallelism
Source: https://github.com/Zurybr/lefarma-skills/tree/main/letta/benchmarks/trajectory-only/torch-tensor-parallelism
Command: npx skills add https://github.com/Zurybr/lefarma-skills --skill torch-tensor-parallelism

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill addresses the complexities of implementing and debugging tensor parallelism in PyTorch, specifically for column-parallel and row-parallel linear layers, enabling the training of larger models.

Core Features & Use Cases

  • Tensor Parallelism Implementation: Provides guidance for ColumnParallelLinear and RowParallelLinear classes.
  • Distributed Training Support: Essential for splitting weights and activations across multiple devices/processes using torch.distributed.
  • Use Case: When building large language models that exceed single-GPU memory, this skill helps correctly shard model weights and manage communication for efficient distributed training.

Quick Start

Implement the ColumnParallelLinear class in PyTorch following the provided guidance.

Frequently Asked Questions about torch-tensor-parallelism

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I implement tensor parallelism in PyTorch for distributed linear layers?▼

Implement tensor parallelism in PyTorch by constructing ColumnParallelLinear and RowParallelLinear classes. This approach splits model weights and manages activation distribution across multiple devices using torch.distributed.

What is the difference between column-parallel and row-parallel linear layers in model parallelism?▼

Model parallelism splits neural network layers across devices: column-parallel divides weight matrices by columns, while row-parallel divides them by rows. Both manage weight splitting and bias handling differently across distributed ranks.

When do I need tensor parallelism instead of standard PyTorch distributed training?▼

You need tensor parallelism when building large language models that exceed single-GPU memory. It enables efficient distributed training by sharding model weights and managing communication for linear layers across multiple processes.

Why does my PyTorch distributed linear layer fail due to incorrect weight splitting?▼

Incorrect weight splitting causes distributed linear layer failures because tensor parallelism requires precise management of weight and activation splitting across ranks. Critical pitfalls include incorrect sharding and improper bias handling in distributed settings.

How do I use torch.distributed to manage weight and activation splitting across ranks?▼

Use torch.distributed to manage weight and activation splitting by initializing process groups and executing communication collectives. This coordinates tensor parallelism across ranks, ensuring correct weight distribution and bias handling for distributed linear layers.

Related Skills

Explore More Software Engineering Skills→
davpatel605-beepdavpatel605-beep

distributed-llm-pretraining-torchtitan

Configure distributed LLM pretraining with PyTorch parallelism and optimization techniques.

Community
Advanced
lazyFrogLOLlazyFrogLOL
127

torch-tensor-parallelism

Implement tensor-parallel linear layers in PyTorch with sharded weights and bias handling.

Community
Intermediate
icyzhicyzh
1

pytorch-fsdp

Implement fully sharded data-parallel training for PyTorch models.

Community
Advanced
Supporter09Supporter09

distributed-llm-pretraining-torchtitan

Scale LLM pretraining with PyTorch-native distributed training and 4D parallelism.

Community
Advanced