skills.rest
hsliuustc0106
hsliuustc0106COMMUNITY
Explore all skills by this author →
WebsiteGitHub

vllm-omni-distributed

Distribute large models across GPUs or nodes with tensor and pipeline parallelism.

84|27|Updated Mar 3, 2026
Search Tags:#ray#distributed-inference#multinode#tensor-parallelism#pipeline-parallelism#gpu-cluster#omniconnector
One-click install
npx skills add https://github.com/hsliuustc0106/vllm-omni-skills --skill vllm-omni-distributed
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: vllm-omni-distributed
Source: https://github.com/hsliuustc0106/vllm-omni-skills/tree/main/skills/vllm-omni-distributed
Command: npx skills add https://github.com/hsliuustc0106/vllm-omni-skills --skill vllm-omni-distributed

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Distributes large inference workloads across multiple GPUs or machines, enabling scalable deployments and efficient resource utilization.

Core Features & Use Cases

  • Tensor Parallelism (TP): Split model weights across GPUs to reduce latency and improve throughput.
  • Pipeline Parallelism (PP): Divide the model across sequential GPU groups to boost overall throughput.
  • Disaggregation / OmniConnector: Run Encode, Prefill, Decode, and Generate stages on separate GPU pools for independent scaling.
  • Multi-node with Ray: Orchestrate distributed serving across a Ray cluster for larger deployments.
  • Sequence Parallelism: Accelerate diffusion-based generation by splitting steps across GPUs.

Quick Start

Launch a multi-node Ray cluster and start the vLLM-Omni server with your model and desired tensor- and pipeline-parallel settings.

Frequently Asked Questions about vllm-omni-distributed

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scale distributed inference across multiple GPUs and nodes?▼

Tensor parallelism splits model weights across multiple GPUs to reduce latency and improve inference throughput, while pipeline parallelism divides the model across sequential GPU groups to boost overall processing capacity.

How do I deploy a multi-node vLLM cluster using Ray?▼

You can deploy multi-node inference by launching a Ray cluster and starting the vLLM server with your model, configuring tensor-parallel and pipeline-parallel sizes to orchestrate distributed serving across the connected machines.

Can I run prefill and decode stages on separate GPU pools?▼

Yes, you can use OmniConnector-based disaggregation to run Encode, Prefill, Decode, and Generate stages on separate GPU pools, enabling independent scaling and efficient resource utilization for mixed workloads.

Does this support sequence parallelism for diffusion-based generation?▼

Sequence parallelism accelerates diffusion-based generation by splitting sequential steps across multiple GPUs, increasing processing speed for diffusion models.

What is the best way to configure parallelism sizes for a multi-GPU deployment?▼

Configuring parallelism involves setting tensor-parallel and pipeline-parallel sizes when starting the server, allowing you to balance latency and throughput based on your available GPU pools and model architecture.

Related Skills

Explore More Software Engineering Skills→
Devsoul2026Devsoul2026
150

lambda-labs-gpu-cloud

Launch and manage Lambda Labs GPU clusters for distributed ML training.

Community
Advanced
thisismynewfmail-uithisismynewfmail-ui

tensorrt-llm

Optimizes LLM inference with NVIDIA TensorRT-LLM on A100/H100 GPUs.

Community
Advanced
ovachieverovachiever
52

ray-train

Orchestrate distributed training across clusters with Ray Train and Ray Tune.

Community
Advanced
carterwayneskhizeinecarterwayneskhizeine
19

distributed-llm-pretraining-torchtitan

Orchestrate PyTorch-native 4D parallelism for distributed LLM pretraining with torchtitan.

Community
Advanced