skills.rest
Brilly-Bohyun
Brilly-BohyunCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Serve LLMs with vLLM using OpenAI-compatible endpoints and tensor parallelism.

Updated Jun 17, 2026
Search Tags:#quantization#OpenAI API#tensor parallelism#high throughput#LLM serving
One-click install
npx skills add https://github.com/Brilly-Bohyun/skill-repository --skill serving-llms-vllm-brilly-bohyun
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/Brilly-Bohyun/skill-repository/tree/main/mlops/inference/vllm
Command: npx skills add https://github.com/Brilly-Bohyun/skill-repository --skill serving-llms-vllm-brilly-bohyun

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, torch, transformers, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill solves the problem of deploying production LLM APIs with high throughput, optimizing inference latency, and serving models with limited GPU memory. It supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.

Core Features & Use Cases

  • High Throughput LLM Serving: Achieve 24x higher throughput than standard transformers.
  • OpenAI API Compatibility: Supports OpenAI-compatible endpoints.
  • Quantization: Offers quantization options (GPTQ/AWQ/FP8) to optimize memory usage.
  • Tensor Parallelism: Supports tensor parallelism for larger models.
  • Use Case: Deploy a high-performance LLM serving solution for production environments, optimizing inference latency and throughput while handling large models with limited GPU memory.

Quick Start

Use the vLLM skill to serve the 'meta-llama/Llama-3-8B-Instruct' model with tensor parallelism on your local machine.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy high throughput LLM serving for production environments?▼

Deploy high throughput LLM serving by using vLLM to optimize inference latency and achieve up to 24x higher throughput than standard transformers. It supports OpenAI-compatible endpoints, quantization, and tensor parallelism for production environments.

Can I use OpenAI API endpoints with my locally hosted large language models?▼

Yes, you can use OpenAI API endpoints with locally hosted models. This serving solution supports OpenAI-compatible endpoints, allowing you to integrate your local large language models seamlessly into existing API workflows.

What is the best way to serve large models with limited GPU memory?▼

The best way to serve large models with limited GPU memory is applying quantization techniques like GPTQ, AWQ, or FP8. These options optimize memory usage and enable efficient inference on hardware-constrained environments.

How does tensor parallelism help with LLM serving?▼

Tensor parallelism helps LLM serving by distributing large language models across multiple GPUs. This allows you to handle larger models that exceed a single GPU's memory capacity while maintaining optimized inference latency.

Does vllm require torch and transformers dependencies to work?▼

Yes, vLLM requires torch and transformers dependencies to work. These foundational frameworks provide the necessary model loading and tensor manipulation capabilities required to execute high throughput inference and quantization.

Related Skills

Explore More Software Engineering Skills→
huggingfacehuggingface
10.9k

huggingface-community-evals

Evaluate Hugging Face Hub models locally with inspect-ai and lighteval.

Official
Intermediate
superfhpsuperfhp

serving-llms-vllm

Serve LLMs with OpenAI-compatible endpoints using vLLM.

Community
Advanced
moltis-orgmoltis-org
2.8k

serving-llms-vllm

Deploy a vLLM-based LLM serving stack with OpenAI-compatible endpoints.

Official
Advanced
unclehowellunclehowell

serving-llms-vllm

Serve LLMs with high throughput and low latency using vLLM and PyTorch on GPU clusters.

Community
Advanced