skills.rest
moltis-org
moltis-orgOFFICIAL
Explore all skills by this author →
WebsiteGitHub

serving-llms-vllm

Deploy a vLLM-based LLM serving stack with OpenAI-compatible endpoints.

2.8k|332|Updated Jan 29, 2026
Search Tags:#llm#gpu#quantization#inference#vllm#serving
One-click install
npx skills add https://github.com/moltis-org/moltis --skill serving-llms-vllm-moltis-org
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/moltis-org/moltis/tree/main/crates/skills/src/assets/mlops/inference/serving-llms-vllm
Command: npx skills add https://github.com/moltis-org/moltis --skill serving-llms-vllm-moltis-org

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Serves large language models at production scale by leveraging vLLM for high throughput, low latency, and memory-efficient inference across OpenAI-compatible endpoints, with support for quantization and tensor parallelism.

Core Features & Use Cases

  • High-throughput; OpenAI-compatible API endpoints
  • Model quantization options (AWQ, GPTQ, FP8) to fit large models into limited GPU memory
  • Tensor parallelism and multi-GPU deployment for large deployments
  • Use cases include production LLM APIs, enterprise chatbots, and research experiments at scale.

Quick Start

Run a vLLM server with your model to expose an OpenAI-compatible endpoint and test a prompt quickly.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve large language models with high throughput and low latency?▼

Serving large language models with high throughput and low latency is achieved by deploying a vLLM-based stack that exposes OpenAI-compatible endpoints and leverages memory-efficient inference for production scale.

Does vLLM support model quantization to fit large models into limited GPU memory?▼

vLLM supports model quantization to fit large models into limited GPU memory by offering AWQ, GPTQ, and FP8 options, reducing the memory footprint required for production deployments.

Can I use tensor parallelism for multi-GPU LLM deployment?▼

You can use tensor parallelism for multi-GPU LLM deployment in vLLM to distribute model inference across a compatible GPU cluster, enabling large-scale production deployments.

Do I need a compatible GPU cluster to deploy an OpenAI-compatible LLM serving endpoint?▼

A compatible GPU cluster is required to deploy an OpenAI-compatible LLM serving endpoint with vLLM, as the stack targets production deployments needing high throughput and memory efficiency for large models.

What is the best way to expose an OpenAI-compatible API for enterprise chatbots?▼

The best way to expose an OpenAI-compatible API for enterprise chatbots is deploying a vLLM serving stack, which handles high-throughput inference and basic configuration for memory utilization and parallelism.

Why does serving large models require memory-efficient inference and parallelism configuration?▼

Serving large models requires memory-efficient inference and parallelism configuration because vLLM must optimize GPU memory utilization and distribute workloads via tensor parallelism to maintain low latency during production deployments.

Related Skills

Explore More Software Engineering Skills→
zerwizzerwiz

tensorrt-llm

Optimizes LLM inference on NVIDIA GPUs via TensorRT-LLM configuration and deployment.

Community
Advanced
lxt070409lxt070409
1

serving-llms-vllm

Automate high-throughput LLM inference with vLLM and OpenAI-compatible endpoints.

Community
Advanced
cxnaivecxnaive

serving-llms-vllm

Deploy high-throughput LLM inference servers with the vLLM engine.

Community
Advanced
superfhpsuperfhp

serving-llms-vllm

Serve LLMs with OpenAI-compatible endpoints using vLLM.

Community
Advanced