skills.rest
TitoPrausee
TitoPrauseeCOMMUNITY
Explore all skills by this author →
GitHub

serving-llms-vllm

Deploy vLLM for high-throughput LLM serving with OpenAI-compatible APIs.

Updated Jul 7, 2026
Search Tags:#monitoring#kubernetes#quantization#inference#vllm#tensor parallelism#llm serving
One-click install
npx skills add https://github.com/TitoPrausee/nexus-toti --skill serving-llms-vllm-titoprausee
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serving-llms-vllm
Source: https://github.com/TitoPrausee/nexus-toti/tree/main/data/skills/mlops/inference/vllm
Command: npx skills add https://github.com/TitoPrausee/nexus-toti --skill serving-llms-vllm-titoprausee

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps you deploy large language models with high throughput, low latency, and OpenAI-compatible APIs without having to design the serving stack from scratch.

Core Features & Use Cases

  • Production LLM serving: Run model endpoints for chatbots, assistants, and application backends with continuous batching and PagedAttention.
  • Performance tuning: Improve latency and throughput with prefix caching, speculative decoding, and GPU memory optimization.
  • Scaled deployments: Handle quantized models, tensor parallelism, Docker, Kubernetes, load balancing, and multi-node serving.
  • Use Case: A platform team can launch a model API for customer support, monitor request latency, and tune concurrency to keep GPU usage high during peak traffic.

Quick Start

Use this Skill to serve a chosen model with vLLM and enable the recommended production settings for your deployment.

Frequently Asked Questions about serving-llms-vllm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy LLMs for high-throughput production serving?▼

Scale LLM serving across multiple GPUs using tensor parallelism, or extend to multi-node deployments with Docker and Kubernetes to handle load balancing for high traffic volumes.

What is the best way to reduce LLM inference latency and improve throughput?▼

Deploy quantized LLMs using 4-bit or FP8 quantization to reduce GPU memory requirements while maintaining high-throughput inference and low-latency model serving.

Can I serve LLMs across multiple GPUs using tensor parallelism?▼

Scale LLM serving across multiple GPUs using tensor parallelism, or extend to multi-node deployments with Docker and Kubernetes to handle load balancing for high traffic volumes.

Does vLLM support quantized 4-bit and FP8 model deployments?▼

Deploy quantized LLMs using 4-bit or FP8 quantization to reduce GPU memory requirements while maintaining high-throughput inference and low-latency model serving.

How do I monitor LLM serving request latency in Kubernetes?▼

Monitor LLM serving request latency in Kubernetes by integrating Prometheus monitoring and health checks to track endpoint performance and tune concurrency during peak traffic.

Related Skills

Explore More Software Engineering Skills→
x-TheFoxx-TheFox

serving-llms-vllm

Serve LLMs via an OpenAI-compatible API using vLLM.

Community
Advanced
giuseppe-trisciuogliogiuseppe-trisciuoglio
322

spring-boot-actuator

Configure Spring Boot Actuator endpoints, health probes, and Micrometer metrics.

Community
Intermediate
overviewlabsoverviewlabs

serving-llms-vllm

Deploy and scale LLM inference with vLLM for low-latency production serving.

Community
Advanced
bailynlovebailynlove
1

serving-llms-vllm

Serve OpenAI-compatible LLM APIs with PagedAttention and continuous batching.

Community
Advanced