model-serving

Name: model-serving
Availability: InStock
Author: MaserMediaGroup

Deploy LLM and ML models for production inference with vLLM, BentoML, and streaming APIs.

1|Updated Feb 24, 2026

Search Tags:#fastapi #rag #kubernetes #streaming #inference #vllm #llm serving

Ready to run

npx skills add https://github.com/masermediagroup-stack/maser-media --skill model-serving-masermediagroup-stack

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: model-serving
Source: https://github.com/masermediagroup-stack/maser-media/tree/main/.cursor/skills/community/ai-design-components/skills/model-serving
Command: npx skills add https://github.com/masermediagroup-stack/maser-media --skill model-serving-masermediagroup-stack

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires aiohttp, numpy, and includes scripts (resource) and references (resource) components.

What problem does it solve? Deploying LLMs and ML models to production requires choosing serving engines, configuring GPU memory, implementing streaming responses, and setting up monitoring, which involves many error-prone decisions and boilerplate configuration. ## Core Features & Use Cases - Serving Engine Selection: Decision frameworks for vLLM, TensorRT-LLM, Ollama, BentoML, and Triton based on throughput, GPU, and model-type requirements. - Streaming Inference APIs: FastAPI Server-Sent Events patterns for token-by-token LLM responses with React frontend integration. - Production Deployment: Kubernetes manifests with GPU resources, HPA autoscaling, Kong API gateway rate limiting, and Prometheus/Grafana monitoring. - Use Case: Self-host Llama-3.1-8B with vLLM behind an OpenAI-compatible API, stream tokens to a chat frontend via SSE, and benchmark throughput with the included script before scaling on Kubernetes. ## Quick Start Set up a vLLM server for Llama-3.1-8B with a FastAPI streaming endpoint and run the benchmark script to measure throughput and latency.

Frequently Asked Questions about model-serving

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I serve a Llama model with vLLM?▼

Install vLLM with pip and run vllm serve with the model ID, setting dtype, max-model-len, and gpu-memory-utilization parameters. It exposes an OpenAI-compatible API on port 8000 that works with the standard OpenAI Python client.

vLLM vs TensorRT-LLM for LLM inference?▼

vLLM offers PagedAttention memory management and continuous batching for 20-30x throughput with simple setup. TensorRT-LLM is 2-8x faster but requires model conversion and optimization, suiting workloads needing maximum GPU efficiency.

How much GPU memory does Llama-3.1-8B need?▼

Llama-3.1-8B in FP16 requires about 19.2 GB (8B parameters x 2 bytes x 1.2 overhead). INT4 quantization with AWQ reduces this roughly 4x, while 70B models in FP16 need around 168 GB across multiple GPUs.

How do I stream LLM responses to a React frontend?▼

Use FastAPI StreamingResponse with text/event-stream media type, yielding SSE-formatted JSON tokens from the vLLM stream. On the frontend, read the response body with getReader and parse lines prefixed with data: to append tokens.

Why is my vLLM server running out of GPU memory?▼

Reduce max-model-len, lower gpu-memory-utilization to 0.8, or enable AWQ quantization to cut memory usage. If the model still does not fit, use tensor parallelism across multiple GPUs or switch to a smaller variant.

When should I use BentoML instead of vLLM?▼

Use BentoML for traditional ML models like scikit-learn, XGBoost, or PyTorch classifiers, where its adaptive batching improves throughput. vLLM is designed specifically for LLM text generation and does not serve conventional ML models.