skills.rest
Rachasumanth
RachasumanthCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

inference-deploy

Deploy trained machine learning models for inference using vLLM, TGI, Ollama, and llama.cpp.

Updated Feb 17, 2026
Search Tags:#api#docker#model deployment#quantization#inference#ollama#vllm
One-click install
npx skills add https://github.com/Rachasumanth/text2llm001 --skill inference-deploy
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: inference-deploy
Source: https://github.com/Rachasumanth/text2llm001/tree/main/workspace/skills/inference-deploy
Command: npx skills add https://github.com/Rachasumanth/text2llm001 --skill inference-deploy

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires vllm, text-generation-inference, huggingface_hub, auto-gptq, autoawq, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill streamlines the process of deploying trained machine learning models for inference, making them accessible for real-time predictions and applications.

Core Features & Use Cases

  • Multiple Serving Frameworks: Supports vLLM, TGI, Ollama, and llama.cpp for flexible deployment.
  • Quantization: Enables model optimization through various quantization techniques (GGUF, GPTQ, AWQ).
  • API Endpoint Setup: Configures OpenAI-compatible API endpoints for easy integration.
  • Containerization: Generates Docker configurations for reproducible deployments.
  • Performance Validation: Includes load testing to benchmark inference speed and throughput.
  • Use Case: Deploy a fine-tuned LLM for a customer support chatbot using vLLM for high throughput and an OpenAI-compatible API.

Quick Start

Use the inference-deploy skill to deploy the model located at '/models/my-llm' using vLLM.

Frequently Asked Questions about inference-deploy

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy a fine-tuned LLM for inference using vLLM?▼

Deploy a fine-tuned LLM using vLLM by generating Docker configurations and setting up OpenAI-compatible API endpoints for high-throughput, scalable model serving in production environments.

What is the best way to quantize models for deployment with GPTQ and AWQ?▼

Model quantization for deployment uses GPTQ and AWQ techniques to optimize model size and inference speed. This process reduces memory requirements, enabling efficient deployment of large language models across serving frameworks.

Can I set up an OpenAI-compatible API endpoint with Ollama or TGI?▼

Yes, you can set up OpenAI-compatible API endpoints with Ollama or TGI. This Skill configures serving frameworks to facilitate easy integration of deployed machine learning models into applications.

Does this model deployment approach support Docker containerization?▼

Yes, this model deployment approach supports Docker containerization. It generates Docker configurations to ensure reproducible deployments of quantized models across environments using frameworks like vLLM and llama.cpp.

How do I benchmark inference speed and throughput after deploying a model?▼

Benchmark inference speed and throughput after deploying a model by running included performance validation and load testing scripts. This validates that vLLM or TGI deployments meet required latency and scalability targets.

Related Skills

Explore More Software Engineering Skills→
PeterHddPeterHdd
10

engineering-ml-engineer

Develop machine learning models with PyTorch, HuggingFace Transformers, scikit-learn, and XGBoost.

Community
Advanced
ax-llmax-llm
2.9k

ax-rust-ai

Generate Rust AxAI provider client code for axllm workflows.

Official
Intermediate
pluginagentmarketplacepluginagentmarketplace
2

model-deployment

Deploy LLMs to production with vLLM, TGI, and Ollama configurations.

Community
Advanced
davpatel605-beepdavpatel605-beep

llama-cpp

Configure llama.cpp deployments and select GGUF quantizations for local inference.

Community
Advanced