skills.rest
drdave-flexnetos
drdave-flexnetosCOMMUNITY
Explore all skills by this author →
GitHub

AI Inference & Model Serving

Deploy LocalAI and vLLM servers for OpenAI-compatible chat and embeddings.

Updated Jan 12, 2026
Search Tags:#vllm#gguf#llama.cpp#model serving#localai#openai-compatible api
One-click install
npx skills add https://github.com/drdave-flexnetos/ripple-env --skill ai-inference-model-serving-drdave-flexnetos
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: AI Inference & Model Serving
Source: https://github.com/drdave-flexnetos/ripple-env/tree/main/.claude/skills/inference
Command: npx skills add https://github.com/drdave-flexnetos/ripple-env --skill ai-inference-model-serving-drdave-flexnetos

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

It solves the problem of running large language models locally and exposing them via an HTTP API for chat, completions, and embeddings without relying on external hosted services.

Core Features & Use Cases

  • LocalAI + OpenAI-Compatible API: Deploy models behind a local endpoint (including GGUF-based backends) so your apps can call standard /v1/* routes.
  • vLLM OpenAI Server: Host Hugging Face models with high-throughput server mode and configurable sampling for interactive generation.
  • GGUF/GGML Workflows (llama.cpp): Manage quantized model formats and run efficient inference pipelines using llama.cpp for CPU/GPU offload tuning.

Use case example: You need an offline-capable assistant for a robotics or research workflow and want a consistent OpenAI-style API regardless of whether you use LocalAI (GGUF) or vLLM (HF models).

Quick Start

Start a LocalAI container that serves your GGUF-backed model by mapping your models/ directory into the container and publishing port 8080 to access http://localhost:8080/v1/chat/completions.

Frequently Asked Questions about AI Inference & Model Serving

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve local LLMs with an OpenAI-compatible API?▼

To serve local LLMs with an OpenAI-compatible API, you can deploy LocalAI or vLLM servers to expose standard /v1/* HTTP routes for chat, completions, and embeddings without relying on external hosted services.

What is the best way to run GGUF quantized models locally?▼

The best way to run GGUF quantized models locally is by using LocalAI with llama.cpp backends, which allows efficient inference pipelines and CPU/GPU offload tuning while exposing a local OpenAI-compatible API endpoint.

Does vLLM support hosting Hugging Face models for high-throughput inference?▼

Yes, vLLM supports hosting Hugging Face models by utilizing its high-throughput server mode with configurable sampling, providing an OpenAI-compatible HTTP interface for interactive text generation pipelines.

Can I use LocalAI to generate embeddings locally?▼

Yes, you can use LocalAI to generate embeddings locally by configuring model backends behind a local HTTP endpoint, allowing your applications to call standard OpenAI-compatible /v1/* routes.

How do I configure CPU and GPU offload settings for llama.cpp workflows?▼

You configure CPU and GPU offload settings for llama.cpp workflows by managing quantized GGUF/GGML model formats within LocalAI, tuning runtime constraints to execute efficient local inference pipelines.

Related Skills

Explore More Software Engineering Skills→
evalstateevalstate

trl

Submit TRL training jobs on Hugging Face Jobs and convert models to GGUF.

Community
Advanced
hsliuustc0106hsliuustc0106
84

vllm-omni-api

Bridge client applications with the vLLM-Omni API using OpenAI-compatible endpoints.

Community
Intermediate
datathingsdatathings
10

llamacpp

Reference the llama.cpp C API for model loading and inference workflows.

Official
Advanced
huggingfacehuggingface
10.9k

huggingface-community-evals

Evaluate Hugging Face Hub models locally with inspect-ai and lighteval.

Official
Intermediate