skills.rest
jayll1303
jayll1303COMMUNITY
Explore all skills by this author →
GitHub

llama-cpp-inference

Run local GGUF model inference and OpenAI-compatible serving with llama.cpp.

17|3|Updated Apr 3, 2026
Search Tags:#inference#gguf#llama.cpp#gpu-offload#continuous-batching#llama-server#llama-cpp-python
One-click install
npx skills add https://github.com/jayll1303/AIEKit --skill llama-cpp-inference
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp-inference
Source: https://github.com/jayll1303/AIEKit/tree/main/.kiro/skills/llama-cpp-inference
Command: npx skills add https://github.com/jayll1303/AIEKit --skill llama-cpp-inference

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill removes the complexity of running GGUF-format large language models locally by providing clear patterns for launching servers, using CLIs, embedding Python bindings, configuring GPU offload, and tuning multi-user batching so models run reliably and efficiently on desktop and server hardware.

Core Features & Use Cases

  • OpenAI-compatible serving: Launch llama-server with health checks, embeddings, chat/completion endpoints and slot monitoring for multi-user deployments.
  • CLI and programmatic inference: Use llama-cli for interactive/batch runs and llama-cpp-python bindings to embed GGUF models inside Python applications.
  • GPU and backend configuration: Build and run with CUDA, Metal, or Vulkan backends, set n_gpu_layers, flash-attn, and tune context and parallel slots for throughput or low-memory setups.
  • Performance tuning & troubleshooting: Guidance for context sizing, continuous batching, speculative decoding, OOM recovery, CUDA detection, and model format validation.
  • Use Case: Deploy a local OpenAI-compatible GGUF server for development, perform low-latency interactive chat on a laptop with Metal/CUDA, or run multi-user inference with continuous batching in a small team.

Quick Start

Launch a local GGUF model server using llama-server with the model path, appropriate context size, and -ngl set to control GPU offload.

Frequently Asked Questions about llama-cpp-inference

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a local GGUF model server with OpenAI-compatible API endpoints?▼

Run local GGUF model inference and OpenAI-compatible serving using llama-server. Configure model paths, context size, and n_gpu_layers to launch chat completions, embeddings, and health check endpoints for local development.

How do I configure GPU offload for llama.cpp to prevent out of memory errors?▼

Configure GPU offload by adjusting the n_gpu_layers parameter in llama.cpp to balance VRAM usage and prevent out of memory errors. This Skill guides tuning context size, flash-attn, and parallel slots for low-memory setups or maximum throughput.

Does llama-cpp-python support continuous batching for multi-user inference?▼

Yes, llama-cpp-python and llama-server support continuous batching for multi-user inference. This Skill provides guidance on configuring parallel slots and context sizing to handle concurrent requests efficiently in small team deployments.

Can I use llama.cpp with Metal or Vulkan backends instead of CUDA?▼

Yes, llama.cpp supports Metal and Vulkan backends in addition to CUDA. This Skill covers build options and runtime configuration for these backends, enabling low-latency interactive chat on laptops or systems without NVIDIA GPUs.

What is the best way to embed GGUF models inside a Python application?▼

The best way to embed GGUF models inside a Python application is using llama-cpp-python bindings. This Skill provides implementation patterns for programmatic inference, embedding generation, and configuring hardware acceleration within your codebase.

Why does llama-server fail to detect my CUDA GPU during local inference?▼

llama-server may fail to detect CUDA GPUs due to incorrect build options or format validation issues. This Skill offers troubleshooting for CUDA detection, OOM recovery, and format validation to ensure hardware acceleration works correctly.

Related Skills

Explore More Software Engineering Skills→
fikriaffikriaf

llama-cpp

Automate discovery, download, and local inference of llama.cpp models from Hugging Face Hub.

Community
Advanced
copilot-ldcopilot-ld
1

libllm

Expose an HTTP LLM API client for OpenAI-compatible chat completions and embeddings.

Official
Advanced
Supporter09Supporter09

serving-llms-vllm

Serve large language models with vLLM high-throughput inference and OpenAI-compatible endpoints.

Community
Advanced
guccangguccang
3

llama-cpp

Select and deploy GGUF models for local llama.cpp inference.

Community
Advanced