skills.rest
davpatel605-beep
davpatel605-beepCOMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Configure llama.cpp deployments and select GGUF quantizations for local inference.

Updated Jun 25, 2026
Search Tags:#quantization#gguf#llama.cpp#model serving#local inference#hugging face#llama-cpp-python
One-click install
npx skills add https://github.com/davpatel605-beep/hermusagent --skill llama-cpp-davpatel605-beep
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/davpatel605-beep/hermusagent/tree/main/backend/vendor/hermes/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/davpatel605-beep/hermusagent --skill llama-cpp-davpatel605-beep

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps users run local large language models efficiently by simplifying GGUF model discovery, quant selection, and llama.cpp inference setup across different hardware environments.

Core Features & Use Cases

  • Local Model Inference: Configure llama.cpp and llama-cpp-python workflows for CPU, Apple Silicon, CUDA, ROCm, and Intel GPU deployments.
  • GGUF Discovery and Optimization: Find Hugging Face llama.cpp-compatible models, select suitable quantizations, inspect GGUF files, and build launch commands.
  • Use Case: A developer with limited GPU memory can use this Skill to identify the right GGUF quantization for a model and deploy an OpenAI-compatible local inference server.

Quick Start

Use the llama-cpp skill to find a suitable GGUF model from Hugging Face and generate the correct llama-server command for my hardware.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run local GGUF model inference using llama.cpp?▼

To run local GGUF model inference, configure llama.cpp workflows by discovering compatible Hugging Face models, selecting appropriate quantizations for your hardware, and generating llama-server launch commands.

What's the best way to choose a GGUF quantization for limited GPU memory?▼

Choosing a GGUF quantization for limited GPU memory involves analyzing hardware-specific constraints and selecting a compatible model from Hugging Face to ensure efficient local inference deployment.

How do I set up an OpenAI-compatible local inference server with llama-cpp-python?▼

Setting up an OpenAI-compatible local inference server with llama-cpp-python requires configuring llama.cpp deployments and generating correct launch commands tailored to your specific hardware environment.

Does llama.cpp support inference across different hardware like Apple Silicon and CUDA?▼

Yes, llama.cpp supports local model inference across diverse hardware environments including Apple Silicon, CUDA, ROCm, Intel GPU, and CPU deployments for flexible local AI serving.

How do I find compatible GGUF models on Hugging Face for local serving?▼

Finding compatible GGUF models on Hugging Face involves using discovery techniques to identify llama.cpp-compatible models and inspecting GGUF files to ensure they meet your local serving requirements.

What is GGUF quantization and when do I need it for local AI serving?▼

GGUF quantization compresses large language models to reduce memory requirements, needed when deploying local AI serving on hardware with limited VRAM to maintain inference performance.

Related Skills

Explore More Software Engineering Skills→
DonZzzillaDonZzzilla

llama-cpp

Run local GGUF inference for llama.cpp models with Hugging Face Hub discovery.

Community
Advanced
ovachieverovachiever
52

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM for high-throughput inference.

Community
Advanced
SignmanalSignmanal
1

llama-cpp

Run local GGUF model inference with llama.cpp across CPU and GPU hardware.

Community
Intermediate
GitnappGitnapp

serving-llms-vllm

Deploy large language model inference services with vLLM and OpenAI-compatible endpoints.

Community
Advanced