skills.rest
loteiron
loteironCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

llama-cpp

Run local GGUF inference and discover Hugging Face models for llama.cpp.

Updated Sep 10, 2026
Search Tags:#GGUF#quantization#edge deployment#Hugging Face#llama.cpp#local inference#llama-server
One-click install
npx skills add https://github.com/loteiron/ZeusAgent --skill llama-cpp-loteiron
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/loteiron/ZeusAgent/tree/main/optional-skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/loteiron/ZeusAgent --skill llama-cpp-loteiron

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, and includes references (resource) components.

What problem does it solve? Running large language models locally requires choosing the right GGUF quantization, finding compatible models on Hugging Face, and configuring llama.cpp for your hardware, which involves many manual steps and platform-specific details. ## Core Features & Use Cases - Local GGUF Inference: Run models on CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs using llama-cli, llama-server, or the llama-cpp-python bindings. - Hugging Face Model Discovery: Search the Hub for llama.cpp-compatible repos, enumerate available .gguf files via the tree API, and build exact llama-server commands from URL-only workflows. - Quantization Guidance: Choose between Q4/Q5/Q6/IQ variants based on RAM or VRAM budgets, with repo-specific labels like UD-Q4_K_M preserved exactly. - Use Case: A developer wants to run a Qwen model on a laptop with 16 GB RAM. Use this Skill to search the Hub, identify the recommended quant and exact filename, and generate a working llama-server command. ## Quick Start Ask the agent to find a llama.cpp-compatible GGUF model on Hugging Face for a given model family and generate the llama-server command to run it locally.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a GGUF model locally with llama.cpp?▼

Run a GGUF model locally using llama-cli or llama-server with the -hf flag pointing to a Hugging Face repo and quant, such as llama-server -hf repo:Q4_K_M. You can also load a local .gguf file with the llama-cpp-python bindings by passing model_path to the Llama class.

How to find GGUF models on Hugging Face for llama.cpp?▼

Search Hugging Face with the apps=llama.cpp filter, then open the repo with ?local-app=llama.cpp to see the recommended quant and command. Use the tree API endpoint to list exact .gguf filenames and byte sizes for the repo.

Which GGUF quantization should I use for my RAM?▼

For general chat, start with Q4_K_M, which balances quality and size. Prefer Q5_K_M or Q6_K for code or technical work if memory allows, and use Q3 or IQ variants only for very tight RAM budgets.

Does llama.cpp support Apple Silicon and NVIDIA GPUs?▼

llama.cpp supports Apple Silicon via Metal, NVIDIA GPUs via CUDA, plus ROCm and Intel GPUs. For Python bindings, build with CMAKE_ARGS="-DGGML_METAL=on" for Metal or "-DGGML_CUDA=on" for CUDA, and set n_gpu_layers to offload layers.

Why does llama-cpp-python fail to build with CUDA?▼

Build failures usually come from missing CUDA toolkit headers or build dependencies. Install cmake and scikit-build-core, then reinstall with CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python --force-reinstall --no-cache-dir.

Can llama.cpp serve an OpenAI-compatible API?▼

llama-server exposes an OpenAI-compatible /v1/chat/completions endpoint supporting streaming. Point an OpenAI client at http://localhost:8080/v1 and send standard chat completion requests.

Related Skills

Explore More Software Engineering Skills→
Tyler-R-KendrickTyler-R-Kendrick
1

huggingface-local-models

Search Hugging Face Hub and launch GGUF models via llama.cpp.

Community
Intermediate
qcmuuqcmuu

ollama

Run local LLM inference via REST API and CLI.

Community
Basic
FutureMLS-LabFutureMLS-Lab
556

sglang-diffusion-performance

Select SGLang Diffusion performance flags for generation speed, VRAM, and quality.

Official
Advanced
AevonixAevonix

gguf-quantization

Convert models to GGUF and apply bit-depth quantization for local inference.

Official
Intermediate