skills.rest
AvaTar-ArTs
AvaTar-ArTsCOMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Runs local GGUF model inference and discovers Hugging Face repos for llama.cpp.

Updated Jul 10, 2026
Search Tags:#quantization#edge deployment#gguf#llama.cpp#local inference#hugging face#llama-server
One-click install
npx skills add https://github.com/AvaTar-ArTs/.Agent-skills --skill llama-cpp-avatar-arts
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/AvaTar-ArTs/.Agent-skills/tree/main/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/AvaTar-ArTs/.Agent-skills --skill llama-cpp-avatar-arts

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, and includes references (resource) components.

What problem does it solve? Running large language models locally requires choosing the right GGUF quantization, finding compatible models on Hugging Face, and configuring llama.cpp for your hardware, which involves many manual steps and platform-specific details. ## Core Features & Use Cases - Local GGUF Inference: Run models on CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs using llama-cli, llama-server, or the llama-cpp-python bindings with chat, streaming, and embedding support. - Hugging Face Model Discovery: Search the Hub for llama.cpp-compatible repos, extract exact GGUF filenames and sizes via the tree API, and build ready-to-run llama-server commands. - Quantization Guidance: Choose between Q4_K_M, Q5_K_M, Q6_K, IQ variants, and repo-specific labels based on RAM/VRAM budgets and quality tradeoffs. - Use Case: A developer wants to run a 24B-parameter model on a laptop with 16 GB of VRAM. The skill finds a suitable GGUF repo on Hugging Face, recommends the exact quant that fits, and outputs the precise llama-server command to launch an OpenAI-compatible endpoint. ## Quick Start Ask the assistant to find a llama.cpp-compatible GGUF model on Hugging Face for your hardware and generate the llama-server command to run it locally.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a GGUF model locally with llama.cpp?▼

Install llama.cpp via brew, winget, or a CMake build, then run llama-cli or llama-server with the -hf flag pointing to a Hugging Face repo and quant, such as llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0. You can also load a local .gguf file path directly.

How do I find GGUF models on Hugging Face for llama.cpp?▼

Search huggingface.co/models with the apps=llama.cpp filter, optionally adding a search term and num_parameters bounds. Open the repo with ?local-app=llama.cpp for the recommended quant, and use the tree API to list exact .gguf filenames and sizes.

Which GGUF quantization should I choose for my RAM or VRAM?▼

Q4_K_M is the recommended balance of quality and size for general chat. Use Q5_K_M or Q6_K for code or technical work if memory allows, and Q3 or IQ variants only when fitting into tight RAM budgets matters more than quality.

Does llama-cpp-python support GPU acceleration with CUDA or Metal?▼

Yes. Install with CMAKE_ARGS="-DGGML_CUDA=on" for NVIDIA CUDA or "-DGGML_METAL=on" for Apple Silicon Metal, using --force-reinstall --no-cache-dir. Then set n_gpu_layers to control how many layers offload to the GPU.

Why does llama.cpp run out of memory during inference?▼

Out-of-memory errors occur when the model, context, and KV cache exceed available RAM or VRAM. Reduce n_gpu_layers, lower the context size with -c, use a smaller quantization like Q4_K_S, or quantize the KV cache with type_k and type_v settings.

Can llama-server provide an OpenAI-compatible API?▼

Yes. llama-server exposes /v1/chat/completions endpoints compatible with OpenAI clients, supporting streaming, parallel request slots, and continuous batching. Point your OpenAI client base_url to http://localhost:8080/v1 to connect.

Related Skills

Explore More Software Engineering Skills→
AlexKonceptAlexKoncept

llama-cpp

Locates GGUF files on Hugging Face Hub and launches a local llama.cpp inference server for testing.

Community
Advanced
sylin-orgsylin-org
4

koan-ai-integration

Integrate chat endpoints, embeddings, and RAG workflows with vector search.

Official
Advanced
NousResearchNousResearch
239.1k

serving-llms-vllm

Deploy high-throughput LLM inference servers with vLLM, PagedAttention, and quantization.

Official
Advanced
samuelmukotisamuelmukoti

llama-cpp

Run local LLM inference with llama.cpp and GGUF models on CPU or GPU.

Community
Advanced