skills.rest
xu1713
xu1713COMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Run local GGUF inference and discover Hugging Face models for llama.cpp.

Updated Jun 5, 2026
Search Tags:#quantization#edge deployment#gguf#llama.cpp#local inference#hugging face#llama-server
One-click install
npx skills add https://github.com/xu1713/openhorse --skill llama-cpp-xu1713
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/xu1713/openhorse/tree/main/openhorse/openhorse/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/xu1713/openhorse --skill llama-cpp-xu1713

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, and includes references (resource) components.

What problem does it solve? Running large language models locally requires choosing the right GGUF quantization, finding compatible model files on Hugging Face, and configuring llama.cpp for your hardware, which involves fragmented documentation and trial-and-error. ## Core Features & Use Cases - Hub Model Discovery: Search Hugging Face for llama.cpp-compatible repos, read local-app pages for recommended quants, and enumerate exact GGUF files via the tree API. - Local Inference: Run models with llama-cli, llama-server, or llama-cpp-python on CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs. - Quant Selection Guidance: Choose between Q4_K_M, Q5_K_M, Q6_K, IQ variants, and repo-specific labels based on RAM or VRAM constraints. - Use Case: A developer wants to run a 35B model on a 24GB GPU. Use this Skill to find the repo's hardware compatibility section, pick UD-Q4_K_M at 22.1 GB, and generate the exact llama-server command. ## Quick Start Ask the assistant to find a GGUF version of a Hugging Face model and generate the llama-server command to run it locally.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a Hugging Face model locally with llama.cpp?▼

Use llama-cli or llama-server with the -hf flag pointing to a repo and quant, such as llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0. For exact files, use --hf-repo and --hf-file with the filename from the repo's tree API.

Which GGUF quantization should I choose for my hardware?▼

Start with Q4_K_M for general chat, Q5_K_M or Q6_K for code and technical work, and Q3 or IQ variants for tight RAM budgets. Prefer the exact quant labels shown in the Hugging Face local-app page's hardware compatibility section over generic tables.

Does llama.cpp work on Apple Silicon and AMD GPUs?▼

Yes, llama.cpp supports Apple Silicon via Metal, NVIDIA GPUs via CUDA, AMD GPUs via ROCm, and Intel GPUs. For Python bindings on macOS, install with CMAKE_ARGS="-DGGML_METAL=on" pip install llama-cpp-python.

How do I list all GGUF files in a Hugging Face repo?▼

Query the tree API at https://huggingface.co/api/models/<repo>/tree/main?recursive=true and filter entries where type is file and path ends with .gguf. Separate main quantized checkpoints from mmproj projector files and BF16 shards.

Why does llama-cpp-python fail to build with CUDA support?▼

The build fails when CUDA toolkit paths are missing or the wheel was compiled without GPU flags. Reinstall with CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python --force-reinstall --no-cache-dir and ensure nvcc is on your PATH.

What should I do when llama.cpp runs out of memory?▼

Reduce GPU offloaded layers with -ngl, lower the context size with -c, switch to a smaller quantization like Q4_K_S or Q3_K_M, or quantize the KV cache. On constrained hardware, start with few GPU layers and increase gradually.

Related Skills

Explore More Software Engineering Skills→
DarkArty07DarkArty07
3

llama-cpp

Deploy local LLM inference with GGUF quantized weights via llama.cpp.

Community
Advanced
TehminanazTehminanaz
1

openai-agents-sdk-groq-integration

Integrate the OpenAI Agents SDK with Groq's endpoint in FastAPI applications.

Community
Advanced
agentic-inagentic-in
577

Inference

Select inference paths, serving stacks, and benchmarking plans for ML models.

Official
Intermediate
mahg-esmahg-es
3

llm-local-deploy

Deploy local LLMs with llama.cpp, Ollama, or vLLM via docker-compose.

Official
Intermediate