skills.rest
Chensihakniroth
ChensihaknirothCOMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Runs local GGUF model inference and discovers llama.cpp-compatible models on Hugging Face Hub.

Updated Jun 7, 2026
Search Tags:#GGUF#quantization#Hugging Face#llama.cpp#local inference#llama-server#gpu offload
One-click install
npx skills add https://github.com/Chensihakniroth/ANAKOT-AGENT --skill llama-cpp-chensihakniroth
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/Chensihakniroth/ANAKOT-AGENT/tree/main/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/Chensihakniroth/ANAKOT-AGENT --skill llama-cpp-chensihakniroth

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, and includes references (resource) components.

What problem does it solve? Running large language models locally requires choosing the right GGUF quantization, finding compatible model files on Hugging Face, and configuring llama.cpp for your hardware, which involves many manual steps and platform-specific details. ## Core Features & Use Cases - Local GGUF Inference: Run models on CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs using llama-cli, llama-server, or the llama-cpp-python bindings. - Hugging Face Hub Discovery: Search for llama.cpp-compatible repos, extract exact GGUF filenames and sizes via the tree API, and build launch commands directly from Hub URLs. - Quantization Guidance: Choose between Q4/Q5/Q6/IQ variants based on RAM or VRAM constraints, with repo-specific labels preserved. - Use Case: A developer wants to run a 24B-parameter model on a laptop with 16 GB of RAM. Use this Skill to find a suitable GGUF repo on Hugging Face, pick the right quant, and generate the exact llama-server command. ## Quick Start Ask the agent to find a llama.cpp-compatible GGUF model for a given Hugging Face repo and generate the llama-server command to run it locally.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a GGUF model locally with llama.cpp?▼

Install llama.cpp via brew, winget, or a CMake build, then run llama-cli or llama-server with the -hf flag pointing to a Hugging Face repo and quant, such as llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0.

How to find GGUF files for a Hugging Face model repo?▼

Query the tree API at https://huggingface.co/api/models/<repo>/tree/main?recursive=true and keep entries where type is file and path ends with .gguf. The path and size fields give exact filenames and byte sizes.

What GGUF quantization should I use for my RAM or VRAM?▼

Start with Q4_K_M for general chat, Q5_K_M or Q6_K for code and technical work, and Q3 or IQ variants only for tight memory budgets. Prefer the exact quant labels shown in the repo's Hardware compatibility section.

Does llama-cpp-python support CUDA and Metal GPU acceleration?▼

Yes, install with CMAKE_ARGS="-DGGML_CUDA=on" for CUDA or CMAKE_ARGS="-DGGML_METAL=on" for Apple Silicon Metal, using pip install llama-cpp-python --force-reinstall --no-cache-dir. Set n_gpu_layers to control GPU offloading.

Why does llama.cpp run out of memory during inference?▼

Out of memory happens when too many layers are offloaded to GPU, the context is too large, or the quantization is too big for available RAM. Reduce n_gpu_layers, lower n_ctx, or switch to a smaller quant like Q4_K_S.

Can llama-server provide an OpenAI-compatible API?▼

Yes, llama-server exposes /v1/chat/completions endpoints compatible with OpenAI clients. Point your client's base_url to http://localhost:8080/v1 and send standard chat completion requests with optional streaming.

Related Skills

Explore More Software Engineering Skills→
faramirezsfaramirezs

llama-cpp

Automate local GGUF inference and model discovery for llama.cpp.

Community
Advanced
sylin-orgsylin-org
4

koan-ai-integration

Integrate chat endpoints, embeddings, and RAG workflows with vector search.

Official
Advanced
FutureMLS-LabFutureMLS-Lab
556

sglang-diffusion-performance

Select SGLang Diffusion performance flags for generation speed, VRAM, and quality.

Official
Advanced
cuba6112cuba6112

unsloth-gguf

Export fine-tuned machine learning models to GGUF format with quantization and LoRA merging.

Community
Intermediate