skills.rest
CHENHUI-X
CHENHUI-XCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

llama-cpp

Run local GGUF inference and discover llama.cpp-compatible models on Hugging Face Hub.

Updated Jul 3, 2026
Search Tags:#quantization#edge deployment#gguf#llama.cpp#local inference#hugging face#llama-server
One-click install
npx skills add https://github.com/CHENHUI-X/toolbox --skill llama-cpp-chenhui-x
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/CHENHUI-X/toolbox/tree/main/custom-skills/inference/llama-cpp
Command: npx skills add https://github.com/CHENHUI-X/toolbox --skill llama-cpp-chenhui-x

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python>=0.2.0, and includes references (resource) components.

What problem does it solve? Running large language models locally requires choosing the right GGUF quantization, finding compatible model files on Hugging Face, and configuring llama.cpp for your hardware, which involves scattered documentation and trial-and-error. ## Core Features & Use Cases - Hub-First Model Discovery: Search Hugging Face for llama.cpp-compatible repos, read the local-app page for recommended quants, and confirm exact GGUF filenames and sizes via the tree API. - Local Inference: Run models with llama-cli, llama-server, or the llama-cpp-python bindings on CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs. - Quant Selection Guidance: Choose between Q4_K_M, Q5_K_M, Q6_K, IQ variants, and repo-specific labels based on RAM or VRAM constraints. - Use Case: A developer wants to run a Qwen model on a laptop with 16 GB RAM. The skill finds the right GGUF repo, identifies the UD-Q4_K_M quant with its exact file size, and produces a ready-to-run llama-server command. ## Quick Start Ask the assistant to find a llama.cpp-compatible GGUF for a Hugging Face model and generate the llama-server command for your hardware.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a GGUF model locally with llama.cpp?▼

Install llama.cpp via brew, winget, or a CMake build, then run llama-cli or llama-server with the -hf flag pointing to a Hugging Face repo and quant, such as llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0. You can also load a local .gguf file directly with the -m flag.

How do I find GGUF files for a Hugging Face model?▼

Open the repo with ?local-app=llama.cpp to see the recommended quant and launch snippet, then query the tree API at /api/models/<repo>/tree/main?recursive=true to list exact .gguf filenames and byte sizes. Separate main checkpoints from mmproj projector files and BF16 shards.

Which GGUF quantization should I choose for my RAM?▼

Q4_K_M is the recommended balance of quality and size for general chat, while Q5_K_M or Q6_K suit code and technical work if memory allows. For tight budgets consider Q3_K_M or IQ variants, and always prefer the exact quant label shown on the repo's Hugging Face local-app page.

Does llama-cpp-python support GPU acceleration?▼

Yes, install with CMAKE_ARGS="-DGGML_CUDA=on" for NVIDIA CUDA or "-DGGML_METAL=on" for Apple Silicon Metal, then set n_gpu_layers in the Llama constructor to offload layers to the GPU. Use 0 for CPU-only or a high value like 99 to offload everything.

Why does my GGUF model output gibberish after quantization?▼

Gibberish usually means the quantization is too aggressive, such as Q2_K, or the model converted incorrectly. Try Q4_K_M or Q5_K_M, use an importance matrix for calibration, and verify the FP16 version produces sensible output first.

Can llama-server provide an OpenAI-compatible API?▼

Yes, llama-server exposes /v1/chat/completions endpoints compatible with OpenAI clients, including streaming responses. Point your OpenAI client base_url to http://localhost:8080/v1 and use any model name string.

Related Skills

Explore More Software Engineering Skills→
waybarrioswaybarrios
507

huggingface-local-models

Selects and runs GGUF models from Hugging Face locally with llama.cpp.

Community
Intermediate
copilot-ldcopilot-ld
1

libllm

Expose an HTTP LLM API client for OpenAI-compatible chat completions and embeddings.

Official
Advanced
agentic-inagentic-in
577

Inference

Select inference paths, serving stacks, and benchmarking plans for ML models.

Official
Intermediate
drdave-flexnetosdrdave-flexnetos

AI Inference & Model Serving

Deploy LocalAI and vLLM servers for OpenAI-compatible chat and embeddings.

Community
Intermediate