skills.rest
Junkfooooood
JunkfoooooodCOMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Run local GGUF inference and discover Hugging Face models for llama.cpp.

Updated May 4, 2026
Search Tags:#quantization#edge deployment#gguf#llama.cpp#local inference#hugging face#llama-server
One-click install
npx skills add https://github.com/Junkfooooood/hermes-governance --skill llama-cpp-junkfooooood
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/Junkfooooood/hermes-governance/tree/main/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/Junkfooooood/hermes-governance --skill llama-cpp-junkfooooood

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python>=0.2.0, and includes references (resource) components.

What problem does it solve? Running large language models locally requires choosing the right GGUF quantization, finding compatible model files on Hugging Face, and configuring llama.cpp for your specific hardware, which involves many manual steps and platform-specific details. ## Core Features & Use Cases - Hub-First Model Discovery: Search Hugging Face for llama.cpp-compatible repos, read the local-app page for recommended quants, and confirm exact GGUF filenames and sizes via the tree API. - Local Inference on Any Hardware: Run models on CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs using llama-cli, llama-server, or the llama-cpp-python bindings with chat, streaming, and embedding support. - Quantization Guidance: Choose between Q4_K_M, Q5_K_M, Q6_K, IQ variants, and repo-specific labels like UD-Q4_K_M based on RAM/VRAM budgets and quality needs. - Use Case: A developer wants to run a 24B-parameter model on a laptop with 16 GB of VRAM. The skill finds the right GGUF repo on Hugging Face, identifies the quant that fits, and produces the exact llama-server command to launch an OpenAI-compatible endpoint. ## Quick Start Ask the assistant to find a GGUF version of a Hugging Face model and give you the llama-server command to run it locally.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a Hugging Face model locally with llama.cpp?▼

Use llama-cli or llama-server with the -hf flag pointing to a GGUF repo and quant, for example llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0. For exact filenames, use --hf-repo and --hf-file instead.

How do I find GGUF files for a Hugging Face repo?▼

Query the tree API at https://huggingface.co/api/models/<repo>/tree/main?recursive=true and keep entries where type is file and path ends with .gguf. The path and size fields give exact filenames and byte sizes.

Which GGUF quantization should I choose for my hardware?▼

Start with Q4_K_M for general chat, Q5_K_M or Q6_K for code and technical work, and Q3 or IQ variants only for tight RAM budgets. Prefer the exact quant shown in the repo's Hardware compatibility section when available.

Does llama-cpp-python support GPU acceleration?▼

Yes, install with CMAKE_ARGS="-DGGML_CUDA=on" for CUDA or "-DGGML_METAL=on" for Apple Silicon Metal, then set n_gpu_layers to offload layers to the GPU. Use 0 for CPU-only or a high value to offload everything.

Why does my GGUF model output gibberish after quantization?▼

Aggressive quantization like Q2_K degrades quality significantly. Regenerate with Q4_K_M or Q5_K_M, use an importance matrix (imatrix) with representative calibration data, and verify the FP16 base model works first.

Can llama-server provide an OpenAI-compatible API?▼

Yes, llama-server exposes /v1/chat/completions endpoints compatible with OpenAI clients. Point your client's base_url to http://localhost:8080/v1 and send standard chat completion requests with optional streaming.

Related Skills

Explore More Software Engineering Skills→
DarkArty07DarkArty07
3

llama-cpp

Deploy local LLM inference with GGUF quantized weights via llama.cpp.

Community
Advanced
sylin-orgsylin-org
4

koan-ai-integration

Integrate chat endpoints, embeddings, and RAG workflows with vector search.

Official
Advanced
samuelmukotisamuelmukoti

tensorrt-llm

Optimizes LLM inference throughput and latency on NVIDIA GPUs using TensorRT-LLM with quantization and multi-GPU deployment.

Community
Advanced
Shubh2310-developerShubh2310-developer
2

gguf-quantization

Convert and quantize HuggingFace models to GGUF format for llama.cpp inference.

Community
Advanced