skills.rest
vivekgoquest
vivekgoquestCOMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Runs local GGUF model inference and discovers llama.cpp-compatible models on Hugging Face Hub.

Updated Aug 22, 2026
Search Tags:#GGUF#quantization#edge deployment#Hugging Face#llama.cpp#local inference#llama-server
One-click install
npx skills add https://github.com/vivekgoquest/hermes-agent-stable --skill llama-cpp-vivekgoquest
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/vivekgoquest/hermes-agent-stable/tree/main/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/vivekgoquest/hermes-agent-stable --skill llama-cpp-vivekgoquest

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, and includes references (resource) components.

What problem does it solve? Running large language models locally requires choosing the right GGUF quantization, finding compatible model files on Hugging Face, and configuring llama.cpp for your specific hardware, which involves many manual steps and platform-specific details. ## Core Features & Use Cases - Hub Model Discovery: Search Hugging Face for llama.cpp-compatible repos via URL workflows, extract exact GGUF filenames and sizes from the tree API, and build ready-to-run llama-server or llama-cli commands. - Quantization Guidance: Choose between Q4_K_M, Q5_K_M, Q6_K, IQ variants, and repo-native labels based on RAM/VRAM budgets and use case (chat, code, edge devices). - Local Inference & Deployment: Run models on CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs using llama-cpp-python bindings or an OpenAI-compatible server with Docker and load balancing. - Use Case: A developer wants to run a 24B-parameter model on a laptop with 16 GB of RAM. The skill finds the right GGUF repo on Hugging Face, recommends a Q4_K_M quant that fits, and outputs the exact llama-server command. ## Quick Start Ask the assistant to find a llama.cpp-compatible GGUF model for a given Hugging Face repo and generate the llama-server command with a recommended quantization for your hardware.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a GGUF model locally with llama.cpp?▼

Install llama.cpp via brew, winget, or a CMake build, then run llama-cli or llama-server with the -hf flag pointing to a Hugging Face repo and quant, such as llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0.

How to find GGUF files for a Hugging Face model repo?▼

Query the Hub tree API at https://huggingface.co/api/models/<repo>/tree/main?recursive=true and keep entries where type is file and path ends with .gguf. The path and size fields give exact filenames and byte sizes.

What GGUF quantization should I use for my RAM or VRAM?▼

Start with Q4_K_M for general chat, Q5_K_M or Q6_K for code and technical work, and Q3_K_M or IQ variants for tight memory budgets. Prefer the exact quant labels shown in the repo's Hardware compatibility section when available.

Does llama-cpp-python support GPU acceleration with CUDA or Metal?▼

Yes, install with CMAKE_ARGS="-DGGML_CUDA=on" for CUDA or "-DGGML_METAL=on" for Apple Silicon Metal, then set n_gpu_layers to offload layers to the GPU. Use 0 for CPU-only or a high value to offload everything.

Why does llama.cpp run out of memory during inference?▼

Out-of-memory errors come from too many offloaded GPU layers, an overly large context, or a quant too big for available memory. Reduce n_gpu_layers, lower the context size, or switch to a smaller quantization like Q4_K_S.

Can llama-server provide an OpenAI-compatible API?▼

Yes, llama-server exposes /v1/chat/completions with streaming support, so OpenAI clients work by setting base_url to http://localhost:8080/v1. It also supports parallel slots, continuous batching, and prompt caching.

Related Skills

Explore More Software Engineering Skills→
faramirezsfaramirezs

llama-cpp

Automate local GGUF inference and model discovery for llama.cpp.

Community
Advanced
sylin-orgsylin-org
4

koan-ai-integration

Integrate chat endpoints, embeddings, and RAG workflows with vector search.

Official
Advanced
spacecat2002spacecat2002

sglang-diffusion-performance

Select SGLang Diffusion configuration flags for generation speed and GPU memory usage.

Community
Advanced
azaanalirazaazaanaliraza

llama-cpp

Run and quantize GGUF LLMs with llama.cpp across CPU, Apple Silicon, and GPU hardware.

Community
Advanced