skills.rest
Chia1104
Chia1104COMMUNITY
Explore all skills by this author →
Website𝕏@chia7423GitHub

llama-cpp

Runs local GGUF model inference and discovers llama.cpp-compatible models on Hugging Face Hub.

1|Updated Aug 11, 2026
Search Tags:#GGUF#quantization#edge deployment#Hugging Face#llama.cpp#local inference#llama-server
One-click install
npx skills add https://github.com/Chia1104/agent-air --skill llama-cpp-chia1104
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/Chia1104/agent-air/tree/main/skills/hermes/mlops/inference/llama-cpp
Command: npx skills add https://github.com/Chia1104/agent-air --skill llama-cpp-chia1104

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python>=0.2.0, and includes references (resource) components.

What problem does it solve? Running large language models locally requires choosing the right GGUF quantization, finding compatible model files on Hugging Face, and configuring llama.cpp for your hardware, which involves many manual steps and error-prone guesswork. ## Core Features & Use Cases - Hub Model Discovery: Search Hugging Face for llama.cpp-compatible repos via URL workflows, extract exact GGUF filenames and sizes from the tree API, and build ready-to-run llama-server or llama-cli commands. - Quant Selection Guidance: Choose between Q4_K_M, Q5_K_M, Q6_K, IQ variants, and repo-native labels based on the user's RAM or VRAM budget. - Local Inference & Deployment: Run models on CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs using llama-cli, llama-server with an OpenAI-compatible API, or the llama-cpp-python bindings. - Use Case: A user with 16 GB of VRAM asks which Qwen model to run locally; the skill searches the Hub, lists available GGUF quants with sizes, and outputs the exact llama-server command. ## Quick Start Ask the assistant to find a llama.cpp-compatible GGUF model on Hugging Face for your hardware and generate the llama-server command to run it locally.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a GGUF model locally with llama.cpp?▼

Install llama.cpp via brew, winget, or a CMake build, then run llama-cli or llama-server with the -hf flag pointing to a Hugging Face repo and quant, for example llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0.

How do I find GGUF models on Hugging Face for llama.cpp?▼

Search huggingface.co/models with the apps=llama.cpp filter, then open the repo with ?local-app=llama.cpp to see the recommended quant. Confirm exact filenames and sizes using the repo's tree API endpoint.

Which GGUF quantization should I choose for my RAM or VRAM?▼

Q4_K_M is the recommended balance of quality and size for general chat. Use Q5_K_M or Q6_K for code and technical work, and Q3 or IQ variants only when memory is very tight.

Does llama.cpp support GPU acceleration on Apple Silicon and NVIDIA?▼

Yes, llama.cpp supports Metal on Apple Silicon, CUDA on NVIDIA, ROCm on AMD, and Intel GPUs. Set n_gpu_layers to offload layers to the GPU, using 99 to offload everything.

Why does llama-cpp-python fail to build with CUDA support?▼

The build fails when CUDA toolkit paths are missing or the wheel lacks GPU support. Reinstall with CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python --force-reinstall --no-cache-dir after installing build dependencies.

Can llama-server provide an OpenAI-compatible API?▼

Yes, llama-server exposes an OpenAI-compatible /v1/chat/completions endpoint on port 8080 by default. Point an OpenAI client at http://localhost:8080/v1 to use it with streaming support.

Related Skills

Explore More Software Engineering Skills→
aws-samplesaws-samples
1

bench-open-model

Benchmarks open-weight Hugging Face models on AWS EC2 GPU instances with vLLM.

Official
Advanced
sylin-orgsylin-org
4

koan-ai-integration

Integrate chat endpoints, embeddings, and RAG workflows with vector search.

Official
Advanced
air-gappedair-gapped
5

aiperf

Benchmark latency, throughput, and telemetry across OpenAI-compatible inference servers.

Official
Advanced
azaanalirazaazaanaliraza

llama-cpp

Run and quantize GGUF LLMs with llama.cpp across CPU, Apple Silicon, and GPU hardware.

Community
Advanced