skills.rest
TylerSimons1127
TylerSimons1127COMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Runs local GGUF model inference and discovers llama.cpp-compatible models on Hugging Face Hub.

Updated Aug 21, 2026
Search Tags:#GGUF#quantization#edge deployment#Hugging Face#llama.cpp#local inference#llama-server
One-click install
npx skills add https://github.com/TylerSimons1127/vibe --skill llama-cpp-tylersimons1127
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/TylerSimons1127/vibe/tree/main/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/TylerSimons1127/vibe --skill llama-cpp-tylersimons1127

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python>=0.2.0, and includes references (resource) components.

What problem does it solve? Running large language models locally requires choosing the right GGUF quantization, finding compatible models on Hugging Face, and configuring llama.cpp for your specific hardware, which involves many manual steps and platform-specific pitfalls. ## Core Features & Use Cases - Hub-First Model Discovery: Search Hugging Face for llama.cpp-compatible models via URL workflows, extract recommended quants from local-app pages, and confirm exact GGUF filenames and sizes through the tree API. - Local Inference Setup: Install and run llama-cli or llama-server on CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs, including direct-from-Hub commands and OpenAI-compatible server endpoints. - Quant Selection Guidance: Choose between Q4/Q5/Q6/IQ variants based on RAM or VRAM budgets, with repo-native labels preserved exactly as Hugging Face shows them. - Use Case: A user wants to run a 24B-parameter model on a laptop with 16 GB of VRAM. The skill finds the right GGUF repo, identifies the UD-Q4_K_M quant with its exact file size, and produces a ready-to-run llama-server command. ## Quick Start Ask the agent to find a llama.cpp-compatible GGUF for a Hugging Face model and generate the llama-server command for your hardware.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a GGUF model locally with llama.cpp?▼

Install llama.cpp via brew, winget, or a CMake build, then run llama-cli or llama-server with the -hf flag pointing to a Hugging Face repo and quant, such as llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0.

How do I find GGUF files for a Hugging Face model?▼

Open the repo with ?local-app=llama.cpp to see the recommended quant, then query the tree API at huggingface.co/api/models/<repo>/tree/main?recursive=true to list exact .gguf filenames and byte sizes.

Which GGUF quantization should I choose for my RAM?▼

Prefer the quant Hugging Face marks as compatible for your hardware profile. For general chat start with Q4_K_M, use Q5_K_M or Q6_K for code work, and consider Q3_K_M or IQ variants only for very tight memory budgets.

Does llama.cpp support GPU acceleration on Apple Silicon?▼

Yes, llama.cpp supports Metal on Apple Silicon. Build with GGML_METAL=1 or install llama-cpp-python with CMAKE_ARGS="-DGGML_METAL=on", then set n_gpu_layers to 99 to offload all layers.

Why does llama-cpp-python fail to build with CUDA?▼

Build failures usually come from missing CUDA toolkit headers or build dependencies. Install cmake and scikit-build-core, then reinstall with CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python --force-reinstall --no-cache-dir.

Can llama-server provide an OpenAI-compatible API?▼

Yes, llama-server exposes /v1/chat/completions endpoints compatible with OpenAI clients. Point your client's base_url to http://localhost:8080/v1 and send standard chat completion requests with optional streaming.

Related Skills

Explore More Software Engineering Skills→
waybarrioswaybarrios
507

hf-mem

Estimates inference memory requirements for Hugging Face Safetensors and GGUF models without downloading weights.

Community
Basic
sylin-orgsylin-org
4

koan-ai-integration

Integrate chat endpoints, embeddings, and RAG workflows with vector search.

Official
Advanced
matthew-johnsonmatthew-johnson

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT and quantization.

Community
Advanced
azaanalirazaazaanaliraza

llama-cpp

Run and quantize GGUF LLMs with llama.cpp across CPU, Apple Silicon, and GPU hardware.

Community
Advanced