skills.rest
luckybbjason1
luckybbjason1COMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Run local GGUF inference and discover Hugging Face models for llama.cpp.

Updated Sep 9, 2026
Search Tags:#quantization#edge deployment#gguf#llama.cpp#local inference#hugging face#llama-cpp-python
One-click install
npx skills add https://github.com/luckybbjason1/trading --skill llama-cpp-luckybbjason1
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/luckybbjason1/trading/tree/main/.hermes/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/luckybbjason1/trading --skill llama-cpp-luckybbjason1

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, and includes references (resource) components.

What problem does it solve? Running large language models locally requires choosing the right GGUF quantization, finding compatible models on Hugging Face, and configuring llama.cpp for your hardware, which involves many manual steps and platform-specific details. ## Core Features & Use Cases - Local GGUF Inference: Run models on CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs using llama-cli, llama-server, or the llama-cpp-python bindings with chat, streaming, and embedding support. - Hugging Face Model Discovery: Use URL-first workflows with the local-app page and tree API to find GGUF repos, enumerate exact filenames and sizes, and build launch commands without extra tooling. - Quantization Guidance: Choose between Q4_K_M, Q5_K_M, Q6_K, IQ variants, and repo-specific labels based on RAM or VRAM constraints. - Use Case: A developer wants to run a 24B-parameter model on a laptop with 16 GB of VRAM. The skill searches the Hub for llama.cpp-compatible repos, reads the hardware compatibility section, recommends UD-Q4_K_M, and outputs the exact llama-server command. ## Quick Start Ask the assistant to find a GGUF version of a Hugging Face model and generate the llama-server command to run it locally.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a GGUF model locally with llama.cpp?▼

Install llama.cpp via brew, winget, or CMake build, then run llama-cli or llama-server with the -hf flag pointing to a Hugging Face repo and quant, such as llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0. You can also load a local .gguf file path directly.

How do I find GGUF models on Hugging Face for llama.cpp?▼

Search https://huggingface.co/models?apps=llama.cpp with optional search terms and parameter bounds, then open the repo with ?local-app=llama.cpp to see the recommended quant. Use the tree API at /api/models/<repo>/tree/main?recursive=true to list exact .gguf filenames and sizes.

Which GGUF quantization should I choose for my RAM or VRAM?▼

Q4_K_M is the recommended balance of quality and size for general chat, while Q5_K_M or Q6_K suit code and technical work. For tight memory budgets use Q3_K_M or IQ variants, and always prefer the exact quant label shown in the repo's hardware compatibility section.

Does llama-cpp-python support GPU acceleration on CUDA and Metal?▼

Yes, install with CMAKE_ARGS="-DGGML_CUDA=on" for NVIDIA CUDA or "-DGGML_METAL=on" for Apple Silicon Metal, using --force-reinstall --no-cache-dir. Then set n_gpu_layers in the Llama constructor to offload layers to the GPU.

Why does llama.cpp run out of memory during inference?▼

Out of memory occurs when the model, context, and KV cache exceed available RAM or VRAM. Reduce n_gpu_layers, lower the context size with -c, switch to a smaller quantization like Q4_K_S, or quantize the KV cache with type_k and type_v settings.

Can llama-server provide an OpenAI-compatible API?▼

Yes, llama-server exposes /v1/chat/completions endpoints compatible with OpenAI clients, supporting streaming, parallel slots, and continuous batching. Point your OpenAI client base_url to http://localhost:8080/v1 to use it.

Related Skills

Explore More Software Engineering Skills→
DonZzzillaDonZzzilla

llama-cpp

Run local GGUF inference for llama.cpp models with Hugging Face Hub discovery.

Community
Advanced
sylin-orgsylin-org
4

koan-ai-integration

Integrate chat endpoints, embeddings, and RAG workflows with vector search.

Official
Advanced
agentic-inagentic-in
577

Inference

Select inference paths, serving stacks, and benchmarking plans for ML models.

Official
Intermediate
big4council-progbig4council-prog

serving-llms-vllm

Deploy high-throughput LLM inference endpoints with vLLM and OpenAI-compatible APIs.

Community
Intermediate