skills.rest
perasyudha
perasyudhaCOMMUNITY
Explore all skills by this author →
Website𝕏@Nyxora_AIGitHub

llama-cpp

Runs local GGUF model inference and discovers quantized models on Hugging Face Hub.

5|2|Updated May 26, 2026
Search Tags:#quantization#edge deployment#gguf#llama.cpp#local inference#hugging face#llama-server
One-click install
npx skills add https://github.com/perasyudha/Nyxora --skill llama-cpp-perasyudha
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/perasyudha/Nyxora/tree/main/packages/core/playbooks/mlops/inference/llama-cpp
Command: npx skills add https://github.com/perasyudha/Nyxora --skill llama-cpp-perasyudha

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, and includes references (resource) components.

What problem does it solve? Running large language models locally requires choosing the right GGUF quantization, finding compatible model files on Hugging Face, and configuring llama.cpp for your hardware, which involves many manual steps and platform-specific pitfalls. ## Core Features & Use Cases - Hub-First Model Discovery: Search Hugging Face for llama.cpp-compatible models, read the local-app page for recommended quants, and confirm exact GGUF filenames and sizes via the tree API. - Local Inference Setup: Build llama-cli and llama-server commands, use llama-cpp-python bindings for chat, streaming, and embeddings across CPU, Apple Silicon, CUDA, ROCm, and Intel GPUs. - Quantization Guidance: Choose between Q4_K_M, Q5_K_M, Q6_K, IQ variants, and repo-specific labels based on RAM or VRAM constraints. - Use Case: A developer wants to run a 35B model on a machine with 24 GB VRAM. The skill finds the repo's hardware compatibility section, recommends UD-Q4_K_M at 22.1 GB, and outputs the exact llama-server command. ## Quick Start Ask the assistant to find a GGUF version of a Hugging Face model and generate the llama-server command to run it locally.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I find GGUF models for llama.cpp on Hugging Face?▼

Search huggingface.co/models with the apps=llama.cpp filter, optionally adding a search term and num_parameters bounds. Open the repo with ?local-app=llama.cpp to see the recommended quant, then confirm exact filenames via the tree API endpoint.

How do I run a GGUF model directly from Hugging Face Hub?▼

Use llama-server -hf <repo>:<QUANT> for shorthand quant selection, or llama-server --hf-repo <repo> --hf-file <filename.gguf> to pin an exact file. The same -hf syntax works with llama-cli for interactive use.

Which GGUF quantization should I choose for my RAM or VRAM?▼

Start with Q4_K_M for general chat, Q5_K_M or Q6_K for code and technical work, and Q3 or IQ variants only for tight memory budgets. Prefer the exact quant labels shown in the repo's hardware compatibility section over generic tables.

Does llama-cpp-python support GPU acceleration on CUDA and Metal?▼

Yes, install with CMAKE_ARGS="-DGGML_CUDA=on" for NVIDIA GPUs or "-DGGML_METAL=on" for Apple Silicon, using --force-reinstall --no-cache-dir. Then set n_gpu_layers to control how many layers offload to the GPU.

Why does my quantized GGUF model output gibberish?▼

Gibberish usually means the quantization is too aggressive, such as Q2_K, or the conversion was faulty. Test the FP16 version first, regenerate with an importance matrix, or move up to Q4_K_M or Q5_K_M.

What are mmproj GGUF files in a model repo?▼

mmproj-*.gguf files are projector weights for multimodal models, not the main language model checkpoint. When listing available GGUFs, separate them from quantized checkpoints and BF16 shard files.

Related Skills

Explore More Software Engineering Skills→
waybarrioswaybarrios
507

huggingface-best

Finds and ranks Hugging Face models for a task using benchmark leaderboards and device constraints.

Community
Intermediate
sylin-orgsylin-org
4

koan-ai-integration

Integrate chat endpoints, embeddings, and RAG workflows with vector search.

Official
Advanced
NousResearchNousResearch
239.1k

serving-llms-vllm

Deploy high-throughput LLM inference servers with vLLM, PagedAttention, and quantization.

Official
Advanced
NVIDIA-AI-IOTNVIDIA-AI-IOT
110

jetson-llm-serve

Deploy OpenAI-compatible LLM or VLM serving endpoints on Jetson devices.

Official
Advanced