skills.rest
Signmanal
SignmanalCOMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Run local GGUF model inference with llama.cpp across CPU and GPU hardware.

1|Updated Jun 25, 2026
Search Tags:#llm#quantization#huggingface#gpu-acceleration#gguf#llama.cpp#local-inference
One-click install
npx skills add https://github.com/Signmanal/VIGIL --skill llama-cpp-signmanal
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/Signmanal/VIGIL/tree/main/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/Signmanal/VIGIL --skill llama-cpp-signmanal

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill eliminates the friction of setting up local large language model inference with llama.cpp, finding compatible GGUF checkpoints on Hugging Face, and selecting the right quantization for your available hardware, removing the need for manual trial and error or cloud inference costs.

Core Features & Use Cases

  • Local GGUF Inference: Run quantized LLMs on CPU, Apple Silicon, NVIDIA, AMD, or Intel GPUs with full local control over your data and compute.
  • Hugging Face Model Discovery: Find llama.cpp-compatible GGUF models, view available quantization options and file sizes, and generate exact launch commands directly from Hub repos without extra tooling.
  • Use Case: For example, if you need a 7B parameter chat model that fits in 8GB of RAM, use this Skill to locate the optimal Q4_K_M quant, get the precise llama-server command, and launch a local OpenAI-compatible API endpoint in seconds.

Quick Start

Use the llama-cpp skill to find a Q4_K_M quantized Llama-3.2-3B GGUF model on Hugging Face and generate the exact llama-server command to run it on your local machine.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run local LLM inference using GGUF models with llama.cpp?▼

Find GGUF models on Hugging Face Hub by using this Skill to discover compatible checkpoints, view available quantization options and file sizes, and generate exact launch commands directly from the repository.

How do I choose the right GGUF quantization for my available hardware?▼

Choose the right GGUF quantization by aligning your selection with specific hardware constraints like memory limits. This Skill helps you identify the optimal quant, such as Q4_K_M, to fit your available RAM and GPU resources.

Can I use llama.cpp for GPU acceleration on Apple Silicon, NVIDIA, and AMD?▼

Yes, you can use llama.cpp for GPU acceleration on Apple Silicon, NVIDIA, AMD, and Intel GPUs. The Skill supports configuring local inference deployment to utilize your specific graphics hardware for improved performance.

What is the best way to serve an OpenAI-compatible API locally from a quantized model?▼

The best way to serve an OpenAI-compatible API locally is by running the llama-server command with a GGUF model. This Skill provides the exact command configuration needed to deploy your local API endpoint in seconds.

Related Skills

Explore More Software Engineering Skills→
CyapstayeCyapstaye
9

llama-cpp

Discover GGUF models on Hugging Face and run local inference with llama.cpp.

Community
Advanced
rawvegrawveg
40

ollama

Run and interact with local AI models via the Ollama API.

Community
Advanced
sheawinklersheawinkler
78

llama-cpp

Run local GGUF inference and discover Hugging Face Hub models with llama.cpp.

Community
Advanced
maystudiosmaystudios
15

llama-cpp

Build, configure, and deploy llama.cpp for local LLM inference.

Community
Advanced