skills.rest
iani-kuli
iani-kuliCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

llama-cpp

Discover and run quantized GGUF models locally via llama.cpp and Hugging Face.

Updated May 5, 2026
Search Tags:#quantization#inference#gguf#llama.cpp#local llm#hugging face
One-click install
npx skills add https://github.com/iani-kuli/harness_bro --skill llama-cpp-iani-kuli
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/iani-kuli/harness_bro/tree/main/.claude/skills/ported/llama-cpp
Command: npx skills add https://github.com/iani-kuli/harness_bro --skill llama-cpp-iani-kuli

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This skill simplifies the complex process of discovering, selecting, and running quantized GGUF models locally, removing the guesswork from hardware compatibility and model performance.

Core Features & Use Cases

  • Model Discovery: Efficiently search and identify the best GGUF models on Hugging Face using URL-based workflows.
  • Hardware Optimization: Provides clear guidance on quantization levels (Q4/Q5/Q6) and GPU offloading to match your specific RAM and VRAM constraints.
  • Deployment: Enables rapid setup of local inference servers or CLI tools for chat, embeddings, and API-compatible services.

Quick Start

Use the llama-cpp skill to find and run a quantized version of the Llama-3.2-3B model that fits within my available system memory.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run local GGUF models using llama.cpp?▼

Run local GGUF models using llama.cpp by discovering quantized models on Hugging Face and deploying them via local inference servers or CLI tools optimized for your specific CPU and GPU hardware.

What quantization level should I choose for local LLM inference?▼

Choose quantization levels like Q4, Q5, or Q6 for local LLM inference based on your available system RAM and VRAM, balancing memory constraints against model performance and inference speed.

Can I deploy a local inference server with Hugging Face GGUF models?▼

Deploy a local inference server with Hugging Face GGUF models by identifying suitable quantized versions and running API-compatible services, chat interfaces, or embedding generation through llama.cpp.

Does llama.cpp support GPU offloading for memory-constrained model execution?▼

llama.cpp supports GPU offloading for memory-constrained model execution, providing clear guidance on hardware optimization to match your specific VRAM constraints and improve local inference performance.

What's the best way to find GGUF models on Hugging Face for edge deployment?▼

Find GGUF models on Hugging Face for edge deployment using URL-based discovery workflows to efficiently search and identify the best quantized models that fit within your available system memory.

Why use GGUF format for local LLM hosting instead of other model formats?▼

Use GGUF format for local LLM hosting because it enables hardware-aware quantization selection and optimized inference across CPU and GPU environments, satisfying strict memory constraints for edge deployment.

Related Skills

Explore More Software Engineering Skills→
GeargrindadminGeargrindadmin

llmfit-advisor

Analyze local CPU, RAM, and GPU/VRAM to recommend LLM models with quantization and fit scores.

Community
Intermediate
SignmanalSignmanal
1

llama-cpp

Run local GGUF model inference with llama.cpp across CPU and GPU hardware.

Community
Intermediate
aivos-xieaivos-xie
1

llama-cpp

Run local GGUF inference with llama.cpp models from the Hugging Face Hub.

Community
Intermediate
JKhyroJKhyro

llama-cpp

Run local LLM inference with GGUF models on non-NVIDIA hardware.

Community
Advanced