skills.rest
vikrant-project
vikrant-projectCOMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Execute GGUF inference locally with llama.cpp and Python bindings.

2|Updated Jun 8, 2026
Search Tags:#GGUF#quantization#inference#model discovery#llama.cpp
One-click install
npx skills add https://github.com/vikrant-project/devil-agent-ai-platform --skill llama-cpp-vikrant-project
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/vikrant-project/devil-agent-ai-platform/tree/main/agent_core/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/vikrant-project/devil-agent-ai-platform --skill llama-cpp-vikrant-project

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides a streamlined interface for executing local GGUF inference with llama.cpp, simplifying the process of running quantized models and facilitating model discovery on the Hugging Face Hub.

Core Features & Use Cases

  • Local GGUF Inference: Perform inference on quantized models stored locally.
  • Model Discovery: Search and retrieve models from the Hugging Face Hub using llama.cpp.
  • Quant Selection: Determine the appropriate quantization level for different hardware setups.
  • Use Case: Imagine you need to perform inference on a specific model with limited memory. Use this Skill to find the right model and quantization level for your system and execute the inference locally.

Quick Start

Execute the llama.cpp model with the specified quantization level: llama-server -hf <repo>:<QUANT>

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run local GGUF inference with llama.cpp?▼

To run local GGUF inference with llama.cpp, execute `llama-server -hf <repo>:<QUANT>` using the Python bindings. This enables local execution of quantized models with fine-grained control over hardware compatibility and memory usage.

How do I find and retrieve GGUF models from the Hugging Face Hub?▼

You can find and retrieve GGUF models from the Hugging Face Hub using the model discovery feature. This functionality integrates with llama.cpp to search and download quantized models directly for local execution.

How do I choose the right quantization level for my hardware?▼

Choosing the right quantization level for your hardware involves using the quant selection feature to match model size with your available memory. This ensures efficient inference when running local GGUF models with llama.cpp and prevents out-of-memory errors.

Do I need llama-cpp-python installed to execute GGUF models?▼

Yes, you need the llama-cpp-python package installed to execute GGUF models. This dependency provides the necessary Python bindings for running quantized inference with llama.cpp locally on your machine.

Can I run inference on quantized models with limited memory?▼

Yes, you can run inference on quantized models with limited memory by selecting a lower quantization level. This Skill facilitates finding the appropriate model and quantization setup for your specific hardware constraints to ensure smooth execution.

Related Skills

Explore More Software Engineering Skills→
AlexiosBluffMaraAlexiosBluffMara
2

llama-cpp

Run local GGUF inference and discover models via Hugging Face Hub.

Community
Intermediate
djankiesdjankies

using-runtime-checks

Validate external data at runtime using Zod schemas in TypeScript applications.

Community
Advanced
ToqsickToqsick

llama-cpp

Run local GGUF inference for llama.cpp models with quantization selection and Hugging Face Hub discovery.

Community
Advanced
nanocoainanocoai
30.5k

add-atomic-chat-tool

Integrate local Atomic Chat models with OpenAI-compatible APIs for on-device inference.

Official
Intermediate