llama-cpp

Name: llama-cpp
Availability: InStock
Author: LynxLab

Load, quantize, and run llama.cpp models for local inference.

Updated Jul 3, 2026

Search Tags:#quantization #inference #model discovery #Hugging Face Hub #llama.cpp

Ready to run

npx skills add https://github.com/LynxLabVN/office-agent --skill llama-cpp-lynxlabvn

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/LynxLabVN/office-agent/tree/main/agent-core/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/LynxLabVN/office-agent --skill llama-cpp-lynxlabvn

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill automates local inference with llama.cpp models, supports model quantization for efficient computation, and assists in discovering the right llama.cpp models from the Hugging Face Hub.

Core Features & Use Cases

Local Inference: Perform inference with llama.cpp models on CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs.
Quantization: Apply various quantization methods (Q4, Q5, Q6, IQ) to reduce model size and speed up inference.
Model Discovery: Find suitable llama.cpp models from the Hugging Face Hub with size and hardware compatibility constraints.
Use Case: When you need a lightweight, high-performance inference engine for your AI application and you have a specific model size or hardware requirement.

Quick Start

Use the llama-cpp skill to run the 'bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0' model on your local machine.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I run local inference with llama.cpp models on my machine?▼

Local inference with llama.cpp models is automated by loading and running models directly on your hardware. The process supports execution across CPU, Apple Silicon, CUDA, ROCm, and Intel GPUs for high-performance text generation.

What is the best way to quantize Hugging Face Hub models for local inference?▼

Quantizing Hugging Face models involves applying methods like Q4, Q5, Q6, or IQ to reduce model size. This speeds up inference and lowers memory requirements while maintaining compatibility with the llama.cpp format.

Can I use llama.cpp with CUDA or Apple Silicon hardware?▼

Yes, llama.cpp supports multiple hardware platforms including CUDA, Apple Silicon, ROCm, and Intel GPUs. You can run local inference and quantize models effectively across these distinct hardware accelerators.

How do I find suitable llama.cpp models on the Hugging Face Hub?▼

Model discovery from the Hugging Face Hub is automated by filtering based on your specific size and hardware compatibility constraints. This ensures you find the right llama.cpp models for your local inference needs.

Do I need llama-cpp-python to run quantized models locally?▼

Yes, you need the llama-cpp-python dependency to interface with the models. It provides the necessary Python bindings to load, quantize, and run llama.cpp models for your local inference tasks.

What are the limitations of running quantized llama.cpp models?▼

Running quantized llama.cpp models requires the llama-cpp-python dependency and sufficient hardware resources. While quantization reduces model size, extreme compression may impact generation quality.