skills.rest
LynxLabVN
LynxLabVNOFFICIAL
Explore all skills by this author →
GitHub

llama-cpp

Load, quantize, and run llama.cpp models for local inference.

Updated Jul 3, 2026
Search Tags:#quantization#inference#model discovery#Hugging Face Hub#llama.cpp
One-click install
npx skills add https://github.com/LynxLabVN/office-agent --skill llama-cpp-lynxlabvn
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/LynxLabVN/office-agent/tree/main/agent-core/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/LynxLabVN/office-agent --skill llama-cpp-lynxlabvn

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill automates local inference with llama.cpp models, supports model quantization for efficient computation, and assists in discovering the right llama.cpp models from the Hugging Face Hub.

Core Features & Use Cases

  • Local Inference: Perform inference with llama.cpp models on CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs.
  • Quantization: Apply various quantization methods (Q4, Q5, Q6, IQ) to reduce model size and speed up inference.
  • Model Discovery: Find suitable llama.cpp models from the Hugging Face Hub with size and hardware compatibility constraints.
  • Use Case: When you need a lightweight, high-performance inference engine for your AI application and you have a specific model size or hardware requirement.

Quick Start

Use the llama-cpp skill to run the 'bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0' model on your local machine.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run local inference with llama.cpp models on my machine?▼

Local inference with llama.cpp models is automated by loading and running models directly on your hardware. The process supports execution across CPU, Apple Silicon, CUDA, ROCm, and Intel GPUs for high-performance text generation.

What is the best way to quantize Hugging Face Hub models for local inference?▼

Quantizing Hugging Face models involves applying methods like Q4, Q5, Q6, or IQ to reduce model size. This speeds up inference and lowers memory requirements while maintaining compatibility with the llama.cpp format.

Can I use llama.cpp with CUDA or Apple Silicon hardware?▼

Yes, llama.cpp supports multiple hardware platforms including CUDA, Apple Silicon, ROCm, and Intel GPUs. You can run local inference and quantize models effectively across these distinct hardware accelerators.

How do I find suitable llama.cpp models on the Hugging Face Hub?▼

Model discovery from the Hugging Face Hub is automated by filtering based on your specific size and hardware compatibility constraints. This ensures you find the right llama.cpp models for your local inference needs.

Do I need llama-cpp-python to run quantized models locally?▼

Yes, you need the llama-cpp-python dependency to interface with the models. It provides the necessary Python bindings to load, quantize, and run llama.cpp models for your local inference tasks.

What are the limitations of running quantized llama.cpp models?▼

Running quantized llama.cpp models requires the llama-cpp-python dependency and sufficient hardware resources. While quantization reduces model size, extreme compression may impact generation quality.

Related Skills

Explore More Software Engineering Skills→
peteromalletpeteromallet
97

llama-cpp

Run CPU-only LLM inference with llama-cpp on non-NVIDIA hardware.

Community
Intermediate
twjarviszyra-webtwjarviszyra-web
3

llama-cpp

Run local LLM inference from GGUF models on CPUs and GPUs.

Community
Intermediate
evalstateevalstate

trl

Submit TRL training jobs on Hugging Face Jobs and convert models to GGUF.

Community
Advanced
YosokuSolanaYosokuSolana

YOSOKU SDK

Interact with the YOSOKU prediction market protocol on Solana via a typed TypeScript client.

Community
Intermediate