skills.rest
lucasfth
lucasfthCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

llama-cpp

Run local LLM inference and quantize GGUF models with llama.cpp.

1|Updated Apr 30, 2025
Search Tags:#quantization#inference#gguf#apple-silicon#llama.cpp#edge-deployment
One-click install
npx skills add https://github.com/lucasfth/config --skill llama-cpp-lucasfth
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/lucasfth/config/tree/main/.hermes/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/lucasfth/config --skill llama-cpp-lucasfth

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Run LLaMA-compatible inference locally across CPU, Apple Silicon, and non-CUDA hardware, enabling quantization workflows and server integration without cloud dependencies.

Core Features & Use Cases

  • Local inference on CPU/Apple Silicon, with optional GPU offload for AMD/Intel GPUs and NVIDIA CUDA environments.
  • GGUF model conversion and quantization (2–8 bit) including K-quants and imatrix, plus OpenAI-compatible server support.
  • Python bindings and ecosystem integrations (Ollama, LM Studio, text-generation-webui) for flexible deployment.

Quick Start

Install llama.cpp, download a GGUF model, and start the local server or CLI to begin inference.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run local LLM inference on Apple Silicon or CPU-only hardware?▼

Local LLM inference on Apple Silicon or CPU-only machines is performed using GGUF models and llama.cpp, enabling text generation without cloud dependencies or CUDA requirements.

What is GGUF quantization and how do K-quants with imatrix work?▼

GGUF quantization reduces model size to 2-8 bits for local deployment. K-quants with imatrix optimize weight precision during the quantization process to preserve model accuracy.

Can I run an OpenAI-compatible server locally without an NVIDIA GPU?▼

Yes, you can run an OpenAI-compatible server locally on AMD/Intel GPUs, Apple Silicon, or CPU-only environments using llama.cpp, avoiding the need for NVIDIA CUDA hardware.

How do I convert models to GGUF format for edge deployment?▼

Convert models to GGUF format for edge deployment using llama.cpp's model conversion tools, followed by applying 2-8 bit quantization or K-quants to reduce file size for local execution.

Does llama.cpp integrate with Ollama or LM Studio?▼

Yes, llama.cpp supports ecosystem integrations with Ollama, LM Studio, and text-generation-webui, providing Python bindings and flexible deployment options for local model serving.

Related Skills

Explore More Software Engineering Skills→
davpatel605-beepdavpatel605-beep

llama-cpp

Configure llama.cpp deployments and select GGUF quantizations for local inference.

Community
Advanced
SignmanalSignmanal
1

llama-cpp

Run local GGUF model inference with llama.cpp across CPU and GPU hardware.

Community
Intermediate
dennisonbertramdennisonbertram
1

ollama-ops

Orchestrate local LLM inference and model management with Ollama.

Community
Intermediate
gagan114662gagan114662

llama-cpp

Run LLM inference on CPUs, Apple Silicon, and non-NVIDIA GPUs using GGUF quantization.

Community
Intermediate