skills.rest
afel6
afel6COMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Run GGUF-quantized LLM inference on CPUs and non-NVIDIA GPUs.

Updated Mar 25, 2026
Search Tags:#quantization#gguf#apple-silicon#llama.cpp#rocm#cpu-inference#amd-gpu
One-click install
npx skills add https://github.com/afel6/scal-ai-pipeline --skill llama-cpp-afel6
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/afel6/scal-ai-pipeline/tree/main/hermes_skills_library/mlops/inference/llama-cpp
Command: npx skills add https://github.com/afel6/scal-ai-pipeline --skill llama-cpp-afel6

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Enables local and edge LLM inference when CUDA and NVIDIA hardware are unavailable, removing the need for heavy GPU infrastructure and simplifying deployment on CPUs, Apple Silicon, and alternative GPUs.

Core Features & Use Cases

  • Cross-platform inference: Build and run llama.cpp with Metal, ROCm, or CPU-only builds to serve GGUF-quantized models on M1/M2/M3 Macs, AMD/Intel GPUs, or embedded devices.
  • Quantization & conversion guidance: Convert HuggingFace models to GGUF, choose Q2–Q8 quant formats, and balance speed versus quality for 7B–70B models.
  • Server and production patterns: Start an OpenAI-compatible llama-server, tune offloading (-ngl), enable batching and metrics, and deploy with Docker and NGINX for scaling.
  • Use Case: Host a private, low-latency chat API on an M3 Mac or a fleet of edge devices using Q4_K_M quantized Llama-2-7B models for secure on-premises assistants.

Quick Start

Start an interactive CPU chat using the GGUF Q4_K_M Llama-2-7B model in models/ and run with 8 threads for balanced performance.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run LLM inference on Apple Silicon or AMD GPUs without CUDA?▼

Run LLM inference on Apple Silicon or AMD GPUs by building llama.cpp with Metal or ROCm support to serve GGUF-quantized models locally without relying on NVIDIA CUDA hardware.

What is the best way to convert HuggingFace models to GGUF format for local serving?▼

The best way to convert HuggingFace models to GGUF format is by following quantization and conversion guidance to select Q2–Q8 formats, balancing inference speed versus quality for 7B–70B models.

Can I host an OpenAI-compatible API server using GGUF models on edge devices?▼

Yes, you can host an OpenAI-compatible API server on edge devices by starting llama-server with GGUF models, tuning layer offloading with -ngl, and enabling batching for low-latency local inference.

Does llama.cpp support CPU-only inference for embedded devices and cloud instances?▼

llama.cpp supports CPU-only inference for embedded devices and cloud instances, allowing you to execute GGUF-quantized models and run lightweight servers using optimized thread configurations like 8 threads.

How do I scale local LLM inference deployments using Docker and NGINX?▼

Scale local LLM inference deployments by containerizing the OpenAI-compatible llama-server with Docker and routing traffic through NGINX, ensuring efficient load balancing across multiple edge devices.

When should I choose Q4_K_M quantization for running local LLM models?▼

Choose Q4_K_M quantization when running local LLM models to achieve an optimal balance between memory footprint and output quality, making it ideal for secure on-premises assistants on M3 Macs.

Related Skills

Explore More Software Engineering Skills→
Tyler-R-KendrickTyler-R-Kendrick
1

huggingface-local-models

Search Hugging Face Hub and launch GGUF models via llama.cpp.

Community
Intermediate
stackconsultstackconsult
2

nginx-production-infra

Configure Nginx reverse proxy with SSL, rate limiting, and load balancing.

Community
Advanced
fgarofalo56fgarofalo56
6

nginx-proxy

Configure Nginx and Nginx Proxy Manager for reverse proxying, load balancing, and SSL/TLS termination.

Community
Advanced
RheasilviaRheasilvia

tensorrt-llm

Optimize large language model inference on NVIDIA GPUs with quantization and dynamic batching.

Community
Advanced