skills.rest
guccang
guccangCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

llama-cpp

Select and deploy GGUF models for local llama.cpp inference.

3|1|Updated Apr 19, 2024
Search Tags:#model deployment#quantization#gguf#llama.cpp#gpu optimization#local inference#hugging face
One-click install
npx skills add https://github.com/guccang/blogclaw --skill llama-cpp-guccang
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/guccang/blogclaw/tree/main/cmd/hermes-agent/vendor/hermes_runtime/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/guccang/blogclaw --skill llama-cpp-guccang

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps users run local large language models efficiently by simplifying GGUF model discovery, quantization selection, and llama.cpp deployment workflows.

Core Features & Use Cases

  • Local Model Inference: Guides CPU, Apple Silicon, CUDA, ROCm, and Intel GPU deployments with llama.cpp and llama-cpp-python.
  • Model Discovery and Optimization: Helps find compatible Hugging Face GGUF models, choose suitable quantizations, and tune inference performance.
  • Use Case: A developer can use this Skill to select a GGUF model for their hardware, launch a local OpenAI-compatible inference server, and troubleshoot performance issues.

Quick Start

Use the llama-cpp skill to find the best GGUF quantization for my hardware and generate the correct llama-server command.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run local inference with a GGUF model using llama.cpp?▼

Run local inference by discovering a compatible Hugging Face GGUF model, selecting the right quantization, and launching llama-server. This Skill generates the deployment commands and configures GPU offloading for your specific hardware.

What is the best GGUF quantization for my hardware?▼

GGUF quantization choice depends on your available VRAM and desired inference speed. This Skill evaluates your hardware constraints to recommend a compatible quantization level that balances memory usage and model performance.

Does llama.cpp work with Apple Silicon, CUDA, and Intel GPUs?▼

llama.cpp supports local inference across CPU, Apple Silicon, CUDA, ROCm, and Intel GPUs. The Skill guides hardware-specific deployment configurations and GPU offloading to optimize serving performance.

How do I find compatible Hugging Face models for local deployment?▼

Finding compatible Hugging Face models involves filtering for the GGUF format and matching quantizations to your hardware. This Skill streamlines GGUF file discovery to ensure the model runs efficiently on your local setup.

Why is my local model inference slow and how can I optimize it?▼

Slow local model inference often results from incorrect GPU offloading or unsuitable quantization. This Skill troubleshoots performance bottlenecks and tunes llama-server configurations to improve generation speed.

Can I use llama.cpp to serve an OpenAI-compatible local API?▼

llama.cpp can launch a local OpenAI-compatible inference server using llama-server. This Skill provides the exact commands to configure and start the server for your selected GGUF model.

Related Skills

Explore More Software Engineering Skills→
lenadlmlenadlm

llama-cpp

Run local GGUF inference on llama.cpp models from Hugging Face Hub.

Community
Advanced
TehminanazTehminanaz
1

openai-agents-sdk-groq-integration

Integrate the OpenAI Agents SDK with Groq's endpoint in FastAPI applications.

Community
Advanced
superfhpsuperfhp

llama-cpp

Run local GGUF inference for llama.cpp models on CPU, Apple Silicon, CUDA, ROCm, and Intel GPUs.

Community
Advanced
ovachieverovachiever
52

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM for high-throughput inference.

Community
Advanced