skills.rest
tdimino
tdiminoCOMMUNITY
Explore all skills by this author →
Website𝕏@IdaeanDaktylGitHub

llama-cpp

Run GGUF model inference with llama.cpp and apply LoRA adapters.

41|3|Updated Jan 2, 2026
Search Tags:#benchmarking#LLM#GGUF#LoRA#inference#llama.cpp#API server
One-click install
npx skills add https://github.com/tdimino/claude-code-minoan --skill llama-cpp-tdimino
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/tdimino/claude-code-minoan/tree/main/skills/integration-automation/llama-cpp
Command: npx skills add https://github.com/tdimino/claude-code-minoan --skill llama-cpp-tdimino

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cli, llama-server, llama-embedding, llama-quantize, peft, transformers, torch, and includes scripts (resource) components.

What problem does it solve?

This Skill provides direct access to the llama.cpp inference engine for advanced local LLM operations, offering faster inference, LoRA adapter loading, and benchmarking capabilities that complement primary engines like Ollama.

Core Features & Use Cases

  • Direct GGUF Model Inference: Run GGUF models efficiently for faster text generation.
  • Dynamic LoRA Adapter Loading: Test and apply LoRA adapters without merging them into base models.
  • Benchmarking: Compare inference speeds between llama.cpp and Ollama.
  • Custom API Server: Serve models via an OpenAI-compatible API.
  • Kothar LoRA Conversion: Convert Kothar training pipeline LoRAs into GGUF format.

Quick Start

Use the llama-cpp skill to run inference on the 'qwen2.5:7b' model with the prompt 'Explain the theory of general relativity in three paragraphs.'.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run inference on GGUF models locally with GPU offloading?▼

Local GGUF model inference is executed directly through the llama.cpp engine, utilizing optimized parameters like GPU offloading and flash attention for faster text generation.

Can I apply LoRA adapters to a GGUF model without permanently merging them?▼

Dynamic LoRA adapter loading allows you to test and apply adapters on top of GGUF models in memory without permanently merging them into the base model.

What is the best way to benchmark llama.cpp inference speed against Ollama?▼

Benchmarking inference speed against Ollama is handled natively by the skill to compare text generation performance and identify the faster engine for specific hardware.

How do I serve a local GGUF model through an OpenAI-compatible API?▼

Serving GGUF models via an OpenAI-compatible API is achieved using the integrated llama-server component to expose local inference to applications.

Does this workflow support converting Kothar training pipeline LoRAs into GGUF format?▼

Kothar LoRA to GGUF conversion is fully supported to transform training pipeline adapters for direct inference using the optimized llama.cpp engine.

Related Skills

Explore More Software Engineering Skills→
XyHalcyonXyHalcyon

llama-cpp

Run local inference with llama.cpp GGUF models and discover models from Hugging Face Hub.

Community
Advanced
sahiixxsahiixx

ai-tools

Generate embeddings, analyze images, and extract structured JSON from text and documents.

Community
Advanced
RachasumanthRachasumanth

inference-deploy

Deploy trained machine learning models for inference using vLLM, TGI, Ollama, and llama.cpp.

Community
Advanced
SethyPagnaSethyPagna

optimizing-attention-flash

Reduce transformer attention memory use and latency in PyTorch workflows.

Community
Advanced