skills.rest
zangjeicy
zangjeicyCOMMUNITY
Explore all skills by this author →
GitHub

llama-cpp

Discover GGUF models on Hugging Face and run local llama.cpp inference.

Updated Jul 13, 2026
Search Tags:#quantization#inference#gguf#llama.cpp#local llm#hugging face
One-click install
npx skills add https://github.com/zangjeicy/Hermes --skill llama-cpp-zangjeicy
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llama-cpp
Source: https://github.com/zangjeicy/Hermes/tree/main/skills/mlops/inference/llama-cpp
Command: npx skills add https://github.com/zangjeicy/Hermes --skill llama-cpp-zangjeicy

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires llama-cpp-python, and includes references (resource) components.

What problem does it solve?

This skill simplifies the complex process of finding, downloading, and running local GGUF models, removing the technical friction of managing hardware compatibility and inference configurations.

Core Features & Use Cases

  • Model Discovery: Search and identify the best GGUF models on Hugging Face based on your specific hardware constraints.
  • Inference Management: Launch local servers or CLI tools for chat, embeddings, and speculative decoding without manual setup.
  • Use Case: If you need to run a 7B parameter model on a laptop with limited VRAM, this skill helps you select the optimal quantization level and provides the exact command to launch it efficiently.

Quick Start

Use the llama-cpp skill to find and run the best quantized version of Llama-3.2-3B for my current hardware.

Frequently Asked Questions about llama-cpp

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run local GGUF models on hardware with limited VRAM?▼

You can run local GGUF models on limited VRAM by selecting an optimal quantization level to reduce memory footprint. This skill evaluates hardware constraints and provides exact CLI commands to launch inference efficiently on your device.

What's the best way to find and download quantized models on Hugging Face?▼

The best way to find quantized models on Hugging Face is using model discovery features that search repositories based on hardware constraints. This identifies optimal GGUF models for your specific edge-based execution requirements.

Can I launch a local server for llama.cpp inference without manual setup?▼

Yes, you can launch a local server for llama.cpp inference without manual setup. This skill manages inference configurations automatically, enabling chat, embeddings, and speculative decoding via local servers or CLI tools.

Does llama-cpp-python support grammar-constrained generation and batched inference?▼

Yes, llama-cpp-python supports grammar-constrained generation and batched inference. This skill facilitates these advanced features across CPU and GPU architectures, satisfying complex edge-based LLM execution requirements.

When do I need to adjust quantization levels for local LLM execution?▼

You need to adjust quantization levels when targeting specific hardware constraints or optimizing for limited VRAM. This skill helps select optimal quantization to ensure efficient local GGUF model inference across CPU and GPU architectures.

Related Skills

Explore More Software Engineering Skills→
richardnguyen0715richardnguyen0715

llama-cpp

Execute local GGUF inference for llama.cpp models across multiple hardware platforms.

Community
Advanced
ricablericable

sglang

Serve LLMs with prefix caching and constrained decoding for structured outputs.

Community
Advanced
huggingfacehuggingface
10.9k

hugging-face-cli

Manage Hugging Face Hub resources via the hf CLI.

Official
Intermediate
nelohenriqnelohenriq
1

modal-serverless-gpu

Deploy ML models as scalable APIs with serverless GPU compute.

Community
Advanced