skills.rest
eyadsibai
eyadsibaiCOMMUNITY
Explore all skills by this author →
Website𝕏@eyadsibaiGitHub

llm-inference

Compare inference engines like vLLM and llama.cpp for LLM serving.

7|1|Updated Jan 14, 2026
Search Tags:#optimization#llm#inference#vllm#gguf#llama.cpp#model serving
One-click install
npx skills add https://github.com/eyadsibai/ltk --skill llm-inference-eyadsibai
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-inference
Source: https://github.com/eyadsibai/ltk/tree/main/plugins/ltk-data/skills/llm-inference
Command: npx skills add https://github.com/eyadsibai/ltk --skill llm-inference-eyadsibai

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill provides guidance and comparisons for deploying and optimizing Large Language Models (LLMs) for inference, addressing challenges in throughput, latency, and hardware utilization.

Core Features & Use Cases

  • Engine Comparison: Offers a clear overview of popular inference engines like vLLM, llama.cpp, TGI, Ollama, and TensorRT-LLM, detailing their strengths and best use cases.
  • Decision Guide: Helps users select the right engine based on their specific scenario, from production APIs to local development and edge deployments.
  • Optimization Concepts: Explains key techniques such as PagedAttention, continuous batching, quantization, and speculative decoding.
  • Use Case: A developer needs to serve a fine-tuned LLM for a customer-facing application and wants to understand which engine offers the best throughput and lowest latency for their GPU setup.

Quick Start

Use the llm-inference skill to compare vLLM and llama.cpp for serving a 7B parameter model on a local machine with a GPU.

Frequently Asked Questions about llm-inference

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
What is the best way to serve a large language model for high-throughput inference?▼

The best way to serve large language models depends on your hardware, but using optimized engines like vLLM or TensorRT-LLM with continuous batching maximizes throughput and reduces latency for production APIs.

How does vLLM compare to llama.cpp for local LLM deployment?▼

vLLM excels at high-throughput production serving using GPUs with PagedAttention, whereas llama.cpp is optimized for local deployment and edge devices, supporting quantization formats like GGUF for CPU and Apple Silicon.

Can I use Ollama and TGI for customer-facing LLM applications?▼

Yes, TGI is built for production-level serving with features like continuous batching, while Ollama is generally better suited for local development. Selecting between them depends on your specific latency and scalability requirements.

How do I choose an inference engine for my specific GPU setup?▼

Choosing an inference engine requires evaluating your GPU capabilities against engine strengths: vLLM and TensorRT-LLM maximize GPU throughput, while llama.cpp provides flexibility across CPU, GPU, and Apple Silicon environments.

Why does LLM inference run out of memory and how can I optimize it?▼

LLM inference runs out of memory due to inefficient KV cache management, which you can optimize by applying techniques like PagedAttention, continuous batching, and model quantization to reduce the memory footprint.

Related Skills

Explore More Software Engineering Skills→
pmcdowallpmcdowall

gguf-quantization

Convert Hugging Face models into GGUF format with 2-8 bit quantization.

Community
Advanced
afel6afel6

serving-llms-vllm

Deploy LLM serving with vLLM and OpenAI-compatible endpoints.

Community
Advanced
ricablericable

serving-llms-vllm

Serve large language models with vLLM high-throughput inference and OpenAI-compatible endpoints.

Community
Advanced
lengoctuong2005lengoctuong2005
5

serving-llms-vllm

Serve LLMs with optimized throughput via OpenAI-compatible endpoints and quantization options using the vLLM package.

Community
Advanced