skills.rest
uv-xiao
uv-xiaoCOMMUNITY
Explore all skills by this author →
Website𝕏@youwei_xiaoGitHub

uv-tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT-LLM on NVIDIA GPUs.

1|Updated Feb 6, 2026
Search Tags:#performance optimization#quantization#serving#llm inference#tensorrt-llm#nvidia gpu
One-click install
npx skills add https://github.com/uv-xiao/pkbllm --skill uv-tensorrt-llm
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: uv-tensorrt-llm
Source: https://github.com/uv-xiao/pkbllm/tree/main/knowledge/ML/serving/tensorrt-llm
Command: npx skills add https://github.com/uv-xiao/pkbllm --skill uv-tensorrt-llm

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill optimizes Large Language Model (LLM) inference for production environments on NVIDIA GPUs, drastically reducing latency and increasing throughput compared to standard frameworks like PyTorch.

Core Features & Use Cases

  • High-Performance Inference: Achieve 10-100x faster inference speeds for LLMs on NVIDIA hardware.
  • Production Deployment: Ideal for serving models with features like quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.
  • Use Case: Deploying a Llama 3-70B model for a real-time customer support chatbot, requiring sub-100ms response times even under heavy load.

Quick Start

Use the uv-tensorrt-llm skill to serve the meta-llama/Meta-Llama-3-8B model with tensor parallelism across 4 GPUs.

Frequently Asked Questions about uv-tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I accelerate LLM inference on NVIDIA GPUs for production deployment?▼

Accelerate LLM inference on NVIDIA GPUs using TensorRT-LLM to achieve high throughput and low latency. It enables 10-100x faster inference speeds compared to standard PyTorch by leveraging advanced features like FP8/INT4 quantization.

What is in-flight batching and does TensorRT-LLM support it for large language models?▼

In-flight batching is a technique to dynamically manage incoming requests for higher throughput. TensorRT-LLM supports in-flight batching along with FP8/INT4 quantization and multi-GPU scaling to optimize serving models like Llama and Qwen.

How do I deploy a Llama 3-70B model with sub-100ms response times under heavy load?▼

Deploy a Llama 3-70B model with sub-100ms response times by using TensorRT-LLM with multi-GPU scaling and tensor parallelism. This approach provides the high throughput and low latency required for real-time customer support chatbots.

Can I use INT4 and FP8 quantization for LLM inference with TensorRT-LLM?▼

Yes, you can use INT4 and FP8 quantization for LLM inference with TensorRT-LLM. These quantization features reduce memory footprint and increase processing speed on NVIDIA GPUs during production serving.

Does TensorRT-LLM support multi-GPU scaling for Mixtral and Qwen models?▼

TensorRT-LLM supports multi-GPU scaling for Mixtral and Qwen models using tensor parallelism. This allows you to distribute large model inference across multiple NVIDIA GPUs to handle heavy workloads efficiently.

Why is PyTorch LLM inference slower than TensorRT-LLM for production serving?▼

PyTorch LLM inference is slower because it lacks the deep hardware-level optimizations for NVIDIA GPUs. TensorRT-LLM achieves 10-100x faster speeds by utilizing in-flight batching, quantization, and multi-GPU scaling for production serving.

Related Skills

Explore More Software Engineering Skills→
SignmanalSignmanal
1

tensorrt-llm

Optimize NVIDIA GPU inference for large language models with quantization and parallelism.

Community
Advanced
JKhyroJKhyro

serving-llms-vllm

Serve large language models with OpenAI-compatible endpoints and high-throughput inference.

Community
Advanced
Chris-Chai-MinjaeChris-Chai-Minjae

tensorrt-llm

Accelerate large language model inference on NVIDIA GPUs with FP8/INT4 quantization.

Community
Advanced
NousResearchNousResearch
227.8k

llama-cpp

Run LLM inference on CPUs, Apple Silicon, and AMD GPUs with GGUF quantization.

Official
Intermediate