uv-tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT-LLM on NVIDIA GPUs.

1|Updated Feb 6, 2026

Search Tags:#performance optimization #quantization #serving #llm inference #tensorrt-llm #nvidia gpu

Ready to run

npx skills add https://github.com/uv-xiao/pkbllm --skill uv-tensorrt-llm

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: uv-tensorrt-llm
Source: https://github.com/uv-xiao/pkbllm/tree/main/knowledge/ML/serving/tensorrt-llm
Command: npx skills add https://github.com/uv-xiao/pkbllm --skill uv-tensorrt-llm

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill optimizes Large Language Model (LLM) inference for production environments on NVIDIA GPUs, drastically reducing latency and increasing throughput compared to standard frameworks like PyTorch.

Core Features & Use Cases

High-Performance Inference: Achieve 10-100x faster inference speeds for LLMs on NVIDIA hardware.
Production Deployment: Ideal for serving models with features like quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.
Use Case: Deploying a Llama 3-70B model for a real-time customer support chatbot, requiring sub-100ms response times even under heavy load.

Quick Start

Use the uv-tensorrt-llm skill to serve the meta-llama/Meta-Llama-3-8B model with tensor parallelism across 4 GPUs.

Frequently Asked Questions about uv-tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I accelerate LLM inference on NVIDIA GPUs for production deployment?▼

Accelerate LLM inference on NVIDIA GPUs using TensorRT-LLM to achieve high throughput and low latency. It enables 10-100x faster inference speeds compared to standard PyTorch by leveraging advanced features like FP8/INT4 quantization.

What is in-flight batching and does TensorRT-LLM support it for large language models?▼

In-flight batching is a technique to dynamically manage incoming requests for higher throughput. TensorRT-LLM supports in-flight batching along with FP8/INT4 quantization and multi-GPU scaling to optimize serving models like Llama and Qwen.

How do I deploy a Llama 3-70B model with sub-100ms response times under heavy load?▼

Deploy a Llama 3-70B model with sub-100ms response times by using TensorRT-LLM with multi-GPU scaling and tensor parallelism. This approach provides the high throughput and low latency required for real-time customer support chatbots.

Can I use INT4 and FP8 quantization for LLM inference with TensorRT-LLM?▼

Yes, you can use INT4 and FP8 quantization for LLM inference with TensorRT-LLM. These quantization features reduce memory footprint and increase processing speed on NVIDIA GPUs during production serving.

Does TensorRT-LLM support multi-GPU scaling for Mixtral and Qwen models?▼

TensorRT-LLM supports multi-GPU scaling for Mixtral and Qwen models using tensor parallelism. This allows you to distribute large model inference across multiple NVIDIA GPUs to handle heavy workloads efficiently.

Why is PyTorch LLM inference slower than TensorRT-LLM for production serving?▼

PyTorch LLM inference is slower because it lacks the deep hardware-level optimizations for NVIDIA GPUs. TensorRT-LLM achieves 10-100x faster speeds by utilizing in-flight batching, quantization, and multi-GPU scaling for production serving.