skills.rest
tianhao909
tianhao909COMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

1|Updated Feb 21, 2026
Search Tags:#llm optimization#inference#low latency#nvidia#high throughput#tensorrt-llm#gpu serving
One-click install
npx skills add https://github.com/tianhao909/AI-Research-SKILLs-cn --skill tensorrt-llm-tianhao909
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/tianhao909/AI-Research-SKILLs-cn/tree/main/12-inference-serving/tensorrt-llm
Command: npx skills add https://github.com/tianhao909/AI-Research-SKILLs-cn --skill tensorrt-llm-tianhao909

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve?

This Skill optimizes Large Language Model (LLM) inference for maximum throughput and lowest latency on NVIDIA GPUs, making production deployment faster and more efficient.

Core Features & Use Cases

  • High-Performance Inference: Achieve 10-100x faster inference speeds compared to standard frameworks like PyTorch.
  • Production Deployment: Ideal for serving LLMs in production environments on NVIDIA A100/H100/GB200 GPUs.
  • Advanced Optimizations: Supports quantization (FP8, INT4), in-flight batching, multi-GPU scaling, and more.
  • Use Case: Deploying a Llama 3-70B model for a real-time chatbot application where low latency and high throughput are critical.

Quick Start

Use the tensorrt-llm skill to serve the 'meta-llama/Meta-Llama-3-8B' model with tensor parallelism across 4 GPUs.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I accelerate LLM inference on NVIDIA GPUs for production serving?▼

You can accelerate LLM inference on NVIDIA GPUs using TensorRT-LLM, which applies FP8/INT4 quantization, in-flight batching, and multi-GPU scaling to achieve 10-100x faster speeds than standard PyTorch.

Can I deploy Llama and Qwen models with high throughput and low latency?▼

Yes, you can deploy models like Llama, Qwen, and Mixtral with high throughput and low latency using TensorRT-LLM optimized for NVIDIA A100, H100, or GB200 GPUs.

What is the best way to scale large language model inference across multiple GPUs?▼

The best way to scale LLM inference across multiple GPUs is using TensorRT-LLM with tensor parallelism, enabling multi-GPU scaling to efficiently distribute workloads for real-time chatbot applications.

Does TensorRT-LLM support FP8 and INT4 quantization for faster inference?▼

Yes, TensorRT-LLM supports FP8 and INT4 quantization to optimize LLM inference, significantly reducing memory footprint and increasing throughput while maintaining low latency.

Do I need PyTorch and NVIDIA GPUs to use in-flight batching for LLM serving?▼

Yes, you need PyTorch and NVIDIA GPUs because TensorRT-LLM depends on torch and requires NVIDIA hardware to enable in-flight batching and other advanced optimization techniques.

Related Skills

Explore More Software Engineering Skills→
heysuhasheysuhas
2

tensorrt-llm

Optimize large language model inference with NVIDIA TensorRT-LLM on GPU hardware.

Community
Advanced
jayll1303jayll1303
17

tensorrt-llm

Convert HuggingFace checkpoints into optimized TensorRT-LLM engines with quantization.

Community
Advanced
zerwizzerwiz

serving-llms-vllm

Deploy high-throughput LLM inference servers with vLLM and OpenAI-compatible endpoints.

Community
Advanced
informatico-madridinformatico-madrid
1

tensorrt-llm

Optimizes LLM serving on NVIDIA hardware with batched GPU inference.

Community
Advanced