skills.rest
Supporter09
Supporter09COMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Configure TensorRT-LLM inference with quantization, batching, and multi-GPU serving.

Updated May 4, 2026
Search Tags:#batching#quantization#low latency#llm inference#tensorrt-llm#nvidia gpu#multi-gpu
One-click install
npx skills add https://github.com/Supporter09/Face_Anti_Spoofing_Biometric --skill tensorrt-llm-supporter09
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/Supporter09/Face_Anti_Spoofing_Biometric/tree/main/.claude/skills/tensorrt-llm
Command: npx skills add https://github.com/Supporter09/Face_Anti_Spoofing_Biometric --skill tensorrt-llm-supporter09

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

It removes the bottlenecks of slow, memory-heavy large language model inference by making NVIDIA GPU serving faster, lower latency, and more scalable.

Core Features & Use Cases

  • High-Performance Inference: Accelerate LLM generation with TensorRT compilation, in-flight batching, paged KV cache, and CUDA graphs.
  • Production Deployment: Serve models through an OpenAI-compatible API with health checks, metrics, authentication, and autoscaling support.
  • Multi-GPU Scaling: Run large models across multiple GPUs or nodes using tensor, pipeline, and expert parallelism.
  • Use Case: A team deploying a 70B model on H100s can use this Skill to reduce latency, improve throughput, and configure quantization and batching for production traffic.

Quick Start

Ask for a TensorRT-LLM deployment plan or inference configuration for your NVIDIA GPU model, including quantization, batching, and multi-GPU serving settings.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference on NVIDIA GPUs for production traffic?▼

Optimize LLM inference by applying TensorRT compilation, in-flight batching, paged KV cache, and CUDA graphs to reduce latency and increase throughput on NVIDIA GPUs. This Skill configures these features to make large model serving faster and more scalable.

Can I run a 70B model across multiple H100 GPUs?▼

Run large models across multiple H100 GPUs or nodes using tensor, pipeline, and expert parallelism. This multi-GPU scaling approach reduces latency and improves throughput for models ranging from 7B to 405B parameters.

What's the best way to reduce memory usage during LLM deployment?▼

Reduce memory usage during LLM deployment by configuring FP8 or INT4 quantization and paged KV cache. These configurations improve memory efficiency while maintaining high-performance inference on NVIDIA GPUs.

Does TensorRT-LLM support an OpenAI-compatible API for serving?▼

TensorRT-LLM supports production deployment through an OpenAI-compatible API with health checks, metrics, authentication, and autoscaling support. This allows seamless integration of optimized LLM inference into existing application architectures.

How does in-flight batching improve LLM generation throughput?▼

In-flight batching improves LLM generation throughput by dynamically managing incoming requests on NVIDIA GPUs. Combined with TensorRT compilation and CUDA graphs, it removes bottlenecks and maximizes memory efficiency during high-volume production serving.

When do I need to configure TensorRT-LLM for my large language model?▼

Configure TensorRT-LLM when you need to overcome slow, memory-heavy inference bottlenecks by accelerating generation on NVIDIA GPUs. It is essential for production serving, quantized deployment, and multi-GPU scaling scenarios for 7B to 405B parameter models.

Related Skills

Explore More Software Engineering Skills→
SignmanalSignmanal
1

tensorrt-llm

Optimize NVIDIA GPU inference for large language models with quantization and parallelism.

Community
Advanced
superfhpsuperfhp

serving-llms-vllm

Serve LLMs with OpenAI-compatible endpoints using vLLM.

Community
Advanced
AveerAveer
1

zeus-llm

Orchestrate deployment and configuration of local LLM inference servers for OpenCode.

Community
Intermediate
JamesFincherJamesFincher

tensorrt-llm

Optimizes LLM inference serving on NVIDIA GPUs using TensorRT-LLM with quantization and parallelism options.

Community
Advanced