skills.rest
markwang2658
markwang2658COMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize large-language-model inference on NVIDIA GPUs with TensorRT-LLM.

31|3|Updated May 7, 2026
Search Tags:#gpu#quantization#inference#serving#fp8#tensorrt-llm#int4
One-click install
npx skills add https://github.com/markwang2658/hermes-windows-native --skill tensorrt-llm-markwang2658
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/markwang2658/hermes-windows-native/tree/main/hermes-agent/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/markwang2658/hermes-windows-native --skill tensorrt-llm-markwang2658

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

TensorRT-LLM accelerates large-language-model inference on NVIDIA GPUs by applying advanced optimizations and runtime efficiencies to deliver higher throughput and lower latency in production environments.

Core Features & Use Cases

  • FP8/INT4 quantization for faster inference and reduced memory on H100/A100+ GPUs.
  • In-flight batching, CUDA graphs, and KV cache optimizations for sustained throughput.
  • Multi-GPU scaling via tensor/pipeline/expert parallelism for very large models.
  • OpenAI-compatible serving API integration and production-ready deployment workflows.
  • Use cases include high-throughput chat servers, enterprise-grade model serving, and large-document QA pipelines.

Quick Start

Install TensorRT-LLM and start the server with a suitable tensor-parallel size and precision to your GPU setup.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I accelerate large language model inference on NVIDIA GPUs?▼

You can accelerate large language model inference on NVIDIA GPUs by applying FP8 or INT4 quantization, in-flight batching, and KV cache optimizations to sustain high throughput and low latency in production environments.

What is in-flight batching and how does it improve LLM serving throughput?▼

In-flight batching is a runtime execution technique that dynamically processes new inference requests while older requests are still generating tokens, which improves LLM serving throughput by keeping GPU resources fully utilized.

Can I use FP8 and INT4 quantization for LLM inference on A100 and H100 GPUs?▼

Yes, you can use FP8 and INT4 quantization on A100 and H100 GPUs to achieve faster LLM inference and significantly reduced memory consumption while maintaining production-grade accuracy.

How do I scale LLM inference across multiple GPUs for very large models?▼

You can scale LLM inference across multiple GPUs by leveraging tensor parallelism, pipeline parallelism, and expert parallelism to distribute model execution and accommodate very large language models.

Does this approach provide an OpenAI-compatible API for production model serving?▼

Yes, the trtllm-serve interface provides an OpenAI-compatible API, allowing seamless integration into existing production workflows and enabling scalable, enterprise-grade model serving.

What is the best way to deploy high-throughput chat servers with large language models?▼

The best way to deploy high-throughput chat servers is to combine CUDA graphs, multi-GPU scaling, and automatic model compilation to maximize LLM execution efficiency and reduce serving latency.

Related Skills

Explore More Software Engineering Skills→
m0atm0at
1

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
frankxaifrankxai
2

nvidia-nim

Configure and run NVIDIA NIM inference via OpenAI-compatible APIs.

Community
Advanced
OpenLAIROpenLAIR
1.0k

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT and multi-GPU scaling.

Official
Advanced
sheawinklersheawinkler
78

tensorrt-llm

Accelerate LLM inference on NVIDIA GPUs with FP8/INT4 quantization and multi-GPU scaling.

Community
Advanced