skills.rest
handsomelong922
handsomelong922COMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Updated Apr 3, 2026
Search Tags:#production#llm#gpu#quantization#inference#parallelism#tensorrt
One-click install
npx skills add https://github.com/handsomelong922/my-codex-skills --skill tensorrt-llm-handsomelong922
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/handsomelong922/my-codex-skills/tree/main/skills/tensorrt-llm
Command: npx skills add https://github.com/handsomelong922/my-codex-skills --skill tensorrt-llm-handsomelong922

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Optimizes LLM inference on NVIDIA GPUs using TensorRT-LLM.

Core Features & Use Cases

  • Quantization support: FP8/INT4 enabling faster inference and memory savings for production deployments.
  • Multi-GPU & multi-node scaling: Tensor parallelism, pipeline parallelism, and expert parallelism for large models.
  • Production-grade serving: OpenAI-compatible API, built-in metrics, and automatic model compilation for quick deployment.

Quick Start

Install the needed runtime and start trtllm-serve with your target model to begin serving.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference on NVIDIA GPUs for production deployments?▼

TensorRT-LLM optimizes LLM inference on NVIDIA GPUs for production deployments requiring maximum throughput and low latency. It applies quantization, multi-GPU scaling, and OpenAI-compatible serving APIs to accelerate model execution.

What's the best way to scale large language models across multiple GPUs?▼

Scale large language models across multiple GPUs using tensor parallelism, pipeline parallelism, and expert parallelism. This multi-GPU and multi-node scaling approach efficiently distributes large models to achieve faster inference throughput.

Does TensorRT-LLM support INT4 and FP8 quantization for reducing memory usage?▼

Yes, TensorRT-LLM supports FP8 and INT4 quantization for faster inference and memory savings. These quantization techniques enable efficient large model deployment on NVIDIA GPUs in production environments.

Can I serve models with an OpenAI-compatible API using TensorRT-LLM?▼

Yes, you can serve models with an OpenAI-compatible API using the trtllm-serve serving API. It provides production-grade serving with built-in metrics and automatic model compilation for quick deployment.

Do I need CUDA-enabled GPUs and TensorRT runtime to run LLM inference?▼

Yes, you need CUDA-enabled GPUs, TensorRT runtime, and compatible model formats to run LLM inference. These prerequisites ensure the environment can fully utilize NVIDIA hardware acceleration for optimized execution performance.

How do I start serving a model with trtllm-serve?▼

Install the needed runtime and start trtllm-serve with your target model to begin serving. The process includes automatic model compilation, enabling quick deployment with production-grade serving capabilities and built-in metrics.

Related Skills

Explore More Software Engineering Skills→
GregsGreyCodeGregsGreyCode
6

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT.

Community
Advanced
gagan114662gagan114662

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM quantization and batching.

Community
Advanced
jayll1303jayll1303
17

tensorrt-llm

Convert HuggingFace checkpoints into optimized TensorRT-LLM engines with quantization.

Community
Advanced
lxt070409lxt070409
1

serving-llms-vllm

Automate high-throughput LLM inference with vLLM and OpenAI-compatible endpoints.

Community
Advanced