skills.rest
SethyPagna
SethyPagnaCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize large language model inference on NVIDIA GPUs with TensorRT-LLM.

Updated May 14, 2026
Search Tags:#quantization#model serving#llm inference#tensorrt-llm#multi-gpu#nvidia gpus
One-click install
npx skills add https://github.com/SethyPagna/Secretary-Jarvis --skill tensorrt-llm-sethypagna
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/SethyPagna/Secretary-Jarvis/tree/main/src/capabilities/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/SethyPagna/Secretary-Jarvis --skill tensorrt-llm-sethypagna

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

TensorRT-LLM removes the performance bottlenecks of standard LLM serving by turning large models into highly optimized inference workloads for NVIDIA GPUs, making them faster, cheaper, and more production-ready.

Core Features & Use Cases

  • High-throughput inference: Improve token generation speed and reduce latency for chatbots, copilots, and real-time AI services.
  • Production serving: Deploy models with OpenAI-compatible APIs, health checks, metrics, load balancing, and autoscaling support.
  • GPU optimization: Use FP8, INT4, in-flight batching, paged KV cache, speculative decoding, and multi-GPU parallelism for demanding workloads.
  • Use case: Run a Llama-class model on an A100 or H100 cluster with efficient batching, monitoring, and sharding to support user traffic at scale.

Quick Start

Use the tensorrt-llm skill to design a production-ready NVIDIA GPU serving setup for your model, including quantization, batching, parallelism, and monitoring.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference on NVIDIA GPUs for production serving?▼

LLM inference on NVIDIA GPUs is optimized through TensorRT-based compilation, FP8 and INT4 quantization, in-flight batching, and paged KV cache to maximize throughput and minimize latency for production serving.

What is the best way to scale large language model serving across multiple GPUs?▼

Multi-GPU scaling for large language model serving uses parallelism and sharding to distribute workloads, ensuring demanding user traffic is handled efficiently on A100 or H100 clusters with autoscaling support.

How does FP8 and INT4 quantization improve LLM inference speed?▼

FP8 and INT4 quantization improves LLM inference speed by reducing model memory footprint and computational overhead, allowing faster token generation while maintaining high throughput for real-time AI services.

Can I deploy an OpenAI-compatible API for my large language model on NVIDIA GPUs?▼

Yes, you can deploy an OpenAI-compatible API for your large language model on NVIDIA GPUs, integrating health checks, metrics, load balancing, and autoscaling for production-ready serving.

Does in-flight batching and paged KV cache work for high-throughput chatbot inference?▼

In-flight batching and paged KV cache work for high-throughput chatbot inference by dynamically managing concurrent requests and memory, significantly improving token generation speed and reducing latency.

When do I need TensorRT-based compilation for large language model deployment?▼

TensorRT-based compilation is needed when standard LLM serving faces performance bottlenecks, requiring optimized inference workloads with monitoring, speculative decoding, and multi-GPU parallelism for demanding traffic.

Related Skills

Explore More Software Engineering Skills→
ovachieverovachiever
52

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
hung-phanhung-phan
1

ray-serve

Deploy ML models to Ray Serve with autoscaling and request batching.

Community
Advanced
attentiondotnetattentiondotnet

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
davpatel605-beepdavpatel605-beep

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM quantization and batching.

Community
Advanced