skills.rest
hochoa13
hochoa13COMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT-LLM on NVIDIA GPUs.

Updated Mar 13, 2026
Search Tags:#performance optimization#production deployment#quantization#llm inference#tensorrt-llm#nvidia gpu
One-click install
npx skills add https://github.com/hochoa13/Asistente-Harlest --skill tensorrt-llm-hochoa13
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/hochoa13/Asistente-Harlest/tree/main/skills/mlops/inference/tensorrt-llm
Command: npx skills add https://github.com/hochoa13/Asistente-Harlest --skill tensorrt-llm-hochoa13

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill optimizes Large Language Model (LLM) inference for production environments, significantly reducing latency and increasing throughput on NVIDIA GPUs.

Core Features & Use Cases

  • High-Performance Inference: Achieve 10-100x faster inference speeds compared to standard frameworks like PyTorch.
  • Production Deployment: Ideal for serving LLMs at scale with features like quantization (FP8, INT4), in-flight batching, and multi-GPU scaling.
  • Use Case: Deploying a customer-facing chatbot that requires near real-time responses. This Skill ensures the LLM can handle thousands of requests per second with minimal delay, even with large models.

Quick Start

Use the tensorrt-llm skill to optimize and serve the 'meta-llama/Meta-Llama-3-8B' model for high-throughput inference.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I accelerate LLM inference on NVIDIA GPUs for production?▼

To accelerate LLM inference on NVIDIA GPUs, optimize your models with TensorRT-LLM to achieve 10-100x faster speeds than PyTorch. This Skill enables high-throughput production serving using in-flight batching and multi-GPU scaling.

What's the best way to reduce LLM latency for customer-facing chatbots?▼

Reduce LLM latency for customer-facing chatbots by deploying models with TensorRT-LLM. It supports near real-time response generation and handles thousands of requests per second with minimal delay, even for large models.

Can I use FP8 or INT4 quantization for LLM inference?▼

Yes, you can use FP8 or INT4 quantization for LLM inference. This Skill supports production deployment scenarios with quantization to maximize throughput and lower latency on NVIDIA GPUs.

Does TensorRT-LLm work with PyTorch models?▼

Yes, TensorRT-LLM works with PyTorch models. It requires PyTorch as a dependency and significantly accelerates inference speeds beyond standard PyTorch frameworks for production environments.

How does in-flight batching improve LLM throughput?▼

In-flight batching improves LLM throughput by dynamically managing concurrent requests during generation. TensorRT-LLM uses this technique to maximize hardware utilization on NVIDIA GPUs for high-performance production inference.

Related Skills

Explore More Software Engineering Skills→
KappTech88KappTech88

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM quantization and batching.

Community
Advanced
Z43LZ43L

tensorrt-llm

Deploys TensorRT-LLM to optimize LLM inference on NVIDIA GPUs.

Community
Advanced
ovachieverovachiever
52

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
BagelHoleBagelHole
46

vllm-server

Deploy and manage vLLM for high-throughput LLM inference with OpenAI-compatible API endpoints.

Community
Advanced