trt-onnx-quickstart

Convert ONNX models to TensorRT engines and verify numerical parity.

13.2k|2.4k|Updated May 2, 2019

Search Tags:#machine learning #gpu #inference #onnx #tensorrt

Ready to run

npx skills add https://github.com/NVIDIA/TensorRT --skill trt-onnx-quickstart

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: trt-onnx-quickstart
Source: https://github.com/NVIDIA/TensorRT/tree/main/.agents/skills/trt-onnx-quickstart
Command: npx skills add https://github.com/NVIDIA/TensorRT --skill trt-onnx-quickstart

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt, python, polygraphy, onnxruntime, onnxsim, onnx_graphsurgeon, optimum-onnx, sentence-transformers, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill enables developers to convert ONNX models to TensorRT, ensuring efficient and accurate inference on NVIDIA GPUs.

Core Features & Use Cases

ONNX to TensorRT Conversion: Build and verify TensorRT engines from ONNX models for high-performance inference.
Numerical Parity Check: Ensures that TensorRT engines match the outputs of ONNX Runtime within specified tolerances.
Dynamic Shapes and Multi-Input Support: Handles multi-input models and dynamic shapes, including batch size and sequence length variations.

Quick Start

Run the trt-onnx-quickstart skill with the Hugging Face model ID or ONNX file to generate a TensorRT engine and verify its numerical parity against ONNX Runtime.

Frequently Asked Questions about trt-onnx-quickstart

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I convert ONNX models to TensorRT for GPU inference?▼

To convert ONNX models to TensorRT for GPU inference, you can use a conversion process that builds optimized engines and verifies numerical parity against ONNX Runtime. This requires an NVIDIA GPU, Python, and Polygraphy.

Does TensorRT support dynamic shapes and multi-input ONNX models?▼

TensorRT supports dynamic shapes and multi-input ONNX models by handling variations in batch size and sequence length. The conversion process builds engines that accommodate these dimensions during inference.

How do I verify numerical accuracy when converting ONNX to TensorRT?▼

You verify numerical accuracy during ONNX to TensorRT conversion by performing a numerical parity check. This ensures the generated TensorRT engine matches ONNX Runtime outputs within specified tolerances.

Do I need an NVIDIA GPU to run TensorRT engines?▼

An NVIDIA GPU is required to run TensorRT engines, as TensorRT is specifically designed to optimize and execute inference on NVIDIA hardware. You also need TensorRT and Python installed.

What is the best way to optimize ONNX inference performance?▼

The best way to optimize ONNX inference performance is converting the model to a TensorRT engine. This process leverages NVIDIA GPU hardware capabilities for high-performance execution while maintaining output accuracy.