skills.rest
NVIDIA
NVIDIAOFFICIAL
Explore all skills by this author →
WebsiteGitHub

perf-host-optimization

Profile and reduce host CPU overhead in TensorRT-LLM PyTorch inference pipelines.

2.8k|332|Updated Feb 25, 2026
Search Tags:#python#optimization#profiling#line_profiler#tensorrt-llm#nsight#host-overhead
One-click install
npx skills add https://github.com/NVIDIA/skills --skill perf-host-optimization
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: perf-host-optimization
Source: https://github.com/NVIDIA/skills/tree/main/skills/TensorRT-LLM/perf-host-optimization
Command: npx skills add https://github.com/NVIDIA/skills --skill perf-host-optimization

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Profiles and optimizes host CPU overhead in TensorRT-LLM's PyTorch backend, enabling higher GPU utilization during inference.

Core Features & Use Cases

  • Automated host profiling with line_profiler (and planned nsys integration) to locate CPU bottlenecks.
  • Iterative profile-analyze-optimize-validate workflow across multiple optimization rounds.
  • Safe, incremental improvements with unit tests and results comparison to quantify gains.

Quick Start

Start the profiling loop to identify CPU bottlenecks, then apply a minimal optimization and re-profile to verify throughput gains.

Frequently Asked Questions about perf-host-optimization

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I reduce CPU overhead in TensorRT-LLM inference pipelines?▼

To reduce CPU overhead in TensorRT-LLM, profile CPU-bound hotspots using line_profiler. This identifies bottlenecks in core paths like request scheduling, enabling iterative optimizations for higher GPU utilization.

How do I profile host CPU bottlenecks in a PyTorch TensorRT-LLM deployment?▼

Profile host CPU bottlenecks by enabling line_profiler with TLLM_LINE_PROFILER_ENABLED. This targets hot paths such as _prepare_tp_inputs and _forward_step to locate inefficiencies within the PyTorch backend.

Does TensorRT-LLM profiling work with NSight and NVTX for system-level analysis?▼

Yes, TensorRT-LLM profiling supports optional NSight and NVTX integration. This provides deeper system-level context alongside line_profiler to analyze CPU overhead during inference workflows.

What is the workflow for optimizing TensorRT-LLM host overhead?▼

The workflow involves iterative profile-analyze-optimize-validate rounds. After profiling to locate bottlenecks, you apply minimal optimizations and run unit tests to quantify throughput gains safely.

Why does my TensorRT-LLM deployment have low GPU utilization during inference?▼

Low GPU utilization often stems from host CPU overhead in the PyTorch backend. Profiling with line_profiler identifies CPU-bound hotspots in request scheduling, allowing you to optimize and restore throughput.

Do I need line_profiler to optimize TensorRT-LLM CPU overhead?▼

Yes, enabling line_profiler via TLLM_LINE_PROFILER_ENABLED is required. It automates host profiling to pinpoint CPU bottlenecks, which is essential for the iterative optimization process.

Related Skills

Explore More Software Engineering Skills→
Fudan-SMI-labFudan-SMI-lab
39

torch-npu-venv-setup-cpu-base

Pin CPU-base PyTorch and install matching torch-npu and torchvision.

Official
Intermediate
yo-stevenyo-steven

perf-host-optimization

Profile and optimize TensorRT-LLM host CPU overhead with line_profiler.

Community
Advanced
NVIDIANVIDIA
2.8k

perf-nsight-systems

Profile CPU-GPU timelines, API calls, OS events, and NVTX ranges.

Official
Advanced
jralphjralph

performance-ts

Profile CPU and memory usage to identify TypeScript/JavaScript performance bottlenecks.

Community
Intermediate