skills.rest
yo-steven
yo-stevenCOMMUNITY
Explore all skills by this author →
GitHub

ad-accuracy-debug

Debug TensorRT-LLM AutoDeploy accuracy regressions by reproducing evaluator sampling and isolating components.

Updated May 23, 2026
Search Tags:#tensorrt-llm#autodeploy#accuracy debugging#fp8 quantization#sharding correctness#evaluation harness#kernel investigation
One-click install
npx skills add https://github.com/yo-steven/skills-exploration-20260522 --skill ad-accuracy-debug
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ad-accuracy-debug
Source: https://github.com/yo-steven/skills-exploration-20260522/tree/main/skills/TensorRT-LLM/ad-accuracy-debug
Command: npx skills add https://github.com/yo-steven/skills-exploration-20260522 --skill ad-accuracy-debug

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps you debug when an AutoDeploy TensorRT-LLM model’s evaluation accuracy drops sharply versus a known-good reference, without knowing the root cause.

Core Features & Use Cases

  • Phase-based accuracy triage: validate the eval harness, reproduce on a small sample, and classify the error pattern before deep investigation.
  • Targeted isolation knobs: simplify configuration (e.g., TP/world_size, transforms, multi-streaming) and switch compile backends (CUDA graphs vs torch-simple) to narrow the failing component.
  • Root-cause investigation paths: cover common categories like quantization/FP8 or wrapper assumptions, and sharding issues that only appear at world_size > 1.
  • Actionable outputs: produce an identified root cause, a minimal reproducer approach, and concrete next steps for a code/config fix.

Quick Start

Use the ad-accuracy-debug Skill to compare an AutoDeploy model run against a PyTorch backend reference on the same eval task, then run a 50–100 sample diagnostic that reproduces the evaluator’s exact prompt formatting.

Frequently Asked Questions about ad-accuracy-debug

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I debug TensorRT-LLM AutoDeploy accuracy regressions on MMLU or GSM8K?▼

Debug AutoDeploy accuracy regressions by reproducing the evaluator’s exact prompt formatting and sampling behavior, then comparing the output against a PyTorch backend reference to classify the failure pattern.

Why does my TensorRT-LLM evaluation score drop when using FP8 quantization?▼

FP8 quantization can cause evaluation score drops due to wrapper assumption mismatches. Isolate the offending component by switching compile backends, simplifying transforms, and running targeted root-cause investigation paths.

What is the best way to isolate sharding correctness issues in TensorRT-LLM AutoDeploy?▼

Isolate sharding correctness issues by adjusting isolation knobs like TP/world_size and multi-streaming configurations. This narrows the failing component when regressions only appear at world_size greater than 1.

How do I reproduce an evaluator's exact prompt formatting for TensorRT-LLM accuracy debugging?▼

Reproduce the evaluator’s exact prompt formatting by running a 50–100 sample diagnostic that compares the AutoDeploy model run against a PyTorch backend reference on the same evaluation task.

Can I use AutoDeploy accuracy debugging for models outside of TensorRT-LLM backends?▼

AutoDeploy accuracy debugging applies specifically to TensorRT-LLM evaluation scenarios where the eval score is meaningfully below a PyTorch backend or published baseline across tasks like MMLU and GSM8K.

What should I do if TensorRT-LLM AutoDeploy accuracy triage does not identify the root cause?▼

If initial triage does not identify the root cause, perform structured investigation for quantization, kernel-wrapper assumptions, and sharding-related correctness to produce a minimal reproducer and concrete next steps.

Related Skills

Explore More Software Engineering Skills→
haolangehaolange
15

capture-repro

Creates deterministic capture baselines and environment snapshots for runtime debugging.

Community
Advanced
jonathanrtuckjonathanrtuck
1

kernel-auditor

Audit kernel source files for bugs using a six-category checklist and TDD.

Community
Advanced
qcmuuqcmuu

nemo-evaluator-sdk

Evaluate LLM endpoints across standardized benchmarks using YAML-configured multi-backend setups.

Community
Advanced
ovachieverovachiever
52

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced