skills.rest
BBuf
BBufCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

llm-pipeline-analysis

Inspect LLM torch profiler traces at forward-pass, layer, and kernel level.

783|67|Updated Apr 1, 2026
Search Tags:#perfetto#trace analysis#llm inference#kernel analysis#DeepSeek#torch profiler#layer timing
One-click install
npx skills add https://github.com/BBuf/AI-Infra-Auto-Driven-SKILLS --skill llm-pipeline-analysis
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-pipeline-analysis
Source: https://github.com/BBuf/AI-Infra-Auto-Driven-SKILLS/tree/main/skills/llm-pipeline-analysis
Command: npx skills add https://github.com/BBuf/AI-Infra-Auto-Driven-SKILLS --skill llm-pipeline-analysis

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

Whole-trace profiler summaries are too coarse to explain where time goes inside an LLM forward pass. This Skill parses Chrome-trace JSON files from torch.profiler, detects layer boundaries via anchor kernels, and produces per-layer timing tables, kernel breakdowns, and Perfetto navigation ranges.

Core Features & Use Cases

  • Per-Layer Timeline Analysis: Group GPU kernels into forward passes and layers, classify layers by type (C4_LIGHT, C128_HEAVY, HASH, FULL_ATTN), and compare cold-start versus steady-state passes.
  • Kernel-Level Breakdown: Dump every kernel in a specific layer with category, duration, and input dims, export as text, compute-flow tables, or JSON, and diff two layers side-by-side.
  • Perfetto Time Mapping: Convert trace timestamps into Perfetto-relative time ranges for any forward pass or layer so you can jump directly to the region of interest.
  • Use Case: You profiled a DeepSeek-V4 decode run on TP=8 and need to know whether C128_HEAVY layers dominate. Run the timeline analyzer to find the steady-state pass, then break down a representative layer to see MLA, MoE, GEMM, and NCCL shares.

Quick Start

Analyze my torch profiler trace at /path/to/TP-0.trace.json.gz with the model config at /path/to/config.json and show me the per-layer timing breakdown for the steady-state forward pass.

Frequently Asked Questions about llm-pipeline-analysis

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I analyze per-layer timing in a torch profiler trace?▼

Run layer_timeline_analyzer.py with --trace pointing to your Chrome-trace JSON file and --config pointing to the model config.json. Use --show-all-passes to see all forward passes, then --fwd-pass N for a detailed per-layer breakdown of one pass.

How to find which layers are slowest in an LLM forward pass?▼

The timeline analyzer prints per-layer wall-clock and sum-duration tables plus cluster statistics grouped by layer type such as C4_LIGHT, C128_HEAVY, and HASH. The cluster table shows which layer type dominates and breaks down MLA, MoE, GEMM, and NCCL percentages.

What trace format does torch profiler analysis require?▼

The scripts accept Chrome-trace JSON format produced by torch.profiler, either plain .json or gzipped .json.gz. The trace must contain GPU kernel events and a recognizable layer-boundary anchor kernel, auto-detected or set via --anchor-kernel.

Which models are supported for layer boundary detection?▼

Built-in profiles cover DeepSeek-V4 CSA/HCA (anchor mhc_post_tilelang, 2 blocks per layer) and DeepSeek-V3 MLA (anchor flash_fwd_mla_combine). Other models use the generic profile with --anchor-kernel, and profiles are auto-inferred from config.json fields like compress_ratios and kv_lora_rank.

Why does layer boundary auto-detection fail on my trace?▼

Auto-detection fails when no candidate anchor kernel (mhc_post_tilelang, flash_fwd_mla_combine, RMSNorm, AllReduce) appears at least four times in the trace. Fix it by passing --profile generic together with --anchor-kernel naming a kernel that repeats once per layer.

Can I compare kernels between two layers in the same trace?▼

Yes, layer_kernel_breakdown.py supports --compare-layer to dump two layers side-by-side and print a kernel diff showing types unique to each layer. With --format json the comparison is exported as one machine-readable document with primary, comparison, and kernel_diff sections.

Related Skills

Explore More Software Engineering Skills→
sgl-projectsgl-project
33.0k

llm-torch-profiler-analysis

Analyzes torch.profiler traces from LLM serving frameworks into kernel, overlap, and fusion tables.

Official
Advanced
Nabilhassan12345Nabilhassan12345

sglang-diffusion-ako4all-kernel

Optimizes existing SGLang diffusion kernels using AKO4ALL harnesses with benchmarking and validation workflows.

Community
Advanced
kali20gakkikali20gakki
7

op-mfu-calculator

Compute MFU for matmul/GEMM operators from dimensions, time, and peak FLOPs.

Community
Intermediate
NVIDIANVIDIA
2.8k

perf-nsight-compute-analysis

Analyze Nsight Compute .ncu-rep reports to classify CUDA kernel bottlenecks.

Official
Advanced