profiling-collect

Collect NPU profiling data from remote Ascend devices via SSH and patch-based instrumentation.

14|5|Updated Mar 30, 2026
One-click install
npx skills add https://github.com/Ascend/MindIE-SD --skill profiling-collect-ascend
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: profiling-collect
Source: https://github.com/Ascend/MindIE-SD/tree/main/.agents/skills/profiling-collect
Command: npx skills add https://github.com/Ascend/MindIE-SD --skill profiling-collect-ascend

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires paramiko, torch, torch_npu, and includes scripts (resource) and references (resource) components.

What problem does it solve? Collecting operator-level performance profiling data from remote Ascend NPU devices is error-prone: warmup steps pollute traces, multi-card runs duplicate output, and missing utilization metrics silently invalidate downstream bottleneck analysis. This Skill standardizes the entire collection workflow so the captured data is always valid input for profiling analysis. ## Core Features & Use Cases - Dual collection entry points: Use collect_profile.py for mindiesd's own scripts (SSH connect, run profiler, compress, download) or collect_patch_template.py to wrap third-party frameworks like LightX2V, DiffSynth-Engine, and vLLM-Omni with minimal-invasion patches launched via torchrun. - Fail-closed output gate: check_output.py validates that every capture contains execution-order columns and the four utilization ratio families (vec/mac/mte2/mte3), rejecting N/A-only results before they reach downstream analysis. - Warmup and isolation discipline: Enforces warmup outside the profiler (5 steps default, 10+ for JIT compile scenarios), rank0-only collection, and per-run profile directory isolation for parallel captures. - Use Case: You need kernel-level timing for a vLLM-Omni HTTP service. The Skill wraps the transformer forward, warms up via real HTTP requests, profiles one forward pass on rank0, packages ASCEND_PROFILER_OUTPUT as tar.gz, and transfers it back for analysis. ## Quick Start Ask the AI to collect profiling data from your remote Ascend server for a given inference script, and it will run the profiler with proper warmup, validate the output, and return the compressed trace archive.

Frequently Asked Questions about profiling-collect

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I collect profiling data from a remote Ascend NPU server?

Run collect_profile.py with SSH credentials, container name, and the remote inference script. It connects via paramiko, sources the CANN environment, runs the script with profiling enabled, compresses the output directory, and downloads the tar.gz archive locally.

How to profile third-party frameworks like vLLM or LightX2V on Ascend NPU?

Use the collect_patch_template.py approach: wrap the framework's top-level inference method with a minimal patch that enables torch_npu.profiler only after warmup, then launch via torchrun. Only rank0 collects, and output matches the standard ASCEND_PROFILER_OUTPUT format.

Why is the first profiling step abnormally slow on Ascend NPU?

The first step usually includes JIT compilation overhead because warmup was not completed outside the profiler. Run at least 5 warmup steps before enabling the profiler, or 10 or more for MindieSDBackend compile scenarios, otherwise downstream analysis flags WARMUP_NOT_STRIPPED.

What profiling metrics are required for operator bottleneck analysis?

Collection must use task-time l1 with task-based AIC mode and PipeUtilization metrics. Without these, the vec, mac, mte2, and mte3 ratio columns are all N/A, making memory-bound versus compute-bound classification impossible.

Can I run multiple profiling jobs in parallel on the same host?

Yes, but each run must use an isolated output directory named by model and configuration. The default profile directory is relative to the working directory, so parallel runs sharing it will overwrite each other's kernel_details.csv files.

Why does profiling fail inside Docker containers on Ascend servers?

Common causes include HCCL port conflicts with existing processes, container /tmp differing from the host /tmp so artifacts must be retrieved via docker cp, and Windows line endings in uploaded scripts. Set HCCL_NPU_SOCKET_PORT_RANGE and convert files to LF before uploading.