skills.rest
WangWindow
WangWindowCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

ops-evaluation

Builds, installs, and benchmarks AscendC operators from ops repositories against baseline versions.

Updated Sep 15, 2026
Search Tags:#npu#cann#baseline comparison#performance benchmarking#ascendc#pybind#operator evaluation
One-click install
npx skills add https://github.com/WangWindow/CANN-BatchMatMulMaxsum --skill ops-evaluation-wangwindow
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ops-evaluation
Source: https://github.com/WangWindow/CANN-BatchMatMulMaxsum/tree/main/.agents/skills/ops-evaluation
Command: npx skills add https://github.com/WangWindow/CANN-BatchMatMulMaxsum --skill ops-evaluation-wangwindow

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, torch_npu, and includes scripts (resource) components.

What problem does it solve? Evaluating whether an optimized AscendC operator actually outperforms its baseline requires a repeatable pipeline: compiling the ops repository, installing both versions, generating PyBind bindings, and running correctness plus performance comparisons on NPU hardware. This Skill automates that entire workflow and produces a structured comparison report. ## Core Features & Use Cases - Automated Build and Install: Wraps the ops repository build.sh flow for ops-nn/cv/math/transformer and omni-ops repos, with automatic repo-type detection and .run package installation to absolute paths. - Baseline vs Evolved Comparison: Evaluates both versions in isolated subprocesses (since CANN runtime cannot switch OPP libraries in one process), measuring precision, kernel time, pipeline ratios, and bottleneck classification. - Multi-Shape Gating: Supports target and generalization shape sets with per-shape speedup aggregation and a five-state gating verdict (failed, target_regression, generalization_regression, partial_passed, fully_passed). - Use Case: After optimizing a BatchMatmulMaxSum cube operator, run the full evaluation to confirm precision passes and obtain a speedup report comparing the evolved kernel against the original baseline on an Ascend 910B device. ## Quick Start Ask the agent to build the baseline and evolved versions of your operator from the ops repository, generate PyBind bindings, and run the correctness and performance comparison to produce evaluation_results.json.

Frequently Asked Questions about ops-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate an AscendC operator against its baseline version?▼

Build both versions with build_ops.py into separate absolute install paths, generate PyBind bindings once with generate_pybind.py, then run evaluate_ops.py with the baseline and evolved paths. The script compares precision and kernel time in isolated subprocesses and writes evaluation_results.json.

How to build operators from ops-nn or ops-transformer repositories?▼

Run build_ops.py with --repo-root, --op-name, --soc, and an absolute --install-path. The script auto-detects the repo type from build.sh or the .run filename, cleans old build directories, runs build.sh, and installs the resulting .run package.

Why must baseline and evolved operators be evaluated in separate subprocesses?▼

The CANN runtime loads the OPP library once per process and cannot switch to another version afterward. evaluate_ops.py spawns one subprocess per version with a distinct ASCEND_CUSTOM_OPP_PATH so each version loads its own operator package.

Why does the build fail with ASCEND custom OPP directory not found?▼

This error occurs when --install-path is a relative path instead of an absolute path. Pass a fully qualified absolute path to build_ops.py and verify ASCEND_HOME_PATH is set before building.

Can multiple operator evaluations share the same NPU device?▼

Yes, device_lease.py provides session-level device leases and an evaluation queue lock using flock. Each evolution session binds one NPU card for its full lifecycle, and concurrent evaluations serialize through the eval lock to keep timing comparable.

What are the limitations of the python_npu_event timing backend?▼

This backend measures host-side end-to-end time including framework overhead, not pure kernel time. Results are not directly comparable with msprof kernel-level measurements, and abnormally high baseline times near evolved times indicate this overhead.

Related Skills

Explore More Software Engineering Skills→
AscendAscend
31

cann-operator-env-config

Install and configure the CANN toolkit on Ascend NPU hardware.

Official
Advanced
TongmingLAICTongmingLAIC
344

ako4all

Automate iterative GPU kernel optimization with benchmarking and correctness checks.

Official
Advanced
AscendAscend
31

triton-operator-performance-eval

Diagnose Triton kernel performance bottlenecks on Ascend NPUs using msprof data.

Official
Advanced
endorlabsendorlabs
4

endor-project-retrieval-bundle

Build a deterministic retrieval bundle for one Endor Labs project with indexed versions and dependency artifacts.

Official
Advanced