skills.rest
NVIDIA
NVIDIAOFFICIAL
Explore all skills by this author →
WebsiteGitHub

doca-dpa-hl-tracer

Capture and decode DPA-side execution traces at the programming-events layer on BlueField devices.

3.2k|370|Updated Feb 25, 2026
Search Tags:#tracing#Debugging#doca#bluefield#dpa#rdma#dpacc
One-click install
npx skills add https://github.com/NVIDIA/skills --skill doca-dpa-hl-tracer
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: doca-dpa-hl-tracer
Source: https://github.com/NVIDIA/skills/tree/main/skills/doca-dpa-hl-tracer
Command: npx skills add https://github.com/NVIDIA/skills --skill doca-dpa-hl-tracer

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

When a DPA kernel on a BlueField device returns wrong results or shows unexpected latency while host-side completions look clean, developers lack visibility into what the DPA is actually doing. This Skill guides agents through using the doca_dpa_hl_tracer CLI to capture, decode, and interpret DPA-side execution traces at the programming-events layer rather than raw cycle counts.

Core Features & Use Cases

  • Capture mode selection: Choose between TRACE (full per-event capture, higher overhead) and CRIT (critical-events-only, lower overhead) modes based on the bug-versus-overhead tradeoff.
  • JSON config tuning: Configure file-size limits, file_size_limit_policy, and thread priorities/core affinities so the capture does not perturb the workload more than the bug being investigated.
  • Decode against matching ELF: Decode binary traces against the exact DPACC-built DPA-side ELF, with SHA verification to prevent wrong-symbol output from mismatched builds.
  • Layered error diagnosis: Walk an eight-layer error taxonomy (install, device-binding, image-instrumented, capture-window, decode, overhead-saturated, version, cross-cutting) to diagnose empty or noisy captures.
  • Use Case: A developer's DPA kernel produces wrong results with clean host-side completions; the agent captures a CRIT-mode trace, decodes it against the matching ELF, and identifies missing sync-point events as evidence for the host-side doca-dpa debug ladder.

Quick Start

Ask your agent to capture a DPA-side trace with doca_dpa_hl_tracer to diagnose why your DPA kernel returns wrong results while host completions look clean.

Frequently Asked Questions about doca-dpa-hl-tracer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I trace DPA kernel execution on a BlueField device?▼

Use doca_dpa_hl_tracer with a mandatory --device flag, a capture mode (TRACE or CRIT), a JSON config file, and an output file path. The DPA workload must already be running via the host-side doca-dpa lifecycle, since the tracer observes events rather than driving the workload.

What is the difference between TRACE and CRIT capture modes?▼

TRACE captures full per-event detail across all event classes with higher overhead that can shift kernel timing, while CRIT captures only critical events like errors and lifecycle transitions with minimal perturbation. Start with CRIT and widen to TRACE only when the bug demands per-event detail.

Why does my DPA trace decode show noise or wrong symbols?▼

Decode failures almost always mean the ELF passed to --elf-file differs from the build that produced the trace. Recompute the ELF SHA before decoding and compare it with the capture-time SHA; on mismatch, abort decode and locate the exact capture-time ELF rather than rebuilding.

Why is my captured DPA trace file empty?▼

Empty traces come from layered causes: missing binary or DPA Tools component, device-binding failure, a DPA image built without instrumentation hooks, or a capture window that missed the workload phase. Walk the error taxonomy in order from install through capture-window before assuming a deeper fault.

Can doca_dpa_hl_tracer be used for production DPA monitoring?▼

No, the tracer is a diagnostic capture tool, not a production observability surface, and TRACE mode measurably perturbs kernel timing. For continuous production telemetry of DPA workloads, route to the doca-telemetry skill instead.

When should I use raw cycle profiling instead of the high-level tracer?▼

Use raw cycle profiling when the performance question is at per-instruction granularity, such as cache misses or branch behavior. The high-level tracer answers questions at the DPA programming-events layer like sync-point dwell and comm-call latency, not individual cycles.

Related Skills

Explore More Software Engineering Skills→
jrhueiuengjrhueiueng

dpa-review

Review DPAs against a privacy playbook with direction-aware clause analysis.

Community
Advanced
ROCmROCm
260

debug-flydsl-kernel

Diagnose incorrect FlyDSL GPU kernel outputs, NaNs, infs, and crashes.

Official
Advanced
air-gappedair-gapped
5

nvidia-datacenter-bringup

Automate NVIDIA datacenter GPU bring-up on Ubuntu 24.04 with health validation.

Official
Advanced
guardiatechnologyguardiatechnology

kata-pov-value-track

Log PoV review cycles with metric readings and telemetry SHA evidence.

Official
Advanced