troubleshoot-nvidia-gpu

Triage Nvidia GPU failures by mapping Netdata MCP metrics to signal-domain rule files.

1|Updated Apr 17, 2026
One-click install
npx skills add https://github.com/netdata/skills --skill troubleshoot-nvidia-gpu
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: troubleshoot-nvidia-gpu
Source: https://github.com/netdata/skills/tree/main/skills/troubleshoot-nvidia-gpu
Command: npx skills add https://github.com/netdata/skills --skill troubleshoot-nvidia-gpu

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

It helps you triage and remediate Nvidia GPU issues by identifying the dominant failure archetype (OOM, thermal-throttle cascade, silent degradation, interconnect straggler, or GPU hang) using Netdata signals accessed through an MCP query flow.

Core Features & Use Cases

  • Structured Nvidia Gpu triage tree: Routes an agent through domain-specific rule files aligned to the Netdata operator playbook’s signal domains (availability, thermal/power, memory, compute/utilization, interconnect, errors/Xid, plus support domains).
  • MCP-driven verification: Uses MCP calls to discover nodes, pull the last-window metrics, rank anomalies, correlate signals, and confirm whether remediation actually returns signals to expected ranges.
  • Failure-mode coverage: Targets common real-world incidents like memory exhaustion (OOM), thermal runaway with throttling, ECC-related silent corruption, PCIe/NVLink degradation patterns, and catastrophic “fallen off bus” behaviors (including XID 79).
  • Operational on-call workflow: Designed for situations where an on-call engineer needs a paging-ready, evidence-based triage path tied to Netdata alerts on a specific GPU host.

Quick Start

Use the troubleshoot-nvidia-gpu skill to diagnose a failing Nvidia GPU by querying the Netdata instance through MCP for the last 15 to 30 minutes of nvidia_smi signals, matching anomalies to the appropriate rule-file domain, and running the verification queries to confirm remediation effectiveness.

Frequently Asked Questions about troubleshoot-nvidia-gpu

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I troubleshoot an Nvidia GPU that has fallen off the bus or hung?

To troubleshoot an Nvidia GPU hang or fallen-off-bus scenario, map observed Xid errors to the Netdata operator playbook’s signal-domain triage tree using MCP queries for nvidia_smi metrics to verify remediation effectiveness.

What is the best way to diagnose silent ECC memory degradation on Nvidia GPUs?

Diagnosing silent ECC memory degradation requires querying Netdata through MCP for recent nvidia_smi signals, ranking anomalies, and cross-correlating memory errors against the operator playbook's triage tree.

How can I identify thermal runaway and throttling cascades on an Nvidia GPU?

Identify thermal runaway and throttling cascades by applying the Netdata operator playbook’s thermal/power signal domain rules to nvidia_smi metrics retrieved via MCP, correlating temperature spikes with throttling behaviors.

Does this Nvidia GPU triage workflow support NVLink and PCIe interconnect stragglers?

Yes, the Nvidia GPU triage workflow supports NVLink and PCIe interconnect stragglers by mapping interconnect degradation patterns to the Netdata operator playbook and verifying signal normalization through MCP-based metric retrieval.

How do I verify if an OOM remediation resolved an Nvidia GPU memory exhaustion issue?

Verify OOM remediation by running post-remediation MCP verification queries against the Netdata instance to confirm that nvidia_smi memory metrics and anomalies return to expected load-bearing ranges.