offline-GPU-fault-diagnosis

Correlate iBMC, OS, and InfoCollect logs to diagnose GPU hardware failures.

3|2|Updated Apr 24, 2026
One-click install
npx skills add https://github.com/openeuler-mirror/witty-diagnosis-agent --skill offline-gpu-fault-diagnosis-openeuler-mirror
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: offline-GPU-fault-diagnosis
Source: https://github.com/openeuler-mirror/witty-diagnosis-agent/tree/main/skills/offline-GPU-fault-diagnosis
Command: npx skills add https://github.com/openeuler-mirror/witty-diagnosis-agent --skill offline-gpu-fault-diagnosis-openeuler-mirror

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This skill resolves the ambiguity and complexity of diagnosing GPU failures in offline server environments by correlating disparate hardware and software logs into a clear, actionable root cause.

Core Features & Use Cases

  • Multi-Source Correlation: Automatically aligns iBMC hardware events, OS kernel messages, and NVIDIA driver XID errors to reconstruct the exact fault propagation chain.
  • Physical Localization: Pinpoints the specific GPU slot and BDF address, distinguishing between genuine hardware failures and software-induced pseudo-faults.
  • Use Case: When a server reports a GPU fallen off the bus, use this skill to determine if the issue is a physical PCIe link degradation, a power supply fluctuation, or a driver-level software hang.

Quick Start

Use the offline-GPU-fault-diagnosis skill to analyze the logs located in /tmp/server-logs and generate a root cause report.

Frequently Asked Questions about offline-GPU-fault-diagnosis

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I diagnose a GPU that has fallen off the bus using offline server logs?

To diagnose a GPU fallen off the bus, you can analyze offline server logs like iBMC, OS messages, and InfoCollect data. This process correlates hardware and software events to identify whether the root cause is physical PCIe link degradation, power fluctuation, or a driver-level software hang.

What is the best way to determine if an XID error indicates a hardware failure or a software issue?

Determining if an XID error is a hardware or software issue requires multi-dimensional log correlation. By aligning NVIDIA driver XID errors with OS kernel messages and iBMC hardware events, you can validate the actual root cause and distinguish genuine hardware failures from software-induced pseudo-faults.

Can I pinpoint the specific physical slot of a failed GPU from offline collected data?

Yes, you can pinpoint the specific physical slot of a failed GPU from offline collected data. By analyzing InfoCollect data and OS logs, the diagnosis process identifies the exact GPU slot and BDF address to locate the physical source of hardware instability or driver faults.

Does multi-source log correlation help with PCIe link instability and ECC memory faults?

Yes, multi-source log correlation helps with PCIe link instability and ECC memory faults. Aligning iBMC hardware events with OS kernel messages reconstructs the exact fault propagation chain, validating hardware versus software root causes for these specific GPU hardware and driver failures.

How do I analyze iBMC hardware events and OS kernel messages to validate a GPU driver hang?

To analyze iBMC hardware events and OS kernel messages for a GPU driver hang, align the hardware events with OS messages and InfoCollect data. This multi-dimensional correlation validates the root cause by reconstructing the fault propagation chain across the server environment.