offline-NPU-fault-diagnosis

Diagnose NPU hardware failures by analyzing offline server logs including iBMC, OS Messages, and InfoCollect to pinpoint physical component issues like HBM memory errors and PCIe link failures.

Correlate iBMC and OS Messages to reconstruct the fault propagation chain, determining if an Ascend NPU offline status stems from a physical PCIe link training failure or a driver-level firmware mismatch.

Identify thermal throttling and driver-firmware incompatibilities in Ascend NPU environments by correlating hardware events with kernel-level logs to expose complex fault scenarios affecting physical components.

Yes, pinpoint failures to specific NPU IDs, PCIe slots, or HBM memory modules by implementing a multi-step diagnostic workflow that correlates hardware events with kernel-level logs.

Start NPU diagnostics by analyzing logs located in the /tmp/server_logs directory, automatically aligning iBMC, OS Messages, and InfoCollect data to identify physical component failures.

Yes, offline NPU fault diagnosis works without real-time monitoring by analyzing server-side offline logs to reconstruct the exact fault propagation chain and identify physical PCIe or driver failures.