hyperpod-nccl

What problem does it solve?

This Skill helps diagnose and debug NCCL failures on HyperPod GPU clusters, including training hangs, AllReduce/collective-op timeouts, EFA or libfabric errors, rendezvous failures, and other related issues.

Core Features & Use Cases

• NCCL Diagnostics: Identify and diagnose NCCL failures, including timeouts, rendezvous issues, and version mismatches.
• Infrastructure Checks: Validate cluster health, security group rules, and Kubernetes node readiness.
• Hardware Health: Check GPU and EFA hardware health, including XID errors, ECC, and NVLink issues.
• Use Case: If your training job on HyperPod suddenly stops, use this Skill to identify the root cause, such as NCCL timeouts, GPU OOM, or network issues, and take appropriate actions.

Quick Start

Run the hyperpod-nccl skill to diagnose NCCL issues on your cluster.