skills.rest
haozhx23
haozhx23COMMUNITY
Explore all skills by this author →
GitHub

hyperpod-nccl

Diagnose NCCL failures on SageMaker HyperPod clusters via events, security groups, and node status.

15|9|Updated Aug 16, 2025
Search Tags:#gpu#nccl#hyperpod#cluster debugging#training diagnostics
One-click install
npx skills add https://github.com/haozhx23/HyperPod-InstantStart --skill hyperpod-nccl
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hyperpod-nccl
Source: https://github.com/haozhx23/HyperPod-InstantStart/tree/main/.claude/skills/hyperpod-nccl
Command: npx skills add https://github.com/haozhx23/HyperPod-InstantStart --skill hyperpod-nccl

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires awscli, kubectl, python3, unbuffer, jq, nccl-diagnose.sh, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill helps diagnose and debug NCCL failures on HyperPod GPU clusters, including training hangs, AllReduce/collective-op timeouts, EFA or libfabric errors, rendezvous failures, and other related issues.

Core Features & Use Cases

  • NCCL Diagnostics: Identify and diagnose NCCL failures, including timeouts, rendezvous issues, and version mismatches.
  • Infrastructure Checks: Validate cluster health, security group rules, and Kubernetes node readiness.
  • Hardware Health: Check GPU and EFA hardware health, including XID errors, ECC, and NVLink issues.
  • Use Case: If your training job on HyperPod suddenly stops, use this Skill to identify the root cause, such as NCCL timeouts, GPU OOM, or network issues, and take appropriate actions.

Quick Start

Run the hyperpod-nccl skill to diagnose NCCL issues on your cluster.

Frequently Asked Questions about hyperpod-nccl

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I diagnose NCCL failures on SageMaker HyperPod GPU clusters?▼

To diagnose NCCL failures on SageMaker HyperPod, this Skill analyzes cluster events, security group rules, Kubernetes node status, and hardware health. It targets training hangs, AllReduce timeouts, and EFA errors to identify the root cause.

Why does my training job on HyperPod suddenly stop with a rendezvous failure?▼

Training jobs on HyperPod may stop due to NCCL rendezvous failures, version mismatches, or network issues. This Skill diagnoses these specific NCCL failures by checking infrastructure health and security group rules to pinpoint the root cause.

Can I use this to check GPU and EFA hardware health for libfabric errors?▼

Yes, you can check GPU and EFA hardware health for libfabric errors. The Skill inspects hardware status including XID errors, ECC, and NVLink issues to determine if physical hardware problems are causing the NCCL failures.

Do I need kubectl and AWS SDK to debug NCCL timeouts on EKS and Slurm?▼

Yes, you need kubectl and AWS SDK to debug NCCL timeouts on EKS and Slurm. The Skill requires awscli, kubectl, python3, unbuffer, and jq to validate cluster health and perform comprehensive training diagnostics.

What is the best way to troubleshoot collective-op timeouts on HyperPod clusters?▼

The best way to troubleshoot collective-op timeouts on HyperPod clusters is to run a diagnostic Skill that analyzes Kubernetes node readiness and security group rules. This identifies whether network configuration or hardware issues cause the timeout.

Related Skills

Explore More Software Engineering Skills→
haozhx23haozhx23
15

hyperpod-node-debugger

Diagnose and remediate per-node issues on HyperPod clusters.

Community
Advanced
ovachieverovachiever
52

nemo-curator

Curate multimodal LLM training data with GPU-accelerated filtering and deduplication.

Community
Advanced
openeuler-mirroropeneuler-mirror
3

offline-GPU-fault-diagnosis

Correlate iBMC, OS, and InfoCollect logs to diagnose GPU hardware failures.

Official
Advanced
ovachieverovachiever
52

ray-data

Process distributed ML data pipelines with Ray Data across CPU/GPU clusters.

Community
Advanced