skills.rest
awslabs
awslabsOFFICIAL
Explore all skills by this author →
WebsiteGitHub

hyperpod-slurm-debugger

Diagnose Slurm scheduler and node-daemon issues on SageMaker HyperPod clusters.

881|152|Updated Feb 5, 2026
Search Tags:#AWS#Diagnostics#ssm#cluster troubleshooting#Slurm#SageMaker HyperPod#slurmctld
One-click install
npx skills add https://github.com/awslabs/agent-plugins --skill hyperpod-slurm-debugger
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hyperpod-slurm-debugger
Source: https://github.com/awslabs/agent-plugins/tree/main/plugins/sagemaker-ai/skills/hyperpod-slurm-debugger
Command: npx skills add https://github.com/awslabs/agent-plugins --skill hyperpod-slurm-debugger

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

Slurm clusters on Amazon SageMaker HyperPod can fail in opaque ways: nodes stuck in down or drain states, jobs pending despite idle nodes, jobs stuck in COMPLETING after node replacement, or auto-recovery requests that silently never trigger. This Skill performs read-only diagnostics to classify the exact failure mode and point you to the authoritative AWS or Slurm documentation for remediation.

Core Features & Use Cases

  • Automated cluster diagnostics: Runs a read-only script over SSM that checks node states, slurmctld health, slurmdbd connectivity, slurm.conf freshness, and stuck job queues.
  • Symptom-to-doc decision table: Maps findings like "Node unexpectedly rebooted", GRES count mismatches, and Action:Reboot/Replace reason-string mismatches to the correct AWS and Slurm documentation sections.
  • Auto-resume investigation: Explains why HyperPod --auto-resume did not restart a job, covering HMA flagging, NodeRecovery settings, and AMI version requirements.
  • Use Case: A training job is stuck PENDING with REASON=Resources while sinfo shows idle nodes. Invoke this Skill with the cluster name and region; it confirms the orchestrator type, runs the diagnostic script, and links you to the controller-state recovery documentation.

Quick Start

Ask the assistant to diagnose why Slurm nodes are down or jobs are stuck on your HyperPod cluster, providing the cluster name and AWS region.

Frequently Asked Questions about hyperpod-slurm-debugger

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I diagnose a Slurm node stuck in down state on SageMaker HyperPod?▼

Run the diagnostic script with your cluster name and region, optionally scoping to a specific node. It checks slurmd status, disk usage, memory, and Slurm RPC reachability via SSM, then links you to the HyperPod troubleshooting documentation for recovery.

Why are my Slurm jobs stuck PENDING with REASON=Resources while nodes are idle?▼

This typically indicates slurmctld in-memory state has desynced from disk state. The Skill detects this pattern and points to controller-state documentation; a controller restart reloads from StateSaveLocation and clears stale scheduling decisions.

Why didn't HyperPod auto-recovery trigger after I set the node to fail state?▼

HyperPod matches the Slurm node Reason field exactly and case-sensitively: it must be Action:Reboot or Action:Replace with no extra spaces or punctuation. The script detects near-miss reason strings on nodes in fail state.

Does this Skill work with EKS-orchestrated HyperPod clusters?▼

No. It checks DescribeCluster for the orchestrator type and exits if Orchestrator.Eks is present, directing you to EKS-side skills like hyperpod-node-debugger or hyperpod-nccl instead.

Can this Skill fix or restart Slurm nodes automatically?▼

No. It is strictly diagnostic-only and read-only; it never runs, recommends, or prints state-mutating commands. For every finding it links to official AWS or Slurm documentation, and the operator authorizes and executes any remediation.

Why didn't HyperPod auto-resume restart my training job after a node failure?▼

Common causes include setting the flag on sbatch instead of srun, HMA not flagging the node as a hardware fault, cluster NodeRecovery set to None, missing checkpointing, or an AMI predating HMA support released 2025-09-11.

Related Skills

Explore More Software Engineering Skills→
haozhx23haozhx23
15

hyperpod-ssm

Executes remote commands and transfers files on SageMaker HyperPod nodes via AWS Systems Manager.

Community
Advanced
thimsluggathimslugga

systemd-verify

Validate systemd review findings against false-positive patterns with evidence.

Community
Intermediate
haozhx23haozhx23
15

hyperpod-nccl

Diagnose NCCL failures on SageMaker HyperPod clusters via events, security groups, and node status.

Community
Advanced
haozhx23haozhx23
15

hyperpod-node-debugger

Diagnose and remediate per-node issues on HyperPod clusters.

Community
Advanced