sue-lumi-gpu-running-status

What problem does it solve?

This Skill resolves the frustration of LUMI Slurm score-pack or evaluator jobs running far slower than expected, with unclear GPU utilization metrics or idle allocated GPUs, by eliminating guesswork about whether the issue is missing GPU access, unparallelized work, or actual compute bottlenecks.

Core Features & Use Cases

• Three-Mode Failure Diagnosis: Distinguishes between GPU visibility issues (incorrect environment, container, or GPU binding), sequential work granularity problems (work not spread across allocated GPUs), and true GPU compute bottlenecks requiring optimization.
• LUMI-Specific Workflow: Tailored for LUMI MI250X nodes, with steps to inspect work launchers, verify Slurm GPU allocations, check rocm-smi telemetry, and validate PyTorch GPU visibility inside running job steps.
• Use Case: If your LUMI evaluator job with 8 allocated GPUs is projected to take 9 hours instead of the expected 1 hour, this Skill identifies if the work is running sequentially on one GPU, if GPUs are not visible to the job, or if batch size or model loading needs optimization.

Quick Start

Invoke the sue-lumi-gpu-running-status skill to diagnose the root cause of your slow LUMI Slurm evaluator job with low GPU utilization across allocated nodes.