sue-monitor

What problem does it solve?

Manually tracking active ML/HPC scale-up experiment runs across multiple sandbox backends requires cross-referencing scheduler state, progress logs, W&B metrics, and GPU utilization data to get accurate status, ETA, and resource usage estimates, which is time-consuming and error-prone. This skill automates that entire process, enforcing critical guardrails to ensure accurate and compliant monitoring output.

Core Features & Use Cases

• Standardized Monitor Tables: Generates fixed 9-column per-job tables with progress, elapsed time, completion estimates, and remaining runtime, never collapsing array tasks into summary rows to preserve per-task visibility.
• Critical Guardrail Enforcement: Automatically validates GPU vacancy (flagging underutilized allocated GPUs) and full absolute path output compliance per DeepResearch root contracts.
• Flexible Invocation Modes: Supports one-shot ad-hoc snapshots, durable monitoring loops with desktop notifications, and headless subagent mode for cron jobs or parent agent dispatch.
• Use Case: After submitting a large-scale training run on the LUMI supercomputer, use this skill to check if any nodes have idle GPUs while work is queued, get an accurate ETA for run completion, and track projected GPU hour consumption against your initial budget.

Quick Start

Use the sue-monitor skill to pull a live status snapshot of your active ML/HPC scale-up experiment run, including per-job progress, completion ETA, and GPU utilization details.