slurm-self-healing-watchdog

What problem does it solve?

Slurm job workflows can stall or fail silently, requiring manual monitoring. This skill provides an automated watchdog that submits sbatch jobs, watches terminal states via sacct/squeue, diagnoses failures from logs, and triggers minimal, local repairs before resubmitting up to a configurable limit.

Core Features & Use Cases

• Submits an sbatch script and continuously monitors its status until terminal.
• Diagnoses failures using Slurm state and log output, then applies a minimal repair workflow.
• Automatically retries failed jobs up to a configurable limit and emits per-attempt JSON reports.
• Ideal for long-running pipelines and experiments where manual re-submission is error-prone.

Quick Start

From the target repository root, run the watchdog with the sbatch script path and optional --max-attempts to start automatic monitoring and safe repairs.