slurm-auto-retry-watchdog

What problem does it solve?

Submitting Slurm jobs often requires manual monitoring and retries. This skill provides an automated watchdog that submits an sbatch script, monitors terminal state using sacct and squeue, classifies common failures from job state and log output, and automatically resubmits retriable failures up to a fixed attempt limit.

Core Features & Use Cases

• Monitors Slurm job status via sacct and squeue to determine when a job completes or fails.
• Classifies failures from exit codes, stdout, and stderr to separate retriable vs non-retriable conditions.
• Automatically retries retriable failures up to a configurable maximum, and records a detailed JSON report per attempt.
• Useful for long-running batch workflows that need to recover from timeouts, OOMs, node failures, or preemption without manual intervention.

Quick Start

From the project root, run the watchdog against your sbatch script to automatically retry retriable failures with a configurable max-attempts.