slurm-auto-retry-watchdog

You can automatically retry Slurm sbatch jobs by using a watchdog that submits the script, monitors status via sacct and squeue, and resubmits retriable failures like OOM or node issues up to a fixed limit.

A Slurm watchdog classifies retriable vs non-retriable failures by parsing job exit codes, stdout, stderr, and sacct output to distinguish temporary node issues from permanent script errors.

Yes, you can monitor long-running HPC batch jobs and automatically resubmit them after a timeout by applying a watchdog that checks terminal states via squeue and retries within a configured maximum attempt limit.

You need a target sbatch script and a Slurm environment capable of parsing SBATCH options, sacct/squeue output, and log files to properly classify failures before starting the automated retry watchdog.

The main limitation is that automatic resubmission only applies to retriable failures such as preemption or timeouts; non-retriable failures from script logic errors will not be retried beyond the initial attempt classification.