slurm-self-healing-watchdog

To automatically monitor and retry failed Slurm sbatch jobs, you can use a watchdog that submits scripts, watches terminal states via sacct and squeue, diagnoses failures, and resubmits up to a configurable limit.

The best way to prevent Slurm workflows from stalling or failing silently is deploying an automated watchdog that continuously tracks job state and triggers minimal local repairs before safely resubmitting.

A Slurm watchdog diagnoses job failures by checking the terminal state via sacct and squeue, analyzing log output, and emitting structured JSON repair notes to aid debugging and auditing.

Yes, you can configure the maximum retry attempts for sbatch job resubmission. The watchdog applies minimal fixes and resubmits failed jobs up to a configurable limit, defaulting to five attempts in common scenarios.

No, you do not need to manually parse sacct logs to repair long-running Slurm pipelines. The watchdog automatically diagnoses failures from log output and applies minimal repair workflows before retrying.

A limitation of using an automated watchdog for Slurm job repair is that it applies only minimal local fixes and retries up to a configured limit, which may not resolve complex pipeline failures requiring manual intervention.