slurm-auto-retry-watchdog

Name: slurm-auto-retry-watchdog
Availability: InStock
Author: KwongFuk

Submit Slurm sbatch scripts and retry retriable failures up to a fixed limit.

Updated Apr 10, 2026

Search Tags:#monitoring #retry #hpc #slurm #watchdog #sbatch

Ready to run

npx skills add https://github.com/KwongFuk/codex-skills --skill slurm-auto-retry-watchdog

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: slurm-auto-retry-watchdog
Source: https://github.com/KwongFuk/codex-skills/tree/main/global/slurm-auto-retry-watchdog
Command: npx skills add https://github.com/KwongFuk/codex-skills --skill slurm-auto-retry-watchdog

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

Submitting Slurm jobs often requires manual monitoring and retries. This skill provides an automated watchdog that submits an sbatch script, monitors terminal state using sacct and squeue, classifies common failures from job state and log output, and automatically resubmits retriable failures up to a fixed attempt limit.

Core Features & Use Cases

Monitors Slurm job status via sacct and squeue to determine when a job completes or fails.
Classifies failures from exit codes, stdout, and stderr to separate retriable vs non-retriable conditions.
Automatically retries retriable failures up to a configurable maximum, and records a detailed JSON report per attempt.
Useful for long-running batch workflows that need to recover from timeouts, OOMs, node failures, or preemption without manual intervention.

Quick Start

From the project root, run the watchdog against your sbatch script to automatically retry retriable failures with a configurable max-attempts.

Frequently Asked Questions about slurm-auto-retry-watchdog

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I automatically retry Slurm sbatch jobs that fail due to node issues or OOM?▼

You can automatically retry Slurm sbatch jobs by using a watchdog that submits the script, monitors status via sacct and squeue, and resubmits retriable failures like OOM or node issues up to a fixed limit.

How does a Slurm watchdog classify retriable vs non-retriable job failures?▼

A Slurm watchdog classifies retriable vs non-retriable failures by parsing job exit codes, stdout, stderr, and sacct output to distinguish temporary node issues from permanent script errors.

Can I monitor long-running HPC batch jobs and resubmit them automatically after a timeout?▼

Yes, you can monitor long-running HPC batch jobs and automatically resubmit them after a timeout by applying a watchdog that checks terminal states via squeue and retries within a configured maximum attempt limit.

What do I need to set up before automatically retrying Slurm sbatch scripts?▼

You need a target sbatch script and a Slurm environment capable of parsing SBATCH options, sacct/squeue output, and log files to properly classify failures before starting the automated retry watchdog.

Are there limitations to automatically resubmitting failed Slurm jobs?▼

The main limitation is that automatic resubmission only applies to retriable failures such as preemption or timeouts; non-retriable failures from script logic errors will not be retried beyond the initial attempt classification.