slurm-gpu-training

What problem does it solve?

Running GPU-based machine learning workloads on Slurm-managed HPC clusters is error-prone and resource-expensive when jobs are misconfigured, dependencies are missing, or offline access is not handled; this Skill helps avoid wasted GPU hours by guiding submission, environment setup, monitoring, and failure triage.

Core Features & Use Cases

• Preflight validation: Verify dataset caches, model weights, and environment variables before submitting a job.
• Non-interactive environment init: Patterns for sourcing conda in batch shells and exporting LD_LIBRARY_PATH so CUDA libraries resolve properly.
• Submission templates & monitoring: sbatch script conventions, job naming with SLURM_JOB_ID, walltime planning, and quick-use monitoring commands like squeue, sacct, and tail.
• Failure triage & best practices: Detect OOMs, import errors, NaN losses, and implement offline-first workflows for package and model access.
• Use case: Prepare and submit a fastrun smoke test that runs a short validation pass with cached datasets to catch config errors before a fullrun.

Quick Start

Submit a Slurm training job that runs a preflight check, activates conda in a non-interactive shell, enforces offline HF settings, and launches the sbatch template to start training.