What problem does it solve?
Running GPU-based machine learning workloads on Slurm-managed HPC clusters is error-prone and resource-expensive when jobs are misconfigured, dependencies are missing, or offline access is not handled; this Skill helps avoid wasted GPU hours by guiding submission, environment setup, monitoring, and failure triage.
Core Features & Use Cases
- Preflight validation: Verify dataset caches, model weights, and environment variables before submitting a job.
- Non-interactive environment init: Patterns for sourcing conda in batch shells and exporting LD_LIBRARY_PATH so CUDA libraries resolve properly.
- Submission templates & monitoring: sbatch script conventions, job naming with SLURM_JOB_ID, walltime planning, and quick-use monitoring commands like squeue, sacct, and tail.
- Failure triage & best practices: Detect OOMs, import errors, NaN losses, and implement offline-first workflows for package and model access.
- Use case: Prepare and submit a fastrun smoke test that runs a short validation pass with cached datasets to catch config errors before a fullrun.
Quick Start
Submit a Slurm training job that runs a preflight check, activates conda in a non-interactive shell, enforces offline HF settings, and launches the sbatch template to start training.
