skills.rest
NVIDIA
NVIDIAOFFICIAL
Explore all skills by this author →
WebsiteGitHub

exec-local-slurm

Submit and monitor Slurm jobs on a local cluster with persistent node allocation reuse.

14.5k|2.7k|Updated Aug 16, 2023
Search Tags:#sbatch#job monitoring#job scheduling#Slurm#gpu testing#hpc cluster#persistent allocation
One-click install
npx skills add https://github.com/NVIDIA/TensorRT-LLM --skill exec-local-slurm
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: exec-local-slurm
Source: https://github.com/NVIDIA/TensorRT-LLM/tree/main/.claude/skills/exec-local-slurm
Command: npx skills add https://github.com/NVIDIA/TensorRT-LLM --skill exec-local-slurm

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Running tests, evaluations, and benchmarks on a Slurm cluster repeatedly wastes time on queue waits, container imports, and environment installs for every single job. This Skill manages Slurm job submission and monitoring with a persistent allocation mode that allocates nodes once and reuses them across runs.

Core Features & Use Cases

  • Persistent Allocation Mode: Allocates nodes once via nohup salloc, imports the container once, installs dependencies once, and reuses the allocation across runs by exporting SLURM environment variables.
  • One-Shot sbatch Mode: Submits a fully generated Slurm script via sbatch, polls job status, and reads logs on completion.
  • Hang Detection and Failure Diagnosis: Monitors logs for hang indicators, cancels stuck steps without killing the allocation, and maps failures (TIMEOUT, OUT_OF_MEMORY, HANG_DETECTED) to actionable causes.
  • Use Case: An engineer running repeated pytest suites for TensorRT-LLM on an H100 cluster uses this Skill to keep a warm allocation alive, so each test run starts immediately without re-importing the container or reinstalling packages.

Quick Start

Submit my generated llama_auto_test.slurm script to the local Slurm cluster in persistent mode and report the pytest results when it finishes.

Frequently Asked Questions about exec-local-slurm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I submit and monitor a Slurm job from an AI agent?▼

Provide the path to a generated .slurm script, a work directory, and success/failure patterns. The Skill submits via sbatch or reuses a persistent allocation, polls job status with squeue, and reports results parsed from the log files.

What is a persistent Slurm allocation and why use one?▼

A persistent allocation reserves nodes once via nohup salloc and keeps them RUNNING across multiple job runs. It eliminates repeated queue waits, container imports, and package installs, so subsequent jobs start immediately by reusing the named container.

How does the skill detect a hung Slurm job?▼

It checks the job log every 60 seconds for the pattern 'hang detected' (case-insensitive). When found, it cancels the active srun step with scancel while keeping the allocation alive, then reads the last 200 log lines for root cause analysis.

Why does my Slurm job fail with 'Unable to create step'?▼

This error means the persistent allocation expired, typically because its walltime ran out. Delete the stale state file .slurm_alloc.json and re-run; the Skill will allocate fresh nodes and re-import the container automatically.

Can I force one-shot sbatch instead of persistent mode?▼

Yes, set persistent_mode=false in the input fields to opt out. The Skill then submits the script directly via sbatch, polls with squeue every 30 seconds, and uses sacct for final status and exit code.

When is the persistent allocation released?▼

The allocation is never auto-released after job completion. It persists until its walltime expires or the user explicitly sets release_allocation=true, which runs scancel on the stored job ID and deletes the state file.

Related Skills

Explore More Software Engineering Skills→
umich-foreseerumich-foreseer
16

slurm-job

Generate Slurm sbatch scripts with partition, account, GPU, and validation directives.

Official
Intermediate
krzemienskikrzemienski
2

team-validation-dashboard

Aggregate team validation metrics and identify projects with low posture scores.

Community
Intermediate
yo-stevenyo-steven

exec-slurm-compile

Compile TensorRT-LLM on a SLURM cluster via enroot container batch jobs.

Community
Intermediate
matsunagalabmatsunagalab
11

HPC Run

Submit and manage SLURM-based MDClaw workloads on HPC clusters.

Official
Advanced