skills.rest
NVIDIA
NVIDIAOFFICIAL
Explore all skills by this author →
WebsiteGitHub

exec-remote-slurm

Executes builds and jobs on remote SLURM clusters over SSH with pyxis containers.

14.5k|2.7k|Updated Aug 16, 2023
Search Tags:#ssh#remote execution#hpc#sbatch#enroot#gpu cluster#Slurm
One-click install
npx skills add https://github.com/NVIDIA/TensorRT-LLM --skill exec-remote-slurm
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: exec-remote-slurm
Source: https://github.com/NVIDIA/TensorRT-LLM/tree/main/.claude/skills/exec-remote-slurm
Command: npx skills add https://github.com/NVIDIA/TensorRT-LLM --skill exec-remote-slurm

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

Running TensorRT-LLM builds, tests, and benchmarks on a remote SLURM cluster normally requires manual SSH setup, MFA handling, repository syncing, and hand-written sbatch/srun commands. This Skill automates that entire workflow through SSH so jobs run remotely without any filesystem sync tooling.

Core Features & Use Cases

  • SSH connection management: Resolves the cluster, selects direct or MFA authentication based on cluster configuration, and reuses a persistent ControlMaster connection.
  • Two-mechanism code sync: Ships tracked changes via a temporary git branch and untracked files via a tar-over-SSH pipe, then builds TensorRT-LLM as a dedicated SLURM batch job.
  • Ready-made SLURM recipes: Provides srun, sbatch, MPI, distributed training, and tmux-based persistent allocation patterns with pyxis/enroot containers.
  • Use Case: You have local uncommitted changes to TensorRT-LLM and need to compile and run a multi-node benchmark on a B200 cluster. The Skill syncs your code, submits the build job, waits for it to succeed, and runs the workload inside the official container.

Quick Start

Use the exec-remote-slurm skill to sync my local TensorRT-LLM changes to the remote SLURM cluster, build the project, and submit my benchmark job.

Frequently Asked Questions about exec-remote-slurm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run jobs on a remote SLURM cluster over SSH?▼

Establish an SSH connection (reusing a ControlMaster for MFA clusters), sync your code to the remote, then submit jobs with sbatch or srun using --container-image and --container-mounts for pyxis/enroot containers. The Skill provides ready-made recipes for single-process, MPI, and batch workloads.

How do I sync local git changes to a remote cluster without rsync?▼

Commit tracked changes on a temporary branch, push it to origin, and fetch plus checkout on the remote. Ship untracked files separately with git ls-files piped through tar over SSH, then delete the temporary branch on both sides.

How do I keep a SLURM allocation alive for interactive development?▼

Launch salloc inside a detached tmux session on the login node so the allocation persists. Then create a named enroot container with srun --container-name and reuse it across subsequent srun commands, preserving in-container state.

Does SLURM need --gpus-per-node for every job?▼

It depends on the cluster. Exclusive-access clusters allocate all node GPUs automatically, while shared or QOS clusters require --gpus-per-node or jobs may fail with GPU specification or QOSMinGRES errors. Check the cluster's gpus_per_node configuration.

Why does importing tensorrt_llm fail on the remote cluster after syncing code?▼

The remote checkout wipes the cpp/ directory, so compiled C++ extensions are missing until a build completes. Submit the dedicated build job first and verify exit code 0:0 before running any test or benchmark job.

Related Skills

Explore More Software Engineering Skills→
NVIDIANVIDIA
14

k8s-launch-kit-discover

Discover Kubernetes cluster networking hardware and generate a resolved cluster-config.yaml.

Official
Advanced
umich-foreseerumich-foreseer
16

harvest

Detect completed SLURM experiments and collect results into docs.

Official
Advanced
yale-som-hpcyale-som-hpc
5

self-diagnosing-resource-use

Diagnose Slurm job CPU, memory, and GPU utilization using sacct, seff, and nvidia-smi.

Official
Intermediate
twjarviszyra-webtwjarviszyra-web
3

lambda-labs-gpu-cloud

Provision and manage Lambda Labs GPU cloud instances for machine learning workloads.

Community
Advanced