skills.rest
zhuyingqin
zhuyingqinCOMMUNITY
Explore all skills by this author →
GitHub

experiment-queue

Orchestrate SSH-based ML experiment batches with OOM retries and stateful resume.

1|1|Updated May 19, 2026
Search Tags:#machine learning#experiment automation#workflow phases#ssh orchestration#gpu scheduling#oom retry#multi-seed sweeps
One-click install
npx skills add https://github.com/zhuyingqin/ARIS-WEB --skill experiment-queue-zhuyingqin
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: experiment-queue
Source: https://github.com/zhuyingqin/ARIS-WEB/tree/main/crates/runtime/assets/skills/experiment-queue
Command: npx skills add https://github.com/zhuyingqin/ARIS-WEB --skill experiment-queue-zhuyingqin

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nvidia-smi, screen, ssh, scp, jq, and includes scripts (resource) components.

What problem does it solve?

experiment-queue solves the operational pain of running large multi-seed, multi-config ML experiment grids on remote GPUs by preventing common failures like stale hung sessions, GPU OOM stalls, and wave-transition race conditions.

Core Features & Use Cases

  • SSH GPU batch orchestration: schedules 10+ jobs across available GPUs with continuous state tracking and resumability.
  • Wave transitions + phase dependencies: runs jobs in ordered phases (e.g., teacher then student) and only starts the next wave when the previous one fully settles.
  • OOM-aware retry and stale cleanup: detects CUDA OOM in per-job logs, retries up to a configured max, and cleans stale screen sessions that appear alive but have exited.
  • Deterministic manifest-driven launches: accepts a YAML/JSON grid spec or manifest to generate explicit job commands and expected output checks.

Quick Start

Use the experiment-queue skill to run a teacher-to-student multi-seed experiment grid by asking it to orchestrate a batch of 10+ jobs with a manifest and remote SSH GPU server, so it can launch in waves, retry OOM safely, and monitor completion via expected output files.

Frequently Asked Questions about experiment-queue

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate SSH GPU scheduling for multi-seed machine learning sweeps?▼

SSH GPU scheduling for multi-seed machine learning sweeps is automated by parsing grid specs into deterministic detached jobs launched on free GPUs, tracking execution state continuously to ensure resumability. It orchestrates 10+ jobs across available remote GPUs.

How do I automatically retry CUDA OOM errors in remote ML experiments?▼

CUDA OOM errors in remote ML experiments are automatically retried by detecting out-of-memory events in per-job logs and re-launching the failed jobs up to a configured maximum limit. This prevents batch sweeps from stalling due to transient memory spikes.

What is the best way to run phased teacher to student training chains over SSH?▼

Phased teacher to student training chains over SSH are run using wave-based orchestration that enforces dependencies. The system only starts the next wave when the previous one fully settles, preventing race conditions during batch transitions.

Do I need screen and nvidia-smi to orchestrate remote ML experiment batches?▼

Yes, orchestrating remote ML experiment batches requires nvidia-smi, screen, ssh, scp, and jq. These dependencies enable deterministic detached job launching, free GPU detection, remote file copying, and stateful JSON parsing for queue management.

How do I clean up stale screen sessions that hang after a GPU job exits?▼

Stale screen sessions that hang after a GPU job exits are cleaned up by the orchestration loop, which detects sessions that appear alive but have actually exited. This eliminates stale session hangs and frees resources for queued jobs.

How do I resume an interrupted machine learning experiment grid from a checkpoint?▼

An interrupted machine learning experiment grid is resumed via stateful tracking stored in queue_state.json. This file records the execution state of the batch, allowing the system to skip completed jobs and continue the workflow.

Related Skills

Explore More Education & Research Skills→
dingdyandingdyan

windows-remote

Execute PowerShell and CLI commands on remote Windows hosts over SSH.

Community
Intermediate
duypham2801duypham2801

experiment-queue

Orchestrate SSH-hosted batch ML experiments with wave scheduling and OOM-aware retries.

Community
Advanced
oakossoakoss
14

cli-power-tools

Integrates ripgrep, fd, jq, and Nushell for command-line text processing.

Official
Advanced
AissenLiuAissenLiu

modal-serverless-gpu

Deploy ML workloads on auto-scaling serverless GPU infrastructure with REST endpoints.

Community
Advanced