skills.rest
Shallow-W
Shallow-WCOMMUNITY
Explore all skills by this author →
GitHub

experiment-queue

Orchestrate SSH-based ML experiment job queues with OOM retries and idempotent resume.

Updated Apr 21, 2026
Search Tags:#wave scheduling#oom retry#experiment orchestration#stale screen cleanup#ssh gpu#ml grid search
One-click install
npx skills add https://github.com/Shallow-W/llm-wiki --skill experiment-queue-shallow-w
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: experiment-queue
Source: https://github.com/Shallow-W/llm-wiki/tree/main/.claude/skills/experiment-queue
Command: npx skills add https://github.com/Shallow-W/llm-wiki --skill experiment-queue-shallow-w

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

This Skill removes the engineering friction of running large multi-seed or multi-phase ML experiment grids over SSH by automatically scheduling jobs, handling OOM retries, cleaning stale screen sessions, and preventing wave-transition race conditions.

Core Features & Use Cases

  • SSH GPU batch orchestration: Schedules 10+ jobs across available remote GPUs using a manifest with explicit job states.
  • OOM-aware retry & bounded failure: Detects CUDA out-of-memory in per-job logs, retries after a delay up to a configured max, then marks the job stuck for manual follow-up.
  • Stale screen cleanup & idempotent resume: Detects cases where a screen remains but the Python process exited, cleans up safely, continuously persists queue_state.json, and supports restarting the scheduler without losing progress.
  • Wave transitions & dependency phases: Runs jobs in waves and enforces phase dependencies (e.g., teacher training must complete before student distillation).

Quick Start

Ask the AI to run experiment-queue for your multi-seed or multi-phase grid by providing a grid spec or manifest and the remote SSH GPU target (e.g., “Run experiment-queue for a 21-seed sweep across 12 configs with teacher→student phases, retry OOM up to 3 times, and write results to the specified remote run directory”).

Frequently Asked Questions about experiment-queue

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I orchestrate large ML experiment batches over SSH without losing progress?▼

SSH-based ML experiment orchestration schedules multi-seed and multi-phase sweeps across remote GPUs while persisting queue state for idempotent resume, ensuring progress is never lost during long-running batch jobs.

What is the best way to handle CUDA out-of-memory failures during a multi-seed grid search?▼

Handling CUDA out-of-memory during grid search requires detecting OOM errors in per-job logs and applying delayed bounded retries up to a configured maximum before marking the job stuck for manual follow-up.

How do I run wave-chained training pipelines with phase dependencies like teacher to student distillation?▼

Wave-chained pipelines enforce phase dependencies so teacher training fully completes before student distillation starts, preventing race conditions by strictly controlling job waves and dependency transitions within the manifest.

How do I clean up stale screen sessions when a Python process exits unexpectedly on a remote GPU?▼

Stale screen cleanup detects when a screen session remains active but the underlying Python process has exited, safely removing the orphaned session to free the remote GPU and prevent scheduling blockages.

Do I need a YAML or JSON manifest to schedule multi-phase experiment grids?▼

A YAML or JSON manifest is required to define explicit job states, configurations, and phase dependencies, enabling the queue scheduler to orchestrate waves and handle failures properly across the experiment grid.

When should I use a dedicated experiment queue instead of standard run-experiment commands?▼

A dedicated experiment queue is necessary for large job sets involving 10+ jobs, multi-seed sweeps, or complex phase dependencies that exceed standard run-experiment capabilities, providing OOM recovery and state persistence for heavy workloads.

Related Skills

Explore More Education & Research Skills→
duypham2801duypham2801

experiment-queue

Orchestrate SSH-hosted batch ML experiments with wave scheduling and OOM-aware retries.

Community
Advanced
zhuyingqinzhuyingqin
1

experiment-queue

Orchestrate SSH-based ML experiment batches with OOM retries and stateful resume.

Community
Advanced
czh-ee-2023czh-ee-2023

experiment-queue

Orchestrate ML experiment batches on SSH GPU servers with OOM retry.

Community
Advanced
xqinagxqinag

experiment-queue

Orchestrate ML experiment batches across SSH-enabled GPUs with retries and resume.

Community
Advanced