skills.rest
duypham2801
duypham2801COMMUNITY
Explore all skills by this author →
GitHub

experiment-queue

Orchestrate SSH-hosted batch ML experiments with wave scheduling and OOM-aware retries.

Updated May 25, 2026
Search Tags:#stateful automation#oom retry#ml experiment orchestration#ssh job queue#multi-gpu scheduling#grid sweep#teacher-student distillation
One-click install
npx skills add https://github.com/duypham2801/ThS_LLM --skill experiment-queue-duypham2801
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: experiment-queue
Source: https://github.com/duypham2801/ThS_LLM/tree/main/.claude/skills/experiment-queue
Command: npx skills add https://github.com/duypham2801/ThS_LLM --skill experiment-queue-duypham2801

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

This Skill eliminates the engineering friction of running large multi-seed, multi-config ML experiments where GPU contention, OOM failures, stale terminal sessions, and multi-phase dependencies commonly waste hours.

Core Features & Use Cases

  • Wave-based multi-GPU scheduling: runs jobs in GPU-constrained waves while enforcing phase dependencies (e.g., teacher-before-student).
  • OOM-aware retry and state persistence: detects OOM in per-job logs, retries with bounded attempts, and records job state to allow restart after crashes.
  • Stale screen and completion validation: detects screen/Python mismatches, cleans up stale sessions, and marks completion using expected output checks rather than screen state.

Use case example: You have a 40+ job distillation sweep (multiple seeds and hyperparameters) and the teacher phase must finish before students start; this Skill coordinates the full queue on an SSH GPU server, retries OOM-prone cells, and produces a resumable state file.

Quick Start

Ask the system to: “Run the experiment-queue using a grid/manifest with project, cwd, conda env, remote SSH host, GPUs, and a phase dependency, then launch the scheduler and report the total job count and remote run directory.”

Frequently Asked Questions about experiment-queue

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I orchestrate multi-seed ML experiment sweeps across remote SSH GPUs?▼

The scheduler uses wave-based multi-GPU scheduling to execute batch ML experiments defined in a YAML or JSON manifest. It enforces phase dependencies, such as teacher-before-student chains, across remote SSH hosts.

How does OOM-aware retry work for large machine learning experiment queues?▼

OOM-aware retry detects out-of-memory failures in per-job logs and retries the failed jobs with bounded attempts. The system persists job state to allow restartable orchestration after crashes.

Can I run multi-phase teacher-student distillation chains with GPU contention?▼

Yes, you can run multi-phase teacher-student distillation chains by specifying phase dependencies in your manifest. The wave-based scheduler enforces these dependencies, ensuring teacher phases finish before student jobs start under GPU constraints.

Do I need a manifest file to schedule batch ML experiments on an SSH server?▼

Yes, you need a YAML or JSON manifest to schedule batch ML experiments. It must describe project paths, conda environments, remote SSH hosts, GPU constraints, per-job commands, expected outputs, and optional retry policies.

How do you validate job completion for stale screen sessions on remote GPUs?▼

You validate job completion by checking for expected outputs rather than relying on screen state. The system detects screen or Python mismatches, cleans up stale sessions, and marks completion based on expected output checks.

Related Skills

Explore More Education & Research Skills→
Mang30Mang30

experiment-queue

Orchestrate ML experiment batches on SSH GPU servers with OOM retry and state persistence.

Community
Advanced
ovachieverovachiever
52

weights-and-biases

Automate ML experiment tracking and model registry management via the wandb API.

Community
Advanced
MACJONSWMACJONSW

run-experiment

Deploy and execute machine learning experiments on local or remote GPU servers via SSH.

Community
Advanced
jonnabiojonnabio
6

ray-train

Scale machine learning training from single GPU to multi-node clusters.

Community
Advanced