skills.rest
Mang30
Mang30COMMUNITY
Explore all skills by this author →
GitHub

experiment-queue

Orchestrate ML experiment batches on SSH GPU servers with OOM retry and state persistence.

Updated May 29, 2026
Search Tags:#experiment management#ml experiments#ssh orchestration#gpu scheduling#oom retry#grid search#batch job queue
One-click install
npx skills add https://github.com/Mang30/myskills --skill experiment-queue-mang30
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: experiment-queue
Source: https://github.com/Mang30/myskills/tree/main/skills/experiment-queue
Command: npx skills add https://github.com/Mang30/myskills --skill experiment-queue-mang30

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

This Skill solves the operational pain of running large multi-seed or phased ML experiment batches without losing progress to OOM failures, stale detached screens, or wave-order race conditions.

Core Features & Use Cases

  • Wave-based orchestration: Runs jobs in GPU-fit waves and only starts the next wave after processes fully settle, stale screens are cleared, GPU free memory is sufficient, and preconditions pass.
  • OOM-aware retry: Detects CUDA OOM from per-job logs, kills the failed screen, waits, re-queues, and limits retries before marking jobs as stuck.
  • Stale-screen cleanup & resume: Detects “screen exists but Python exited,” verifies expected outputs, cleans up, persists queue state to JSON continuously, and supports restarting/reloading without rebuilding.
  • Phased dependency chaining: Enforces teacher→student or other depends_on phase workflows so downstream jobs wait for checkpoints.
  • Grid/manifest driven execution: Builds a reproducible manifest from declarative grid specs, including phase templates, expected outputs, and resource scheduling parameters.

Quick Start

Use the experiment-queue skill to run batch experiments by asking: “Batch experiments on SSH GPU servers using the provided manifest or grid spec, orchestrate waves with OOM retry and stale-screen cleanup, and resume from state if needed.”

Frequently Asked Questions about experiment-queue

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run multi-seed ML experiments over SSH without losing progress to OOM failures?▼

You can orchestrate multi-seed ML experiments over SSH using wave-based scheduling with OOM-aware retry. It detects CUDA OOM from logs, kills failed screens, waits, and re-queues jobs with bounded retries before marking them as stuck.

How does stale detached screen cleanup work when batch jobs fail on remote GPU servers?▼

Stale detached screen cleanup works by detecting when a screen exists but the Python process has exited. It verifies expected outputs, cleans up the stale screen, and continuously persists queue state to JSON for seamless resumption.

Can I enforce dependency chains for teacher-student training jobs across grid sweeps?▼

Yes, you can enforce phased dependency chaining for teacher-student workflows. Downstream jobs with depends_on conditions wait for required checkpoints to become available before starting their execution phase.

What is the best way to schedule GPU jobs in waves and gate them by free memory thresholds?▼

The best way to schedule GPU jobs in waves is through GPU free-threshold gating. Jobs run in GPU-fit waves, and the next wave only starts after processes fully settle, stale screens clear, and GPU free memory is sufficient.

Do I need to rebuild the experiment queue if a batch job sweep is interrupted?▼

No, you do not need to rebuild the queue if a sweep is interrupted. The system continuously persists queue state to JSON and supports restarting and reloading from that state without rebuilding.

How do I generate a reproducible manifest for a batch of grid search experiments?▼

You generate a reproducible manifest from declarative grid specs. This manifest includes phase templates, expected outputs, and resource scheduling parameters to drive reproducible grid search execution.

Related Skills

Explore More Software Engineering Skills→
lightrain-alightrain-a

experiment-queue

Orchestrate SSH-based ML experiment batches with wave and dependency enforcement.

Community
Advanced
EmaRimoldiEmaRimoldi

result-collector

Aggregate experiment outputs into structured results with gap reporting.

Community
Advanced
caw111caw111

experiment-queue

Orchestrate machine learning experiments on SSH remote GPU servers with OOM retries.

Community
Advanced
xqinagxqinag

experiment-queue

Orchestrate ML experiment batches across SSH-enabled GPUs with retries and resume.

Community
Advanced