use-iris

Submit, monitor, debug, and recover Iris jobs, controllers, and accelerator sessions.

3.2k|266|Updated Mar 22, 2024
One-click install
npx skills add https://github.com/marin-community/marin --skill use-iris
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: use-iris
Source: https://github.com/marin-community/marin/tree/main/.agents/skills/use-iris
Command: npx skills add https://github.com/marin-community/marin --skill use-iris

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Operating the Iris job orchestration platform involves many distinct workflows—job submission, federated scheduling, controller rollouts, accelerator reservations, and stuck-pod recovery—and performing them incorrectly can kill running jobs or corrupt cluster state. This Skill routes each request to the correct operational procedure and enforces safety boundaries.

Core Features & Use Cases

  • Job and Task Operations: Submit, describe, inspect events, and monitor jobs and tasks across clusters, including federated root jobs spanning parent and peer clusters.
  • Controller Rollouts and Rollbacks: Deploy Iris controllers cluster-by-cluster with preflight checks, snapshots, verification gates, and rollback procedures.
  • Dev Accelerator Sessions: Reserve interactive GPU (H100, GB200) or TPU sessions with allocate, connect, execute, and release lifecycle commands.
  • Stuck Pod Recovery: Diagnose and recover CoreWeave pods stuck in terminating state with read-only inspection, cordon, and provider-confirmed node reboot.
  • Use Case: A training job appears stalled on a federated cluster. Use this Skill to inspect the parent-side job list, peer handoff state, and forwarded logs to determine whether the job is genuinely stuck or just waiting on scheduler capacity.

Quick Start

Ask the assistant to describe the status of job /myuser/my-training-run on the marin cluster using Iris.

Frequently Asked Questions about use-iris

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check the status of an Iris job?

Run `uv run iris --cluster=<cluster> job describe <job>` to see job state, and `task describe` or `task events` for individual tasks. For federated root jobs, also check the parent's job list, peer list, and the federated_jobs table.

How do I monitor a long-running Iris training job?

Follow the monitor-job reference: keep one monitoring state file, check logs every 570 seconds after an initial 120-second wait, and confirm progress from live logs, W&B timestamps, and checkpoint movement. A controller RUNNING state alone is not completion.

How do I reserve a dev GPU or TPU with Iris?

Use scripts/iris/dev_gpu.py or dev_tpu.py with the cluster config to allocate, connect, and release a session. The default GPU allocation holds one H100 node; TPU sessions require an explicit --tpu-type such as v5p-8.

Can I restart an Iris cluster to fix a stuck job?

No. `iris cluster restart` kills all workers and jobs and requires explicit approval for the named cluster. Job recovery uses cancel and resubmit with the exact original command, never a cluster restart.

How do I recover a CoreWeave pod stuck in terminating state?

Start read-only with kubectl get pod and events using the kubeconfig from the cluster config. Cordon the node, try ordinary deletion first, and only force-delete after provider-confirmed node reboot for GPU-bound processes.

Where do Iris task output files go?

Write bounded diagnostics to $IRIS_OUTPUT_DIR, which Iris preserves as one outputs.tar.zst archive per attempt. Find the archive URI and its upload state with `attempt describe <task>:<attempt>`; use direct object-storage writes for large outputs.