skills.rest
open-thoughts
open-thoughtsOFFICIAL
Explore all skills by this author →
WebsiteGitHub

eval-agentic-launch-iris

Manages the full lifecycle of Iris evaluation jobs via the command line.

276|37|Updated Dec 5, 2025
Search Tags:#evaluation#cleanup#launch#monitor#agent-harness#marin#iris-cluster
One-click install
npx skills add https://github.com/open-thoughts/OpenThoughts-Agent --skill eval-agentic-launch-iris
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-agentic-launch-iris
Source: https://github.com/open-thoughts/OpenThoughts-Agent/tree/main/.claude/skills/eval-agentic-launch-iris
Command: npx skills add https://github.com/open-thoughts/OpenThoughts-Agent --skill eval-agentic-launch-iris

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Launch, monitor, and manually clean up evaluation runs on Marin's Iris TPU cluster using the OpenThoughts-Agent entrypoint. This enables reproducible model benchmarks and reduces manual cluster management when running eval scripts.

Core Features & Use Cases

  • End-to-end Iris evaluation workflow: launch, monitor, and gracefully terminate or cleanup an eval run.
  • Works with standard presets and custom datasets via the provided harbor config and model identifiers.
  • Provides guidance for recovering partial results and handling common failure modes during long-running evals.

Quick Start

Run the Iris launcher with --preset and --harbor_config to start an eval on Iris.

Frequently Asked Questions about eval-agentic-launch-iris

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I launch an evaluation job on Marin's Iris TPU cluster?▼

To launch an Iris eval job, run the launcher with the iris CLI tooling, providing a preset and a harbor config YAML to initialize the benchmark on the Iris TPU cluster.

What is the best way to monitor and clean up long-running evaluation runs?▼

This workflow manages end-to-end Iris evaluation runs, allowing you to monitor active eval jobs and gracefully terminate or clean up resources to handle common long-running failure modes.

Do I need the iris CLI and harbor config YAMLs to run model benchmarks?▼

Yes, running model benchmarks requires the iris CLI tooling and harbor config YAMLs, along with access to specific model identifiers and datasets as described in the SKILL.md.

Can I recover partial results if an Iris eval job fails on the TPU cluster?▼

Yes, the workflow provides guidance for recovering partial results and handling common failure modes encountered during long-running evals on the Iris TPU cluster.

Does the Iris eval workflow support custom datasets and standard presets?▼

Yes, the Iris evaluation workflow supports both standard presets and custom datasets by utilizing the provided harbor config and model identifiers for reproducible benchmarks.

Related Skills

Explore More Software Engineering Skills→
open-thoughtsopen-thoughts
276

datagen-launch-iris

Orchestrate datagen jobs on Marin's Iris TPU cluster from launch to cleanup.

Official
Advanced
xiaoquqixiaoquqi

evaluating-llms-harness

Benchmark language models across 60+ academic tasks with multiple backends.

Community
Advanced
sorodriguezzsorodriguezz

Monitoring Productions

Monitor InterSystems IRIS production status, message queues, and system performance.

Community
Advanced
neubigneubig
3

eval-with-ci

Trigger CI-based benchmark evaluations for the OpenHands software-agent-sdk suite.

Community
Advanced