skills.rest
marin-community
marin-communityOFFICIAL
Explore all skills by this author →
WebsiteGitHub

triage-canary

Diagnose failed canary ferry runs and file GitHub issues with Slack summaries.

3.2k|266|Updated Mar 22, 2024
Search Tags:#root cause#kubernetes#triage#ci#Canary#incident diagnosis#gitHub issues
One-click install
npx skills add https://github.com/marin-community/marin --skill triage-canary
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: triage-canary
Source: https://github.com/marin-community/marin/tree/main/.agents/skills/triage-canary
Command: npx skills add https://github.com/marin-community/marin --skill triage-canary

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

When a canary ferry run fails in CI, engineers must quickly gather cluster diagnostics, identify the root cause, and report the failure before the cluster is torn down. This Skill automates that triage workflow so failures are diagnosed and reported consistently.

Core Features & Use Cases

  • Diagnostic Collection: Gathers Iris job state, Kubernetes pod status and logs (GPU lane), or Iris process logs (TPU lane) while the cluster is still live.
  • Root Cause Classification: Categorizes failures into infra/scheduling, training crash, metric regression, controller bug, or data/storage using hypothesis-driven diagnosis.
  • Automated Reporting: Files a labeled GitHub issue with canary run context and writes a four-line Slack summary to slack_message.md.
  • Use Case: A GPU canary run fails overnight in GitHub Actions; CI invokes this Skill to collect kubectl logs filtered by job ID, determine the training crashed, file a bug issue, and post a Slack summary before the cluster is torn down.

Quick Start

Triage the failed canary run using the CANARY_LANE, CANARY_JOB_ID, and other context values provided in the CI prompt, then file the issue and write the Slack summary.

Frequently Asked Questions about triage-canary

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I triage a failed canary run in CI?▼

Gather diagnostics while the cluster is live: check Iris job state, collect pod status and logs via kubectl for GPU lanes or iris process logs for TPU lanes. Then classify the root cause, file a GitHub issue, and write a Slack summary.

How do I filter kubectl pods for a specific Iris job?▼

Use the label selector iris.job_id with the job ID's slashes replaced by dots, for example kubectl -n iris-ci get pods -l iris.job_id=runner.iris-run-job-abc123. This isolates the canary's pods from co-tenant CI pods in the shared namespace.

What root cause categories does canary triage use?▼

Failures are classified into five categories: infra/scheduling, training crash, metric regression, controller bug, or data/storage. Diagnosis follows a hypothesis-driven approach of stating a hypothesis, gathering evidence, and narrowing down.

Does this skill make code changes or open pull requests?▼

No. The skill performs diagnosis and reporting only. It files a GitHub issue with the bug-report template and writes a Slack summary, but explicitly does not make code changes or create PRs.

What context variables are required to run canary triage?▼

The CI prompt provides CANARY_LANE, CANARY_JOB_ID, CANARY_RUN_ID, IRIS_CONFIG, IRIS_NAMESPACE, WANDB_ENTITY, WANDB_PROJECT, and GHA_RUN_URL. These prompt values should be used directly rather than inspecting the process environment.

Related Skills

Explore More Software Engineering Skills→
open-thoughtsopen-thoughts
276

datagen-launch-iris

Orchestrate datagen jobs on Marin's Iris TPU cluster from launch to cleanup.

Official
Advanced
carllelandtaylorcarllelandtaylor

watch-and-fix-ci

Monitor open pull request CI status, reproduce failures locally, and push fixes until checks pass.

Community
Advanced
hao-ai-labhao-ai-lab
3.9k

monitor-experiment

Monitor W&B training runs to detect anomalies and generate alerts.

Official
Intermediate
nebinfranebinfra

neb-argocd-debug

Triage ArgoCD deployment issues through a structured 5-phase workflow.

Official
Advanced