incident-triage

Name: incident-triage
Availability: InStock
Author: Wikimedia

Correlate Prometheus signals, deployment changes, and task metadata to diagnose LiftWing incidents.

4|Updated May 12, 2021

Search Tags:#kubernetes #observability #prometheus #gerrit #ml inference #incident triage #liftwing

Ready to run

npx skills add https://github.com/wikimedia/machinelearning-liftwing-inference-services --skill incident-triage-wikimedia

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: incident-triage
Source: https://github.com/wikimedia/machinelearning-liftwing-inference-services/tree/main/.claude/skills/incident-triage
Command: npx skills add https://github.com/wikimedia/machinelearning-liftwing-inference-services --skill incident-triage-wikimedia

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps you triage LiftWing ML inference-services production incidents by correlating alert signals, time-windowed Prometheus metrics, and nearby deployment changes—without needing SSH or kubectl.

Core Features & Use Cases

Time-windowed Prometheus triage: Uses Grafana’s anonymous datasource proxy to run range queries around a precisely pinned timestamp T to find the failing layer (deploy/readiness, pod/container failure, or mesh/Istio error flags).
Change correlation: Queries Gerrit for matching merges in inference-services and deployment-charts around the incident window to identify likely causal commits (and distinguish multi-CL roll forward vs reactive changes).
Structured incident reporting: Produces a consistent report format (Date, Issue description, Root cause with confidence, Potential solutions) and includes a “current state” recovery/regression check.

Quick Start

Provide the LiftWing alert text (or incident timestamp in UTC with year, month, day, hour, minute, timezone) and the affected cluster/namespace, and ask the Skill to triage with the default ±2 hour window and report the root cause and recommended next steps.

Frequently Asked Questions about incident-triage

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I triage Kubernetes ML inference incidents without using kubectl or SSH?▼

You can triage Kubernetes ML inference incidents by correlating time-pinned Prometheus signals and deployment changes. This approach uses Grafana's anonymous proxy and read-only Gerrit lookups to find the failing layer and identify causal commits without cluster access.

What Prometheus metrics help diagnose unavailable replicas and pod crash loops in ML-serve namespaces?▼

Prometheus metrics for unavailable replicas and pod crash loops are retrieved via range-based PromQL queries around a precise incident timestamp. These queries target deploy, readiness, pod, and mesh error flags across ML-serve namespaces to isolate the failing layer.

How do I correlate Gerrit merges with inference-services error-rate spikes?▼

Correlating Gerrit merges with inference-services error-rate spikes involves querying Gerrit for matching commits in inference-services and deployment-charts within a time window. This identifies likely causal commits and distinguishes multi-CL roll forwards from reactive changes.

Does incident triage for LiftWing require a specific timestamp format?▼

Incident triage for LiftWing requires UTC-precise timestamp pinning with year, month, day, hour, minute, and timezone. This exact timestamp anchors the range-based PromQL queries and deployment-change context needed to produce a structured diagnosis.

What's the best way to structure root cause analysis reports for Kubernetes inference-services alerts?▼

The best way to structure root cause analysis reports for Kubernetes inference-services alerts is a consistent format containing Date, Issue description, Root cause with confidence, and Potential solutions, alongside a current state recovery and regression check.

Why does my LiftWing inference error-rate spike triage miss the actual deployment change?▼

Your LiftWing inference error-rate spike triage might miss the actual deployment change if the timestamp is not UTC-precise or the default ±2 hour window fails to capture the causal commit, preventing accurate correlation of Prometheus signals with Gerrit merges.