slo-investigate

Diagnose breaching SLOs with dimensional breakdown, alert correlation, and runbook retrieval via gcx.

1|Updated May 25, 2020
One-click install
npx skills add https://github.com/titaneric/dotfiles --skill slo-investigate-titaneric
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: slo-investigate
Source: https://github.com/titaneric/dotfiles/tree/main/dot_agents/skills/slo-investigate
Command: npx skills add https://github.com/titaneric/dotfiles --skill slo-investigate-titaneric

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? When an SLO starts breaching or burning error budget, operators need to quickly find the root cause instead of manually piecing together Grafana dashboards, PromQL queries, and alert rules. This Skill runs a structured investigation workflow that pinpoints the failing dimension and surfaces relevant runbooks. ## Core Features & Use Cases - Guided Investigation Workflow: Retrieves the SLO definition, checks status with early exits for OK/NODATA states, renders timelines, and performs dimensional breakdown of ratio or freeform queries. - Alert and Runbook Correlation: Searches alert rules by SLO name or UUID, extracts runbook and dashboard URLs from annotations, and fetches runbook content from GitHub via the gh CLI. - PromQL Reference: Includes a reference of Grafana SLO recording rule metrics (SLI, burn rate, error budget) with ready-to-use query patterns. - Use Case: An on-call engineer receives a fast-burn alert for an availability SLO, asks the agent to investigate, and gets a report showing the worst offending cluster, related firing alert rules, and the runbook's remediation steps. ## Quick Start Ask the agent to investigate why your SLO is breaching by providing its name or UUID, for example: investigate why the checkout-availability SLO is burning error budget.

Frequently Asked Questions about slo-investigate

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I investigate why an SLO is breaching?

Provide the SLO name or UUID and the workflow retrieves the definition, checks status, renders a timeline, and runs dimensional breakdown queries on the success and total metrics. It finishes with a structured report covering the worst dimension, related alert rules, and runbook links.

How do I find which dimension is causing SLO errors?

The investigation queries the ratio success and total metrics grouped by the SLO's groupByLabels to compute error rates per dimension. If no groupByLabels exist, it tries common dimensions like cluster, namespace, service, endpoint, and status_code.

Why does my SLO status show NODATA in Grafana?

NODATA usually means the destination datasource is misconfigured, recording rules have not evaluated yet (allow 1-2 minutes after creation), or a Prometheus federation issue exists. Verify the destination datasource UID in the SLO spec matches an actual Prometheus datasource.

Can I fetch runbook content from SLO alert annotations?

Yes, runbook and dashboard URLs are extracted from fastBurn and slowBurn annotations plus metadata annotations. If a GitHub URL is found and the gh CLI is authenticated, the runbook content is fetched via the GitHub API and decoded.

What is the difference between slo-investigate and slo-check-status?

slo-investigate performs deep root cause analysis on a specific breaching SLO with dimensional breakdown and alert correlation. slo-check-status is intended for a broad status overview across SLOs rather than single-SLO deep dives.