investigate-alert

Diagnose firing Grafana alerts by querying Prometheus and Loki datasources with gcx.

1|Updated May 25, 2020
One-click install
npx skills add https://github.com/titaneric/dotfiles --skill investigate-alert-titaneric
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: investigate-alert
Source: https://github.com/titaneric/dotfiles/tree/main/dot_agents/skills/investigate-alert
Command: npx skills add https://github.com/titaneric/dotfiles --skill investigate-alert-titaneric

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? When a Grafana alert fires, operators need to quickly determine why it triggered, what it affects, and what to do next. This Skill automates that investigation by fetching alert state, querying the underlying metrics and logs, and producing a concise root-cause analysis with actionable next steps. ## Core Features & Use Cases - Alert State Triage: Fetches alert rules via gcx, distinguishes recording rules and healthy inactive alerts from genuinely firing or pending alerts, and stops early when no action is needed. - Datasource Investigation: Queries Prometheus metrics and Loki logs using the alert's datasourceUID, rendering ASCII graph visualizations to reveal spikes, trends, and anomalies. - Actionable Analysis: Extracts runbook and dashboard links from annotations, suggests follow-up queries by error class (connection errors, latency, error rates, resource exhaustion), and recommends incident creation when there is customer impact. - Use Case: An on-call engineer sees a HighErrorRate alert firing in production. The Skill retrieves the alert, graphs the 5xx error rate over the last hour, correlates with Loki error logs, identifies a recent deployment as the likely cause, and lists the next queries to run. ## Quick Start Ask the assistant to investigate why the HighErrorRate Grafana alert is firing and show the current trend and likely causes.

Frequently Asked Questions about investigate-alert

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I investigate a firing Grafana alert from the command line?

Use gcx to list alert rules with gcx alert rules list -o json, filter by name and state with jq, then query the alert's datasourceUID with gcx metrics query or gcx logs query. Graph output reveals whether the issue is a spike, gradual trend, or sustained condition.

How do I query Prometheus metrics and Loki logs for alert troubleshooting?

Run gcx metrics query <datasource-uid> '<PromQL>' --from now-1h --to now -o graph for metrics, and gcx logs query <loki-uid> '<LogQL>' for logs. Resolve datasource names to UIDs first with gcx datasources list --type loki.

What is the difference between a recording rule and an alerting rule in Grafana?

A recording rule pre-calculates metrics and never fires alerts, while an alerting rule evaluates a condition and transitions between inactive, pending, and firing states. Check the type field in the alert JSON to distinguish them before investigating.

Why does my Loki query hit series limits or return no results?

Loki metric queries like rate() or count_over_time() produce one series per label combination and hit the default 20K series limit. Wrap them in sum(), sum by(label), or topk(), and filter only on stream labels in the {} selector rather than extracted labels.

What prerequisites are needed before investigating alerts with gcx?

You need the gcx CLI installed with a configured context and appropriate permissions to list alert rules and query datasources. If gcx is not configured, run the setup-gcx skill first, and specify which context to use when multiple exist.