What problem does it solve?
This Skill reduces alert fatigue by replacing noisy threshold alerts with SLO-based multi-window burn-rate alerting, and by improving AlertManager routing, inhibition, and deduplication so on-call teams get fewer, more actionable pages.
Core Features & Use Cases
- SLO-based burn rate alerting: Implements critical and warning alerts using short/long evaluation windows to detect both fast and sustained error budget burn.
- Alert quality standards: Enforces required labels (e.g., severity, ownership, paging intent) and runbook-linked annotations to keep alerts actionable.
- Noise reduction with AlertManager: Adds routing by severity (PagerDuty for critical, Slack for warning), groups/deduplicates alerts, and uses inhibition rules to suppress derived child alerts when parent alerts fire.
- Flapping and duration tuning: Uses appropriate for durations and timing guidance to avoid transient spikes triggering pages.
- Weekly alert noise audit & review process: Supports a repeatable workflow to identify top-firing noisy alerts and decide whether to fix, demote, rewrite, or delete them.
- Alert ownership model and escalation: Encourages team labels and ticket-based escalation for warning-level noise to keep warning storms from becoming page storms.
Quick Start
Use the infra-alert-fatigue-reduction Skill to generate SLO burn-rate Prometheus alert rules plus an AlertManager configuration that routes critical pages to PagerDuty, warnings to Slack, and suppresses redundant/derived alerts with inhibition rules.
