infra-alert-fatigue-reduction

What problem does it solve?

This Skill reduces alert fatigue by replacing noisy threshold alerts with SLO-based multi-window burn-rate alerting, and by improving AlertManager routing, inhibition, and deduplication so on-call teams get fewer, more actionable pages.

Core Features & Use Cases

• SLO-based burn rate alerting: Implements critical and warning alerts using short/long evaluation windows to detect both fast and sustained error budget burn.
• Alert quality standards: Enforces required labels (e.g., severity, ownership, paging intent) and runbook-linked annotations to keep alerts actionable.
• Noise reduction with AlertManager: Adds routing by severity (PagerDuty for critical, Slack for warning), groups/deduplicates alerts, and uses inhibition rules to suppress derived child alerts when parent alerts fire.
• Flapping and duration tuning: Uses appropriate for durations and timing guidance to avoid transient spikes triggering pages.
• Weekly alert noise audit & review process: Supports a repeatable workflow to identify top-firing noisy alerts and decide whether to fix, demote, rewrite, or delete them.
• Alert ownership model and escalation: Encourages team labels and ticket-based escalation for warning-level noise to keep warning storms from becoming page storms.

Quick Start

Use the infra-alert-fatigue-reduction Skill to generate SLO burn-rate Prometheus alert rules plus an AlertManager configuration that routes critical pages to PagerDuty, warnings to Slack, and suppresses redundant/derived alerts with inhibition rules.