monitoring-and-alerting

To reduce alert fatigue, monitoring and alerting should page humans only for actionable user-impact symptoms. This involves applying symptom-over-cause alerting, using thresholds with time windows, hysteresis, and suppression to ensure alerts fire only when truly needed.

Defining SLI, SLO, and SLA targets requires translating user tolerance into measurable metrics using RED or USE methodologies. This ensures success targets are derived from user experience rather than internal metrics, maintaining a correct SLI/SLO/SLA chain and calculating error budgets.

Designing dashboards and runbooks for incident readiness requires a dashboard hierarchy and alert design checklists covering ownership and escalation. This provides responders with immediate 2am context, ensuring they can quickly diagnose and resolve actionable user-impact symptoms.

Setting up RED and USE metrics for new service launch observability involves translating what users feel, such as speed and failures, into SRE golden signals. This grounds the monitoring system in user impact and prepares actionable alerts before the service goes live.

An alert audit plan using action-rate metrics evaluates whether existing alerts are actually actionable. It identifies noisy or missing alerts, verifies SLI/SLO correctness, and drives alert fatigue reduction by ensuring pages only trigger for genuine user-impact symptoms.

Symptom-over-cause alerting applies to existing systems with missing alerts by shifting focus to user-impact symptoms rather than underlying causes. This approach uses RED/USE-based metrics and SLI/SLO targets to catch incidents that traditional cause-based monitoring misses.