Alerting & Incident Management

Automate alerting and incident management setup with Prometheus, Alertmanager, and PagerDuty.

1|Updated Dec 18, 2024
One-click install
npx skills add https://github.com/altrupets/monorepo --skill alerting-incident-management-altrupets
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: Alerting & Incident Management
Source: https://github.com/altrupets/monorepo/tree/main/skills/system-reliability-engineering/alerting-incident-management
Command: npx skills add https://github.com/altrupets/monorepo --skill alerting-incident-management-altrupets

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the critical need for maintaining reliable services by establishing effective alerting mechanisms and robust incident response workflows, minimizing downtime and impact.

Core Features & Use Cases

  • Alerting Configuration: Design and implement effective alerts using Prometheus and Alertmanager, covering various scenarios like error budget exhaustion, high latency, and service downtime.
  • Incident Management: Define on-call rotations, create runbooks for troubleshooting, and establish clear incident response procedures.
  • Use Case: When a critical service experiences a spike in errors, this Skill can automatically trigger an alert to the on-call engineer via PagerDuty, provide a link to a runbook for immediate troubleshooting, and facilitate communication within a dedicated incident channel.

Quick Start

Configure a complete alerting and incident management system using PagerDuty and Prometheus.

Frequently Asked Questions about Alerting & Incident Management

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I configure Prometheus alert rules and Alertmanager routing for production services?

You can define Prometheus alert rules for error budget exhaustion or high latency, then configure Alertmanager routing and notifications to establish reliable service alerting. The system automates this setup to minimize downtime and impact.

What is the best way to set up on-call rotations and incident management with PagerDuty?

The best way to establish incident management is to define on-call rotations via PagerDuty and create actionable runbooks for troubleshooting. This ensures the on-call engineer receives immediate alerts and clear response procedures during critical service incidents.

How does incident management automation handle critical service error spikes?

Incident management automation triggers an alert to the on-call engineer via PagerDuty during a critical service error spike, provides a runbook link for immediate troubleshooting, and facilitates communication within a dedicated incident channel.

Do I need Prometheus to use this alerting and incident management setup?

Yes, Prometheus is required as the primary system for metrics collection and alerting. The setup integrates directly with Prometheus to define alert rules and relies on Alertmanager to route notifications for production service reliability.

When do I need actionable runbooks for SRE incident response?

You need actionable runbooks for SRE incident response when establishing clear troubleshooting procedures for production services. They guide on-call engineers through resolving critical alerts like service downtime or high latency triggered by Prometheus and Alertmanager.