skills.rest
chmonitor
chmonitorOFFICIAL
Explore all skills by this author →
WebsiteGitHub

incident-response

Triage common ClickHouse operational incidents with diagnostic queries and remediation steps.

255|42|Updated Nov 16, 2023
Search Tags:#replication#triage#clickhouse#incident-response#slow-query#database-operations#disk-management
One-click install
npx skills add https://github.com/chmonitor/chmonitor --skill incident-response-chmonitor
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: incident-response
Source: https://github.com/chmonitor/chmonitor/tree/main/.agents/skills/incident-response
Command: npx skills add https://github.com/chmonitor/chmonitor --skill incident-response-chmonitor

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Unplanned ClickHouse outages, performance degradation, and operational errors cause downtime and data pipeline disruptions, and most teams lack pre-defined, step-by-step procedures to diagnose and fix these issues quickly under pressure.

Core Features & Use Cases

  • Structured Incident Recipes: Pre-written triage workflows for 6 common ClickHouse incident types: disk capacity exhaustion, high error rates, replication lag, stuck mutations/merges, cluster health degradation, and slow query performance.
  • Exact Diagnostic Guidance: Each recipe includes ready-to-run system table queries and tool calls to identify root causes, plus error code interpretation guides to speed up troubleshooting.
  • Safe Remediation Steps: Standard fix procedures with safety notes for irreversible actions, plus cross-references to related optimization and verification skills for complex multi-step incidents.
  • Use Case: If your ClickHouse cluster is reporting disk usage over 85% and insert failures, follow the disk filling recipe to identify the largest tables, estimate time to full capacity, and apply TTL policies or partition drops to free space safely.

Quick Start

Use the incident-response skill to diagnose and resolve a high error rate spike on your self-hosted ClickHouse cluster.

Frequently Asked Questions about incident-response

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I troubleshoot ClickHouse replication lag across my self-hosted cluster?▼

To troubleshoot ClickHouse replication lag, use structured triage procedures that query system tables to identify root causes and apply standard remediation steps to restore sync. This incident response approach covers self-hosted deployments across Docker, Kubernetes, and bare metal.

What is the best way to resolve ClickHouse disk capacity exhaustion and prevent insert failures?▼

The best way to resolve ClickHouse disk capacity exhaustion is to follow a disk filling recipe that identifies the largest tables, estimates time to full capacity, and safely applies TTL policies or partition drops to free space.

How do I diagnose a high error rate spike in ClickHouse?▼

To diagnose a high error rate spike in ClickHouse, run ready-to-use diagnostic system table queries provided by the incident response recipes, then interpret the specific error codes to speed up troubleshooting and apply safe remediation steps.

Does this incident response triage work with ClickHouse Cloud and Kubernetes deployments?▼

Yes, this incident response triage works with ClickHouse Cloud and Kubernetes deployments, as well as self-hosted Docker and bare metal environments. It provides structured procedures covering multiple incident types across these platforms.

How do I fix stuck background mutations and merges in ClickHouse?▼

To fix stuck background mutations and merges in ClickHouse, apply standard remediation steps from the triage recipes which include safety guards for irreversible actions, ensuring you resolve the blockage without risking data integrity.

Why is my ClickHouse slow query performance degrading and how do I triage it?▼

ClickHouse slow query performance degradation can be triaged using pre-written recipes that include exact diagnostic system table queries to identify root causes. The recipes cross-reference related optimization skills for complex multi-step resolution.

Related Skills

Explore More Data & Analytics Skills→
chmonitorchmonitor
255

storage-optimization

Tune compression codecs, TTL policies, and tiered storage for ClickHouse clusters.

Official
Intermediate
chmonitorchmonitor
255

anomaly-detection

Detect abnormal behavior in self-hosted ClickHouse clusters using 1-hour metrics against a 24-hour baseline.

Official
Intermediate
jeremylongshorejeremylongshore
2.6k

monitoring-database-health

Monitor PostgreSQL and MySQL database health with real-time metrics and alerts.

Community
Advanced
hieutrtrhieutrtr
8

monitoring-setup

Implement logging, metrics, tracing, health checks, alerting, and dashboards for Python/React services.

Community
Advanced