skills.rest
chmonitor
chmonitorOFFICIAL
Explore all skills by this author →
WebsiteGitHub

replication-guide

Diagnose and resolve ClickHouse ReplicatedMergeTree replication lag, replica failures, and split brain scenarios.

255|42|Updated Nov 16, 2023
Search Tags:#replication#failover#clickhouse#keeper#quorum#replicatedmergetree#lag-diagnosis
One-click install
npx skills add https://github.com/chmonitor/chmonitor --skill replication-guide
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: replication-guide
Source: https://github.com/chmonitor/chmonitor/tree/main/.agents/skills/replication-guide
Command: npx skills add https://github.com/chmonitor/chmonitor --skill replication-guide

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill eliminates operational friction for self-hosted ClickHouse operators managing ReplicatedMergeTree clusters, addressing common pain points like replication lag, stuck readonly replicas, split brain scenarios, and Keeper connectivity failures that disrupt data availability and cluster consistency.

Core Features & Use Cases

  • Replication Health Monitoring: Quickly diagnose cluster health using system tables to check replication delay, queue size, and replica status.
  • Failover & Recovery Workflows: Follow safe, step-by-step procedures to restore readonly replicas, remove corrupted replicas, and sync diverged data without accidental data loss.
  • Quorum & Keeper Configuration: Configure quorum writes to protect critical data from node failures, and manage ClickHouse Keeper (the modern ZooKeeper replacement) for reliable metadata storage.
  • Use Case: If a ClickHouse replica becomes readonly after a temporary network outage, use this Skill to verify Keeper connectivity, restore write access, and confirm all replicas are fully caught up.

Quick Start

Use the replication-guide skill to diagnose why your ClickHouse ReplicatedMergeTree replica is stuck in readonly mode and restore normal write access.

Frequently Asked Questions about replication-guide

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I fix ClickHouse ReplicatedMergeTree replication lag and stuck readonly replicas?▼

To fix ClickHouse ReplicatedMergeTree replication lag and readonly replicas, diagnose cluster health using system tables to check replication delay, queue size, and replica status, then follow safe failover procedures to restore write access and sync diverged data.

Why does my ClickHouse replica become readonly after a network outage?▼

A ClickHouse replica becomes readonly after a network outage due to lost Keeper connectivity. You must verify Keeper connectivity, restore write access, and confirm all replicas are fully caught up to resolve the split brain scenario.

How do I configure quorum writes in ClickHouse to protect critical data from node failures?▼

To configure quorum writes in ClickHouse, set appropriate quorum parameters on your ReplicatedMergeTree tables to ensure data consistency and protect critical data from node failures during temporary network outages or hardware issues.

Does this ClickHouse replication guide apply to self-hosted deployments on Kubernetes and Docker?▼

Yes, this ClickHouse replication guide applies to self-hosted deployments on Docker, Kubernetes, or bare metal, covering failover procedures, quorum write configuration, and ClickHouse Keeper management for production workloads.

What is the best way to manage ClickHouse Keeper for reliable metadata storage and recover from split brain?▼

The best way to manage ClickHouse Keeper and recover from split brain is to use step-by-step recovery workflows to verify Keeper connectivity, remove corrupted replicas, and sync diverged data without accidental data loss.

How do I monitor ClickHouse replication health and diagnose queue backlog?▼

To monitor ClickHouse replication health and diagnose queue backlog, query system tables to check replication delay, queue size, and replica status, enabling reliable health monitoring and data consistency recovery for production clusters.

Related Skills

Explore More Software Engineering Skills→
chmonitorchmonitor
255

version-upgrade-advisor

Plan ClickHouse version upgrades with pre-upgrade checks and post-upgrade validation.

Official
Intermediate
imMamdouhaboammarimMamdouhaboammar

agency-database-reliability-engineer

Guide database reliability engineering for high availability, disaster recovery, and safe schema migrations.

Community
Advanced
coreweavecoreweave
7

deploy-sunk-on-gke

Automate SUNK deployment on Google Kubernetes Engine with cluster and Slurm setup.

Official
Advanced
microwindmicrowind
65

CAP定理应用

Analyze CAP trade-offs in distributed systems to guide architecture decisions.

Official
Advanced