skills.rest
chmonitor
chmonitorOFFICIAL
Explore all skills by this author →
WebsiteGitHub

storage-optimization

Tune compression codecs, TTL policies, and tiered storage for ClickHouse clusters.

255|42|Updated Nov 16, 2023
Search Tags:#compression#clickhouse#ttl#storage-optimization#merge-tree#tiered-storage#disk-recovery
One-click install
npx skills add https://github.com/chmonitor/chmonitor --skill storage-optimization
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: storage-optimization
Source: https://github.com/chmonitor/chmonitor/tree/main/.agents/skills/storage-optimization
Command: npx skills add https://github.com/chmonitor/chmonitor --skill storage-optimization

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Self-hosted ClickHouse deployments often suffer from unnecessarily high storage costs, slow query performance, and disk space outages caused by unoptimized compression settings, unmanaged data lifecycle policies, and unmonitored MergeTree part counts.

Core Features & Use Cases

  • Compression Codec Tuning: Select optimal per-column codecs (LZ4, ZSTD, Delta, DoubleDelta) to reduce storage footprint without sacrificing query speed for different data types.
  • TTL & Tiered Storage Management: Configure automated data retention and tiering rules to move cold data to cheaper storage volumes and delete expired data to free up disk space.
  • Part & Disk Health Monitoring: Track MergeTree part counts, detached parts, and disk usage to prevent performance degradation and disk-full outages.
  • Use Case: A team running a 10TB time-series ClickHouse cluster can use this skill to implement Delta+ZSTD compression on timestamp columns, move 90-day-old data to S3 cold storage, and set TTL on system logs to reclaim 60% of disk space.

Quick Start

Use the storage-optimization skill to analyze your ClickHouse cluster's current storage configuration and recommend compression codecs, TTL policies, and tiered storage rules to reduce costs and improve query performance.

Frequently Asked Questions about storage-optimization

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I reduce ClickHouse storage costs and query latency for large time-series datasets?▼

Reduce ClickHouse storage costs by tuning per-column compression codecs like LZ4, ZSTD, Delta, and DoubleDelta, configuring TTL policies for automated data retention, and moving cold data to tiered storage volumes to improve query performance.

What's the best way to configure ClickHouse TTL and tiered storage for cold data?▼

Configure ClickHouse TTL and tiered storage by setting automated data retention rules that move older data to cheaper storage volumes like S3 and delete expired records, freeing disk space and cutting storage overhead for time-series clusters.

How do I select the right compression codec for different ClickHouse column data types?▼

Select ClickHouse compression codecs by matching data type characteristics to algorithms like Delta for timestamps, DoubleDelta for monotonic values, and ZSTD for general compression, balancing storage footprint reduction against query speed requirements.

Why does ClickHouse slow down or run out of disk space with high MergeTree part counts?▼

ClickHouse slows down or faces disk-full outages due to unmonitored MergeTree part counts, detached parts, and system table bloat, requiring part health tracking and TTL configuration to mitigate storage degradation.

Does this ClickHouse storage optimization approach work on Kubernetes and Docker deployments?▼

This approach supports ClickHouse storage optimization across Kubernetes, Docker, bare metal, and ClickHouse Cloud deployments, enabling database administrators to implement compression tuning and tiered storage regardless of the underlying infrastructure.

Can I reclaim disk space from system logs in ClickHouse to prevent outages?▼

Reclaim ClickHouse disk space from system logs by setting TTL policies to automatically delete expired data, monitoring detached MergeTree parts, and mitigating system table bloat to prevent disk-full outages.

Related Skills

Explore More Data & Analytics Skills→
ntk148vntk148v
42

altinity-expert-clickhouse-storage

Diagnose storage and IO bottlenecks in ClickHouse using SQL queries on system tables.

Community
Intermediate
hiromailyhiromaily
126

verify-buildkit-cache

Verify BuildKit cache persistence and restoration in GitHub Actions CI logs.

Community
Advanced
kevinnguyen271090kevinnguyen271090
15

databases

Design and optimize MongoDB and PostgreSQL databases for schema, queries, and performance.

Community
Intermediate
patricio0312revpatricio0312rev
5

data-retention-archiving-planner

Plan and enforce data retention and archival policies across hot and cold storage.

Community
Advanced