generating-clickhouse-query-performance-reports

Generate slow-query performance reports for PostHog production ClickHouse clusters.

713|118|Updated Aug 11, 2020
One-click install
npx skills add https://github.com/PostHog/posthog-foss --skill generating-clickhouse-query-performance-reports
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: generating-clickhouse-query-performance-reports
Source: https://github.com/PostHog/posthog-foss/tree/main/.agents/skills/generating-clickhouse-query-performance-reports
Command: npx skills add https://github.com/PostHog/posthog-foss --skill generating-clickhouse-query-performance-reports

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Investigating slow queries, OOMs, and timeouts across PostHog's production ClickHouse clusters (US and EU) requires knowing the right data source, attribution columns, and report structure; this Skill provides the complete methodology so the analysis is consistent and unbiased.

Core Features & Use Cases

  • Slow-query analysis over query_log_archive: Query the multi-week query_log_archive table with typed lc_* columns instead of the hours-only system.query_log, with ready-to-run SQL for headline summaries, daily distributions, and category breakdowns.
  • Attribution and root-cause workflows: Rank offenders by team, API key, and product using cluster-hours, OOM counts, bytes, and CPU-seconds, then form hypotheses about causes like unmaterialized JSONExtract or high-cardinality breakdowns.
  • Materialization candidate analysis: Identify JSON-extracted event and person properties worth materializing and materialized columns safe to drop, across both regions.
  • Use Case: When asked for a slow query report over the last 14 days, run the step-by-step queries via the query-clickhouse-via-metabase skill, write a structured markdown report with linked example queries, and diff it against the previous report.

Quick Start

Generate a ClickHouse slow query performance report for the US cluster covering the last 14 days, including OOM attribution and materialization candidates.

Frequently Asked Questions about generating-clickhouse-query-performance-reports

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I analyze slow ClickHouse queries over multiple days?

Query the posthog.query_log_archive Distributed table, which retains roughly three weeks of data, instead of system.query_log which only keeps a few hours. Filter is_initial_query and use the slow predicate query_duration_ms > 30000 OR exception_code IN (159, 160, 241).

How do I attribute ClickHouse OOMs and timeouts to specific teams?

Group OOM rows (exception_code = 241) by team_id, lc_product, lc_access_method, and lc_api_key_label from query_log_archive. If one team or API key dominates, the cluster-wide spike is really a single-tenant incident rather than a systemic problem.

Why should I not use system.query_log for slow query reports?

system.query_log on production clusters retains only a few hours, so it cannot answer multi-day questions. The query_log_archive table retains about three weeks and exposes log_comment as typed lc_* columns, avoiding JSONExtract.

How do I find ClickHouse properties worth materializing?

Run the JSON-extracted property breakdown query that extracts property names from JSONExtract calls in slow queries, split by event, person, and group property columns. Properties hit by many slow queries across several teams are the strongest materialization candidates.

Can I identify AI-written HogQL queries in the query log?

Yes, filter on lc_product IN ('max_ai','mcp') or lc_feature IN ('mcp','posthog_ai') to find queries from the Max assistant and MCP server. Do not use ai_query_source, which records the LLM-analytics table resolver choice instead.

Where should the finished query performance report be saved?

Save it as analysis/<YYYY-MM-DD>-<topic>.md in the sibling private query-performance-analysis repo, never in the public posthog repo. If that repo is not checked out, write to a temp folder such as /tmp and tell the user the path.