optimizing-clickhouse-and-hogql-queries

Diagnoses and rewrites slow ClickHouse and HogQL queries using EXPLAIN plans and measured benchmarks.

713|118|Updated Aug 11, 2020
One-click install
npx skills add https://github.com/PostHog/posthog-foss --skill optimizing-clickhouse-and-hogql-queries
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: optimizing-clickhouse-and-hogql-queries
Source: https://github.com/PostHog/posthog-foss/tree/main/.agents/skills/optimizing-clickhouse-and-hogql-queries
Command: npx skills add https://github.com/PostHog/posthog-foss --skill optimizing-clickhouse-and-hogql-queries

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Slow ClickHouse and HogQL queries in PostHog cause timeouts, out-of-memory errors, and sluggish insights, and it is hard to know whether the fix belongs in the query runner, the HogQL printer, or a schema migration. This Skill provides a structured workflow to extract the actual ClickHouse SQL, identify common performance smells, measure rewrites against a real cluster, and apply the fix at the correct layer.

Core Features & Use Cases

  • Smell detection: Scans generated ClickHouse SQL for known anti-patterns such as FROM ... FINAL, JSONExtract over raw properties, missing skip indexes, self-joins on events, and inlined CTE blow-ups.
  • Measurement-driven optimization: Guides running EXPLAIN variants and timing original versus rewritten queries on a test cluster, reporting query_duration_ms, read_bytes, and memory_usage from system.query_log.
  • Layered fixes: Directs the fix to the right layer — query runner, new HogQL aggregation function, printer rule, or ClickHouse migration — and includes a case-study learnings log and an investigation playbook for root-causing individual slow queries.
  • Use Case: A trends insight times out in production. Pull the slow query from query_log_archive, find it wraps the timestamp sort key in a function that defeats granule pruning, confirm with a side-by-side EXPLAIN diff, and fix the printer rule with a snapshot test.

Quick Start

Ask the agent to optimize a slow HogQL query or insight by providing the query text or a slow query ID from the production query log.

Frequently Asked Questions about optimizing-clickhouse-and-hogql-queries

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize a slow HogQL query in PostHog?

First get the ClickHouse SQL the HogQL compiles to via execute_hogql_query's response.clickhouse or prepare_and_print_ast. Scan it for smells like JSONExtract on properties or FROM FINAL, then measure a rewrite against the test cluster and apply the fix in the query runner, printer, or a migration.

How do I find slow ClickHouse queries in production?

Query posthog.query_log_archive (or system.query_log for recent rows) filtering is_initial_query, type = 'QueryFinish', and a query_duration_ms threshold. The lc_* tag columns identify the query kind, route, dashboard, or background job that triggered it.

Does this skill cover Postgres or Django ORM query optimization?

No. It explicitly excludes Postgres, Django ORM querysets, and personhog gRPC calls. Those require pganalyze and the Postgres section of the query-performance-optimization handbook instead.

Why is FROM person FINAL slow in ClickHouse?

FINAL on a ReplacingMergeTree forces an on-the-fly merge across every part read, defeating parallel reads and inflating memory. Depending on the case, the rewrite is argMax per group, LIMIT 1 BY with ORDER BY version DESC, or filtering before FINAL.

When should JSONExtract be replaced with properties.X in HogQL?

Always, for printer-path queries. The HogQL printer emits the best available form (materialized column, property group, or JSONExtract fallback) at print time, so properties.X is never worse than hand-written JSONExtract and improves automatically when a column is later materialized.

Why is my ClickHouse query not using its skip index?

Common causes are a function wrapper like nullIf hiding the materialized column, a comparison against a stringified NULL, or the column missing from test fixtures. Confirm with EXPLAIN PLAN indexes=1 and assert index use with get_index_from_explain in tests.