skills.rest
PostHog
PostHogOFFICIAL
Explore all skills by this author →
Website𝕏@PostHogGitHub

monitoring-ingestion-pipeline

Diagnose PostHog ingestion pipeline health using Grafana MCP metrics, logs, and dashboards.

713|118|Updated Aug 11, 2020
Search Tags:#grafana#prometheus#Kafka#clickhouse#incident diagnosis#ingestion pipeline#Monitoring
One-click install
npx skills add https://github.com/PostHog/posthog-foss --skill monitoring-ingestion-pipeline
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: monitoring-ingestion-pipeline
Source: https://github.com/PostHog/posthog-foss/tree/main/.agents/skills/monitoring-ingestion-pipeline
Command: npx skills add https://github.com/PostHog/posthog-foss --skill monitoring-ingestion-pipeline

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Investigating production ingestion issues at PostHog requires knowing hundreds of metric names, label values, datasource UIDs, and topology details across Kafka, Redis, Postgres, and ClickHouse. This Skill encodes that operational knowledge so an AI agent can monitor and diagnose the Node.js ingestion pipeline workers without memorizing unstable metric names.

Core Features & Use Cases

  • Live metric discovery: Uses Grafana MCP tools like list_prometheus_metric_names and list_loki_label_names to discover current metrics rather than relying on hardcoded names.
  • Full topology reference: Documents deployment roles, Kafka/MSK/WarpStream clusters, Redis instances, Postgres Aurora clusters, ClickHouse cluster roles, Pyroscope services, and dashboard UIDs for prod-us and prod-eu.
  • Investigation playbooks: Twelve step-by-step workflows covering health checks, dropped events, latency diagnosis, consumer lag, person processing, Kafka/Redis/Postgres failures, session replay, ClickHouse lag, and single-partition hot-key incidents.
  • Use Case: When on-call and paged for rising event lag in prod-eu, ask the agent to check ingestion health; it will query E2E lag, consumer group lag, pipeline results, pod restarts, and ClickHouse consumer lag in the correct order.

Quick Start

Ask the agent to check whether the ingestion pipeline is healthy in prod-us and summarize any lag, drops, or restarts.

Frequently Asked Questions about monitoring-ingestion-pipeline

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check if the PostHog ingestion pipeline is healthy?▼

Open the ingestion-health Grafana dashboard, then verify E2E event lag via ingestion_lag_ms, consumer group lag via kminion_kafka_consumer_group_topic_lag_seconds, pipeline results by result label, pod restarts, and ClickHouse consumer lag on clickhouse_events_json.

How to diagnose growing Kafka consumer group lag in ingestion workers?▼

Identify the lagging group with kminion_kafka_consumer_group_topic_lag_seconds, check partition-level lag for hot partitions, compare batch duration against configured limits, verify consumer assignment, and look for rebalance events in Loki logs.

Does this monitoring approach work for both prod-us and prod-eu?▼

Yes, datasource UIDs, dashboard UIDs, and label values are identical across both environments. Differences are documented: ingestion-analytics-turbo exists only in prod-us, and CloudWatch cluster IDs and Kafka partition counts differ by region.

Why is a single Kafka partition lagging while others are healthy?▼

A single lagging partition means cost tied to one team or distinct_id key, since events are keyed by token:distinct_id. Compute events per second and ms per event from batch logs to distinguish volume hot keys from expensive per-event work, then rank by cost using tophog.

What causes ClickHouse ingestion to fall behind even when workers are healthy?▼

ClickHouse consumes from the WarpStream ingestion VC, so check consumer lag scoped to kminion-warpstream-ingestion, KafkaRowsRejected and FailedInsertQuery rates, part count growth toward the 300-part limit, merge pressure, and readonly replicas.

Which Grafana Loki datasource should I use for ingestion logs?▼

Always use the Loki-logs datasource with UID P44D702D3E93867EC. The primary Loki datasource returns intermittent 502 errors in both prod-us and prod-eu and should not be used.

Related Skills

Explore More Software Engineering Skills→
grafanagrafana
535

explore-datasources

Discover Grafana datasources, metrics, and logs for monitoring coverage.

Official
Intermediate
rohitg00rohitg00
944

k8s-autoscaling

Automate Kubernetes autoscaling configuration and validation across HPA, VPA, and KEDA.

Community
Advanced
FaysilAlshareefFaysilAlshareef
4

partition-strategy

Optimize Cosmos DB partition key selection and cross-partition query handling.

Community
Intermediate
stacklokstacklok
2.1k

deploy-otel

Deploys an OpenTelemetry observability stack with Prometheus, Grafana, and Tempo to a Kind cluster.

Official
Intermediate