debugging-signals-pipeline

Diagnose PostHog signals pipeline failures across Temporal workflows, Docker sandboxes, and ClickHouse embeddings.

713|118|Updated Aug 11, 2020
One-click install
npx skills add https://github.com/PostHog/posthog-foss --skill debugging-signals-pipeline
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: debugging-signals-pipeline
Source: https://github.com/PostHog/posthog-foss/tree/main/products/signals/skills/debugging-signals-pipeline
Command: npx skills add https://github.com/PostHog/posthog-foss --skill debugging-signals-pipeline

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

When a signal never reaches the inbox, a signal-report-summary workflow fails, or a sandbox task run times out, engineers need a systematic way to trace the failure across Temporal workflows, object storage logs, Docker containers, and ClickHouse embeddings instead of guessing.

Core Features & Use Cases

  • End-to-end pipeline tracing: Documents the full flow from signal emission through buffering, grouping, semantic search, and report summary workflows.
  • Temporal workflow inspection: Provides REST API commands to list workflows, inspect history, and examine continued-as-new runs.
  • Sandbox and log debugging: Shows how to read agent JSONL logs from object storage and inspect ephemeral Docker sandbox containers before they are removed.
  • Common failure playbooks: Covers stale ClickHouse embeddings, agentsh network denials, inactivity timeouts, and lost buffered signals with concrete fixes.
  • Use Case: A signal emitted from a Zendesk fixture never produces a report. Use this Skill to check the buffer workflow history, read the sandbox agent log tail for network denials, and verify ClickHouse embeddings were cleaned up.

Quick Start

Ask the assistant to debug why a signal emitted with emit_signals_from_fixture is not reaching the inbox for team 1.

Frequently Asked Questions about debugging-signals-pipeline

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I debug a Temporal workflow that failed locally?

Query the Temporal REST API at localhost:8081 to list recent workflows and fetch workflow history by ID. The history events show which activity failed, including failure messages, timers, and signal names.

How do I emit test signals from fixtures in PostHog?

Run python manage.py emit_signals_from_fixture with a type like zendesk, a team ID, offset, and limit. Always run cleanup_signals first to avoid stale embeddings causing phantom report matches.

Why does SignalReport matching query does not exist happen?

This error occurs when stale embeddings remain in ClickHouse after cleanup because CLICKHOUSE_DATABASE is unset, so cleanup targets the default database instead of posthog. Set CLICKHOUSE_DATABASE=posthog in .env and manually delete stale embedding rows.

How do I read sandbox agent logs from object storage?

Fetch the log_url from the most recent TaskRun model in the Django shell, then read it with object_storage.read. The JSONL log tail reveals agentsh network denials, progress events, and sandbox provisioning messages.

Why does select_repository_activity time out due to inactivity?

The sandbox Claude agent went idle longer than TASKS_INACTIVITY_TIMEOUT_SECONDS, which defaults to two hours. Set it to 30 seconds locally for fast failures, then check the agent log for agentsh network denials or invalid ANTHROPIC_API_KEY.

Can I inspect a Docker sandbox container after the task finishes?

No, sandbox containers are ephemeral and removed after the task run completes. Inspect them while running using docker ps with the task-sandbox name filter, docker exec for processes, and reading /tmp/agent-server.log.