skills.rest
PostHog
PostHogOFFICIAL
Explore all skills by this author →
Website𝕏@PostHogGitHub

signals-scout-inbox-validation

Re-measures resolved PostHog Signals reports after a soak window to detect fixes that did not hold.

713|118|Updated Aug 11, 2020
Search Tags:#regression detection#error tracking#PostHog#fix validation#Monitoring#inbox reports#soak window
One-click install
npx skills add https://github.com/PostHog/posthog-foss --skill signals-scout-inbox-validation
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: signals-scout-inbox-validation
Source: https://github.com/PostHog/posthog-foss/tree/main/products/signals/skills/signals-scout-inbox-validation
Command: npx skills add https://github.com/PostHog/posthog-foss --skill signals-scout-inbox-validation

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Merged fix PRs are assumed to work, but nobody verifies them: a resolved report is a promise that the underlying problem stopped, and broken promises go unnoticed. This Skill closes the loop by re-measuring the data behind recently resolved inbox reports after a deploy soak window and filing a report when a fix failed.

Core Features & Use Cases

  • Validation queue management: Enqueues newly resolved reports with merge timestamps, pre-fix baselines, and probe plans, capped per run and persisted in a scratchpad.
  • Multi-ladder re-probing: Re-measures error-tracking issues, log patterns, experiments, and other source surfaces post-soak, then checks fresh-signal recurrence via document_embeddings SQL and sibling reports.
  • Verdict discipline: Classifies outcomes as held, failed, inconclusive, or unverifiable with disqualifier rules (deploy lag, quiet surfaces, cold backlog), authoring a fresh cited report only on failed validations.
  • Use Case: A PR merged two days ago claiming to fix a 310-occurrences/day error issue; the scout re-queries the issue post-soak, finds it still firing at 290/day, and files a P2 failed-validation report citing the original report and PR.

Quick Start

Run the inbox validation scout to check whether any recently resolved reports have fixes that did not hold.

Frequently Asked Questions about signals-scout-inbox-validation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I verify that a merged fix PR actually resolved the reported problem?▼

Enqueue the resolved report with its merge timestamp and a pre-fix baseline, wait at least 24 hours as a soak window, then re-measure the same entities. Compare post-soak rates against the baseline; comparable or rising rates mean the fix failed.

What is a soak window in fix validation?▼

A soak window is the delay between a PR merging and validation, defaulting to 24 hours, because a merged PR is not necessarily deployed. Client-side and mobile fixes need 72 hours or more since app store rollouts take days to weeks.

When should a failed validation become a new report instead of an edit?▼

A failed validation is always authored as a fresh report citing the original resolved report, because edit_report cannot reopen a resolved report. Edits are only used to append fresh numbers to a still-open failed-validation report you authored earlier.

Why does a quiet surface not confirm a fix worked?▼

Zero post-merge events may mean the surface has no traffic at all, not that the fix worked. Check a denominator such as overall event volume or the service's total log rate before concluding the fix held.

What are the limitations of validating dismissed reports?▼

Dismissal rationale and timing are not readable, so dismissed reports are left alone unless the underlying entity escalates to at least twice its report-era rate at meaningful volume. Only then is one P3 report per run authored.

Related Skills

Explore More Software Engineering Skills→
elasticelastic
71

docs-crosslink-validator

Validate cross-links in Elastic documentation markdown files and report broken links.

Official
Intermediate
wimpysworldwimpysworld
716

review-code-follow-up

Rechecks prior code review findings against the author's response delta and writes a follow-up report.

Official
Advanced
PostHogPostHog
58

feature-flags-nextjs

Manage and evaluate PostHog feature flags in Next.js apps.

Official
Advanced
agentmattersagentmatters
3

canary-watch

Monitor deployed URLs for HTTP status, console errors, and performance regressions.

Official
Advanced