signals-scout-replay-vision

Monitors PostHog Replay Vision scanners for observation gaps and aggregate cross-session signal shifts.

713|118|Updated Aug 11, 2020
One-click install
npx skills add https://github.com/PostHog/posthog-foss --skill signals-scout-replay-vision
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: signals-scout-replay-vision
Source: https://github.com/PostHog/posthog-foss/tree/main/products/signals/skills/signals-scout-replay-vision
Command: npx skills add https://github.com/PostHog/posthog-foss --skill signals-scout-replay-vision

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Replay Vision scanners judge one session at a time, so two failure modes stay invisible: an enabled scanner silently stops observing (throughput cliffs, failures, quota exhaustion), and aggregate trends across sessions (rising yes-rates, shifting scores, concentrating tags) never get surfaced. This Skill watches both.

Core Features & Use Cases

  • Observing integrity checks: Detects enabled scanners whose observation throughput collapses, whose observations fail or land ineligible, or whose org quota is exhausted, using SQL over $recording_observed plus the vision MCP tools.
  • Aggregate signal detection: Compares each scanner's verdict rate, mean score, or tag distribution against its own prior-week baseline to surface cross-session shifts no per-session scan can emit.
  • Report authoring with dedupe: Authors or edits prioritized inbox reports via the report channel, deduping against the per-session push path and sibling scouts, and persists scanner baselines in a scratchpad.
  • Use Case: A monitor scanner's yes-rate creeps from 8% to 22% week-over-week across 200 sessions. The scout quantifies the shift against the scanner's own baseline, links example recordings, and files a P2 report for the owning team.

Quick Start

Run the replay vision scout to check whether all enabled scanners are still observing and whether any scanner's aggregate output has shifted from its baseline.

Frequently Asked Questions about signals-scout-replay-vision

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I detect when a Replay Vision scanner stops observing sessions?

Compare each enabled scanner's observation count this week versus the prior week on `$recording_observed` events, grouped by stable scanner_id. Then use vision-scanners-observations-list with status filters to distinguish a stopped scanner from one whose observations all fail or land ineligible.

How do I find aggregate trends across Replay Vision scanner sessions?

Query daily yes-rates for monitors, mean scores for scorers, or tag shares for classifiers over the past 28 days, comparing the latest week against the scanner's own prior weeks. Require roughly 30 sessions per week before treating a shift as meaningful.

Why does arrayJoin on scanner_output_tags fail in HogQL?

scanner_output_tags is a JSON-encoded string, not a native array, so a bare arrayJoin errors or returns garbage. Wrap it in JSONExtract(..., 'Array(String)') first, and union scanner_output_tags_freeform to avoid missing freeform tags.

Why does a scanner show zero $recording_observed events?

Only succeeded observations write events, so zero rows can mean no scanners exist or that every observation is failing, ineligible, or quota-skipped. Check vision-scanners-list for enabled scanners and vision-quota-retrieve for org-wide exhaustion before concluding non-adoption.

Can this scout modify or trigger Replay Vision scanners?

No, its scopes are read-only for scanner configuration and observations. When an aggregate finding warrants a sharper standing watch, it recommends a scanner change in the report and lets the team decide.

When should a scanner shift be skipped instead of reported?

Skip disabled scanners, drops explained by a dated config edit, low-volume scanners under 30 sessions per week, single-session findings owned by the push path, and anything already covered by a dedupe, noise, or addressed scratchpad entry.