What problem does it solve?
Teams running LLM features in PostHog need continuous oversight of cost, latency, errors, volume, and evaluation performance, but manually watching these metrics across models, products, and users is impractical and regressions go unnoticed.
Core Features & Use Cases
- Multi-lens LLM monitoring: Watches eight observability lenses (cost, latency, errors, volume, eval performance, eval/enrichment config, clusters, tool usage) against per-team baselines learned over time.
- Report authoring and deduplication: Authors validated regression reports directly to the PostHog inbox via emit-report/edit-report, editing existing reports instead of duplicating known issues.
- Durable cross-run memory: Persists baselines, noise patterns, addressed incidents, and reviewer routing in a scratchpad so each run builds on prior findings.
- Use Case: A team's LLM spend doubles after a model swap; the scout localizes the cost step to a specific model and ai_product slice, samples representative traces as evidence, and files a report routed to the owning engineer.
Quick Start
Run the AI observability scout against this PostHog project and report any validated LLM cost, latency, error, or eval regressions you find.
