aps-product-metrics-operations

Start from the product promise and apply Goals-Signals-Metrics: state the goal, identify observable user signals, then pick metrics. Favor outcome metrics like independent solves and delayed re-solves over raw AC counts, and pair every optimization target with a guardrail metric.

Track attempts started, review completion, re-solve after delay, variation solves for transfer, recommendation start and completion rates, and hint dependency over time. These distinguish actual learning and independent solving from mere activity or assistance-inflated results.

Use a consistent Object-Action naming convention, prefer few events with rich properties over many near-duplicate events, and separate exposure, start, completion, and outcome. Maintain a tracking plan documenting triggers, required properties, and owners as the contract between product and engineering.

Define an SLI such as sync success rate or p95 latency, then set an SLO like 99.5% of accounts synced within 5 minutes over 28 days. Treat the error budget as a signal to pause feature work, and surface data freshness to users since stale sync corrupts downstream metrics.

Small user bases produce underpowered tests that miss real effects, so compute required sample size before running. Watch for novelty effects, sample ratio mismatch, and network interference in social features like study groups, which may require cluster-level randomization.

Total AC mixes assisted and independent solving and says nothing about retention or transfer. Goodhart's law applies: once targeted, users game it with easy problems, so pair volume metrics with independence and delayed re-solve counter-metrics.