addy-observability-and-instrumentation

Structured logging means emitting JSON log events with stable event names and machine-readable fields instead of string-interpolated prose. Attach a correlation ID generated at the system boundary to every log line, and never log secrets, tokens, or unredacted PII.

Use the RED pattern for every endpoint and external dependency: Rate, Errors, and Duration as a latency histogram. Labels must come from small fixed sets like route templates and status classes, never user IDs or raw URLs, to avoid cardinality explosions.

OpenTelemetry is the vendor-neutral standard and its auto-instrumentation covers HTTP, gRPC, and common database clients with near-zero code. Add manual spans only around meaningful internal units of work and propagate trace context across every async boundary.

Symptom-based alerts fire exactly when users are hurt, regardless of the underlying cause, while cause-based alerts like CPU thresholds fire when nothing is wrong and miss unpredicted failures. Every alert must be actionable, link to a runbook, and use only page and ticket severities.

Do not use it to diagnose a failure happening right now; that belongs to a debugging skill. It also does not cover performance profiling of measured slowness or launch-day monitoring checklists, which are handled by separate skills.