observability-and-instrumentation

Structured logging emits JSON objects with a stable event name and machine-readable fields instead of interpolated strings. Attach a child logger per request carrying a correlation ID, and never log secrets, tokens, or full request bodies.

Use RED metrics on every endpoint and external dependency: rate, errors, and duration as a histogram. Keep label cardinality bounded to small fixed sets like route template and status class, and read p50/p95/p99 rather than averages.

OpenTelemetry auto-instrumentation covers HTTP, gRPC, and common database clients with near-zero code when the SDK is imported before anything else. Add manual spans only around meaningful internal units of work and propagate context across async boundaries.

Symptom-based alerts fire exactly when users are hurt, regardless of cause, while cause-based alerts like CPU thresholds fire when nothing is wrong and miss unpredicted failures. Every alert must be actionable, link to a runbook, and use only page and ticket severities.

Do not use it to diagnose a failure happening right now; that belongs to debugging workflows. It also does not cover performance profiling of measured slowness or launch-day monitoring checklists, which are separate concerns.