observability-and-instrumentation

Structured logging emits JSON objects with stable event names and machine-readable fields instead of interpolated strings. Attach a correlation ID via a child logger per request, and use consistent levels: error for broken invariants, warn for handled degradation, info for business events.

Track RED metrics on every endpoint and external dependency: Rate, Errors, and Duration as a latency histogram. Use labels from small fixed sets like route template and status class, and read p50/p95/p99 percentiles rather than averages.

OpenTelemetry is the vendor-neutral standard for tracing and metrics with auto-instrumentation for HTTP, gRPC, and common DB clients. Prometheus prom-client is one common metrics backend; the RED/USE and cardinality rules apply identically either way.

Every unique label combination creates a separate time series, so unbounded values like user IDs, raw URLs, or error messages explode cardinality and overload the metrics backend. High-cardinality lookups belong in logs and traces instead.

Do not use it to diagnose a failure happening right now; that belongs to debugging workflows. It also does not cover profiling measured slowness or launch-day monitoring checklists, which are separate concerns.