observability-and-instrumentation

Structured logging means emitting JSON log events with stable event names and machine-readable fields instead of string-interpolated prose. Attach a correlation ID at the system boundary, use consistent log levels, and never log secrets, tokens, or full PII.

Use the RED pattern: rate, errors, and duration as a latency histogram for every endpoint and external dependency. For resources like queues and pools, use USE: utilization, saturation, and errors. Keep label cardinality bounded to small fixed sets like route templates and status classes.

OpenTelemetry is the vendor-neutral standard for tracing and metrics, with auto-instrumentation covering HTTP, gRPC, and common database clients. Prometheus prom-client is one common metrics backend; the RED/USE and cardinality rules apply identically either way.

Every unique label combination creates a separate time series, so unbounded values like user IDs, raw URLs, or error messages cause cardinality explosions that overwhelm the metrics backend. High-cardinality lookups belong in logs and traces instead.

Do not use it to diagnose an active failure; that belongs to debugging and error recovery workflows. It also does not cover performance profiling of measured slowness or launch-day monitoring checklists, which are separate concerns.