observability-and-instrumentation

Structured logging emits JSON objects with a stable event name and machine-readable fields instead of interpolated strings. Attach a child logger per request carrying a correlation ID, and use consistent levels: error for broken invariants, warn for handled degradation, info for business events.

Use the RED method for request-driven services: rate, errors, and duration as a histogram with p50/p95/p99 percentiles. For resources like queues and pools, use USE: utilization, saturation, and errors. Keep label values from small fixed sets to avoid cardinality explosions.

Initialize the OpenTelemetry NodeSDK with auto-instrumentations before importing anything else, which covers HTTP, gRPC, and common database clients. Add manual spans around meaningful internal work and propagate context across HTTP headers and queue message metadata.

Every unique label combination creates a separate time series, so unbounded values like user IDs, raw URLs, or error messages cause cardinality explosions that overwhelm the metrics backend. High-cardinality identifiers belong in logs and traces instead.

Page only for user-facing symptoms requiring immediate action, such as error rate above 1% for five minutes. Use tickets for degradations actionable within the week. Every alert needs a threshold, a duration, and a linked runbook, and cause-based alerts like CPU usage belong on dashboards.