observability-and-instrumentation

Use a JSON logger that emits stable event names with machine-readable fields instead of string interpolation. Create a child logger per request carrying a request ID, and attach that ID to every log line, span, and outbound call.

Track RED metrics on every endpoint and external dependency: rate, errors, and duration as a histogram with p50/p95/p99 percentiles. Keep labels to small fixed sets like route template and status class, never user IDs or raw URLs.

Import the OpenTelemetry NodeSDK with auto-instrumentations before any other module so HTTP, gRPC, and database clients are traced automatically. Add manual spans only around meaningful internal work and propagate context across async boundaries.

Alert on symptoms users feel, such as error rate above 1% or p99 latency over 2 seconds, not causes like CPU. Cause-based alerts fire when nothing is wrong and miss unpredicted failures; causes belong on dashboards.

Every unique label combination creates a separate time series, so unbounded values like user IDs or request IDs cause cardinality explosions that overwhelm the metrics backend. High-cardinality lookups belong in logs and traces instead.

Do not use it to diagnose an active failure, which belongs to debugging workflows, or to profile measured slowness, which belongs to performance optimization. Instrumentation is written alongside the feature to make future diagnosis fast.