observability-and-instrumentation

Emit JSON log events with stable event names and machine-readable fields instead of string interpolation. Attach a correlation ID at the request boundary via middleware and use a child logger per request so every line can be traced back to one request.

Use the RED method: Rate, Errors, and Duration for every endpoint and external dependency. Track latency as a histogram so p95 and p99 percentiles are queryable, and keep label sets bounded to route templates and status classes.

Yes, OpenTelemetry auto-instrumentation covers HTTP, gRPC, and common database clients with near-zero code. Initialize the NodeSDK with getNodeAutoInstrumentations before importing anything else, then add manual spans only around meaningful internal units of work.

Every unique label combination creates a separate time series, so unbounded values like user IDs, raw URLs, or error messages cause cardinality explosions that overwhelm the metrics backend. High-cardinality lookups belong in logs and traces instead.

Alert on user-facing symptoms like error rate above 1% or p99 latency above 2 seconds, since cause-based alerts like CPU usage fire when nothing is wrong and miss unpredicted failures. Every alert must be actionable, link a runbook, and have a justified threshold.

It is not for diagnosing an active failure, which belongs to debugging workflows, nor for profiling measured slowness, which belongs to performance optimization. It covers the instrumentation written alongside features that makes those later investigations fast.