observability-and-instrumentation

Structured logging means emitting JSON log events with stable event names and machine-readable fields instead of interpolated strings. Attach a correlation or request ID at the system boundary and include it on every log line, and never log secrets, tokens, or full PII.

Use the RED method: rate, errors, and duration as a latency histogram for every endpoint and external dependency. Keep label cardinality bounded to small fixed sets like route template and status class, and read p95/p99 percentiles rather than averages.

OpenTelemetry is the vendor-neutral standard for tracing and metrics, with auto-instrumentation covering HTTP, gRPC, and common database clients. Prometheus prom-client is one common metrics backend; the RED/USE and cardinality rules apply identically either way.

Symptom-based alerts fire exactly when users are hurt, such as error rate above 1% or p99 latency above 2 seconds, regardless of cause. Cause-based alerts like CPU at 85% fire when nothing is wrong and miss failures you did not predict.

Do not use it to diagnose a failure happening right now; that belongs to debugging and error recovery workflows. It also does not cover profiling measured slowness or launch-day monitoring checklists, which are separate concerns.