observability-and-instrumentation

Structured logging means emitting JSON objects with stable event names and machine-readable fields instead of interpolated strings. Attach a correlation ID at the system boundary via a child logger, use consistent log levels, and never log secrets or full PII.

Use the RED method for request-driven services: rate, errors, and duration as a latency histogram with p50/p95/p99 percentiles. For resources like queues and pools, use USE: utilization, saturation, and errors. Keep label sets bounded to avoid cardinality explosions.

OpenTelemetry is the vendor-neutral standard for tracing and metrics, with auto-instrumentation for HTTP, gRPC, and common database clients. Prometheus prom-client is one common metrics backend; the RED/USE and cardinality rules apply identically either way.

Symptom-based alerts fire exactly when users are hurt, such as error rate above 1% for five minutes, regardless of cause. Cause-based alerts on CPU or disk fire when nothing is wrong and miss unpredicted failures. Every alert must be actionable and link to a runbook.

Do not use it to diagnose a failure happening right now; that belongs to debugging and error recovery workflows. It also does not cover profiling measured slowness or launch-day monitoring checklists, which are handled by performance and shipping skills.