principle-observability

Use logs for discrete events, metrics for time-based aggregations, and distributed tracing for causal latency and dependency mapping. Selecting the right observability signal prevents unreliable monitoring and ensures you capture the correct system behavior.

Design SLO alerts by defining SLIs with error budgets and alerting on symptoms tied to user pain. Use the RED or USE methods for consistent instrumentation to ensure alerts are actionable and reflect real user impact.

Metric cardinality refers to the number of unique label combinations in time-based aggregations. Control it by applying strict cardinality-safe metric design, preventing observability failure modes and backend resource exhaustion during service instrumentation.

Implement distributed tracing by applying OpenTelemetry-compatible trace propagation across services. This enables causal latency attribution and dependency mapping, ensuring you can track requests through complex distributed systems.

Use histogram-based duration metrics when monitoring latency to support SLO error budget calculations. They provide accurate time-based aggregations for latency attribution, which are essential for debugging user-visible incidents.

The RED method measures Rate, Errors, and Duration for consistent observability instrumentation. Apply it alongside SLIs and SLOs to define error-budget thinking and create symptom-driven alert criteria for user-visible incidents.