logging-strategy

Define log categories (application, access, error, audit, security), use structured JSON with required fields like request_id and service, and set retention tiers. Aggregate centrally with ELK, Datadog, or CloudWatch rather than reading logs on individual servers.

Sample 1-10% of successful responses on high-volume endpoints like health checks, but always retain 100% of ERROR and CRITICAL logs. Consistent sampling based on request_id ensures complete traces for sampled requests.

ELK offers full control without per-GB pricing but requires operational overhead. Datadog provides managed dashboards and APM integration at roughly $0.10/GB ingested. CloudWatch Logs is cheap and AWS-native but has a limited query interface.

Keep 7-30 days in hot storage for active debugging, 90 days in warm storage like S3 Standard, and 1+ years in cold storage like S3 Glacier. Audit logs for compliance may require 7-year retention and must be immutable.

Hash fields like email and phone with MD5 before logging, and fully redact highly sensitive data like SSNs. Automate redaction with a sanitize function that processes known sensitive fields before any log write occurs.

Synchronous logging blocks request handling while waiting for I/O. Use asynchronous handlers like Python's QueueHandler, batch log shipping every 10 seconds or 1000 entries, and compress logs with gzip to reduce overhead.