log-aggregation

Standardize all services on structured JSON logs with common fields like trace_id and service, then deploy shippers such as Filebeat or Vector to forward logs to Elasticsearch or Loki. Configure index templates, retention policies, and alerts on error rates.

ELK with Elasticsearch offers full-text search and rich indexing but costs more storage and memory. Loki with Grafana is Kubernetes-native, indexes only labels, and stores chunks in S3, making it cheaper for high-volume workloads.

Run Filebeat as a DaemonSet reading /var/log/pods with the container parser and add_kubernetes_metadata processor to enrich events with pod and namespace labels. Alternatively, use Vector with the kubernetes_logs transform for the same enrichment.

Define an ILM policy with hot, warm, cold, and delete phases, for example rollover at 50GB, shrink after 2 days, freeze after 30 days, and delete after 90 days. Attach the policy to your index template so all matching indices inherit it.

Common causes include misconfigured shipper output hosts, JSON parsing failures that silently drop events, or index template mismatches. Monitor the pipeline itself by alerting on shipper errors, dropped events, and queue lag.

Include timestamp, level, service, environment, message, trace_id, request_id, and user_id as standard fields across all services. Never log PII such as passwords, tokens, or credit card numbers, and keep DEBUG level out of production.