observability-apm-expert

What problem does it solve?

Distributed systems often lack coherent telemetry, leaving teams blind to cross-service performance issues, missed error traces, and noisy alerts; this Skill provides structured guidance to design tracing, metrics, and logging so incidents are detectable and actionable.

Core Features & Use Cases

• Sampling Strategy & Trace Retention: Tailored guidance to choose 100% error retention, tail-based sampling for high-cardinality services, and policies for slow-trace capture.
• Backend Selection & Integration: Recommendations for self-hosted Grafana stack (Tempo, Mimir, Loki) or SaaS vendors (Datadog, Honeycomb) plus OTLP collector configuration and fallbacks.
• Alerting and SLOs: SLO definition, error-budget calculation, burn-rate thresholds, and runbook linkage to reduce alert fatigue and speed response.
• Incident Investigation Playbooks: Stepwise trace-first triage, correlation with infrastructure metrics, and remediation actions for common root causes like DB pool exhaustion.
• Instrumentation Guidance: Practical advice for OpenTelemetry SDK usage, context propagation, log->trace correlation, and business-metric instrumentation with bounded cardinality.

Quick Start

Describe your service topology and ask for a recommended OpenTelemetry sampling strategy, backend choice, and the SLO/alert configuration to apply.