skills.rest
lurodrisilva
lurodrisilvaCOMMUNITY
Explore all skills by this author →
GitHub

observability-stack

Configures Prometheus, Grafana, OpenTelemetry, Loki, Tempo, and Alertmanager observability with SLO burn-rate alerting.

Updated Apr 11, 2026
Search Tags:#monitoring#observability#Prometheus#Grafana#opentelemetry#SLO#alertmanager
One-click install
npx skills add https://github.com/lurodrisilva/personal-skills --skill observability-stack-lurodrisilva
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: observability-stack
Source: https://github.com/lurodrisilva/personal-skills/tree/main/operations/observability-stack
Command: npx skills add https://github.com/lurodrisilva/personal-skills --skill observability-stack-lurodrisilva

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Teams running self-hosted observability struggle to correlate metrics, logs, and traces, control cardinality costs, and design alerts that page on real user impact instead of noisy thresholds. This Skill encodes the full OSS observability discipline — three signals in one context, OpenTelemetry instrumentation, everything as code, and SLO-driven alerting — so you build and operate the stack correctly from the start. ## Core Features & Use Cases - Metrics & Rules: PromQL recording and alerting rules, ServiceMonitor/PodMonitor discovery, relabeling, cardinality control, and long-term storage via remote-write, Thanos, or Mimir. - Collection & Correlation: OpenTelemetry Collector pipelines (memory_limiter first, batch last), tail-based sampling with a load-balancer tier, Loki/LogQL and Tempo/TraceQL with exemplar-driven trace-to-log correlation. - Dashboards, SLOs & Alerting: Grafana dashboards-as-code (JSON, provisioning, grafana-operator, Terraform), RED/USE methods, Sloth/OpenSLO error budgets, multi-window multi-burn-rate alerts, and Alertmanager routing trees with inhibition. - Read-only validators: Three bash tools wrap promtool, otelcol validate, and amtool to check configs and simulate routing before changes reach CI. - Use Case: A service's p99 latency spikes. You follow an exemplar from the Prometheus histogram into the exact Tempo trace, pivot to its structured Loki logs via trace_id, find the failing downstream call, and confirm the SLO burn-rate alert paged the right on-call team. ## Quick Start Ask the AI to design an SLO with multi-window burn-rate alerts and an Alertmanager routing tree for your checkout service using Prometheus and Sloth.

Frequently Asked Questions about observability-stack

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up SLO burn-rate alerts with Prometheus?▼

Define the SLO declaratively with Sloth or OpenSLO, which generates the recording rules and multi-window multi-burn-rate alerting rules. A typical page alert uses 1h and 5m windows at roughly 14.4x burn, while a ticket alert uses 6h and 30m at about 6x, routed by severity in Alertmanager.

How to correlate metrics, logs, and traces in Grafana?▼

Correlation works through exemplars linking metric samples to a trace_id, structured logs carrying trace_id and span_id, and consistent OpenTelemetry resource labels across all signals. Grafana derived fields and trace-to-logs datasource config turn a trace_id into one-click jumps between Tempo traces and Loki log lines.

Why is my Prometheus running out of memory?▼

Prometheus OOMs are usually caused by cardinality explosions from unbounded label values like user IDs, full URLs, or request IDs. Find offenders with topk queries or /tsdb-status, then drop high-cardinality metrics and labels using metric_relabel_configs before ingestion.

Does tail sampling in the OpenTelemetry Collector need special setup?▼

Yes, tail sampling requires every span of a trace to reach the same collector instance. Place a load-balancing exporter tier that routes by trace_id in front of the sampling collectors, otherwise partial traces are silently dropped.

Should trace_id be a Loki stream label?▼

No, trace_id as a Loki stream label explodes the indexed stream count because Loki indexes labels. Keep trace_id inside the structured log line and extract it at query time with the json parser instead.

When should I use Dynatrace instead of the OSS observability stack?▼

Use Dynatrace when you want a commercial managed APM with OneAgent, Grail storage, and DQL querying rather than assembling and operating Prometheus, Grafana, Loki, and Tempo yourself. The OSS stack suits teams wanting vendor-neutral, self-hosted tooling they fully own.

Related Skills

Explore More Software Engineering Skills→
trailofbitstrailofbits
6.5k

debug-buttercup

Diagnose Kubernetes pod issues by analyzing Redis, queues, and logs.

Official
Advanced
Shakudo-ioShakudo-io

k8s-troubleshooter

Diagnose and remediate Kubernetes incidents and performance issues in production clusters.

Official
Advanced
fzymgc-housefzymgc-house

grafana

Manage Grafana, Loki, and Prometheus operations for Kubernetes observability.

Official
Advanced
mf-dynatracemf-dynatrace

dt-obs-azure

Monitor Azure resources using Dynatrace Smartscape and DQL queries.

Community
Advanced