monitoring-setup

Use the prom-client library to define counters and histograms, add middleware that records request counts and durations, and expose a /metrics endpoint. Then configure a Prometheus scrape job targeting that endpoint every 15 seconds.

Define rules in an alerts.yml file with a PromQL expression, a for duration, severity labels, and annotations. Reference the file under rule_files in prometheus.yml and route notifications through Alertmanager to PagerDuty or Slack.

Histograms bucket observations into configurable ranges and support server-side quantile calculation via histogram_quantile, making them aggregable across instances. Summaries calculate quantiles client-side and cannot be aggregated, so histograms are preferred for latency tracking.

Define an availability target such as 99.9%, which allows 43 minutes of downtime per month. Create a burn-rate alert comparing the observed error ratio against the allowed budget, paging when the budget is being consumed faster than sustainable.

Yes, pair Grafana with Loki and promtail to collect structured JSON logs, then query them with LogQL such as {job="app"} |= "error". This lets you correlate log lines with metrics and traces in the same dashboard.

High-cardinality labels create a new time series per unique value, exploding memory usage and slowing queries in Prometheus. Use bounded labels such as endpoint, method, and status code instead of unbounded identifiers.