prometheus-configuration

Use kubernetes_sd_configs with the pod role and relabel_configs that keep pods annotated with prometheus.io/scrape: true. Additional relabel rules map the prometheus.io/path and prometheus.io/port annotations to the metrics path and target address.

Define rule groups in a YAML file loaded via rule_files, with each rule specifying a record name and a PromQL expression. Common examples include pre-computing 5-minute request rates, error rate percentages, and P95 latency using histogram_quantile.

Add the prometheus-community Helm repository and install the kube-prometheus-stack chart into a monitoring namespace. You can set retention and storage size via values such as prometheus.prometheusSpec.retention and storageVolumeSize.

Run promtool check config prometheus.yml to validate the main configuration and promtool check rules on your rules files to verify recording and alert rule syntax. You can also test queries with promtool query instant against a running server.

Static configs hardcode targets directly in prometheus.yml, requiring a reload for changes. File-based service discovery reads target lists from JSON or YAML files on a refresh interval, letting you update targets without restarting Prometheus.