Prometheus, Alertmanager and Grafana

The common open-source monitoring stack, three separate components:

• Prometheus scrapes and stores time series, and evaluates rules. Pull-based: it fetches /metrics from targets.
• Alertmanager receives alerts from Prometheus and handles grouping, deduplication, silencing and routing to receivers (email, Slack, PagerDuty, …).
• Grafana queries Prometheus and draws dashboards.

Everything below was replayed on Prometheus 3.15.0 and Alertmanager 0.34.1: the configs through promtool check config and amtool check-config, the queries against a running server or through promtool test rules.

Prometheus config

prometheus.yml sets the scrape interval, the rule files, where Alertmanager lives, and what to scrape:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - rules.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: node
    static_configs:
      - targets: ["localhost:9100"]   # node_exporter

static_configs is the simplest target source; in a real fleet you use service discovery (kubernetes_sd_configs, ec2_sd_configs, …) instead. Validate before shipping:

promtool check config prometheus.yml

PromQL, the basics

• up - 1 if a target is scrapeable, 0 otherwise (per instance/job).
• rate(http_requests_total[5m]) - per-second average increase of a counter over 5m.
• sum by (job) (rate(http_requests_total[5m])) - the same, aggregated per job.
• histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) - p95 latency.

Recording and alerting rules

Recording rules precompute expensive queries; alerting rules fire when an expression holds for a duration.

# rules.yml
groups:
  - name: examples
    rules:
      - record: job:up:ratio
        expr: avg by (job) (up)
      - alert: InstanceDown
        expr: up == 0
        for: 5m                      # must hold 5 minutes before firing
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.instance }} of job {{ $labels.job }} is down"

An alert inherits the series labels (job, instance) plus its own labels, and Alertmanager routes on them.

Test the rules

Rules are logic, so test them. promtool test rules feeds synthetic series and asserts which alerts fire:

# tests.yml
rule_files:
  - rules.yml
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: 'up{job="node", instance="a"}'
        values: '0x10'              # up=0 for 10 minutes
    alert_rule_test:
      - eval_time: 6m
        alertname: InstanceDown
        exp_alerts:
          - exp_labels: { severity: page, job: node, instance: a }
            exp_annotations: { summary: "a of job node is down" }

promtool check rules rules.yml
promtool test rules tests.yml

At eval_time: 6m the alert has held past its for: 5m, so the test passes. At eval_time: 4m it has not, and exp_alerts: [] is what passes instead.

A failing test prints both sides, which is also the fastest way to find out what an expression really returns: assert something absurd once, read the answer, then write the assertion.

  FAILED:
    expr: "increase(c_total[5m])", time: 5m,
        exp: {job="x"} 9.99E+02
        got: {job="x"} 3.75E+00

Gotchas

A range shorter than two scrape intervals returns nothing

rate() and increase() need at least two samples inside the range. Scraping every 5 seconds:

rate(prometheus_http_requests_total[5s])   # empty result
rate(prometheus_http_requests_total[1m])   # a value

No error and no warning, just an empty panel or an alert that can never fire. Take a range of at least four scrape intervals, so that one missed scrape does not empty it.

increase() extrapolates, it does not count

Five consecutive samples of a counter, scraped every 5 seconds:

12  13  14  15  16

increase(prometheus_http_requests_total{handler="/metrics"}[25s])   # 5

The counter moved by exactly 4, the function answers 5. rate() and increase() extrapolate to the edges of the range, because samples never land exactly on them, and the result is often not even an integer. It is an estimate of a rate, so do not use it to count events and do not round it into a number you then treat as exact.

rate() on a gauge is silently wrong

rate() assumes a counter, so every decrease is read as a counter reset and the new value is added on top. A gauge that oscillates, one sample per minute:

10  5  10  5  10  5

rate(g_bytes[5m])    # 0.0833 per second, that is 25 over the window

The gauge ends below where it started and the answer is firmly positive. Nothing complains. For gauges, delta() and deriv() are the functions that allow the value to go down.

Counter resets themselves are handled correctly: on 0 1 2 0 1 2, a process restart in the middle of the window, increase(c_total[5m]) gives 3.75, close to the true 4, not the naive last minus first of 2.

histogram_quantile cannot exceed your largest finite bucket

Buckets le="0.1" and le="+Inf", with every observation landing above 0.1:

histogram_quantile(0.95, sum by (le) (rate(h_seconds_bucket[5m])))   # 0.1

Not the real value and not +Inf: the upper bound of the highest finite bucket. A quantile read from a classic histogram is capped by the bucket layout chosen when the metric was instrumented, and it goes quietly flat the day latency grows past the last bucket. A p95 sitting exactly on a bucket boundary is a bucket problem until proven otherwise.

A removed job keeps answering for five minutes

An instant query returns the most recent sample within the lookback window, five minutes by default. Verified on a series whose last sample is at t=2m: at t=6m it still resolves, at t=7m, exactly five minutes later, it is gone.

Removing one target from a job is not affected: Prometheus writes a staleness marker and the series was gone within 15 seconds of the reload. Removing the whole scrape_config job is: its up series kept its last value for the full five minutes. Two consequences:

• An InstanceDown alert on a job you just removed keeps firing for up to five minutes, then resolves on its own.
• The opposite case is worse: up == 0 never fires for a target that stopped being discovered, or that was never discovered, because there is no series at all. absent(up{job="node"}) is what catches that, and it is the alert people write only after the first outage nobody was paged for.

pending is not firing

for: 5m means the expression must hold at every evaluation for five minutes. Until then the alert exists and is visible in Prometheus, but Alertmanager has heard nothing about it:

ALERTS{alertname="InstanceDown", alertstate="pending", instance="a", job="node", severity="page"}   1

alertstate becomes firing once for has elapsed. Querying ALERTS is the quickest way to find out why the page you were expecting never arrived.

Alertmanager

alertmanager.yml groups related alerts, waits before firing, and routes by label to receivers:

route:
  group_by: ["alertname", "job"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: default
  routes:
    - matchers: ['severity="page"']
      receiver: pager
receivers:
  - name: default
  - name: pager
    # slack_configs / pagerduty_configs / email_configs ...

Check it, then walk the routing tree with the labels an alert would carry:

amtool check-config alertmanager.yml
amtool config routes test --config.file=alertmanager.yml severity=page      # pager
amtool config routes test --config.file=alertmanager.yml severity=warning   # default

config routes test prints the receiver that would get the alert. On a nested tree it is the only cheap way to find out that a route never matches, and the alternative is an incident finding out for you. Silences are managed with amtool silence.

Grafana

Point Grafana at Prometheus as a data source, then build or import dashboards (the community dashboard for node_exporter is a good start). In infra-as-code setups, provision data sources and dashboards from files under /etc/grafana/provisioning/ rather than clicking in the UI.

On Kubernetes

Do not wire this by hand on a cluster. The kube-prometheus-stack Helm chart installs Prometheus, Alertmanager and Grafana together via the Prometheus Operator, where you declare scrape targets and rules as ServiceMonitor and PrometheusRule custom resources.

See also

• Official docs: https://prometheus.io/docs/
• Kubernetes and its Kubernetes tooling.