Catalog My learning Blog
Observability Course: Metrics, Logs, Traces, OpenTelemetry · Phase 4 · Lesson 24 · 3 min read

On-Call Best Practices and Alert Routing

On this page
  1. Why alert routing matters
  2. Alertmanager routing, grouping and inhibition
  3. Escalation policies
  4. Making on-call sustainable
  5. Check yourself

Why alert routing matters

A network partition takes down a cluster. Every service in it fires its own alerts: 200 pages in two minutes, to three teams, none of whom can fix the network. The real signal is buried. Routing is about delivering one actionable notification to the team that can act.

Alertmanager routing, grouping and inhibition

Prometheus sends firing alerts to Alertmanager, which routes them by their labels. This config passes amtool check-config:

YAML
route:
  receiver: team-tickets              # default: nothing pages unless a rule below says so
  group_by: [alertname, service]
  group_wait: 30s                     # collect related alerts before the first notification
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: [severity="page", team="payments"]
      receiver: payments-pager
    - matchers: [severity="page"]
      receiver: platform-pager
    - matchers: [severity="ticket"]
      receiver: team-tickets

inhibit_rules:
  # If a whole cluster is down, don't also page for every service in it
  - source_matchers: [alertname="ClusterUnreachable"]
    target_matchers: [severity="page"]
    equal: [cluster]

receivers:
  - name: payments-pager
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/payments-pd-key
  - name: platform-pager
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/platform-pd-key
  - name: team-tickets
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack-webhook
        channel: "#alerts-tickets"
        send_resolved: true

amtool config routes test shows where alerts with given labels go:

Config
severity=page team=payments alertname=CheckoutErrorBudgetFastBurn -> payments-pager
severity=page team=search alertname=SearchLatency                 -> platform-pager
severity=ticket team=payments alertname=DiskWillFill              -> team-tickets
alertname=Unlabeled                                               -> team-tickets

Routes match top to bottom, first match wins, so the specific payments route sits above the general paging route. An alert missing labels falls to the safe default, a ticket channel, rather than waking someone. Credentials live in files, not in the config.

  • Grouping combines alerts with the same alertname and service into one notification.
  • Inhibition suppresses symptom alerts while a cause alert is firing in the same cluster.
  • Silences mute known alerts during planned maintenance.
Alertmanager groups related alerts, suppresses dependent ones with inhibition, routes by team and severity, and escalates if a page isn't acknowledged.
Alertmanager groups related alerts, suppresses dependent ones with inhibition, routes by team and severity, and escalates if a page isn't acknowledged.

Escalation policies

Paging tools (PagerDuty, Opsgenie, incident.io, Grafana OnCall) add schedules and escalation:

Code
step 1, at 0 min:   primary on-call (push notification, then phone call)
step 2, at 5 min:   secondary on-call, if not acknowledged
step 3, at 15 min:  engineering manager

Every page needs an acknowledgment; an unacknowledged page escalates automatically rather than being lost.

Making on-call sustainable

  • Every page links a runbook and a dashboard (Days 11 and 12).
  • Measure the load: pages per shift, out-of-hours pages, time to acknowledge. Google's SRE guidance suggests at most about two incidents per 12-hour shift; more means the system or the alerts need work.
  • Review alerts weekly: for each page, was it actionable? Tune or delete the ones that weren't.
  • Follow the sun where teams span time zones, so nights are covered by someone's daytime.
  • Hand off with a short note at each shift change.
  • Compensate and recover: time off after bad nights, and on-call pay.
  • Developers who build a service share its on-call, which tends to make services quieter fast (see the CI/CD course).

Check yourself

Alerts missing routing labels then land somewhere safe and visible, like a team channel, instead of paging the wrong person or disappearing. Review that receiver to fix the labels.

It suppresses some alerts while another is firing, such as per-service pages during a whole-cluster outage, so responders see the one alert that matters instead of dozens of symptoms.

A page that nobody acknowledges, because the on-call person is asleep or their phone is dead, would otherwise be lost. Escalation to a secondary and then a manager ensures someone responds.

It batches alerts with the same labels, such as cluster and alert name, into one notification, so 50 failing pods send one page instead of 50.

Track pages per shift, out-of-hours pages and repeat alerts. Regular sleep interruptions or pages without action point to alerts to fix.