Incident Response and On-Call Best Practices
On this page
Why incidents need a process
Without a process, an incident looks like this: five people investigate the same log, nobody tells customers, two people roll back different things at once, and the fix that worked isn't written down. A simple, practiced process fixes most of that.
Incident severity levels
Agree in advance what counts as what, so nobody debates it at 2 a.m.:
| Level | Meaning | Response |
|---|---|---|
| SEV1 | Core function down or data at risk for many users (checkout failing) | Page immediately; incident commander; status page update |
| SEV2 | Major feature degraded, or a workaround exists | Page the owning team |
| SEV3 | Minor issue, few users affected | Handle in working hours |
Incident roles: commander, comms and scribe
- Incident commander (IC): coordinates, decides and delegates. The IC doesn't debug; their job is to keep the response organized.
- Operations / subject experts: investigate and apply fixes, as directed.
- Communications lead: updates stakeholders and the status page on a fixed cadence (say every 30 minutes), so engineers aren't interrupted.
- Scribe: keeps a timeline in the incident channel; it becomes the postmortem's backbone.
In a small team, one person may hold several roles, but someone must clearly be in charge.
Mitigate first, then understand
The goal during an incident is to stop the user impact, not to find the root cause. Rolling back, turning off a feature flag, failing over, or adding capacity often restores service in minutes, while diagnosis can wait until users are fine. The root cause is the postmortem's job (Day 23).
An incident, minute by minute
00:00 Alert: checkout-svc error rate 6% (SLO burn rate alert)
00:02 On-call acknowledges, opens #inc-checkout-errors, posts the alert link
00:04 Declares SEV1 (checkout is the revenue path); pages an incident commander
00:06 IC assigns: Engineer A checks logs and traces; Engineer B checks recent changes;
comms lead posts "investigating elevated checkout errors" on the status page
00:09 Engineer B: errors began 3 minutes after the 23:54 deploy marker (Day 21)
00:10 IC: roll back now, investigate later
00:12 Rollback to the previous image complete; error rate falling
00:16 Error rate at baseline for 4 minutes; IC declares mitigated
00:20 Status page: resolved. IC schedules the postmortem for the next working dayNotice what made this fast: a symptom-based alert, deploy markers on dashboards, a rehearsed rollback, and one person deciding.
Sustainable on-call
- Clear escalation: primary, secondary, then a manager, with automatic escalation if a page isn't acknowledged (PagerDuty, Opsgenie, incident.io, Grafana OnCall).
- Runbooks linked from every alert: what it means, how to check, how to mitigate.
- Reasonable load: track pages per shift; if nights are regularly interrupted, fixing alerts and reliability becomes priority work.
- Handoffs: a short note at each shift change about open issues.
- Compensation and recovery time after rough nights.
- Practice: game days and incident drills, so the first real SEV1 isn't the first time anyone has run the process.
Check yourself
Their job is coordination: assigning work, making decisions such as rolling back, and keeping communication flowing. Someone deep in logs loses track of the overall picture, and the response becomes uncoordinated.
Roll back to restore service, then investigate the cause once users are no longer affected. Mitigation comes before diagnosis; a rollback that turns out unnecessary costs little.
Regular updates, such as every 30 minutes, show customers someone is working on it and stop stakeholders from interrupting the responders to ask for information.
Alerts that are actionable and rare, a rotation large enough for rest, compensation or time off, and fixing the causes of repeat pages.
They decide who's paged, how fast to respond and how to communicate, based on user impact, so the response matches the problem.