Catalog My learning Blog
CI/CD and DevOps Tutorial: Build a Delivery Pipeline · Phase 4 · Lesson 23 · 3 min read

Blameless Postmortems: How to Run Them

On this page
  1. Why blame makes systems worse
  2. What goes in a postmortem
  3. A five whys example
  4. Postmortem action items that get done
  5. Making blameless culture real
  6. Check yourself

Why blame makes systems worse

After an outage, it's tempting to find the person who pushed the bad change. But that person was working inside a system: a test suite that missed the case, a review that didn't catch it, a deploy that went to everyone at once. Blaming them changes nothing in that system, and teaches everyone to hide mistakes. Next time, nobody mentions the risky shortcut, and the same failure happens again.

A blameless postmortem assumes people acted reasonably with the information and tools they had, and asks what made the failure possible.

What goes in a postmortem

  1. Summary: what happened and the user impact, in a few sentences.
  2. Impact: duration, users and requests affected, revenue or SLO budget lost.
  3. Timeline: from the incident channel (Day 22), including when the problem started, when it was detected, and when it was mitigated.
  4. Contributing factors: the conditions that combined to cause it. Usually several, rarely one "root cause".
  5. What went well, and what was lucky.
  6. Action items: specific, owned, dated changes.

Write it within a few days, review it in a meeting focused on learning, and share it widely.

A blameless postmortem traces an incident through its timeline to the system conditions behind it, and ends in owned, dated action items.
A blameless postmortem traces an incident through its timeline to the system conditions behind it, and ends in owned, dated action items.

A five whys example

Code
Why did checkout errors spike?
  -> A deploy changed payment-retry logic, which failed when retries ran out.
Why did that reach production?
  -> No test covered the retry-exhausted path.
Why was there no test?
  -> The retry logic was added earlier without tests for its failure branch,
     and reviews don't prompt for error-path tests.
Why didn't the canary stop it?
  -> The canary watched overall error rate; the new errors were 2% of checkout
     traffic, below the 5% threshold.
Why was the threshold 5%?
  -> It was set once, for a noisier service, and never revisited.

The answer isn't "Sam broke checkout". It's a set of gaps: missing error-path tests, a review checklist without that prompt, and a canary threshold that was too loose. Each becomes an action item. Five is a rule of thumb; stop when you reach things the team can change.

Postmortem action items that get done

WeakStrong
"Be more careful with retries""Add tests for retry exhaustion in payment client (owner: A, due Oct 18)"
"Improve monitoring""Canary analysis compares error rate against the stable version, threshold 1% (owner: B, due Oct 25)"
"Review checklists""Add 'error paths tested?' to the PR template (owner: C, due Oct 14)"

Track them in the normal backlog with priority, and review open items monthly. A postmortem whose actions are never done is a ritual, not learning.

Making blameless culture real

  • Language: "the deploy introduced", not "Sam introduced". Ask "what made this seem like the right call at the time?"
  • Leaders go first: managers sharing their own mistakes makes it safe for others.
  • Blameless isn't consequence-free for negligence, but honest mistakes in a system that allowed them are the system's problem.
  • Share postmortems across teams; another team's incident is a free lesson. Google, Cloudflare and GitHub publish theirs.

Check yourself

When people aren't punished, they share what really happened, including shortcuts, confusing tools and near-misses. That reveals the system weaknesses that let a mistake cause an incident, which blame would hide.

It's vague, has no owner or due date, and relies on human vigilance, which fails under pressure. A good action changes the system, such as adding a pipeline check or an automatic canary rollback.

Incidents usually need several conditions to line up, such as a bug, a missing alert and a risky deploy time. Fixing only one leaves the others in place for the next failure.

For significant incidents, by severity or user impact, within a few days while details are fresh. Build the timeline from logs, alerts and chat rather than memory, and include near-misses that could have been worse.

Give each one an owner and a due date, track them in the team's normal backlog rather than a forgotten document, and review open items regularly. Prioritize them like other work, or they quietly expire.