Stop 13 · Site Reliability Engineering PracticesTopics · 3/8

Postmortems & Blameless Culture

In short

After an outage, you can either find someone to blame or find out why the system let it happen. Only the second one stops it happening again.

A structured review after an incident focused on understanding contributing factors and preventing recurrence — 'blameless' means focusing on systems and processes, not punishing the individual who happened to trigger the issue.

The mascot moves a magnet on a tactics board while two cricket players sit with it and nod.

Think of it like this

The cricket team's match review

A team loses because a catch was dropped. A good coach doesn't shout at the fielder. They ask why he was placed there, whether the sun was in his eyes, how practice went. Fix the field setting and the next fielder won't face the same trap.

Watch it happen

Asking why, not who

Site down 40 min

Bad config

No review needed

No config test

Slow to notice

CPU-only alerts

Require review

Validate in CI

Alert on errors

1/4Start with the impact: what users saw, for how long. Not a name.

The big ideas

What you need to know

Blameless means safe

People explain honestly only when they won't be punished. Honest timelines are the only useful ones.

Timeline first

Write what happened minute by minute from logs and chat. Facts before opinions.

Several causes, not one

Big outages usually need several things to go wrong together. List them all, not just the last click.

Actions with owners

Every postmortem ends with specific tasks, an owner and a due date. Otherwise it's just a sad story.

See it in code

Walk through the code

text

1Postmortem: Checkout errors, 14 Mar, 40 minutes2 3Impact:   ~12% of checkout requests failed (5xx)4Detected: 19:22 by customer tweets, not by alerts5 6Timeline7  19:02  Config change merged and deployed8  19:22  Support reports failures9  19:31  On-call identifies config, starts rollback10  19:42  Errors back to normal11 12Contributing factors13  - Config changes skip review and tests14  - Alerting watched CPU, not error rate15 16Actions17  - Require review on config repo        (owner: Ravi, 21 Mar)

1/4Start with impact and how it was found. Being told by customers is itself a finding.

Notice what's missing: no 'Ravi broke it'. The person who made the change is often the best one to help fix the gaps.

Save yourself the pain

Common mistakes

Oops

Writing 'root cause: human error' and closing the document.

Do this instead

Ask why the system allowed that error and why it wasn't caught. Humans will always slip; systems should catch it.

Oops

Holding postmortems only for huge outages.

Do this instead

Short reviews of small incidents and near misses catch problems while they're still cheap.

Your mission

Write a postmortem for a real slip

~30 min

Quick check

Why do blameless postmortems lead to better fixes?

Go deeper

1Read a public blameless postmortem from a well-known company's engineering blog.
2Learn why blameless culture actually surfaces more honest information than a blame-focused one.
3Draft a postmortem template: timeline, impact, root cause, and action items.