Stop 13 · Site Reliability Engineering PracticesTopics · 2/8

On-Call Practices

In short

Someone has to answer when the site breaks at 2am. Good on-call means that person has clear alerts, a runbook, backup to call, and still gets to sleep most nights.

The rotation and process for responding to production issues outside business hours — good on-call practices include clear escalation paths, reasonable rotation frequency, and protecting engineers from burnout.

The mascot in a doctor's coat walks at night holding a beeping pager and a cup of chai.

Think of it like this

The night-duty doctor

Hospitals don't make every doctor stay awake every night. One doctor takes night duty this week, with a senior on call if things get serious. Clear handover notes in the morning. Next week someone else takes the pager. On-call rotations borrow exactly that idea.

Watch it happen

A page at 2:14 am

Alerting

Primary

Secondary

Incident chat

PAGE: checkout 5xx > 5%

1/6The alert names the service, the symptom and links to a runbook. It goes to this week's primary.

The big ideas

What you need to know

Fair rotation

Spread on-call across enough people that each gets a week every month or two, not every other week.

Clear escalation

Primary, then secondary, then a manager or expert. Everyone knows who to call next and when.

Page only for action

A page must mean 'a human must act now'. Everything else becomes a ticket or a daytime notification.

Protect rest

After a rough night, the on-call person comes in late or hands off. Tired engineers make worse calls.

How it works

A healthy on-call week

1
Handover in
The previous on-call shares open issues, recent deploys and anything flaky to watch.
2
Check your setup
Phone alerts working, laptop charged, VPN and dashboards logging in fine.
3
Respond and note
Acknowledge quickly, follow the runbook, write short notes as you go.
4
Handover out
Pass on what happened and file tickets for alerts that were noisy or unclear.

Save yourself the pain

Common mistakes

Oops

Paging on every CPU spike, so the on-call person gets 40 pages a night.

Do this instead

Page on user-facing symptoms like errors and latency. Too many pages and real ones get missed.

Oops

Expecting one hero who knows everything to handle every incident.

Do this instead

Write runbooks, rotate fairly and train new people with shadow shifts. Knowledge in one head is a risk.

Your mission

Design a rotation for a small team

~20 min

Quick check

Which alert deserves to wake someone up at night?

Go deeper

1Research a public on-call policy (several companies publish theirs) and note its structure.
2Learn why alert quality (Phase 11) directly determines on-call quality of life.
3Consider the trade-offs of different rotation lengths (weekly vs daily) for sustainability.