Stop 13 · Site Reliability Engineering PracticesTopics · 8/8

Toil Reduction & Automation Mindset

In short

If you spend every Monday resetting the same test accounts by hand, that's toil. It never ends, it doesn't make the system better, and it grows with traffic. SREs write code to make it disappear.

Toil is manual, repetitive operational work that doesn't add lasting value — SRE culture explicitly prioritizes automating toil away, freeing engineering time for higher-leverage reliability and platform work.

The mascot sips chai beside a running washing machine while an old bucket and washing stone sit aside.

Think of it like this

From bucket to washing machine

Washing clothes by hand every day takes an hour, and the pile only grows with more family members. Buying a washing machine costs money and setup once, then the work runs itself. Automating toil is spending effort once to stop paying it every week.

Watch it happen

Is this task worth automating?

Manual

Repetitive

Automatable

No lasting value

Grows with scale

1/5Task: manually renewing TLS certificates on 12 servers every few months. Let's check it against the signs of toil.

The big ideas

What you need to know

Toil has a shape

Manual, repetitive, automatable, reactive, with no lasting value, and it grows as the service grows.

Cap it

Google's SRE book suggests keeping toil under about half of an SRE's time. The rest goes to engineering work.

Do the maths

Ten minutes a day is about 60 hours a year. If a script takes four hours to write, it pays back fast.

Automate safely

Start with a script you run yourself, add checks and logging, then schedule it. Bad automation repeats mistakes quickly.

See it in code

Walk through the code

bash

1#!/usr/bin/env bash2# Delete app log files older than 14 days (was a weekly manual job)3set -euo pipefail4 5LOG_DIR="/var/log/myapp"6DAYS=147 8echo "Before: $(du -sh "$LOG_DIR" | cut -f1)"9find "$LOG_DIR" -name "*.log" -type f -mtime +"$DAYS" -print -delete10echo "After:  $(du -sh "$LOG_DIR" | cut -f1)"11 12# Schedule with cron, every day at 03:00:13# 0 3 * * * /usr/local/bin/clean-logs.sh >> /var/log/clean-logs.out 2>&1

1/4set -euo pipefail stops the script on the first error or unset variable, so it fails loudly instead of half-working.

Run it once with -print but without -delete to see what it would remove. For many apps logrotate does this job, so check before writing your own.

Save yourself the pain

Common mistakes

Oops

Automating a messy manual process exactly as it is.

Do this instead

First simplify the process, remove steps that aren't needed, then automate what remains.

Oops

Writing a clever script only you understand and keeping it on your laptop.

Do this instead

Put it in Git with a README, run it from a shared scheduler and make failures send an alert.

Your mission

Kill one chore

~30 min

Quick check

Which of these is toil?

Go deeper

1List 3 recurring manual tasks from your own practice work and rank them by automation priority.
2Learn Google's SRE definition of toil and why it's tracked as a specific, minimized category of work.
3Practice automating one small recurring task from earlier phases of this roadmap.