Stop 12 · Monitoring & ObservabilityTopics · 4/8

Alerting & Notification Systems

In short

An alert that fires ten times a night for nothing teaches your team to ignore alerts. Good alerting means few alerts, each one worth getting out of bed for.

Alerts should be actionable, not noisy — every alert that fires should require a real human response, connected to a notification channel (email, Slack, PagerDuty) the right people actually see.

The blob character jumps up in a kitchen as a pressure cooker whistles with steam.

Think of it like this

The pressure cooker whistle

When the cooker whistles, someone in the house gets up and turns off the gas. It means "act now". If it whistled every two minutes for no reason, everyone would stop listening, and one day the dal would burn.

Watch it happen

From a bad number to a human

Metric

Condition

Alarm

SNS topic

On-call

Runbook

1/6The load balancer reports 5xx errors every minute.

The big ideas

What you need to know

Alert on symptoms

Page on what users feel, like errors and latency, not every high CPU reading.

Avoid flapping

Require several bad data points in a row, like 3 out of 5, before firing.

Severity levels

Page a human only for urgent issues. Send the rest to a ticket queue or a channel.

Every alert has a runbook

If nobody knows what to do when it fires, it isn't ready to page anyone.

See it in code

Walk through the code

bash

1aws cloudwatch put-metric-alarm \2  --alarm-name api-5xx-high \3  --namespace AWS/ApplicationELB --metric-name HTTPCode_Target_5XX_Count \4  --dimensions Name=LoadBalancer,Value=app/web/abc123 \5  --statistic Sum --period 60 --threshold 20 \6  --comparison-operator GreaterThanThreshold \7  --evaluation-periods 5 --datapoints-to-alarm 3 \8  --treat-missing-data notBreaching \9  --alarm-actions arn:aws:sns:ap-south-1:123456789012:oncall

1/4Watch 5xx responses from targets behind this specific load balancer.

"3 out of 5" filters out single blips. notBreaching means no data, like no traffic at night, doesn't trigger the alarm.

Save yourself the pain

Common mistakes

Oops

Alerting on CPU above 70% on every server.

Do this instead

High CPU can be normal. Alert on user-facing errors and latency, and graph CPU on a dashboard.

Oops

Sending all alerts to one busy Slack channel nobody owns.

Do this instead

Route each alert to a clear owner, and page only for urgent ones.

Your mission

Get paged by your own app

~30 min

Quick check

Which alert is most likely to be worth paging someone at 3 a.m.?

Go deeper

1Set up an alert on a key metric with a notification to a real channel.
2Learn the concept of alert fatigue and why over-alerting is more dangerous than under-alerting.
3Practice writing an alert with clear context about what to do when it fires.