Monitoring & Alerting Questions
Part of the DevOps Roadmap
In short
'What makes a good alert?' and 'how would you monitor a new service?' test judgement. Anyone can install Prometheus; interviewers want to know you'd page people only for things that matter.
'How would you set up monitoring for a new service?' or 'what makes a good alert?' — testing Phase 11's observability judgment, not just tool familiarity.
Think of it like this
The pressure cooker whistle
You don't stand staring at the cooker. You go read a book, and the whistle tells you when to act. A good alert is that whistle: it sounds only when something needs you. A cooker that whistled every minute would just be ignored.
Watch it happen
Two ways to alert
Cause-based
- CPU above 80%
- Fires even when users are fine
- Many alerts, easy to ignore
Symptom-based
- Error rate above 2% for 10 minutes
- Fires only when users are hurting
- Few alerts, each worth acting on
Page on symptoms users feel. Keep cause metrics like CPU on dashboards for diagnosis, or as low-priority tickets, not 3am pages.
The big ideas
What you need to know
Golden signals
Latency, traffic, errors and saturation. Start every new service's monitoring with these four.
Use percentiles
Averages hide pain. p95 or p99 latency shows what your slowest users actually experience.
Every alert has a runbook
An alert should say what's wrong and link to steps. If nobody knows what to do, it's not ready.
Tie alerts to SLOs
Alerting on how fast you're burning the error budget catches real problems and ignores harmless blips.
See it in code
Walk through the code
1# p95 latency over 5 minutes (PromQL)2histogram_quantile(0.95,3 sum by (le) (rate(http_request_duration_seconds_bucket[5m])))4 5# Error ratio over 5 minutes6sum(rate(http_requests_total{status=~"5.."}[5m]))7 / sum(rate(http_requests_total[5m]))1/2histogram_quantile estimates the 95th percentile from latency buckets. le is the bucket's upper bound label.
Being able to write or read these two queries covers a lot of monitoring interview ground. The metric names follow common client library defaults.
Save yourself the pain
Common mistakes
Oops
Answering 'I'd set alerts on CPU, memory and disk for every server'.
Do this instead
Start from user impact: errors and latency on key endpoints. Resource alerts only where they predict real trouble, like disk filling.
Oops
Talking only about tools.
Do this instead
Explain thresholds, durations, routing and who gets woken up. That's the judgement they're testing.
Your mission
Design monitoring for one service
Quick check
Why use p95 latency instead of average latency for alerts?
Go deeper
- 1Review Phase 11 and practice explaining the 3 pillars (metrics, logs, traces) and when each helps.
- 2Prepare to discuss what makes an alert actionable versus noisy.
- 3Practice explaining SLIs/SLOs/error budgets in plain, non-jargon language.
More incident & reliability at this stop
Troubleshooting Scenarios
Incident Response Questions
Stuck on this topic? Ask an Insider
Get 1:1 guidance from people who've walked this exact path — on the InsideEdge app.
