Stop 23 · Interview PreparationTechnical · 8/8

Debugging distributed systems

In short

"Checkout fails for some users, sometimes" is a nightmare when five services are involved. Interviewers ask about this to see if you debug with evidence or with guesses.

Being able to describe how you'd track down a bug in a system spanning multiple services — logs, tracing, and systematic isolation — since debugging a single process is a different skill than debugging a distributed one.

The purple blob detective follows footprints from a food stall past a scooter to a house.

Think of it like this

Tracing a lost food order

Your biryani never arrived. Did the restaurant get the order? Did the rider pick it up? Did the app crash? With one order ID you can check each step's record and find exactly where it broke. Trace IDs do that for requests.

Watch it happen

Following one failed checkout

App

Gateway

Orders

Payments

POST /checkout

1/6A user taps Pay. The gateway creates request ID r-7f3a.

The big ideas

What you need to know

Correlation IDs

Give every request an ID and pass it to every service. OpenTelemetry and the traceparent header do this as a standard.

Structured logs

JSON logs with requestId, userId, route and duration can be searched and filtered, unlike free text.

Metrics and percentiles

Averages hide pain. p95 and p99 latency show the slow requests that real users complain about.

Usual suspects

Timeouts, retry storms, exhausted connection pools, partial deploys and clock differences between machines.

See it in code

Walk through the code

1import pino from "pino";2import { randomUUID } from "node:crypto";3const logger = pino();4app.use((req, res, next) => {5  const requestId = req.header("x-request-id") ?? randomUUID(); // reuse the caller's ID6  req.log = logger.child({ requestId });7  res.setHeader("x-request-id", requestId);8  next();9});10 11// Pass it on to every downstream call12await fetch("http://payments:4003/charge", {13  method: "POST",14  headers: { "x-request-id": req.log.bindings().requestId, "content-type": "application/json" },15  signal: AbortSignal.timeout(3000),16});

1/3Reuse an incoming ID or create one, attach it to a child logger, and echo it back.

Every log line from this request now carries the same requestId, in every service. AbortSignal.timeout makes the downstream call fail fast.

Save yourself the pain

Common mistakes

Oops

Adding retries everywhere to make errors go away.

Do this instead

Retries on top of retries can multiply load on a struggling service. Retry with backoff, limits and only for safe operations.

Oops

Guessing the cause and redeploying until it seems fixed.

Do this instead

Collect evidence first: logs for one request ID, metrics around the time, and what changed recently.

Your mission

Trace one request end to end

~40 min

Quick check

Average latency is 120 ms, but users complain checkout is often slow. What should you look at?

Go deeper

1Prepare a real story of debugging an issue that spanned more than one service or component.
2Learn the basic idea of distributed tracing — a trace ID following a request across services.
3Practice a structured answer: reproduce, isolate which service is at fault, form a hypothesis, verify.

More technical at this stop