Stop 15 · Message Queues & Event-Driven ArchitectureTopics · 1/8

Message Queue Concepts

In short

Why should a user stare at a spinner while your server sends an email and builds a PDF? Queues let you reply fast and do the slow work a moment later.

A middleman that holds messages between producers and consumers, decoupling them in time and allowing work to be processed asynchronously, reliably and at a different pace than it's created.

The blob at a bank counter picks the front envelope from a neat line of envelopes on a tray.

Think of it like this

The bank token counter

At the bank you take a token and sit down. The person at the counter calls tokens one by one, and if a second counter opens, tokens get served twice as fast. Nobody crowds the window, and you're free to scroll your phone. A message queue is that token system for work.

Watch it happen

Jobs waiting in line

enqueue(email #1)

email #1

Front of the queue is coloured

1/6The signup API adds a welcome-email job to the back of the queue and replies to the user straight away.

The big ideas

What you need to know

Producer, broker, consumer

The producer adds messages, the broker stores them, and consumers (workers) take them off and do the work.

Acknowledgements

A worker says 'done' after finishing. If it crashes first, the broker gives the message to another worker.

Smooths out spikes

A sudden rush fills the queue instead of crashing your app. Workers drain it at a steady pace.

At-least-once delivery

Most queues may deliver a message twice, for example after a crash. Design workers so doing a job twice is safe.

See it in code

Walk through the code

1// Without a queue: the user waits for every slow step2app.post("/signup", async (req, res) => {3  const user = await createUser(req.body);4  await sendWelcomeEmail(user);   // slow5  await generateInvoicePdf(user); // slower6  res.status(201).json(user);7});8 9// With a queue: do the must-have part, hand off the rest10app.post("/signup", async (req, res) => {11  const user = await createUser(req.body);12  await queue.add("welcome-email", { userId: user.id });13  await queue.add("invoice-pdf", { userId: user.id });14  res.status(201).json(user);15});

1/3Everything runs inside the request. If the email provider is slow or down, signup is slow or fails.

Send ids, not whole objects, in job data. The worker loads fresh data when it runs, so it never works on stale copies.

Save yourself the pain

Common mistakes

Oops

Putting work in a queue that the user needs in the response.

Do this instead

Queue only what can happen later. Creating the user stays in the request; the welcome email can wait.

Oops

Assuming each message is delivered exactly once.

Do this instead

Plan for duplicates. Make workers idempotent, so a repeat does no harm.

Your mission

Time the difference

~25 min

Quick check

A worker crashes halfway through a job before acknowledging it. What does a typical queue do?

Go deeper

1Diagram a scenario where a queue smooths out a traffic spike, e.g. sending emails.
2Learn the basic vocabulary: producer, consumer, queue, message broker.
3Understand why queues improve reliability — a message waits until a consumer is ready.