Stop 08 · Serverless & Event-Driven ArchitectureTopics · 5/9

Message Queues (SQS, Pub/Sub)

In short

When a sale starts and 10,000 orders land in a minute, your email service can't keep up. A queue lets orders pile up safely and get processed at a pace the workers can handle.

A buffer between producers and consumers of messages — decoupling services so a slow or temporarily unavailable consumer doesn't block the producer, and enabling reliable, asynchronous processing.

The blob character as a cook pulls an order slip from a spike beside a dosa griddle.

Think of it like this

The dosa counter spike

Waiters jam order slips onto a metal spike. The cook pulls one, makes the dosa, pulls the next. A rush just means a taller pile, not a crashed kitchen. Waiters never wait for the cook. That spike is a queue.

Watch it happen

Orders through an SQS queue

enqueue(order #101)

order #101

Front of the queue is coloured

1/6The checkout service sends a message and immediately replies to the customer.

The big ideas

What you need to know

Buffer and decouple

The producer doesn't need the consumer to be up, fast or even deployed yet. Messages wait.

Visibility timeout

A received message hides for a while. Delete it when done, or it comes back for another try.

Dead-letter queue

After a set number of failed attempts, a message moves to a DLQ so one bad message doesn't block everyone.

Queue vs pub/sub

SQS gives each message to one worker. Pub/sub topics like SNS or Google Pub/Sub copy it to every subscriber.

See it in code

Walk through the code

python

1import json, boto32 3sqs = boto3.client("sqs")4URL = "https://sqs.ap-south-1.amazonaws.com/123456789012/orders"5 6sqs.send_message(QueueUrl=URL, MessageBody=json.dumps({"orderId": 101}))7 8resp = sqs.receive_message(QueueUrl=URL, MaxNumberOfMessages=10, WaitTimeSeconds=20)9for msg in resp.get("Messages", []):10    order = json.loads(msg["Body"])11    print("sending receipt for", order["orderId"])12    sqs.delete_message(QueueUrl=URL, ReceiptHandle=msg["ReceiptHandle"])

1/4Producer side: send a message and move on.

WaitTimeSeconds=20 turns on long polling: fewer empty responses and fewer billed requests.

Save yourself the pain

Common mistakes

Oops

Deleting the message as soon as it's received, before processing.

Do this instead

Delete after the work succeeds. Otherwise a crash mid-way loses the order forever.

Oops

Setting visibility timeout shorter than the job takes.

Do this instead

Make it longer than your slowest processing time, or two workers will handle the same message.

Your mission

Feel the buffer

~30 min

Quick check

A worker crashes after receiving a message but before deleting it. What happens?

Go deeper

1Set up a message queue and practice sending and receiving a message.
2Learn the difference between a queue (one consumer per message) and a topic/pub-sub (many consumers per message).
3Understand dead-letter queues and why they matter for handling repeatedly failing messages.