Stop 15 · Message Queues & Event-Driven ArchitectureTopics · 8/8

Idempotent Consumers

In short

Queues promise 'at least once', which quietly means 'sometimes twice'. If your worker credits a wallet twice, that's real money gone. Idempotent workers make repeats harmless.

Designing message consumers so processing the same message twice — which queues can occasionally guarantee only 'at-least-once' delivery of — produces the same result as processing it once.

The blob keeps pressing a glowing lift call button while a single lift calmly arrives.

Think of it like this

Pressing the lift button again

You press the lift call button, then press it five more times because you're late. The lift still comes once. Pressing again doesn't call five lifts. An idempotent consumer works the same way: the same message, handled twice, gives the same result as handling it once.

Watch it happen

The same payment event, delivered twice

Queue

Worker

Database

payment evt_9, Rs 500

1/6The worker receives a payment.captured event with a unique id evt_9.

The big ideas

What you need to know

Every message needs an id

Producers attach a unique event id. Without one, a consumer can't tell a duplicate from a new message.

Record and act together

Mark the id as processed in the same database transaction as the change. Two separate steps can be split by a crash.

Naturally idempotent writes

"Set status to shipped" is safe to repeat. "Add 500 to balance" isn't, so it needs the id check.

External calls too

For payment or SMS APIs, pass an idempotency key if the provider supports one, so they skip repeats on their side.

See it in code

Walk through the code

1await db.$transaction(async (tx) => {2  const inserted = await tx.$executeRaw`3    INSERT INTO processed_events (event_id) VALUES (${event.id})4    ON CONFLICT (event_id) DO NOTHING`;5  if (inserted === 0) return; // seen it before, skip6 7  await tx.wallet.update({8    where: { userId: event.data.userId },9    data: { balance: { increment: event.data.amount } },10  });11});12ack(message);

1/3Try to record the event id. If it already exists, zero rows are inserted and we stop here.

processed_events needs a UNIQUE (or primary key) constraint on event_id. Clean out old rows after a few days to keep it small.

Save yourself the pain

Common mistakes

Oops

Checking 'have I seen this id?' with a SELECT, then inserting later.

Do this instead

Two workers can both pass the SELECT. Let a unique constraint decide, inside the transaction.

Oops

Using the queue's internal delivery id as the dedupe key.

Do this instead

Redelivered messages may get a new delivery id. Use an id the producer put in the message.

Your mission

Prove the double credit, then stop it

~30 min

Quick check

Which operation is already idempotent without any extra checks?

Go deeper

1Design a consumer that uses a unique message ID to detect and skip duplicates.
2Learn why 'at-least-once' delivery is far more common than 'exactly-once' in real systems.
3Identify a real operation, e.g. charging a card, where non-idempotent processing would be dangerous.