Every event-driven system eventually hits a wall. You’re processing messages, things are humming along, and then, a retry. Or a consumer crash mid-processing. Or a network blip that makes your message queue replay something you already handled. Suddenly, you’ve debited the same account twice, shipped the same order again, or sent that welcome email for the fifth time.
The “at-least-once delivery” promise in your queue’s documentation sounds reasonable until you’re explaining to your CTO why customer data got duplicated. Your message queue is not your salvation, it’s the start of the problem.
The Inbox Pattern is the most battle-tested solution for consumer-side idempotency, but most implementations get the critical detail wrong: the transaction boundary. You cannot separate your deduplication check from your business logic. Do it, and you’re back to square one.
Why Your Queue’s “Exactly-Once” Label Is Marketing, Not Engineering
Let’s clear this up immediately. Amazon SQS FIFO queues advertise “exactly-once processing.” AWS calls it that right in the documentation. Here’s what it actually means: the queue won’t introduce duplicates from producer retries within a five-minute deduplication window.
That’s it.
The consumer can still run the same message twice. Here’s the scenario that breaks the promise:
Worker A receives the message
Worker A commits status=shipped to the database
Worker A crashes before DeleteMessage
Visibility timeout expires
Worker B receives the same message
Worker B runs the handler again
The queue hasn’t duplicated the message. It redelivered it because your worker never confirmed it was done. But to your business logic, that looks exactly like a duplicate. The queue-level guarantee stops at the queue boundary.
This isn’t just an SQS issue. Kafka’s idempotent producer works the same way, it prevents duplicates within a single producer session, but if your consumer application restarts and gets a new Producer ID, sequence numbers reset to zero and the broker can’t detect duplicates anymore. The producer-side deduplication table below shows exactly where the gaps live:
| Limitation | Description |
|---|---|
| Session-based | Idempotency only guaranteed within single producer session |
| Partition scope | No deduplication across different partitions |
| Topic scope | No deduplication across different topics |
| Memory | Brokers maintain state per producer-partition |
Your queue is doing its job. The problem is that you need something between the queue and your data.
The Inbox Pattern: Brute Force and Transactional Elegance
The Inbox Pattern is deceptively simple on paper: before processing a message, record that you’ve seen it. If you see it again, skip it. The trick is when and how you record that fact.
The naive implementation looks like this:
Receive message with ID: evt_abc123
Check inbox table: is evt_abc123 already there? → No
Process business logic (update account balance, create order, etc.)
Insert evt_abc123 into inbox table
Delete message from queue
Spot the problem. If your service crashes between step 3 and step 4, you’ve already committed the business effect but failed to record the deduplication marker. When the message gets redelivered, your check in step 2 says “not processed yet”, and you run the business logic again. Congratulations, you’ve just doubled the order.
The entire pattern hinges on making steps 3 and 4 atomic. You can’t split the transaction.
The Only Implementation That Works: One Transaction, One Truth
Here’s the correct approach. Your inbox table and your business data must live in the same database, and the deduplication check and business update must execute in the same database transaction.
BEGIN;
WITH claimed AS (
INSERT INTO consumed_message (consumer_name, message_id, processed_at)
VALUES ('order-worker', 'evt_abc123', now())
ON CONFLICT (consumer_name, message_id) DO NOTHING
RETURNING message_id
)
UPDATE orders
SET status = 'shipped', version = version + 1
WHERE order_id = 'order-417'
AND EXISTS (SELECT 1 FROM claimed);
COMMIT;
The ON CONFLICT DO NOTHING (or INSERT ... WHERE NOT EXISTS in databases without upsert support) is doing the heavy lifting. If this message_id has already been claimed, the INSERT does nothing, the subquery returns empty, and your UPDATE doesn’t execute. Entirely inside one transaction, either both happen or neither does.
Delete the queue message only after this transaction commits. A replay becomes a duplicate no-op because the inbox row is already there.
Unique Constraints: Your Real Defense Against Race Conditions
Every design discussion about the Inbox Pattern eventually hits the same question: what about concurrent consumers trying to process the same message? If two workers grab the same message and both check the inbox table before either commits, you get a race condition.
The database’s unique constraint is your weapon here. Put a UNIQUE or PRIMARY KEY constraint on (consumer_name, message_id). When two transactions attempt the same insert, only one succeeds. The other rolls back and its consumer can see the committed row on retry.
This is the same principle that makes distributed locking work without a lock manager. You’re not trying to coordinate access, you’re using the database’s transaction isolation to guarantee exactly one winner.
Tracking Messages at Kafka Scale
Kafka brings additional complexity. Kafka’s consumer offset management means you can’t simply delete a message after processing, you commit the offset instead. And if you commit the offset before your business transaction, you risk losing messages on crash.
The solution mirrors the SQS approach but with Kafka’s semantics: commit your business transaction and record the offset in your inbox table within the same database transaction, then commit the Kafka offset separately as an at-most-once operation. If the offset commit fails, you reprocess the message on the next poll, and your inbox deduplication prevents double effects.
Kafka itself handles producer-side deduplication via sequence numbers and Producer IDs at the broker level. The image below illustrates how the broker tracks sequence numbers per producer-partition to reject duplicates:
This protects you from duplicate writes to Kafka, but your consumer still needs the Inbox Pattern for end-to-end safety.
Handling Failure Modes That Actually Happen
Every system fails differently. The Inbox Pattern handles the most common ones cleanly, but you need a strategy for each.
Producer retries with new IDs: If a client generates a fresh UUID for every retry attempt, your inbox table sees them as distinct messages. You need the producer to use stable idempotency keys. The canonical approach is for the client to generate a UUID (v7 recommended for time-ordered indexing) and reuse it on retries.
Application crashes mid-transaction: The database handles this. If your process dies before the transaction commits, the inbox insert and business update both roll back. Your consumer will see the message again and reprocess it, correctly this time.
Idempotency key collisions: Two genuinely different messages with the same key. This is a producer bug, and your inbox table should actually help you detect it. Log every collision that results in a no-op and alert on unexpected volumes. It’s better to miss processing one message than to process something twice.
Separate Four Types of Duplicate, Because Not All Are Equal
“Duplicate” is too vague to be useful for debugging. When you’re investigating why something went wrong, ask which flavor you’re dealing with:
-
Producer retry duplicate: The same logical send is attempted again. FIFO queues suppress this within the deduplication window. Your inbox pattern handles it outside that window.
-
Independent producer duplicate: Two upstream workflows create the same business action with different event IDs. No queue can detect this. Your inbox table won’t catch it either unless you use business keys instead of message IDs.
-
Consumer redelivery: One queued message is received again after a failed or incomplete settlement. This is the primary use case for the Inbox Pattern.
-
Repeated downstream effect: Your consumer called an external API that executed twice because your request was ambiguous. The downstream service needs its own idempotency key.
Only the first and third are the Inbox Pattern’s job. The second requires different tooling (a deduplication layer that understands your business domain), and the fourth is someone else’s problem architecture entirely.
The Messaging Infrastructure That Still Needs Work
The Inbox Pattern isn’t plug-and-play. You need:
- A transactional database that supports upserts or conditional inserts within a transaction
- A periodic cleanup job to purge old inbox rows, otherwise that table grows forever
- Monitoring on inbox conflict rates, a sudden spike in deduplication hits means something upstream is retrying aggressively or your consumers are struggling
The overhead is real but manageable. You’re adding one database table and one cleanup job. Compared to the cost of explaining duplicate customer charges, that’s practically free.
When the Pattern Breaks
The Inbox Pattern assumes you’re using a database that supports ACID transactions. If you’re writing to a non-transactional data store, a distributed document database, or an external API, the pattern doesn’t directly apply. You’d need a distributed transaction coordinator, a two-phase commit, or an outbox pattern to propagate your deduplication markers alongside your business effects.
That’s not a limitation of the pattern, it’s a fundamental constraint of distributed systems. You cannot achieve idempotent effects across independent systems without coordination. Accept that constraint before you try to design around it.
Foundational idempotency patterns for message-driven systems cover the tradeoffs between approaches, including when the Inbox Pattern isn’t the right choice.
The Bottom Line on Consumer-Side Idempotency
Your message queue is not your contract. The Inbox Pattern is not elegant. It’s a clunky, transactional brute-force approach to a problem that shouldn’t exist but absolutely does.
It works because it forces a single truth: one message, one transaction, one decision. The database decides whether your message has been processed, and it does so with the same atomicity that protects your business data. No external coordination, no distributed locks, no hand-waving about eventual consistency.
When a new engineer asks why you’re inserting into a tracking table before doing any real work, explain it plainly: “This is how we guarantee that processing a message twice has the same effect as processing it once. It’s not fancy, but it’s correct.”
Sometimes correct beats fancy. Especially when there’s money on the line.
