You’ve seen it a hundred times. A message arrives out of order. The consumer checks a dependency, it’s not there, so it throws an error. The error handler pushes the message into a dead-letter queue, and some cron job replays it every thirty seconds until the dependency appears.
This pattern is everywhere. And it’s almost always wrong.
The most dangerous thing about retries is that they feel smart. When you write a retry loop, you feel like you’re building resilience. But what you’re actually doing is treating a fundamental property of distributed systems as a bug. That’s not engineering. That’s denial.
What We Actually Mean By "Eventual Consistency"
Distributed systems don’t promise that information arrives everywhere simultaneously, or in order. They promise that, eventually, it will. That’s the entire idea behind eventual consistency, and it’s not a failure mode.
Think about what happens when two services emit events for the same user. One fires a "user created" event. The other fires a "subscription created" event. A downstream service needs both before it can process a payment. The subscription event arrives first. The user data isn’t there yet. What happens next?
The conventional answer: treat it as an error, push it to a dead-letter queue, retry later.
But nothing actually failed. The system is behaving exactly according to its guarantees. The data will arrive. Not immediately, not in order, but eventually. Calling that an "error" is like complaining that your pizza delivery takes 30 minutes when the menu said 30 minutes.
The Dead-Letter Queue Industrial Complex
The "revolutionary" retry-based approach mostly revolutionizes your operational load. Here’s what typically happens:
- The subscription event lands first, gets dead-lettered.
- The user event arrives a few milliseconds later. The subscription event is still sitting in the DLQ.
- A replay job fires, pulls the subscription event back, processes it, now both events are handled.
- Nobody noticed the 500ms delay. But you’ve now built: a dead-letter queue, a replay scheduler, a manual retry console, and three Slack alerts that fire every time an event is "misordered."
The kicker? None of this infrastructure was necessary. The events were never lost. They were never corrupted. They simply arrived in an order you didn’t expect.
This problem compounds. Imagine your service goes down for a minute. Messages pile up. Some end up in the dead-letter queue. Newer messages flow through once it recovers. Now you need to replay those older messages in the correct order. Should they jump ahead of newer events? Should processing pause until they’re replayed? What if ordering actually matters?
You’ve created a secondary problem that’s harder to solve than the original one.
A Better Mental Model: Store Then Check
Once you recognize eventual consistency as a property rather than an error, the architecture transforms.
Instead of treating missing information as a failure that requires escalation, you accept it as a legitimate state. The implementation is almost boring:
- Store every incoming piece of data.
- Each time a new event arrives, check whether all required pieces are now present.
- If they are, execute the work.
- If they aren’t, do nothing.
No retries. No dead-letter queues. No humans manually replaying messages.
The system’s internal state naturally progresses as information becomes available. This is the inbox pattern done right: you persist everything first, then figure out what can be processed.
The CAP Theorem Won’t Save You From Yourself
The CAP theorem tells us that in the presence of a network partition, we choose between consistency and availability. But what it doesn’t tell us is how engineers abuse retries to pretend they’ve escaped this trade-off.
Here’s a confession: I’ve seen teams implement five-second retry loops with exponential backoff for cross-service data dependencies. The logic was: "If we retry enough, we’ll get strong consistency anyway." No. You won’t. You’ll get a system that spikes CPU during partitions and presents a false sense of reliability.
The CAP theorem forces a real choice. For many use cases, eventual consistency is the correct answer, but you need to design for it, not paper over it.
Real-world systems make this choice explicitly:
| System Type | Consistency Model | Why It Works |
|---|---|---|
| Banking | Strong (via consensus) | Money must not disappear |
| Social feeds | Eventual | Old posts are fine |
| Seat reservation | Eventual + atomic check | Race conditions must be prevented |
| Payment processing | Strong for state, eventual for notifications | Double charges are unacceptable |
The table is from a deeper guide on distributed systems, but the lesson is simple: pick your model, build for it, don’t try to cheat with retries.
When Retries Actually Make Sense
Lest anyone think this is an anti-retry manifesto: retries are essential. A transient network timeout? Retry it. A packet was dropped? Retry it. A database connection pool exhausted for 200ms? Retry it.
Distributed systems are full of short-lived failures, and retrying once is often enough to smooth over the inevitable hiccups of real infrastructure. The problem isn’t retries. It’s retrying the wrong thing.
Ask yourself one question before adding another retry layer:
If retrying once doesn’t solve the problem, why would retrying five times work? What changes between attempt two and attempt six?
If the answer is "I don’t know, I’m just hoping", you’re not engineering resilience. You’re gambling.
The Real Damage: Complexity You Didn’t Need
One of the most satisfying things about solving the right problem is watching unrelated problems disappear along with it.
Consider a service that emits webhooks. Delivery fails, retries pile up, and soon you’re managing a state machine for each webhook attempt. That complexity is self-inflicted. Webhooks aren’t simple, and the retry behavior you add around them often makes things worse.
Or consider an event-driven pipeline where message ordering matters. The moment you introduce retries for "missing dependencies", you break ordering guarantees. Now you need a sequencer, a deduplication layer, and a reconciliation job. That’s not architecture, it’s a Rube Goldberg machine.
The cleanest systems I’ve seen don’t retry for consistency. They store the events, wait patiently, and process when ready. That’s it. The complexity evaporates because they solved the right problem.
Idempotency: The Actual Solution
Here’s the uncomfortable truth: you need idempotency anyway. Even with perfect ordering and zero retries, your events will be duplicated. At-least-once delivery guarantees duplicates, and no amount of wishful thinking changes that.
The real answer to eventual consistency isn’t retries. It’s stateless processing combined with idempotent handlers. Store the event, check if you’ve already processed it, act only on new ones. When the retry comes (and it will), your handler simply no-ops.
This is why the inbox pattern works. You persist the event, track what’s been applied, and design for idempotency. Everything else is noise.
Solving The Problem You Actually Have
The bigger lesson isn’t really about retries at all. It’s about correctly identifying the problem before reaching for a solution.
Availability problems deserve one set of tools. Eventual consistency deserves another. Treating one as the other usually leads to extra infrastructure, extra operational burden, and systems that are harder to reason about.
The next time you catch yourself adding a retry loop for a "missing" piece of data, stop. Ask: Is the system actually failing? Or is it behaving exactly as designed?
Sometimes the cleanest solution isn’t finding a smarter retry strategy. It’s realizing that there was never anything to retry in the first place.
