Your Circuit Breaker’s Coordinator Just Died. Now What?

The Problem With Shared State

Before we get to the coordinator failure itself, let’s set the stage. Why does a distributed circuit breaker need a coordinator in the first place?

The simple version of the pattern is embarrassingly easy. Keep a sliding window of recent outcomes in memory, count failures, and once the rate crosses a threshold, stop calling the dependency. One process, one window, no network involved. Done.

The moment you scale to twenty replicas, that simplicity evaporates. As the deep dive on implementing distributed circuit breakers points out, each replica keeps its own window when the state is local. Each one needs minimumThroughput observations of its own before it can react. If traffic spreads thinly enough, some replicas never accumulate enough observations to trip at all.

The dependency absorbs roughly twenty times the failures before the first breaker trips. And even then, you’ve got replicas disagreeing about system state, some rejecting calls, others happily sending them through.

Moving the window into shared storage fixes that. It also makes the window itself a dependency with its own failure modes. The storage you just made load-bearing can itself go away. The coordinator becomes a single point of failure in a system designed to prevent them.

This is the classic risk of centralized orchestration in distributed systems problem, just smaller in scale. You don’t need a full integration middleware bus to create coordination fragility, a Redis key is enough.

The Only Clear Answer: Scopes You Know Are Non-Closed

Let’s start with the case where the right decision is actually obvious.

Your replica has previously seen this scope go open or half-open. The breaker was protecting something, a struggling payment service, a flaky database, whatever. Nothing has told you it recovered. The only reason you can’t confirm that is the coordinator being down.

Do you let calls through?

Absolutely not. The breaker was doing its job when Redis died. Letting traffic through because the bookkeeping is unavailable removes the protection at the exact moment it’s needed most. The dependency is still struggling, nothing has changed that, you’ve just lost the ability to confirm it.

So a scope known to be non-closed rejects. Always. Regardless of configuration.

This isn’t a hard call, but it does point at a design requirement: your breaker needs to remember which scopes it has seen in a non-closed state, even when the coordinator is unreachable. That memory is local, so it costs something:

const keyFor = (operation: string, scope: string) =>
  JSON.stringify([operation, scope])
    

The implementation keeps only open and half-open in that map, because “closed” and “never heard of it” lead to the same decision. Storing closed entries would grow the map with every scope the process ever sees, which is a slow leak.

The caveat is that entries only leave when that same scope is later seen closed, which requires traffic. A scope that opens once and then goes quiet stays in the map for the life of the process, holding a belief that may be long out of date. That’s a feature disguised as a bug: a stale “open” belief is safer than a fresh “closed” assumption when you can’t verify either.

The Actually Ambiguous Case: Fail Open or Fail Closed?

Here’s where the architectural controversy kicks in.

What about a scope last seen closed? Or one the replica has never seen at all? You have no evidence the dependency is in trouble. Blocking all traffic because a coordinator is unreachable turns one outage into two.

That’s the argument for fail-open: let traffic through unprotected. The dependency probably isn’t broken, and refusing checkouts during a Redis blip is a product decision, not an implementation detail.

The counter-argument is that you’re now sending traffic to a dependency with zero protection. If it was on the edge of failing, this is the moment it tips over. You’ve turned a coordinator outage into a dependency outage.

The honest answer from the caracal implementation is that this is a configuration option with a documented default, not a decision made silently:

onCoordinatorError: "fail-open",  // or "fail-closed"
    

The reasoning is sound: what the right choice is depends on what the call does, not on anything the library can figure out on its own. Refusing all checkouts during a Redis blip is a product decision, it needs to be made by someone who understands the business cost of both failure modes.

One useful heuristic from the broader resilience toolkit: think about what a single failed call costs versus what a coordinated failure costs. If one checkout failing is annoying but survivable, fail-open. If a burst of uncoordinated traffic hitting a struggling dependency could cascade, fail-closed.

The Trap: “Just Fall Back to a Local Breaker”

The most tempting option when your coordinator goes down is also the most dangerous one.

“Let’s just run a local breaker while Redis is down! It looks like graceful degradation! The process keeps working!”

The implementation article is blunt about why this fails: what you’re actually doing is rebuilding the per-replica window that the shared one existed to replace, at the moment the system is already under stress, while continuing to report itself as a distributed breaker.

A breaker that silently changes what it protects is worse than one that tells you it cannot decide.

Think through what happens with a local fallback breaker and twenty replicas. Each one starts building its own window from scratch. Each one needs minimumThroughput observations before it can react. The exact problem you solved by moving to shared storage comes back, just at the worst possible moment.

Your dependency now absorbs twenty times the failures before anything trips. The “graceful degradation” has quietly recreated the original bug.

What About Recording Outcomes?

The coordinator being down when you’re deciding about a call is one problem. The coordinator being down when a result comes back is a different one entirely.

Suppose a call gets admitted (either because the breaker decided fail-open, or because it was already in flight). The call completes successfully. The dependency is fine. But Redis is still down, so you can’t record the outcome.

What do you do with that result?

Drop it. There is nothing useful to do with it and no reason to fail a call that has already completed. One missing datapoint moves a window of a hundred by one percent. Slightly stale, but not wrong in any meaningful way.

The same logic applies to a probe settlement that can’t be delivered. The claim expires on its own, that’s what the deadline mechanism is for, and the slot returns to the pool without anyone needing to be told.

This is the difference between a system that degrades gracefully and one that compounds failures. Dropping an observation costs you a tiny bit of accuracy. Retrying a failed recording could spin up a retry storm on a coordinator that’s already struggling.

The Deeper Lesson: Epochs, Leases, and Design for Disappearance

Coordinator failure isn’t just about the moment of unavailability. It’s about what happens after Redis comes back and the breaker has to reconcile its state.

The generation-to-epoch pattern is designed for exactly this. Each observation carries the generation it belongs to, so counting only looks at members carrying the current one. A transition costs nothing beyond incrementing a number, and superseded observations age out on their own.

But there’s a nasty failure mode hiding in this arrangement. The state hash carries the current generation, and the observations are just members in another key. If the hash disappears while the observations survive, which an eviction policy or an administrative delete will happily do, then a breaker that restarts its generation from zero adopts every orphaned member whose label happens to be zero.

The window it believes is current is actually a window from the past.

The fix is to restart from a value nothing can be carrying. When the script finds no state hash but does find observations, it derives a fresh epoch from the incoming observation’s own identifier:

if redis.call('EXISTS', KEYS[2]) == 1 then
  gen = tonumber(string.sub(redis.sha1hex(uuid), 1, 11), 16)
  if not gen or gen == 0 then gen = 1 end
else
  gen = 0
end
    

Forty-four bits is at most fourteen decimal digits, which stock Lua prints in full. A wider value can come out in scientific notation, and since the epoch roundtrips through the hash as a decimal string, a generation written as 1e+15 stops matching the members it labels.

There’s also the expiry problem. Since a missing state record reads as closed, letting an open breaker expire silently admits all traffic and loses the epoch that labels the current window. A breaker that is not closed must not be allowed to expire at all. Only the closed state can carry an expiry, and even that is bounded below by the window retention period, because the record that says which epoch is current has to outlive the observations carrying that label.

The same principle applies to probe slot leases. A deadline that’s too short causes live probes to lose their slots while still running. A deadline that’s too long blocks recovery after a replica crash. Both ends of that range move when the wrapped call’s timeout changes.

The Coordination Pattern Is Everywhere

This isn’t just about circuit breakers. The underlying problem, a coordinator that itself becomes a failure point, shows up across distributed systems. Workflow coordination without centralized systems is a direct response to this fragility. Distributed transaction patterns dance around the same coordinator issue, with the same “which failure do you prefer” tradeoffs.

The pattern is always the same: shared state gives you coordination, and coordination gives you a new failure point. The design question is never “how do I make the coordinator reliable.” It’s “what happens when it isn’t.”

The choices that matter are made at the edges:

• Memory: remember which scopes were non-closed, so you know when you’re making that unambiguous decision.
• Configuration: make the ambiguous case explicit, with a documented default that matches your business priorities.
• Dropping data: an unrecorded observation is noise, not corruption. Don’t fail completed calls because you can’t persist their outcome.
• No silent mode changes: a local fallback breaker that reports itself as distributed is a lie. The system can’t defend against a failure it doesn’t know exists.

The rule at the center is still the one from the top: count the recent outcomes, and if too many failed, stop calling for a while. But in distributed systems, the simple rule is now surrounded by complex logic and a dozen ways to get it wrong.

None of these are exotic problems. They’re the same problems that appear whenever shared state replaces local state, just concentrated in a smaller system where you can see all of them at once.

Testing the Unthinkable

One thing the original article gets right is that coordinator failure handling is not something you can verify by reading code carefully and hoping.

When the decisions live in Lua, they don’t run in the language the rest of the code is written in, can’t be stepped through in a debugger, and are exercised only when a real coordinator is present. Reliable message processing during failures has the same problem: the interesting behavior only manifests under conditions that are expensive to reproduce.

The technique that works is keeping a second implementation of the same contract in memory, then running both against the same generated sequences of operations and comparing the results field by field. Any disagreement is a bug in one of them, and it doesn’t matter which, because the two were written from the same specification by different means.

The trap with generated sequences is a generator that only produces boring ones. A run where the breaker never opens will pass happily and prove very little. The suite needs counters for each interesting path reached: opened, admitted, rejected at the probe limit, settled, transitioned, stale. And it needs to fail if any of those counters stayed at zero.

The test verifies the behavior. The counters verify the test.

The Bottom Line

Your coordinator will fail. It’s not a question of if, but when, and whether you’ve made the right choices before it happens.

The scopes you know are non-closed reject traffic, no matter what. The scopes you’re unsure about follow a configured policy that reflects business priorities. Outcomes you can’t record get dropped. And nothing, nothing, silently changes what the breaker protects.

The price of distributed resilience is accepting that some failures will be handled imperfectly. The trick is deciding which imperfection you’ll accept when the coordinator goes down.