Your Transactional Outbox Won’t Save You: The Ordering Problem Nobody Talks About

The transactional outbox pattern sounds deceptively simple after a ten-minute explanation. Write your business entity and an outbox event in the same PostgreSQL transaction. Run a background publisher that sends pending events to Kafka. Mark each event as published after the broker acknowledges it.

No distributed transactions. No dual-write problem. If the application crashes, the publisher resumes from the durable outbox.

What the tutorials don’t tell you is that this pattern only solves the easy part. The moment two transactions touch the same aggregate, or a publisher crashes at the wrong moment, or you need consumers to observe events in business order, your tidy outbox design starts leaking correctness guarantees like a sieve.

Amazon’s 90% cost cut after ditching microservices should be your wake-up call that distributed systems carry hidden costs. The outbox pattern is no exception.

What the Outbox Actually Guarantees (and What It Doesn’t)

Let’s be precise about what the outbox pattern provides. Consider a payment service that updates a payment and publishes an event. The unsafe implementation looks familiar:

@Transactional  
public void settlePayment(UUID paymentId) {  
    Payment payment = repository.findById(paymentId)  
        .orElseThrow(),  

    payment.setStatus(SETTLED),  
    kafkaTemplate.send(  
        "payment-events",  
        paymentId.toString(),  
        new PaymentSettled(paymentId)  
    ),  
}

The database transaction and Kafka publication don’t share an atomic commit boundary. The Kafka record can become visible while the database transaction rolls back. Or the application crashes between commit and publish.

The outbox pattern fixes this by moving the event intent into the database transaction:

BEGIN;

UPDATE payments  
SET status = 'SETTLED',  
    version = version + 1  
WHERE id = :payment_id  
  AND status = 'AUTHORIZED';

INSERT INTO outbox_events (  
    event_id,  
    aggregate_id,  
    event_type,  
    payload  
)  
VALUES (  
    :event_id,  
    :payment_id,  
    'PaymentSettled',  
    :payload  
);

COMMIT;

Note the conditional update, you must verify the payment update affected exactly one row before inserting the event. A rejected state transition must not produce a settlement event.

This is the guarantee the outbox actually provides: a durable event record exists if and only if the associated business transaction commits.

It does not guarantee that Kafka receives the event exactly once. It does not guarantee consumers process it exactly once. And critically, it does not guarantee that multiple events for the same payment arrive in business order.

That last one is where production systems fall apart.

The First Ordering Bug Appears Before Kafka

Imagine two application instances modifying the same payment. Transaction T1 changes its status from AUTHORIZED to SETTLED. Transaction T2 subsequently changes it from SETTLED to REFUNDED. Your consumers should observe:

PaymentSettled  
PaymentRefunded

Now suppose each transaction inserts an outbox record with a database-generated sequence ID. The natural assumption is that ordering by that ID reflects commit order.

That assumption is dead wrong.

PostgreSQL sequences aren’t transactional. Values get allocated before a transaction commits, and rolled-back transactions don’t restore consumed sequence values. Transaction T1 can obtain outbox ID 100, remain open, and commit after T2 has already obtained ID 101 and committed.

A polling publisher observes ID 101 first. Later, ID 100 becomes visible. Sorting by sequence ID doesn’t reconstruct commit order.

Timestamps have the same problem. A timestamp records when it was generated, not when the transaction became durable and visible.

The deeper issue? Event ordering must follow the ordering of authoritative business state transitions, not an incidental property of outbox record creation.

Aggregate Versions: A More Honest Contract

For events representing changes to a single aggregate, make the aggregate’s state version explicit. Add a version column to the payments table:

CREATE TABLE payments (  
    id UUID PRIMARY KEY,  
    status TEXT NOT NULL,  
    version BIGINT NOT NULL  
);

A successful transition increments the version and writes the corresponding event in the same transaction. The event carries:

{  
  "eventId": "e4b7...",  
  "aggregateId": "8c21...",  
  "aggregateVersion": 42,  
  "eventType": "PaymentSettled"  
}

The next valid transition produces version 43. This gives downstream systems a meaningful ordering relationship. A consumer receiving version 43 before version 42 recognizes a gap. Receiving 42 twice recognizes a duplicate.

But the version must be assigned through a concurrency-safe transition. Two writers can’t independently read version 41 and both publish version 42. You need conditional updates, row-level locking, or optimistic concurrency control to serialize changes to that aggregate.

An aggregate version represents business-state progression. It is not a globally ordered event number, and it doesn’t solve ordering across independent aggregates.

That’s usually exactly what your architecture actually needs.

Why SKIP LOCKED Destroys Your Ordering

A common outbox publisher pattern uses PostgreSQL row locking so multiple workers claim different records:

SELECT id, aggregate_id, payload  
FROM outbox_events  
WHERE published_at IS NULL  
ORDER BY id  
LIMIT 100  
FOR UPDATE SKIP LOCKED;

This improves throughput by preventing workers from waiting on locked records. But SKIP LOCKED explicitly permits a worker to bypass locked rows.

Worker A claims event 100 for payment P. Worker B skips that locked row and claims event 101, also for payment P. Worker B publishes event 101 first. Worker A hits a Kafka timeout and retries later.

Your consumer observes version 43 before version 42. The publisher delivered both records, but it didn’t preserve aggregate order.

This is a core tension: scaling out your outbox publisher directly undermines ordering guarantees. A production outbox design needs an explicit answer to whether order matters and which component enforces it.

Kafka Partitioning: Necessary but Not Sufficient

Kafka preserves record order within a partition. Using the aggregate identifier as the Kafka record key seems like the natural solution:

kafkaTemplate.send(  
    "payment-events",  
    paymentId.toString(),  
    event  
);

With stable partition configuration, events for the same payment route to the same partition. Order preserved, right?

Not so fast. Kafka preserves the order in which records are appended to the partition. It cannot reconstruct an order already violated by competing outbox publishers. If version 43 gets appended before version 42, consumers observe them in that order.

Partitioning solves a transport-ordering problem. It doesn’t solve source-ordering or publisher-coordination problems.

Your options are limited:

1. Single ordered publishing path per aggregate, kills throughput but preserves order
2. Per-aggregate publication sequencing using durable state and explicit claims
3. Consumer-side version continuity enforcement with bounded buffering and gap recovery

Each has different throughput and operational costs. The key insight: Kafka partition keys cannot repair ordering guarantees the producer never established.

This is exactly the kind of latency optimization trap where engineers micro-optimize the transport layer while the actual bottleneck, publisher coordination, goes unaddressed.

WAL-Based CDC: A Different Ordering Model

Polling isn’t the only way to publish an outbox. Debezium can read PostgreSQL’s logical replication stream and emit changes from an outbox table to Kafka, using the write-ahead log as the source of committed database changes.

The architecture looks like:

Spring Boot  
    |  
    | One PostgreSQL transaction  
    v  
Payment state + Outbox event  
    |  
    v  
PostgreSQL WAL  
    |  
    v  
Debezium connector  
    |  
    v  
Kafka topic  
    |  
    v  
Consumer

CDC has access to the database’s change stream rather than a snapshot of currently visible rows. A single connector processing the change stream can observe committed changes in the order provided by logical decoding. This makes commit-order preservation easier.

But the guarantees need careful statement. Downstream ordering still depends on routing, partitioning, and connector configuration. If events for one aggregate route to different Kafka partitions, the consumer loses the single ordered stream. Multiple independent pipelines publishing the same logical events complicate ordering and duplication further.

And CDC introduces a new operational hazard: WAL retention.

The WAL Retention Trap

PostgreSQL logical replication slots retain WAL needed by consumers to continue reading. If a Debezium connector stops advancing its replication position, PostgreSQL retains WAL that would otherwise be removable.

During a prolonged connector outage, retained WAL consumes significant disk space. Your platform experiences a database storage incident because an event-publishing component stopped progressing. Monitoring Kafka consumer lag won’t catch this.

You need visibility into replication-slot lag, retained WAL, connector health, and disk capacity. PostgreSQL provides max_slot_wal_keep_size, but setting a retention limit creates its own trade-off: if required WAL gets removed, the connector can’t resume and needs recovery or reinitialization.

Define what happens when the CDC pipeline falls behind beyond its recoverable window before it happens. Otherwise, a mechanism introduced to improve reliability becomes a source of database unavailability. The operational overhead of microservices communication at scale is exactly why FinTech’s compliance challenges get exacerbated by event-driven architectures.

Duplicates Are Inevitable. Embrace Idempotency.

Consider a polling publisher processing event E. It reads the outbox record, sends E to Kafka, receives acknowledgment. Before marking the record as published in PostgreSQL, the process crashes.

After restart, the outbox record still appears pending. The publisher sends E again. Kafka now has two copies of the same logical event.

This is the normal consequence of coordinating two independently committed systems without a shared atomic transaction. Kafka’s idempotent producer can prevent certain duplicate records from producer retries within its session semantics, but it cannot deduplicate a logical business event resent by a restarted publisher as a new publication attempt.

The consumer must be prepared for duplicates. For financial workflows, never assume broker configuration alone establishes business-level exactly-once execution.

Consumer-side deduplication requires atomicity with the business effect:

BEGIN;

INSERT INTO processed_events (  
    consumer_name,  
    event_id  
)  
VALUES (  
    :consumer_name,  
    :event_id  
)  
ON CONFLICT DO NOTHING;

-- Apply the business effect only if the insert succeeded.

COMMIT;

The affected-row count determines whether the event is new. This provides a useful local idempotency boundary when the business effect and deduplication record share the same database transaction.

But it doesn’t protect an external API call. If the consumer charges a payment provider and then commits its processed-event record, a crash between those operations can still cause a duplicate charge. External effects need their own stable operation identifiers, idempotency guarantees, and reconciliation procedures.

Idempotency must be enforced at the boundary where the effect becomes durable.

Ordering and Idempotency Are Different Problems

Here’s the trap: you can build a perfectly idempotent consumer that still produces incorrect state.

A consumer maintaining a payment status projection receives:

Version 43: REFUNDED  
Version 42: SETTLED

Both events have unique IDs. The consumer deduplicates successfully, applies both in arrival order. The final projection says SETTLED, even though the authoritative payment is REFUNDED.

No duplicate was processed. The problem was stale-state application.

For projections requiring monotonically increasing versions, guard the update:

UPDATE payment_projection  
SET status = :status,  
    version = :incoming_version  
WHERE payment_id = :payment_id  
  AND version < :incoming_version;

This prevents older events from overwriting newer projections. But it doesn’t guarantee every intermediate transition gets processed. That matters when events represent actions rather than complete state snapshots.

A projection that only needs the latest status can tolerate gaps. A ledger consumer that must record every financial movement cannot simply discard an older event because a newer version arrived first. Gaps must be detected and resolved before applying dependent operations.

This is why generic advice like “just ignore events with older versions” is dangerous. Sometimes it’s correct. Sometimes it silently loses essential business operations.

Schema Evolution Is Part of the Delivery Contract

An outbox event is a published interface. Once multiple consumers depend on it, changing its structure becomes an API compatibility problem.

Suppose the payment service originally publishes:

{  
  "eventType": "PaymentSettled",  
  "amount": 10000  
}

A later release changes the amount representation or introduces currency-specific behavior. The producer deployment finishes in minutes, but consumers run older versions, recover historical records, or replay events created months ago.

A schema change that works for current events can break replay.

Schema registries with Avro or Protobuf enforce structural compatibility, but they can’t prove semantic compatibility. Changing a field’s meaning while preserving its type still breaks consumers. An amount field that represented minor currency units cannot silently begin representing major units.

That’s a business contract change. The outbox makes events durable, and durability increases the importance of long-term schema compatibility because old records remain operationally relevant.

The Staff Engineer Question

There is no universal outbox architecture. A single polling publisher can suffice for low-volume services with modest ordering requirements. Multiple polling workers provide better throughput when events are independent and reordering is acceptable. CDC makes sense when commit-order observation matters or you already run change-data-capture infrastructure.

Each choice adds operational cost. The design review should establish which guarantees the business requires before selecting the implementation:

• Does the consumer need every transition or only the latest state?
• Is ordering required globally, per aggregate, or not at all?
• How long can publication be delayed?
• Can an event be delivered twice?
• What happens if the publisher is unavailable for twelve hours?

These questions expose a common architectural mistake: spending substantial engineering effort guaranteeing an ordering property that no consumer actually needs. Correctness is essential, but unnecessary coordination reduces throughput and increases complexity.

The Outbox Is a Boundary, Not an Architecture

The transactional outbox pattern connects a committed database change with a durable intent to publish an event. That’s a narrow and powerful guarantee. It doesn’t decide event ordering, consumer idempotency, external side-effect semantics, schema compatibility, or recovery after prolonged infrastructure failures.

PostgreSQL can atomically commit business state and outbox records. A CDC connector can transport committed changes from the WAL. Kafka can provide durable, ordered records within partitions. A consumer can enforce local idempotency and application-specific ordering constraints.

None of those components, by itself, proves the entire business process is correct. That proof comes from defining the invariant, tracing every failure boundary, and making each recovery path preserve the intended outcome.

The outbox pattern is easy to draw. The engineering work begins when the first publisher crashes, or when your database fills up because a connector stopped advancing, or when a consumer applies version 43 before version 42 and your payment projections silently diverge from reality.

The patterns we’ve covered here, event-driven architecture, modular monolith communication, and the lessons from companies reversing microservices decisions, all share the same lesson: distributed coordination costs more than the diagrams suggest.

For a Staff Engineer, the skill isn’t knowing the outbox pattern. It’s identifying exactly which guarantees each component provides, where the remaining gaps close, and which of those gaps the business actually needs closed.