Kafka Isn’t a Message Queue, Stop Treating It Like One

The “Queue” Misconception That Breaks Architectures

Here’s the core confusion: Kafka looks like a message queue. Messages go in, messages come out. But the semantics are fundamentally different.

Traditional message queues are built for task distribution. You’ve got work that needs doing, and the queue decides which worker handles it. A message is consumed, processed, and deleted. One job, one worker, done.

Kafka is built for event streaming. It answers the question “what happened?” and lets anyone who cares know about it. Messages aren’t deleted after consumption, they’re retained and replayed. Multiple consumers can read the same message independently. The broker doesn’t track who “should” process what, it just stores the log.

As the fundamentals guide explains, in traditional queues a message is deleted once processed, but Kafka retains it so you can replay events and rebuild system state. That’s not a minor implementation detail, it’s a completely different philosophical approach to data.

The architectural implications are significant:

• Point-to-point vs. publish-subscribe: Queues route one message to one consumer. Kafka broadcasts to any consumer group that subscribes.
• Ephemeral vs. durable: Queue messages vanish after acknowledgment. Kafka messages persist based on retention policy, not consumption state.
• Processing vs. observation: Queues coordinate work execution. Kafka enables event observation, state reconstruction, and audit trails.

If you’re using Kafka like a job queue, sending a task, waiting for one consumer to pick it up, expecting it to disappear, you’re working against the platform. You’ll hit confusing behavior around offsets, replay, and consumer groups that makes no sense under the queue mental model.

Partitions: The Most Underappreciated Design Decision

Partitions are where Kafka’s scalability lives, and where most architecture mistakes happen.

A partition is a subdivision of a topic, an immutable, append-only log with strict ordering guarantees. If message A is produced before message B in the same partition, consumers will read A before B. Period.

This ordering guarantee is the foundation of Kafka’s parallel processing model. The fundamentals walkthrough notes that if a topic has three partitions, three consumers can read from it in parallel. Need more throughput? Add partitions. But here’s the catch:

Ordering is only guaranteed within a partition, not across them.

This creates a fundamental tension. You can have global ordering or horizontal scalability, but not both. The standard solution is partition keying, ensuring all related messages (say, all events for a specific order ID) land in the same partition. Your ItemEventPublisher in a Spring Boot event-driven setup does exactly this by keying events with the item ID:

@Slf4j
@Component
@RequiredArgsConstructor
public class ItemEventPublisher {

    private final KafkaTemplate<String, ItemCreatedEvent> kafkaTemplate;

    public void publish(ItemCreatedEvent event) {
        var item = event.getItem();
        kafkaTemplate.send(ITEM_CREATED, item.getId(), event);
        log.info("Published ItemCreatedEvent {} for item id {} and name {}", event.getEventId(), item.getId(), item.getName());
    }

}

Key by item ID, and all events for one item stay ordered. Key poorly, and you’ll get out-of-order events that corrupt downstream state. This is the kind of subtle decision that looks trivial in a tutorial and causes production incidents later.

Consumer Groups: The Scaling Paradox

Consumer groups are Kafka’s mechanism for horizontal scaling, and they work in ways that surprise people.

Each partition is processed by exactly one consumer in a group. More consumers than partitions? Some sit idle, wasted resources. Fewer consumers than partitions? Some consumers handle multiple partitions, uneven load distribution.

This has a counterintuitive implication: adding more consumers doesn’t always increase throughput. If you have 3 partitions and 10 consumers, 7 of them do nothing. To actually scale consumption, you need more partitions, which means the partition count is a capacity-planning decision you make upfront, not something you can easily change later.

The rebalancing behavior compounds this. When consumers join or leave a group, Kafka automatically redistributes partitions across active members. That sounds great until a single consumer hiccups and triggers a full group rebalance, temporarily halting all consumption from that topic. In large consumer groups with many partitions, rebalances can take minutes, a painful production incident that event-driven architecture critics don’t often highlight in their “look how scalable this is” pitch.

Delivery Guarantees: Choosing Your Poison

Kafka’s delivery semantics are configurable, and each choice is a trade-off between safety and speed.

• At most once: Consumer commits offset before processing. If processing fails, the message is lost. Fast, but dangerous for anything that requires reliability.
• At least once: Consumer processes first, commits after. If a crash happens between processing and commit, the message is processed again. Duplicates are guaranteed at some point.
• Exactly once: Idempotent producers plus transactional APIs. No duplicates, no loss, but with meaningful performance costs.

Most production systems default to at-least-once, which means your consumers must be idempotent. The inbox pattern for transactional deduplication exists precisely because at-least-once delivery will eventually bite you. A network blip, a consumer crash, a retry, suddenly you’re processing the same event twice.

Yet many teams discover this only after they’ve shipped a consumer that debits accounts, sends notifications, or updates inventory without idempotency checks. The result is a debugging nightmare where you’re tracing duplicate side effects across services that all think they’re doing the right thing.

Kafka vs. Everything Else: An Honest Comparison

The messaging landscape isn’t Kafka-or-nothing. AWS’s comparison of Kafka and RabbitMQ highlights genuinely different use cases:

If you need to process a backlog of background jobs, RabbitMQ is often the better tool. If you’re building event pipelines that feed analytics, machine learning, or multiple downstream consumers, Kafka’s log-based model wins.

The event-driven architecture template demonstrates Kafka’s sweet spot: a producer publishes domain events, Kafka stores them durably, and consumers react independently. The ItemCreatedEvent flows through the system, allowing multiple services to respond without coupling to each other:

Client → Producer → Kafka → Consumer → Store

This is where Kafka shines, event distribution at scale, not task coordination.

When Event-Driven Architecture Becomes Overkill

Here’s the uncomfortable truth the memes won’t tell you: most systems don’t need Kafka.

The critical evaluation of event-driven architecture traps makes the case bluntly. When your traffic barely taxes a single Postgres instance, running Kafka introduces staggering complexity for zero benefit. You’re managing brokers, consumer groups, offsets, retention policies, and schema evolution, all to move messages a simple database trigger could handle.

The broader context on event-driven architecture’s rise captures the pattern: a decade ago everything was microservices, today everything is events. Kafka clusters sprout like mushrooms after rain, applied to workflows that could have been handled with a synchronous API call and a cron job.

The pragmatic test: do you need independent scaling of producers and consumers? Multiple consumers reacting to the same event? Event replay for state reconstruction? If yes, Kafka earns its complexity. If you just need to decouple services with asynchronous communication, a simpler message broker might serve you better. Comparing webhook limitations to robust message brokers shows there’s a spectrum of options between “raw HTTP calls” and “full event streaming platform.”

The Architecture Shifts Nobody Warns You About

Adopting Kafka forces architectural changes beyond the messaging layer.

Read models change. In the Spring Boot template, writes go through events, but reads come from ItemReadController querying an in-memory store, a projection built from consumed events. This CQRS-style pattern means your read model is eventually consistent, which is fine until someone expects immediate visibility of their writes.

Observability gets harder. Tracing an event’s journey through multiple consumers requires distributed tracing tools. The template uses Spring Boot Actuator for health checks, but correlating events across producer, broker, and consumer requires careful event ID propagation. As the monorepo and CQRS deep dive notes, Kafka enables distributed system benefits but also mandates distributed system discipline.

Schema management becomes critical. Events are your contract. The template uses a shared ItemCreatedEvent model across producer and consumer, which works when you control both sides. In larger organizations with independent teams, you need schema registries and versioning strategies. Your events are forever. Once consumers exist, changing an event schema breaks them.

Failure modes change. A Kafka outage doesn’t just block traffic, it breaks the contract between services. Producers can buffer events, but consumers fall behind. When the cluster recovers, downstream systems face a backlog of events they must process, potentially hours of lag representing inconsistent states across services.

KRaft and the Death of ZooKeeper

One operational change deserves attention: Kafka is finally shedding ZooKeeper.

For years, Kafka’s reliance on ZooKeeper for cluster coordination was a constant source of operational pain. Managing topic metadata, broker health, and leader elections required running a ZooKeeper ensemble alongside Kafka, two distributed systems to operate instead of one.

KRaft mode eliminates ZooKeeper entirely. A controller quorum uses the Raft consensus algorithm to manage cluster state internally. Broker registration, topic metadata, and leader election all happen within Kafka itself.

This is the kind of change that matters operationally. One fewer system to monitor, one fewer failure domain, one layer of complexity removed. If you’re starting fresh or planning a major Kafka upgrade, KRaft is worth the migration effort.

What Actually Matters

After all the theory, here’s what separates successful Kafka implementations from disasters:

1. Partition strategy is an architecture decision, not a configuration detail. Key by business entity, plan for growth, and accept the ordering trade-offs.
2. Idempotency is non-negotiable. At-least-once delivery means duplicates will happen. Design your consumers accordingly.
3. Schema evolution needs governance. Events are long-lived contracts. Version them, test compatibility, and plan for migration.
4. Monitoring is a feature, not an afterthought. Track consumer lag, partition health, and replica status like your business depends on it, because it does.
5. Know what Kafka isn’t for. Task distribution, request-reply patterns, and simple job processing have better tools.

The practical implementation approach in the Spring Boot template demonstrates the right pattern: producers and consumers share event models, business logic stays decoupled from messaging infrastructure, and testing uses real Kafka instances via Testcontainers rather than mocks.

The template’s test setup, integration tests with actual Kafka, Allure reports for visibility, and coverage aggregation, is arguably more valuable than the messaging code itself. It forces you to confront how your system behaves with real infrastructure, not just how it looks in isolation.

The Bottom Line

Kafka is powerful because it’s opinionated. Its log-based model enables event replay, independent scaling, and durable state reconstruction in ways that traditional queues can’t match. But those capabilities come with costs: operational complexity, ordering constraints, and a fundamentally different way of thinking about data flow.

Teams that succeed treat Kafka as a distributed commit log with event-streaming semantics. Teams that fail treat it as a big queue and wonder why everything breaks.

The choice isn’t Kafka versus no messaging, it’s understanding what problem you’re actually solving. Build event pipelines for modern data architectures, not because someone on Twitter said “events are the future.” Scale your data infrastructure because your workloads demand it, not because a Telegram-scale architecture lookalike impressed you.

And when you do adopt Kafka, embrace its semantics rather than fighting them. The platform has opinions. They’re usually right.