The Core Misunderstanding: OSI Layer Blindness
The fundamental difference between a load balancer and an API gateway lives in the OSI model, and most teams ignore it until it bites them.
A load balancer operates at Layer 4 (Transport) or sometimes Layer 7 (Application). It looks at packets, IP addresses, TCP ports, and at best, basic HTTP headers. Its entire job is to answer one question: Which healthy backend server should handle this connection?
An API gateway lives exclusively at Layer 7. It reads the full HTTP request, headers, body, query parameters, cookies, and makes routing decisions based on what the request actually says, not just where it’s going.
This isn’t a minor technical distinction. It’s the difference between “send this to any instance of service X” and “send this authenticated user’s checkout request to the payment service, but only if their rate limit hasn’t been exceeded, and cache the product catalog responses.”
What a Load Balancer Actually Does (And Doesn’t)
Load balancers solve a specific, well-defined problem: distribution. When you have three instances of your user service running, a load balancer ensures traffic spreads across them evenly using algorithms like round-robin, least connections, or IP hash.
What it handles:
– TCP/UDP traffic distribution
– Health checks (is the instance responding?)
– SSL termination (offloading encryption)
– Simple session persistence (sticky sessions)
What it absolutely cannot do:
– Route based on request content (e.g., “send JSON requests to v2, XML to v1”)
– Authenticate users or validate tokens
– Rate-limit per API key or client
– Transform request/response formats
– Aggregate responses from multiple services
If you’re asking your load balancer to handle API keys or parse JSON bodies, you’re fighting the tool. It will work, poorly, with custom Lua scripts and hacky configurations, until it breaks at 2 AM during a traffic spike.
What an API Gateway Actually Brings
An API gateway is a full-featured application-layer proxy designed for the chaos of microservices. Think of it as the bouncer, concierge, and translator rolled into one.
From the research, the key capabilities map to:
| Capability | What It Does | Why It Matters |
|---|---|---|
| Request routing | Routes to specific services based on URL paths, headers, or query params | Single endpoint for clients, regardless of backend complexity |
| Authentication & authorization | Validates tokens, API keys, OAuth flows | Keeps auth logic out of each microservice |
| Rate limiting | Throttles requests per client, IP, or endpoint | Prevents abuse and ensures fair usage |
| Caching | Stores responses to reduce backend load | 10x faster responses for repeated queries |
| Protocol translation | Converts between HTTP, WebSocket, gRPC, MQTT | Clients speak one language, backends speak another |
| Response transformation | Aggregates or modifies responses | One client request can fetch from 5 services |
This identity crisis between Kubernetes routing and full API management is exactly where most teams get stuck. They assume that because their Ingress controller or cloud load balancer can route HTTP traffic, it’s functionally equivalent to a gateway. It’s not.
The Real Architecture: You Almost Always Need Both
Here’s where the “vs” in the title becomes misleading. These aren’t competing technologies. They operate at different layers of your stack, and mature architectures use both.
Client → Load Balancer → API Gateway → Microservices
The load balancer sits in front of the API gateway (or multiple gateway instances). It distributes raw TCP/HTTP traffic across healthy gateway nodes. The gateway then handles the application-level routing, security, and transformation.
If you skip the load balancer, your gateway becomes a single point of failure. If you skip the gateway, your load balancer is forced into application-layer decisions it wasn’t designed for.
ByteByteGo’s architecture diagram shows this layered approach clearly:
Three Common Anti-Patterns (And How to Fix Them)
Anti-Pattern 1: The “API Gateway” That’s Just a Reverse Proxy
The symptom: Your “gateway” only routes based on URL paths. No authentication. No rate limiting. No caching. You’re using nginx or a cloud ALB and calling it an API gateway.
The fix: You don’t necessarily need a full gateway. If your services handle their own auth and you don’t need cross-cutting concerns, a load balancer is fine. But if you’ve told your team you have an API gateway, you’re setting expectations that won’t be met. Either upgrade to a proper gateway like Kong, Apigee, or AWS API Gateway, or stop calling it one.
Anti-Pattern 2: The “Load Balancer” Doing Application Logic
The symptom: Your load balancer config file is 500 lines long. It contains custom Lua scripts for JWT validation, regex patterns for content-based routing, and logic to transform response bodies.
The fix: You’re building a custom API gateway inside a load balancer. Products like Envoy blur this line intentionally, it’s a “universal data plane” designed for both Layer 4 and Layer 7. If you’re using Envoy, fine. If you’re twisting an AWS NLB into doing application routing, stop immediately. Use the right tool.
Anti-Pattern 3: Security Theater
The symptom: You have an API gateway, but every microservice also validates the same authentication token independently. You’re paying for rate limiting in the gateway while also implementing it in each service.
The fix: If your gateway handles auth, let it handle auth. Services should trust the gateway’s validated identity, not re-validate. This is why many teams find event-driven architectures quietly bypassing traditional API gateways like Apigee, the overhead of double-processing security becomes untenable at scale.
When the Lines Blur (And When That’s Okay)
A comment on the original research post from a developer using Envoy made a valid point: there are plenty of products that can do both at once. Envoy, Traefik, and modern service meshes operate as combined data planes that handle both load balancing and gateway-style routing.
This works because they’re designed from the ground up as Layer 7 proxies. They’re not load balancers with gateway features bolted on, they’re universal proxies that can serve either role.
The danger isn’t using a unified platform. It’s using one without understanding which layer you’re operating at. If your unified proxy routes based on TCP port (L4), don’t expect it to parse gRPC headers. If it handles HTTP routing (L7), don’t expect it to forward raw UDP packets.
The Decision Framework
Ask yourself three questions:
- Do your routing decisions depend on request content? (URL path, headers, body, authentication state?) → You need a gateway.
- Do you need to distribute traffic across multiple instances of the same service for availability? → You need a load balancer.
- Do you need both? → Yes, almost always. Put the load balancer in front of the gateway.
The API gateway vs. load balancer question isn’t an either/or. It’s a question of scope and responsibility.
Load balancers are infrastructure. They handle scale, availability, and raw traffic distribution. API gateways are application components. They handle routing, security, and API management. Confusing the two leads to internal API governance failures that undermine your architecture, systems that are simultaneously over-engineered and under-designed.
If you’re building a microservices architecture, start with the assumption that you need both. The only real question is what level of gateway features you need and how much routing intelligence you can delegate to your load balancer layer.
And for the love of production stability, stop asking your load balancer to read JSON bodies. It has feelings too.
