Your K8s Load Balancer Is Silently Burning GPU Cycles

Imagine you’re serving a 70B parameter model. In BF16, those weights alone need around 140 GB of GPU memory. You’re running four 80 GB GPUs with tensor parallelism, which means those four GPUs form exactly one model replica.

Now scale to two replicas.

Replica A has processed a customer’s 7,000-token prefix, say, a system prompt with tool definitions and conversation history from an agent loop. The KV cache for that prefix is sitting there, warm and ready. Replica B has never seen those tokens.

With ordinary round-robin routing, the next request in that conversation may land on B. B doesn’t have the cached KV state, so it processes all 7,000 tokens from scratch. The GPU does the same prefill computation that A already completed. Valuable HBM bandwidth wasted. Time-to-first-token (TTFT) spikes. And the cost multiplies with every turn of the conversation.

This isn’t hypothetical. It’s the default behavior of every standard Kubernetes Service in production today.

When this problem comes up, someone inevitably suggests sticky sessions. And sure, they’d help, if the goal was to pin each user to one replica forever.

But sticky sessions are a sledgehammer where you need a scalpel. They don’t know anything about what’s cached, they just know where the user was last. Consider these scenarios:

• Replica A has a massive queue while B sits idle. Sticky routing sends the request to A anyway, and the user waits.
• A different request shares a 5,000-token system prompt with a session on B, but the user’s individual session was on A. Sticky sessions miss the overlap entirely.
• A pod restarts, its cache is cold, and your session affinity config now points to a dead endpoint.

As the Gateway API Inference Extension spec evolves, it’s clear the community recognizes this: routing decisions need to account for cache locality, KV-cache capacity, active requests, and queued token work, not just session identity.

The real question isn’t “which replica did this user hit last?” It’s “which replica has the most useful prefix cached, and is picking it actually cheaper than recomputing somewhere else?”

Here’s the uncomfortable truth: Kubernetes isn’t doing anything wrong. It understands pods, GPUs, networking, and failures. It simply doesn’t understand tokens, KV caches, or inference queues. Those are concepts from the inference layer, not the infrastructure layer.

Consider what a genuinely intelligent routing decision needs to weigh:

A standard kube-proxy round-robin doesn’t evaluate any of these. It doesn’t even know they exist.

The good news? This problem is solvable. Projects like llm-d router implement configurable scoring systems that evaluate exactly these trade-offs, integrating with vLLM replicas via the Gateway API Inference Extension’s Endpoint Picker (EPP) pattern.

To understand why this matters more every year, look at how agentic workloads actually behave.

An agent’s prompt consists of two parts: the prefix, system prompt, tool definitions, accumulated conversation history, and the new content, the latest tool result or user message. In a typical agent loop, that prefix stays largely identical across calls while the new content sliver at the end changes.

Each tool call adds its result to the end of the context, so the prefix only grows as the session runs. For a long agentic workflow, you’re looking at 20, 50, or even 100+ tokens of context that repeat across every request.

Without prefix caching, the serving engine rebuilds KV state for the entire prompt before generating the first token. With it, cached tokens are skipped, and prefill covers only the uncached suffix, dramatically reducing TTFT and computational cost.

But here’s the catch: a warm cache sitting on one node is only useful if the next call actually lands there.

When your load balancer scatters an agent’s turns across workers with no awareness of cache state, you get the worst of both worlds:

The math gets uglier with scale. A recent release note from oMLX illustrates just how expensive naive KV handling gets: legacy cache storage for an 84K-token session consumed 282.7 GB, moving to a linear storage layout dropped it to 15.3 GB for 94K tokens. Cache management is not a footnote, it’s the difference between viable and bankrupt infrastructure.

So what does proper LLM-aware routing look like in practice? The Gateway API Inference Extension (GAIE) defines the InferencePool resource type, which groups inference endpoints running the same model. An Endpoint Picker (EPP) component, implemented by projects like the llm-d Router, selects the optimal endpoint based on inference-specific metrics.

The request flow looks like this:

1. Request arrives at the Gateway (an Envoy Gateway, for example)
2. The HTTPRoute selects an InferencePool as the backend
3. The EPP evaluates endpoints based on prefix cache content, memory utilization, and queue length
4. The EPP returns the best endpoint to the Gateway
5. The Gateway forwards the request to the selected inference server

The key insight? An InferencePool groups model server pods into a routable Gateway API backend, and the EPP learns what’s cached where by streaming metadata from the inference engines. vLLM can publish KV cache events over ZMQ, the EPP consumes those events to build an accurate picture of cache locality across all replicas.

This approach doesn’t just consider one dimension. The llm-d router’s scoring mechanism uses multiple plugins, weighing prefix cache reuse against queue depth. Because purely cache-affinity-based routing can create hot spots, the worker with the best cache might be overwhelmed while an idle worker with a partial cache would serve the request faster.

There’s a legitimate skepticism here. A skeptic might point out that distributed, disaggregated inference is genuinely hard, and that moving KV state over network interconnects (even fast ones like NVLink) can introduce contention that erases caching benefits.

That’s fair. Routing decisions are cost-benefit calculations, not rule-based lookups.

But the alternative, ignoring cache state entirely, guarantees repeated computation. The question isn’t whether to build smarter routing, it’s how quickly you can get there.

The inefficiency is also compounding with the growing economic pressures of LLM inference at scale. Every token recomputed is compute paid for twice. Every user waiting on a slow TTFT is revenue lost.

From the field, here’s what separates teams that handle this well from those that don’t:

The final piece is observability. Routing decisions happen fast, and you need visibility into whether they’re working.

Without these, you’re flying blind, and cache misses become user-visible performance problems you can’t explain.

Traditional Kubernetes load balancing is a fine tool for stateless services. For LLM inference, “stateless” is a myth you can’t afford to believe.

The difference between infrastructure that runs agents well and infrastructure built for how agents actually behave comes down to whether routing considers the state of your inference fleet. The teams that crack this will deliver dramatically better TTFT and slash their GPU bills. The ones that don’t will keep paying for prefill computation they’ve already purchased, once per request, per replica, forever.

Prefix caching is table stakes. Prefix-aware routing is the infrastructure layer that makes it work. The choice isn’t between simple and complex routing, it’s between paying for compute twice or thinking about state. And the hidden cost of naive load balancing is just one of the hidden cost trade-offs in running LLMs efficiently that too many teams discover only after the invoice arrives.