Your AI Workflow Doesn’t Need a Service Mesh (Probably)

I’ve seen it happen dozens of times now. A team builds a simple AI workflow, maybe it summarizes support tickets, classifies customer feedback, or generates draft responses. The core logic is essentially:

def process_ticket(ticket_text):
    classification = llm.classify(ticket_text)
    summary = llm.summarize(ticket_text)
    return {"classification": classification, "summary": summary}

That’s it. One function. Two LLM calls. Maybe 50 lines of code.

But by the time the “production architecture” is planned, that simple function has sprouted an entire ecosystem: an orchestration layer, a vector database, an evaluation pipeline, an observability stack, a gateway, a memory store, and, because why not, an agent protocol. The team spent three weeks building infrastructure around a workflow that takes 400 milliseconds to execute.

This isn’t an isolated incident. A recent discussion on r/softwarearchitecture captured it perfectly: “At some point you have orchestration, vector databases, observability, evals, gateways, memory, agent protocols and a bunch of other layers, even when the underlying workflow isn’t doing anything particularly complicated.”

The uncomfortable question we need to ask: Are we repeating the microservices mistake with AI, just with better marketing?

The Infrastructure Fetish Strikes Again

The pattern is eerily familiar. In the mid-2010s, every startup was decomposing their Rails monolith into “scalable microservices” because Netflix did it, ignoring that Netflix was handling billions of requests daily with hundreds of engineers. The result was a graveyard of distributed monoliths, event-driven spaghetti, and Kafka clusters serving fewer messages than a busy Discord channel.

Now the same groupthink is infecting AI development. The trigger this time isn’t a Netflix talk, it’s an AI-generated architecture diagram. As Devrim Ozcay points out, when you ask an AI to design a scalable architecture, it regurgitates the canonical stack: load balancer, API gateway, microservices, Kafka, Redis, Kubernetes, database replicas, caching, observability. It looks comprehensive. It’s also exactly what you get when you ask for every component that might eventually prove useful.

These diagrams become the blueprint, and suddenly your ticket-classification function is a distributed system with six services, two message queues, and a saga pattern.

The Real Cost of “Production-Ready” AI Infrastructure

Let’s be concrete about what this modularization actually buys you versus what it costs. A developer with a straightforward AI workflow is now responsible for:

• Another deployment pipeline for each new service
• Another codebase to maintain, version, and keep consistent
• Another failure mode, each network boundary means timeouts, retries, and partial failures
• Another set of logs to aggregate, correlate, and debug
• Another ownership question, who’s on-call for the vector database at 2 AM?
• Another migration every time the infrastructure vendor shifts their API

What was once a function call becomes a network request. And as any experienced engineer will tell you, that change transforms a simple processing step into a distributed systems problem. You’re now dealing with:

Service A → gRPC → Kafka → Service B → Redis → Service C → LLM

And asking questions like: “What happens if Kafka is unavailable? What if Service B processes the message twice? What if Redis returns stale context? What if the network partitions?” All these questions apply to a workflow that was originally three lines of sequential Python.

This analysis of latency optimization patterns highlights something even more worrying: agentic AI workloads generate 10-20 times the data-layer traffic of traditional application interactions. Every additional service inherits that multiplier. You’re not just adding a network hop, you’re adding a network hop that gets hit 20 times per agent action.

Does your three-step workflow need that? Almost certainly not.

The Missing Question: What Failure Are We Preventing?

There’s a test that every proposed architectural component should pass before it’s added to an AI system, and it has nothing to do with how impressive the final diagram looks. The question is simple: What specific failure does this component prevent, and have we actually observed that failure occurring?

Memory makes sense when your system genuinely needs continuity across sessions. Evals become necessary when the workflow has enough variability that regressions could go undetected. Multi-agent orchestration earns its keep only when there’s real decomposition or degradation of reliability at play. RAG is justified when the model needs external knowledge that can’t fit in the context window.

Otherwise, and this is the key insight from engineers who’ve gone through the cycle, you’re building infrastructure around the idea of a problem, not a problem you’ve actually encountered.

One developer’s experience sums this up well: “My first go was also to keep adding layers like agent orchestration, memory, RAG, evals, observability, different protocols, multiple agents, etc. On paper it did make the architecture feel much more production ready. But while implementing and testing these things I realised a lot of the complexity was solving problems I had not actually encountered yet.”

They stripped it back to “good context, a well-defined task, deterministic validation and a small amount of state” and found the system was, surprise, easier to understand, debug, and improve.

The Inflection Point: When Modularization Actually Pays Off

Now here’s the part that complicates the purity of the “monolith-first” argument: there’s also a documented pattern where components that initially looked like overengineering became necessary once the scenario changed. Multiple engineers have reported the same trajectory, started simple, hit real scaling issues or multi-user demands, then added infrastructure deliberately and surgically.

The takeaway isn’t “never adopt microservices in AI.” It’s that the adoption should be evolutionary, driven by observed failure modes, not anticipatory and speculative.

This isn’t an anti-modularity stance. It’s an anti-premature-complexity stance. The measurement should be: Does this component prevent a failure you’ve actually observed? Not: Does adding this make us look more enterprise-ready?

The AI-Architecture Catch-22

There’s one more factor unique to AI systems that makes over-modularization particularly insidious: the industry changes so rapidly that the “best practice” you’re building around may be obsolete in six months.

Developers working in the space have noted how quickly the canonical approach shifts. “You about have OK results with one way of doing a thing when the next ‘best practice’ comes down the pipe. ‘What do you mean you’re not GraphRAG-ing?!? That’s not optional, that’s fundamental!'”

Build a six-service architecture around RAG-as-implemented-today, and you may find yourself rearchitecting when GraphRAG (or whatever’s next) becomes the standard. The more modular your system, the more integration points you need to update when the AI landscape shifts. Each “independent” service is now a coupling point to a moving target.

In contrast, a simpler system can adapt to new paradigms far more quickly. If your workflow is a well-structured monolith with clear internal boundaries, swapping the RAG implementation is a contained change. If it’s spread across three services and a message bus, that’s a larger refactoring effort touching multiple codebases.

The Environmental Stressors That Drive the Madness

Before we go full puritanical on the “just keep it simple” message, it’s worth acknowledging why teams over-build. There’s a context that technical analysis often misses.

In many organizations, AI projects exist under “upward pressure to do something AI related.” A product might not earn adoption, but an engineering team can at least build expertise and demonstrate progress by deploying a technically sophisticated architecture. In that context, as one developer observed, “testing out/building expertise with the technology is probably the best use of your time as an engineering team on what is a DOA project.”

The infrastructure isn’t solving the business problem, it’s creating the appearance of forward motion while the actual product direction remains undefined. That’s a real political reality, but it’s worth being honest about what’s happening: you’re not building architecture for the user, you’re building it for the quarterly tech review.

The Five-Question Audit for Your AI Architecture

Before your team adds another service, vector database, or orchestration framework, run it through this checklist:

1. What specific problem are we solving? Not “scalability” or “future-proofing.” A concrete failure or bottleneck. If you can’t name one, you probably don’t have one.
2. Why can’t the current architecture solve it? Document the limitation. Does it actually prevent you from meeting requirements, or is it just inelegant?
3. What complexity are we introducing? Every service adds operational surface area. What’s the full list of new failure modes, not just the features?
4. What happens when this component fails? If the answer is “we’ll have a bigger incident”, that’s a real cost. Include downtime in your analysis.
5. Can the team operate this system at 3 AM? If not, or if you’d have to page three different owners, the architecture is already too complex.

Teams that follow this process are finding that pragmatic modular monoliths, or even well-structured single services, deliver more value in AI contexts than full microservices decompositions. The modular approach gives you clear boundaries and internal contracts without the operational burden of distributed systems. That’s also why even AI-focused startups are starting with monoliths for simplicity, the team can move faster, debug more easily, and only split when there’s a demonstrable need.

The Not-So-Spicy Conclusion

Here’s the thing nobody wants to hear at a tech conference: the simplest architecture that meets your requirements is usually the right one. Not always, but often enough to be the default.

Your three-step AI workflow doesn’t need a service mesh. It doesn’t need six containers, an event broker, and a vector database. It needs clear code, proper error handling, and a sensible way to update the prompt when the model inevitably changes its behavior.

The senior engineering answer, the one that actually distinguishes experienced practitioners, is often, “Not yet.” You can always modularize later. What you can’t easily do is un-modularize an overbuilt system without painful rewrites and coordination costs.

As the case studies from Shopify and Discord demonstrate, the organizations that have truly struggled with microservices dogmas are the ones who treated architectural patterns as status symbols rather than engineering tradeoffs. And the teams that have successfully navigated this territory are converging on a pragmatic middle ground: modular internals, simple deployment, and the discipline to only break things apart when the boundaries provide clear value.

Your AI infrastructure stack should serve the workflow, not the other way around. The anti-hypertrophic design philosophy has a simple starting point: build the smallest system that can prove the workflow, observe the actual failure modes, and then, and only then, purchase additional complexity.

Because in AI development, as in life, the most impressive architecture is the one that doesn’t crash at 2 AM, doesn’t take three weeks to debug, and doesn’t require a distributed systems PhD to understand.

What’s the most over-engineered AI system you’ve encountered? The comment section is now open for architectural confession.