Every ML system starts with the same innocent pattern. One service. It loads the model into memory, fetches features, runs preprocessing, calls .predict(), applies a few business rules, shapes the response, and returns it. One service, one repo, one on-call rotation.
It works beautifully, right up until it doesn’t.
Then growth does what growth does. One model becomes a handful. A handful becomes a mesh of services wired together to enrich a single message. Each new model is one more always-on service bolted onto the last. Nobody designed this mesh deliberately. It grew organically because no layer owned the orchestration between models.
The result? Latency becomes the sum of hops rather than the sum of work. Costs compound with every new deployment. And debugging requires stitching together logs from a dozen services that all speak slightly different dialects.
This is the story of how CRED hit that wall, and what they did about it. Their journey from tangled monolith to a clean two-layer engine offers transferable lessons for anyone running ML systems at scale.
The Three-Pronged Collapse
The Python Runtime Problem
The first place architectural debt showed up was the language itself. CRED trained in Python, so they served in Python. Fine for one model, disastrous for concurrent traffic.
The GIL forced every request through one core, no matter how many the box had. The standard fix, one worker process per core, ate memory they couldn’t spare. Copy-on-write sharing eroded as Python wrote to its own state. Asyncio only helped while a request was purely waiting on I/O. One heavy prediction still blocked every other request stuck behind it on the same worker.
This is a classic trap: the runtime that’s perfect for training is often the wrong tool for serving, but teams keep it because rewriting feels like premature optimization. Until it isn’t.
The Orchestration Problem
The next problem emerged in how services communicated. In CRED’s highest-volume system, a message came in, a lightweight router worked out what enrichment it needed, and then it moved through a mesh of services, some sequential, some parallel. Every queue added another hop and another wait.
Nobody had designed that mesh. It had accreted, one model and one use case at a time. The bottleneck was no longer the models themselves. It was the orchestration required to connect them.
This is the insidious part of architectural debt: it doesn’t announce itself with a single catastrophic failure. It erodes performance gradually, making every new feature slightly harder and every request slightly slower until someone finally measures the end-to-end path and gasps.
The Operational Problem
Eventually, the challenge shifted from technical to organizational. Conflating a model with its service forced compute-heavy execution and orchestration to rely on a single scaling signal. One would starve while the other ran hot. Without clear boundaries between feature fetches, model calls, and downstream dependencies, debugging became nearly impossible.
And here’s the kicker: model owners carried infrastructure issues they had no business owning. The people who should have been improving model accuracy were instead tuning autoscalers and fighting memory leaks.
Separating Hosting from Serving
CRED’s insight was deceptively simple: a single service was handling two distinct responsibilities, model execution and request orchestration. They named these roles:
Hosting is where the model lives and how you call it. Serving is how a request becomes an answer.
The split buys three things:
- Independent scaling. Each layer scales against the load signals that matter to it. Orchestration-heavy workflows spin up serving resources, compute-heavy models spin up hosting resources. No more over-provisioning one to accommodate the other.
- Runtime flexibility. Model runtimes can be upgraded or replaced without touching client-facing services. You can swap a Python model runtime for something faster without rearchitecting the orchestration layer.
- Clear cost attribution. A dedicated
.predict()interface means hosting and orchestration costs are measured independently, not lumped into one ambiguous service.
The results speak for themselves: today the platform serves over 9 million predictions daily with roughly 97% lower end-to-end latency, around 70% lower infrastructure usage, and nearly 60% lower serving costs compared to the previous Python architecture.
The author is careful to note these gains came from removing an architectural bottleneck, not from a faster inference runtime alone.
Building an Engine, Not a Collection of Scripts
The real magic of CRED’s refactor wasn’t just the layer split. It was rebuilding the serving layer as a framework instead of bespoke code per workflow.
The building blocks:
- Orchestrator, a single entry point where HTTP requests and queue consumers follow the same execution path. It converts incoming requests into a Payload, assigns trace identifiers, and invokes the Assembly.
- Assembly, defines the three-stage lifecycle (Preprocess → Pipeline → Postprocess) that every workflow follows.
- Pipeline, declarative composition of Steps and Parallel Blocks, with platform guarantees like per-step timeouts, panic recovery, and conditional short-circuiting implemented once by the framework.
- Step, the smallest unit of work. Stateless, communicating only through the shared Payload.
- Parallel Block, groups independent steps executing concurrently, each on its own Payload copy, with an optional combiner.
- Payload, the single shared object every step reads from and writes to.
A workflow assembles like this:
pipeline.NewPipeline(
validate,
extractMetadata,
Parallel(lookupA, lookupB).WithCombiner(reconcile, "lookupA", "lookupB"),
route,
modelPrimary,
modelFallback,
)
This isn’t just elegant, it’s operationally transformative. Per-step timeouts, panic recovery, and tracing become framework concerns, not per-workflow concerns. Teams stop reinventing error handling every time they onboard a new model.
One Execution Path to Rule Them All
A key decision was forcing both queue and HTTP traffic through the same orchestrator. This eliminated the pattern where HTTP requests, queue consumers, and batch jobs evolve independently, gradually duplicating orchestration logic.
For asynchronous processing, a lightweight layer polls messages, groups them into micro-batches, and handles acknowledgment, retries, and dead-letter queues. For synchronous HTTP requests, there’s no queue buffer, so backpressure must be enforced at the edge through admission control, rate limiting, quotas, and overload responses.
This was one reason CRED implemented the serving layer in Go. Without a natural buffer, framework overhead directly affects request latency, making efficient orchestration critical.
The unified execution path delivers an often-underappreciated benefit: observability as a platform property. Every step is a named execution unit. Every request carries a reference_id threaded through every step. Per-step latency, outcome counters, throughput, and saturation metrics are uniform across queue and HTTP execution. No more stitching together logs from multiple services.
The Broader Lesson
There’s a quote from CRED’s engineering team that captures the architectural principle perfectly: “Serving is not where predictions happen, it’s where requests become answers.”
Once orchestration and model execution have clear ownership boundaries, each layer can evolve independently. The platform becomes simpler to build, easier to operate, and significantly more efficient to scale.
This pattern extends beyond CRED. The growing ecosystem of tools for model serving, from the high-throughput inference engines behind LLM serving to the managed platforms with their traffic splitting and endpoint configuration capabilities, all reinforce the same architectural truth: the model is only one part of the inference pipeline. At scale, the work happening around it has a much bigger impact on latency and efficiency.
The model ecosystem has already embraced separation of concerns at the infrastructure level, with pay-per-token endpoints and provisioned throughput tiers that let teams choose between convenience and control. But those tools don’t fix a fundamentally tangled internal architecture. They just relocate it.
Practical Takeaways Worth Stealing
If you’re not operating at CRED’s scale, these lessons still apply. The principles transfer regardless of your traffic volume:
1. If a single service is doing both model execution and request orchestration, the two responsibilities will fight each other for the same scaling signal. Split them.
Your orchestration-heavy workflows and compute-heavy model predictions have different load profiles. Forcing them to share a scaling signal means one starves while the other runs hot. This is true at 100 requests per day or 9 million.
2. Express workflows declaratively as pipelines of stateless steps sharing a single payload.
When steps are stateless and communicate only through a shared object, they become reusable and composable. Per-step timeouts, panic recovery, and tracing become framework concerns, implemented once, used everywhere.
3. Use one execution path for HTTP and queue ingestion.
The observability win alone is worth it: uniform per-step metrics and a single reference_id threaded through every step. You’ll never again spend hours trying to correlate a request across three different log formats.
The Cost of Not Refactoring
Here’s the uncomfortable truth about the progression of architectural entropy in codebases over time: it compounds silently. Every “temporary” integration becomes load-bearing. Every shortcut becomes a constraint. The systems that feel fastest to build in the short term are often the ones that make you slowest in the long term.
The data is unambiguous. A 97% latency reduction and 60% cost reduction didn’t come from a faster inference engine, they came from removing an architectural bottleneck. CRED didn’t buy better hardware or wait for Python to get faster. They drew a clean line between two responsibilities that never should have been tangled together.
The hardest part of this refactor wasn’t technical. It was acknowledging that an architecture developed incrementally had become the bottleneck, and that fixing it required deliberate, uncomfortable work. That’s the nature of architectural debt.
Interested in how this pattern connects to other architectural challenges? You might want to explore the hidden technical debt in new architectures that appear clean but lack intentional design or consider whether your scaling strategy is addressing symptoms rather than structural inefficiencies.
Your model isn’t the bottleneck. Your architecture is. The good news is that architecture can be fixed, if you’re willing to do the work.
