Optimizing hot spots in your code might feel productive, but unless you wrote truly terrible code, a condition architects politely call “premature pessimization”, you’re not going to see order-of-magnitude improvements from micro-tuning loops. A single system-level optimization, however, can remove unnecessary network calls and file access operations from critical paths entirely. The result: latency drops by an order of magnitude while your main system gets less load, not more. That’s the kind of win that gets you promoted.
The catch? These optimizations require you to challenge the architectural decisions your team made when the system was young and innocent. Let’s dig into the patterns that actually move the needle, beyond the hot path.
Shortcutting: Skip the Middleman (Literally)
The most direct way to make a system fast is to not use parts of it. It sounds obvious, yet most architectures accumulate components like a hoarder accumulates newspapers, each layer justified by some edge case that rarely happens.
Early response is the simplest form. A firewall, rate limiter, or response cache that sits near the front of your request flow can answer immediately without consulting the business logic behind it. A rate limiter blocks disallowed requests outright. A response cache remembers what it saw before. These systems respond faster than anything downstream possibly could.
A more aggressive variant is bypassing the main system entirely. Low-latency systems frequently rely on DPDK, which maps network packets directly into application memory, skipping the kernel’s networking stack. HFT firms encode trading rules into FPGAs mounted on network cards. When a price tick matches a rule, the decision happens in nanoseconds without ever touching the OS.
Component omission is subtler. A half-proxy connects a client to a server and steps out of the way. An open layer participates in some requests but is skipped for others. Telephony servers exist to help clients find each other, then vanish from the conversation. The lesson: components that serve a coordination purpose don’t need to serve the actual payload.
Co-location: The Network Is the Bottleneck
Every network hop adds microsecond-level delays that compound across requests. The obvious fix, put things closer together, gets ignored because “service boundaries” and “team ownership” maps rarely overlap with performance requirements.
The frontend tier running on user devices is the most common co-location example. An ambassador proxy sits next to a client application and acts on the system’s behalf. A sharding proxy lets clients connect directly to the shard containing their data rather than routing everything through a central gateway.
Sidecars in service meshes co-locate generic infrastructure code with business logic. Actor frameworks actively migrate actors between hosts to keep heavily-interacting actors on the same machine. AUTOSAR requires applications and their services to run on the same chip. This is the extreme latency reduction through on-device execution philosophy applied at the infrastructure level: if you can’t eliminate the work, at least eliminate the travel time.
The monolith, that pattern everyone loves to hate, achieves the ultimate co-location because everything shares one process. Properly implemented, a monolith has near-zero inter-component latency. Distributed teams pay for their organizational structure with network round-trips.
Preloading and Preprocessing: Right Data, Right Place, Right Format
If you have the data you need, in memory, adjacent to the code that uses it, formatted for your exact use case, you win. No query. No network call. No deserialization.
Actors and Space-Based Architecture keep state in operating memory next to processing logic.
Response caches store pre-computed answers. CQRS maintains derived OLAP databases optimized for queries, streaming changes from the primary system so the read side never blocks the write side.
Memory images collect event-sourced state changes into in-memory snapshots. External search indexes pre-process documents for efficient lookup.
The pattern here isn’t caching, it’s anticipation. You’re predicting what data will be needed and preparing it in advance, shifting work from request time to idle time.
The Nuclear Option: Injecting Logic at the Edge
Denys Poltorak’s excellent analysis on latency optimizations on the system level describes the ultimate combination: injecting business logic, together with preprocessed data, directly into the component that handles events. This allows the downstream component to make decisions independently, with zero external calls.
High-frequency trading does this by encoding precomputed trading rules into FPGAs. The host operating system’s entire software stack, scheduling, system calls, context switches, disappears from the critical path.
Uber’s Ambassador Plugins (aka Logic Extensions) apply this pattern at the service level. A service publishes an interface that other teams’ code can plug into. The host service calls these plugins as strategies during request processing. Since plugins are co-located with the host process, interservice calls vanish. The plugin owner’s team streams event data to keep the plugin’s decision engine fresh.
The catch? Someone has to build the event feed, maintain the plugin, and handle versioning. It’s a genuine architectural commitment, not a weekend refactor.
Non-Blocking: Don’t Share, Don’t Block
Fast systems either avoid blocking during request processing or avoid sharing resources between clients. The Proactor pattern processes events with non-blocking callbacks, giving real-time properties at the cost of readability that makes junior developers weep.
Shards dedicate database instances to subsets of clients, so one client’s heavy workload can’t impact another’s latency. Actor systems combine both strategies: one non-blocking actor per client, no shared state. Space-Based Architecture maintains multiple in-memory replicas, each serving a subset of requests, with background state synchronization.
The downside of non-blocking everywhere is complexity. Event-driven code with callbacks and continuations is notoriously hard to reason about. It’s a tool for when you’ve exhausted simpler options.
Miscellaneous Knobs Worth Turning
Two optimizations from the miscellaneous drawer deserve particular attention. Request hedging sends a request to multiple replicas in parallel, accepting the first response. It’s the only pattern here that directly attacks tail latency rather than improving median performance. The cost: you’re spending 2x-3x resources on every hedged request. Worth it for p99-sensitive systems.
Parallel execution splits a request into subrequests processed simultaneously by an orchestrator. An API composer fanning out to multiple services cuts total latency from the sum of service times to the max. Your slowest dependency becomes the ceiling, not the total.
Persistence tuning matters more than most engineers realize. Moving historical data to archive storage, using a pair of specialized SQL + NoSQL databases, and shared memory for inter-process communication all produce outsized latency improvements. A microsecond-level network tuning for latency reduction like disabling Nagle’s algorithm with TCP_NODELAY frequently produces the single biggest win for distributed services, and it’s one line of config.
Diagnosis Before Action: The LLM Serving Cautionary Tale
The LLM serving p95 latency analysis demonstrates a critical principle applying to all of these patterns: fix the cause, not the symptom. High p95 latency in LLM serving comes from five distinct sources with five distinct fixes:
| Cause | Diagnostic Signature | Fix |
|---|---|---|
| Queue contention | Rising queue depth + rising p95, stable time-to-first-token | Autoscale, rate limit, or restore capacity |
| KV cache pressure | High cache occupancy + rising p95 and inter-token latency | Cap context, paged attention, evict idle cache |
| Large prompts | Rising p95 TTFT + large prompt lengths | Prompt caps, routing, prefill optimization |
| Batch size | Larger batches + rising p95, rising throughput | Cap batch at latency target |
| Resource saturation | Near-peak utilization + uniformly rising latency | Add capacity, reduce concurrency, optimize |
Adding GPUs to a queue problem increases capacity without removing the bottleneck. The same logic applies to system-level optimizations: pattern-matching without diagnosis produces changes that make you feel productive while the metrics stay flat.
The Proxy You’re Paying For Twice
Consider the Valkey/Redis direct-access architecture case. Sticking a proxy between your application and cache adds a few hundred microseconds per hop. Under a million queries per second, the proxy redlines at 90% CPU while your data nodes sit at 60%. Direct access eliminates the network hop, slashing tail latency from several milliseconds to around 500-600 microseconds, and you stop paying for proxy VMs that do nothing but move packets.
This isn’t just about speed. It’s about cost. The proxy fleet is pure overhead, burning dollars for the privilege of adding latency.
Practical Takeaways
Start with the load-bearing question: For each request type, map the actual path through your system. Find every component that’s involved but unnecessary. Remove it.
Challenge service boundaries. Teams love boundaries, they make ownership clean. But every boundary is a network call. Ask whether the data could be co-located, preloaded, or embedded as an ambassador plugin.
Measure p95, not average. Average latency hides the slow requests. A system with a 200ms average and 3-second p99 has a serious problem that the average completely conceals. The gap between median and p95 is the early warning signal.
Buy your latency victories with architecture, not code. A system-level optimization that removes network calls from the hot path delivers an order-of-magnitude improvement. Code tuning delivers percentages. Do the architecture work first.
The systems that feel fast, the ones users describe as “instant”, aren’t fast because of clever algorithms. They’re fast because they don’t do unnecessary work. The patterns in this post are all variations on that theme: do less, do it closer, or do it in advance. Start there, and the hot-path micro-optimizations can wait.
