There’s open source, and then there’s “we’re publishing the exact code that runs our billion-dollar production service” open source. DeepSeek just did the latter.
On October 20, the Chinese AI lab released DeepInfer, the inference framework that powers its large-scale production services, under the Apache 2.0 license. Alongside it came deepseek-recipe, an MIT-licensed companion library handling token encoding, decoding, and API protocol conversion.
The reaction was immediate. The announcement on X pulled in 3.8 million views within hours. The GitHub repo hit 12.4k stars and 1.1k forks before most people had even read the README. And the community response? A mix of genuine excitement, healthy skepticism, and the usual accusations of “Chinese bots” inflating the star count.
Let’s dig past the hype and figure out what DeepSeek actually shipped.
Breaking Down the Performance Numbers
DeepSeek published Pareto curves for both prefill and decode phases at 131,072-token context. The benchmarks ran on NVIDIA SM100 GPUs, 4 for prefill, 64 for decode, each within a single scale-up domain.
Prefill performance is genuinely impressive. With EP4DP4 parallelism and 140-token bounded replay, the engine hits roughly 60,000 non-cached tokens per second per GPU at 4,096 tokens per rank. Push the batch size to 65,536 tokens and throughput approaches 90,000 tokens/s/GPU with about 730ms per step.
Decode is where things get spicy. Running EP64DP64 with nextn=3 speculative decoding at 75% draft acceptance, DeepInfer sustains over 25,000 output tokens per second per GPU at 256 requests per rank. That translates to roughly 100 tokens per second per user, fast enough that the model feels genuinely responsive.
At lower concurrency, 8 requests per rank, per-user speed passes 300 tokens per second. That’s the kind of throughput that makes interactive AI applications feel like native software instead of network calls.
The Engine Behind the Curtain
DeepInfer isn’t a toy. It’s the production stack that runs DeepSeek-V4.1-Flash at scale, the same infrastructure serving 155 trillion tokens per day across a single cluster. According to the deployment Q&A, that operation generates approximately $38 per GPU-hour in revenue at off-peak pricing. Those numbers aren’t aspirational—they’re the actual economics of a working AI business.
The framework itself is a tightly integrated assembly of components DeepSeek has been dropping piecemeal for months. The kernel stack alone reads like a greatest hits album: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels. What’s new here isn’t the individual pieces—it’s the fact that they’re finally wired together into a coherent engine.
The README is refreshingly honest about the lineage: “DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM.”
That’s a polite way of saying DeepSeek took the best open-source inference engine in existence, ripped out the generic parts, and replaced them with hyper-optimized components designed specifically for their own models. The result is fast. Very fast.
Technical Innovations
The DSpark Difference
The most interesting technical innovation is DSpark, DeepSeek’s confidence-scheduled speculative decoding. Traditional speculative decoding uses a draft model to propose tokens, then verifies them against the main model. The inefficiency? You waste compute verifying tokens that get rejected.
DSpark adds a confidence head that predicts each draft token’s acceptance probability. The scheduler then verifies only the most valuable prefix of the draft—skipping tokens the model is likely to reject anyway. Per the deployment Q&A, this cuts verified draft tokens by about 30% (from 5.00 down to 3.2–4.4) while barely affecting accepted length (3.29 vs 3.20).
The net effect: 4.7% throughput improvement at DeepSeek’s production operating point of ~60 sequences per rank. At 16 sequences per rank, it’s 1.5% slower than fixed 5-token verification—the confidence scheduling overhead isn’t free. But at scale, it pays for itself.
SWA Bounded Replay
The other clever trick is SWA bounded replay, which addresses one of the pain points of sliding-window attention (SWA) models: cache misses. When a request hits a cached prefix, most engines recompute the full attention state. DeepInfer instead replays only the last 140 tokens of the window (DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140 by default).
The result is that recomputation cost becomes constant regardless of context length. DeepSeek claims this nearly halves total prefill compute in the decoder layers. The tradeoff? The output is technically approximate—you’re not replaying the full window. But at 140 tokens of suffix, the divergence is negligible for most workloads.
What’s Missing (And Why It Matters)
Here’s where the “spicy” part kicks in. DeepInfer is offline inference only. It’s not a complete serving stack. The README is admirably direct about what’s absent:
- A KV cache system
- VE/Prefill/Decode disaggregation
- Load balancing across data-parallel replicas
DeepSeek’s rationale: those components are “tightly coupled to internal infrastructure.” Translation: they’re the parts that would let someone actually compete with DeepSeek’s API business on equal footing.
This split is a recurring pattern in DeepSeek’s open-source strategy. The company released the model weights, the training recipes, and now the inference engine—but the glue that makes everything work as a production service stays in-house.
Is that hypocritical? Not really. Anyone who’s run a serious LLM deployment knows the gaps between “offline inference” and “production API” are enormous. The missing pieces represent months of engineering work: building a distributed KV cache, implementing prefill/decode disaggregation, writing load balancers that understand cache locality, and deploying the whole thing with observability and failover.
What DeepInfer gives you is a massive head start. What it doesn’t give you is a turnkey service.
The Hardware Reality Check
Here’s the part that separates the curious from the committed: DeepInfer supports exactly two hardware platforms.
- NVIDIA SM100 GPUs (Blackwell generation)
- Huawei Ascend 950 NPUs
That’s it. No consumer GPUs. No A100s. No H100s. If you don’t have access to SM100s or Ascend 950s, you’re out of luck—at least for now. The community has already opened RFCs for SM90/H20 and Ascend 910C support, but that’s demand, not shipped functionality.
The build requirements reinforce the enterprise-only positioning: Linux, GCC 13+, Python 3.12, CUDA 13 (NVIDIA) or CANN 9.2 (Ascend). The smallest efficient production unit is 64 prefill GPUs plus 64 decode GPUs. This is not a framework for your home lab.
But the Huawei support is the real story here. DeepSeek’s partnership with Huawei to bring its kernel stack to Ascend NPUs represents a significant geopolitical shift in AI infrastructure. In September, DeepSeek open-sourced six software modules for the Ascend platform, including an Ascend-adapted version of TileLang. DeepInfer completes that picture—a full production inference stack that runs on Chinese-designed hardware.
How DeepInfer Compares to the Competition
The framework comparison chart is instructive:
| Feature | DeepInfer | vLLM | SGLang | TensorRT-LLM |
|---|---|---|---|---|
| Maintainer | DeepSeek | vLLM project | SGLang project | NVIDIA |
| License | Apache 2.0 | Apache 2.0 | Apache 2.0 | Apache 2.0 |
| Model coverage | 1 (V4.1-Flash) | 200+ architectures | LLMs, VLMs, diffusion | Support matrix |
| Hardware | SM100, Ascend 950 | Multiple vendors | Multiple vendors | NVIDIA GPUs |
| Expert parallelism | Yes, with EPLB | Yes | Yes | Wide-EP |
| Speculative decoding | DSpark | Multiple methods | Via SpecForge | Yes |
| P/D disaggregation | Not in release | Yes | Via llm-d | Yes |
| Online serving | Offline only | OpenAI-compatible | Yes | Yes |
The pattern is clear: DeepInfer trades generality for raw performance on one specific model and two hardware platforms. vLLM or SGLang will serve 200 model architectures on commodity GPUs today. DeepInfer will serve DeepSeek-V4.1-Flash on SM100s with cost and efficiency advantages that no general-purpose engine can touch.
For developers building custom inference engines—projects like gufo, halogen, strixite, and Strata in the community—this release is a goldmine. It’s a rare look at how a frontier lab actually structures its serving stack. As one developer noted on Reddit, “this is crazy good for studying to improve existing custom engines… by learning their secret sauces.”
The Stratospheric Economics
Let’s put the revenue numbers in perspective. DeepSeek reports $38 per card per hour at off-peak pricing. On 128 GPUs (64 prefill + 64 decode), that’s roughly $4,864 per hour, or about $116,700 per day, assuming 100% utilization at off-peak rates.
Compare that to DeepSeek’s API pricing, which starts at $0.04–$0.15 per million tokens depending on provider. The per-card revenue only makes sense at massive throughput—which is exactly what DeepInfer is designed to deliver.
The cost per token differential between a well-optimized engine and a generic one isn’t a rounding error. As we’ve covered before, DeepSeek’s API is priced 50x–150x below competitors like GPT-5.5 or Claude Opus. The efficiency gains from DeepInfer are a big part of how that pricing is sustainable.
A Brief History of “We Can’t Open Source This”
One detail in the launch notebook is worth highlighting. In April 2025, DeepSeek said a full inference-engine release was “impractical”—the code had diverged too far from vLLM, was too coupled to internal infrastructure, and the team lacked maintenance bandwidth.
Eighteen months later, they shipped it anyway.
What changed? The Ascend groundwork (September 2026) and, probably, a strategic realization that open-sourcing the engine doesn’t meaningfully erode their competitive advantage. The missing pieces—KV cache, disaggregation, load balancing—are the real moat. The engine itself is the portion of the stack where community contributions can actually benefit DeepSeek without threatening its API business.
This is a pattern that should feel familiar if you’ve been following DeepSeek’s AGI-over-profit philosophy. The company consistently open-sources what most labs would consider proprietary, then maintains a small set of genuinely closed components that preserve commercial viability.
What This Means for the Community
The immediate winners here are:
deepseek-recipe library alone is valuable: it guarantees API behavior that matches the official DeepSeek API, which has historically been a pain point for self-hosted deployments.
The losers? Anyone hoping for a drop-in OpenAI-compatible server. DeepInfer doesn’t include one. You’d need to build it yourself or wait for the community to wrap it—and the accelerating AI model release cycles mean the window for DeepSeek-specific optimizations is always narrowing.
The Bottom Line
DeepSeek has done something genuinely unusual here. It published the production inference engine behind one of the most cost-efficient AI services in the world, under a permissive license, with surprisingly candid documentation about what’s included and what isn’t.
The missing pieces aren’t a scandal—they’re a design boundary. DeepSeek kept the parts tightly coupled to its infrastructure and shipped the parts that benefit from community scrutiny. That’s not deception, that’s engineering strategy.
Whether DeepSeek would have made this move on a “level playing field” is an interesting hypothetical. As one Reddit commenter observed, the open-weight strategy makes the most sense for a company that’s structurally unable to compete on distribution. OpenAI and Anthropic can charge premium prices because of brand and ecosystem lock-in. DeepSeek can’t—so it competes on efficiency and openness.
The result is a net win for anyone building AI infrastructure. The code is real. The benchmarks are self-reported but plausible. The throughput numbers are genuinely world-class. And the techniques—DSpark, EPLB, SWA bounded replay—are now in the public domain for anyone to study, adapt, and improve.
Check out the GitHub repo, the deployment Q&A, and deepseek-recipe. Then go read the code. That’s where the real value is.
And if you’re wondering whether DeepSeek’s V4-Pro launch and open-source harness was a fluke or a pattern—this release should settle that question. It’s a pattern.
