For the last three years, the AI industry has operated on a simple article of faith: large language models require large infrastructure. You want a frontier-class model? You need a GPU cluster. You need distributed inference. You need to be on your knees praying to the cloud provider gods every time your token latency spikes.
Then DeepSeek V4 Flash showed up and ran a 284-billion-parameter model on a single AMD MI300X GPU.
Not a cluster. Not a rack. One card. With a million-token context window.
The “revolutionary” cloud-centric AI architecture just got a reality check, and it’s not coming from a blog post, it’s coming from a GitHub repo and a vLLM recipe that anyone can run.
The MoE Magic Trick: 284B Parameters, 13B Active
Let’s talk about why this is possible, because it’s not magic, it’s architecture.
DeepSeek V4 Flash is a Mixture-of-Experts (MoE) model with 284B total parameters but only 13B activated per token. The difference between those two numbers is the entire ballgame. In a dense model, every parameter fires on every token. In an MoE model, a router network picks which experts are relevant and only those get activated.
Think of it like a consulting firm with 257 specialists on staff. Most firms bill you for everyone’s time whether they’re in the room or not. MoE only bills you for the 6 experts who actually showed up to the meeting.
DeepSeek’s architecture takes this further with hybrid attention: Compressed Sparse Attention (CSA) combined with Heavily Compressed Attention (HCA). The result? At 1M-token context, the V4 architecture requires only 27% of the single-token inference FLOPs and 10% of the KV cache compared to DeepSeek-V3.2. Those numbers are stated for the Pro variant, but the architectural innovations carry over to Flash.
Here’s the breakdown from the technical report:
| Model | Total Params | Activated Params | Context Length | Precision |
|---|---|---|---|---|
| DeepSeek-V4-Flash-Base | 284B | 13B | 1M | FP8 Mixed |
| DeepSeek-V4-Flash | 284B | 13B | 1M | FP4 + FP8 Mixed |
| DeepSeek-V4-Pro-Base | 1.6T | 49B | 1M | FP8 Mixed |
| DeepSeek-V4-Pro | 1.6T | 49B | 1M | FP4 + FP8 Mixed |
The FP4 + FP8 mixed precision is another piece of the puzzle. MoE expert parameters live in FP4, which is aggressive quantization by most standards. But DeepSeek compensates with architecture, Manifold-Constrained Hyper-Connections (mHC) replace conventional residual connections to stabilize signal propagation across layers.
What Actually Runs on One MI300X
The vLLM recipe for the MI325X (the MI300X’s successor variant) shows exactly what’s needed to get this running on a single card:
export VLLM_ROCM_USE_AITER=1
vllm serve deepseek-ai/DeepSeek-V4-Flash \
--host 0.0.0.0 \
--port 8002 \
--tensor-parallel-size 1 \
--kv-cache-dtype fp8_e4m3 \
--max-model-len 4096 \
--enable-chunked-prefill \
--max-num-batched-tokens 256 \
--kv-cache-memory-bytes 10000000000 \
--distributed-executor-backend mp \
--trust-remote-code \
--tokenizer-mode deepseek_v4 \
--moe-backend triton_unfused \
--enforce-eager
That’s it. --tensor-parallel-size 1. One GPU. The configuration loaded the 148.66 GiB checkpoint and passed model-list and chat probes in non-thinking and Think High modes during a 24-hour allocation.
Now, let’s be honest about the constraints: this is verified at 4K context, not 1M. And Think Max mode requires at least 384K context, which means the single-GPU configuration doesn’t unlock the full reasoning ceiling. But 4K context with a 284B-parameter MoE on one card is still a statement.
For the MI355X (4×288GB), the vLLM team recommends going up to --tensor-parallel-size 4 for lower latency, and they’ve validated it on GSM8K, scoring 0.9439 exact_match:
export VLLM_ROCM_USE_AITER=1
vllm serve deepseek-ai/DeepSeek-V4-Flash \
--host localhost \
--port 8001 \
--dtype auto \
--kv-cache-dtype fp8 \
--tensor-parallel-size 4 \
--max-num-seqs 512 \
--max-num-batched-tokens 8192 \
--distributed-executor-backend mp \
--trust-remote-code \
--gpu-memory-utilization 0.9 \
--tokenizer-mode deepseek_v4 \
--reasoning-parser deepseek_v4 \
--tool-call-parser deepseek_v4 \
--enable-auto-tool-choice \
--compilation-config '{"mode": 3, "cudagraph_mode": "FULL_DECODE_ONLY"}'
The Speculative Decoding Multiplier
Here’s where things get spicy. DeepSeek V4 Flash-0731 ships with DSpark speculative decoding, a draft module that predicts multiple future tokens so the main model only validates them.
The DSpark paper reports 60-85% faster per-user generation on V4-Flash versus the MTP-1 baseline at matched aggregate throughput. That’s not incremental. That’s transformative for interactive workloads.
You enable it with one flag:
--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'
The DSpark-fused checkpoints weigh in at ~167 GB on disk versus ~160 GB for the preview, the draft module is the difference. Both require vLLM 0.25.0, with ROCm support for DSpark landing in 0.26.0.
This matters for single-GPU deployment because speculative decoding essentially trades compute for latency. On a card with 192GB of HBM3 memory and massive bandwidth, that trade works.
Performance That Embarrasses Bigger Models
Here’s the part that should make cloud providers nervous. The 0731 update showed that the same 284B/13B architecture, upgraded purely through post-training, now beats DeepSeek V4 Pro (Preview) on every agentic benchmark DeepSeek published:
| Benchmark | V4-Flash-0731 | V4-Flash (Preview) | V4-Pro (Preview) | GLM-5.2 | Opus-4.8 |
|---|---|---|---|---|---|
| Terminal Bench 2.1 | 82.7 | 61.8 | 72.1 | 81.0 | 85.0 |
| DeepSWE | 54.4 | 7.3 | 12.8 | 46.2 | 58.0 |
| Cybergym | 76.7 | 38.7 | 52.7 | , | 83.1 |
| Toolathlon-Verified | 70.3 | 49.7 | 55.9 | 59.9 | 76.2 |
That’s a model with 13B activated parameters posting numbers that trail a frontier closed model like Opus-4.8 by single digits. On DeepSWE, a software engineering benchmark, the improved Flash outperforms its own Pro sibling by 4.2x.
And the API pricing tells the same story: Flash runs at $0.14 per 1M input tokens (cache miss) and $0.28 per 1M output tokens. That’s roughly a third of V4-Pro’s output pricing. At that rate, agent loops become economically viable for startups that previously couldn’t touch frontier-quality reasoning.
Graph: The MoE Advantage
The MoE efficiency curve is stark. A dense 284B model would require roughly 570GB of weights at FP8, spanning multiple GPUs with tensor parallelism, inter-GPU communication overhead, and all the latency that comes with it. Flash’s 13B active parameters plus aggressive FP4 quantization collapses the memory footprint to a range where one high-memory card suffices.
The tradeoff is subtle but real: MoE models have a router overhead per token, and the expert parallel loading pattern means your GPU’s memory bandwidth becomes the bottleneck, not compute. That’s why AMD’s MI300X with its 5.3 TB/s HBM3 bandwidth and 192GB capacity is a natural fit, it’s a bandwidth monster, not just a FLOP monster.
What This Means for Your Architecture
Let me be clear about what’s happening here: the assumption that frontier-class LLMs require distributed cloud infrastructure is cracking. The pressure is coming from three directions simultaneously.
First, the hardware.
AMD’s MI300X has been the underdog in the AI accelerator race, but its memory capacity (192GB) and bandwidth give it a genuine advantage for MoE inference where parameter locality matters more than raw compute. NVIDIA’s H200 has similar capacity but at a NVIDIA-tax price premium. DGX Station single-GPU configs run V4-Flash with --tensor-parallel-size 1, we’re even seeing a GB300 variant at 1×252G that handles this workload.
Second, the inference software stack.
vLLM 0.25.0+ added DSpark support, deep_gemm kernels, expert parallel weight filtering, and a host of optimizations specifically for MoE models. The --enable-ep-weight-filter flag alone speeds up weight loading for large MoE models by skipping expert weights that don’t belong to the current EP rank. The software stack has finally caught up with the model architecture.
Third, the economics.
Unsloth’s dynamic GGUFs put the lossless 8-bit build at 162 GB and a 3-bit build at 103 GB, needing roughly 110 GB of combined RAM plus VRAM. That’s one well-specced workstation. Not a cluster. A workstation.
This aligns perfectly with the broader pattern we’ve covered before: small language models are enabling an edge AI revolution, and training models on a single GPU is becoming more feasible by the quarter.
The Limits You Should Know About
I’m not going to sell you a fantasy. There are real constraints on single-GPU deployment.
Memory residency. Even though only 13B parameters activate per token, every expert stays resident in memory. The GPU needs to hold ~160 GB of weights regardless of how many experts actually fire. The DeepInfra deployment notes make clear this is an efficiency-focused model, not a small model.
Context window tradeoffs. The 1M-token context window exists on paper, but single-GPU serving at that context length requires the full 192GB of memory dedicated to KV cache. The vLLM recipe for MI325X only validates up to 4K context on a single card. For million-token workloads, you still need multi-GPU disaggregated serving with KV cache transfer, the vLLM recipe shows this on H200 with MooncakeConnector or NixlConnector over RDMA.
Agentic reasoning still needs compute. The performance gains on benchmarks like Terminal Bench and DeepSWE came from post-training on the 0731 checkpoint. That post-training pipeline involved domain-specific expert cultivation through SFT and RL with GRPO, followed by on-policy distillation, compute that’s far beyond a single GPU.
Harness sensitivity. All the headline agentic benchmark numbers are DeepSeek-reported on their unreleased DeepSeek Harness. As the MarktechPost analysis notes, agent scores are harness-sensitive, so independent runs may diverge. Run your own evals before betting your architecture on it.
The Cloud vs. Edge Question Gets More Interesting
None of this means cloud AI is dead. It means the calculus changes.
For bursty, unpredictable workloads, the cloud still wins on elasticity. For high-throughput, consistent loads, especially reasoning-heavy agentic workloads, single-GPU serving becomes increasingly attractive. The economics are simply different when your marginal inference cost drops to the amortized cost of a GPU you already own.
This is the same trajectory we’ve seen play out in the distillation wars, where tiny models humiliate frontier LLMs. The pattern keeps repeating: innovation at the efficiency frontier, not just the scale frontier.
If you’re wondering whether you can still afford to run LLMs locally, the V4-Flash answer is: increasingly, yes, if you pick the right card.
The Bottom Line
DeepSeek V4 Flash on a single AMD MI300X isn’t just another model release. It’s evidence that the AI industry’s default assumption, that frontier intelligence requires frontier infrastructure, is a choice, not a law of physics.
The architectural decisions DeepSeek made, MoE with aggressive sparsity, hybrid compressed attention, FP4 expert quantization, speculative decoding, manifold-constrained hyper-connections, compound into a system that collapses the infrastructure footprint of frontier-class AI by an order of magnitude.
And the fact that a community contributor got it running on one MI300X card, with a working vLLM recipe and documented verification, means this isn’t theoretical. It’s deployable, today, on hardware that’s a fraction of the cost of a GPU cluster.
The question now isn’t “can it run on one GPU?” It’s “why did we assume it couldn’t?”
