The CUDA monopoly narrative just took a hit. A developer going by sayamss posted benchmark results showing Qwen3.8-27B in FP8 running on four Radeon AI PRO R9700 GPUs, hitting an aggregate prefill of 7,025 tokens per second and a decode rate of 813.9 tok/s with 8 concurrent requests. The numbers are spicy enough that the Reddit thread’s top comment was literally just “What the fuck?”
Here’s the kicker: every GPU was capped at 210W. All four cards fit into a standard 15-amp wall outlet. No data-center power infrastructure, no liquid cooling, no 4U chassis. Just a workstation with four workstation cards, doing what used to require a rack.
The Numbers That Should Make NVIDIA Nervous
Let’s break down what this actually means. The setup was a full custom inference stack: an engine, on-device training, and routing. The test ran with BF16 KV cache and MTP (multi-token prediction) enabled but no speculative decoding.
| Metric | Mean | P50 | P90 | P95 | P99 |
|---|---|---|---|---|---|
| TTFT | 4.605s | 4.674s | 4.682s | 4.683s | 4.690s |
| TPOT | 9.93ms | 10.00ms | 10.97ms | 11.15ms | 11.62ms |
| End-to-end | 9.677s | 9.726s | 10.102s | 10.227s | 10.550s |
The TPOT (time per output token) of ~10ms translates to roughly 100 tokens per second per session. That’s the number that matters for interactive use, and it’s solid. The aggregate throughput numbers are what make people pause.
Here’s the full model lineup this rig handled:
| Model Profile | Quant | KV Type | Users | Prefill | Peak Decode at 4K | Context Window |
|---|---|---|---|---|---|---|
| Qwen3.8-27B MXFP4 | MXFP4 | BF16 | 16 | 9,830.96 tok/s | 1,426.02 tok/s | 131,072 tokens |
| Qwen3.8 Next Flash Int4/FP8 | Int4/FP8 | BF16 | 16 | 7,243 tok/s | 849 tok/s | 131,072 tokens |
| Qwen3.8-27B FP8 | FP8 | BF16 | 8 | 7,025 tok/s | 813.9 tok/s | 139,264 tokens |
| DeepSeek V4.0 Flash 0731 | native | FP8 | 4 | 2,268 tok/s | 105.99 tok/s | 262,144 tokens |
Note the MXFP4 configuration hitting 9,830 tok/s prefill with 16 concurrent users. That’s not a typo.
The Skeptic’s Guide to Aggregate Numbers
Before we crown AMD king of inference, let’s address the elephant in the thread. Several commenters pointed out that “aggregate” throughput can be misleading if you’re not careful about the test conditions. The 813 tok/s decode figure came from 8 parallel requests, each running at about 100 tok/s per session. That’s honest math.
The more interesting challenge came from someone questioning whether the VRAM cache could even hold 8 full 130K-token contexts. The developer’s response? Each session has a 130K context window, and the tests ran at a 4K prompt. That’s a realistic workload for a small team serving coding agents, not a synthetic worst-case benchmark designed to flatter the hardware.
The system is designed for exactly that: small teams. The developer builds and sells these machines to enterprises, priced at roughly $18K depending on RAM configuration. The board is an Asus SAGE TRX50, which matters because PCIe lanes and bandwidth are the real differentiator here.
Why This Is Actually Possible Now
The R9700 specs tell the story. It’s an RDNA 4 card with 32 GB GDDR6, 640 GB/s bandwidth, and FP8 matrix throughput of 383 TFLOPS (766 with sparsity). On paper, that’s modest compared to an RTX 5090’s 1,792 GB/s bandwidth. And yet, the community was rightly confused: one commenter wondered how the R9700 manages this when its VRAM bandwidth is two-thirds that of a 3090.
The answer is that raw bandwidth isn’t the only variable. The software stack has been catching up in a big way. ROCm 10.1 fixed GPU resets and crashes during training on these cards. vLLM 0.32.0 added official support via prebuilt AMD wheels, though AWQ and GPTQ remain unsupported. Tools like Strata now list the R9700 as a supported target. The ecosystem isn’t just usable anymore, it’s becoming genuinely competitive.
There’s also a dedicated SGLang repository running FP8 inference of Qwen3.8-27B on dual R9700s with 82 local RDNA4 patches, focused on long-context (256K) coding and agentic workloads. The developer in our thread built their own stack. That’s the pattern: people aren’t waiting for official support anymore, they’re writing their own.
The Cost Question Nobody’s Asking Correctly
The hardware for the 4-card rig costs about $18K. That sounds like a lot until you price the NVIDIA equivalent. An RTX PRO 6000 Blackwell Workstation goes for $16,000 per card on NVIDIA’s marketplace, when it’s in stock. It has three times the memory and 2.8 times the bandwidth, but at more than twelve times the R9700’s $1,299 launch price.
A single RTX 5090 costs $1,999 and can’t hold a Qwen3.8-27B in FP8 with meaningful context. Two R9700s at $2,598 combined can serve FP8 models with long context, as Puget Systems demonstrated with their dual-card benchmark. Four R9700s at $5,196 plus the necessary workstation hardware gets you a machine that handles 8 concurrent users at ~100 tok/s each.
Compare that to what a single enterprise GPU costs. The economics of small-team inference are shifting, and NVIDIA doesn’t have a great answer at this price point.
The Software Gap Is Closing, But Not Closed
Let’s not pretend everything is rosy. The VDF AI deep-dive on the R9700 makes clear that prompt processing is where NVIDIA still pulls away. A community benchmark showed an RTX 5090 reading prompts 2.6 to 3.4 times faster than the R9700 as context grew. The paper numbers look close on paper, AMD lists 191 dense FP16 TFLOPS against the RTX 5090’s 209.5, but real-world prompt throughput tells a different story.
Also, multi-GPU vLLM hangs under KVM/VFIO passthrough and needs bare metal. Containers need specific device mappings. The ecosystem is usable, but it’s not turnkey.
ROCm vs Vulkan is also still a genuine choice. Phoronix found Vulkan generates tokens slightly faster while ROCm delivers the first token sooner. The correct answer is to benchmark both on your workload.
What This Means for Teams Building Local Inference
If you’re running a team of 5-15 people who need a capable coding model with long context, this class of hardware is now genuinely viable. You can serve real-world performance of Qwen 3.8 27B that rivals cloud providers without the monthly subscription treadmill.
The practical implications extend to autonomous agents and local tool-calling workloads where context windows and token throughput directly impact how many parallel agent runs you can sustain. This isn’t just about chat, it’s about serving actual production workloads.
There are caveats. The tuned frameworks achieving these numbers are custom forks, not stock software. The performance optimization patterns for Qwen3.8 that work on Radeon hardware aren’t documented in vendor whitepapers. You have to be willing to experiment.
The Bottom Line
AMD’s Radeon AI PRO R9700 isn’t going to replace NVIDIA in data centers. That’s not the point. The point is that the “NVIDIA or nothing” assumption for local AI infrastructure is now demonstrably false.
For roughly $18K, you can build a four-card inference server that runs Qwen3.8-27B in FP8 at 7K tok/s prefill, serves 8 concurrent users at 100 tok/s each, and plugs into a standard wall outlet. That’s not a paper benchmark from a vendor lab, it’s a production machine that a developer sells to enterprise customers.
The “revolutionary” part isn’t the hardware specs. It’s that this is a $1,299 workstation card, not a $30K data-center part, doing work that would have required enterprise infrastructure two years ago. The AI hardware monopoly narrative was already cracking. These numbers are the sound of it breaking.
For teams evaluating whether to build local inference, the Qwen3.8-27B configurations that fit on 16GB VRAM prove the model itself is efficient, but the R9700 rig shows what happens when you give it real hardware. The answer is: a lot more than most people assumed.
