1000 Tokens Per Second on a 1T Model? Xiaomi Just Broke Physics (or At Least the Latency Barrier)

1. FP4 Quantization: Squeezing the Model Until It Fits

The first problem with a trillion-parameter model is simple physics: those parameters take up a lot of memory. Even at FP8, a 1T model requires roughly 1 TB of memory just for the weights. Moving that much data across the PCIe bus and through the memory hierarchy is the dominant bottleneck in inference.

The obvious answer is to use fewer bits per parameter. FP4 (MXFP4 format) cuts the memory footprint in half compared to FP8. The not-so-obvious answer is that applying FP4 uniformly across the entire model destroys accuracy, especially on complex reasoning and code generation tasks.

Xiaomi’s insight was beautifully pragmatic: don’t quantize everything. In a Mixture of Experts architecture, the experts account for the vast majority of parameters (and are the most tolerant to reduced precision). So they quantized only the MoE experts to FP4, leaving everything else, attention mechanisms, projections, the works, at higher precision.

The results are almost too good to believe. On SWE-Bench Verified, the FP4-quantized model actually improved over the FP8 baseline (77.4% vs 78.9%, a minor dip, but essentially on par). On Claw-Eval (pass@3), it jumped from 63.8 to 67.8. The quantization-aware training (QAT) process squeezed the model down without squeezing out its intelligence.

The key takeaway here: mixed-precision quantization isn’t just about saving bits. It’s about understanding which parts of your architecture are resilient to compression and which aren’t. The MoE experts, with their specialized knowledge distributions, turn out to be remarkably robust to 4-bit quantization.

2. DFlash Speculative Decoding: Breaking the Autoregressive Bottleneck

FP4 solved the memory bandwidth problem, but even with the weights loaded faster, autoregressive decoding still generates one token at a time. Each forward pass produces a single token, then you shift the window and do it again. At 1000 TPS, that means each token is generated in 1 millisecond, but the serial nature of the process means you’re constantly waiting on the next compute cycle.

Traditional speculative decoding tackles this by using a lightweight “draft” model to guess several tokens ahead, then having the large model verify them in parallel. The problem is that the draft model’s quality determines the acceptance rate, and improving the draft model increases its compute cost, a fundamental tension.

DFlash, the approach Xiaomi adopted, breaks this trade-off entirely. Instead of generating tokens autoregressively, the draft model fills an entire block of masked positions in a single forward pass. This eliminates the serial constraint of drafting entirely.

The real engineering magic, though, is how they optimized this for a 1T MoE model at scale:

• Sliding Window Attention (SWA): The draft model exclusively uses SWA, which aligns perfectly with the MiMo-V2 series architecture. This eliminates the dependency on full context prefixes, reducing per-prediction compute from linear-in-context-length to constant. Massive win for long-context scenarios.

• Localized training signals: During training, mask-signal sampling is pushed down to GPU-local shards. A single sequence can produce tens of thousands of independent training signals covering diverse context positions, all without cross-device communication.

The acceptance lengths tell the real story:

In coding scenarios, the model accepts 6-7 out of every 8 draft tokens. That means the large model verifies almost an entire block in one shot, effectively multiplying throughput by 6x in practice. The block size is capped at 8 to keep verification overhead low, but the acceptance rates mean those 8 slots are almost fully utilized.

3. TileRT: The Microsecond-Level Execution Engine

FP4 and DFlash are necessary conditions, but they aren’t sufficient. To hit 1000 TPS, you need an inference runtime that doesn’t waste a single microsecond.

At this frequency, each operator’s lifecycle is measured in microseconds. The traditional inference engine model, launch an operator, wait for synchronization, move data, launch the next operator, fractures the execution flow at exactly the wrong granularity. Those “operator boundaries” become the bottleneck, exposing visible “Execution Gaps” where the GPU sits idle.

TileRT’s solution is a paradigm shift in runtime architecture:

• Persistent Engine Kernel: Instead of launching operators one at a time, the entire compute pipeline remains resident in the GPU. While the current Tile is computing on Tensor Cores, subsequent data is already flowing through the memory hierarchy. Full-pipeline continuous prefetching, all the time.

• Warp Specialization: Different warps (thread groups) handle communication, data movement, and tensor computation independently but in precise coordination. The GPU becomes a continuously flowing heterogeneous execution system rather than a homogenous lock-stepped compute array.

This is deep hardware-software co-design. The TileRT team didn’t just optimize a runtime, they fundamentally rethought how a GPU’s execution model should work for low-latency inference. The result is an engine that can sustain 1000 TPS because it never waits for anything.