Local Inference Just Got a Whole Lot Faster
Here’s the thing about speculative decoding: it’s been the theoretical answer to slow local LLM inference for years. “Just use a small draft model to guess tokens, then verify in batches!” Except the draft models were never good enough, the overhead ate the gains, and you’d end up with a 1.05x speedup that wasn’t worth the setup pain.
Then MTP (Multi-Token Prediction) changed the game. Instead of a separate, smaller draft model, the main model ships with its own dedicated prediction head that’s trained to guess the next few tokens. No separate model to load. No quality mismatch. Just a specialized module that knows exactly how the main model thinks.
And now, for the first time, MTP is available for Qwen3.8-Flash-Next in GGUF format. Not just “available”, the numbers are genuinely absurd. On a single B200 with the UD-Q4_K_XL quant, throughput jumps from 83.2 tok/s to 138.8 tok/s. That’s a 1.67x improvement. No quality loss. No sampling changes. Just the draft head doing what it was trained to do, verified exactly by the main model.
Why MTP Beat Traditional Speculative Decoding
The always-question: “Why didn’t we do this before?” The answer is more nuanced than “nobody thought of it.”
Classic speculative decoding pairs a large target model with a tiny draft model. The draft model, something like a 1B parameter LM, proposes tokens, and the target verifies them in parallel. It works, but there’s an inherent mismatch. The draft model’s understanding of language is fundamentally different from the target’s. It guesses wrong more often, and every wrong guess is wasted compute.
MTP eliminates that by embedding the prediction head inside the model architecture. For Qwen3.8-Flash-Next, that’s a single extra layer trained with multi-step objectives specifically to predict subsequent tokens. It’s not a different model trying to imitate a smarter one, it’s the same model’s weights, extended with a module that knows exactly what comes next.
The verification process is exact, meaning output quality is bit-for-bit identical to running without MTP. No “sometimes the output quality degrades slightly” caveats. The draft head guesses, the main model verifies every token, and only accepted tokens make it into the output.
The MTP README on Hugging Face is refreshingly honest about the mechanics: “Verification is exact, so the output is unchanged, only the speed.”
The Numbers: What 1.67x Actually Feels Like
Let’s put those benchmarks in context. From the measured results on a single B200 with the shared-Q8_0 head:
| Main model quant | MTP off | MTP on | Speedup |
|---|---|---|---|
UD-Q4_K_XL |
83.2 tok/s | 138.8 tok/s | 1.67x |
UD-IQ1_S |
90.1 tok/s | 120.9 tok/s | 1.34x |
A note on those numbers: they’re greedy generation, the default for most coding and structured output tasks. The README warns that “higher temperature makes the target less predictable, so fewer guesses are accepted and the speedup shrinks.” That’s the honest engineering answer to “does this work everywhere?” No. But for the workloads that matter, agentic coding, structured generation, function calling, greedy generation is exactly what you want.
Community benchmarks on the ggml-org/llama.cpp pull request #28123 confirm the improvements, though the earlier builds showed MTP could actually hurt performance. The pre-optimization numbers tell the story: before a critical fix, MTP was running at 83 tok/s on prose vs 108 tok/s without drafting at all. After the optimization: 183 tok/s code, 144 tok/s prose.
That’s the difference between “MTP is a gimmick” and “MTP is the biggest local inference win since quantization.”
The Dirty Secret: It Doesn’t Work on Mainline llama.cpp
Here’s the catch that’ll bite you if you’re not paying attention. These MTP files do nothing on standard llama.cpp builds.
As of upstream commit 0eadefebd, there’s no MTP graph for the qwen4exp architecture, no cross-model tensor borrowing, and no --spec-type draft-mtp option. If you download the draft head, pass it to a mainline build, and expect speed, you’ll get silence. The MTP tensors are simply ignored.
You need one of:
- Prebuilt binaries from unslothai/llama.cpp releases, tag
b10715-mix-86bd2d3or newer - Build from source using unslothai/llama.cpp pull request #144
Building from source isn’t bad. The PR is clean and well-structured:
git clone https://github.com/unslothai/llama.cpp && cd llama.cpp
git fetch origin pull/144/head:mtp && git checkout mtp
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON # omit -DGGML_CUDA for CPU
cmake --build build -j
The good news: the PR includes significant optimizations beyond just MTP support. The CUDA graph cache was reworked to key by shape rather than first node, which fixes a real performance bug, “a captured cuda graph hard-codes the shapes, so a caller that alternates shapes, a speculative verify batch, for example, needs a separate instance per shape.” This matters more than most people realize for speculative decoding workloads that alternate between draft and verify batch sizes.
Which File Should You Download?
The README is refreshingly direct. Use mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf (2.60 GB). Here’s the full comparison:
| File | Size | Notes |
|---|---|---|
shared-Q8_0 |
2.60 GB | Recommended. Best speed/quality tradeoff |
shared-Q4_K_M |
1.78 GB | Smaller, ~2 points less acceptance |
shared-BF16 |
4.87 GB | Bigger and slower than Q8_0 |
Q8_0 / Q4_K_M / BF16 |
3.85 / 2.60 / 7.24 GB | Self-contained variants with own tensor copies |
The shared- variants are the interesting innovation. They borrow the token embedding and output projection from the main model instead of carrying their own copies, saving about 1.3 GB per head. The README notes they “draft identically to the self-contained files”, so unless you’re on a build without borrowing support, there’s no reason to grab the bigger files.
One counterintuitive finding: BF16 is bigger and slower. The draft step is dominated by the output projection, which is cheaper to execute at 8 bits. Precision doesn’t buy you acceptance when the head is making quick multi-token guesses that get verified anyway.
Acceptance rates by quant:
| Head | Acceptance Rate |
|---|---|
shared-BF16 |
66.5% |
shared-Q8_0 |
66.1% |
shared-Q4_K_M |
64.4% |
The delta between Q8_0 and BF16 is 0.4 percentage points. Not worth 2+ GB of extra load time.
Running It: The Exact Command
llama-cli \
-m Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
-md mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf \
--spec-type draft-mtp --spec-draft-n-max 2 -ngl 999
The --spec-draft-n-max 2 flag is worth understanding. This is how many tokens the draft head tries to predict ahead. Two is the sweet spot, the README notes “higher drafts more but each guess is accepted less often.” At a draft length of 3+, you’re generating tokens that have low probability of acceptance, wasting compute on the draft path.
For server deployments, the same flags work with llama-server. The quimmedes/Qwen3.8-Flash-Next-MTP-GGUF repo shows a production example with additional optimization flags.
How to verify it’s actually running, check the log for:
draft acceptance = 0.66139 (325 accepted / 491 generated), mean len = 2.76
If you don’t see this line, speculation is off, and you’re probably on a mainline build missing the MTP support.
The Shared-Tensor Architecture Is the Real Innovation
The PR contains a genuinely clever piece of engineering that deserves more attention than the headline speed numbers.
For a draft head to work, it needs the token embeddings and output projection layer. Traditionally, you’d include them in the draft GGUF file, self-contained but wasteful, since the main model already has a copy loaded.
The new --mtp-shared-embd flag changes this. The draft GGUF declares nextn_shared_target_tensors and simply… doesn’t include the embeddings, output norm, or LM head. At load time, the draft model’s loader calls borrow_shared_tensor() to grab them from the already-loaded target model.
// a draft head can leave out the embeddings and lm head and use the target's
mparams.model_shared = model_tgt;
The shape validation is strict, a draft that expects different dimensions from the target throws an error. And the loader adds a helpful error if you try to load a shared draft head standalone:
"this model is a draft head without its own '%s', load it as a draft of its target model, not on its own"
This is exactly the failure mode you want: explicit and immediate, rather than silent corruption. It’s the same philosophy behind the MTP head’s verification guarantee. “A mismatched pairing is rejected with an error rather than producing bad output.”
When Not to Use MTP
The README’s honesty here is a welcome change from the usual hype. Skip MTP for concurrent serving.
Measured at concurrency 8, MTP is a net loss, about 0.81x to 0.87x performance. The reason is elegant: “a busy model has little idle capacity for a draft to exploit.” Speculative decoding works when you have spare compute to run the draft head while the main model is between steps. Under load, that spare capacity disappears, and every draft token becomes pure overhead.
This aligns with broader Qwen3.5 performance measurements in local agentic coding environments, single-stream performance is where these models shine, and multi-user serving is a different engineering problem entirely.
What’s Still Coming
The MTP support in this release is significant, but the pipeline of improvements is still moving. The ggml-org/llama.cpp upstream PRs list ongoing work: KV-cache indexing for O(log n) ngram lookups, direct reads for the PLE table promising “>>2x prefill performance improvement on GB10”, and the port of NextN/MTP speculative decoding to master with “+50% decode at 70k” context.
For those keeping score, the Qwen3.6-27B MTP preservation efforts show that MTP support is becoming the new baseline expectation for local first model releases. A release without MTP is increasingly seen as a release that’s leaving performance on the table.
The Bottom Line
MTP for Qwen3.8-Flash-Next isn’t a marginal optimization. It’s the difference between an interactive coding assistant and a tool that feels like waiting for a remote API. At 138 tok/s on a single GPU with a 4-bit quant, local inference is genuinely approaching the responsiveness of cloud-hosted models.
The “MTP is so 2025” comment in the Reddit thread is premature, as the DSpark draft model work shows, even better speculative approaches are already in development. For now, the only question that matters: are you on a build that supports it?
