After more than a year in development, turboderp’s ExLlamaV3 hit v1.0.0 on July 14, 2026, and the benchmarks are genuinely shocking. We’re talking decode speeds that nearly doubled on certain configurations, a complete rewrite of the attention kernel, and the kind of optimization depth that makes you question why anyone building locally would bother with anything else.
If your entire local LLM workflow runs through llama.cpp or vLLM, it’s time to pay attention. The GPU-poor among us just got a major upgrade.
What Actually Changed (The Meat, Not the Hype)
Let’s cut through the release notes and focus on what matters. Turbo and Fable have been pulling ten-hour days, and the changelog reads like a manifesto:
- Removed flash-attention-2 and xformers dependencies, ExLlamaV3 is now fully self-contained. No more version hell between CUDA, flash-attn, and xformers.
- New attention kernel with online cache quantization, This isn’t incremental. It’s a fundamental rethinking of how attention handles KV cache memory.
- Greatly improved GEMM/GEMV performance on Ampere, RTX 3090 owners, this one’s for you.
- New INT8 GEMV kernel, Because not everyone needs FP16 precision for fast inference.
- New MoE kernel ticket scheduler, Mixture-of-Experts models just got smarter about routing computation.
- C++ and CUDA graph paths for all attention/GDN modules, This eliminates the CPU bottleneck entirely on modern systems.
- Extended tensor-parallel support to most models, including Gemma4.
The full release with charts is available on html.cafe, and the repository itself is on GitHub.
The Decode Numbers That Matter
The performance tables tell a story that’s hard to ignore. Here’s the RTX 5090 decode comparison between v0.0.43 and v1.0.0:
| Model / Bitrate | v0.0.43 | v1.0.0 (mul1) | Improvement |
|---|---|---|---|
| Qwen 3.5 0.8B 4.00 bpw | 261 tok/s | 545 tok/s | +109% |
| Llama 3.1 8B 2.00 bpw | 191 tok/s | 259 tok/s | +36% |
| Qwen 3.6 27B 3.00 bpw | 67 tok/s | 86 tok/s | +28% |
| Gemma 4 12B 3.50 bpw | 99 tok/s | 134 tok/s | +35% |
| Llama 3.2 1B 4.00 bpw | 536 tok/s | 623 tok/s | +16% |
The 109% gain on the 0.8B model isn’t a fluke, it’s a direct result of eliminating the CPU bottleneck via CUDA graphs. When your CPU can’t feed the GPU fast enough for small models, the entire pipeline stalls. ExLlamaV3 v1.0.0 doesn’t just optimize, it restructures the entire compute path.
Even on the aging RTX 3090 (Ampere architecture), the improvements are dramatic:
| Model / Bitrate | v0.0.43 | v1.0.0 (mul1) | Improvement |
|---|---|---|---|
| Qwen 3.5 0.8B 4.00 bpw | 268 tok/s | 444 tok/s | +66% |
| Qwen 3.6 27B 3.00 bpw | 29 tok/s | 50 tok/s | +72% |
| Llama 3.1 8B 2.00 bpw | 94 tok/s | 146 tok/s | +55% |
That 72% gain on the 27B Qwen model at 3.00 bpw is transformative. What was previously a borderline unusable experience now feels snappy.
The New Attention Kernel: Online Cache Quantization Isn’t a Gimmick
The headline feature here is the new attention kernel with online cache quantization. In plain English: ExLlamaV3 now quantizes the KV cache on-the-fly during attention computation rather than staging temporary FP16 cache tensors. This means cache quantization no longer slows down inference.
The impact is visible across every context length tested. On a Qwen 3.6 27B model at 3.00 bpw running on an RTX 5090, the FP16 cache baseline drops off a cliff as context grows. The K6/V4 and K8/V8 quantized cache variants maintain far higher throughput:
- At 32K context, FP16 decode drops to roughly 35 tok/s
- K6/V4 cache maintains around 50 tok/s
- K8/V8 cache holds at approximately 55 tok/s
This is the kind of optimization that makes long-context inference viable on consumer hardware. For anyone building RAG pipelines or working with large document contexts, this changes the calculus significantly. If you’re evaluating whether Apple Silicon M-series chips are viable for local LLM inference, these numbers put NVIDIA’s advantage in stark relief.
Dropping Dependencies: A Bigger Deal Than It Sounds
“Removed flash-attention-2 and xformers dependencies” sounds like a minor housekeeping note. It’s not.
Flash-attention-2 has been a pain point for local LLM deployers since its inception. It requires specific CUDA versions, has compatibility issues with different PyTorch builds, and frequently breaks when upgrading either framework. By building a native attention kernel that doesn’t rely on these external packages, ExLlamaV3 eliminates an entire class of deployment headaches.
The project now ships as a single-step converter: give it a Hugging Face checkpoint and a target bits-per-weight, and it produces a quantized model without a separate calibration pass for most model sizes. This is a stark contrast to the multi-step workflows required by GPTQ or the calibration-aware quantization in llama.cpp.
What This Means for the GPU-Poor
The cost of running LLMs locally has always been a numbers game. The real cost of investing in local inference hardware often doesn’t break even for years, especially when you factor in electricity, depreciation, and the fact that your $20,000 rig competes with a $200/month cloud API subscription.
ExLlamaV3 doesn’t solve the hardware cost problem, but it dramatically improves the performance-per-dollar equation. A 3-bit-per-weight EXL3 quantization with a 4096-token cache can run in well under 16 GB of VRAM. That means an RTX 3090 (which can be found used for under $1,000) can run a 70B model at reasonable speeds, something that was firmly in datacenter territory just two years ago.
The catch? ExLlamaV3 is NVIDIA-only, and it doesn’t support CPU offloading like llama.cpp. Everything must fit in your GPU memory. For users with AMD cards or limited VRAM, the llama.cpp ecosystem remains the better choice. But if you’re rocking an RTX 3090, 4090, or 5090, ExLlamaV3 v1.0.0 is likely the best inference engine available today.
Quantization Quality: Why EXL3 Stands Apart
The EXL3 format is a streamlined derivative of the QTIP method developed at Cornell’s RelaxML lab. It computes Hessians on the fly and uses a fused Viterbi kernel so a model can be converted to a target bitrate in one pass. The result is quantization that maintains remarkably low divergence from the baseline model.
Independent analysis suggests that 5 bpw in EXL3 is effectively lossless, comparable to Q8 in llama.cpp. Even at 2.5-3 bpw, the models remain reasonably coherent, which isn’t true for most quantization methods at those extreme compression levels. This is why the community often recommends EXL3 when the question is “what model + quantization gets me the most value out of my VRAM?”
The new INT8 GEMV kernel adds even more precision options for users who don’t need the extreme compression of 3 bpw but want faster inference than FP16 provides. The mul1 codebook (now the default) consistently outperforms the mcg codebook across every benchmark, often by significant margins.
New Model Support: GPT-OSS and NemotronH
Two notable additions to the architecture support list:
GptOssForCausalLM adds support for OpenAI’s first open-weight release since GPT-2. The gpt-oss-20b model is a 21B-parameter mixture-of-experts model with only about 3.6B active parameters per token. A Q4_K_M build sits at around 12GB, meaning it can run on a 24GB card alongside a browser and IDE without VRAM contention. The larger 120B sibling still requires datacenter hardware.
NemotronHForCausalLM brings support for NVIDIA’s Nemotron-H models, which use a Mamba2 module for state-space-based sequence modeling alongside traditional attention. This hybrid architecture approach is increasingly common, and ExLlamaV3’s support puts it ahead of some competing backends.
The architecture support table now covers 30+ model families, with tensor-parallel support extended to most of them. Gemma4, Qwen 3.5/3.6, and the latest Mistral variants all work out of the box.
The Verdict: Should You Switch?
If you’re running a single NVIDIA GPU for local LLM inference and you haven’t tried ExLlamaV3, v1.0.0 is the moment to make the jump. The performance improvements are not theoretical, they’re visible in every benchmark, across every model size, at every quantization level.
The downsides remain: no CPU offloading, NVIDIA-only, and a smaller ecosystem of supporting tools compared to llama.cpp’s vast community. Tool calling via TabbyAPI can be less reliable than production-hardened alternatives. New model architectures take longer to receive support.
But for the specific use case of “I have X VRAM on an NVIDIA GPU, and I want maximum performance from quantized models”, ExLlamaV3 is now the definitive answer. The gap between v0.0.43 and v1.0.0 isn’t incremental, it’s generational.
Turbo and Fable have delivered something remarkable. The local LLM community owes them more than a GitHub star.
Join the ExLlama Discord for ongoing discussion, or check out the v1.0.0 release page for the full changelog and download links. Your RTX 3090 is waiting for its upgrade.