GGUF Just Got a PyTorch Upgrade: Transformers Now Runs llama.cpp Quants Natively
For the past year, local AI enthusiasts have lived in a world of forced choices. Want blazing-fast inference on your MacBook? You reach for llama.cpp, Ollama, or LM Studio. Want to debug, fine-tune, or actually inspect what the model is doing? You’re stuck with PyTorch and the transformers library, which meant downloading separate, dequantized checkpoints that eat your RAM for breakfast.
That wall just crumbled.
Hugging Face has landed native GGUF support in transformers, and it’s not just a token nod to compatibility. They’ve gone deep: reusing ggml’s Metal kernels, optimizing the generation loop, and hitting performance numbers that make llama.cpp sweat. The announcement dropped on the Hugging Face blog and the local AI community collectively lost its mind.
What Actually Changed
The headline feature is deceptively simple: you can now load a GGUF file directly through the standard transformers API. No conversion scripts, no separate runtime, no “please install llama-cpp-python and pray.”
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "unsloth/Qwen3.5-4B-GGUF"
filename = "Qwen3.5-4B-Q4_K_M.gguf"
model = AutoModelForCausalLM.from_pretrained(
model_id,
gguf_file=filename,
)
That’s it. One extra parameter. Everything after that, tokenization, generation, chat templates, logits processors, is the standard transformers API you already know.
But here’s where it gets interesting: this isn’t just “we’ll dequantize the weights and load them into PyTorch.” That would be the lazy path, and it would defeat the entire purpose of quantization. Instead, Hugging Face integrated the actual ggml kernels through their new kernels library, allowing the model to run directly from its packed quantized weights on Apple Silicon.
The Performance Numbers That Matter
Let’s talk benchmarks, because that’s where this either proves itself or collapses. Hugging Face tested three checkpoints on a MacBook Pro M2 Max: a small dense model, a larger dense model, and a mixture-of-experts model. The results are remarkably close to llama.cpp:
| Model | Transformers (tok/s) | llama.cpp (tok/s) |
|---|---|---|
| Qwen3.5-4B Q4_K_M | 70.4 | 71.8 |
| Qwen3.8-27B UD-Q4_K_M | 15.9 | 13.4 |
| Qwen3.5-35B-A3B UD-IQ4_XS | 60.2 | 61.3 |
The 27B model actually beats llama.cpp, by 18%. That’s not a rounding error, that’s the transformers team optimizing the generation loop hard enough to overcome llama.cpp’s runtime advantages.
It’s worth noting the benchmark methodology: transformers includes prefill in its measurement while llama-bench reports decode-only throughput, so the comparison isn’t perfectly apples-to-apples. But even being in the same ballpark is a massive achievement for a general-purpose library.
The Kernel Strategy: Stealing the Good Parts
The real technical meat is in how they made this fast. Rather than reinventing the wheel, Hugging Face’s team did something smarter: they took ggml’s Metal kernels and wrapped them for PyTorch consumption.
| Kernel | Purpose |
|---|---|
ggml-quantization |
Reads packed quantized weights for matrix ops, including MoE expert selection |
ggml-norm |
Fused normalization, including Qwen3.5’s zero-centered RMSNorm |
ggml-attn |
ggml’s Metal flash attention for prompt processing and decoding |
ggml-gated-delta-net |
Accelerates linear-attention layers in hybrid architectures |
topk |
Custom Metal implementation for MoE routing (softmax + top-k) |
This matters beyond GGUF files. A kernel operates on tensors, it doesn’t care where those tensors came from. The same building blocks could accelerate any transformers model that uses similar operations, even without quantization. That’s the bigger opportunity: bringing ggml’s performance to architectures that llama.cpp doesn’t support, and never will.
If the model can’t fetch a compatible kernel, it falls back to dequantizing on-the-fly with a warning. Still more memory-efficient than loading a full-precision checkpoint, but not the optimal path. On Apple Silicon with the right kernels, the weights stay packed on Metal and the speed difference is dramatic.
The Generation Loop Gets an Overhaul Too
Kernels are only half the story. The transformers team also identified that generate was leaving performance on the table through unnecessary CPU-GPU synchronization. Two PRs address this:
Dropping the attention mask early (#48814): For decoder-only inputs without padding, the all-ones mask is removed at the start. Downstream attention code stops repeatedly checking whether it can be skipped.
Deferring the stopping check (#47975): The stopping decision is copied asynchronously and consumed on the next step, letting the CPU keep scheduling while the GPU runs.
These changes benefit all transformers models, not just GGUF ones. The team’s philosophy here is solid: kernels reduce the cost of individual operations, while fewer sync points let CPU scheduling and GPU execution overlap. Together, they close most of the gap with llama.cpp.
What This Unlocks: Beyond Just Faster Local Chats
The performance is nice, but the real value is capability. Here’s what native GGUF support in transformers enables:
Debugging and introspection. You can attach hooks to intermediate activations, inspect tensors mid-forward-pass, and actually understand what a quantized model is doing. llama.cpp is a black box by design, PyTorch is the opposite.
Fine-tuning from GGUF checkpoints. This is potentially huge. Training frameworks like Unsloth and Axolotl are built on transformers. With GgufConfig(dequantize=True), you can load a GGUF file, dequantize it, and continue training:
import torch
from transformers import AutoModelForCausalLM, GgufConfig
model = AutoModelForCausalLM.from_pretrained(
"unsloth/Qwen3.5-4B-GGUF",
gguf_file="Qwen3.5-4B-Q4_K_M.gguf",
quantization_config=GgufConfig(dequantize=True),
dtype=torch.bfloat16,
)
The community is already exploring LoRA training over GGUF. This is interesting for memory-constrained setups, bitsandbytes 4-bit still doesn’t support MoE architectures, but GGUF quantization handles them natively.
Validation and conversion checking. For the Hugging Face team themselves, being able to load the original checkpoint and its GGUF conversion in the same framework makes it easier to verify that conversions preserve the intended behavior.
Custom generation logic. Need a custom logits processor, a bespoke stopping criteria, or a fully custom generation loop? That’s trivial in transformers and nearly impossible in llama.cpp.
Evaluation workflows. Existing evaluation pipelines built on transformers can now measure quantized models with the same harness they use for full-precision checkpoints.
Debugging & Introspection
Hooks, tensor inspection, full transparency.
Fine-tuning from GGUF
Load quantized checkpoints for LoRA and more.
Validation & Conversion
Verify GGUF conversions seamlessly.
Custom Generation
Tailored logits, stopping, and loops.
Serving: OpenAI-Compatible API in One Command
For those who want a proper interface, transformers serve exposes an OpenAI-compatible API:
pip install -U "transformers[serving] @ git+https://github.com/huggingface/transformers.git" kernels
transformers serve "unsloth/Qwen3.5-4B-GGUF:Qwen3.5-4B-Q4_K_M.gguf"
The model argument syntax is <model_id>:<filename>.gguf, which lets you pick a specific quantization from a repository that might contain several. Then point Jan or Pi at http://localhost:8000/v1 and you’ve got a local AI assistant with a nice UI, powered by transformers.
The Honest Limitations
Let’s not get carried away. The initial release has real constraints:
- Apple Silicon only for the packed-inference path. The MPS-only support means Linux GPU users and Windows folks are stuck with dequantization fallback for now.
- Qwen3.5 architecture only (plus compatible Qwen3.8 checkpoints). Other architectures require additional kernel integration work, though the team says it’s “relatively straightforward.”
- No padding or batching optimizations yet. Unpadded inputs benefit from the mask optimization, but padded batches can have noticeably lower performance.
- Single interactive conversation is the target. This isn’t a production serving solution yet.
And crucially: Hugging Face explicitly states that llama.cpp remains the recommended engine for maximum local inference performance. This integration isn’t meant to replace it, it’s for when you need PyTorch’s flexibility without sacrificing quantized efficiency.
That said, for the developers who want to experiment, evaluate, and build on top of GGUF models, this is a game-changer.
The Training Angle Nobody Saw Coming
The most interesting downstream effect might be in training. A proof-of-concept already exists for training LoRA adapters over GGUF weights, which could save significant memory compared to bitsandbytes 4-bit quantization. The community is optimistic that Unsloth will extend its tooling to support this natively.
There’s also synergy with tools like Heretic, which does “model surgery” on transformers models. The ARA-LoRA modifications become more accessible when quantized weights can be loaded directly.
What This Means for the Ecosystem
This move represents a strategic bet by Hugging Face. They’ve been consolidating their position in the open-source AI stack, and the ggml acquisition was a major piece of that. Native GGUF support in transformers completes the circle: llama.cpp’s file format, ggml’s kernels, transformers’ API, and the Hub’s distribution all working together.
The architecture improvements from Transformers v5, those 6x to 11x MoE speedups, compound with this work. Combined with the open-source voice AI pipeline, Hugging Face is building toward a unified local AI platform where every modality runs efficiently on consumer hardware.
The Bottom Line
GGUF support in transformers is a big deal because it removes a fundamental tradeoff. You no longer have to choose between speed and flexibility, between quantization and debuggability, between llama.cpp and PyTorch.
The initial release is scoped, Apple Silicon, Qwen3.5, single conversation, but the trajectory is clear. As kernel coverage expands to more architectures and hardware, the distinction between “local inference runtime” and “research framework” will keep blurring.
If you’re developing local AI applications, now is the time to experiment with GGUF checkpoints in transformers. Start with Q4_K_M, evaluate on your actual use case, and see whether the flexibility costs you anything on your hardware.
The Transformers v5 speedups already made the library dramatically faster for MoE models. Native GGUF support removes the quantization barrier. Together, they might just make transformers the default choice for local inference, not just for research, but for production.
The only question left is whether llama.cpp can keep up.
