The 284B Model on a 32GB GPU Trick: How FreeToken Proves Hardware Is a Software Problem

Let’s get the elephant out of the room: a 284-billion-parameter model does not fit inside 32 GB of VRAM. The math simply doesn’t work. At FP4 precision, that’s roughly 140 GB of weights, still four times what your gaming GPU can hold.

Yet researchers from UC Berkeley and UT Austin are running 753B GLM-5.2 on a single workstation card at 14.9 tokens per second. Not through quantum magic. Not through lossy compression tricks. Through something far more interesting: an inference engine called FreeToken that treats your entire computer, CPU, GPU, RAM, and VRAM, as one unified, elastic inference machine.

The “revolutionary” part isn’t that it offloads weights to system memory. That trick is older than llama.cpp. What FreeToken does is smarter: it performs a real-time cost-benefit analysis for every single token to decide whether it’s faster to ship a missing expert across the PCIe bus to the GPU, or just compute it on the CPU and merge the results. Sometimes the CPU wins. Often, it does.

Here’s how the dance actually works.

Sparsity Is a Lie (But a Useful One)

Mixture-of-Experts models don’t work like dense transformers. A dense model like Llama-3-70B activates every parameter for every token, 70B weights, 70B computations, no shortcuts. MoE models are different: they contain hundreds of small feed-forward networks called “experts”, and a tiny router module selects only a handful for each token.

DeepSeek-V4-Flash, the 284B model that started this conversation, activates 6 of 256 routed experts per MoE layer. Across 43 layers, that’s roughly 13 billion active parameters out of 284 billion total. Only 4.5% of the model does work for any given token.

This sparsity makes local inference arithmetically feasible. It does not make it practically feasible. Here’s the catch that everyone glosses over: those 271 inactive experts still occupy ~127 GB of memory. They don’t go away. They just sit there, waiting for the router to call them.

Where do they wait?

That’s the architectural question that separates FreeToken from its predecessors. Traditional engines like llama.cpp assign MoE tensors at load time and pray. KTransformers pins a “hot” subset of experts to VRAM and hopes the router behaves. Both approaches break down in practice because the router doesn’t care about your cache placement strategy, it shifts which experts it selects on every single token.

The Three Failure Modes of Existing Engines

The research team behind FreeToken identified three systematic failures in current MoE inference systems:

Prefill destroys sparsity. During the prefill phase, when the model processes your entire prompt at once, thousands of tokens per layer route to nearly the whole expert set. The model needs almost every expert, which means the entire pool streams across PCIe. That’s roughly two seconds on an RTX 5090, five seconds on PCIe 4.0 desktops, and ten or more seconds on the x8 PCIe links common in laptops.

Static placement misses decode traffic. During token-by-token generation, the router’s selections shift constantly. A placement strategy frozen at load time becomes stale within seconds. Most expert evaluations end up on the CPU while the GPU and PCIe link sit idle.

Consumer CPUs can’t carry the load. Dual-channel DDR5 delivers 80, 90 GB/s of memory bandwidth. An RTX 4090 draws 1, 1.8 TB/s from its on-package memory. The gap isn’t a factor of two, it’s an order of magnitude. When you offload everything to the CPU, you trade a fast GPU bottleneck for a slow CPU bottleneck.

These failure modes explain why existing engines produce genuinely terrible numbers. In MarkTechPost’s benchmarks, worst-case time-to-first-token for llama.cpp hits 232 seconds. Ollama: 179 seconds. KTransformers: 946 seconds. At those latencies, agentic clients have already timed out and moved on.

The q* Policy: Math Instead of Guesses

FreeToken’s core insight is that expert cache misses don’t require a single response strategy. When a selected expert isn’t in VRAM, the engine has two options:

1. Copy the weights across PCIe and execute on the GPU
2. Leave the weights in system RAM and execute on the CPU

The GPU is faster at the actual computation. But that computation has to wait for the weights to arrive. Meanwhile, the CPU can start computing immediately from system RAM. For a small expert, the CPU might finish before the transfer-and-GPU path even starts.

The decision comes down to a quantity FreeToken calls q*. Since DMA transfers and CPU expert computations both read from the same host-memory subsystem, a saturated PCIe link leaves residual bandwidth of B_H - B_P (host bandwidth minus PCIe bandwidth). For each step with m cache misses, FreeToken calculates:

q* ≈ m × B_P / B_H

This gives the number of experts to fill into the GPU cache versus compute on CPU. The two partial sums merge exactly after the layer completes, no approximation, no router modification, no precision relaxation.

The critical detail: both bandwidths are measured on your specific machine, not assumed from spec sheets. The team’s profiling shows why this matters. On an RTX 5090 server, the measured ratio B_P:B_H is 52.7:77.3 GB/s. On an RTX 4060 laptop, it’s 11.8:47.5. A fixed offload policy tuned for one machine would perform terribly on the other.

The ft bench bw command profiles your hardware and tunes the policy accordingly. This is the difference between a heuristic and an architecture.

Semantic-Aware Caching: Because Agents Lie

FreeToken’s second major mechanism addresses something that should terrify anyone running agentic workloads: context truncation. When a coding agent hits its context window, the harness truncates the conversation, typically at special token boundaries like thinking blocks, tool calls, or tool outputs. If the inference engine has been caching state naively, that truncation invalidates everything and forces a full re-prefill.

FreeToken anchors recurrent-state checkpoints at these special-token boundaries. When the agent truncates and edits context, the engine re-prefills only the new suffix. For agentic workloads where contexts are constantly edited and re-prefilled, this is the difference between an interactive tool and a job queue.

During prefill, full-layer double buffering streams layer l+1 while the GPU computes layer l, hiding PCIe transfer latency behind compute. During decode, a shared LRU expert cache spanning all MoE layers follows the router in real-time, rather than relying on a placement decision made at load time.

The numbers back this up. At equal cache capacity (37% of the Qwen3.6 expert pool), FreeToken’s global LRU misses 16% of decode-time expert reads. KTransformers misses 41%. llama.cpp misses 62%.

Real Results, Real Hardware

Benchmarks from the FreeToken team show consistent gains across the board:

Those laptop numbers deserve a second look. 39.3 tok/s on an 8 GB laptop GPU running a 35B model. For context, the median decode speed measured for Codex in production traces is 33 tok/s. FreeToken beat that on consumer laptop hardware with a quantized model. This isn’t just “runs”, it’s “usable as a daily driver.”

Worst-case TTFT stays under 44 seconds across every workload tested. That’s the difference between an engine you can actually build products on and a toy that works until you try to do something real.

Elastic Memory: The VRAM Budget Is a Suggestion

FreeToken’s final mechanism is the one that breaks most existing inference engines: dynamic VRAM reallocation. At scheduler safe points, the engine rebuilds the GPU expert cache under a revised VRAM budget, without restarting the engine or reloading the host pool.

This matters because your VRAM budget changes as conversations grow. KV caches expand with context length. Long prompts route to more experts. The engine needs the flexibility to shrink the expert cache to make room for an expanding conversation, or grow it back when the context resets.

The elastic memory manager reads experts from disk directly into their final host layout, then pins them. No GPU warmup required, the first request is served with a cold cache. The memory hierarchy (disk → RAM → VRAM) is treated as one continuous system, not separate silos with hard boundaries.

This is the same philosophy behind the emerging High Bandwidth Flash (HBF) memory standard. At Hot Chips 2026, OXMIQ demonstrated that MoE expert pools are the ideal HBF use case, large, write-once, infrequently read. And indeed, expects running billions of tokens per day are actively evaluating HBF as a new tier specifically for these workloads. The hardware is starting to meet the software where it lives.

What Does “Running on Consumer GPU” Actually Mean?

Here’s the uncomfortable truth the AI industry doesn’t want to discuss: “runs on consumer hardware” has never been the same as “runs well.” The FreeToken benchmarks are promising, but they come with a generous set of asterisks:

• The RTX 4060 laptop runs the 35B model, not the 284B or 753B models
• The 284B and 753B runs require workstation-class GPUs with 24-32 GB VRAM
• Quantization is doing a lot of heavy lifting (MXFP4 is 4-bit precision)
• Generation speeds of 15-25 tok/s are usable but far from what a cloud GPU cluster provides

The engine’s own benchmarks show GLM-5.2 at 14.9 tok/s on the RTX PRO 6000. That’s a $7,000+ workstation card. The “consumer GPU” in the headline is doing some heavy lifting.

But here’s what the benchmarks don’t capture: the capability shift. CPU-first AI approaches are increasingly demonstrating that local inference is achievable for models that were previously cloud-only. For healthcare, legal, finance, and IP-heavy R&D, the ability to run frontier-class models without data leaving the machine isn’t a luxury, it’s a compliance requirement. The 14.9 tok/s is slow. It’s also private, free, and always available.

Who Actually Wins?

FreeToken is open source under Apache-2.0, available on PyPI as freetoken, and ships as a one-click desktop app for Windows and Linux. ft serve exposes OpenAI- and Anthropic-compatible endpoints on port 1919. ft launch claude wires up Claude Code, Codex, OpenCode, or OpenClaw against your own box.

The project even offers a simple, plain-language explanation in its README, for those who are new to all this.

For solo developers and small teams whose agent token bills exceed the cost of a GPU, the math is simple: buy the hardware once, run unlimited local inference. For enterprises, it’s the air-gapped path. For the rest of us, it’s a preview of where the industry is heading.

The memory wall isn’t getting thinner, frontier models are getting larger, and HBM capacity is growing slower than parameter counts. The future of inference isn’t bigger GPUs. It’s smarter orchestration of the hardware you already own. FreeToken is the first system that treats your entire computer as one machine, rather than a GPU with accessories.

That’s not just a technical achievement. It’s a philosophical one. The future belongs to the systems that make the best use of what’s already on your desk, not the ones that assume you’ll buy your way out of hardware limitations with ever-more-expensive silicon.

Install it, run ft bench bw, and see what your machine can actually do. The results might surprise you.