Your Browser Just Got a 27B-Node Brain: Why Ternary Bonsai 2 Changes the Local AI Game

So you want to run a 27-billion-parameter language model but your GPU budget is basically a laptop and a prayer. Prism ML just dropped Ternary Bonsai 2, and it’s making some audacious claims: a 27B model compressed to under 6GB, running entirely in your browser via WebGPU. The “revolutionary” part? It claims to retain 98.2% of the original FP16 model’s intelligence.

Let’s dig into whether this is genuinely impressive engineering or just another carefully packaged benchmark story.

Wait, 27B Parameters in Under 6GB? How?

The short answer: ternary quantization. But not the naive kind.

Standard quantization compresses weights to 4-bit or 8-bit integers. Bonsai 2 takes this to an extreme, every weight in the model becomes one of exactly three values: -1, 0, or +1. Since log₂(3) ≈ 1.585, each weight theoretically carries under 1.6 bits of information instead of 16.

But here’s where it gets clever. A single scale factor can’t capture the dynamic range of a 27B model. So Bonsai 2 uses “g128” grouping: one FP16 scale value shared across every 128 weights. The math works out to about 1.71 bits per weight for the language model, and 1.72 bits when you account for the tiny slice of parameters (~0.1%) kept at higher precision for recurrent state and normalization weights.

That’s how a model that normally demands ~54GB in FP16 shrinks to a package that fits on your laptop, or even in your browser tab.

The Hadamard Rotation: The Devil in the Details

Here’s the part that makes Bonsai 2 different from your average quantized model, and the source of most setup pain.

Before quantization, each weight matrix is passed through a blockwise Hadamard rotation, an orthogonal transform applied in fixed 1024-element blocks. This rotation is folded into the stored weights ahead of time, so it adds no extra bits or runtime cost on the weight side. But the corresponding inverse transform must be applied to activations at inference time.

This single detail makes Bonsai 2 incompatible with stock loaders. Skip the rotation, and the model doesn’t error out, it just produces wrong output silently. That’s a worse failure mode than a crash, because you might not notice until you’re three paragraphs into a confidently wrong answer.

The model card declares its architecture as prism_hadamard_qwen35, signaling to any loader that it needs the special activation transform. But practically speaking, you need the project’s llama.cpp fork or the bundled MLX runtime. This isn’t a drop-in GGUF you point a standard binary at.

The 98.2% Claim: Benchmark Math or Reality?

The model card reports an 84.78 average across 14 thinking-mode benchmarks against an FP16 baseline. The authors frame this as 98.2% retention. Here’s the full comparison:

Category-level numbers reinforce the story: math benchmarks hit 96.57 (within half a point of full precision), coding scores 89.42, and agentic tool-calling lands at 74.92.

But here’s where skepticism is warranted. A single benchmark average can hide significant regressions. Some early testers have reported the model looping endlessly on simple questions about Rust, while others found it surprisingly capable on reasoning tasks. The model’s defaults matter too, it ships with xhigh reasoning effort and thinking-mode sampling (temperature 1.0, top_p 0.95, top_k 20), which means it burns through tokens thinking before answering.

One particularly sharp observation from the community: the 98.2% figure only makes sense if you understand the “potato paradox.” A model composed of 99% “intelligence” (per the benchmark) that drops to 98% has lost proportionally more than the raw numbers suggest, just like potatoes that are 99% water, then 98% water, weigh half as much afterward.

Real-World Performance: What Actually Matters

Here’s what the throughput tables look like on Apple Silicon with the GGUF packs via llama.cpp Metal:

On the M5 Pro, decode throughput streams roughly 204 GB/s of weight data, confirming this is a memory-bandwidth-bound model. Power draw is modest too: 27.5W on the GPU rail during decode, versus 300-455W for datacenter GPUs.

But here’s the honest comparison that matters: a 27B FP16 model doesn’t fit on a laptop at all. So the relevant benchmark isn’t “how much slower than FP16”, it’s “can a model of this capability class run interactively on consumer hardware in the first place?” The answer is yes, and that’s genuinely notable.

GGUF Packings: Size vs. Speed Tradeoffs

The same ternary weights ship in multiple containers, and the packaging choice matters more than you’d think:

• PTQ1_0: 1.75 bits/weight, 5.95GB, packs trits densely, close to the theoretical minimum
• PQ2_0: 2.13 bits/weight, 7.21GB, stores each value in a 2-bit slot, cheaper to unpack
• MLX 2-bit: 2.25 bits/weight, 7.67GB, container stores both scale and bias per group, which is structural overhead

Neither GGUF packing wins universally. PTQ1_0 tends to win on memory-bandwidth-limited GPUs (RTX 4090, RTX 6000 Ada, L4) because it moves 17% less data per step. PQ2_0 wins on compute-limited cards (RTX 5090, H100, Blackwell) where decode is bottlenecked by instruction throughput rather than bandwidth.

The MLX version is larger for a structural reason, not a quality difference, the grouped low-bit container stores both a scale and a bias per 128 weights when the ternary format only strictly needs the scale. The model card says the underlying weights are verified bit-for-bit identical across formats.

Running It: The Setup Reality

If you want to try this on Apple Silicon, the MLX path looks like this:

hf download prism-ml/Ternary-Bonsai-2-27B-mlx-2bit --local-dir bonsai2-27b-mlx
pip install -r bonsai2-27b-mlx/runtime/requirements.txt

Then load it through the bundled runtime module rather than a stock loader:

import sys
sys.path.insert(0, "bonsai2-27b-mlx/runtime")
from vision_artifact import load_vl_model, chat_config
from mlx_vlm import generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor, config = load_vl_model("bonsai2-27b-mlx")
prompt = apply_chat_template(processor, chat_config(config), "What is in this image?", num_images=1)
print(generate(model, processor, prompt, ["photo.jpg"], max_tokens=256, temperature=1.0))

The MLX pack also bundles the Qwen3.8-27B vision tower at full FP16 precision (0.92GB), so image inputs work alongside the compressed language model.

The WebGPU demo runs entirely in-browser, but expect the same custom kernel requirements. This isn’t a Transformers.js drop-in yet.

What This Means for Edge AI

The significance here extends beyond one model. Ternary quantization combined with WebGPU browser inference could break the hardware dependency cycle that’s dominated AI deployment. This follows the precedent of running a 27B-parameter model on mobile with extreme quantization and builds on WebGPU-powered browser-based AI inference with highly compressed models.

The broader context: Apple’s reportedly interested in running large AI models locally using advanced compression. And if you’re tracking edge AI performance milestones for running large models on-device, Bonsai 2 represents a meaningful leap in what consumer hardware can handle.

Still, the ecosystem is fragmented. You need custom loaders, patched llama.cpp forks, and a willingness to debug silent wrongness. The alternative approaches to efficient LLM inference without GPUs are getting more viable, but none of them are turnkey yet.

The Verdict: Impressive Engineering, Measure the Claims

The honest take: getting a 27B model under 6GB with benchmark performance that doesn’t collapse is genuinely hard engineering. The Hadamard rotation scheme is clever, the g128 grouping is well-designed, and the WebGPU implementation represents real progress for browser-based inference.

But the “98.2% intelligence” framing deserves scrutiny. Early user reports show the model can loop badly on straightforward questions, and the reasoning-mode defaults mean you’re paying for output in tokens even when you just want a simple answer. The model’s creators acknowledge these are meant for thinking-mode tasks, not code generation or quick Q&A.

If you’re deploying this in production, treat the demo repository as the reference implementation, pin your runtime versions, and validate outputs before trusting them. And note that Apple’s reported throughput figures were measured on an earlier pre-rotation build, directionally accurate, but not final.

The easy part, fitting a 27B model in under 6GB, is solved. The hard part, making it consistently useful across real-world queries, is still being tested. But for a 27B model that runs in your browser tab? That’s a future I’m willing to poke at.