Ever since Qwen 3.8 Flash Next dropped, a very specific misconception has been doing the rounds: N-gram tables will let you run 1T+ parameter models on a single server with 980B parameters offloaded to SSD. I’m here to disappoint you, it won’t. But what it will actually do for local models is arguably even better.
The Engram architecture, built on N-gram embedding tables, represents a fundamental challenge to how we think about transformer computation. Instead of scaling up, it’s about working smarter, moving the boring parts of language understanding into a lookup table so neural layers can focus on what actually matters: reasoning.
The Core Idea: It’s Just an Embedding Table with a Longer Key
Let’s strip away the hype. At its core, Engram is embarrassingly simple: an embedding table indexed by the last 2-3 tokens instead of a single token ID.
Instead of fetching a static vector for “New” and another for “York”, you fetch a single memorized vector for “New York”. “The United” gets its own vector. “import std” gets its own too. Hash the N-gram, fetch the vector, feed it into the network. O(1), constant time, essentially zero FLOPs.
This simplicity is precisely why it’s powerful. The table is cheap to query, so you can make it enormous and have it live in RAM or even SSD storage. Qwen 3.8 Flash Next carries 51 billion parameters of N-gram embeddings while only activating around 6 billion per token. For context, that’s roughly 8x more static knowledge stored per active parameter than a traditional dense model.
Why Transformers Waste Layers on Database Lookups
Here’s the uncomfortable truth about transformer architecture: a surprising amount of what happens in those early layers is reconstructing static crap from scratch. How entity names are spelled. Formulaic phrases. Common collocations. “New” + “York” = Wall Street, delis, rats, subways.
Every time the model needs to recall a multi-token entity, it burns several layers of attention and feed-forward networks re-assembling something that is, frankly, a database lookup. Engram moves that job to an actual database lookup so the neural layers can spend their depth on actual reasoning.
This isn’t just a theoretical inefficiency. The parameter budget of smaller models, 4B, 7B, even 27B, has always been split between two competing jobs: actually reasoning, and memorizing static patterns that a lookup table could hold. That’s a big part of why smaller models feel “dumb” even on tasks well within their reasoning ability. They’re spending their limited capacity on spelling “Massachusetts” correctly instead of thinking.
The Part Nobody Understands: The Lookup Is Dumb
Here’s the critical limitation that gets glossed over in most discussions: the lookup is “dumb.” The key is just the last 2-3 tokens. Your 200k tokens of context have zero influence on what gets retrieved.
The wider context can accept or reject whatever vector the N-gram fetched, but it can’t change what was fetched. Engrams are used to store “meaning”, similar to embeddings. It doesn’t replace reasoning or computation.
When an Engram model sees “import std”, it doesn’t suddenly gain years of C++ programming experience from the Engram vectors. The table memorizes, the transformer reasons. This separation of concerns isn’t a bug, it’s the entire point.
And you can’t fix this by cranking N up either. This is where the physics of language modeling kicks in. The higher the N, the rarer that specific N-gram is in training data, so each entry gets less and less training signal. The paper’s own ablation found that allocating capacity to 4-grams “dilutes capacity from the more frequent 2/3-gram patterns.” You can’t scale Engram embeddings up to 500B without it literally becoming a waste of space.
| N-gram Size | Training Signal | Optimal Use |
|---|---|---|
| 2-gram | Dense, frequent patterns | Core vocabulary, common collocations |
| 3-gram | Good coverage | Entity names, formulaic phrases |
| 4-gram+ | Sparse, diluted | Diminishing returns, capacity waste |
What This Means for Local Models
The real breakthrough here isn’t about running trillion-parameter models on consumer hardware. It’s about making small models dramatically smarter.
A 27B model has always had to spend its parameter budget on doing two jobs at once: actually reasoning, and memorizing static patterns that a lookup table could hold. Engram splits those jobs. The knowledge moves into a table that costs nothing to query, and every active parameter gets freed for reasoning.
This is why smaller 4B or 7B models feel “dumb” even on tasks well within their reasoning ability. They’re spending their limited capacity on spelling “Massachusetts” correctly instead of thinking. Engram changes that math entirely.
This architectural shift aligns with other alternative approaches to efficient transformer inference we’ve covered, where the industry is increasingly questioning whether the traditional transformer stack is the most efficient way to compute.
The Practical Reality: It’s Already Working
The theoretical promise is one thing, but the implementation is further along than most people realize. llama.cpp already has pull requests for N-gram offloading, and users have demonstrated it working. One user with a 3060 Ti (8GB VRAM) and 48GB of system RAM was able to run the Q2 quantized version of Qwen 3.8 Flash Next at 5-6 tokens per second, with their NVMe drive never exceeding 50% usage at peak.
Daniel Hanchen from Unsloth handled the pull request, which means the implementation has serious engineering chops behind it. The VLLM_PLE_CPU_OFFLOAD=1 environment variable already exists, allowing the PLE/n-gram embedding table to sit in system RAM instead of eating VRAM.
This isn’t theoretical. It’s shipping.
Why the Big Labs Are Quiet (For Now)
A fair question keeps surfacing: if Engrams are so great, why hasn’t anyone except Qwen adopted them? DeepSeek introduced the concept at scale, and there are hints that Long Cat used something similar. Google’s E2B and E4B models utilize related concepts through Per-Layer Embeddings (PLE), which some researchers describe as a simplified form of N-gram lookup tables (1-grams).
The answer might be simpler than conspiracy theories suggest: this is an architecture preview release. Other labs might just not be ready for public previews with their own N-gram implementations yet. Rather than attempting it straight away on a 1-2T model, they’re training a cheaper smaller model and getting it out the door to prime the community to adapt to the new architecture. Low risk, big gain.
There’s also the uncomfortable possibility that frontier labs have been doing this for years in their closed models. The hardware purchases of all those SSDs don’t need to be explained by training data alone.
The Research Frontier: Where This Goes Next
The most interesting experiments are happening at the intersection of Engrams and other efficiency innovations. Projects like Needle, a 14MB foundation model for tiny devices, are combining “engram key-value memory” with “hashed n-gram tables” and multi-lane hyper-connections. This isn’t just about making existing architectures slightly faster, it’s about rethinking what a model can be when memory and computation are fully decoupled.
One particularly compelling idea from the community: a proof of concept for a consumer GPU-sized model (around 20-30B parameters) loaded in VRAM, with many more parameters as Engrams (perhaps 100B or more) offloaded to RAM or NVMe storage. For VRAM-constrained systems, which is most consumer systems, having more Engram parameters than the theoretical optimum might be the pragmatic choice, even if it’s not what the ablations suggest.
This connects to a broader inference-first architectural shift in AI models we’re seeing across the industry. The question is no longer just “how big can we make it” but “how efficiently can we compute with what we have.”
The Bottom Line: Smaller Models Get Smarter
The hype around Engrams has been focused on the wrong metric. It’s not about running 1T models on a single server. It’s about smaller models becoming dramatically more capable because they no longer waste their parameter budget on memorization.
The table memorizes, the transformer reasons. That division of labor might be the most important architectural innovation for local models in years. A 4B model with an Engram table could be as intelligent as a 27B model without one, because every single active parameter is now devoted to actual cognition rather than rote recall.
This is one of several architectural innovations for efficient local inference that are collectively reshaping what’s possible on consumer hardware. We’re seeing a fundamental rethinking of state and memory in transformer architectures, and the trend is unmistakable.
The revolution isn’t bigger models. It’s smarter use of the models we already have.
