What actually happened is that a research team led by Jacky Kwok (Stanford CS PhD, Berkeley EECS) dropped CLM-8B, an open-weights projection head for Qwen3-8B that implements the exact same “System One” decision interface as TypeSafe AI’s proprietary Jev. It supports the full primitive set (Choice, Noul, Score), ships with a drop-in compatible Python client, and runs on a single NVIDIA GPU.
The benchmarks show something that looks suspiciously like a knockout punch: CLM matches Jev on T-Rex and Super Mario while running 4x to 13x faster. On interactive browser agents and gaming tasks, the latency gap is brutal. In the T-Rex game specifically, CLM takes 16.5 ms per decision versus Jev’s 149.8 ms.
That’s the headline. The asterisk is where things get spicy.
The Architecture Isn’t a Clone, It’s a Rewrite
Jev evaluates state and question choices jointly in a single forward pass. CLM fundamentally disaggregates the problem: it trains a state encoder and an action encoder with bidirectional InfoNCE loss. Each encoder is a frozen Qwen3-8B backbone plus a 20M-parameter trainable projection head.
The implications of this design choice ripple through everything:
1. Caching becomes the killer feature. In an agent loop, the state changes every step but the action set stays mostly fixed. CLM embeds those actions once and caches them. A clm-serve deployment reserves a slab of GPU memory similar to vLLM’s KV cache and reuses cached vectors. On a single RTX 4090 with three actions, revisited states drop from 1.7 ms to 0.6 ms.
2. Fine-tuning is actually possible. Jev is a closed API with no user fine-tuning, you can only prompt it via state and question instructions. CLM’s heads are tiny open weights (~75 MB). The team provides a fine-tuning tutorial that walks through training your own heads on agent trajectories.
3. The scaling story is predictable. The authors report that test InfoNCE loss decreases as a power law with training compute, dataset size, projection-head size, and encoder size. This isn’t vibes-based engineering, it’s the kind of predictability that lets teams forecast whether a bigger encoder is worth the compute cost.
The disaggregation is the part builders should steal. A loop scores the same state against many candidate actions, so the state embedding caches perfectly and each candidate collapses to a cheap dot product. That’s the economics that makes per-turn decisions affordable.
Training Data: The Three-Stage Recipe
CLM-8B wasn’t trained on vibes either. The recipe is refreshingly concrete:
- Pre-training on ~60M Nemotron DQA question-answer pairs for broad question-to-answer alignment
- Mid-training on ~30M synthetic hard negatives generated by Gemini 2.5 Flash-Lite to distinguish near-miss options from correct ones
- Post-training on ~1M agent trajectories from Agent Data Protocol, Endless-Terminals, and LiteCoder-Terminal-SFT
The hard negatives matter more than most people realize. A contrastive model trained only on easy wrong answers learns shallow features. Hard negatives force it to attend to what actually distinguishes the right action from the tempting-but-wrong one.
The ablation data is worth studying: on ~100K held-out questions, pre-training alone reaches 52.1% top-1 accuracy. Mid-training lifts it to 69.2%. Training on hard negatives from the start peaks at 62.4% before overfitting. Timing matters.
The Zero-Shot Numbers: Speed Everywhere, Accuracy Mostly
Here’s the full published comparison table:
| Task | CLM-8B Latency | Jev Latency | CLM-8B Success | Jev Success |
|---|---|---|---|---|
| T-Rex Game | 16.5 ms | 149.8 ms | 5/5 | 5/5 |
| Tool Calling (BFCL v4) | 76.8 ms | 125.5 ms | 95.2% | 99.2% |
| WikiRacing | 79.8 ms | 225 ms | 26/30 | 30/30 |
| Super Mario | 33.5 ms | 132.6 ms | 5/5 | 5/5 |
CLM wins on latency everywhere, 1.6x to 9x. It matches on the gaming tasks. But it trails on tool calling (95.2% vs 99.2%) and WikiRacing (26/30 vs 30/30).
The “9x faster” claim comes specifically from the T-Rex row. Across the other tasks, speedups range from roughly 1.6x to 4x. And several of these benchmarks are tiny (5 trials, 30 trials), so treat the success columns as directional.
The counterargument from Jev defenders is legitimate: the entire point of Jev is zero-shot broad knowledge backed by a larger proprietary model. CLM-8B runs a nimbler Qwen skeleton, and it shows on edge cases. As one commenter put it, this isn’t “Jev but open”, it’s “Jev for a specific slice of the problem space.”
But here’s the thing about that counterargument: it cuts both ways. The sentiment in developer forums is that Jev’s zero-shot broad knowledge is precisely why it’s backwards. If you’re already engineering options and instructions for Jev, you can also fine-tune a small model to perform even better on your specific domain. The zero-shot broad knowledge with constrained outputs pattern doesn’t scale for teams with actual data pipelines.
The Verifier Results: Where CLM Actually Embarrasses Jev
The flashier claim is the coding benchmark verifier results. For each task, a strong coding model generates several candidate solutions (Opus 5 for DeepSWE, Fable 5 for Terminal-Bench 2.1), then CLM or Jev acts as the verifier that picks the best one.
| Benchmark | Candidates | Pass@1 Baseline | CLM (Fine-Tuned) | Jev | CLM Latency | Jev Latency |
|---|---|---|---|---|---|---|
| DeepSWE | Best of 4 from Opus 5 | 73.7% | 81.6% | 71.1% | 79 ms (5.7x faster) | 449 ms |
| Terminal-Bench 2.1 | Best of 5 from Fable 5 | 84.0% | 87.6% | 83.1% | 32 ms (4.1x faster) | 131 ms |
Three things jump out:
Jev scored below pass@1 on both benchmarks. Selecting with Jev is worse than randomly picking one sample. For a model whose entire pitch is calibrated decision-making, that’s damning on long-horizon coding tasks.
The gains are genuine but modest. Picking the best of several attempts lifts DeepSWE from 73.7% to 81.6% and Terminal-Bench 2.1 from 84.0% to 87.6%. That’s a real verifier effect, about 8 and 3.6 points respectively, not a model beating frontier models from scratch.
The eval sets are small. 38 held-out DeepSWE tasks and 30 held-out Terminal-Bench 2.1 tasks. 81.6% is 31 of 38. 87.6% is roughly 26 of 30. A change of one or two tasks moves the percentage by several points. The “SOTA” label is the authors’ own.
The Honest Limitations
Let’s not pretend this is a clean win. CLM has three genuine gaps against Jev:
Context budget. Jev accepts requests up to 64K token context out-of-the-box. CLM-8B was tested and calibrated at 2K to 8K context. While its Qwen3 backbone can accept longer prompts, representations past 8K haven’t been calibrated for the reference head. That’s a real limitation for tasks requiring long-horizon state tracking.
Probability normalization. CLM calculates probabilities via dot products and softmax over the candidates passed in that request. Its probabilities are inherently relative to the candidate set provided, whereas Jev’s scoring is calibrated internally against absolute criteria. A proposed fix exists, add a garbage candidate to absorb the “rest” of the confidence, but it hasn’t been implemented.
Zero-shot generalization. Jev is backed by a larger proprietary model. On zero-shot open-domain tasks, Jev holds a clear edge in edge-case accuracy. If your problem is “handle arbitrary inputs I haven’t seen before”, CLM-8B’s Qwen3-8B backbone will feel smaller than it looks on paper.
Where This Fits in the Broader Open-Weights Story
The practical reality: with CLM, you don’t lose API features. You get the full primitive set (Choice, Noul, Score) with massive latency gains and zero API costs. What you sacrifice is some zero-shot generalization on niche out-of-domain tasks.
For teams that route intents, verify agent checkpoints, rerank best-of-N outputs, or make real-time decisions in games and UI agents, CLM is a legitimate option. For classification with changing labels where you can rewrite option text instead of retraining, it’s arguably better than Jev.
The deeper architectural implications of open-weights models in distributed systems are exactly what’s playing out here. Alibaba’s strategic commitment to Qwen’s open-source ecosystem just enabled a research team to build what is effectively a Jev competitor using Qwen as the backbone.
And that’s the real story. This isn’t about CLM versus Jev. It’s about what happens when open-weights models get cheap and predictable enough that a small team can bolt a specialized head onto an existing LLM and compete with a company that raised hundreds of millions. The practical challenges of deploying large open-weights models locally are diminishing, and the sovereign AI deployment using open models playbook is expanding.
The Bottom Line
CLM-8B is a well-documented, genuinely fast, open take on System One decision-making. The speed claims hold in the published charts. The accuracy claims are “on par” zero-shot and a real but modest verifier lift on small benchmarks. A multimodal CLM-35B is already in training with more data, compute, and parameters.
If you build agents that make many cheap decisions, routing, verifying, reranking, CLM is worth benchmarking against your current LLM calls. Run it on your own tasks before you trust the leaderboard numbers.
Jev isn’t dead. But the era of “Jev is the only way to do System One decisions” just ended. The open-weights ecosystem now has a legitimate alternative that’s faster, cheaper, and fine-tunable. Whether that matters depends on whether your workflows look more like CLM’s tied game results or its lower-success tool and navigation results.
The question for developers is stark: how much decision quality can your agent afford to trade for speed? For some workloads, the answer will be “a lot.” For others, Jev’s zero-shot broad knowledge will remain the moat.
