They Cracked the Encryption on AI’s Private Thoughts, All Three of Them

The “Encryption” That Wasn’t

Here’s the setup: OpenAI, Anthropic, and Google all realized that their models’ final answers are copyable, but the process, how the model thinks, is the real intellectual property. So they stopped showing users the chain-of-thought. Instead, APIs return reasoning traces as encrypted blocks. The client can’t read them, but can send them back on subsequent requests so the model can continue reasoning from where it left off.

Sounds secure enough. Until you realize those encrypted blocks have a fatal flaw: they’re interchangeable across sessions, users, and even different models within the same provider’s ecosystem.

The attack is almost embarrassingly simple:

1. Let the strongest model reason through a problem, capturing its encrypted CoT.
2. Take that encrypted block and feed it to a weaker, easier-to-jailbreak model from the same provider.
3. Get the weaker model to decode and output the trace verbatim.

For example, Claude Opus 4.8’s hidden reasoning could be handed to Haiku, Anthropic’s smaller, speed-optimized model, which could then be induced to “read it out” in plaintext. You don’t need to break Opus itself. You just go after its little brother.

One Hacker News user neatly summarized the absurdity: “If you didn’t allow it, you wouldn’t be able to change models in the same conversation, as key parts of the context would be lost.” The feature that lets you seamlessly switch from a frontier model to a cheaper one mid-conversation is the same feature that makes the entire anti-distillation protection worthless.

Four Attack Vectors, Zero Subtlety

The paper identifies four distinct ways this vulnerability gets exploited, and honestly, each one deserves its own panic.

The Distillation Elephant in the Room

Here’s where the paper gets genuinely spicy. The researchers used their extracted traces to study open-source models, particularly Kimi K3, which benchmark tests show exceeding Claude Opus 4.8 in several categories.

The findings are damning. Prefilling Kimi K3’s reasoning with a few tokens of Opus reasoning measurably shifts its responses toward Opus’s style. A small memorization analysis showed that specific Claude and GPT reasoning spans are up to six orders of magnitude easier to extract from Kimi K3 than from the next-closest model.

Translation: it looks like a Chinese model was trained on the reasoning traces of US frontier models. The “distillation debate” that’s been simmering for years just got some very concrete evidence.

The Hacker News comment section erupted, of course. “Wow, almost certainly the approach that alternative labs use to distill Claude”, one user noted. “I always wondered how far they could get with just the answer missing the reasoning. They probably actually also had the reasoning.”

The irony isn’t lost on anyone. As one sharp commenter put it, the companies that built their empires on “the theft and misuse of intellectual property” are now on the receiving end. If you train your models on the sum of human knowledge, much of it copyrighted, then cry foul when someone trains on your model’s outputs, you’re not making a principled argument. You’re making a power grab.

Others pushed back on the term “stealing” itself. “Stealing” something you already paid for (tokens) but can’t have access to? The EU doesn’t recognize copyright on LLM outputs. You’re paying per token, and the reasoning blocks are generated as part of the service you purchased. Calling this “theft” is, at best, a stretch.

Why Did This Happen?

The architectural decision that enabled this attack wasn’t malicious, it was lazy, or perhaps pragmatic.

Providers chose to return encrypted reasoning blocks to clients rather than storing them server-side. Why? Two reasons stand out. First, zero data retention promises to enterprise clients become much easier when sensitive reasoning data never touches your servers. Second, scaling server-side storage for billions of tokens of reasoning would be a nightmare of disk space, latency, and disaster recovery.

The problem is they used a single encryption key across all models, sessions, and users. Per-user keys would have prevented cross-session replay. Per-model keys would have prevented the “ask Haiku to read Opus’s homework” trick. A proper key hierarchy would have made the entire attack impractical.

Instead, they optimized for the happy path: let users seamlessly switch models mid-conversation. The result is what happens when security takes a backseat to product convenience.

The paper notes all three providers have since patched the vulnerability, “All model providers acknowledged the receipt of our report and subsequently we were unable to launch the same attacks”, but providers denied any “security implications arising from side channels or replay attacks.”

Classy.

What This Means for the Rest of Us

For developers and companies building on proprietary APIs, this paper should change how you think about your data. Those encrypted blocks in your session logs aren’t just opaque noise, they’re a potential liability. If you’re sharing logs publicly, you might be leaking private reasoning that contains PII, credentials, or proprietary information from your own codebase.

The broader question is whether the “encrypted thinking” model makes sense at all. If the LLM needs to access the raw CoT to continue reasoning, and can be persuaded to output it, then the encryption is just obfuscation with extra steps. One HN commenter shared that a simple two-sentence developer prompt made all of Codex’s models output their encrypted compaction data in plaintext. Not a weaker model. Not a jailbreak. Just a prompt.

The real takeaway: if a model can read something, a determined attacker can get it to say something. This is a fundamental property of LLMs, not a bug that can be patched. The illusion of security through metrics applies here too, the encryption looked like protection until someone actually tested it.

This also raises uncomfortable questions about the ethical failures in AI model transparency and user trust that have become routine in this industry. Companies promise transparency and safety, then deploy opaque reasoning processes that they actively hide from users. The “safety” justification for encrypted CoT is partly real, hazardous information can lurk in reasoning even when outputs are safe. But the dominant motivation is clearly anti-competitive: preventing distillation.

The Cat-and-Mouse Game Continues

The paper’s authors demonstrated that the fix, per-model or per-session encryption keys, stops this specific attack. But the underlying problem remains. As one researcher noted, you can’t permanently prevent motivated actors from extracting reasoning from models that have to expose that reasoning to function.

There’s already follow-up work on “trace inversion”, reconstructing plausible reasoning traces from final answers alone, without any access to encrypted blocks. If you can approximate the thinking process from the outputs, then hiding the actual traces becomes a speed bump, not a wall.

The broader implication for supply chain and AI infrastructure security risks is clear: the AI stack has too many moving parts, too many trust assumptions, and too many shortcuts taken in the name of speed and convenience. This paper isn’t an anomaly, it’s a preview.

The Real Question Nobody’s Asking

Beyond the technical wizardry and the geopolitical finger-pointing, there’s a deeper issue here. The AI industry runs on a fundamental contradiction: these models were trained on the collective intellectual output of humanity, but their owners treat the models’ own outputs as proprietary secrets.

Whether you call it “stealing” or “distillation” or “recovery”, the underlying dynamic is the same. The companies that scraped the entire internet without permission are now scrambling to protect their models from the same treatment. The pot is calling the kettle black, and the kettle has a very good lawyer.

One HN commenter captured the sentiment perfectly: “If you steal all of human knowledge and shove it behind a paywall, you get exactly what you deserve.”

Is this a breakthrough or an ethical breach? It’s both, and that’s precisely why it’s so uncomfortable. The technique is clever, the science is solid, and the implications are genuinely concerning for model privacy, intellectual property, and AI safety. But the moral high ground is very much occupied, and it’s not held by the companies playing victim.

The bottom line: If you’re building on proprietary AI APIs, assume your reasoning is readable. Audit your session logs. Stop sharing them publicly. And remember that the “encryption” protecting your thoughts is about as robust as a password written on a sticky note.

The AI industry just learned that transparency isn’t optional, it’s inevitable. You can try to hide your model’s thinking behind encryption, but as this paper demonstrates, the harder you try to keep secrets, the harder someone will work to reveal them.