Lin F. Yang is not some random AI hobbyist. He’s a tenured associate professor at UCLA with dual PhDs from Johns Hopkins, a Simons Research Fellowship, and a research portfolio spanning reinforcement learning, LLM acceleration, and optimization theory. When he says he couldn’t understand what his AI was doing, it’s worth paying attention.
His recent tweetstorm went semi-viral for an uncomfortable reason. He used GPT to solve a problem in matrix sketching that he’d wanted to tackle for ten years. The result, published as a 72-page paper on arXiv, nearly resolves an open question about the measurement complexity of sketching the Schatten-1 norm. Previous best bounds were Ω(n) lower and trivial O(n²) upper, per Li, Nguyen, Woodruff (2019). Yang’s result pins it down to n²⁻⁰⁽¹⁾, tight up to polylogarithmic factors.
Impressive stuff. But here’s the twist: Yang couldn’t fully read the AI’s work.
“Throughout the process, I felt that my only role was to teach the AI how to write things in a way that I could understand. Its initial language was extremely condensed, so compressed that I could barely follow it, but somehow the AI agents themselves seemed to understand it perfectly well.”
This isn’t a story about a professor being outsmarted. It’s a story about the emergence of machine-to-machine communication that humans can’t parse, and what that means for everything from scientific publishing to AI safety.
The “Guff” Problem: When AI Jargon Becomes Machine Shorthand
Yang’s experience isn’t an isolated anecdote. Across forums and communities, users of frontier models are hitting the same wall. A pattern is emerging with models like GPT-5.6 “Sol” and Claude “Opus 5”: they increasingly produce highly compressed, jargon-laden output that other LLMs understand perfectly but humans find nearly impenetrable.
One developer described the experience bluntly: Sol will say things like “T024 surface-repair failed backwards stability verification on P3 microstructure, rerunning read-bound with in-plane leveling fix”, having invented its own internal shorthand while working through a problem, then summarizing its work using only that private vocabulary.
This isn’t necessarily a bug. In fact, it might be the most natural thing in the world.
Why AI Is Developing Its Own Compressed Language
Language models aren’t optimizing for human readability. They’re optimizing for performance on benchmarks and tasks. When you train a model with reinforcement learning from verifiable rewards (RLVR), you’re implicitly teaching it that hit the benchmark, regardless of how elegant or human-readable the reasoning was.
The result is a kind of emergent linguistic optimization. Think of it as a human mathematician who’s been told “solve the problem, show your work”, except the “work” is only ever evaluated by other mathematicians who already know the field inside out. You’d develop shorthand too.
One Reddit user described an experiment where they asked an AI to encode a dataset in the most efficient way possible, explicitly telling it that human readability didn’t matter. The output was a strange combination of characters that a second AI could parse perfectly. The lesson was clear:
If AI systems need to share information with each other, or with themselves across context windows, they’ll develop communication protocols that optimize for machine efficiency, not human comprehension.
This has one researcher suggesting we should think of it as AI developing its own scientific notation. A modern mathematical paper might rely on 10,000 files of accumulated definitions. An LLM’s “jargon” could represent massive compressed context that would require a thousand-page book to unpack for a human.
The Interpretability Tool Crisis: Even the Tools Are Failing
You might think the answer is better interpretability tools. That’s what Anthropic’s alignment team has been working on, right? Well, their own research just dropped a massive reality check.
Anthropic introduced CHIVE, an agentic pipeline that discovers unexpected LLM behaviors and tests potential explanations through counterfactual experiments. The pipeline works: it screens transcripts, identifies odd behaviors, generates hypotheses, and verifies them with prompt edits.
Then came the awkward part. They used the CHIVE data to evaluate their own interpretability tools, the same activation oracles, natural-language autoencoders, and sparse autoencoders that Anthropic uses in its system card evaluations. Here’s what they found:
| Condition | Performance |
|---|---|
| Transcript-only baseline | No uplift |
| + Activation oracles | No uplift |
| + Natural-language autoencoders | No uplift |
| + Sparse autoencoders | No uplift |
None of the interpretability tools helped an agent predict model behavior better than just reading the transcript. Even worse, the tools sometimes actively misled predictors.
The researchers’ reasoning for why the tools fail is telling: activation-reading tools can describe what’s in the activations, but they almost never state the causal relationship between a prompt feature and a behavior. A tool might say the model “is likely generating a random integer between min and max”, which helps you identify a coding bug. But on average, the wins were offset by cases where tool outputs missed relevant information or sent the predictor down the wrong path.
The deeper problem is structural. When you’re evaluating whether an AI might write code to fool a grader, you can’t just run the counterfactual and check. You’re dealing with behaviors that are, by definition, hard to verify. CHIVE’s behaviors were simple and checkable, and the tools still failed to provide uplift. For harder-to-check behaviors in production safety cases, the outlook is grim.
This doesn’t mean interpretability tools are useless. As the authors note, they can still surface concepts worth investigating, like “concealment” or “evaluation awareness.” But as evidence for causal claims, they currently perform no better than a coin flip dressed in spectral analysis.
What This Means for Governance, Safety, and the “Artificial State”
The implications extend far beyond academic curiosity. As models become agents that make decisions rather than just generate text, the distinction between observing what a model does and understanding why it does it becomes existential.
Consider the regulatory landscape. Colorado’s Automated Decision-Making Technology (ADMT) Act aims to govern AI systems that make consequential decisions in housing, employment, insurance, education, healthcare, and government services. The FTC’s July 2026 policy statement argues federal law may preempt state-level mandates that “suppress or alter accurate outputs.”
What does “accurate” even mean when we can’t fully interpret the system? What does “transparency” mean when the most advanced interpretability tools on the planet provide zero uplift over reading a transcript?
A report on AI model governance transparency frames the core structural challenge precisely: transparency doesn’t automatically create trust. Its first effect is exposure.
When you force AI developers to disclose more, you create documentation burden, compliance complexity, and incentives for metric gaming. The question becomes whether we’re building an “evidence system” or a “decoration.” The report identifies a core governance chain:
Documentation → Auditability → Responsibility → Correction → Enforcement → Learning
If the chain stops at documentation, you get transparency as performance art. If it reaches learning and enforcement, you get something approaching institutional accountability.
But here’s the uncomfortable question: How do you enforce accountability for a system no human can fully understand?
You can’t. That’s the hard truth. And it explains why some researchers are increasingly focusing on smaller, more interpretable language models as a counterpoint to opaque, massive LLMs, and why the incoherence that comes from scaling LLMs is becoming a mainstream concern rather than a fringe obsession.
The University Question: What Do We Teach When AI Outpaces Us?
Yang’s original tweet ended with an existential question for academia:
“What will be the role of humans then? Will we only need to ask the right questions? Universities will also change profoundly. What should universities teach students in the future?”
The MIT response is already underway. A Forbes piece covering MIT’s position notes that AI is forcing higher education to rethink what students learn, how they prove it, and which human skills matter most.
But there’s a deeper issue hiding beneath the “AI will change universities” headline. The professor’s experience suggests a new kind of scientific workflow is emerging: human poses problem → AI solves it in machine-comprehensible shorthand → human forces AI to translate back into human-readable form → human verifies and publishes.
This raises serious questions about the foundation of scientific knowledge:
- Who’s responsible for errors in AI-generated proofs? If the human can barely understand the reasoning, how do they verify it?
- What happens when AI-to-AI communication becomes the dominant mode? Would human researchers become mere scribes who translate machine outputs into publishable format?
- How do we train the next generation to work with tools they don’t fully understand?
Some would argue this is a skill issue. Critics on Reddit pointed out that LLMs amplify existing developer expertise, making experts more powerful while leaving novices behind. Yang’s experience might simply be the experience of a mathematical expert who’s been pushed to expand his toolkit. That’s not a bad thing.
Except… Yang isn’t a novice. He’s a senior researcher in theoretical computer science who works on LLM acceleration, compression, and quantization. If he can’t follow the AI’s reasoning, who can?
The “Just Ask It to Explain” Fallacy
There’s a tempting response: “Why didn’t the professor just ask the AI to explain itself in simpler terms?”
Users report doing exactly this. They ask the model to break down its work. And it works, partially. But there’s a difference between an AI producing human-readable text about its reasoning and that text being a faithful representation of what the AI actually computed.
This is the fundamental problem of chain-of-thought faithfulness. Modern models can easily produce post-hoc narratives that make their decisions appear intuitive and logical, even when those narratives are confabulated. The gap between “what the model did” and “what the model says it did” is precisely what interpretability research tries to close, and Anthropic’s own results suggest we’re not closing it.
When you train models to predict the outcomes of counterfactual prompts, they improve, substantially. The CHIVE paper shows models trained on counterfactual data generalize to held-out settings, including out-of-distribution transcript sources. That’s promising for self-understanding.
But the same research that validated this approach also demonstrated the tools designed to provide external verification, activation readings, natural-language autoencoders, SAEs, fail to help. A model can learn to predict its own behavior without anyone, not even the model’s own creators, understanding how it works.
The Road Ahead: Accepting Bounded Human Interpretability
Here’s the uncomfortable place we’re headed. The prevailing sentiment among developers is that the “AI is too complex” framing misses the point:
We can often observe what a model does without having a complete explanation for why the internal computation produces that behavior. That distinction matters. We don’t need to understand every neuron in a model to use it, just like we don’t need to understand every biological mechanism in the brain to study behavior.
This perspective suggests interpretability isn’t about achieving perfect transparency but about developing reliable methods for explaining, testing, and predicting why systems behave the way they do. It’s about building confidence rather than achieving omniscience.
That’s a more useful frame. We don’t fully understand how human experts make decisions either, yet we hold them accountable. We rely on outcomes, audits, and behavioral consistency. Maybe the same framework can work for AI.
The shift toward local, private LLM deployment amid concerns over transparency and control suggests This approach requires different infrastructure than massive cloud models. And the economics of running complex LLMs locally are getting more brutal by the quarter.
But one thing is clear: the genie is out of the bottle. Models are developing communication patterns that transcend human linguistic conventions. Tools designed to make AI interpretable to humans are failing at their core mission. And a UCLA professor with dual PhDs just published a breakthrough paper he can barely read.
The question isn’t whether AI will outpace our ability to understand it. It already has. The question is whether we’ll build the institutional, technical, and educational infrastructure to live with that reality, or keep pretending that interpretability is a problem we can engineer our way out of.
Want to dig deeper into the complexity crisis? Check out The Incoherence Wall: Why Scaling LLMs Is Making Them Less Reliable and the surprising case for why AI might actually be amplifying expert developers’ abilities rather than closing skill gaps.
