The M1’s Neural Engine was supposed to be the future of on-device AI. Instead, it turns out to be a beautifully engineered monument to a past that ended around 2018. Three years after shelving a reverse-engineered ANE driver, Eileen Yoon came back to finish the job, and the resulting retrospective reads like an archaeological dig through silicon that Apple has already abandoned.
The kicker? The M5’s headline feature folded the ANE cores directly into the GPU. The standalone NPU is officially on life support. But before we write its obituary, there’s a lot to learn from why it died, and what that says about how we design hardware for a moving target like modern AI.
The Real Innovation Was Never the Math
Here’s the thing about the ANE’s 16 compute cores: they’re boring. Each one packs 128 FP16 (or 256 INT8) parallel multiply-accumulate lanes, giving you 2,048 parallel MAC lanes across the whole chip. But a MAC unit is just a MAC unit, a dot product is a dot product, whether you’re running a convolution or an attention head.
The ANE die layout shows where the real architectural decisions were made. It wasn’t in the arithmetic, it was in the dataflow surrounding it: when and where inputs and outputs enter, stay, and move.
What made the ANE special for its era was the assumption embedded in its DNA: predictable reuse patterns. Convolutions slide the same kernel across an entire input tensor. That’s an asymmetric workload where one operand (the weights) can be loaded once and reused thousands of times. Apple designed its MAC datapath around this asymmetry, and it worked brilliantly, for CNN-era models in 2017.
The problem? Transformers don’t work that way, especially during autoregressive decode, where you stream the whole model’s weights to produce one token.
What the Register File Reveals About Apple’s Priorities
The most revealing part of this reverse-engineering effort comes from decoding the hardware register files. When Eileen compiled a CoreML model with a simple TANH activation and inspected the resulting compiled hardware, she found a 33-entry lookup table at offset 0x4288:
00004280: 0000 0044 0000 003c 0000 f52f d633 bc35 # 0.000000 0.124329 0.244873 0.358398
00004290: 6537 7038 1539 a239 183a 793a c93a 0a3b # 0.462158 0.554688 0.635254 0.704102 0.761719 0.809082 0.848145 0.879883
000042a0: 3e3b 673b 883b a23b b63b c63b d33b dd3b # 0.905273 0.925293 0.941406 0.954102 0.963867 0.971680 0.978027 0.982910Those 33 FP16 words are quantized samples of tanh(x) at intervals of 1/8. But here’s the clever part: adjacent entries are linearly interpolated, meaning that single lookup table gives smooth, continuous output across its entire range.
This is classic fixed-function hardware design: do the common case in dedicated silicon, make it fast and efficient, and accept that you can’t change it later. The same trade-off that made the ANE brilliant for convolutions is what sealed its fate for transformers.
A Task Descriptor Is Not an Instruction
Now here’s where things get philosophically interesting. When you look at what the driver actually submits to the ANE, there are no CONV, MATMUL, or RELU opcodes. Instead, the driver loads a “task descriptor” (TD) into memory, sets a pointer via (TM_ADDR, TM_SIZE), and rings the doorbell with TM_PUSH:
static void ane_tm_push_tq(struct ane_device *ane, struct ane_request *req)
{
int qid = req->qid;
tm_write32(ane, TM_ADDR, tq_read32(ane, TQ_ADDR1(qid)));
tm_write32(ane, TM_INFO, tq_read32(ane, TQ_SIZE1(qid)) | req->td_count);
tm_write32(ane, TM_PUSH, TQ_PRTY_TABLE[qid] | (qid & 7) << 8), // magic
}That TD is essentially a serialized dump of the ANE’s datapath configuration registers. Each section, KernelDMASrc, Common, TileDMASrc, L2, PE, NE, TileDMADst, writes to a specific MMIO register block. The hardware has no ISA. It’s a fixed-function dataflow engine, and the “program” is just the configuration for one pass through the datapath.
This is a fundamental philosophical divide from how GPUs work. NVIDIA’s architecture uses pushbuffers and PBDMA to issue commands that the GPU’s command processor walks through, but the actual compute hardware is still executing instructions. The ANE is more like a configurable ASIC: you can tweak the knobs it exposes, but you can’t tell it to do something new.
The Memory Hierarchy Betrayal
The ANE’s memory hierarchy tells you exactly what Apple believed about ML workloads in 2017:
- KMem: 16x per-core 64 KiB SRAM for kernels (1 MiB total)
- L1: per-core MAC input staging
- L2: 2 MiB shared across all cores
The numbers work out to a roofline ridge point of 162 operations per byte of DRAM traffic. The M1 ANE specs of 11 TOP/s at 68 GB/s system DRAM bandwidth mean that to sustain peak MAC throughput from DRAM, you’d need to stream 22 TB/s, over 300x what’s physically available. Everything depends on on-chip reuse.
Here’s where the design assumptions get exposed: KernelDMA is load-only. Apple committed to the idea that weights would never be written back. TileDMA is bidirectional, so intermediate activations can be fed back through L2. But if your model has weights that need to be updated during inference, say, a KV cache that grows during autoregressive decode, you’re out of luck. The kernel path becomes a bottleneck.
The measured DRAM throughput confirms the problem:
| Path | Bandwidth |
|---|---|
| ANE KernelDMA | 37.99 GB/s |
| ANE TileDMA | 59.08 GB/s |
| GPU | 77.70 GB/s |
And the worst part? The kernel and tile DMA requests are serial. When you run them together, execution time is strictly additive:
T_AB = 0.001 + 0.939*T_A + 0.981*T_BApple’s “unified memory” doesn’t mean zero-copy direct access. Every byte has to be staged through ANE’s local SRAM first, and the DMA engines can’t even overlap their transfers.
The Compiler’s Invisible Hand
Perhaps the most interesting insight from this reverse-engineering effort is about the compile-time folding that Core ML performs. When you define a ReLU with constant scale and offset like z = 4x – 2, y = ReLU(z/2 + 1), the compiler doesn’t just pass those parameters through:
authored convolution: W = 4, b = -2
activation affine: s = 0.5, c = 1
compiled convolution: W' = 2, b' = 0The constant scale and offset get folded directly into the convolution weights and bias before the ReLU even happens. This is exactly the kind of aggressive static optimization that a fixed-function datapath enables, the compiler knows everything at build time, so it bakes the math into the weights.
This is the double-edged sword of hardware abstraction. The abstraction hides enormous complexity from the developer, but it also hides the assumptions baked into the silicon. When those assumptions break, like when transformers arrived, the abstraction becomes a cage.
What This Means for AI Hardware’s Future
The ANE story isn’t just about Apple. The same pattern plays out across reverse-engineering high-performance AI hardware and the broader NPU ecosystem. Every vendor building dedicated AI accelerators faces the same fundamental question: how do you build hardware that’s efficient for known workloads without painting yourself into a corner?
The M5’s decision to fold ANE cores into the GPU is really an admission that the era of standalone NPUs optimized for CNNs is over. Transformers need flexibility, and GPUs, with their general-purpose dataflow and lean, efficient AI system design and performance optimization, are better positioned to provide it.
The ANE’s 33-entry tanh lookup table, the serial DMA engines, the load-only kernel path, these are all monuments to a 2017 worldview where models were smaller, more predictable, and could be fully compiled at build time. The industry has moved on.
The Lessons For Hardware Abstraction
If you’re designing hardware or building abstractions on top of it, the ANE retrospective offers some uncomfortable truths:
- Performance isolation is a feature, not a bug. The ANE’s deterministic timing and dedicated DMA paths made it predictable, but they also made it rigid.
- The compiler is the architecture. When Core ML folds constants into weights, it’s not just an optimization, it’s a statement about what the hardware can and cannot do.
- Memory movement is destiny. The ANE’s 162:1 roofline ratio means that for most workloads, DRAM bandwidth, not compute, is the bottleneck. Any hardware design that doesn’t solve the memory problem first is just moving deck chairs.
The open hardware and low-level system transparency movement gets this right: you can’t optimize what you can’t inspect. Apple’s opacity forced the reverse-engineering community to spend hundreds of hours just to understand the register file layout. That effort produced real insights about how the ANE actually works, but it also highlighted how much freedom is lost when hardware becomes a black box.
The forensic analysis of AI model weights and reverse engineering shows a similar pattern: the more you understand what’s actually happening under the hood, the better equipped you are to make intelligent decisions about what to build on top.
As for the ANE itself? It was a brilliant piece of engineering for its time. But like all hardware that’s optimized for a specific generation of models, it was destined to become obsolete. The real lesson of the ANE isn’t about Apple’s Neural Engine at all, it’s about the price of committing to silicon before you know what the software will need.
