3 articles found
Blindly using Adam defaults is costing you convergence, stability, and generalization. Here’s the math, the failure modes, and the fix.
Xiaomi’s MiMo V2.5 hits 3000 tps with a 1-trillion-parameter model using a radical FP4 quantization and a ‘block-diffusion’ drafter. Here’s the tech that made it happen and the catch.
Moonshot AI’s Attention Residuals architecture replaces decade-old residual connections with selective depth-wise attention, delivering 1.25x compute efficiency and breaking the PreNorm dilution bottleneck that has plagued deep transformers.
