Your Adam Optimizer Defaults Are Raising Your GPU Bill, And Your Blood Pressure

The Three Numbers That Broke My Model

A practitioner working on a hard RL problem spent weeks chasing a dead model. They simplified the architecture, added layers, removed layers, swapped LSTMs for Transformers, added attention, removed it, rebuilt input features twenty times, and burned thousands of GPU hours on pathological hyperparameter searches.

The fix, when it finally arrived, was three numbers:

optimizer = optim.AdamW(
    model.parameters(), 
    lr=3e-4, 
    betas=(0.5, 0.95),  # The magic numbers
    eps=1e-4
)

Changing β‍₂ from 0.999 to 0.95 and β‍₁ from 0.9 to 0.5, plus bumping ε from 1e-8 to 1e-4. That was it. The model that had been stone-dead for weeks suddenly worked.

This isn’t an edge case. It’s a signal that the defaults you trust are actively hiding catastrophic dynamics.

What Adam Actually Does (And Why Most People Get It Wrong)

Vanilla SGD applies the same learning rate to every parameter. Adam fixes two problems at once: it smooths noisy gradients via momentum (first moment), and adapts step sizes per parameter based on observed gradient statistics (second moment).

m_t = β₁ · m_{t-1} + (1 - β₁) · g_t

v_t = β₂ · v_{t-1} + (1 - β₂) · g_t²

The update then becomes:

θ_t = θ_{t-1} - α · m̂_t / (√v̂_t + ε)

Elegant. Adaptive. And the source of nearly every failure mode in modern deep learning.

Where Adam Fails Spectacularly

Adam Can Fail on Trivial Convex Problems

Reddi, Kale, and Kumar’s 2018 ICLR paper On the Convergence of Adam and Beyond constructed an explicit simple convex optimization problem where Adam does not converge to the optimal solution. The proof of convergence in the original Adam paper? Flat out wrong.

The short-term memory of the exponential moving average kills off informative gradients too fast. They even show a case where Adam converges to the worst possible solution.

Adaptive Methods Generalize Worse Than SGD

Wilson et al.’s 2017 NeurIPS paper The Marginal Value of Adaptive Gradient Methods in Machine Learning is damning. On a simple binary classification task, SGD achieves zero test error while adaptive methods, including Adam, have much larger errors. Model architecture doesn’t matter. The result holds across different architectures: adaptive models perform worse, not better, than vanilla SGD on the test set.

The Bigger the Model, the More Adam Breaks

Researchers at Meta (Molybog et al.) published A Theory on Adam Instability in Large-Scale Machine Learning, which documents unexplained loss spikes during large language model training. Models from 7 billion to 546 billion parameters all exhibit the same pathology.

Here’s the mechanism: some parameters enter states where the gradient update is tiny, shrinking v_t. Adam’s adaptive normalization then divides by √v_t + ε. When v_t has decayed over ~1000 steps of near-zero gradients, and a large gradient suddenly arrives, the update magnitude explodes. Catastrophic loss spike. Unexplainable dynamics.

This is almost certainly what was happening in the RL training that burned thousands of GPU hours.

RL Destroys Adam

Reinforcement learning is where Adam’s assumptions break most catastrophically.

The core assumption that makes Adam a reliable default is that the gradient distribution is stationary: the volatility of a parameter update now reasonably predicts the volatility of future updates. In supervised learning, this mostly holds.

In RL, it doesn’t. The policy that generated the data ten steps ago may be completely different from the policy generating data now. The optimization landscape is constantly shifting. v_t is always chasing a moving target.

Google’s Dopamine framework, a deep RL library, silently accounts for this by using dramatically different ε values:

The Intuition Behind Fixing the Defaults

Adam’s defaults, β‍₁ = 0.9, β‍₂ = 0.999, ε = 1e-8, were chosen because they work reasonably well across a wide spectrum of problems. But “reasonably well” is not the same as “optimal”, and “wide spectrum” does not include every regime.

The PyTorch Trap

PyTorch’s Adam has a default weight_decay=0. But AdamW defaults to weight_decay=0.01. These are not the same optimizer, and assuming they are will silently change your training dynamics. If you switch from Adam to AdamW, which most modern recipes recommend, you’re also changing regularization strength.

Why This Matters More Than Ever

The frontier of AI training is shifting to models with billions of parameters trained on trillions of tokens. At this scale, the optimizer isn’t a footnote in the training script, it’s the primary decision handling every weight update across 100 billion parameters.

Recent work from JPMorgan Chase’s Global Technology Applied Research center on Muon-SW shows that a one-line change to the weight decay rule can accelerate training by 30% on mixture-of-experts models ranging from 72 to 930 million parameters. The principle, that blind use of optimization defaults is costly, applies far beyond Adam.

The Bottom Line

The next time you encounter inexplicable loss spikes, unstable reinforcement learning, or a model that won’t converge despite weeks of architectural changes, don’t immediately redesign the network.

Open the optimizer. Inspect the hyperparameters. Question defaults. And think critically, now armed with better intuition.

The problem might not be your model at all. It might be three little numbers.

For a deeper dive into how autonomous optimization and hyperparameter tuning are reshaping AI research, check out the work on self-modifying training agents. And if you’re training large models on limited hardware, understanding optimizer dynamics becomes even more critical.

The “just use Adam” era needs to end. Not because Adam is bad, but because blind trust in any default is the fastest path to a broken model and an empty GPU budget.