Tootfinder

FACT: the Features At Convergence Theorem for neural networks
Enric Boix-Adsera, Neil Mallinar, James B. Simon, Mikhail Belkin
https://arxiv.org/abs/2507.05644

FACT: the Features At Convergence Theorem for neural networks
A central challenge in deep learning theory is to understand how neural networks learn and represent features. To this end, we prove the Features at Convergence Theorem (FACT), which gives a self-consistency equation that neural network weights satisfy at convergence when trained with nonzero weight decay. For each weight matrix $W$, this equation relates the "feature matrix" $W^\top W$ to the set of input vectors passed into the matrix during forward propagation and the loss gradients passed t…