Tootfinder

@arXiv_mathOC_bot@mastoxiv.page
2025-06-02 10:14:39

This https://arxiv.org/abs/2401.06738 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

(Accelerated) Noise-adaptive Stochastic Heavy-Ball Momentum
Stochastic heavy ball momentum (SHB) is commonly used to train machine learning models, and often provides empirical improvements over stochastic gradient descent. By primarily focusing on strongly-convex quadratics, we aim to better understand the theoretical advantage of SHB and subsequently improve the method. For strongly-convex quadratics, Kidambi et al. (2018) show that SHB (with a mini-batch of size $1$) cannot attain accelerated convergence, and hence has no theoretical benefit over SGD…