Tootfinder

Low-rank Momentum Factorization for Memory Efficient Training
Pouria Mahdavinia, Mehrdad Mahdavi
https://arxiv.org/abs/2507.08091 https://arxiv.org/pdf/2507.08091 https://arxiv.org/html/2507.08091
arXiv:2507.08091v1 Announce Type: new
Abstract: Fine-tuning large foundation models presents significant memory challenges due to stateful optimizers like AdamW, often requiring several times more GPU memory than inference. While memory-efficient methods like parameter-efficient fine-tuning (e.g., LoRA) and optimizer state compression exist, recent approaches like GaLore bridge these by using low-rank gradient projections and subspace moment accumulation. However, such methods may struggle with fixed subspaces or computationally costly offline resampling (e.g., requiring full-matrix SVDs). We propose Momentum Factorized SGD (MoFaSGD), which maintains a dynamically updated low-rank SVD representation of the first-order momentum, closely approximating its full-rank counterpart throughout training. This factorization enables a memory-efficient fine-tuning method that adaptively updates the optimization subspace at each iteration. Crucially, MoFaSGD leverages the computed low-rank momentum factors to perform efficient spectrally normalized updates, offering an alternative to subspace moment accumulation. We establish theoretical convergence guarantees for MoFaSGD, proving it achieves an optimal rate for non-convex stochastic optimization under standard assumptions. Empirically, we demonstrate MoFaSGD's effectiveness on large language model alignment benchmarks, achieving a competitive trade-off between memory reduction (comparable to LoRA) and performance compared to state-of-the-art low-rank optimization methods. Our implementation is available at https://github.com/pmahdavi/MoFaSGD.
toXiv_bot_toot

🇺🇦 #NowPlaying on KEXP's #90TEEN
Freckle:
🎵 Taraval
#Freckle
https://freckledfreckles.bandcamp.com/track/taraval
https://open.spotify.com/track/5m0fBSdQ9f9EJRhZ0IGjm9

"Rubbing one out" has different meanings to a mobster and his moll.

The German radio station "Bayern 2" has devoted the latest episode of its show “IQ” to the #Sun - and to Solar Orbiter’s first views of the solar poles. Interview partner is Sami Solanki of #mpsgoettingen :

We had leftover home fries Sunday from brunch takeaway. Today they’re latkes, (with added oil egg and milk). Egg and bacon, tea and coffee. Our mahjongg buddy is busy but we will@probably play tonight anyway. #TogetherBreakfast https://photos.app.goo.gl/Mhw6iFA4k2Wdbm3T8

How to use SDKMAN with MobaXterm on Win11: https://blog.rmendes.net/2025/07/11/how-to-use-sdkman-with.html

🇺🇦 Auf radioeins läuft...
Peter and Kerry:
🎵 All I've Got Are My Dreams
#NowPlaying #PeterandKerry
https://peterandkerry.bandcamp.com/track/all-ive-got-are-my-dreams
https://open.spotify.com/track/3GvMPgHM6pEK20uD0WjOQC

#ndwgoecountdown Ein Blick in den Himmel: Ausstellung Astrofotografie
Unser Nachthimmel ist voller atemberaubender Ansichten, die über die Astrofotografie sichtbar gemacht werden können, mit einer einfachen Kamera oder einem hochtechnischen Teleskop. Unsere Ausstellung zeigt Bilder von Sonne und Planeten über interstellare Gasnebel bis hin zu fernen Galaxien, aufgenommen von u…

#ndwgoecountdown Exoplaneten – Entdeckung von Planeten um ferne Sterne
Tausende Planeten, die um ferne Sterne kreisen, sind bisher bekannt. Doch wie entdeckt man einen Planeten, der viele Lichtjahre von uns entfernt ist? Im Mitmachexperiment erläutern Forscher*innen eine der gängigsten Methoden und berichten von der künftigen Weltraummission PLATO und ihrer Suche nach erdähnli…

🇺🇦 Auf radioeins läuft...
Peter and Kerry:
🎵 All I've Got Are My Dreams
#NowPlaying #PeterandKerry
https://peterandkerry.bandcamp.com/track/all-ive-got-are-my-dreams
https://open.spotify.com/track/3GvMPgHM6pEK20uD0WjOQC