Tootfinder

Low-rank Momentum Factorization for Memory Efficient Training
Pouria Mahdavinia, Mehrdad Mahdavi
https://arxiv.org/abs/2507.08091 https://arxiv.org/pdf/2507.08091 https://arxiv.org/html/2507.08091
arXiv:2507.08091v1 Announce Type: new
Abstract: Fine-tuning large foundation models presents significant memory challenges due to stateful optimizers like AdamW, often requiring several times more GPU memory than inference. While memory-efficient methods like parameter-efficient fine-tuning (e.g., LoRA) and optimizer state compression exist, recent approaches like GaLore bridge these by using low-rank gradient projections and subspace moment accumulation. However, such methods may struggle with fixed subspaces or computationally costly offline resampling (e.g., requiring full-matrix SVDs). We propose Momentum Factorized SGD (MoFaSGD), which maintains a dynamically updated low-rank SVD representation of the first-order momentum, closely approximating its full-rank counterpart throughout training. This factorization enables a memory-efficient fine-tuning method that adaptively updates the optimization subspace at each iteration. Crucially, MoFaSGD leverages the computed low-rank momentum factors to perform efficient spectrally normalized updates, offering an alternative to subspace moment accumulation. We establish theoretical convergence guarantees for MoFaSGD, proving it achieves an optimal rate for non-convex stochastic optimization under standard assumptions. Empirically, we demonstrate MoFaSGD's effectiveness on large language model alignment benchmarks, achieving a competitive trade-off between memory reduction (comparable to LoRA) and performance compared to state-of-the-art low-rank optimization methods. Our implementation is available at https://github.com/pmahdavi/MoFaSGD.
toXiv_bot_toot

What's the opposite of a snowbird? I'm going to start summering in Montreal. Or even further north..
https://flipboard.com/@gothamist/gothamist-igeoft02z/-/a-8mMvqdG0TBu8rLQV9dpBmg:a:3807498986-/0

Heavy storms, possible flooding expected for NYC area Monday amid ‘tropical-like’ air
A “tropical-like” air mass could yield heavy showers, thunderstorms and wind gusts Monday across the New York City region, the National Weather …

🇺🇦 #NowPlaying on KEXP's #90TEEN
Freckle:
🎵 Taraval
#Freckle
https://freckledfreckles.bandcamp.com/track/taraval
https://open.spotify.com/track/5m0fBSdQ9f9EJRhZ0IGjm9

Genndy Tartakovsky's developing a new show 'Heist Safari' that looks pretty great. Loved what he did with Dexter's Lab. This one's about three estranged frog brothers who meet up at their dad's funeral, have to rob a bank to inherit their late dad's fortune, and have issues when they cut side deals with mobsters. Every episode will be executed as just one shot, the events won't be in chronological order, and it'll have an EDM score. Looks & sounds prom…

Genndy Tartakovsky Developing 'Heist Safari' Animated Series for Adult Swim
Celebrated animator Genndy Tartakovsky — the tooned-up brain behind the adult-comedy movie Fixed (coming to Netflix on August 13), Sony's Hotel Transylvania

#ndwgoecountdown Exoplaneten – Entdeckung von Planeten um ferne Sterne
Tausende Planeten, die um ferne Sterne kreisen, sind bisher bekannt. Doch wie entdeckt man einen Planeten, der viele Lichtjahre von uns entfernt ist? Im Mitmachexperiment erläutern Forscher*innen eine der gängigsten Methoden und berichten von der künftigen Weltraummission PLATO und ihrer Suche nach erdähnli…

How to use SDKMAN with MobaXterm on Win11: https://blog.rmendes.net/2025/07/11/how-to-use-sdkman-with.html

#ndwgoecountdown Vortrag – Die Entdeckung von Planeten um ferne Sterne mit der PLATO-Mission
Neben den acht Planeten unseres Sonnensystems wurden um andere Sterne bereits über 5800 Planeten entdeckt. Am MPS bereiten wir in Zusammenarbeit u.a. mit der ESA die Weltraummission PLATO vor. Mit PLATO werden wir bald tausende neuer Exoplaneten entdecken.

Confirmada la visita de Morrissey a Santiago este 2025
No serš en Fauna Primavera. Se presentarš en el Movistar Arena el 16 de Noviembre.
Preventa Caja Los Andes desde el miércoles 11 de junio. Venta general nueve 12 de junio por puntoticket.
https://www.

The German radio station "Bayern 2" has devoted the latest episode of its show “IQ” to the #Sun - and to Solar Orbiter’s first views of the solar poles. Interview partner is Sami Solanki of #mpsgoettingen :

#ndwgoecountdown Ein Blick in den Himmel: Ausstellung Astrofotografie
Unser Nachthimmel ist voller atemberaubender Ansichten, die über die Astrofotografie sichtbar gemacht werden können, mit einer einfachen Kamera oder einem hochtechnischen Teleskop. Unsere Ausstellung zeigt Bilder von Sonne und Planeten über interstellare Gasnebel bis hin zu fernen Galaxien, aufgenommen von u…