Tootfinder

Stochastic gradient descent based variational inference for infinite-dimensional inverse problems
Jiaming Sui, Junxiong Jia, Jinglai Li
https://arxiv.org/abs/2506.08380

Stochastic gradient descent based variational inference for infinite-dimensional inverse problems
This paper introduces two variational inference approaches for infinite-dimensional inverse problems, developed through gradient descent with a constant learning rate. The proposed methods enable efficient approximate sampling from the target posterior distribution using a constant-rate stochastic gradient descent (cSGD) iteration. Specifically, we introduce a randomization strategy that incorporates stochastic gradient noise, allowing the cSGD iteration to be viewed as a discrete-time process.…

Online Quantum State Tomography via Stochastic Gradient Descent
Jian-Feng Cai, Yuling Jiao, Yinan Li, Xiliang Lu, Jerry Zhijian Yang, Juntao You
https://arxiv.org/abs/2507.07601

Online Quantum State Tomography via Stochastic Gradient Descent
We initiate the study of online quantum state tomography (QST), where the matrix representation of an unknown quantum state is reconstructed by sequentially performing a batch of measurements and updating the state estimate using only the measurement statistics from the current round. Motivated by recent advances in non-convex optimization algorithms for solving low-rank QST, we propose non-convex mini-batch stochastic gradient descent (SGD) algorithms to tackle online QST, which leverage the l…

Almost Sure Convergence for the Last Iterate of Stochastic Gradient Descent Schemes
Marcel Hudiani
https://arxiv.org/abs/2507.07281 https://

Almost Sure Convergence for the Last Iterate of Stochastic Gradient Descent Schemes
We study the almost sure convergence rate for the last iterate of stochastic gradient descent (SGD) and stochastic heavy ball (SHB) in the parametric setting when the objective function $F$ is globally convex or non-convex whose gradient is $γ$-Hölder. Using only discrete Gronwall's inequality without Robbins-Siegmund theorem nor martingale convergence theory, we recover results for both SGD and SHB: $\min_{s\leq t} \|\nabla F(w_s)\|^2 = o(t^{p-1})$ for non-convex objectives and $F(…

An Adaptive Order Caputo Fractional Gradient Descent Method for Multi-objective Optimization Problems
Barsha Shaw, Md Abu Talhamainuddin Ansary
https://arxiv.org/abs/2507.07674

An Adaptive Order Caputo Fractional Gradient Descent Method for Multi-objective Optimization Problems
This article introduces the multi-objective adaptive order Caputo fractional gradient descent (MOAOCFGD) algorithm for solving unconstrained multi-objective problems. The proposed method performs equally well for both smooth and non-smooth multi-objective optimization problems. Moreover, the proposed method does not require any a priori chosen parameters or ordering information of the objective functions. At every iteration of the proposed method, a subproblem is solved to identify a suitable d…

This https://arxiv.org/abs/2409.08469 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Improved Finite-Particle Convergence Rates for Stein Variational Gradient Descent
We provide finite-particle convergence rates for the Stein Variational Gradient Descent (SVGD) algorithm in the Kernelized Stein Discrepancy ($\mathsf{KSD}$) and Wasserstein-2 metrics. Our key insight is that the time derivative of the relative entropy between the joint density of $N$ particle locations and the $N$-fold product target measure, starting from a regular initial distribution, splits into a dominant `negative part' proportional to $N$ times the expected $\mathsf{KSD}^2$ and a smalle…

Gradient Similarity Surgery in Multi-Task Deep Learning
Thomas Borsani, Andrea Rosani, Giuseppe Nicosia, Giuseppe Di Fatta
https://arxiv.org/abs/2506.06130

Gradient Similarity Surgery in Multi-Task Deep Learning
The multi-task learning ($MTL$) paradigm aims to simultaneously learn multiple tasks within a single model capturing higher-level, more general hidden patterns that are shared by the tasks. In deep learning, a significant challenge in the backpropagation training process is the design of advanced optimisers to improve the convergence speed and stability of the gradient descent learning rule. In particular, in multi-task deep learning ($MTDL$) the multitude of tasks may generate potentially conf…

Non-Asymptotic Analysis of Online Local Private Learning with SGD
Enze Shi, Jinhan Xie, Bei Jiang, Linglong Kong, Xuming He
https://arxiv.org/abs/2507.07041

Non-Asymptotic Analysis of Online Local Private Learning with SGD
Differentially Private Stochastic Gradient Descent (DP-SGD) has been widely used for solving optimization problems with privacy guarantees in machine learning and statistics. Despite this, a systematic non-asymptotic convergence analysis for DP-SGD, particularly in the context of online problems and local differential privacy (LDP) models, remains largely elusive. Existing non-asymptotic analyses have focused on non-private optimization methods, and hence are not applicable to privacy-preservin…

Stochastic Gradient-Descent Calibration of Pyragas Delayed-Feedback Control for Chaos Suppression in the Sprott Circuit
Adib Kabir, Onil Morshed, Oishi Kabir
https://arxiv.org/abs/2506.06639

Stochastic Gradient-Descent Calibration of Pyragas Delayed-Feedback Control for Chaos Suppression in the Sprott Circuit
This paper investigates chaos control in the Sprott circuit, a minimal electronic system exhibiting complex nonlinear dynamics. Using the third-order nonlinear differential equation from Kaveh Merat paper, we model the circuit and implement delayed feedback control to suppress chaos. Experimental voltage data were extracted from published figures via WebPlotDigitizer. Then we explore two calibration techniques: Minimizing sum of squared errors (SSE), and stochastic gradient descent (SGD) with f…

This https://arxiv.org/abs/2504.14730 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csIT_…

Optimizing Noise Distributions for Differential Privacy
We propose a unified optimization framework for designing continuous and discrete noise distributions that ensure differential privacy (DP) by minimizing Rényi DP, a variant of DP, under a cost constraint. Rényi DP has the advantage that by considering different values of the Rényi parameter $α$, we can tailor our optimization for any number of compositions. To solve the optimization problem, we reduce it to a finite-dimensional convex formulation and perform preconditioned gradient descent…

Decentralized Optimization with Amplified Privacy via Efficient Communication
Wei Huo, Changxin Liu, Kemi Ding, Karl Henrik Johansson, Ling Shi
https://arxiv.org/abs/2506.07102

Decentralized Optimization with Amplified Privacy via Efficient Communication
Decentralized optimization is crucial for multi-agent systems, with significant concerns about communication efficiency and privacy. This paper explores the role of efficient communication in decentralized stochastic gradient descent algorithms for enhancing privacy preservation. We develop a novel algorithm that incorporates two key features: random agent activation and sparsified communication. Utilizing differential privacy, we demonstrate that these features reduce noise without sacrificing…

Can Biologically Plausible Temporal Credit Assignment Rules Match BPTT for Neural Similarity? E-prop as an Example
Yuhan Helena Liu, Guangyu Robert Yang, Christopher J. Cueva
https://arxiv.org/abs/2506.06904

Can Biologically Plausible Temporal Credit Assignment Rules Match BPTT for Neural Similarity? E-prop as an Example
Understanding how the brain learns may be informed by studying biologically plausible learning rules. These rules, often approximating gradient descent learning to respect biological constraints such as locality, must meet two critical criteria to be considered an appropriate brain model: (1) good neuroscience task performance and (2) alignment with neural recordings. While extensive research has assessed the first criterion, the second remains underexamined. Employing methods such as Procruste…

Simple Convergence Proof of Adam From a Sign-like Descent Perspective
Hanyang Peng, Shuang Qin, Yue Yu, Fangqing Jiang, Hui Wang, Zhouchen Lin
https://arxiv.org/abs/2507.05966

Simple Convergence Proof of Adam From a Sign-like Descent Perspective
Adam is widely recognized as one of the most effective optimizers for training deep neural networks (DNNs). Despite its remarkable empirical success, its theoretical convergence analysis remains unsatisfactory. Existing works predominantly interpret Adam as a preconditioned stochastic gradient descent with momentum (SGDM), formulated as $\bm{x}_{t+1} = \bm{x}_t - \frac{γ_t}{{\sqrt{\bm{v}_t}+ε}} \circ \bm{m}_t$. This perspective necessitates strong assumptions and intricate techniques, resulti…

Linear Discriminant Analysis with Gradient Optimization on Covariance Inverse
Cencheng Shen, Yuexiao Dong
https://arxiv.org/abs/2506.06845 https://

Linear Discriminant Analysis with Gradient Optimization on Covariance Inverse
Linear discriminant analysis (LDA) is a fundamental method in statistical pattern recognition and classification, achieving Bayes optimality under Gaussian assumptions. However, it is well-known that classical LDA may struggle in high-dimensional settings due to instability in covariance estimation. In this work, we propose LDA with gradient optimization (LDA-GO), a new approach that directly optimizes the inverse covariance matrix via gradient descent. The algorithm parametrizes the inverse co…

This https://arxiv.org/abs/2503.16398 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

The global convergence time of stochastic gradient descent in non-convex landscapes: Sharp estimates via large deviations
In this paper, we examine the time it takes for stochastic gradient descent (SGD) to reach the global minimum of a general, non-convex loss function. We approach this question through the lens of randomly perturbed dynamical systems and large deviations theory, and we provide a tight characterization of the global convergence time of SGD via matching upper and lower bounds. These bounds are dominated by the most "costly" set of obstacles that the algorithm may need to overcome in order to reach…

Kahan's Automatic Step-Size Control for Unconstrained Optimization
Yifeng Meng, Chungen Shen, Linuo Xue, Lei-Hong Zhang
https://arxiv.org/abs/2508.06002 https://

Kahan's Automatic Step-Size Control for Unconstrained Optimization
The Barzilai and Borwein (BB) gradient method is one of the most widely-used line-search gradient methods. It computes the step-size for the current iterate by using the information carried in the previous iteration. Recently, William Kahan [Kahan, Automatic Step-Size Control for Minimization Iterations, Technical report, University of California, Berkeley CA, USA, 2019] proposed new Gradient Descent (KGD) step-size strategies which iterate the step-size itself by effectively utilizing the info…

Constrained Stein Variational Gradient Descent for Robot Perception, Planning, and Identification
Griffin Tabor, Tucker Hermans
https://arxiv.org/abs/2506.00589

Constrained Stein Variational Gradient Descent for Robot Perception, Planning, and Identification
Many core problems in robotics can be framed as constrained optimization problems. Often on these problems, the robotic system has uncertainty, or it would be advantageous to identify multiple high quality feasible solutions. To enable this, we present two novel frameworks for applying principles of constrained optimization to the new variational inference algorithm Stein variational gradient descent. Our general framework supports multiple types of constrained optimizers and can handle arbitra…

This https://arxiv.org/abs/2505.17282 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csLG_…

Attention with Trained Embeddings Provably Selects Important Tokens
Token embeddings play a crucial role in language modeling but, despite this practical relevance, their theoretical understanding remains limited. Our paper addresses the gap by characterizing the structure of embeddings obtained via gradient descent. Specifically, we consider a one-layer softmax attention model with a linear head for binary classification, i.e., $\texttt{Softmax}( p^\top E_X^\top ) E_X v = \frac{ \sum_{i=1}^T \exp(p^\top E_{x_i}) E_{x_i}^\top v}{\sum_{j=1}^T \exp(p^\top E_{x_{j…

This month's newsletter/digest is on it's way to the email box of all the amazing friendly good people who asked for it.
The rest of you bitchy vicious augmentative people in public spats can read it here:
#newsletter #digest

Toroidal area-preserving parameterizations of genus-one closed surfaces
Marco Sutti, Mei-Heng Yueh
https://arxiv.org/abs/2508.05111 https://arxiv.org/pdf/2…

Toroidal area-preserving parameterizations of genus-one closed surfaces
We consider the problem of computing toroidal area-preserving parameterizations of genus-one closed surfaces. We propose four algorithms based on Riemannian geometry: the projected gradient descent method, the projected conjugate gradient method, the Riemannian gradient method, and the Riemannian conjugate gradient method. Our objective function is based on the stretch energy functional, and the minimization is constrained on a power manifold of ring tori embedded in three-dimensional Euclidean…

Statistical Inference for Differentially Private Stochastic Gradient Descent
Xintao Xia, Linjun Zhang, Zhanrui Cai
https://arxiv.org/abs/2507.20560 https://

Statistical Inference for Differentially Private Stochastic Gradient Descent
Privacy preservation in machine learning, particularly through Differentially Private Stochastic Gradient Descent (DP-SGD), is critical for sensitive data analysis. However, existing statistical inference methods for SGD predominantly focus on cyclic subsampling, while DP-SGD requires randomized subsampling. This paper first bridges this gap by establishing the asymptotic properties of SGD under the randomized rule and extending these results to DP-SGD. For the output of DP-SGD, we show that th…

This https://arxiv.org/abs/2503.14353 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_ees…

Unified Analysis of Decentralized Gradient Descent: a Contraction Mapping Framework
The decentralized gradient descent (DGD) algorithm, and its sibling, diffusion, are workhorses in decentralized machine learning, distributed inference and estimation, and multi-agent coordination. We propose a novel, principled framework for the analysis of DGD and diffusion for strongly convex, smooth objectives, and arbitrary undirected topologies, using contraction mappings coupled with a result called the mean Hessian theorem (MHT). The use of these tools yields tight convergence bounds, b…

A Dynamical Systems Perspective on the Analysis of Neural Networks
Dennis Chemnitz, Maximilian Engel, Christian Kuehn, Sara-Viola Kuntz
https://arxiv.org/abs/2507.05164

A Dynamical Systems Perspective on the Analysis of Neural Networks
In this chapter, we utilize dynamical systems to analyze several aspects of machine learning algorithms. As an expository contribution we demonstrate how to re-formulate a wide variety of challenges from deep neural networks, (stochastic) gradient descent, and related topics into dynamical statements. We also tackle three concrete challenges. First, we consider the process of information propagation through a neural network, i.e., we study the input-output map for different architectures. We ex…

Comparative Analysis of Novel NIRMAL Optimizer Against Adam and SGD with Momentum
Nirmal Gaud, Surej Mouli, Preeti Katiyar, Vaduguru Venkata Ramya
https://arxiv.org/abs/2508.04293

Comparative Analysis of Novel NIRMAL Optimizer Against Adam and SGD with Momentum
This study proposes NIRMAL (Novel Integrated Robust Multi-Adaptation Learning), a novel optimization algorithm that combines multiple strategies inspired by the movements of the chess piece. These strategies include gradient descent, momentum, stochastic perturbations, adaptive learning rates, and non-linear transformations. We carefully evaluated NIRMAL against two widely used and successful optimizers, Adam and SGD with Momentum, on four benchmark image classification datasets: MNIST, Fashion…

On the Inherent Privacy of Zeroth Order Projected Gradient Descent
Devansh Gupta, Meisam Razaviyayn, Vatsal Sharan
https://arxiv.org/abs/2507.05610 https:/…

On the Inherent Privacy of Zeroth Order Projected Gradient Descent
Differentially private zeroth-order optimization methods have recently gained popularity in private fine tuning of machine learning models due to their reduced memory requirements. Current approaches for privatizing zeroth-order methods rely on adding Gaussian noise to the estimated zeroth-order gradients. However, since the search direction in the zeroth-order methods is inherently random, researchers including Tang et al. (2024) and Zhang et al. (2024a) have raised an important question: is t…

Solving the Gross-Pitaevskii Equation with Quantic Tensor Trains: Ground States and Nonlinear Dynamics
Qian-Can Chen, I-Kang Liu, Jheng-Wei Li, Chia-Min Chung
https://arxiv.org/abs/2507.04279

Solving the Gross-Pitaevskii Equation with Quantic Tensor Trains: Ground States and Nonlinear Dynamics
We develop a tensor network framework based on the quantic tensor train (QTT) format to efficiently solve the Gross-Pitaevskii equation (GPE), which governs Bose-Einstein condensates under mean-field theory. By adapting time-dependent variational principle (TDVP) and gradient descent methods, we accurately handle the GPE's nonlinearities within the QTT structure. Our approach enables high-resolution simulations with drastically reduced computational cost. We benchmark ground states and dynamics…

Riemannian Inexact Gradient Descent for Quadratic Discrimination
Uday Talwar, Meredith K. Kupinski, Afrooz Jalilzadeh
https://arxiv.org/abs/2507.04670 http…

Riemannian Inexact Gradient Descent for Quadratic Discrimination
We propose an inexact optimization algorithm on Riemannian manifolds, motivated by quadratic discrimination tasks in high-dimensional, low-sample-size (HDLSS) imaging settings. In such applications, gradient evaluations are often biased due to limited sample sizes. To address this, we introduce a novel Riemannian optimization algorithm that is robust to inexact gradient information and prove an $\mathcal O(1/K)$ convergence rate under standard assumptions. We also present a line search variant …

A gradient flow that is none: Heat flow with Wentzell boundary condition
Marie Bormann, L\'eonard Monsaingeon, D. R. Michiel Renger, Max von Renesse
https://arxiv.org/abs/2506.22093

A gradient flow that is none: Heat flow with Wentzell boundary condition
We establish a representation of the heat flow with Wentzell boundary conditions on smooth domains as gradient descent dynamics for the entropy in a suitably extended Otto manifold of probability measures with additional boundary parts. Yet it is shown that for weak boundary diffusion, the associated Fokker-Planck dynamics cannot be recovered from any entropy-driven metric JKO-Wasserstein scheme, at least if the underlying point metric satisfies certain natural regularity assumptions. This disc…

Fast and Provable Hankel Tensor Completion for Multi-measurement Spectral Compressed Sensing
Jinsheng Li, Xu Zhang, Shuang Wu, Wei Cui
https://arxiv.org/abs/2507.04847

Fast and Provable Hankel Tensor Completion for Multi-measurement Spectral Compressed Sensing
In this paper, we introduce a novel low-rank Hankel tensor completion approach to address the problem of multi-measurement spectral compressed sensing. By lifting the multiple signals to a Hankel tensor, we reformulate this problem into a low-rank Hankel tensor completion task, exploiting the spectral sparsity via the low multilinear rankness of the tensor. Furthermore, we design a scaled gradient descent algorithm for Hankel tensor completion (ScalHT), which integrates the low-rank Tucker deco…

Multi-objective Portfolio Optimization Via Gradient Descent
Christian Oliva, Pedro R. Ventura, Luis F. Lago-Fern\'andez
https://arxiv.org/abs/2507.16717 https://

Multi-objective Portfolio Optimization Via Gradient Descent
Traditional approaches to portfolio optimization, often rooted in Modern Portfolio Theory and solved via quadratic programming or evolutionary algorithms, struggle with scalability or flexibility, especially in scenarios involving complex constraints, large datasets and/or multiple conflicting objectives. To address these challenges, we introduce a benchmark framework for multi-objective portfolio optimization (MPO) using gradient descent with automatic differentiation. Our method supports any …

On gradient descent-ascent flows in metric spaces
Noboru Isobe, Sho Shimoyama
https://arxiv.org/abs/2506.20258 https://arxiv.org/pdf/…

On gradient descent-ascent flows in metric spaces
Gradient descent-ascent (GDA) flows play a central role in finding saddle points of bivariate functionals, with applications in optimization, game theory, and robust control. While they are well-understood in Hilbert and Banach spaces via maximal monotone operator theory, their extension to general metric spaces, particularly Wasserstein spaces, has remained largely unexplored. In this paper, we develop a mathematical theory of GDA flows on the product of two complete metric spaces, formulating…

On the Benefits of Accelerated Optimization in Robust and Private Estimation
Laurentiu Andrei Marchis, Po-Ling Loh
https://arxiv.org/abs/2506.03044 https:/…

On the Benefits of Accelerated Optimization in Robust and Private Estimation
We study the advantages of accelerated gradient methods, specifically based on the Frank-Wolfe method and projected gradient descent, for privacy and heavy-tailed robustness. Our approaches are as follows: For the Frank-Wolfe method, our technique is based on a tailored learning rate and a uniform lower bound on the gradient of the $\ell_2$-norm over the constraint set. For accelerating projected gradient descent, we use the popular variant based on Nesterov's momentum, and we optimize our obje…

Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness
Thomas Pethick, Wanyun Xie, Mete Erdogan, Kimon Antonakopoulos, Tony Silveti-Falls, Volkan Cevher
https://arxiv.org/abs/2506.01913

Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness
This work introduces a hybrid non-Euclidean optimization method which generalizes gradient norm clipping by combining steepest descent and conditional gradient approaches. The method achieves the best of both worlds by establishing a descent property under a generalized notion of ($L_0$,$L_1$)-smoothness. Weight decay is incorporated in a principled manner by identifying a connection to the Frank-Wolfe short step. In the stochastic case, we show an order optimal $O(n^{-1/4})$ convergence rate b…

The Glider Equation for Asymptotic Lenia
Hiroki Kojima, Ivan Yevenko, Takashi Ikegami
https://arxiv.org/abs/2508.04167 https://arxiv.org/pdf/2508.04167

The Glider Equation for Asymptotic Lenia
Lenia is a continuous extension of Conway's Game of Life that exhibits rich pattern formations including self-propelling structures called gliders. In this paper, we focus on Asymptotic Lenia, a variant formulated as partial differential equations. By utilizing this mathematical formulation, we analytically derive the conditions for glider patterns, which we term the ``Glider Equation.'' We demonstrate that by using this equation as a loss function, gradient descent methods can successfully dis…

Enhancing Biosecurity in Tamper-Resistant Large Language Models With Quantum Gradient Descent
Fahmida Hai, Saif Nirzhor, Rubayat Khan, Don Roosan
https://arxiv.org/abs/2506.19086 …

Enhancing Biosecurity in Tamper-Resistant Large Language Models With Quantum Gradient Descent
This paper introduces a tamper-resistant framework for large language models (LLMs) in medical applications, utilizing quantum gradient descent (QGD) to detect malicious parameter modifications in real time. Integrated into a LLaMA-based model, QGD monitors weight amplitude distributions, identifying adversarial fine-tuning anomalies. Tests on the MIMIC and eICU datasets show minimal performance impact (accuracy: 89.1 to 88.3 on MIMIC) while robustly detecting tampering. PubMedQA evaluations co…

Multilevel Stochastic Gradient Descent for Optimal Control Under Uncertainty
Niklas Baumgarten, David Schneiderhan
https://arxiv.org/abs/2506.02647 https:/…

Multilevel Stochastic Gradient Descent for Optimal Control Under Uncertainty
We present a multilevel stochastic gradient descent method for the optimal control of systems governed by partial differential equations under uncertain input data. The gradient descent method used to find the optimal control leverages a parallel multilevel Monte Carlo method as stochastic gradient estimator. As a result, we achieve precise control over the stochastic gradient's bias, introduced by numerical approximation, and its sampling error, arising from the use of incomplete gradients, wh…

A Proximal Variable Smoothing for Minimization of Nonlinearly Composite Nonsmooth Function -- Maxmin Dispersion and MIMO Applications
Keita Kume, Isao Yamada
https://arxiv.org/abs/2506.05974

A Proximal Variable Smoothing for Minimization of Nonlinearly Composite Nonsmooth Function -- Maxmin Dispersion and MIMO Applications
We propose a proximal variable smoothing algorithm for a nonsmooth optimization problem whose cost function is the sum of three functions including a weakly convex composite function. The proposed algorithm has a single-loop structure inspired by a proximal gradient-type method. More precisely, the proposed algorithm consists of two steps: (i) a gradient descent of a time-varying smoothed surrogate function designed partially with the Moreau envelope of the weakly convex function; (ii) an appli…

Antenna Q-Factor Topology Optimization with Auxiliary Edge Resistivities
Stepan Bosak, Miloslav Capek, Jiri Matas
https://arxiv.org/abs/2506.00595 https://…

Antenna Q-Factor Topology Optimization with Auxiliary Edge Resistivities
This paper presents a novel bi-level topology optimization strategy within the method-of-moments paradigm. The proposed approach utilizes an auxiliary variables called edge resistivities related to the Rao-Wilton-Glisson method-of-moments basis functions, for a definition of a fast local optimization algorithm. The local algorithm combines automatic differentiation with adaptive gradient descent. A Bayesian optimization scheme is applied on top of the local algorithm to search for an optimum po…

A Parallelizable Approach for Characterizing NE in Zero-Sum Games After a Linear Number of Iterations of Gradient Descent
Taemin Kim, James P. Bailey
https://arxiv.org/abs/2507.11366

A Parallelizable Approach for Characterizing NE in Zero-Sum Games After a Linear Number of Iterations of Gradient Descent
We study online optimization methods for zero-sum games, a fundamental problem in adversarial learning in machine learning, economics, and many other domains. Traditional methods approximate Nash equilibria (NE) using either regret-based methods (time-average convergence) or contraction-map-based methods (last-iterate convergence). We propose a new method based on Hamiltonian dynamics in physics and prove that it can characterize the set of NE in a finite (linear) number of iterations of altern…

Experimental Multiport-Network Parameter Estimation and Optimization for Multi-Bit RIS
Philipp del Hougne
https://arxiv.org/abs/2507.02168 https://

Experimental Multiport-Network Parameter Estimation and Optimization for Multi-Bit RIS
Physics-consistent theoretical studies on RIS-parametrized wireless channels use models from multiport-network theory (MNT) to capture mutual-coupling (MC) effects. However, in practice, RIS design and radio environment are partially or completely unknown. We fill a research gap on how to estimate the MNT model parameters in such experimentally relevant scenarios. Our technique efficiently combines closed-form and gradient-descent steps, and it can be applied to multi-bit-programmable RIS eleme…

Replaced article(s) found for stat.ML. https://arxiv.org/list/stat.ML/new
[1/1]:
- Rank-1 Matrix Completion with Gradient Descent and Small Random Initialization
Daesung Kim, Hye Won Chung

Rethinking LLM Training through Information Geometry and Quantum Metrics
Riccardo Di Sipio
https://arxiv.org/abs/2506.15830 https://a…

Rethinking LLM Training through Information Geometry and Quantum Metrics
Optimization in large language models (LLMs) unfolds over high-dimensional parameter spaces with non-Euclidean structure. Information geometry frames this landscape using the Fisher information metric, enabling more principled learning via natural gradient descent. Though often impractical, this geometric lens clarifies phenomena such as sharp minima, generalization, and observed scaling laws. We argue that curvature-aware approaches deepen our understanding of LLM training. Finally, we specula…

Perturbed Gradient Descent Algorithms are Small-Disturbance Input-to-State Stable
Leilei Cui, Zhong-Ping Jiang, Eduardo D. Sontag, Richard D. Braatz
https://arxiv.org/abs/2507.02131

Perturbed Gradient Descent Algorithms are Small-Disturbance Input-to-State Stable
This article investigates the robustness of gradient descent algorithms under perturbations. The concept of small-disturbance input-to-state stability (ISS) for discrete-time nonlinear dynamical systems is introduced, along with its Lyapunov characterization. The conventional linear Polyak-Lojasiewicz (PL) condition is then extended to a nonlinear version, and it is shown that the gradient descent algorithm is small-disturbance ISS provided the objective function satisfies the generalized nonli…

Emergent universal long-range structure in random-organizing systems
Satyam Anand, Guanming Zhang, Stefano Martiniani
https://arxiv.org/abs/2505.22933 http…

Emergent universal long-range structure in random-organizing systems
Self-organization through noisy interactions is ubiquitous across physics, mathematics, and machine learning, yet how long-range structure emerges from local noisy dynamics remains poorly understood. Here, we investigate three paradigmatic random-organizing particle systems drawn from distinct domains: models from soft matter physics (random organization, biased random organization) and machine learning (stochastic gradient descent), each characterized by distinct sources of noise. We discover …

Multilevel Bregman Proximal Gradient Descent
Yara Elshiaty, Stefania Petra
https://arxiv.org/abs/2506.03950 https://arxiv.org/pdf/250…

Multilevel Bregman Proximal Gradient Descent
We present the Multilevel Bregman Proximal Gradient Descent (ML-BPGD) method, a novel multilevel optimization framework tailored to constrained convex problems with relative Lipschitz smoothness. Our approach extends the classical multilevel optimization framework (MGOPT) to handle Bregman-based geometries and constrained domains. We provide a rigorous analysis of ML BPGD for multiple coarse levels and establish a global linear convergence rate. We demonstrate the effectiveness of ML BPGD in th…

Extreme Learning Machines for Exoplanet Simulations: A Faster, Lightweight Alternative to Deep Learning
Tara P. A. Tahseen, Lu\'is F. Sim\~oes, Kai Hou Yip, Nikolaos Nikolaou, Jo\~ao M. Mendon\c{c}a, Ingo P. Waldmann
https://arxiv.org/abs/2506.19679

Extreme Learning Machines for Exoplanet Simulations: A Faster, Lightweight Alternative to Deep Learning
Increasing resolution and coverage of astrophysical and climate data necessitates increasingly sophisticated models, often pushing the limits of computational feasibility. While emulation methods can reduce calculation costs, the neural architectures typically used--optimised via gradient descent--are themselves computationally expensive to train, particularly in terms of data generation requirements. This paper investigates the utility of the Extreme Learning Machine (ELM) as a lightweight, no…

This https://arxiv.org/abs/2502.03701 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

First-ish Order Methods: Hessian-aware Scalings of Gradient Descent
Gradient descent is the primary workhorse for optimizing large-scale problems in machine learning. However, its performance is highly sensitive to the choice of the learning rate. A key limitation of gradient descent is its lack of natural scaling, which often necessitates expensive line searches or heuristic tuning to determine an appropriate step size. In this paper, we address this limitation by incorporating Hessian information to scale the gradient direction. By accounting for the curvatur…

A Smoothing Newton Method for Rank-one Matrix Recovery
Tyler Maunu, Gabriel Abreu
https://arxiv.org/abs/2507.23017 https://arxiv.org/pdf/2507.23017

A Smoothing Newton Method for Rank-one Matrix Recovery
We consider the phase retrieval problem, which involves recovering a rank-one positive semidefinite matrix from rank-one measurements. A recently proposed algorithm based on Bures-Wasserstein gradient descent (BWGD) exhibits superlinear convergence, but it is unstable, and existing theory can only prove local linear convergence for higher rank matrix recovery. We resolve this gap by revealing that BWGD implements Newton's method with a nonsmooth and nonconvex objective. We develop a smoothing f…

Can SGD Handle Heavy-Tailed Noise?
Ilyas Fatkhullin, Florian H\"ubler, Guanghui Lan
https://arxiv.org/abs/2508.04860 https://arxiv.org/pdf/2508.04860

Can SGD Handle Heavy-Tailed Noise?
Stochastic Gradient Descent (SGD) is a cornerstone of large-scale optimization, yet its theoretical behavior under heavy-tailed noise -- common in modern machine learning and reinforcement learning -- remains poorly understood. In this work, we rigorously investigate whether vanilla SGD, devoid of any adaptive modifications, can provably succeed under such adverse stochastic conditions. Assuming only that stochastic gradients have bounded $p$-th moments for some $p \in (1, 2]$, we establish sha…

Replaced article(s) found for cs.LG. https://arxiv.org/list/cs.LG/new
[4/5]:
- SPGD: Steepest Perturbed Gradient Descent Optimization
Amir M. Vahedi, Horea T. Ilies

This https://arxiv.org/abs/2502.16492 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Convergence of Clipped SGD on Convex $(L_0,L_1)$-Smooth Functions
We study stochastic gradient descent (SGD) with gradient clipping on convex functions under a generalized smoothness assumption called $(L_0,L_1)$-smoothness. Using gradient clipping, we establish a high probability convergence rate that matches the SGD rate in the $L$ smooth case up to polylogarithmic factors and additive terms. We also propose a variation of adaptive SGD with gradient clipping, which achieves the same guarantee. We perform empirical experiments to examine our theory and algor…

SLASH: Self-Supervised Speech Pitch Estimation Leveraging DSP-derived Absolute Pitch
Ryo Terashima, Yuma Shirahata, Masaya Kawamura
https://arxiv.org/abs/2507.17208 https://

SLASH: Self-Supervised Speech Pitch Estimation Leveraging DSP-derived Absolute Pitch
We present SLASH, a pitch estimation method of speech signals based on self-supervised learning (SSL). To enhance the performance of conventional SSL-based approaches that primarily depend on the relative pitch difference derived from pitch shifting, our method incorporates absolute pitch values by 1) introducing a prior pitch distribution derived from digital signal processing (DSP), and 2) optimizing absolute pitch through gradient descent with a loss between the target and differentiable DSP…

From Sublinear to Linear: Fast Convergence in Deep Networks via Locally Polyak-Lojasiewicz Regions
Agnideep Aich, Ashit Baran Aich, Bruce Wade
https://arxiv.org/abs/2507.21429 h…

From Sublinear to Linear: Fast Convergence in Deep Networks via Locally Polyak-Lojasiewicz Regions
The convergence of gradient descent (GD) on the non-convex loss landscapes of deep neural networks (DNNs) presents a fundamental theoretical challenge. While recent work has established that GD converges to a stationary point at a sublinear rate within locally quasi-convex regions (LQCRs), this fails to explain the exponential convergence rates consistently observed in practice. In this paper, we resolve this discrepancy by proving that under a mild assumption on Neural Tangent Kernel (NTK) sta…

Learning to Solve Parametric Mixed-Integer Optimal Control Problems via Differentiable Predictive Control
J\'an Boldock\'y, Shahriar Dadras Javan, Martin Gulan, Martin M\"onnigmann, J\'an Drgo\v{n}a
https://arxiv.org/abs/2506.19646

Learning to Solve Parametric Mixed-Integer Optimal Control Problems via Differentiable Predictive Control
We propose a novel approach to solving input- and state-constrained parametric mixed-integer optimal control problems using Differentiable Predictive Control (DPC). Our approach follows the differentiable programming paradigm by learning an explicit neural policy that maps control parameters to integer- and continuous-valued decision variables. This policy is optimized via stochastic gradient descent by differentiating the quadratic model predictive control objective through the closed-loop fin…

A Study of Hybrid and Evolutionary Metaheuristics for Single Hidden Layer Feedforward Neural Network Architecture
Gautam Siddharth Kashyap, Md Tabrez Nafis, Samar Wazir
https://arxiv.org/abs/2506.15737

A Study of Hybrid and Evolutionary Metaheuristics for Single Hidden Layer Feedforward Neural Network Architecture
Training Artificial Neural Networks (ANNs) with Stochastic Gradient Descent (SGD) frequently encounters difficulties, including substantial computing expense and the risk of converging to local optima, attributable to its dependence on partial weight gradients. Therefore, this work investigates Particle Swarm Optimization (PSO) and Genetic Algorithms (GAs) - two population-based Metaheuristic Optimizers (MHOs) - as alternatives to SGD to mitigate these constraints. A hybrid PSO-SGD strategy is …

On Universality of Non-Separable Approximate Message Passing Algorithms
Max Lovig, Tianhao Wang, Zhou Fan
https://arxiv.org/abs/2506.23010 https://

On Universality of Non-Separable Approximate Message Passing Algorithms
Mean-field characterizations of first-order iterative algorithms -- including Approximate Message Passing (AMP), stochastic and proximal gradient descent, and Langevin diffusions -- have enabled a precise understanding of learning dynamics in many statistical applications. For algorithms whose non-linearities have a coordinate-separable form, it is known that such characterizations enjoy a degree of universality with respect to the underlying data distribution. However, mean-field characterizat…

Worst-case convergence analysis of relatively inexact gradient descent on smooth convex functions
Pierre Vernimmen, Fran\c{c}ois Glineur
https://arxiv.org/abs/2506.17145

Worst-case convergence analysis of relatively inexact gradient descent on smooth convex functions
We consider the classical gradient descent algorithm with constant stepsizes, where some error is introduced in the computation of each gradient. More specifically, we assume some relative bound on the inexactness, in the sense that the norm of the difference between the true gradient and its approximate value is bounded by a certain fraction of the gradient norm. This paper presents a worst-case convergence analysis of this so-called relatively inexact gradient descent on smooth convex functio…

Faithful-Newton Framework: Bridging Inner and Outer Solvers for Enhanced Optimization
Alexander Lim, Fred Roosta
https://arxiv.org/abs/2506.13154 https://

Faithful-Newton Framework: Bridging Inner and Outer Solvers for Enhanced Optimization
While Newton-type methods are known for their fast local convergence and strong empirical performance, achieving theoretically favorable global convergence compared to first-order methods remains a key challenge. For instance, surprisingly, for simple strongly convex problems, no straightforward variant of Newton's method matches the global complexity of gradient descent. Although sophisticated variants can improve iteration complexity over gradient descent for various problems, they often invo…

Replaced article(s) found for cs.LG. https://arxiv.org/list/cs.LG/new
[2/4]:
- Convergence Properties of Natural Gradient Descent for Minimizing KL Divergence
Adwait Datar, Nihat Ay

Efficient Feedback Design for Unsourced Random Access with Integrated Sensing and Communication
Mohammad Javad Ahmadi, Mohammad Kazemi, Rafael F. Schaefer
https://arxiv.org/abs/2506.20262

Efficient Feedback Design for Unsourced Random Access with Integrated Sensing and Communication
We consider an unsourced random access (URA) system enhanced with a feedback mechanism that serves both communication and sensing tasks. While traditional URA systems do not incorporate feedback, we propose a novel feedback signal design that announces the decoding status of users and simultaneously enables target sensing. To design this dual-purpose feedback, we introduce a modified projected gradient descent algorithm that minimizes a weighted combination of communication and sensing errors. …

Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon
Tongtong Liang, Dan Qiao, Yu-Xiang Wang, Rahul Parhi
https://arxiv.org/abs/2506.20779

Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon
We study the implicit bias of flatness / low (loss) curvature and its effects on generalization in two-layer overparameterized ReLU networks with multivariate inputs -- a problem well motivated by the minima stability and edge-of-stability phenomena in gradient-descent training. Existing work either requires interpolation or focuses only on univariate inputs. This paper presents new and somewhat surprising theoretical results for multivariate inputs. On two natural settings (1) generalization g…

Beyond Rate Coding: Surrogate Gradients Enable Spike Timing Learning in Spiking Neural Networks
Ziqiao Yu, Pengfei Sun, Dan F. M. Goodman
https://arxiv.org/abs/2507.16043 https:…

Beyond Rate Coding: Surrogate Gradients Enable Spike Timing Learning in Spiking Neural Networks
We investigate the extent to which Spiking Neural Networks (SNNs) trained with Surrogate Gradient Descent (Surrogate GD), with and without delay learning, can learn from precise spike timing beyond firing rates. We first design synthetic tasks isolating intra-neuron inter-spike intervals and cross-neuron synchrony under matched spike counts. On more complex spike-based speech recognition datasets (Spiking Heidelberg Digits (SHD) and Spiking Speech Commands (SSC), we construct variants where spi…

Information Entropy-Based Scheduling for Communication-Efficient Decentralized Learning
Jaiprakash Nagar, Zheng Chen, Marios Kountouris, Photios A. Stavrou
https://arxiv.org/abs/2507.17426

Information Entropy-Based Scheduling for Communication-Efficient Decentralized Learning
This paper addresses decentralized stochastic gradient descent (D-SGD) over resource-constrained networks by introducing node-based and link-based scheduling strategies to enhance communication efficiency. In each iteration of the D-SGD algorithm, only a few disjoint subsets of nodes or links are randomly activated, subject to a given communication cost constraint. We propose a novel importance metric based on information entropy to determine node and link scheduling probabilities. We validate …

Stochastic gradient with least-squares control variates
Fabio Nobile, Matteo Raviola, Nathan Schaeffer
https://arxiv.org/abs/2507.20981 https://arxiv.org/p…

Stochastic gradient with least-squares control variates
The stochastic gradient descent (SGD) method is a widely used approach for solving stochastic optimization problems, but its convergence is typically slow. Existing variance reduction techniques, such as SAGA, improve convergence by leveraging stored gradient information; however, they are restricted to settings where the objective functional is a finite sum, and their performance degrades when the number of terms in the sum is large. In this work, we propose a novel approach which is well suit…

Gradient descent avoids strict saddles with a simple line-search method too
Andreea-Alexandra Mu\c{s}at, Nicolas Boumal
https://arxiv.org/abs/2507.13804 ht…

Gradient descent avoids strict saddles with a simple line-search method too
It is known that gradient descent (GD) on a $C^2$ cost function generically avoids strict saddle points when using a small, constant step size. However, no such guarantee existed for GD with a line-search method. We provide one for a modified version of the standard Armijo backtracking method with generic, arbitrarily large initial step size. In contrast to previous works, our analysis does not require a globally Lipschitz gradient. We extend this to the Riemannian setting (RGD), assuming the…

Power-Constrained Policy Gradient Methods for LQR
Ashwin Verma, Aritra Mitra, Lintao Ye, Vijay Gupta
https://arxiv.org/abs/2507.15806 https://

Power-Constrained Policy Gradient Methods for LQR
Consider a discrete-time Linear Quadratic Regulator (LQR) problem solved using policy gradient descent when the system matrices are unknown. The gradient is transmitted across a noisy channel over a finite time horizon using analog communication by a transmitter with an average power constraint. This is a simple setup at the intersection of reinforcement learning and networked control systems. We first consider a communication-constrained optimization framework, where gradient descent is applie…

Random feature approximation for general spectral methods
Mike Nguyen, Nicole M\"ucke
https://arxiv.org/abs/2506.16283 https://a…

Random feature approximation for general spectral methods
Random feature approximation is arguably one of the most widely used techniques for kernel methods in large-scale learning algorithms. In this work, we analyze the generalization properties of random feature methods, extending previous results for Tikhonov regularization to a broad class of spectral regularization techniques. This includes not only explicit methods but also implicit schemes such as gradient descent and accelerated algorithms like the Heavy-Ball and Nesterov method. Through this…

An Enhanced Privacy-preserving Federated Few-shot Learning Framework for Respiratory Disease Diagnosis
Ming Wang, Zhaoyang Duan, Dong Xue, Fangzhou Liu, Zhongheng Zhang
https://arxiv.org/abs/2507.08050 https://arxiv.org/pdf/2507.08050 https://arxiv.org/html/2507.08050
arXiv:2507.08050v1 Announce Type: new
Abstract: The labor-intensive nature of medical data annotation presents a significant challenge for respiratory disease diagnosis, resulting in a scarcity of high-quality labeled datasets in resource-constrained settings. Moreover, patient privacy concerns complicate the direct sharing of local medical data across institutions, and existing centralized data-driven approaches, which rely on amounts of available data, often compromise data privacy. This study proposes a federated few-shot learning framework with privacy-preserving mechanisms to address the issues of limited labeled data and privacy protection in diagnosing respiratory diseases. In particular, a meta-stochastic gradient descent algorithm is proposed to mitigate the overfitting problem that arises from insufficient data when employing traditional gradient descent methods for neural network training. Furthermore, to ensure data privacy against gradient leakage, differential privacy noise from a standard Gaussian distribution is integrated into the gradients during the training of private models with local data, thereby preventing the reconstruction of medical images. Given the impracticality of centralizing respiratory disease data dispersed across various medical institutions, a weighted average algorithm is employed to aggregate local diagnostic models from different clients, enhancing the adaptability of a model across diverse scenarios. Experimental results show that the proposed method yields compelling results with the implementation of differential privacy, while effectively diagnosing respiratory diseases using data from different structures, categories, and distributions.
toXiv_bot_toot

Keep the beat going: Automatic drum transcription with momentum
Alisha L. Foster, Robert J. Webber
https://arxiv.org/abs/2507.12596 https://

Keep the beat going: Automatic drum transcription with momentum
A simple, interpretable way to perform automatic drum transcription is by factoring the magnitude spectrogram of a recorded musical piece using a partially fixed nonnegative matrix factorization. There are two natural ways to optimize the nonnegative matrix factorization, including a multiplicative update rule and projected gradient descent with momentum. The methods differ in their empirical accuracies and theoretical convergence guarantees. This paper summarizes the methods and their time com…

High Probability Convergence of Distributed Clipped Stochastic Gradient Descent with Heavy-tailed Noise
Yuchen Yang, Kaihong Lu, Long Wang
https://arxiv.org/abs/2506.11647

High Probability Convergence of Distributed Clipped Stochastic Gradient Descent with Heavy-tailed Noise
In this paper, the problem of distributed optimization is studied via a network of agents. Each agent only has access to a noisy gradient of its own objective function, and can communicate with its neighbors via a network. To handle this problem, a distributed clipped stochastic gradient descent algorithm is proposed, and the high probability convergence of the algorithm is studied. Existing works on distributed algorithms involving stochastic gradients only consider the light-tailed noises. Di…

Replaced article(s) found for math.OC. https://arxiv.org/list/math.OC/new
[1/1]:
- Low-rank optimization methods based on projected-projected gradient descent that accumulate at Bo...
Guillaume Olikier, Kyle A. Gallivan, P. -A. Absil

This https://arxiv.org/abs/2505.08408 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

A three-term Polak-Ribière-Polyak conjugate gradient method for vector optimization
A three-term Polak-Ribière-Polyak conjugate gradient method is first proposed in this paper for solving vector optimization problems. This method can autonomously generate descent directions independent of line search procedures, while retaining the conjugate property. The global convergence of the proposed scheme is established by the generalized Wolfe line search procedure without self-adjusting strategies, regular restarts and convexity assumptions. Finally, numerical experiments illustrati…

This https://arxiv.org/abs/2505.08408 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

A three-term Polak-Ribière-Polyak conjugate gradient method for vector optimization
A three-term Polak-Ribière-Polyak conjugate gradient method is first proposed in this paper for solving vector optimization problems. This method can autonomously generate descent directions independent of line search procedures, while retaining the conjugate property. The global convergence of the proposed scheme is established by the generalized Wolfe line search procedure without self-adjusting strategies, regular restarts and convexity assumptions. Finally, numerical experiments illustrati…

Replaced article(s) found for stat.CO. https://arxiv.org/list/stat.CO/new
[1/1]:
- Error bounds for particle gradient descent, and extensions of the log-Sobolev and Talagrand inequ...
Rocco Caprio, Juan Kuntz, Samuel Power, Adam M. Johansen

Primal-Dual Coordinate Descent for Nonconvex-Nonconcave Saddle Point Problems Under the Weak MVI Assumption
Iyad Walwil, Olivier Fercoq
https://arxiv.org/abs/2506.15597

Primal-Dual Coordinate Descent for Nonconvex-Nonconcave Saddle Point Problems Under the Weak MVI Assumption
We introduce two novel primal-dual algorithms for addressing nonconvex, nonconcave, and nonsmooth saddle point problems characterized by the weak Minty Variational Inequality (MVI). The first algorithm, Nonconvex-Nonconcave Primal-Dual Hybrid Gradient (NC-PDHG), extends the well-known Primal-Dual Hybrid Gradient (PDHG) method to this challenging problem class. The second algorithm, Nonconvex-Nonconcave Stochastic Primal-Dual Hybrid Gradient (NC-SPDHG), incorporates a randomly extrapolated prima…

Efficient Online Mirror Descent Stochastic Approximation for Multi-Stage Stochastic Programming
Junhui Zhang, Patrick Jaillet
https://arxiv.org/abs/2506.15392

Efficient Online Mirror Descent Stochastic Approximation for Multi-Stage Stochastic Programming
We study the unconstrained and the minimax saddle point variants of the convex multi-stage stochastic programming problem, where consecutive decisions are coupled through the objective functions, rather than through the constraints. Based on the analysis of deterministic mirror descent algorithms with inexact gradients, we introduce the idea of \textit{stochastic conditional gradient oracles}, a multi-stage analog of the stochastic gradient oracles used in (classical) stochastic programming. We…

This https://arxiv.org/abs/2401.06738 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

(Accelerated) Noise-adaptive Stochastic Heavy-Ball Momentum
Stochastic heavy ball momentum (SHB) is commonly used to train machine learning models, and often provides empirical improvements over stochastic gradient descent. By primarily focusing on strongly-convex quadratics, we aim to better understand the theoretical advantage of SHB and subsequently improve the method. For strongly-convex quadratics, Kidambi et al. (2018) show that SHB (with a mini-batch of size $1$) cannot attain accelerated convergence, and hence has no theoretical benefit over SGD…

Numerical Design of Optimized First-Order Algorithms
Yassine Kamri, Julien M. Hendrickx, Fran\c{c}ois Glineur
https://arxiv.org/abs/2507.20773 https://arxi…

Numerical Design of Optimized First-Order Algorithms
We derive several numerical methods for designing optimized first-order algorithms in unconstrained convex optimization settings. Our methods are based on the Performance Estimation Problem (PEP) framework, which casts the worst-case analysis of optimization algorithms as an optimization problem itself. We benchmark our methods against existing approaches in the literature on the task of optimizing the step sizes of memoryless gradient descent (which uses only the current gradient for updates) …

Replaced article(s) found for math.OC. https://arxiv.org/list/math.OC/new
[1/2]:
- Projected gradient descent accumulates at Bouligand stationary points
Guillaume Olikier, Ir\`ene Waldspurger

Non-smooth stochastic gradient descent using smoothing functions
Tommaso Giovannelli, Jingfu Tan, Luis Nunes Vicente
https://arxiv.org/abs/2507.10901 https…

Non-smooth stochastic gradient descent using smoothing functions
In this paper, we address stochastic optimization problems involving a composition of a non-smooth outer function and a smooth inner function, a formulation frequently encountered in machine learning and operations research. To deal with the non-differentiability of the outer function, we approximate the original non-smooth function using smoothing functions, which are continuously differentiable and approach the original function as a smoothing parameter goes to zero (at the price of increasin…

Deep Equilibrium models for Poisson Imaging Inverse problems via Mirror Descent
Christian Daniele, Silvia Villa, Samuel Vaiter, Luca Calatroni
https://arxiv.org/abs/2507.11461

Deep Equilibrium models for Poisson Imaging Inverse problems via Mirror Descent
Deep Equilibrium Models (DEQs) are implicit neural networks with fixed points, which have recently gained attention for learning image regularization functionals, particularly in settings involving Gaussian fidelities, where assumptions on the forward operator ensure contractiveness of standard (proximal) Gradient Descent operators. In this work, we extend the application of DEQs to Poisson inverse problems, where the data fidelity term is more appropriately modeled by the Kullback-Leibler dive…

Empirical and computer-aided robustness analysis of long-step and accelerated methods in smooth convex optimization
Pierre Vernimmen, Fran\c{c}ois Glineur
https://arxiv.org/abs/2506.09730

Empirical and computer-aided robustness analysis of long-step and accelerated methods in smooth convex optimization
This work assesses both empirically and theoretically, using the performance estimation methodology, how robust different first-order optimization methods are when subject to relative inexactness in their gradient computations. Relative inexactness occurs, for example, when compressing the gradient using fewer bits of information, which happens when dealing with large-scale problems on GPUs. Three major families of methods are analyzed: constant step gradient descent, long-step methods, and acc…

Learning Acceleration Algorithms for Fast Parametric Convex Optimization with Certified Robustness
Rajiv Sambharya, Jinho Bok, Nikolai Matni, George Pappas
https://arxiv.org/abs/2507.16264

Learning Acceleration Algorithms for Fast Parametric Convex Optimization with Certified Robustness
We develop a machine-learning framework to learn hyperparameter sequences for accelerated first-order methods (e.g., the step size and momentum sequences in accelerated gradient descent) to quickly solve parametric convex optimization problems with certified robustness. We obtain a strong form of robustness guarantee -- certification of worst-case performance over all parameters within a set after a given number of iterations -- through regularization-based training. The regularization term is …

Last-Iterate Complexity of SGD for Convex and Smooth Stochastic Problems
Guillaume Garrigos, Daniel Cortild, Lucas Ketels, Juan Peypouquet
https://arxiv.org/abs/2507.14122

Last-Iterate Complexity of SGD for Convex and Smooth Stochastic Problems
Most results on Stochastic Gradient Descent (SGD) in the convex and smooth setting are presented under the form of bounds on the ergodic function value gap. It is an open question whether bounds can be derived directly on the last iterate of SGD in this context. Recent advances suggest that it should be possible. For instance, it can be achieved by making the additional, yet unverifiable, assumption that the variance of the stochastic gradients is uniformly bounded. In this paper, we show that …

An Extended Variational Barzilai-Borwein Method
Xin Xu
https://arxiv.org/abs/2506.12731 https://arxiv.org/pdf/2506.12731

An Extended Variational Barzilai-Borwein Method
The Barzilai-Borwein (BB) method is an effective gradient descent algorithm for solving unconstrained optimization problems. Based on the observation of two classical BB step sizes, by constructing a variational least squares model, we propose a novel class of BB step sizes, each of which still retains the quasi-Newton property, with the original two BB step sizes being their two extreme cases.