Tootfinder

This https://arxiv.org/abs/2311.00521 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Gaussian smoothing gradient descent for minimizing functions (GSmoothGD)
This work analyzes the convergence of a class of smoothing-based gradient descent methods when applied to optimization problems. In particular, Gaussian smoothing is employed to define a nonlocal gradient that reduces high-frequency noise, small variations, and rapid fluctuations in the computation of the descent directions while preserving the structure and features of the loss landscape. The resulting Gaussian smoothing gradient descent (GSmoothGD) approach can facilitate gradient descent in …

This https://arxiv.org/abs/2311.00521 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Gaussian smoothing gradient descent for minimizing functions (GSmoothGD)
This work analyzes the convergence of a class of smoothing-based gradient descent methods when applied to optimization problems. In particular, Gaussian smoothing is employed to define a nonlocal gradient that reduces high-frequency noise, small variations, and rapid fluctuations in the computation of the descent directions while preserving the structure and features of the loss landscape. The resulting Gaussian smoothing gradient descent (GSmoothGD) approach can facilitate gradient descent in …

Wirtinger gradient descent methods for low-dose Poisson phase retrieval
Benedikt Diederichs, Frank Filbir, Patricia R\"omer
https://arxiv.org/abs/2403.18527

Wirtinger gradient descent methods for low-dose Poisson phase retrieval
The problem of phase retrieval has many applications in the field of optical imaging. Motivated by imaging experiments with biological specimens, we primarily consider the setting of low-dose illumination where Poisson noise plays the dominant role. In this paper, we discuss gradient descent algorithms based on different loss functions adapted to data affected by Poisson noise, in particular in the low-dose regime. Starting from the maximum log-likelihood function for the Poisson distribution, …

Failures and Successes of Cross-Validation for Early-Stopped Gradient Descent
Pratik Patil, Yuchen Wu, Ryan J. Tibshirani
https://arxiv.org/abs/2402.16793 …

Failures and Successes of Cross-Validation for Early-Stopped Gradient Descent
We analyze the statistical properties of generalized cross-validation (GCV) and leave-one-out cross-validation (LOOCV) applied to early-stopped gradient descent (GD) in high-dimensional least squares regression. We prove that GCV is generically inconsistent as an estimator of the prediction risk of early-stopped GD, even for a well-specified linear model with isotropic features. In contrast, we show that LOOCV converges uniformly along the GD trajectory to the prediction risk. Our theory requir…

Faster Convergence for Transformer Fine-tuning with Line Search Methods
Philip Kenneweg, Leonardo Galli, Tristan Kenneweg, Barbara Hammer
https://arxiv.org/abs/2403.18506

Faster Convergence for Transformer Fine-tuning with Line Search Methods
Recent works have shown that line search methods greatly increase performance of traditional stochastic gradient descent methods on a variety of datasets and architectures [1], [2]. In this work we succeed in extending line search methods to the novel and highly popular Transformer architecture and dataset domains in natural language processing. More specifically, we combine the Armijo line search with the Adam optimizer and extend it by subdividing the networks architecture into sensible units…

This https://arxiv.org/abs/2303.10599 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_sta…

Convergence Analysis of Stochastic Gradient Descent with MCMC Estimators
Understanding stochastic gradient descent (SGD) and its variants is essential for machine learning. However, most of the preceding analyses are conducted under amenable conditions such as unbiased gradient estimator and bounded objective functions, which does not encompass many sophisticated applications, such as variational Monte Carlo, entropy-regularized reinforcement learning and variational inference. In this paper, we consider the SGD algorithm that employ the Markov Chain Monte Carlo (MC…

This https://arxiv.org/abs/2311.18426 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Convergence Analysis of Fractional Gradient Descent
Fractional derivatives are a well-studied generalization of integer order derivatives. Naturally, for optimization, it is of interest to understand the convergence properties of gradient descent using fractional derivatives. Convergence analysis of fractional gradient descent is currently limited both in the methods analyzed and the settings analyzed. This paper aims to fill in these gaps by analyzing variations of fractional gradient descent in smooth and convex, smooth and strongly convex, an…

This https://arxiv.org/abs/2402.14423 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_qu…

The Universe as a Learning System
At its microscopic level, the universe follows the laws of quantum mechanics. Focusing on the quantum trajectories of particles as followed from the hydrodynamical formulation of quantum mechanics, we propose that under general requirements, quantum systems follow a disrupted version of the gradient descent model, a basic machine learning algorithm, where the learning is distorted due to the self-organizing process of the quantum system. Such a learning process is possible only when we assume d…

MegaParticles: Range-based 6-DoF Monte Carlo Localization with GPU-Accelerated Stein Particle Filter
Kenji Koide, Shuji Oishi, Masashi Yokozuka, Atsuhiko Banno
https://arxiv.org/abs/2404.16370

MegaParticles: Range-based 6-DoF Monte Carlo Localization with GPU-Accelerated Stein Particle Filter
This paper presents a 6-DoF range-based Monte Carlo localization method with a GPU-accelerated Stein particle filter. To update a massive amount of particles, we propose a Gauss-Newton-based Stein variational gradient descent (SVGD) with iterative neighbor particle search. This method uses SVGD to collectively update particle states with gradient and neighborhood information, which provides efficient particle sampling. For an efficient neighbor particle search, it uses locality sensitive hashin…

This https://arxiv.org/abs/2311.13967 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_ees…

Unconstrained learning of networked nonlinear systems via free parametrization of stable interconnected operators
This paper characterizes a new parametrization of nonlinear networked incrementally $L_2$-bounded operators in discrete time. The distinctive novelty is that our parametrization is \emph{free} -- that is, a sparse large-scale operator with bounded incremental $L_2$ gain is obtained for any choice of the real values of our parameters. This property allows one to freely search over optimal parameters via unconstrained gradient descent, enabling direct applications in large-scale optimal control a…

Organizing Physics with Open Energy-Driven Systems
Matteo Capucci, Owen Lynch, David I. Spivak
https://arxiv.org/abs/2404.16140 https://

Organizing Physics with Open Energy-Driven Systems
Organizing physics has been a long-standing preoccupation of applied category theory, going back at least to Lawvere. We contribute to this research thread by noticing that Hamiltonian mechanics and gradient descent depend crucially on a consistent choice of transformation -- which we call a reaction structure -- from the cotangent bundle to the tangent bundle. We then construct a compositional theory of reaction structures. Reaction-based systems offer a different perspective on composition in…

Improving Line Search Methods for Large Scale Neural Network Training
Philip Kenneweg, Tristan Kenneweg, Barbara Hammer
https://arxiv.org/abs/2403.18519 ht…

Improving Line Search Methods for Large Scale Neural Network Training
In recent studies, line search methods have shown significant improvements in the performance of traditional stochastic gradient descent techniques, eliminating the need for a specific learning rate schedule. In this paper, we identify existing issues in state-of-the-art line search methods, propose enhancements, and rigorously evaluate their effectiveness. We test these methods on larger datasets and more complex data domains than before. Specifically, we improve the Armijo line search by inte…

This https://arxiv.org/abs/2402.11858 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_sta…

Stochastic Hessian Fittings on Lie Groups
This paper studies the fitting of Hessian or its inverse for stochastic optimizations using a Hessian fitting criterion from the preconditioned stochastic gradient descent (PSGD) method, which is intimately related to many commonly used second order and adaptive gradient optimizers, e.g., BFGS, Gaussian-Newton and natural gradient descent, AdaGrad, etc. Our analyses reveal the efficiency and reliability differences among a wide range of preconditioner fitting methods, from closed-form to iterat…

On the Variational Interpretation of Mirror Play in Monotone Games
Yunian Pan, Tao Li, Quanyan Zhu
https://arxiv.org/abs/2403.15636 https://

On the Variational Interpretation of Mirror Play in Monotone Games
Mirror play (MP) is a well-accepted primal-dual multi-agent learning algorithm where all agents simultaneously implement mirror descent in a distributed fashion. The advantage of MP over vanilla gradient play lies in its usage of mirror maps that better exploit the geometry of decision domains. Despite extensive literature dedicated to the asymptotic convergence of MP to equilibrium, the understanding of the finite-time behavior of MP before reaching equilibrium is still rudimentary. To facilit…

This https://arxiv.org/abs/2402.12493 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

On Averaging and Extrapolation for Gradient Descent
This work considers the effect of averaging, and more generally extrapolation, of the iterates of gradient descent in smooth convex optimization. After running the method, rather than reporting the final iterate, one can report either a convex combination of the iterates (averaging) or a generic combination of the iterates (extrapolation). For several common stepsize sequences, including recently developed accelerated periodically long stepsize schemes, we show averaging cannot improve gradient…

This https://arxiv.org/abs/2401.11176 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_ees…

Data-Driven Target Localization: Benchmarking Gradient Descent Using the Cramer-Rao Bound
In modern radar systems, precise target localization using azimuth and velocity estimation is paramount. Traditional unbiased estimation methods have utilized gradient descent algorithms to reach the theoretical limits of the Cramer Rao Bound (CRB) for the error of the parameter estimates. As an extension, we demonstrate on a realistic simulated example scenario that our earlier presented data-driven neural network model outperforms these traditional methods, yielding improved accuracies in tar…

MegaParticles: Range-based 6-DoF Monte Carlo Localization with GPU-Accelerated Stein Particle Filter
Kenji Koide, Shuji Oishi, Masashi Yokozuka, Atsuhiko Banno
https://arxiv.org/abs/2404.16370

MegaParticles: Range-based 6-DoF Monte Carlo Localization with GPU-Accelerated Stein Particle Filter
This paper presents a 6-DoF range-based Monte Carlo localization method with a GPU-accelerated Stein particle filter. To update a massive amount of particles, we propose a Gauss-Newton-based Stein variational gradient descent (SVGD) with iterative neighbor particle search. This method uses SVGD to collectively update particle states with gradient and neighborhood information, which provides efficient particle sampling. For an efficient neighbor particle search, it uses locality sensitive hashin…

JaxDecompiler: Redefining Gradient-Informed Software Design
Pierrick Pochelu
https://arxiv.org/abs/2403.10571 https://arxiv.org/pdf/2…

JaxDecompiler: Redefining Gradient-Informed Software Design
Among numerical libraries capable of computing gradient descent optimization, JAX stands out by offering more features, accelerated by an intermediate representation known as Jaxpr language. However, editing the Jaxpr code is not directly possible. This article introduces JaxDecompiler, a tool that transforms any JAX function into an editable Python code, especially useful for editing the JAX function generated by the gradient function. JaxDecompiler simplifies the processes of reverse engineer…

Algorithmic unfolding for image reconstruction and localization problems in fluorescence microscopy
Silvia Bonettini, Luca Calatroni, Danilo Pezzi, Marco Prato
https://arxiv.org/abs/2403.17506

Algorithmic unfolding for image reconstruction and localization problems in fluorescence microscopy
We propose an unfolded accelerated projected-gradient descent procedure to estimate model and algorithmic parameters for image super-resolution and molecule localization problems in image microscopy. The variational lower-level constraint enforces sparsity of the solution and encodes different noise statistics (Gaussian, Poisson), while the upper-level cost assesses optimality w.r.t.~the task considered. In more detail, a standard $\ell_2$ cost is considered for image reconstruction (e.g., deco…

Projected Gradient Descent for Spectral Compressed Sensing via Symmetric Hankel Factorization
Jinsheng Li, Wei Cui, Xu Zhang
https://arxiv.org/abs/2403.09031

Projected Gradient Descent for Spectral Compressed Sensing via Symmetric Hankel Factorization
Current spectral compressed sensing methods via Hankel matrix completion employ symmetric factorization to demonstrate the low-rank property of the Hankel matrix. However, previous non-convex gradient methods only utilize asymmetric factorization to achieve spectral compressed sensing. In this paper, we propose a novel nonconvex projected gradient descent method for spectral compressed sensing via symmetric factorization named Symmetric Hankel Projected Gradient Descent (SHGD), which updates on…

This https://arxiv.org/abs/2306.10529 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Dropout Regularization Versus $\ell_2$-Penalization in the Linear Model
We investigate the statistical behavior of gradient descent iterates with dropout in the linear regression model. In particular, non-asymptotic bounds for the convergence of expectations and covariance matrices of the iterates are derived. The results shed more light on the widely cited connection between dropout and l2-regularization in the linear model. We indicate a more subtle relationship, owing to interactions between the gradient descent dynamics and the additional randomness induced by …

This https://arxiv.org/abs/2307.12943 has been replaced.
link: https://scholar.google.com/scholar?q=a

Gaussian Cooling and Dikin Walks: The Interior-Point Method for Logconcave Sampling
The connections between (convex) optimization and (logconcave) sampling have been considerably enriched in the past decade with many conceptual and mathematical analogies. For instance, the Langevin algorithm can be viewed as a sampling analogue of gradient descent and has condition-number-dependent guarantees on its performance. In the early 1990s, Nesterov and Nemirovski developed the Interior-Point Method (IPM) for convex optimization based on self-concordant barriers, providing efficient al…

How Transformers Learn Causal Structure with Gradient Descent
Eshaan Nichani, Alex Damian, Jason D. Lee
https://arxiv.org/abs/2402.14735 https://

How Transformers Learn Causal Structure with Gradient Descent
The incredible success of transformers on sequence modeling tasks can be largely attributed to the self-attention mechanism, which allows information to be transferred between different parts of a sequence. Self-attention allows transformers to encode causal structure which makes them particularly suitable for sequence modeling. However, the process by which transformers learn such causal structure via gradient-based training algorithms remains poorly understood. To better understand this proce…

Global convergence of iterative solvers for problems of nonlinear magnetostatics
Herbert Egger, Felix Engertsberger, Bogdan Radu
https://arxiv.org/abs/2403.18520

Global convergence of iterative solvers for problems of nonlinear magnetostatics
We consider the convergence of iterative solvers for problems of nonlinear magnetostatics. Using the equivalence to an underlying minimization problem, we can establish global linear convergence of a large class of methods, including the damped Newton-method, fixed-point iteration, and the Kacanov iteration, which can all be interpreted as generalized gradient descent methods. Armijo backtracking isconsidered for an adaptive choice of the stepsize. The general assumptions required for our analy…

This https://arxiv.org/abs/2311.00944 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_sta…

Stochastic Smoothed Gradient Descent Ascent for Federated Minimax Optimization
In recent years, federated minimax optimization has attracted growing interest due to its extensive applications in various machine learning tasks. While Smoothed Alternative Gradient Descent Ascent (Smoothed-AGDA) has proved its success in centralized nonconvex minimax optimization, how and whether smoothing technique could be helpful in federated setting remains unexplored. In this paper, we propose a new algorithm termed Federated Stochastic Smoothed Gradient Descent Ascent (FESS-GDA), which…

Sub-Micrometer Particles Remote Detection in Enceladus Plume Based on Cassinis UV Spectrograph Data
Jan Kotlarz, Katarzyna Kubiak, Natalia Zalewska
https://arxiv.org/abs/2403.15727

Sub-Micrometer Particles Remote Detection in Enceladus Plume Based on Cassinis UV Spectrograph Data
Enceladus is the Saturnian satellite known to have water vapor erupting from its south pole region called Tiger Stripes. Data collected by Cassini Ultraviolet Imaging Spectrograph during Enceladus transiting Saturn allow us to estimate water plume absorption from 1115.35 - 1912.50 Angstrom and compare it to the Mie solutions of Maxwell equations for particles with a diameter in the range from 10 nm up to 2 um. The best fit performed using Gradient Descent method indicates a presence of submicro…

This https://arxiv.org/abs/2104.02857 has been replaced.
link: https://scholar.google.com/scholar?q=a

Soft-Label Anonymous Gastric X-ray Image Distillation
This paper presents a soft-label anonymous gastric X-ray image distillation method based on a gradient descent approach. The sharing of medical data is demanded to construct high-accuracy computer-aided diagnosis (CAD) systems. However, the large size of the medical dataset and privacy protection are remaining problems in medical data sharing, which hindered the research of CAD systems. The idea of our distillation method is to extract the valid information of the medical dataset and generate a…

This https://arxiv.org/abs/2403.07124 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_sta…

Stochastic gradient descent-based inference for dynamic network models with attractors
In Coevolving Latent Space Networks with Attractors (CLSNA) models, nodes in a latent space represent social actors, and edges indicate their dynamic interactions. Attractors are added at the latent level to capture the notion of attractive and repulsive forces between nodes, borrowing from dynamical systems theory. However, CLSNA reliance on MCMC estimation makes scaling difficult, and the requirement for nodes to be present throughout the study period limit practical applications. We address …

This https://arxiv.org/abs/2311.00944 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_sta…

Stochastic Smoothed Gradient Descent Ascent for Federated Minimax Optimization
In recent years, federated minimax optimization has attracted growing interest due to its extensive applications in various machine learning tasks. While Smoothed Alternative Gradient Descent Ascent (Smoothed-AGDA) has proved its success in centralized nonconvex minimax optimization, how and whether smoothing technique could be helpful in federated setting remains unexplored. In this paper, we propose a new algorithm termed Federated Stochastic Smoothed Gradient Descent Ascent (FESS-GDA), which…

This https://arxiv.org/abs/2306.17782 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csIT_…

Efficient Federated Low Rank Matrix Recovery via Alternating GD and Minimization: A Simple Proof
This note provides a significantly simpler and shorter proof of our sample complexity guarantee for solving the low rank column-wise sensing problem using the Alternating Gradient Descent (GD) and Minimization (AltGDmin) algorithm. AltGDmin was developed and analyzed for solving this problem in our recent work. We also provide an improved guarantee.

This https://arxiv.org/abs/2306.11246 has been replaced.
link: https://scholar.google.com/scholar?q=a

Neural Inventory Control in Networks via Hindsight Differentiable Policy Optimization
We argue that inventory management presents unique opportunities for reliably applying and evaluating deep reinforcement learning (DRL). Toward reliable application, we emphasize and test two techniques. The first is Hindsight Differentiable Policy Optimization (HDPO), which performs stochastic gradient descent to optimize policy performance while avoiding the need to repeatedly deploy randomized policies in the environment-as is common with generic policy gradient methods. Our second technique…

This https://arxiv.org/abs/2211.10777 has been replaced.
link: https://scholar.google.com/scholar?q=a

Non-Coherent Over-the-Air Decentralized Gradient Descent
Decentralized Gradient Descent (DGD) is a popular algorithm used to solve decentralized optimization problems in diverse domains such as remote sensing, distributed inference, multi-agent coordination, and federated learning. Yet, executing DGD over wireless systems affected by noise, fading and limited bandwidth presents challenges, requiring scheduling of transmissions to mitigate interference and the acquisition of topology and channel state information -- complex tasks in wireless decentral…

Generalized Gradient Descent is a Hypergraph Functor
Tyler Hanks, Matthew Klawonn, James Fairbanks
https://arxiv.org/abs/2403.19845 https://

Generalized Gradient Descent is a Hypergraph Functor
Cartesian reverse derivative categories (CRDCs) provide an axiomatic generalization of the reverse derivative, which allows generalized analogues of classic optimization algorithms such as gradient descent to be applied to a broad class of problems. In this paper, we show that generalized gradient descent with respect to a given CRDC induces a hypergraph functor from a hypergraph category of optimization problems to a hypergraph category of dynamical systems. The domain of this functor consists…

The Universe as a Learning System
Tomer Shushi
https://arxiv.org/abs/2402.14423 https://arxiv.org/pdf/2402.14423

The Universe as a Learning System
At its microscopic level, the universe follows the laws of quantum mechanics. Focusing on the quantum trajectories of particles as followed from the hydrodynamical formulation of quantum mechanics, we propose that under general requirements, quantum systems follow a disrupted version of the gradient descent model, a basic machine learning algorithm, where the learning is distorted due to the self-organizing process of the quantum system. Such a learning process is possible only when we assume d…

Analysing heavy-tail properties of Stochastic Gradient Descent by means of Stochastic Recurrence Equations
Ewa Damek, Sebastian Mentemeier
https://arxiv.org/abs/2403.13868

Analysing heavy-tail properties of Stochastic Gradient Descent by means of Stochastic Recurrence Equations
In recent works on the theory of machine learning, it has been observed that heavy tail properties of Stochastic Gradient Descent (SGD) can be studied in the probabilistic framework of stochastic recursions. In particular, Gürbüzbalaban et al. (arXiv:2006.04740) considered a setup corresponding to linear regression for which iterations of SGD can be modelled by a multivariate affine stochastic recursion $X_k=A_k X_{k-1}+B_k$, for independent and identically distributed pairs $(A_k, B_k)$, whe…

Riemannian Optimization and the Hartree-Fock Method
Caio O. da Silva
https://arxiv.org/abs/2403.15024 https://arxiv.org/pdf/2403.1502…

Riemannian Optimization and the Hartree-Fock Method
In the present work we studied a subfield of Applied Mathematics called Riemannian Optimization. The main goal of this subfield is to generalize algorithms, theorems and tools from Mathematical Optimization to the case in which the optimization problem is defined on a Riemannian manifold. As a case study, we implemented some of the main algorithms described in the literature (Gradient Descent, Newton-Raphson and Conjugate Gradient) to solve an optimization problem known as Hartree-Fock. This me…

Generalized Gradient Descent is a Hypergraph Functor
Tyler Hanks, Matthew Klawonn, James Fairbanks
https://arxiv.org/abs/2403.19845 https://

Generalized Gradient Descent is a Hypergraph Functor
Cartesian reverse derivative categories (CRDCs) provide an axiomatic generalization of the reverse derivative, which allows generalized analogues of classic optimization algorithms such as gradient descent to be applied to a broad class of problems. In this paper, we show that generalized gradient descent with respect to a given CRDC induces a hypergraph functor from a hypergraph category of optimization problems to a hypergraph category of dynamical systems. The domain of this functor consists…

This https://arxiv.org/abs/2103.15130 has been replaced.
link: https://scholar.google.com/scholar?q=a

Consensus-Based Optimization Methods Converge Globally
In this paper we study consensus-based optimization (CBO), which is a multi-agent metaheuristic derivative-free optimization method that can globally minimize nonconvex nonsmooth functions and is amenable to theoretical analysis. Based on an experimentally supported intuition that, on average, CBO performs a gradient descent of the squared Euclidean distance to the global minimizer, we devise a novel technique for proving the convergence to the global minimizer in mean-field law for a rich clas…

This https://arxiv.org/abs/2403.16829 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csLG_…

Convergence of a model-free entropy-regularized inverse reinforcement learning algorithm
Given a dataset of expert demonstrations, inverse reinforcement learning (IRL) aims to recover a reward for which the expert is optimal. This work proposes a model-free algorithm to solve entropy-regularized IRL problem. In particular, we employ a stochastic gradient descent update for the reward and a stochastic soft policy iteration update for the policy. Assuming access to a generative model, we prove that our algorithm is guaranteed to recover a reward for which the expert is $\varepsilon$-…

This https://arxiv.org/abs/2306.05896 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

One-step corrected projected stochastic gradient descent for statistical estimation
A generic, fast and asymptotically efficient method for parametric estimation is described. It is based on the projected stochastic gradient descent on the log-likelihood function corrected by a single step of the Fisher scoring algorithm. We show theoretically and by simulations that it is an interesting alternative to the usual stochastic gradient descent with averaging or the adaptative stochastic gradient descent.

This https://arxiv.org/abs/2305.12568 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Det-CGD: Compressed Gradient Descent with Matrix Stepsizes for Non-Convex Optimization
This paper introduces a new method for minimizing matrix-smooth non-convex objectives through the use of novel Compressed Gradient Descent (CGD) algorithms enhanced with a matrix-valued stepsize. The proposed algorithms are theoretically analyzed first in the single-node and subsequently in the distributed settings. Our theoretical results reveal that the matrix stepsize in CGD can capture the objective's structure and lead to faster convergence compared to a scalar stepsize. As a byproduct of …

This https://arxiv.org/abs/2309.09882 has been replaced.
link: https://scholar.google.com/scholar?q=a

Differentiable Boustrophedon Paths That Enable Optimization Via Gradient Descent
This paper introduces a differentiable representation for the optimization of boustrophedon path plans in convex polygons, explores an additional parameter of these path plans that can be optimized, discusses the properties of this representation that can be leveraged during the optimization process and shows that the previously published attempt at optimization of these path plans was too coarse to be practically useful. Experiments were conducted to show that this differentiable representatio…

Stochastic gradient descent-based inference for dynamic network models with attractors
Hancong Pan, Xiaojing Zhu, Cantay Caliskan, Dino P. Christenson, Konstantinos Spiliopoulos, Dylan Walker, Eric D. Kolaczyk
https://arxiv.org/abs/2403.07124

Stochastic gradient descent-based inference for dynamic network models with attractors
In Coevolving Latent Space Networks with Attractors (CLSNA) models, nodes in a latent space represent social actors, and edges indicate their dynamic interactions. Attractors are added at the latent level to capture the notion of attractive and repulsive forces between nodes, borrowing from dynamical systems theory. However, CLSNA reliance on MCMC estimation makes scaling difficult, and the requirement for nodes to be present throughout the study period limit practical applications. We address …

Attacking Large Language Models with Projected Gradient Descent
Simon Geisler, Tom Wollschl\"ager, M. H. I. Abdalla, Johannes Gasteiger, Stephan G\"unnemann
https://arxiv.org/abs/2402.09154

Attacking Large Language Models with Projected Gradient Descent
Current LLM alignment methods are readily broken through specifically crafted adversarial prompts. While crafting adversarial prompts using discrete optimization is highly effective, such attacks typically use more than 100,000 LLM calls. This high computational cost makes them unsuitable for, e.g., quantitative analyses and adversarial training. To remedy this, we revisit Projected Gradient Descent (PGD) on the continuously relaxed input prompt. Although previous attempts with ordinary gradien…

This https://arxiv.org/abs/2401.15663 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_ees…

Low-resolution Prior Equilibrium Network for CT Reconstruction
The unrolling method has been investigated for learning variational models in X-ray computed tomography. However, it has been observed that directly unrolling the regularization model through gradient descent does not produce satisfactory results. In this paper, we present a novel deep learning-based CT reconstruction model, where the low-resolution image is introduced to obtain an effective regularization term for improving the network`s robustness. Our approach involves constructing the backb…

Symmetry-guided gradient descent for quantum neural networks
Kaiming Bian, Shitao Zhang, Fei Meng, Wen Zhang, Oscar Dahlsten
https://arxiv.org/abs/2404.06108

Symmetry-guided gradient descent for quantum neural networks
Many supervised learning tasks have intrinsic symmetries, such as translational and rotational symmetry in image classifications. These symmetries can be exploited to enhance performance. We formulate the symmetry constraints into a concise mathematical form. We design two ways to adopt the constraints into the cost function, thereby shaping the cost landscape in favour of parameter choices which respect the given symmetry. Unlike methods that alter the neural network circuit ansatz to impose s…

The Blind Normalized Stein Variational Gradient Descent-Based Detection for Intelligent Massive Random Access
Xin Zhu, Ahmet Enis Cetin
https://arxiv.org/abs/2403.18846

The Blind Normalized Stein Variational Gradient Descent-Based Detection for Intelligent Massive Random Access
The lack of an efficient preamble detection algorithm remains a challenge for solving preamble collision problems in intelligent massive random access (RA) in practical communication scenarios. To solve this problem, we present a novel early preamble detection scheme based on a maximum likelihood estimation (MLE) model at the first step of the grant-based RA procedure. A novel blind normalized Stein variational gradient descent (SVGD)-based detector is proposed to obtain an approximate solution…

On the Convergence Rate of the Stochastic Gradient Descent (SGD) and application to a modified policy gradient for the Multi Armed Bandit
Stefana Anita, Gabriel Turinici
https://arxiv.org/abs/2402.06388

On the Convergence Rate of the Stochastic Gradient Descent (SGD) and application to a modified policy gradient for the Multi Armed Bandit
We present a self-contained proof of the convergence rate of the Stochastic Gradient Descent (SGD) when the learning rate follows an inverse time decays schedule; we next apply the results to the convergence of a modified form of policy gradient Multi-Armed Bandit (MAB) with $L2$ regularization.

Convergence and Trade-Offs in Riemannian Gradient Descent and Riemannian Proximal Point
David Mart\'inez-Rubio, Christophe Roux, Sebastian Pokutta
https://arxiv.org/abs/2403.10429

Convergence and Trade-Offs in Riemannian Gradient Descent and Riemannian Proximal Point
In this work, we analyze two of the most fundamental algorithms in geodesically convex optimization: Riemannian gradient descent and (possibly inexact) Riemannian proximal point. We quantify their rates of convergence and produce different variants with several trade-offs. Crucially, we show the iterates naturally stay in a ball around an optimizer, of radius depending on the initial distance and, in some cases, on the curvature. In contrast, except for limited cases, previous works bounded the…

Tractable Local Equilibria in Non-Concave Games
Yang Cai, Constantinos Daskalakis, Haipeng Luo, Chen-Yu Wei, Weiqiang Zheng
https://arxiv.org/abs/2403.08171

Tractable Local Equilibria in Non-Concave Games
While Online Gradient Descent and other no-regret learning procedures are known to efficiently converge to coarse correlated equilibrium in games where each agent's utility is concave in their own strategy, this is not the case when the utilities are non-concave, a situation that is common in machine learning applications where the agents' strategies are parameterized by deep neural networks, or the agents' utilities are computed by a neural network, or both. Indeed, non-concave games present a…

Symmetry-guided gradient descent for quantum neural networks
Kaiming Bian, Shitao Zhang, Fei Meng, Wen Zhang, Oscar Dahlsten
https://arxiv.org/abs/2404.06108

Symmetry-guided gradient descent for quantum neural networks
Many supervised learning tasks have intrinsic symmetries, such as translational and rotational symmetry in image classifications. These symmetries can be exploited to enhance performance. We formulate the symmetry constraints into a concise mathematical form. We design two ways to adopt the constraints into the cost function, thereby shaping the cost landscape in favour of parameter choices which respect the given symmetry. Unlike methods that alter the neural network circuit ansatz to impose s…

This https://arxiv.org/abs/2401.11176 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_ees…

Data-Driven Target Localization: Benchmarking Gradient Descent Using the Cramér-Rao Bound
In modern radar systems, precise target localization using azimuth and velocity estimation is paramount. Traditional unbiased estimation methods have utilized gradient descent algorithms to reach the theoretical limits of the Cramer Rao Bound (CRB) for the error of the parameter estimates. As an extension, we demonstrate on a realistic simulated example scenario that our earlier presented data-driven neural network model outperforms these traditional methods, yielding improved accuracies in tar…

Generative Adversarial Network with Soft-Dynamic Time Warping and Parallel Reconstruction for Energy Time Series Anomaly Detection
Hardik Prabhu, Jayaraman Valadi, Pandarasamy Arjunan
https://arxiv.org/abs/2402.14384

Generative Adversarial Network with Soft-Dynamic Time Warping and Parallel Reconstruction for Energy Time Series Anomaly Detection
In this paper, we employ a 1D deep convolutional generative adversarial network (DCGAN) for sequential anomaly detection in energy time series data. Anomaly detection involves gradient descent to reconstruct energy sub-sequences, identifying the noise vector that closely generates them through the generator network. Soft-DTW is used as a differentiable alternative for the reconstruction loss and is found to be superior to Euclidean distance. Combining reconstruction loss and the latent space's …

Projected Block Coordinate Descent for sparse spike estimation
Pierre-Jean B\'enardIMB, Yann TraonmilinIMB, Jean Fran\c{c}ois AujolIMB
https://arxiv.org/abs/2402.12021

Projected Block Coordinate Descent for sparse spike estimation
We consider the problem of recovering off-the-grid spikes from linear measurements. The state of the art Over-Parametrized Continuous Orthogonal Matching Pursuit (OP-COMP) with Projected Gradient Descent (PGD) successfully recovers those signals. In most cases, the main computational cost lies in a unique global descent on all parameters (positions and amplitudes). In this paper, we propose to improve this algorithm by accelerating this descent step. We introduce a new algorithm, based on Blo…

This https://arxiv.org/abs/2303.07983 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Parameter estimation of stochastic SIR model driven by small Lévy noise with time-dependent periodic transmission
We investigate the parameter estimation and prediction of two forms of the stochastic SIR model driven by small Lévy noise with time-dependent periodic transmission. We present consistency and rate of convergence results for the least-squares estimators. We include simulation studies using the method of projected gradient descent.

Gauss-Newton Natural Gradient Descent for Physics-Informed Computational Fluid Dynamics
Anas Jnini, Flavio Vella, Marius Zeinhofer
https://arxiv.org/abs/2402.10680

Gauss-Newton Natural Gradient Descent for Physics-Informed Computational Fluid Dynamics
We propose Gauss-Newton's method in function space for the solution of the Navier-Stokes equations in the physics-informed neural network (PINN) framework. Upon discretization, this yields a natural gradient method that provably mimics the function space dynamics. Our computational results demonstrate close to single-precision accuracy measured in relative $L^2$ norm on a number of benchmark problems. To the best of our knowledge, this constitutes the first contribution in the PINN literature t…

Interactive Byzantine-Resilient Gradient Coding for General Data Assignments
Shreyas Jain, Luis Maßny, Christoph Hofmeister, Eitan Yaakobi, Rawad Bitar
https://arXiv.org/abs/2401.16915

Interactive Byzantine-Resilient Gradient Coding for General Data Assignments
We tackle the problem of Byzantine errors in distributed gradient descent within the Byzantine-resilient gradient coding framework. Our proposed solution can recover the exact full gradient in the presence of $s$ malicious workers with a data replication factor of only $s+1$. It generalizes previous solutions to any data assignment scheme that has a regular replication over all data samples. The scheme detects malicious workers through additional interactive communication and a small number of …

This https://arxiv.org/abs/2402.11858 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_sta…

Stochastic Hessian Fittings with Lie Groups
This paper studies the fitting of Hessian or its inverse for stochastic optimizations using a Hessian fitting criterion from the preconditioned stochastic gradient descent (PSGD) method, which is intimately related to many commonly used second order and adaptive gradient optimizers, e.g., BFGS, Gaussian-Newton and natural gradient descent, AdaGrad, etc. Our analyses reveal the efficiency and reliability differences among a wide range of preconditioner fitting methods, from closed-form to iterat…

This https://arxiv.org/abs/2403.05070 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

A global Barzilai and Borwein's gradient normalization descent method for multiobjective optimization
In this paper, we consider the unconstrained multiobjective optimization problem. In recent years, researchers pointed out that the steepest decent method may generate small stepsize which leads to slow convergence rates. To address the issue, we propose a global Barzilai and Borwein's gradient normalization descent method for multiobjective optimization (GBBN). In our method, we propose a new normalization technique to generate new descent direction. We demonstrate that line search can achieve…

This https://arxiv.org/abs/2402.11858 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_sta…

Stochastic Hessian Fittings with Lie Groups
This paper studies the fitting of Hessian or its inverse for stochastic optimizations using a Hessian fitting criterion from the preconditioned stochastic gradient descent (PSGD) method, which is intimately related to many commonly used second order and adaptive gradient optimizers, e.g., BFGS, Gaussian-Newton and natural gradient descent, AdaGrad, etc. Our analyses reveal the efficiency and reliability differences among a wide range of preconditioner fitting methods, from closed-form to iterat…

Spline-Interpolated Model Predictive Path Integral Control with Stein Variational Inference for Reactive Navigation
Takato Miura, Naoki Akai, Kohei Honda, Susumu Hara
https://arxiv.org/abs/2404.10395

Spline-Interpolated Model Predictive Path Integral Control with Stein Variational Inference for Reactive Navigation
This paper presents a reactive navigation method that leverages a Model Predictive Path Integral (MPPI) control enhanced with spline interpolation for the control input sequence and Stein Variational Gradient Descent (SVGD). The MPPI framework addresses a nonlinear optimization problem by determining an optimal sequence of control inputs through a sampling-based approach. The efficacy of MPPI is significantly influenced by the sampling noise. To rapidly identify routes that circumvent large and…

Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models
Frederik Kunstner, Robin Yadav, Alan Milligan, Mark Schmidt, Alberto Bietti
https://arxiv.org/abs/2402.19449

Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models
Adam has been shown to outperform gradient descent in optimizing large language transformers empirically, and by a larger margin than on other tasks, but it is unclear why this happens. We show that the heavy-tailed class imbalance found in language modeling tasks leads to difficulties in the optimization dynamics. When training with gradient descent, the loss associated with infrequent words decreases slower than the loss associated with frequent ones. As most samples come from relatively infr…

Tractable Local Equilibria in Non-Concave Games
Yang Cai, Constantinos Daskalakis, Haipeng Luo, Chen-Yu Wei, Weiqiang Zheng
https://arxiv.org/abs/2403.08171

Tractable Local Equilibria in Non-Concave Games
While Online Gradient Descent and other no-regret learning procedures are known to efficiently converge to coarse correlated equilibrium in games where each agent's utility is concave in their own strategy, this is not the case when the utilities are non-concave, a situation that is common in machine learning applications where the agents' strategies are parameterized by deep neural networks, or the agents' utilities are computed by a neural network, or both. Indeed, non-concave games present a…

This https://arxiv.org/abs/2403.14045 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Accelerated Objective Gap and Gradient Norm Convergence for Gradient Descent via Long Steps
This work considers gradient descent for L-smooth convex optimization with stepsizes larger than the classic regime where descent can be ensured. The stepsize schedules considered are similar to but differ slightly from the recent silver stepsizes of Altschuler and Parrilo. For one of our stepsize sequences, we prove a $O\left(N^{- 1.2716\dots}\right)$ convergence rate in terms of objective gap decrease and for the other, we show the same rate of decrease for squared-gradient-norm decrease. Thi…

Learning of deep convolutional network image classifiers via stochastic gradient descent and over-parametrization
Michael Kohler, Adam Krzyzak, Alisha S\"anger
https://arxiv.org/abs/2404.07128

Learning of deep convolutional network image classifiers via stochastic gradient descent and over-parametrization
Image classification from independent and identically distributed random variables is considered. Image classifiers are defined which are based on a linear combination of deep convolutional networks with max-pooling layer. Here all the weights are learned by stochastic gradient descent. A general result is presented which shows that the image classifiers are able to approximate the best possible deep convolutional network. In case that the a posteriori probability satisfies a suitable hierarchi…

This https://arxiv.org/abs/2403.14045 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Accelerated Objective Gap and Gradient Norm Convergence for Gradient Descent via Long Steps
This work considers gradient descent for L-smooth convex optimization with stepsizes larger than the classic regime where descent can be ensured. The stepsize schedules considered are similar to but differ slightly from the recent silver stepsizes of Altschuler and Parrilo. For one of our stepsize sequences, we prove a $O\left(N^{- 1.2716\dots}\right)$ convergence rate in terms of objective gap decrease and for the other, we show the same rate of decrease for squared-gradient-norm decrease. Thi…

This https://arxiv.org/abs/2307.12441 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Swarm-based optimization with random descent
We extend our study of the swarm-based gradient descent method for non-convex optimization, [Lu, Tadmor & Zenginoglu, arXiv:2211.17157], to allow random descent directions. We recall that the swarm-based approach consists of a swarm of agents, each identified with a position, ${\mathbf x}$, and mass, $m$. The key is the transfer of mass from high ground to low(-est) ground. The mass of an agent dictates its step size: lighter agents take larger steps. In this paper, the essential new feature is…

This https://arxiv.org/abs/2310.06689 has been replaced.
link: https://scholar.google.com/scholar?q=a

Approximating Nash Equilibria in Normal-Form Games via Stochastic Optimization
We propose the first loss function for approximate Nash equilibria of normal-form games that is amenable to unbiased Monte Carlo estimation. This construction allows us to deploy standard non-convex stochastic optimization techniques for approximating Nash equilibria, resulting in novel algorithms with provable guarantees. We complement our theoretical analysis with experiments demonstrating that stochastic gradient descent can outperform previous state-of-the-art approaches.

Dissipative Gradient Descent Ascent Method: A Control Theory Inspired Algorithm for Min-max Optimization
Tianqi Zheng, Nicolas Loizou, Pengcheng You, Enrique Mallada
https://arxiv.org/abs/2403.09090

Dissipative Gradient Descent Ascent Method: A Control Theory Inspired Algorithm for Min-max Optimization
Gradient Descent Ascent (GDA) methods for min-max optimization problems typically produce oscillatory behavior that can lead to instability, e.g., in bilinear settings. To address this problem, we introduce a dissipation term into the GDA updates to dampen these oscillations. The proposed Dissipative GDA (DGDA) method can be seen as performing standard GDA on a state-augmented and regularized saddle function that does not strictly introduce additional convexity/concavity. We theoretically show …

This https://arxiv.org/abs/2309.05955 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csRO_…

Trust-Region Neural Moving Horizon Estimation for Robots
Accurate disturbance estimation is essential for safe robot operations. The recently proposed neural moving horizon estimation (NeuroMHE), which uses a portable neural network to model the MHE's weightings, has shown promise in further pushing the accuracy and efficiency boundary. Currently, NeuroMHE is trained through gradient descent, with its gradient computed recursively using a Kalman filter. This paper proposes a trust-region policy optimization method for training NeuroMHE. We achieve th…

This https://arxiv.org/abs/2402.04691 has been replaced.
link: https://scholar.google.com/scholar?q=a

Learning Operators with Stochastic Gradient Descent in General Hilbert Spaces
This study investigates leveraging stochastic gradient descent (SGD) to learn operators between general Hilbert spaces. We propose weak and strong regularity conditions for the target operator to depict its intrinsic structure and complexity. Under these conditions, we establish upper bounds for convergence rates of the SGD algorithm and conduct a minimax lower bound analysis, further illustrating that our convergence analysis and regularity conditions quantitatively characterize the tractabili…

This https://arxiv.org/abs/2403.02967 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Non-Convex Stochastic Composite Optimization with Polyak Momentum
The stochastic proximal gradient method is a powerful generalization of the widely used stochastic gradient descent (SGD) method and has found numerous applications in Machine Learning. However, it is notoriously known that this method fails to converge in non-convex settings where the stochastic noise is significant (i.e. when only small or bounded batch sizes are used). In this paper, we focus on the stochastic proximal gradient method with Polyak momentum. We prove this method attains an opt…

This https://arxiv.org/abs/2401.00691 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_sta…

Stochastic Gradient Descent for Additive Nonparametric Regression
This paper introduces an iterative algorithm for training additive models that enjoys favorable memory storage and computational requirements. The algorithm can be viewed as the functional counterpart of stochastic gradient descent, applied to the coefficients of a truncated basis expansion of the component functions. We show that the resulting estimator satisfies an oracle inequality that allows for model mis-specification. In the well-specified setting, by choosing the learning rate carefully…

This https://arxiv.org/abs/2309.05955 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csRO_…

Trust-Region Neural Moving Horizon Estimation for Robots
Accurate disturbance estimation is essential for safe robot operations. The recently proposed neural moving horizon estimation (NeuroMHE), which uses a portable neural network to model the MHE's weightings, has shown promise in further pushing the accuracy and efficiency boundary. Currently, NeuroMHE is trained through gradient descent, with its gradient computed recursively using a Kalman filter. This paper proposes a trust-region policy optimization method for training NeuroMHE. We achieve th…

A global Barzilai and Borwein's gradient normalization descent method for multiobjective optimization
Yingxue Yang
https://arxiv.org/abs/2403.05070 htt…

A global Barzilai and Borwein's gradient normalization descent method for multiobjective optimization
In this paper, we consider the unconstrained multiobjective optimization problem. In recent years, researchers pointed out that the steepest decent method may generate small stepsize which leads to slow convergence rates. To address the issue, we propose a global Barzilai and Borwein's gradient normalization descent method for multiobjective optimization (GBBN). In our method, we propose a new normalization technique to generate new descent direction. We demonstrate that line search can achieve…

This https://arxiv.org/abs/2308.10547 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Decentralized Riemannian Conjugate Gradient Method on the Stiefel Manifold
The conjugate gradient method is a crucial first-order optimization method that generally converges faster than the steepest descent method, and its computational cost is much lower than that of second-order methods. However, while various types of conjugate gradient methods have been studied in Euclidean spaces and on Riemannian manifolds, there is little study for those in distributed scenarios. This paper proposes a decentralized Riemannian conjugate gradient descent (DRCGD) method that aims…

This https://arxiv.org/abs/2305.16242 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Two-timescale Extragradient for Finding Local Minimax Points
Minimax problems are notoriously challenging to optimize. However, we present that the two-timescale extragradient method can be a viable solution. By utilizing dynamical systems theory, we show that it converges to points that satisfy the second-order necessary condition of local minimax points, under mild conditions that the two-timescale gradient descent ascent fails to work. This work provably improves upon all previous results on finding local minimax points, by eliminating a crucial assum…

SOFIM: Stochastic Optimization Using Regularized Fisher Information Matrix
Gayathri C, Mrinmay Sen, A. K. Qin, Raghu Kishore N, Yen-Wei Chen, Balasubramanian Raman
https://arxiv.org/abs/2403.02833

SOFIM: Stochastic Optimization Using Regularized Fisher Information Matrix
This paper introduces a new stochastic optimization method based on the regularized Fisher information matrix (FIM), named SOFIM, which can efficiently utilize the FIM to approximate the Hessian matrix for finding Newton's gradient update in large-scale stochastic optimization of machine learning models. It can be viewed as a variant of natural gradient descent (NGD), where the challenge of storing and calculating the full FIM is addressed through making use of the regularized FIM and directly …

This https://arxiv.org/abs/2402.12090 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Characterization of optimization problems that are solvable iteratively with linear convergence
In this work, we state a general conjecture on the solvability of optimization problems via algorithms with linear convergence guarantees. We make a first step towards examining its correctness by fully characterizing the problems that are solvable via Riemannian gradient descent with linear convergence.

Sliding down the stairs: how correlated latent variables accelerate learning with neural networks
Lorenzo Bardone, Sebastian Goldt
https://arxiv.org/abs/2404.08602

Sliding down the stairs: how correlated latent variables accelerate learning with neural networks
Neural networks extract features from data using stochastic gradient descent (SGD). In particular, higher-order input cumulants (HOCs) are crucial for their performance. However, extracting information from the $p$th cumulant of $d$-dimensional inputs is computationally hard: the number of samples required to recover a single direction from an order-$p$ tensor (tensor PCA) using online SGD grows as $d^{p-1}$, which is prohibitive for high-dimensional inputs. This result raises the question of h…

This https://arxiv.org/abs/2312.01127 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Symmetric Mean-field Langevin Dynamics for Distributional Minimax Problems
In this paper, we extend mean-field Langevin dynamics to minimax optimization over probability distributions for the first time with symmetric and provably convergent updates. We propose mean-field Langevin averaged gradient (MFL-AG), a single-loop algorithm that implements gradient descent ascent in the distribution spaces with a novel weighted averaging, and establish average-iterate convergence to the mixed Nash equilibrium. We also study both time and particle discretization regimes and pro…

Sliding down the stairs: how correlated latent variables accelerate learning with neural networks
Lorenzo Bardone, Sebastian Goldt
https://arxiv.org/abs/2404.08602

Sliding down the stairs: how correlated latent variables accelerate learning with neural networks
Neural networks extract features from data using stochastic gradient descent (SGD). In particular, higher-order input cumulants (HOCs) are crucial for their performance. However, extracting information from the $p$th cumulant of $d$-dimensional inputs is computationally hard: the number of samples required to recover a single direction from an order-$p$ tensor (tensor PCA) using online SGD grows as $d^{p-1}$, which is prohibitive for high-dimensional inputs. This result raises the question of h…

A modified Polak-Ribiere-Polyak type conjugate gradient method with two stepsize strategies for vector optimization
Yushan Bai, Jiawei Chen, Kaiping Liu
https://arxiv.org/abs/2404.08503

A modified Polak-Ribiere-Polyak type conjugate gradient method with two stepsize strategies for vector optimization
In this paper, in order to find critical points of vector-valued functions with respect to the partial order induced by a closed, convex, and pointed cone with nonempty interior, we propose a nonlinear modified Polak-Ribiere-Polyak type conjugate gradient method with a nonnegative conjugate parameter. We show that the search direction in our method satisfies the sufficient descent condition independent of any line search. Furthermore, under mild assumptions, we obtain the results of global conv…

A modified Polak-Ribiere-Polyak type conjugate gradient method with two stepsize strategies for vector optimization
Yushan Bai, Jiawei Chen, Kaiping Liu
https://arxiv.org/abs/2404.08503

A modified Polak-Ribiere-Polyak type conjugate gradient method with two stepsize strategies for vector optimization
In this paper, in order to find critical points of vector-valued functions with respect to the partial order induced by a closed, convex, and pointed cone with nonempty interior, we propose a nonlinear modified Polak-Ribiere-Polyak type conjugate gradient method with a nonnegative conjugate parameter. We show that the search direction in our method satisfies the sufficient descent condition independent of any line search. Furthermore, under mild assumptions, we obtain the results of global conv…

This https://arxiv.org/abs/2308.10630 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

A Homogenization Approach for Gradient-Dominated Stochastic Optimization
Gradient dominance property is a condition weaker than strong convexity, yet sufficiently ensures global convergence even in non-convex optimization. This property finds wide applications in machine learning, reinforcement learning (RL), and operations management. In this paper, we propose the stochastic homogeneous second-order descent method (SHSODM) for stochastic functions enjoying gradient dominance property based on a recently proposed homogenization approach. Theoretically, we provide it…

A Single-Mode Quasi Riemannian Gradient Descent Algorithm for Low-Rank Tensor Recovery
Yuanwei Zhang, Ya-Nan Zhu, Xiaoqun Zhang
https://arXiv.org/abs/2401.15925

A Single-Mode Quasi Riemannian Gradient Descent Algorithm for Low-Rank Tensor Recovery
This paper focuses on recovering a low-rank tensor from its incomplete measurements. We propose a novel algorithm termed the Single Mode Quasi Riemannian Gradient Descent (SM-QRGD). By exploiting the benefits of both fixed-rank matrix tangent space projection in Riemannian gradient descent and sequentially truncated high-order singular value decomposition (ST-HOSVD), SM-QRGD achieves a much faster convergence speed than existing state-of-the-art algorithms. Theoretically, we establish the conve…

This https://arxiv.org/abs/2403.14045 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Accelerated Objective Gap and Gradient Norm Convergence for Gradient Descent via Long Steps
This work considers gradient descent for L-smooth convex optimization with stepsizes larger than the classic regime where descent can be ensured. The stepsize schedules considered are similar to but differ slightly from the concurrently developed silver stepsizes of Altschuler and Parillo. For one of our stepsize sequences, we prove a $O\left(\frac{1}{N^{1.2716\dots}}\right)$ convergence rate in terms of objective gap decrease and for the other, we show the same rate of decrease for the squared…

Gradient Descent is Pareto-Optimal in the Oracle Complexity and Memory Tradeoff for Feasibility Problems
Moise Blanchard
https://arxiv.org/abs/2404.06720 h…

Gradient Descent is Pareto-Optimal in the Oracle Complexity and Memory Tradeoff for Feasibility Problems
In this paper we provide oracle complexity lower bounds for finding a point in a given set using a memory-constrained algorithm that has access to a separation oracle. We assume that the set is contained within the unit $d$-dimensional ball and contains a ball of known radius $ε>0$. This setup is commonly referred to as the feasibility problem. We show that to solve feasibility problems with accuracy $ε\geq e^{-d^{o(1)}}$, any deterministic algorithm either uses $d^{1+δ}$ bits of memory or m…

This https://arxiv.org/abs/2403.14045 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Accelerated Objective Gap and Gradient Norm Convergence for Gradient Descent via Long Steps
This work considers gradient descent for L-smooth convex optimization with stepsizes larger than the classic regime where descent can be ensured. The stepsize schedules considered are similar to but differ slightly from the concurrently developed silver stepsizes of Altschuler and Parillo. For one of our stepsize sequences, we prove a $O\left(\frac{1}{N^{1.2716\dots}}\right)$ convergence rate in terms of objective gap decrease and for the other, we show the same rate of decrease for the squared…

This https://arxiv.org/abs/2209.05564 has been replaced.
link: https://scholar.google.com/scholar?q=a

Deep Relaxation of Controlled Stochastic Gradient Descent via Singular Perturbations
We consider a singularly perturbed system of stochastic differential equations proposed by Chaudhari et al. (Res. Math. Sci. 2018) to approximate the Entropic Gradient Descent in the optimization of deep neural networks, via homogenisation. We embed it in a much larger class of two-scale stochastic control problems and rely on convergence results for Hamilton-Jacobi-Bellman equations with unbounded data proved recently by ourselves (ESAIM Control Optim. Calc. Var. 2023). We show that the limit …

This https://arxiv.org/abs/2402.07793 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Tuning-Free Stochastic Optimization
Large-scale machine learning problems make the cost of hyperparameter tuning ever more prohibitive. This creates a need for algorithms that can tune themselves on-the-fly. We formalize the notion of "tuning-free" algorithms that can match the performance of optimally-tuned optimization algorithms up to polylogarithmic factors given only loose hints on the relevant problem parameters. We consider in particular algorithms that can match optimally-tuned Stochastic Gradient Descent (SGD). When the …

Faster Convergence of Stochastic Accelerated Gradient Descent under Interpolation
Aaron Mishkin, Mert Pilanci, Mark Schmidt
https://arxiv.org/abs/2404.02378

Faster Convergence of Stochastic Accelerated Gradient Descent under Interpolation
We prove new convergence rates for a generalized version of stochastic Nesterov acceleration under interpolation conditions. Unlike previous analyses, our approach accelerates any stochastic gradient method which makes sufficient progress in expectation. The proof, which proceeds using the estimating sequences framework, applies to both convex and strongly convex functions and is easily specialized to accelerated SGD under the strong growth condition. In this special case, our analysis reduces …

This https://arxiv.org/abs/2305.17224 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Fast and Accurate Estimation of Low-Rank Matrices from Noisy Measurements via Preconditioned Non-Convex Gradient Descent
Non-convex gradient descent is a common approach for estimating a low-rank $n\times n$ ground truth matrix from noisy measurements, because it has per-iteration costs as low as $O(n)$ time, and is in theory capable of converging to a minimax optimal estimate. However, the practitioner is often constrained to just tens to hundreds of iterations, and the slow and/or inconsistent convergence of non-convex gradient descent can prevent a high-quality estimate from being obtained. Recently, the techn…

This https://arxiv.org/abs/2312.01334 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Optimization Methods Rooting in Optimal Control
In the paper, we propose solving optimization problems (OPs) and understanding the Newton method from the optimal control view. We propose a new optimization algorithm based on the optimal control problem (OCP). The algorithm features converging more rapidly than gradient descent, meanwhile, it is superior to Newton's method because it is not divergent in general and can be applied in the case of a singular Hessian matrix. These merits are supported by the convergence analysis for the algorithm…

This https://arxiv.org/abs/2210.15531 has been replaced.
link: https://scholar.google.com/scholar?q=a

Anisotropic Proximal Gradient
This paper studies a novel algorithm for nonconvex composite minimization which can be interpreted in terms of dual space nonlinear preconditioning for the classical proximal gradient method. The proposed scheme can be applied to additive composite minimization problems whose smooth part exhibits an anisotropic descent inequality relative to a reference function. It is proved that the anisotropic descent property is closed under pointwise average if the dual Bregman distance is jointly convex a…

Constrained Stochastic Recursive Momentum Successive Convex Approximation
Basil M. Idrees, Lavish Arora, Ketan Rajawat
https://arxiv.org/abs/2404.11790 htt…

Constrained Stochastic Recursive Momentum Successive Convex Approximation
We consider stochastic optimization problems with functional constraints. If the objective and constraint functions are not convex, the classical stochastic approximation algorithms such as the proximal stochastic gradient descent do not lead to efficient algorithms. In this work, we put forth an accelerated SCA algorithm that utilizes the recursive momentum-based acceleration which is widely used in the unconstrained setting. Remarkably, the proposed algorithm also achieves the optimal SFO com…

Clipped SGD Algorithms for Privacy Preserving Performative Prediction: Bias Amplification and Remedies
Qiang Li, Michal Yemini, Hoi-To Wai
https://arxiv.org/abs/2404.10995

Clipped SGD Algorithms for Privacy Preserving Performative Prediction: Bias Amplification and Remedies
Clipped stochastic gradient descent (SGD) algorithms are among the most popular algorithms for privacy preserving optimization that reduces the leakage of users' identity in model training. This paper studies the convergence properties of these algorithms in a performative prediction setting, where the data distribution may shift due to the deployed prediction model. For example, the latter is caused by strategical users during the training of loan policy for banks. Our contributions are two-fo…

Shuffling Momentum Gradient Algorithm for Convex Optimization
Trang H. Tran, Quoc Tran-Dinh, Lam M. Nguyen
https://arxiv.org/abs/2403.03180 https://…

Shuffling Momentum Gradient Algorithm for Convex Optimization
The Stochastic Gradient Descent method (SGD) and its stochastic variants have become methods of choice for solving finite-sum optimization problems arising from machine learning and data science thanks to their ability to handle large-scale applications and big datasets. In the last decades, researchers have made substantial effort to study the theoretical performance of SGD and its shuffling variants. However, only limited work has investigated its shuffling momentum variants, including shuffl…

Adaptive multi-gradient methods for quasiconvex vector optimization and applications to multi-task learning
Nguyen Anh Minh, Le Dung Muu, Tran Ngoc Thang
https://arxiv.org/abs/2402.06224

Adaptive multi-gradient methods for quasiconvex vector optimization and applications to multi-task learning
We present an adaptive step-size method, which does not include line-search techniques, for solving a wide class of nonconvex multiobjective programming problems on an unbounded constraint set. We also prove convergence of a general approach under modest assumptions. More specifically, the convexity criterion might not be satisfied by the objective function. Unlike descent line-search algorithms, it does not require an initial step-size to be determined by a previously determined Lipschitz cons…

This https://arxiv.org/abs/2308.15145 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Limited memory gradient methods for unconstrained optimization
The limited memory steepest descent method (Fletcher, 2012) for unconstrained optimization problems stores a few past gradients to compute multiple stepsizes at once. We review this method and propose new variants. For strictly convex quadratic objective functions, we study the numerical behavior of different techniques to compute new stepsizes. In particular, we introduce a method to improve the use of harmonic Ritz values. We also show the existence of a secant condition associated with LMSD,…