Tootfinder

This https://arxiv.org/abs/2211.07861 has been replaced.
link: https://scholar.google.com/scholar?q=a

Regularized Stein Variational Gradient Flow
The Stein Variational Gradient Descent (SVGD) algorithm is a deterministic particle method for sampling. However, a mean-field analysis reveals that the gradient flow corresponding to the SVGD algorithm (i.e., the Stein Variational Gradient Flow) only provides a constant-order approximation to the Wasserstein Gradient Flow corresponding to the KL-divergence minimization. In this work, we propose the Regularized Stein Variational Gradient Flow, which interpolates between the Stein Variational Gr…

A global Barzilai and Borwein's gradient normalization descent method for multiobjective optimization
Yingxue Yang
https://arxiv.org/abs/2403.05070 htt…

A global Barzilai and Borwein's gradient normalization descent method for multiobjective optimization
In this paper, we consider the unconstrained multiobjective optimization problem. In recent years, researchers pointed out that the steepest decent method may generate small stepsize which leads to slow convergence rates. To address the issue, we propose a global Barzilai and Borwein's gradient normalization descent method for multiobjective optimization (GBBN). In our method, we propose a new normalization technique to generate new descent direction. We demonstrate that line search can achieve…

Symmetry-guided gradient descent for quantum neural networks
Kaiming Bian, Shitao Zhang, Fei Meng, Wen Zhang, Oscar Dahlsten
https://arxiv.org/abs/2404.06108

Symmetry-guided gradient descent for quantum neural networks
Many supervised learning tasks have intrinsic symmetries, such as translational and rotational symmetry in image classifications. These symmetries can be exploited to enhance performance. We formulate the symmetry constraints into a concise mathematical form. We design two ways to adopt the constraints into the cost function, thereby shaping the cost landscape in favour of parameter choices which respect the given symmetry. Unlike methods that alter the neural network circuit ansatz to impose s…

Learning of deep convolutional network image classifiers via stochastic gradient descent and over-parametrization
Michael Kohler, Adam Krzyzak, Alisha S\"anger
https://arxiv.org/abs/2404.07128

Learning of deep convolutional network image classifiers via stochastic gradient descent and over-parametrization
Image classification from independent and identically distributed random variables is considered. Image classifiers are defined which are based on a linear combination of deep convolutional networks with max-pooling layer. Here all the weights are learned by stochastic gradient descent. A general result is presented which shows that the image classifiers are able to approximate the best possible deep convolutional network. In case that the a posteriori probability satisfies a suitable hierarchi…

Symmetry-guided gradient descent for quantum neural networks
Kaiming Bian, Shitao Zhang, Fei Meng, Wen Zhang, Oscar Dahlsten
https://arxiv.org/abs/2404.06108

Symmetry-guided gradient descent for quantum neural networks
Many supervised learning tasks have intrinsic symmetries, such as translational and rotational symmetry in image classifications. These symmetries can be exploited to enhance performance. We formulate the symmetry constraints into a concise mathematical form. We design two ways to adopt the constraints into the cost function, thereby shaping the cost landscape in favour of parameter choices which respect the given symmetry. Unlike methods that alter the neural network circuit ansatz to impose s…

This https://arxiv.org/abs/2402.02731 has been replaced.
link: https://scholar.google.com/scholar?q=a

Computing Augustin Information via Hybrid Geodesically Convex Optimization
We propose a Riemannian gradient descent with the Poincaré metric to compute the order-$α$ Augustin information, a widely used quantity for characterizing exponential error behaviors in information theory. We prove that the algorithm converges to the optimum at a rate of $\mathcal{O}(1 / T)$. As far as we know, this is the first algorithm with a non-asymptotic optimization error guarantee for all positive orders. Numerical experimental results demonstrate the empirical efficiency of the algor…

Gradient Descent is Pareto-Optimal in the Oracle Complexity and Memory Tradeoff for Feasibility Problems
Moise Blanchard
https://arxiv.org/abs/2404.06720 h…

Gradient Descent is Pareto-Optimal in the Oracle Complexity and Memory Tradeoff for Feasibility Problems
In this paper we provide oracle complexity lower bounds for finding a point in a given set using a memory-constrained algorithm that has access to a separation oracle. We assume that the set is contained within the unit $d$-dimensional ball and contains a ball of known radius $ε>0$. This setup is commonly referred to as the feasibility problem. We show that to solve feasibility problems with accuracy $ε\geq e^{-d^{o(1)}}$, any deterministic algorithm either uses $d^{1+δ}$ bits of memory or m…

This https://arxiv.org/abs/2209.05564 has been replaced.
link: https://scholar.google.com/scholar?q=a

Deep Relaxation of Controlled Stochastic Gradient Descent via Singular Perturbations
We consider a singularly perturbed system of stochastic differential equations proposed by Chaudhari et al. (Res. Math. Sci. 2018) to approximate the Entropic Gradient Descent in the optimization of deep neural networks, via homogenisation. We embed it in a much larger class of two-scale stochastic control problems and rely on convergence results for Hamilton-Jacobi-Bellman equations with unbounded data proved recently by ourselves (ESAIM Control Optim. Calc. Var. 2023). We show that the limit …

Generalized Gradient Descent is a Hypergraph Functor
Tyler Hanks, Matthew Klawonn, James Fairbanks
https://arxiv.org/abs/2403.19845 https://

Generalized Gradient Descent is a Hypergraph Functor
Cartesian reverse derivative categories (CRDCs) provide an axiomatic generalization of the reverse derivative, which allows generalized analogues of classic optimization algorithms such as gradient descent to be applied to a broad class of problems. In this paper, we show that generalized gradient descent with respect to a given CRDC induces a hypergraph functor from a hypergraph category of optimization problems to a hypergraph category of dynamical systems. The domain of this functor consists…

Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models
Frederik Kunstner, Robin Yadav, Alan Milligan, Mark Schmidt, Alberto Bietti
https://arxiv.org/abs/2402.19449

Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models
Adam has been shown to outperform gradient descent in optimizing large language transformers empirically, and by a larger margin than on other tasks, but it is unclear why this happens. We show that the heavy-tailed class imbalance found in language modeling tasks leads to difficulties in the optimization dynamics. When training with gradient descent, the loss associated with infrequent words decreases slower than the loss associated with frequent ones. As most samples come from relatively infr…

This https://arxiv.org/abs/2309.05955 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csRO_…

Trust-Region Neural Moving Horizon Estimation for Robots
Accurate disturbance estimation is essential for safe robot operations. The recently proposed neural moving horizon estimation (NeuroMHE), which uses a portable neural network to model the MHE's weightings, has shown promise in further pushing the accuracy and efficiency boundary. Currently, NeuroMHE is trained through gradient descent, with its gradient computed recursively using a Kalman filter. This paper proposes a trust-region policy optimization method for training NeuroMHE. We achieve th…

This https://arxiv.org/abs/1901.09057 has been replaced.
link: https://scholar.google.com/scholar?q=a

Discretized Gradient Flow for Manifold Learning in the Space of Embeddings
Gradient descent, or negative gradient flow, is a standard technique in optimization to find minima of functions. Many implementations of gradient descent rely on discretized versions, i.e., moving in the gradient direction for a set step size, recomputing the gradient, and continuing. In this paper, we present an approach to manifold learning where gradient descent takes place in the infinite dimensional space $\mathcal{E} = {\rm Emb}(M,\mathbb{R}^N)$ of smooth embeddings $ϕ$ of a manifold $M…

This https://arxiv.org/abs/2308.02958 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_ees…

K-band: Self-supervised MRI Reconstruction via Stochastic Gradient Descent over K-space Subsets
Although deep learning (DL) methods are powerful for solving inverse problems, their reliance on high-quality training data is a major hurdle. This is significant in high-dimensional (dynamic/volumetric) magnetic resonance imaging (MRI), where acquisition of high-resolution fully sampled k-space data is impractical. We introduce a novel mathematical framework, dubbed k-band, that enables training DL models using only partial, limited-resolution k-space data. Specifically, we introduce training …

Finite Sample Analysis and Bounds of Generalization Error of Gradient Descent in In-Context Linear Regression
Karthik Duraisamy
https://arxiv.org/abs/2405.02462

Finite Sample Analysis and Bounds of Generalization Error of Gradient Descent in In-Context Linear Regression
Recent studies show that transformer-based architectures emulate gradient descent during a forward pass, contributing to in-context learning capabilities - an ability where the model adapts to new tasks based on a sequence of prompt examples without being explicitly trained or fine tuned to do so. This work investigates the generalization properties of a single step of gradient descent in the context of linear regression with well-specified models. A random design setting is considered and anal…

Generalized Gradient Descent is a Hypergraph Functor
Tyler Hanks, Matthew Klawonn, James Fairbanks
https://arxiv.org/abs/2403.19845 https://

Generalized Gradient Descent is a Hypergraph Functor
Cartesian reverse derivative categories (CRDCs) provide an axiomatic generalization of the reverse derivative, which allows generalized analogues of classic optimization algorithms such as gradient descent to be applied to a broad class of problems. In this paper, we show that generalized gradient descent with respect to a given CRDC induces a hypergraph functor from a hypergraph category of optimization problems to a hypergraph category of dynamical systems. The domain of this functor consists…

On the Origins of Linear Representations in Large Language Models
Yibo Jiang, Goutham Rajendran, Pradeep Ravikumar, Bryon Aragam, Victor Veitch
https://arxiv.org/abs/2403.03867

On the Origins of Linear Representations in Large Language Models
Recent works have argued that high-level semantic concepts are encoded "linearly" in the representation space of large language models. In this work, we study the origins of such linear representations. To that end, we introduce a simple latent variable model to abstract and formalize the concept dynamics of the next token prediction. We use this formalism to show that the next token prediction objective (softmax with cross-entropy) and the implicit bias of gradient descent together promote the…

This https://arxiv.org/abs/2309.05955 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csRO_…

Trust-Region Neural Moving Horizon Estimation for Robots
Accurate disturbance estimation is essential for safe robot operations. The recently proposed neural moving horizon estimation (NeuroMHE), which uses a portable neural network to model the MHE's weightings, has shown promise in further pushing the accuracy and efficiency boundary. Currently, NeuroMHE is trained through gradient descent, with its gradient computed recursively using a Kalman filter. This paper proposes a trust-region policy optimization method for training NeuroMHE. We achieve th…

Projected gradient descent algorithm for $\textit{ab initio}$ crystal structure relaxation under a fixed unit cell volume
Yukuan Hu, Junlei Yin, Xingyu Gao, Xin Liu, Haifeng Song
https://arxiv.org/abs/2405.02934

Projected gradient descent algorithm for $\textit{ab initio}$ crystal structure relaxation under a fixed unit cell volume
This paper is concerned with $\textit{ab initio}$ crystal structure relaxation under a fixed unit cell volume, which is a step in calculating the static equations of state and forms the basis of thermodynamic property calculations for materials. The task can be formulated as an energy minimization with a determinant constraint. Widely used line minimization-based methods (e.g., conjugate gradient method) lack both efficiency and convergence guarantees due to the nonconvex nature of the feasible…

This https://arxiv.org/abs/1901.09057 has been replaced.
link: https://scholar.google.com/scholar?q=a

Discretized Gradient Flow for Manifold Learning in the Space of Embeddings
Gradient descent, or negative gradient flow, is a standard technique in optimization to find minima of functions. Many implementations of gradient descent rely on discretized versions, i.e., moving in the gradient direction for a set step size, recomputing the gradient, and continuing. In this paper, we present an approach to manifold learning where gradient descent takes place in the infinite dimensional space $\mathcal{E} = {\rm Emb}(M,\mathbb{R}^N)$ of smooth embeddings $ϕ$ of a manifold $M…

This https://arxiv.org/abs/2405.03073 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Convergence and Complexity Guarantee for Inexact First-order Riemannian Optimization Algorithms
We analyze inexact Riemannian gradient descent (RGD) where Riemannian gradients and retractions are inexactly (and cheaply) computed. Our focus is on understanding when inexact RGD converges and what is the complexity in the general nonconvex and constrained setting. We answer these questions in a general framework of tangential Block Majorization-Minimization (tBMM). We establish that tBMM converges to an $ε$-stationary point within $O(ε^{-2})$ iterations. Under a mild assumption, the result…

This https://arxiv.org/abs/2401.02565 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_ees…

Demonstration of an Adversarial Attack Against a Multimodal Vision Language Model for Pathology Imaging
In the context of medical artificial intelligence, this study explores the vulnerabilities of the Pathology Language-Image Pretraining (PLIP) model, a Vision Language Foundation model, under targeted attacks. Leveraging the Kather Colon dataset with 7,180 H&E images across nine tissue types, our investigation employs Projected Gradient Descent (PGD) adversarial perturbation attacks to induce misclassifications intentionally. The outcomes reveal a 100% success rate in manipulating PLIP's predict…

SOFIM: Stochastic Optimization Using Regularized Fisher Information Matrix
Gayathri C, Mrinmay Sen, A. K. Qin, Raghu Kishore N, Yen-Wei Chen, Balasubramanian Raman
https://arxiv.org/abs/2403.02833

SOFIM: Stochastic Optimization Using Regularized Fisher Information Matrix
This paper introduces a new stochastic optimization method based on the regularized Fisher information matrix (FIM), named SOFIM, which can efficiently utilize the FIM to approximate the Hessian matrix for finding Newton's gradient update in large-scale stochastic optimization of machine learning models. It can be viewed as a variant of natural gradient descent (NGD), where the challenge of storing and calculating the full FIM is addressed through making use of the regularized FIM and directly …

This https://arxiv.org/abs/2107.12416 has been replaced.
link: https://scholar.google.com/scholar?q=a

Asynchronous Distributed Reinforcement Learning for LQR Control via Zeroth-Order Block Coordinate Descent
Recently introduced distributed zeroth-order optimization (ZOO) algorithms have shown their utility in distributed reinforcement learning (RL). Unfortunately, in the gradient estimation process, almost all of them require random samples with the same dimension as the global variable and/or require evaluation of the global cost function, which may induce high estimation variance for large-scale networks. In this paper, we propose a novel distributed zeroth-order algorithm by leveraging the netwo…

This https://arxiv.org/abs/2404.19027 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_qu…

Better Optimization of Variational Quantum Eigensolvers by combining the Unitary Block Optimization Scheme with Classical Post-Processing
Variational Quantum Eigensolvers (VQE) are a promising approach for finding the classically intractable ground state of a Hamiltonian. The Unitary Block Optimization Scheme (UBOS) is a state-of-the-art VQE method which works by sweeping over gates and finding optimal parameters for each gate in the environment of other gates. UBOS improves the convergence time to the ground state by an order of magnitude over Stochastic Gradient Descent (SGD). It nonetheless suffers in both rate of convergence …

This https://arxiv.org/abs/2210.15531 has been replaced.
link: https://scholar.google.com/scholar?q=a

Anisotropic Proximal Gradient
This paper studies a novel algorithm for nonconvex composite minimization which can be interpreted in terms of dual space nonlinear preconditioning for the classical proximal gradient method. The proposed scheme can be applied to additive composite minimization problems whose smooth part exhibits an anisotropic descent inequality relative to a reference function. It is proved that the anisotropic descent property is closed under pointwise average if the dual Bregman distance is jointly convex a…

JaxDecompiler: Redefining Gradient-Informed Software Design
Pierrick Pochelu
https://arxiv.org/abs/2403.10571 https://arxiv.org/pdf/2…

JaxDecompiler: Redefining Gradient-Informed Software Design
Among numerical libraries capable of computing gradient descent optimization, JAX stands out by offering more features, accelerated by an intermediate representation known as Jaxpr language. However, editing the Jaxpr code is not directly possible. This article introduces JaxDecompiler, a tool that transforms any JAX function into an editable Python code, especially useful for editing the JAX function generated by the gradient function. JaxDecompiler simplifies the processes of reverse engineer…

This https://arxiv.org/abs/2310.14085 has been replaced.
link: https://scholar.google.com/scholar?q=a

Adaptive, Doubly Optimal No-Regret Learning in Strongly Monotone and Exp-Concave Games with Gradient Feedback
Online gradient descent (OGD) is well known to be doubly optimal under strong convexity or monotonicity assumptions: (1) in the single-agent setting, it achieves an optimal regret of $Θ(\log T)$ for strongly convex cost functions; and (2) in the multi-agent setting of strongly monotone games, with each agent employing OGD, we obtain last-iterate convergence of the joint action to a unique Nash equilibrium at an optimal rate of $Θ(\frac{1}{T})$. While these finite-time guarantees highlight its…

Projected Gradient Descent for Spectral Compressed Sensing via Symmetric Hankel Factorization
Jinsheng Li, Wei Cui, Xu Zhang
https://arxiv.org/abs/2403.09031

Projected Gradient Descent for Spectral Compressed Sensing via Symmetric Hankel Factorization
Current spectral compressed sensing methods via Hankel matrix completion employ symmetric factorization to demonstrate the low-rank property of the Hankel matrix. However, previous non-convex gradient methods only utilize asymmetric factorization to achieve spectral compressed sensing. In this paper, we propose a novel nonconvex projected gradient descent method for spectral compressed sensing via symmetric factorization named Symmetric Hankel Projected Gradient Descent (SHGD), which updates on…

This https://arxiv.org/abs/2305.19394 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_qbi…

Synaptic Weight Distributions Depend on the Geometry of Plasticity
A growing literature in computational neuroscience leverages gradient descent and learning algorithms that approximate it to study synaptic plasticity in the brain. However, the vast majority of this work ignores a critical underlying assumption: the choice of distance for synaptic changes - i.e. the geometry of synaptic plasticity. Gradient descent assumes that the distance is Euclidean, but many other distances are possible, and there is no reason that biology necessarily uses Euclidean geome…

Wirtinger gradient descent methods for low-dose Poisson phase retrieval
Benedikt Diederichs, Frank Filbir, Patricia R\"omer
https://arxiv.org/abs/2403.18527

Wirtinger gradient descent methods for low-dose Poisson phase retrieval
The problem of phase retrieval has many applications in the field of optical imaging. Motivated by imaging experiments with biological specimens, we primarily consider the setting of low-dose illumination where Poisson noise plays the dominant role. In this paper, we discuss gradient descent algorithms based on different loss functions adapted to data affected by Poisson noise, in particular in the low-dose regime. Starting from the maximum log-likelihood function for the Poisson distribution, …

The Blind Normalized Stein Variational Gradient Descent-Based Detection for Intelligent Massive Random Access
Xin Zhu, Ahmet Enis Cetin
https://arxiv.org/abs/2403.18846

The Blind Normalized Stein Variational Gradient Descent-Based Detection for Intelligent Massive Random Access
The lack of an efficient preamble detection algorithm remains a challenge for solving preamble collision problems in intelligent massive random access (RA) in practical communication scenarios. To solve this problem, we present a novel early preamble detection scheme based on a maximum likelihood estimation (MLE) model at the first step of the grant-based RA procedure. A novel blind normalized Stein variational gradient descent (SVGD)-based detector is proposed to obtain an approximate solution…

This https://arxiv.org/abs/1905.12948 has been replaced.
link: https://scholar.google.com/scholar?q=a

Global Momentum Compression for Sparse Communication in Distributed Learning
With the rapid growth of data, distributed momentum stochastic gradient descent~(DMSGD) has been widely used in distributed learning, especially for training large-scale deep models. Due to the latency and limited bandwidth of the network, communication has become the bottleneck of distributed learning. Communication compression with sparsified gradient, abbreviated as \emph{sparse communication}, has been widely employed to reduce communication cost. All existing works about sparse communicati…

This https://arxiv.org/abs/2401.11176 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_ees…

Data-Driven Target Localization: Benchmarking Gradient Descent Using the Cramer-Rao Bound
In modern radar systems, precise target localization using azimuth and velocity estimation is paramount. Traditional unbiased estimation methods have utilized gradient descent algorithms to reach the theoretical limits of the Cramer Rao Bound (CRB) for the error of the parameter estimates. As an extension, we demonstrate on a realistic simulated example scenario that our earlier presented data-driven neural network model outperforms these traditional methods, yielding improved accuracies in tar…

This https://arxiv.org/abs/2304.00707 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

High-dimensional scaling limits and fluctuations of online least-squares SGD with smooth covariance
We derive high-dimensional scaling limits and fluctuations for the online least-squares Stochastic Gradient Descent (SGD) algorithm by taking the properties of the data generating model explicitly into consideration. Our approach treats the SGD iterates as an interacting particle system, where the expected interaction is characterized by the covariance structure of the input. Assuming smoothness conditions on moments of order up to eight orders, and without explicitly assuming Gaussianity, we e…

Optimisation challenge for superconducting adiabatic neural network implementing XOR and OR boolean functions
D. S. Pashin, M. V. Bastrakova, D. A. Rybin, I. I. Soloviev, A. E. Schegolev, N. V. Klenov
https://arxiv.org/abs/2405.03521

Optimisation challenge for superconducting adiabatic neural network implementing XOR and OR boolean functions
In this article, we consider designs of simple analog artificial neural networks based on adiabatic Josephson cells with a sigmoid activation function. A new approach based on the gradient descent method is developed to adjust the circuit parameters, allowing efficient signal transmission between the network layers. The proposed solution is demonstrated on the example of the system implementing XOR and OR logical operations.

Fast Quantum Process Tomography via Riemannian Gradient Descent
Daniel Volya, Andrey Nikitin, Prabhat Mishra
https://arxiv.org/abs/2404.18840 https://

Fast Quantum Process Tomography via Riemannian Gradient Descent
Constrained optimization plays a crucial role in the fields of quantum physics and quantum information science and becomes especially challenging for high-dimensional complex structure problems. One specific issue is that of quantum process tomography, in which the goal is to retrieve the underlying quantum process based on a given set of measurement data. In this paper, we introduce a modified version of stochastic gradient descent on a Riemannian manifold that integrates recent advancements i…

This https://arxiv.org/abs/2211.08212 has been replaced.
link: https://scholar.google.com/scholar?q=a

A Homogeneous Second-Order Descent Method for Nonconvex Optimization
In this paper, we introduce a Homogeneous Second-Order Descent Method (HSODM) using the homogenized quadratic approximation to the original function. The merit of homogenization is that only the leftmost eigenvector of a gradient-Hessian integrated matrix is computed at each iteration. Therefore, the algorithm is a single-loop method that does not need to switch to other sophisticated algorithms and is easy to implement. We show that HSODM has a global convergence rate of $O(ε^{-3/2})$ to find…

How Transformers Learn Causal Structure with Gradient Descent
Eshaan Nichani, Alex Damian, Jason D. Lee
https://arxiv.org/abs/2402.14735 https://

How Transformers Learn Causal Structure with Gradient Descent
The incredible success of transformers on sequence modeling tasks can be largely attributed to the self-attention mechanism, which allows information to be transferred between different parts of a sequence. Self-attention allows transformers to encode causal structure which makes them particularly suitable for sequence modeling. However, the process by which transformers learn such causal structure via gradient-based training algorithms remains poorly understood. To better understand this proce…

This https://arxiv.org/abs/2211.17157 has been replaced.
link: https://scholar.google.com/scholar?q=a

Swarm-Based Gradient Descent Method for Non-Convex Optimization
We introduce a new Swarm-Based Gradient Descent (SBGD) method for non-convex optimization. The swarm consists of agents, each is identified with a position, ${\mathbf x}$, and mass, $m$. The key to their dynamics is communication: masses are being transferred from agents at high ground to low(-est) ground. At the same time, agents change positions with step size, $h=h({\mathbf x},m)$, adjusted to their relative mass: heavier agents proceed with small time-steps in the direction of local gradien…

Noise misleads rotation invariant algorithms on sparse targets
Manfred K. WarmuthGoogle Inc, Wojciech Kot{\l}owskiInstitute of Computing Science, Poznan University of Technology, Poznan, Poland, Matt JonesUniversity of Colorado Boulder, Colorado, USA, Ehsan AmidGoogle Inc
https://arxiv.org/abs/2403.02697

Noise misleads rotation invariant algorithms on sparse targets
It is well known that the class of rotation invariant algorithms are suboptimal even for learning sparse linear problems when the number of examples is below the "dimension" of the problem. This class includes any gradient descent trained neural net with a fully-connected input layer (initialized with a rotationally symmetric distribution). The simplest sparse problem is learning a single feature out of $d$ features. In that case the classification error or regression loss grows with $1-k/n$ wh…

On a Family of Relaxed Gradient Descent Methods for Quadratic Minimization
Liam MacDonald, Rua Murray, Rachael Tappenden
https://arxiv.org/abs/2404.19255 h…

On a Family of Relaxed Gradient Descent Methods for Quadratic Minimization
This paper studies the convergence properties of a family of Relaxed $\ell$-Minimal Gradient Descent methods for quadratic optimization; the family includes the omnipresent Steepest Descent method, as well as the Minimal Gradient method. Simple proofs are provided that show, in an appropriately chosen norm, the gradient and the distance of the iterates from optimality converge linearly, for all members of the family. Moreover, the function values decrease linearly, and iteration complexity resu…

Efficient simulations of Hartree--Fock equations by an accelerated gradient descent method
Y. Ohno, A. Del Maestro, T. I. Lakoba
https://arxiv.org/abs/2402.17843

Efficient simulations of Hartree--Fock equations by an accelerated gradient descent method
We develop convergence acceleration procedures that enable a gradient descent-type iteration method to efficiently simulate Hartree--Fock equations for atoms interacting both with each other and with an external potential. Our development focuses on three aspects: (i) optimization of a parameter in the preconditioning operator; (ii) adoption of a technique that eliminates the slowest-decaying mode to the case of many equations (describing many atoms); and (iii) a novel extension of the above te…

This https://arxiv.org/abs/2403.14045 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Accelerated Objective Gap and Gradient Norm Convergence for Gradient Descent via Long Steps
This work considers gradient descent for L-smooth convex optimization with stepsizes larger than the classic regime where descent can be ensured. The stepsize schedules considered are similar to but differ slightly from the concurrently developed silver stepsizes of Altschuler and Parillo. For one of our stepsize sequences, we prove a $O\left(\frac{1}{N^{1.2716\dots}}\right)$ convergence rate in terms of objective gap decrease and for the other, we show the same rate of decrease for the squared…

Noise misleads rotation invariant algorithms on sparse targets
Manfred K. WarmuthGoogle Inc, Wojciech Kot{\l}owskiInstitute of Computing Science, Poznan University of Technology, Poznan, Poland, Matt JonesUniversity of Colorado Boulder, Colorado, USA, Ehsan AmidGoogle Inc
https://arxiv.org/abs/2403.02697

Noise misleads rotation invariant algorithms on sparse targets
It is well known that the class of rotation invariant algorithms are suboptimal even for learning sparse linear problems when the number of examples is below the "dimension" of the problem. This class includes any gradient descent trained neural net with a fully-connected input layer (initialized with a rotationally symmetric distribution). The simplest sparse problem is learning a single feature out of $d$ features. In that case the classification error or regression loss grows with $1-k/n$ wh…

Subdifferentially polynomially bounded functions and Gaussian smoothing-based zeroth-order optimization
Ming Lei, Ting Kei Pong, Shuqin Sun, Man-Chung Yue
https://arxiv.org/abs/2405.04150

Subdifferentially polynomially bounded functions and Gaussian smoothing-based zeroth-order optimization
We introduce the class of subdifferentially polynomially bounded (SPB) functions, which is a rich class of locally Lipschitz functions that encompasses all Lipschitz functions, all gradient- or Hessian-Lipschitz functions, and even some non-smooth locally Lipschitz functions. We show that SPB functions are compatible with Gaussian smoothing (GS), in the sense that the GS of any SPB function is well-defined and satisfies a descent lemma akin to gradient-Lipschitz functions, with the Lipschitz co…

Failures and Successes of Cross-Validation for Early-Stopped Gradient Descent
Pratik Patil, Yuchen Wu, Ryan J. Tibshirani
https://arxiv.org/abs/2402.16793 …

Failures and Successes of Cross-Validation for Early-Stopped Gradient Descent
We analyze the statistical properties of generalized cross-validation (GCV) and leave-one-out cross-validation (LOOCV) applied to early-stopped gradient descent (GD) in high-dimensional least squares regression. We prove that GCV is generically inconsistent as an estimator of the prediction risk of early-stopped GD, even for a well-specified linear model with isotropic features. In contrast, we show that LOOCV converges uniformly along the GD trajectory to the prediction risk. Our theory requir…

This https://arxiv.org/abs/2211.10777 has been replaced.
link: https://scholar.google.com/scholar?q=a

Non-Coherent Over-the-Air Decentralized Gradient Descent
Decentralized Gradient Descent (DGD) is a popular algorithm used to solve decentralized optimization problems in diverse domains such as remote sensing, distributed inference, multi-agent coordination, and federated learning. Yet, executing DGD over wireless systems affected by noise, fading and limited bandwidth presents challenges, requiring scheduling of transmissions to mitigate interference and the acquisition of topology and channel state information -- complex tasks in wireless decentral…

Attacking Large Language Models with Projected Gradient Descent
Simon Geisler, Tom Wollschl\"ager, M. H. I. Abdalla, Johannes Gasteiger, Stephan G\"unnemann
https://arxiv.org/abs/2402.09154

Attacking Large Language Models with Projected Gradient Descent
Current LLM alignment methods are readily broken through specifically crafted adversarial prompts. While crafting adversarial prompts using discrete optimization is highly effective, such attacks typically use more than 100,000 LLM calls. This high computational cost makes them unsuitable for, e.g., quantitative analyses and adversarial training. To remedy this, we revisit Projected Gradient Descent (PGD) on the continuously relaxed input prompt. Although previous attempts with ordinary gradien…

Shuffling Momentum Gradient Algorithm for Convex Optimization
Trang H. Tran, Quoc Tran-Dinh, Lam M. Nguyen
https://arxiv.org/abs/2403.03180 https://…

Shuffling Momentum Gradient Algorithm for Convex Optimization
The Stochastic Gradient Descent method (SGD) and its stochastic variants have become methods of choice for solving finite-sum optimization problems arising from machine learning and data science thanks to their ability to handle large-scale applications and big datasets. In the last decades, researchers have made substantial effort to study the theoretical performance of SGD and its shuffling variants. However, only limited work has investigated its shuffling momentum variants, including shuffl…

MegaParticles: Range-based 6-DoF Monte Carlo Localization with GPU-Accelerated Stein Particle Filter
Kenji Koide, Shuji Oishi, Masashi Yokozuka, Atsuhiko Banno
https://arxiv.org/abs/2404.16370

MegaParticles: Range-based 6-DoF Monte Carlo Localization with GPU-Accelerated Stein Particle Filter
This paper presents a 6-DoF range-based Monte Carlo localization method with a GPU-accelerated Stein particle filter. To update a massive amount of particles, we propose a Gauss-Newton-based Stein variational gradient descent (SVGD) with iterative neighbor particle search. This method uses SVGD to collectively update particle states with gradient and neighborhood information, which provides efficient particle sampling. For an efficient neighbor particle search, it uses locality sensitive hashin…

This https://arxiv.org/abs/2303.10599 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_sta…

Convergence Analysis of Stochastic Gradient Descent with MCMC Estimators
Understanding stochastic gradient descent (SGD) and its variants is essential for machine learning. However, most of the preceding analyses are conducted under amenable conditions such as unbiased gradient estimator and bounded objective functions, which does not encompass many sophisticated applications, such as variational Monte Carlo, entropy-regularized reinforcement learning and variational inference. In this paper, we consider the SGD algorithm that employ the Markov Chain Monte Carlo (MC…

This https://arxiv.org/abs/2403.14045 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Accelerated Objective Gap and Gradient Norm Convergence for Gradient Descent via Long Steps
This work considers gradient descent for L-smooth convex optimization with stepsizes larger than the classic regime where descent can be ensured. The stepsize schedules considered are similar to but differ slightly from the concurrently developed silver stepsizes of Altschuler and Parillo. For one of our stepsize sequences, we prove a $O\left(\frac{1}{N^{1.2716\dots}}\right)$ convergence rate in terms of objective gap decrease and for the other, we show the same rate of decrease for the squared…

Organizing Physics with Open Energy-Driven Systems
Matteo Capucci, Owen Lynch, David I. Spivak
https://arxiv.org/abs/2404.16140 https://

Organizing Physics with Open Energy-Driven Systems
Organizing physics has been a long-standing preoccupation of applied category theory, going back at least to Lawvere. We contribute to this research thread by noticing that Hamiltonian mechanics and gradient descent depend crucially on a consistent choice of transformation -- which we call a reaction structure -- from the cotangent bundle to the tangent bundle. We then construct a compositional theory of reaction structures. Reaction-based systems offer a different perspective on composition in…

This https://arxiv.org/abs/2403.03473 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csLG_…

Inverse-Free Fast Natural Gradient Descent Method for Deep Learning
Second-order optimization techniques have the potential to achieve faster convergence rates compared to first-order methods through the incorporation of second-order derivatives or statistics. However, their utilization in deep learning is limited due to their computational inefficiency. Various approaches have been proposed to address this issue, primarily centered on minimizing the size of the matrix to be inverted. Nevertheless, the necessity of performing the inverse operation iteratively p…

Faster Convergence of Stochastic Accelerated Gradient Descent under Interpolation
Aaron Mishkin, Mert Pilanci, Mark Schmidt
https://arxiv.org/abs/2404.02378

Faster Convergence of Stochastic Accelerated Gradient Descent under Interpolation
We prove new convergence rates for a generalized version of stochastic Nesterov acceleration under interpolation conditions. Unlike previous analyses, our approach accelerates any stochastic gradient method which makes sufficient progress in expectation. The proof, which proceeds using the estimating sequences framework, applies to both convex and strongly convex functions and is easily specialized to accelerated SGD under the strong growth condition. In this special case, our analysis reduces …

MegaParticles: Range-based 6-DoF Monte Carlo Localization with GPU-Accelerated Stein Particle Filter
Kenji Koide, Shuji Oishi, Masashi Yokozuka, Atsuhiko Banno
https://arxiv.org/abs/2404.16370

MegaParticles: Range-based 6-DoF Monte Carlo Localization with GPU-Accelerated Stein Particle Filter
This paper presents a 6-DoF range-based Monte Carlo localization method with a GPU-accelerated Stein particle filter. To update a massive amount of particles, we propose a Gauss-Newton-based Stein variational gradient descent (SVGD) with iterative neighbor particle search. This method uses SVGD to collectively update particle states with gradient and neighborhood information, which provides efficient particle sampling. For an efficient neighbor particle search, it uses locality sensitive hashin…

This https://arxiv.org/abs/2311.00944 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_sta…

Stochastic Smoothed Gradient Descent Ascent for Federated Minimax Optimization
In recent years, federated minimax optimization has attracted growing interest due to its extensive applications in various machine learning tasks. While Smoothed Alternative Gradient Descent Ascent (Smoothed-AGDA) has proved its success in centralized nonconvex minimax optimization, how and whether smoothing technique could be helpful in federated setting remains unexplored. In this paper, we propose a new algorithm termed Federated Stochastic Smoothed Gradient Descent Ascent (FESS-GDA), which…

This https://arxiv.org/abs/2310.12140 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Online Estimation with Rolling Validation: Adaptive Nonparametric Estimation with Streaming Data
Online nonparametric estimators are gaining popularity due to their efficient computation and competitive generalization abilities. An important example includes variants of stochastic gradient descent. These algorithms often take one sample point at a time and instantly update the parameter estimate of interest. In this work we consider model selection and hyperparameter tuning for such online algorithms. We propose a weighted rolling-validation procedure, an online variant of leave-one-out cr…

Projected Block Coordinate Descent for sparse spike estimation
Pierre-Jean B\'enardIMB, Yann TraonmilinIMB, Jean Fran\c{c}ois AujolIMB
https://arxiv.org/abs/2402.12021

Projected Block Coordinate Descent for sparse spike estimation
We consider the problem of recovering off-the-grid spikes from linear measurements. The state of the art Over-Parametrized Continuous Orthogonal Matching Pursuit (OP-COMP) with Projected Gradient Descent (PGD) successfully recovers those signals. In most cases, the main computational cost lies in a unique global descent on all parameters (positions and amplitudes). In this paper, we propose to improve this algorithm by accelerating this descent step. We introduce a new algorithm, based on Blo…

This https://arxiv.org/abs/2311.00944 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_sta…

Stochastic Smoothed Gradient Descent Ascent for Federated Minimax Optimization
In recent years, federated minimax optimization has attracted growing interest due to its extensive applications in various machine learning tasks. While Smoothed Alternative Gradient Descent Ascent (Smoothed-AGDA) has proved its success in centralized nonconvex minimax optimization, how and whether smoothing technique could be helpful in federated setting remains unexplored. In this paper, we propose a new algorithm termed Federated Stochastic Smoothed Gradient Descent Ascent (FESS-GDA), which…

This https://arxiv.org/abs/2311.00521 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Gaussian smoothing gradient descent for minimizing functions (GSmoothGD)
This work analyzes the convergence of a class of smoothing-based gradient descent methods when applied to optimization problems. In particular, Gaussian smoothing is employed to define a nonlocal gradient that reduces high-frequency noise, small variations, and rapid fluctuations in the computation of the descent directions while preserving the structure and features of the loss landscape. The resulting Gaussian smoothing gradient descent (GSmoothGD) approach can facilitate gradient descent in …

This https://arxiv.org/abs/2311.00521 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Gaussian smoothing gradient descent for minimizing functions (GSmoothGD)
This work analyzes the convergence of a class of smoothing-based gradient descent methods when applied to optimization problems. In particular, Gaussian smoothing is employed to define a nonlocal gradient that reduces high-frequency noise, small variations, and rapid fluctuations in the computation of the descent directions while preserving the structure and features of the loss landscape. The resulting Gaussian smoothing gradient descent (GSmoothGD) approach can facilitate gradient descent in …

Analysing heavy-tail properties of Stochastic Gradient Descent by means of Stochastic Recurrence Equations
Ewa Damek, Sebastian Mentemeier
https://arxiv.org/abs/2403.13868

Analysing heavy-tail properties of Stochastic Gradient Descent by means of Stochastic Recurrence Equations
In recent works on the theory of machine learning, it has been observed that heavy tail properties of Stochastic Gradient Descent (SGD) can be studied in the probabilistic framework of stochastic recursions. In particular, Gürbüzbalaban et al. (arXiv:2006.04740) considered a setup corresponding to linear regression for which iterations of SGD can be modelled by a multivariate affine stochastic recursion $X_k=A_k X_{k-1}+B_k$, for independent and identically distributed pairs $(A_k, B_k)$, whe…

This https://arxiv.org/abs/2306.11246 has been replaced.
link: https://scholar.google.com/scholar?q=a

Neural Inventory Control in Networks via Hindsight Differentiable Policy Optimization
We argue that inventory management presents unique opportunities for reliably applying and evaluating deep reinforcement learning (DRL). Toward reliable application, we emphasize and test two techniques. The first is Hindsight Differentiable Policy Optimization (HDPO), which performs stochastic gradient descent to optimize policy performance while avoiding the need to repeatedly deploy randomized policies in the environment-as is common with generic policy gradient methods. Our second technique…

This https://arxiv.org/abs/2306.05896 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

One-step corrected projected stochastic gradient descent for statistical estimation
A generic, fast and asymptotically efficient method for parametric estimation is described. It is based on the projected stochastic gradient descent on the log-likelihood function corrected by a single step of the Fisher scoring algorithm. We show theoretically and by simulations that it is an interesting alternative to the usual stochastic gradient descent with averaging or the adaptative stochastic gradient descent.

This https://arxiv.org/abs/2006.07769 has been replaced.
link: https://scholar.google.com/scholar?q=a

Variance-Reduced Accelerated First-order Methods: Central Limit Theorems and Confidence Statements
In this paper, we study a stochastic strongly convex optimization problem and propose three classes of variable sample-size stochastic first-order methods including the standard stochastic gradient descent method, its accelerated variant, and the stochastic heavy ball method. In the iterates of each scheme, the unavailable exact gradients are approximated by averaging across an increasing batch size of sampled gradients. We prove that when the sample-size increases geometrically, the generated …

This https://arxiv.org/abs/2402.11858 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_sta…

Stochastic Hessian Fittings on Lie Groups
This paper studies the fitting of Hessian or its inverse for stochastic optimizations using a Hessian fitting criterion from the preconditioned stochastic gradient descent (PSGD) method, which is intimately related to many commonly used second order and adaptive gradient optimizers, e.g., BFGS, Gaussian-Newton and natural gradient descent, AdaGrad, etc. Our analyses reveal the efficiency and reliability differences among a wide range of preconditioner fitting methods, from closed-form to iterat…

This https://arxiv.org/abs/2311.18426 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Convergence Analysis of Fractional Gradient Descent
Fractional derivatives are a well-studied generalization of integer order derivatives. Naturally, for optimization, it is of interest to understand the convergence properties of gradient descent using fractional derivatives. Convergence analysis of fractional gradient descent is currently limited both in the methods analyzed and the settings analyzed. This paper aims to fill in these gaps by analyzing variations of fractional gradient descent in smooth and convex, smooth and strongly convex, an…

This https://arxiv.org/abs/2402.14423 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_qu…

The Universe as a Learning System
At its microscopic level, the universe follows the laws of quantum mechanics. Focusing on the quantum trajectories of particles as followed from the hydrodynamical formulation of quantum mechanics, we propose that under general requirements, quantum systems follow a disrupted version of the gradient descent model, a basic machine learning algorithm, where the learning is distorted due to the self-organizing process of the quantum system. Such a learning process is possible only when we assume d…

On the Convergence Rate of the Stochastic Gradient Descent (SGD) and application to a modified policy gradient for the Multi Armed Bandit
Stefana Anita, Gabriel Turinici
https://arxiv.org/abs/2402.06388

On the Convergence Rate of the Stochastic Gradient Descent (SGD) and application to a modified policy gradient for the Multi Armed Bandit
We present a self-contained proof of the convergence rate of the Stochastic Gradient Descent (SGD) when the learning rate follows an inverse time decays schedule; we next apply the results to the convergence of a modified form of policy gradient Multi-Armed Bandit (MAB) with $L2$ regularization.

This https://arxiv.org/abs/2305.17224 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Fast and Accurate Estimation of Low-Rank Matrices from Noisy Measurements via Preconditioned Non-Convex Gradient Descent
Non-convex gradient descent is a common approach for estimating a low-rank $n\times n$ ground truth matrix from noisy measurements, because it has per-iteration costs as low as $O(n)$ time, and is in theory capable of converging to a minimax optimal estimate. However, the practitioner is often constrained to just tens to hundreds of iterations, and the slow and/or inconsistent convergence of non-convex gradient descent can prevent a high-quality estimate from being obtained. Recently, the techn…

Faster Convergence for Transformer Fine-tuning with Line Search Methods
Philip Kenneweg, Leonardo Galli, Tristan Kenneweg, Barbara Hammer
https://arxiv.org/abs/2403.18506

Faster Convergence for Transformer Fine-tuning with Line Search Methods
Recent works have shown that line search methods greatly increase performance of traditional stochastic gradient descent methods on a variety of datasets and architectures [1], [2]. In this work we succeed in extending line search methods to the novel and highly popular Transformer architecture and dataset domains in natural language processing. More specifically, we combine the Armijo line search with the Adam optimizer and extend it by subdividing the networks architecture into sensible units…

This https://arxiv.org/abs/2306.10529 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Dropout Regularization Versus $\ell_2$-Penalization in the Linear Model
We investigate the statistical behavior of gradient descent iterates with dropout in the linear regression model. In particular, non-asymptotic bounds for the convergence of expectations and covariance matrices of the iterates are derived. The results shed more light on the widely cited connection between dropout and l2-regularization in the linear model. We indicate a more subtle relationship, owing to interactions between the gradient descent dynamics and the additional randomness induced by …

Convergence and Complexity Guarantee for Inexact First-order Riemannian Optimization Algorithms
Yuchen Li, Laura Balzano, Deanna Needell, Hanbaek Lyu
https://arxiv.org/abs/2405.03073

Convergence and Complexity Guarantee for Inexact First-order Riemannian Optimization Algorithms
We analyze inexact Riemannian gradient descent (RGD) where Riemannian gradients and retractions are inexactly (and cheaply) computed. Our focus is on understanding when inexact RGD converges and what is the complexity in the general nonconvex and constrained setting. We answer these questions in a general framework of tangential Block Majorization-Minimization (tBMM). We establish that tBMM converges to an $ε$-stationary point within $O(ε^{-2})$ iterations. Under a mild assumption, the result…

Improving Line Search Methods for Large Scale Neural Network Training
Philip Kenneweg, Tristan Kenneweg, Barbara Hammer
https://arxiv.org/abs/2403.18519 ht…

Improving Line Search Methods for Large Scale Neural Network Training
In recent studies, line search methods have shown significant improvements in the performance of traditional stochastic gradient descent techniques, eliminating the need for a specific learning rate schedule. In this paper, we identify existing issues in state-of-the-art line search methods, propose enhancements, and rigorously evaluate their effectiveness. We test these methods on larger datasets and more complex data domains than before. Specifically, we improve the Armijo line search by inte…

This https://arxiv.org/abs/2302.05797 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_sta…

Global Convergence Rate of Deep Equilibrium Models with General Activations
In a recent paper, Ling et al. investigated the over-parametrized Deep Equilibrium Model (DEQ) with ReLU activation. They proved that the gradient descent converges to a globally optimal solution for the quadratic loss function at a linear convergence rate. This paper shows that this fact still holds for DEQs with any generally bounded activation with bounded first and second derivatives. Since the new activation function is generally non-homogeneous, bounding the least eigenvalue of the Gram m…

This https://arxiv.org/abs/2306.10529 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Dropout Regularization Versus $\ell_2$-Penalization in the Linear Model
We investigate the statistical behavior of gradient descent iterates with dropout in the linear regression model. In particular, non-asymptotic bounds for the convergence of expectations and covariance matrices of the iterates are derived. The results shed more light on the widely cited connection between dropout and l2-regularization in the linear model. We indicate a more subtle relationship, owing to interactions between the gradient descent dynamics and the additional randomness induced by …

This https://arxiv.org/abs/2305.18502 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_sta…

Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGD
This study explores the sample complexity for two-layer neural networks to learn a generalized linear target function under Stochastic Gradient Descent (SGD), focusing on the challenging regime where many flat directions are present at initialization. It is well-established that in this scenario $n=O(d \log d)$ samples are typically needed. However, we provide precise results concerning the pre-factors in high-dimensional contexts and for varying widths. Notably, our findings suggest that overp…

This https://arxiv.org/abs/2403.16829 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csLG_…

Convergence of a model-free entropy-regularized inverse reinforcement learning algorithm
Given a dataset of expert demonstrations, inverse reinforcement learning (IRL) aims to recover a reward for which the expert is optimal. This work proposes a model-free algorithm to solve entropy-regularized IRL problem. In particular, we employ a stochastic gradient descent update for the reward and a stochastic soft policy iteration update for the policy. Assuming access to a generative model, we prove that our algorithm is guaranteed to recover a reward for which the expert is $\varepsilon$-…

A Penalty-Based Guardrail Algorithm for Non-Decreasing Optimization with Inequality Constraints
Ksenija Stepanovic, Wendelin B\"ohmer, Mathijs de Weerdt
https://arxiv.org/abs/2405.01984

A Penalty-Based Guardrail Algorithm for Non-Decreasing Optimization with Inequality Constraints
Traditional mathematical programming solvers require long computational times to solve constrained minimization problems of complex and large-scale physical systems. Therefore, these problems are often transformed into unconstrained ones, and solved with computationally efficient optimization approaches based on first-order information, such as the gradient descent method. However, for unconstrained problems, balancing the minimization of the objective function with the reduction of constraint …

This https://arxiv.org/abs/2402.11858 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_sta…

Stochastic Hessian Fittings with Lie Groups
This paper studies the fitting of Hessian or its inverse for stochastic optimizations using a Hessian fitting criterion from the preconditioned stochastic gradient descent (PSGD) method, which is intimately related to many commonly used second order and adaptive gradient optimizers, e.g., BFGS, Gaussian-Newton and natural gradient descent, AdaGrad, etc. Our analyses reveal the efficiency and reliability differences among a wide range of preconditioner fitting methods, from closed-form to iterat…

Generative Adversarial Network with Soft-Dynamic Time Warping and Parallel Reconstruction for Energy Time Series Anomaly Detection
Hardik Prabhu, Jayaraman Valadi, Pandarasamy Arjunan
https://arxiv.org/abs/2402.14384

Generative Adversarial Network with Soft-Dynamic Time Warping and Parallel Reconstruction for Energy Time Series Anomaly Detection
In this paper, we employ a 1D deep convolutional generative adversarial network (DCGAN) for sequential anomaly detection in energy time series data. Anomaly detection involves gradient descent to reconstruct energy sub-sequences, identifying the noise vector that closely generates them through the generator network. Soft-DTW is used as a differentiable alternative for the reconstruction loss and is found to be superior to Euclidean distance. Combining reconstruction loss and the latent space's …

This https://arxiv.org/abs/2402.12493 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

On Averaging and Extrapolation for Gradient Descent
This work considers the effect of averaging, and more generally extrapolation, of the iterates of gradient descent in smooth convex optimization. After running the method, rather than reporting the final iterate, one can report either a convex combination of the iterates (averaging) or a generic combination of the iterates (extrapolation). For several common stepsize sequences, including recently developed accelerated periodically long stepsize schemes, we show averaging cannot improve gradient…

This https://arxiv.org/abs/2402.11858 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_sta…

Stochastic Hessian Fittings with Lie Groups
This paper studies the fitting of Hessian or its inverse for stochastic optimizations using a Hessian fitting criterion from the preconditioned stochastic gradient descent (PSGD) method, which is intimately related to many commonly used second order and adaptive gradient optimizers, e.g., BFGS, Gaussian-Newton and natural gradient descent, AdaGrad, etc. Our analyses reveal the efficiency and reliability differences among a wide range of preconditioner fitting methods, from closed-form to iterat…

This https://arxiv.org/abs/2310.07922 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Polyak Minorant Method for Convex Optimization
In 1963 Boris Polyak suggested a particular step size for gradient descent methods, now known as the Polyak step size, that he later adapted to subgradient methods. The Polyak step size requires knowledge of the optimal value of the minimization problem, which is a strong assumption but one that holds for several important problems. In this paper we extend Polyak's method to handle constraints and, as a generalization of subgradients, general minorants, which are convex functions that tightly l…

This https://arxiv.org/abs/2403.14045 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Accelerated Objective Gap and Gradient Norm Convergence for Gradient Descent via Long Steps
This work considers gradient descent for L-smooth convex optimization with stepsizes larger than the classic regime where descent can be ensured. The stepsize schedules considered are similar to but differ slightly from the recent silver stepsizes of Altschuler and Parrilo. For one of our stepsize sequences, we prove a $O\left(N^{- 1.2716\dots}\right)$ convergence rate in terms of objective gap decrease and for the other, we show the same rate of decrease for squared-gradient-norm decrease. Thi…

Online Policy Learning and Inference by Matrix Completion
Congyuan Duan, Jingyang Li, Dong Xia
https://arxiv.org/abs/2404.17398 https://

Online Policy Learning and Inference by Matrix Completion
Making online decisions can be challenging when features are sparse and orthogonal to historical ones, especially when the optimal policy is learned through collaborative filtering. We formulate the problem as a matrix completion bandit (MCB), where the expected reward under each arm is characterized by an unknown low-rank matrix. The $ε$-greedy bandit and the online gradient descent algorithm are explored. Policy learning and regret performance are studied under a specific schedule for explor…

This https://arxiv.org/abs/2403.14045 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Accelerated Objective Gap and Gradient Norm Convergence for Gradient Descent via Long Steps
This work considers gradient descent for L-smooth convex optimization with stepsizes larger than the classic regime where descent can be ensured. The stepsize schedules considered are similar to but differ slightly from the recent silver stepsizes of Altschuler and Parrilo. For one of our stepsize sequences, we prove a $O\left(N^{- 1.2716\dots}\right)$ convergence rate in terms of objective gap decrease and for the other, we show the same rate of decrease for squared-gradient-norm decrease. Thi…

Convergence and Trade-Offs in Riemannian Gradient Descent and Riemannian Proximal Point
David Mart\'inez-Rubio, Christophe Roux, Sebastian Pokutta
https://arxiv.org/abs/2403.10429

Convergence and Trade-Offs in Riemannian Gradient Descent and Riemannian Proximal Point
In this work, we analyze two of the most fundamental algorithms in geodesically convex optimization: Riemannian gradient descent and (possibly inexact) Riemannian proximal point. We quantify their rates of convergence and produce different variants with several trade-offs. Crucially, we show the iterates naturally stay in a ball around an optimizer, of radius depending on the initial distance and, in some cases, on the curvature. In contrast, except for limited cases, previous works bounded the…

This https://arxiv.org/abs/2402.04691 has been replaced.
link: https://scholar.google.com/scholar?q=a

Learning Operators with Stochastic Gradient Descent in General Hilbert Spaces
This study investigates leveraging stochastic gradient descent (SGD) to learn operators between general Hilbert spaces. We propose weak and strong regularity conditions for the target operator to depict its intrinsic structure and complexity. Under these conditions, we establish upper bounds for convergence rates of the SGD algorithm and conduct a minimax lower bound analysis, further illustrating that our convergence analysis and regularity conditions quantitatively characterize the tractabili…

Gauss-Newton Natural Gradient Descent for Physics-Informed Computational Fluid Dynamics
Anas Jnini, Flavio Vella, Marius Zeinhofer
https://arxiv.org/abs/2402.10680

Gauss-Newton Natural Gradient Descent for Physics-Informed Computational Fluid Dynamics
We propose Gauss-Newton's method in function space for the solution of the Navier-Stokes equations in the physics-informed neural network (PINN) framework. Upon discretization, this yields a natural gradient method that provably mimics the function space dynamics. Our computational results demonstrate close to single-precision accuracy measured in relative $L^2$ norm on a number of benchmark problems. To the best of our knowledge, this constitutes the first contribution in the PINN literature t…

This https://arxiv.org/abs/2401.00691 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_sta…

Stochastic Gradient Descent for Additive Nonparametric Regression
This paper introduces an iterative algorithm for training additive models that enjoys favorable memory storage and computational requirements. The algorithm can be viewed as the functional counterpart of stochastic gradient descent, applied to the coefficients of a truncated basis expansion of the component functions. We show that the resulting estimator satisfies an oracle inequality that allows for model mis-specification. In the well-specified setting, by choosing the learning rate carefully…

Dissipative Gradient Descent Ascent Method: A Control Theory Inspired Algorithm for Min-max Optimization
Tianqi Zheng, Nicolas Loizou, Pengcheng You, Enrique Mallada
https://arxiv.org/abs/2403.09090

Dissipative Gradient Descent Ascent Method: A Control Theory Inspired Algorithm for Min-max Optimization
Gradient Descent Ascent (GDA) methods for min-max optimization problems typically produce oscillatory behavior that can lead to instability, e.g., in bilinear settings. To address this problem, we introduce a dissipation term into the GDA updates to dampen these oscillations. The proposed Dissipative GDA (DGDA) method can be seen as performing standard GDA on a state-augmented and regularized saddle function that does not strictly introduce additional convexity/concavity. We theoretically show …

This https://arxiv.org/abs/2305.12568 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Det-CGD: Compressed Gradient Descent with Matrix Stepsizes for Non-Convex Optimization
This paper introduces a new method for minimizing matrix-smooth non-convex objectives through the use of novel Compressed Gradient Descent (CGD) algorithms enhanced with a matrix-valued stepsize. The proposed algorithms are theoretically analyzed first in the single-node and subsequently in the distributed settings. Our theoretical results reveal that the matrix stepsize in CGD can capture the objective's structure and lead to faster convergence compared to a scalar stepsize. As a byproduct of …

This https://arxiv.org/abs/2403.05070 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

A global Barzilai and Borwein's gradient normalization descent method for multiobjective optimization
In this paper, we consider the unconstrained multiobjective optimization problem. In recent years, researchers pointed out that the steepest decent method may generate small stepsize which leads to slow convergence rates. To address the issue, we propose a global Barzilai and Borwein's gradient normalization descent method for multiobjective optimization (GBBN). In our method, we propose a new normalization technique to generate new descent direction. We demonstrate that line search can achieve…

This https://arxiv.org/abs/2307.12441 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Swarm-based optimization with random descent
We extend our study of the swarm-based gradient descent method for non-convex optimization, [Lu, Tadmor & Zenginoglu, arXiv:2211.17157], to allow random descent directions. We recall that the swarm-based approach consists of a swarm of agents, each identified with a position, ${\mathbf x}$, and mass, $m$. The key is the transfer of mass from high ground to low(-est) ground. The mass of an agent dictates its step size: lighter agents take larger steps. In this paper, the essential new feature is…

This https://arxiv.org/abs/2308.10547 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Decentralized Riemannian Conjugate Gradient Method on the Stiefel Manifold
The conjugate gradient method is a crucial first-order optimization method that generally converges faster than the steepest descent method, and its computational cost is much lower than that of second-order methods. However, while various types of conjugate gradient methods have been studied in Euclidean spaces and on Riemannian manifolds, there is little study for those in distributed scenarios. This paper proposes a decentralized Riemannian conjugate gradient descent (DRCGD) method that aims…

This https://arxiv.org/abs/2403.02967 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Non-Convex Stochastic Composite Optimization with Polyak Momentum
The stochastic proximal gradient method is a powerful generalization of the widely used stochastic gradient descent (SGD) method and has found numerous applications in Machine Learning. However, it is notoriously known that this method fails to converge in non-convex settings where the stochastic noise is significant (i.e. when only small or bounded batch sizes are used). In this paper, we focus on the stochastic proximal gradient method with Polyak momentum. We prove this method attains an opt…

This https://arxiv.org/abs/2402.04689 has been replaced.
link: https://scholar.google.com/scholar?q=a

Stein Boltzmann Sampling: A Variational Approach for Global Optimization
In this paper, we introduce a new flow-based method for global optimization of Lipschitz functions, called Stein Boltzmann Sampling (SBS). Our method samples from the Boltzmann distribution that becomes asymptotically uniform over the set of the minimizers of the function to be optimized. Candidate solutions are sampled via the \emph{Stein Variational Gradient Descent} algorithm. We prove the asymptotic convergence of our method, introduce two SBS variants, and provide a detailed comparison wit…

Riemannian Optimization and the Hartree-Fock Method
Caio O. da Silva
https://arxiv.org/abs/2403.15024 https://arxiv.org/pdf/2403.1502…

Riemannian Optimization and the Hartree-Fock Method
In the present work we studied a subfield of Applied Mathematics called Riemannian Optimization. The main goal of this subfield is to generalize algorithms, theorems and tools from Mathematical Optimization to the case in which the optimization problem is defined on a Riemannian manifold. As a case study, we implemented some of the main algorithms described in the literature (Gradient Descent, Newton-Raphson and Conjugate Gradient) to solve an optimization problem known as Hartree-Fock. This me…