Tootfinder

Saddle Hierarchy in Dense Associative Memory
Robin Th\'eriault, Daniele Tantari
https://arxiv.org/abs/2508.19151 https://arxiv.org/pdf/2508.19151

Saddle Hierarchy in Dense Associative Memory
Dense associative memory (DAM) models have been attracting renewed attention since they were shown to be robust to adversarial examples and closely related to state-of-the-art machine learning paradigms, such as the attention mechanisms in transformers and generative diffusion models. We study a DAM built upon a three-layer Boltzmann machine with Potts hidden units, which represent data clusters and classes. Through a statistical mechanics analysis, we derive saddle-point equations that charact…

Mean-Field Generalisation Bounds for Learning Controls in Stochastic Environments
Boris Baros, Samuel N. Cohen, Christoph Reisinger
https://arxiv.org/abs/2508.16001 https://

Mean-Field Generalisation Bounds for Learning Controls in Stochastic Environments
We consider a data-driven formulation of the classical discrete-time stochastic control problem. Our approach exploits the natural structure of many such problems, in which significant portions of the system are uncontrolled. Employing the dynamic programming principle and the mean-field interpretation of single-hidden layer neural networks, we formulate the control problem as a series of infinite-dimensional minimisation problems. When regularised carefully, we provide practically verifiable a…

A Study of Hybrid and Evolutionary Metaheuristics for Single Hidden Layer Feedforward Neural Network Architecture
Gautam Siddharth Kashyap, Md Tabrez Nafis, Samar Wazir
https://arxiv.org/abs/2506.15737

A Study of Hybrid and Evolutionary Metaheuristics for Single Hidden Layer Feedforward Neural Network Architecture
Training Artificial Neural Networks (ANNs) with Stochastic Gradient Descent (SGD) frequently encounters difficulties, including substantial computing expense and the risk of converging to local optima, attributable to its dependence on partial weight gradients. Therefore, this work investigates Particle Swarm Optimization (PSO) and Genetic Algorithms (GAs) - two population-based Metaheuristic Optimizers (MHOs) - as alternatives to SGD to mitigate these constraints. A hybrid PSO-SGD strategy is …

Solar Cycle Variations in Meridional Flows and Rotational Shear within the Sun's Near-surface Shear Layer: #solar rotation: https://www.nature.com/articles/d44151-025-00101-6 - scientists uncover how deep plasma flows reshape the Sun’s rotation and hint at mysteries below its surface.

Rank Inspired Neural Network for solving linear partial differential equations
Wentao Peng, Yunqing Huang, Nianyu Yi
https://arxiv.org/abs/2506.17654 https…

Rank Inspired Neural Network for solving linear partial differential equations
This paper proposes a rank inspired neural network (RINN) to tackle the initialization sensitivity issue of physics informed extreme learning machines (PIELM) when numerically solving partial differential equations (PDEs). Unlike PIELM which randomly initializes the parameters of its hidden layers, RINN incorporates a preconditioning stage. In this stage, covariance-driven regularization is employed to optimize the orthogonality of the basis functions generated by the last hidden layer. The key…

Hybrid Least Squares/Gradient Descent Methods for DeepONets
Jun Choi, Chang-Ock Lee, Minam Moon
https://arxiv.org/abs/2508.15394 https://arxiv.org/pdf/2508…

Hybrid Least Squares/Gradient Descent Methods for DeepONets
We propose an efficient hybrid least squares/gradient descent method to accelerate DeepONet training. Since the output of DeepONet can be viewed as linear with respect to the last layer parameters of the branch network, these parameters can be optimized using a least squares (LS) solve, and the remaining hidden layer parameters are updated by means of gradient descent form. However, building the LS system for all possible combinations of branch and trunk inputs yields a prohibitively large line…

On the Effect of Uncertainty on Layer-wise Inference Dynamics
Sunwoo Kim, Haneul Yoo, Alice Oh
https://arxiv.org/abs/2507.06722 https://

On the Effect of Uncertainty on Layer-wise Inference Dynamics
Understanding how large language models (LLMs) internally represent and process their predictions is central to detecting uncertainty and preventing hallucinations. While several studies have shown that models encode uncertainty in their hidden states, it is underexplored how this affects the way they process such hidden states. In this work, we demonstrate that the dynamics of output token probabilities across layers for certain and uncertain outputs are largely aligned, revealing that uncerta…

KKT-based optimality conditions for neural network approximation
Vinesha Peiris, Nadezda Sukhorukova, Julien Ugon
https://arxiv.org/abs/2506.17305 https://…

KKT-based optimality conditions for neural network approximation
In this paper, we obtain necessary optimality conditions for neural network approximation. We consider neural networks in Manhattan ($l_1$ norm) and Chebyshev ($\max$ norm). The optimality conditions are based on neural networks with at most one hidden layer. We reformulate nonsmooth unconstrained optimisation problems as larger dimension constrained problems with smooth objective functions and constraints. Then we use KKT conditions to develop the necessary conditions and present the optimalit…

Widely Linear Augmented Extreme Learning Machine Based Impairments Compensation for Satellite Communications
Yang Luo, Arunprakash Jayaprakash, Gaojie Chen, Chong Huang, Qu Luo, De Mi, Pei Xiao
https://arxiv.org/abs/2506.14557

Widely Linear Augmented Extreme Learning Machine Based Impairments Compensation for Satellite Communications
Satellite communications are crucial for the evolution beyond fifth-generation networks. However, the dynamic nature of satellite channels and their inherent impairments present significant challenges. In this paper, a novel post-compensation scheme that combines the complex-valued extreme learning machine with augmented hidden layer (CELMAH) architecture and widely linear processing (WLP) is developed to address these issues by exploiting signal impropriety in satellite communications. Althoug…

Out & about in the Bernina Range, on our way to the Labyrinth, surrounded by majestic 4000ers (meters)... A dream! 😍
(#ThrowbackThursday, preemptive #FootpathFriday and my personal hiking highlight of 2024)

Regional, Lattice and Logical Representations of Neural Networks
Sandro Preto (Federal University of ABC, Brazil), Marcelo Finger (University of Sao Paulo, Brazil)
https://arxiv.org/abs/2506.05834

Regional, Lattice and Logical Representations of Neural Networks
A possible path to the interpretability of neural networks is to (approximately) represent them in the regional format of piecewise linear functions, where regions of inputs are associated to linear functions computing the network outputs. We present an algorithm for the translation of feedforward neural networks with ReLU activation functions in hidden layers and truncated identity activation functions in the output layer. We also empirically investigate the complexity of regional representati…

Generalization performance of narrow one-hidden layer networks in the teacher-student setting
Jean Barbier, Federica Gerace, Alessandro Ingrosso, Clarissa Lauditi, Enrico M. Malatesta, Gibbs Nwemadji, Rodrigo P\'erez Ortiz
https://arxiv.org/abs/2507.00629

Generalization performance of narrow one-hidden layer networks in the teacher-student setting
Understanding the generalization abilities of neural networks for simple input-output distributions is crucial to account for their learning performance on real datasets. The classical teacher-student setting, where a network is trained from data obtained thanks to a label-generating teacher model, serves as a perfect theoretical test bed. In this context, a complete theoretical account of the performance of fully connected one-hidden layer networks in the presence of generic activation functio…

Uncertainty Quantification for Large-Scale Deep Networks via Post-StoNet Modeling
Yan Sun, Faming Liang
https://arxiv.org/abs/2508.01217 https://arxiv.org/…

Uncertainty Quantification for Large-Scale Deep Networks via Post-StoNet Modeling
Deep learning has revolutionized modern data science. However, how to accurately quantify the uncertainty of predictions from large-scale deep neural networks (DNNs) remains an unresolved issue. To address this issue, we introduce a novel post-processing approach. This approach feeds the output from the last hidden layer of a pre-trained large-scale DNN model into a stochastic neural network (StoNet), then trains the StoNet with a sparse penalty on a validation dataset and constructs prediction…

SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning
Lingkun Long, Rubing Yang, Yushi Huang, Desheng Hui, Ao Zhou, Jianlei Yang
https://arxiv.org/abs/2508.06447

SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning
Long-context inference for Large Language Models (LLMs) is heavily limited by high computational demands. While several existing methods optimize attention computation, they still process the full set of hidden states at each layer, limiting overall efficiency. In this work, we propose SlimInfer, an innovative framework that aims to accelerate inference by directly pruning less critical prompt tokens during the forward pass. Our key insight is an information diffusion phenomenon: As information…

An em algorithm for quantum Boltzmann machines
Takeshi Kimura, Kohtaro Kato, Masahito Hayashi
https://arxiv.org/abs/2507.21569 https://arxiv.org/pdf/2507.2…

An em algorithm for quantum Boltzmann machines
We develop a quantum version of the em algorithm for training quantum Boltzmann machines. The em algorithm is an information-geometric extension of the well-known expectation-maximization (EM) algorithm, offering a structured alternative to gradient-based methods with potential advantages in stability and convergence. We implement the algorithm on a semi-quantum restricted Boltzmann machine, where quantum effects are confined to the hidden layer. This structure enables analytical update rules w…

Efficient Interleaved Speech Modeling through Knowledge Distillation
Mohammadmahdi Nouriborji, Morteza Rohanian
https://arxiv.org/abs/2506.23670 https://…

Efficient Interleaved Speech Modeling through Knowledge Distillation
Current speech language models exceed the size and latency constraints of many deployment environments. We build compact, expressive speech generation models through layer-aligned distillation, matching hidden states, attention maps, and softened logits to compress large multimodal transformers by 3x with minimal loss in performance. We introduce TinyWave, a family of 2B-parameter models for speech-to-speech and interleaved speech-text generation, trained on 50,000 hours of public audio. TinyWa…

This https://arxiv.org/abs/2403.05809 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_mat…

Shallow ReLU neural networks and finite elements
We point out that (continuous or discontinuous) piecewise linear functions on a convex polytope mesh can be represented by two-hidden-layer ReLU neural networks in a weak sense. In addition, the numbers of neurons of the two hidden layers required to weakly represent are accurately given based on the numbers of polytopes and hyperplanes involved in this mesh. The results naturally hold for constant and linear finite element functions. Such weak representation establishes a bridge between shallo…

Sharp uniform approximation for spectral Barron functions by deep neural networks
Yulei Liao, Pingbing Ming, Hao Yu
https://arxiv.org/abs/2507.06789 https:…

Sharp uniform approximation for spectral Barron functions by deep neural networks
This work explores the neural network approximation capabilities for functions within the spectral Barron space $\mathscr{B}^s$, where $s$ is the smoothness index. We demonstrate that for functions in $\mathscr{B}^{1/2}$, a shallow neural network (a single hidden layer) with $N$ units can achieve an $L^p$-approximation rate of $\mathcal{O}(N^{-1/2})$. This rate also applies to uniform approximation, differing by at most a logarithmic factor. Our results significantly reduce the smoothness requi…

Game-Theoretic Gradient Control for Robust Neural Network Training
Maria Zaitseva, Ivan Tomilov, Natalia Gusarova
https://arxiv.org/abs/2507.19143 https://…

Game-Theoretic Gradient Control for Robust Neural Network Training
Feed-forward neural networks (FFNNs) are vulnerable to input noise, reducing prediction performance. Existing regularization methods like dropout often alter network architecture or overlook neuron interactions. This study aims to enhance FFNN noise robustness by modifying backpropagation, interpreted as a multi-agent game, and exploring controlled target variable noising. Our "gradient dropout" selectively nullifies hidden layer neuron gradients with probability 1 - p during backpropagation, w…