Tootfinder

Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Time
Blake Bordelon, Mary I. Letey, Cengiz Pehlevan
https://arxiv.org/abs/2510.01098 https://

Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Time
We study in-context learning (ICL) of linear regression in a deep linear self-attention model, characterizing how performance depends on various computational and statistical resources (width, depth, number of training steps, batch size and data per context). In a joint limit where data dimension, context length, and residual stream width scale proportionally, we analyze the limiting asymptotics for three ICL settings: (1) isotropic covariates and tasks (ISO), (2) fixed and structured covarianc…

Assumption-lean Inference for Network-linked Data
Wei Li, Nilanjan Chakraborty, Robert Lunde
https://arxiv.org/abs/2510.00287 https://arxiv.org/pdf/2510.00…

Assumption-lean Inference for Network-linked Data
We consider statistical inference for network-linked regression problems, where covariates may include network summary statistics computed for each node. In settings involving network data, it is often natural to posit that latent variables govern connection probabilities in the graph. Since the presence of these latent features makes classical regression assumptions even less tenable, we propose an assumption-lean framework for linear regression with jointly exchangeable regression arrays. We …

Mathematical Theory of Collinearity Effects on Machine Learning Variable Importance Measures
Kelvyn K. Bladen, D. Richard Cutler, Alan Wisler
https://arxiv.org/abs/2510.00557 ht…

Mathematical Theory of Collinearity Effects on Machine Learning Variable Importance Measures
In many machine learning problems, understanding variable importance is a central concern. Two common approaches are Permute-and-Predict (PaP), which randomly permutes a feature in a validation set, and Leave-One-Covariate-Out (LOCO), which retrains models after permuting a training feature. Both methods deem a variable important if predictions with the original data substantially outperform those with permutations. In linear regression, empirical studies have linked PaP to regression coefficie…

Guaranteed Noisy CP Tensor Recovery via Riemannian Optimization on the Segre Manifold
Ke Xu, Yuefeng Han
https://arxiv.org/abs/2510.00569 https://arxiv.org…

Guaranteed Noisy CP Tensor Recovery via Riemannian Optimization on the Segre Manifold
Recovering a low-CP-rank tensor from noisy linear measurements is a central challenge in high-dimensional data analysis, with applications spanning tensor PCA, tensor regression, and beyond. We exploit the intrinsic geometry of rank-one tensors by casting the recovery task as an optimization problem over the Segre manifold, the smooth Riemannian manifold of rank-one tensors. This geometric viewpoint yields two powerful algorithms: Riemannian Gradient Descent (RGD) and Riemannian Gauss-Newton (R…

Information-Computation Tradeoffs for Noiseless Linear Regression with Oblivious Contamination
Ilias Diakonikolas, Chao Gao, Daniel M. Kane, John Lafferty, Ankit Pensia
https://arxiv.org/abs/2510.10665

Information-Computation Tradeoffs for Noiseless Linear Regression with Oblivious Contamination
We study the task of noiseless linear regression under Gaussian covariates in the presence of additive oblivious contamination. Specifically, we are given i.i.d.\ samples from a distribution $(x, y)$ on $\mathbb{R}^d \times \mathbb{R}$ with $x \sim \mathcal{N}(0,\mathbf{I}_d)$ and $y = x^\top β+ z$, where $z$ is drawn independently of $x$ from an unknown distribution $E$. Moreover, $z$ satisfies $\mathbb{P}_E[z = 0] = α>0$. The goal is to accurately recover the regressor $β$ to small $\ell_2…

On function-on-function linear quantile regression
Muge Mutis, Ufuk Beyaztas, Filiz Karaman, Han Lin Shang
https://arxiv.org/abs/2510.10792 https://arxiv.o…

On function-on-function linear quantile regression
We present two innovative functional partial quantile regression algorithms designed to accurately and efficiently estimate the regression coefficient function within the function-on-function linear quantile regression model. Our algorithms utilize functional partial quantile regression decomposition to effectively project the infinite-dimensional response and predictor variables onto a finite-dimensional space. Within this framework, the partial quantile regression components are approximated …

An efficient algorithm for kernel quantile regression
Shengxiang Deng, Xudong Li, Yangjing Zhang
https://arxiv.org/abs/2510.07929 https://arxiv.org/pdf/251…

An efficient algorithm for kernel quantile regression
Kernel Quantile Regression (KQR) extends classical quantile regression to nonlinear settings using kernel methods, offering powerful tools for modeling conditional distributions. However, its application to large-scale datasets is severely limited by the computational burden of the large, dense kernel matrix and the need to efficiently solve large, often ill-conditioned linear systems. Existing state-of-the-art solvers usually struggle with scalability. In this paper, we propose a novel and hig…

Cellular Learning: Scattered Data Regression in High Dimensions via Voronoi Cells
Shankar Prasad Sastry
https://arxiv.org/abs/2510.03810 https://arxiv.org/…

Cellular Learning: Scattered Data Regression in High Dimensions via Voronoi Cells
I present a regression algorithm that provides a continuous, piecewise-smooth function approximating scattered data. It is based on composing and blending linear functions over Voronoi cells, and it scales to high dimensions. The algorithm infers Voronoi cells from seed vertices and constructs a linear function for the input data in and around each cell. As the algorithm does not explicitly compute the Voronoi diagram, it avoids the curse of dimensionality. An accuracy of around 98.2% on the MN…

Learning Linear Regression with Low-Rank Tasks in-Context
Kaito Takanami, Takashi Takahashi, Yoshiyuki Kabashima
https://arxiv.org/abs/2510.04548 https://a…

Learning Linear Regression with Low-Rank Tasks in-Context
In-context learning (ICL) is a key building block of modern large language models, yet its theoretical mechanisms remain poorly understood. It is particularly mysterious how ICL operates in real-world applications where tasks have a common structure. In this work, we address this problem by analyzing a linear attention model trained on low-rank regression tasks. Within this setting, we precisely characterize the distribution of predictions and the generalization error in the high-dimensional li…

Uncertainty in Machine Learning
Hans Weytjens, Wouter Verbeke
https://arxiv.org/abs/2510.06007 https://arxiv.org/pdf/2510.06007

Uncertainty in Machine Learning
This book chapter introduces the principles and practical applications of uncertainty quantification in machine learning. It explains how to identify and distinguish between different types of uncertainty and presents methods for quantifying uncertainty in predictive models, including linear regression, random forests, and neural networks. The chapter also covers conformal prediction as a framework for generating predictions with predefined confidence intervals. Finally, it explores how uncerta…

Robust Functional Logistic Regression
Berkay Akturk, Ufuk Beyaztas, Han Lin Shang
https://arxiv.org/abs/2510.12048 https://arxiv.org/pdf/2510.12048

Robust Functional Logistic Regression
Functional logistic regression is a popular model to capture a linear relationship between binary response and functional predictor variables. However, many methods used for parameter estimation in functional logistic regression are sensitive to outliers, which may lead to inaccurate parameter estimates and inferior classification accuracy. We propose a robust estimation procedure for functional logistic regression, in which the observations of the functional predictor are projected onto a set …

Locally Linear Convergence for Nonsmooth Convex Optimization via Coupled Smoothing and Momentum
Reza Rahimi Baghbadorani, Sergio Grammatico, Peyman Mohajerin Esfahani
https://arxiv.org/abs/2511.10239 https://arxiv.org/pdf/2511.10239 https://arxiv.org/html/2511.10239
arXiv:2511.10239v1 Announce Type: new
Abstract: We propose an adaptive accelerated smoothing technique for a nonsmooth convex optimization problem where the smoothing update rule is coupled with the momentum parameter. We also extend the setting to the case where the objective function is the sum of two nonsmooth functions. With regard to convergence rate, we provide the global (optimal) sublinear convergence guarantees of O(1/k), which is known to be provably optimal for the studied class of functions, along with a local linear rate if the nonsmooth term fulfills a so-call locally strong convexity condition. We validate the performance of our algorithm on several problem classes, including regression with the l1-norm (the Lasso problem), sparse semidefinite programming (the MaxCut problem), Nuclear norm minimization with application in model free fault diagnosis, and l_1-regularized model predictive control to showcase the benefits of the coupling. An interesting observation is that although our global convergence result guarantees O(1/k) convergence, we consistently observe a practical transient convergence rate of O(1/k^2), followed by asymptotic linear convergence as anticipated by the theoretical result. This two-phase behavior can also be explained in view of the proposed smoothing rule.
toXiv_bot_toot

Bayesian Transfer Learning for High-Dimensional Linear Regression via Adaptive Shrinkage
Parsa Jamshidian, Donatello Telesca
https://arxiv.org/abs/2510.03449 https://

Bayesian Transfer Learning for High-Dimensional Linear Regression via Adaptive Shrinkage
We introduce BLAST, Bayesian Linear regression with Adaptive Shrinkage for Transfer, a Bayesian multi-source transfer learning framework for high-dimensional linear regression. The proposed analytical framework leverages global-local shrinkage priors together with Bayesian source selection to balance information sharing and regularization. We show how Bayesian source selection allows for the extraction of the most useful data sources, while discounting biasing information that may lead to negat…

Crosslisted article(s) found for physics.geo-ph. https://arxiv.org/list/physics.geo-ph/new
[1/1]:
- Rethinking deep learning: linear regression remains a key benchmark in predicting terrestrial wat...
Nie, Kumar, Chen, Zhao, Skulovich, Yoo, Pflug, Ahmad, Konapala

A mesh-free, derivative-free, matrix-free, and highly parallel localized stochastic method for high-dimensional semilinear parabolic PDEs
Shuixin Fang, Changtao Sheng, Bihao Su, Tao Zhou
https://arxiv.org/abs/2510.02635

A mesh-free, derivative-free, matrix-free, and highly parallel localized stochastic method for high-dimensional semilinear parabolic PDEs
We develop a mesh-free, derivative-free, matrix-free, and highly parallel localized stochastic method for high-dimensional semilinear parabolic PDEs. The efficiency of the proposed method is built upon four essential components: (i) a martingale formulation of the forward backward stochastic differential equation (FBSDE); (ii) a small scale stochastic particle method for local linear regression (LLR); (iii) a decoupling strategy with a matrix-free solver for the weighted least-squares system us…

Risk Phase Transitions in Spiked Regression: Alignment Driven Benign and Catastrophic Overfitting
Jiping Li, Rishi Sonthalia
https://arxiv.org/abs/2510.01414 https://

Risk Phase Transitions in Spiked Regression: Alignment Driven Benign and Catastrophic Overfitting
This paper analyzes the generalization error of minimum-norm interpolating solutions in linear regression using spiked covariance data models. The paper characterizes how varying spike strengths and target-spike alignments can affect risk, especially in overparameterized settings. The study presents an exact expression for the generalization error, leading to a comprehensive classification of benign, tempered, and catastrophic overfitting regimes based on spike strength, the aspect ratio $c=d/n…

Bayesian Profile Regression with Linear Mixed Models (Profile-LMM) applied to Longitudinal Exposome Data
Matteo Amestoy, Mark van de Wiel, Jeroen Lakerveld, Wessel van Wieringen
https://arxiv.org/abs/2510.08304

Bayesian Profile Regression with Linear Mixed Models (Profile-LMM) applied to Longitudinal Exposome Data
Exposure to diverse non-genetic factors, known as the exposome, is a critical determinant of health outcomes. However, analyzing the exposome presents significant methodological challenges, including: high collinearity among exposures, the longitudinal nature of repeated measurements, and potential complex interactions with individual characteristics. In this paper, we address these challenges by proposing a novel statistical framework that extends Bayesian profile regression. Our method integr…

Optimality and computational barriers in variable selection under dependence
Ming Gao, Bryon Aragam
https://arxiv.org/abs/2510.03990 https://arxiv.org/pdf/…

Optimality and computational barriers in variable selection under dependence
We study the optimal sample complexity of variable selection in linear regression under general design covariance, and show that subset selection is optimal while under standard complexity assumptions, efficient algorithms for this problem do not exist. Specifically, we analyze the variable selection problem and provide the optimal sample complexity with exact dependence on the problem parameters for both known and unknown sparsity settings. Moreover, we establish a sample complexity lower boun…

Fitting sparse high-dimensional varying-coefficient models with Bayesian regression tree ensembles
Soham Ghosh, Saloni Bhogale, Sameer K. Deshpande
https://arxiv.org/abs/2510.08204

Fitting sparse high-dimensional varying-coefficient models with Bayesian regression tree ensembles
By allowing the effects of $p$ covariates in a linear regression model to vary as functions of $R$ additional effect modifiers, varying-coefficient models (VCMs) strike a compelling balance between interpretable-but-rigid parametric models popular in classical statistics and flexible-but-opaque methods popular in machine learning. But in high-dimensional settings where $p$ and/or $R$ exceed the number of observations, existing approaches to fitting VCMs fail to identify which covariates have a …

dHPR: A Distributed Halpern Peaceman--Rachford Method for Non-smooth Distributed Optimization Problems
Zhangcheng Feng, Defeng Sun, Yancheng Yuan, Guojun Zhang
https://arxiv.org/abs/2511.10069 https://arxiv.org/pdf/2511.10069 https://arxiv.org/html/2511.10069
arXiv:2511.10069v1 Announce Type: new
Abstract: This paper introduces the distributed Halpern Peaceman--Rachford (dHPR) method, an efficient algorithm for solving distributed convex composite optimization problems with non-smooth objectives, which achieves a non-ergodic $O(1/k)$ iteration complexity regarding Karush--Kuhn--Tucker residual. By leveraging the symmetric Gauss--Seidel decomposition, the dHPR effectively decouples the linear operators in the objective functions and consensus constraints while maintaining parallelizability and avoiding additional large proximal terms, leading to a decentralized implementation with provably fast convergence. The superior performance of dHPR is demonstrated through comprehensive numerical experiments on distributed LASSO, group LASSO, and $L_1$-regularized logistic regression problems.
toXiv_bot_toot

High-dimensional Analysis of Synthetic Data Selection
Parham Rezaei, Filip Kovacevic, Francesco Locatello, Marco Mondelli
https://arxiv.org/abs/2510.08123 https://

High-dimensional Analysis of Synthetic Data Selection
Despite the progress in the development of generative models, their usefulness in creating synthetic data that improve prediction performance of classifiers has been put into question. Besides heuristic principles such as "synthetic data should be close to the real data distribution", it is actually not clear which specific properties affect the generalization error. Our paper addresses this question through the lens of high-dimensional regression. Theoretically, we show that, for linear models…

Inference in pseudo-observation-based regression using (biased) covariance estimation and naive bootstrapping
Simon Mack, Morten Overgaard, Dennis Dobler
https://arxiv.org/abs/2510.06815

Inference in pseudo-observation-based regression using (biased) covariance estimation and naive bootstrapping
We demonstrate that the usual Huber-White estimator is not consistent for the limiting covariance of parameter estimates in pseudo-observation regression approaches. By confirming that a plug-in estimator can be used instead, we obtain asymptotically exact and consistent tests for general linear hypotheses in the parameters of the model. Additionally, we confirm that naive bootstrapping can not be used for covariance estimation in the pseudo-observation model either. However, it can be used for…

Crosslisted article(s) found for stat.CO. https://arxiv.org/list/stat.CO/new
[1/1]:
- Bayesian Transfer Learning for High-Dimensional Linear Regression via Adaptive Shrinkage
Parsa Jamshidian, Donatello Telesca

Adaptive randomized pivoting and volume sampling
Ethan N. Epperly
https://arxiv.org/abs/2510.02513 https://arxiv.org/pdf/2510.02513

Adaptive randomized pivoting and volume sampling
Adaptive randomized pivoting (ARP) is a recently proposed and highly effective algorithm for column subset selection. This paper reinterprets the ARP algorithm by drawing connections to the volume sampling distribution and active learning algorithms for linear regression. As consequences, this paper presents new analysis for the ARP algorithm and faster implementations using rejection sampling.

Repro Samples Method for Model-Free Inference in High-Dimensional Binary Classification
Xiaotian Hou, Peng Wang, Minge Xie, Linjun Zhang
https://arxiv.org/abs/2510.01468 https:/…

Repro Samples Method for Model-Free Inference in High-Dimensional Binary Classification
This paper presents a novel method for statistical inference in high-dimensional binary models with unspecified structure, where we leverage a (potentially misspecified) sparsity-constrained working generalized linear model (GLM) to facilitate the inference process. Our method is based on the repro samples framework, which generates artificial samples that mimic the actual data-generating process. Our inference targets include the model support, case probabilities, and the oracle regression coe…