Tootfinder

Bellman Optimality of Average-Reward Robust Markov Decision Processes with a Constant Gain
Shengbo Wang, Nian Si
https://arxiv.org/abs/2509.14203 https://a…

Bellman Optimality of Average-Reward Robust Markov Decision Processes with a Constant Gain
Learning and optimal control under robust Markov decision processes (MDPs) have received increasing attention, yet most existing theory, algorithms, and applications focus on finite-horizon or discounted models. The average-reward formulation, while natural in many operations research and management contexts, remains underexplored. This is primarily because the dynamic programming foundations are technically challenging and only partially understood, with several fundamental questions remaining…

Online Bayesian Risk-Averse Reinforcement Learning
Yuhao Wang, Enlu Zhou
https://arxiv.org/abs/2509.14077 https://arxiv.org/pdf/2509.14077

Online Bayesian Risk-Averse Reinforcement Learning
In this paper, we study the Bayesian risk-averse formulation in reinforcement learning (RL). To address the epistemic uncertainty due to a lack of data, we adopt the Bayesian Risk Markov Decision Process (BRMDP) to account for the parameter uncertainty of the unknown underlying model. We derive the asymptotic normality that characterizes the difference between the Bayesian risk value function and the original value function under the true unknown distribution. The results indicate that the Baye…

Semi-parametric Markov models for multi-type point patterns
Ib Thorsgaard Jensen, Jean-Fran\c{c}ois Coeurjolly, Rasmus Waagepetersen
https://arxiv.org/abs/2510.11226 https://

Semi-parametric Markov models for multi-type point patterns
Multi-type Markov point processes offer a flexible framework for modelling complex multi-type point patterns where it is pertinent to capture both interactions between points as well as large scale trends depending on observed covariates. However, estimation of interaction and covariate effects may be seriously biased in the presence of unobserved spatial confounders. In this paper we introduce a new class of semi-parametric Markov point processes that adjusts for spatial confounding through a …

Sound Value Iteration for Simple Stochastic Games
Muqsit Azeem (Technical University of Munich), Jan Kretinsky (Masaryk University), Maximilian Weininger (Ruhr-University Bochum)
https://arxiv.org/abs/2509.14112

Sound Value Iteration for Simple Stochastic Games
Algorithmic analysis of Markov decision processes (MDP) and stochastic games (SG) in practice relies on value-iteration (VI) algorithms. Since basic VI does not provide guarantees on the precision of the result, variants of VI have been proposed that offer such guarantees. In particular, sound value iteration (SVI) not only provides precise lower and upper bounds on the result, but also converges faster in the presence of probabilistic cycles. Unfortunately, it is neither applicable to SG, nor …

Secret-Key Agreement Through Hidden Markov Modeling of Wavelet Scattering Embeddings
Nora Basha, Bechir Hamdaoui, Attila A. Yavuz, Thang Hoang, Mehran Mozaffari Kermani
https://arxiv.org/abs/2510.09773

Secret-Key Agreement Through Hidden Markov Modeling of Wavelet Scattering Embeddings
Secret-key generation and agreement based on wireless channel reciprocity offers a promising avenue for securing IoT networks. However, existing approaches predominantly rely on the similarity of instantaneous channel measurement samples between communicating devices. This narrow view of reciprocity is often impractical, as it is highly susceptible to noise, asynchronous sampling, channel fading, and other system-level imperfections -- all of which significantly impair key generation performanc…

Long-term unsupervised recalibration of cursor-based intracortical brain–computer interfaces using a hidden Markov model #BCI

Long-term unsupervised recalibration of cursor-based intracortical brain–computer interfaces using a hidden Markov model - Nature Biomedical Engineering
A hidden Markov model that uses probabilistic retrospective inference allows for up to one month of unsupervised recalibration in an online cursor-based intracortical brain–machine interface.

Simulation-based Inference of Massive Black Hole Binaries using Sequential Neural Likelihood
Iv\'an Mart\'in V\'ilchez, Carlos F. Sopuerta
https://arxiv.org/abs/2509.13842

Simulation-based Inference of Massive Black Hole Binaries using Sequential Neural Likelihood
We propose a machine learning-based approach for parameter estimation of Massive Black Hole Binaries (MBHBs), leveraging normalizing flows to approximate the likelihood function. By training these flows on simulated data, we can generate posterior samples via Markov Chain Monte Carlo with a relatively reduced computational cost. Our method enables iterative refinement of smaller models targeting specific MBHB events, with significantly fewer waveform template evaluations. However, dimensionalit…

Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Markov Likelihood
Xingyu Lin, Yilin Wen, En Wang, Du Su, Wenbin Liu, Chenfu Bao, Zhonghou Lv
https://arxiv.org/abs/2510.09369

Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Markov Likelihood
Group Relative Policy Optimization (GRPO) has significantly advanced the reasoning ability of large language models (LLMs), particularly by boosting their mathematical performance. However, GRPO and related entropy-regularization methods still face challenges rooted in the sparse token rewards inherent to chain-of-thought (CoT). Current approaches often rely on undifferentiated token-level entropy adjustments, which frequently lead to entropy collapse or model collapse. In this work, we propose…

A Dobrushin condition for quantum Markov chains: Rapid mixing and conditional mutual information at high temperature
Ainesh Bakshi, Allen Liu, Ankur Moitra, Ewin Tang
https://arxiv.org/abs/2510.08542

A Dobrushin condition for quantum Markov chains: Rapid mixing and conditional mutual information at high temperature
A central challenge in quantum physics is to understand the structural properties of many-body systems, both in equilibrium and out of equilibrium. For classical systems, we have a unified perspective which connects structural properties of systems at thermal equilibrium to the Markov chain dynamics that mix to them. We lack such a perspective for quantum systems: there is no framework to translate the quantitative convergence of the Markovian evolution into strong structural consequences. We…

Feynman Formula for Discrete-time Quantum Walks
Jean-Pierre Fouque, Tomoyuki Ichiba, Ka Lok Lam
https://arxiv.org/abs/2510.12038 https://arxiv.org/pdf/2510…

Feynman Formula for Discrete-time Quantum Walks
We explicitly connect (discrete-time) quantum walks on Z with a four-state Markov additive process via a Feynman-type formula (2.5). Using this representation, we derive a relation between the spectral decomposition of the Markov additive process and the limiting density of the homogeneous quantum walk. In addition, we consider a space-time rescaling of quantum walks, which leads to a system of quantum transport PDEs in continuous time and space with a phase interaction term. Our probabilistic …

Inhomogeneous continuous-time Markov chains to infer flexible time-varying evolutionary rates
Pratyusa Datta, Philippe Lemey, Marc A. Suchard
https://arxiv.org/abs/2510.11982 ht…

Inhomogeneous continuous-time Markov chains to infer flexible time-varying evolutionary rates
Reconstructing evolutionary histories and estimating the rate of evolution from molecular sequence data is of central importance in evolutionary biology and infectious disease research. We introduce a flexible Bayesian phylogenetic inference framework that accommodates changing evolutionary rates over time by modeling sequence character substitution processes as inhomogeneous continuous-time Markov chains (ICTMCs) acting along the unknown phylogeny, where the rate remains as an unknown, positiv…

Bayesian Modelling of Multi-Year Crop Type Classification Using Deep Neural Networks and Hidden Markov Models
Gianmarco Perantoni, Giulio Weikmann, Lorenzo Bruzzone
https://arxiv.org/abs/2510.07008

Bayesian Modelling of Multi-Year Crop Type Classification Using Deep Neural Networks and Hidden Markov Models
The temporal consistency of yearly land-cover maps is of great importance to model the evolution and change of the land cover over the years. In this paper, we focus the attention on a novel approach to classification of yearly satellite image time series (SITS) that combines deep learning with Bayesian modelling, using Hidden Markov Models (HMMs) integrated with Transformer Encoder (TE) based DNNs. The proposed approach aims to capture both i) intricate temporal correlations in yearly SITS and…

Markov kernels in Mathlib's probability library
R\'emy Degenne
https://arxiv.org/abs/2510.04070 https://arxiv.org/pdf/2510.04070

Markov kernels in Mathlib's probability library
The probability folder of Mathlib, Lean's mathematical library, makes a heavy use of Markov kernels. We present their definition and properties and describe the formalization of the disintegration theorem for Markov kernels. That theorem is used to define conditional probability distributions of random variables as well as posterior distributions. We then explain how Markov kernels are used in a more unusual way to get a common definition of independence and conditional independence and, follow…

Generative Consistency Models for Estimation of Kinetic Parametric Image Posteriors in Total-Body PET
Yun Zhao, Qinlin Gu, Georgios I. Angelis, Andrew J. Reader, Yanan Fan, Steven R. Meikle
https://arxiv.org/abs/2509.13614

Generative Consistency Models for Estimation of Kinetic Parametric Image Posteriors in Total-Body PET
Dynamic total body positron emission tomography (TB-PET) makes it feasible to measure the kinetics of all organs in the body simultaneously which may lead to important applications in multi-organ disease and systems physiology. Since whole-body kinetics are highly heterogeneous with variable signal-to-noise ratios, parametric images should ideally comprise not only point estimates but also measures of posterior statistical uncertainty. However, standard Bayesian techniques, such as Markov chain…

Restoring detailed balance in non-Hermitian Markov processes
Tim Van Wesemael, Gilberto Nakamura, Jan Baetens, Odemir M. Bruno, Alexandre S. Martinez, Christophe Deroulers
https://arxiv.org/abs/2510.09467

Restoring detailed balance in non-Hermitian Markov processes
Stochastic processes out-of-equilibrium often involve asymmetric contributions that break detailed balance and lead to non-monotonic entropy production, limiting thermodynamic interpretations and inference techniques. Here we use Dyson maps to restore monotonic entropy growth in those processes, allowing the use of standard tools from statistical physics, providing a general and computationally tractable method applicable to a broad class of Markovian systems.

Crosslisted article(s) found for math.NT. https://arxiv.org/list/math.NT/new
[1/1]:
- Markov's equation is not partition regular
Tianyi Tao, Bohan Yang
http…

I bet there's a billion bucks in figuring out how to get ChatGPT to generate Markov chains. Just think, we could disrupt the global Markov ecosystem with AI!

Linear Recurrent Sequences, Markov Chains and Their Applications in Graph Theory
Rebecca Carter, M. Ram Murty
https://arxiv.org/abs/2510.03537 https://arxi…

Linear Recurrent Sequences, Markov Chains and Their Applications in Graph Theory
After a brief review of the key theorems concerning recurrent sequences, we give an explicit computation of the inverse of the Vandermonde matrix. This will then be used to derive sub-exponential decay error terms in the ergodic theorem of Markov chains. Finally, we apply these results to give estimates for the diameters of directed graphs.

Replaced article(s) found for math.NA. https://arxiv.org/list/math.NA/new
[1/2]:
- Almost sure convergence rates of adaptive increasingly rare Markov chain Monte Carlo
Julian Hofstadler, Krzysztof Latuszynski, Gareth O. Roberts, Daniel Rudolf

Replaced article(s) found for astro-ph.CO. https://arxiv.org/list/astro-ph.CO/new
[1/2]:
- Markov Walk Exploration of Model Spaces: Bayesian Selection of Dark Energy Models with Supernovae
Benedikt Schosser, Tobias R\"ospel, Bjoern Malte Schaefer

Der Film endet dann mit dem 'Rückkauf' der Villa, und unserem ersten gemeinsamen Besuch selbiger. Zwei weitere Mitbesitzer (der vorher mal erwähnte etwas entferntere Cousin, sowie noch ein mittelentfernter Cousin) waren da leider nicht dabei.
Danach: Fragerunde mit der Regisseurin Taťšna Markovš und uns vier im Film vorkommenden Ururenkel*innen von Johann Schicht.
Da sitzen wir also vor dem Publikum, und eigentlich weiss mindestens die Hälfte des Publikums besser bescheid als wir 🫣

Scale dependence in hidden Markov models for animal movement
Th\'eo Michelot, Emma Storey
https://arxiv.org/abs/2510.03958 https://arxiv.org/pdf/2510.0…

Scale dependence in hidden Markov models for animal movement
Hidden Markov models (HMMs) have been used increasingly to understand how movement patterns of animals arise from behavioural states. An animal is assumed to transition between behavioural states through time, as described by transition probabilities. Within each state, the movement typically follows a discrete-time random walk, where steps between successive observed locations are described in terms of step lengths (related to speed) and turning angles (related to tortuosity). HMMs are discret…

Crosslisted article(s) found for q-bio.PE. https://arxiv.org/list/q-bio.PE/new
[1/1]:
- Inhomogeneous continuous-time Markov chains to infer flexible time-varying evolutionary rates
Pratyusa Datta, Philippe Lemey, Marc A. Suchard

from my link log —
A new approach to analyzing Robin Hood hashing. (2014)
https://arxiv.org/abs/1401.7616
saved 2025-10-05 https://dotat.at/:/HOGQW.html…

A New Approach to Analyzing Robin Hood Hashing
Robin Hood hashing is a variation on open addressing hashing designed to reduce the maximum search time as well as the variance in the search time for elements in the hash table. While the case of insertions only using Robin Hood hashing is well understood, the behavior with deletions has remained open. Here we show that Robin Hood hashing can be analyzed under the framework of finite-level finite-dimensional jump Markov chains. This framework allows us to re-derive some past results for the in…

Adaptive Science Operations in Deep Space Missions Using Offline Belief State Planning
Grace Ra Kim, Hailey Warner, Duncan Eddy, Evan Astle, Zachary Booth, Edward Balaban, Mykel J. Kochenderfer
https://arxiv.org/abs/2510.08812

Adaptive Science Operations in Deep Space Missions Using Offline Belief State Planning
Deep space missions face extreme communication delays and environmental uncertainty that prevent real-time ground operations. To support autonomous science operations in communication-constrained environments, we present a partially observable Markov decision process (POMDP) framework that adaptively sequences spacecraft science instruments. We integrate a Bayesian network into the POMDP observation space to manage the high-dimensional and uncertain measurements typical of astrobiology missions…

Analysis of approximate linear programming solution to Markov decision problem with log barrier function
Donghwan Lee, Hyukjun Yang, Bum Geun Park
https://arxiv.org/abs/2509.19800

Analysis of approximate linear programming solution to Markov decision problem with log barrier function
There are two primary approaches to solving Markov decision problems (MDPs): dynamic programming based on the Bellman equation and linear programming (LP). Dynamic programming methods are the most widely used and form the foundation of both classical and modern reinforcement learning (RL). By contrast, LP-based methods have been less commonly employed, although they have recently gained attention in contexts such as offline RL. The relative underuse of the LP-based methods stems from the fact t…

On quantum to classical comparison for Davies generators
Joao Basso, Shirshendu Ganguly, Alistair Sinclair, Nikhil Srivastava, Zachary Stier, Thuy-Duong Vuong
https://arxiv.org/abs/2510.07267

On quantum to classical comparison for Davies generators
Despite extensive study, our understanding of quantum Markov chains remains far less complete than that of their classical counterparts. [Temme'13] observed that the Davies Lindbladian, a well-studied model of quantum Markov dynamics, contains an embedded classical Markov generator, raising the natural question of how the convergence properties of the quantum and classical dynamics are related. While [Temme'13] showed that the spectral gap of the Davies Lindbladian can be much smaller than that…

Replaced article(s) found for math.PR. https://arxiv.org/list/math.PR/new
[1/1]:
- Accuracy criterion for mean field approximations of Markov processes on hypergraphs
Illes Horvath, Daniel Keliger

Replaced article(s) found for q-fin.RM. https://arxiv.org/list/q-fin.RM/new
[1/1]:
- A General Framework for Importance Sampling with Markov Random Walks
Cheng-Der Fuh, Yanwei Jia, Steven Kou

Accepting Normalization via Markov Magmoids
Elena Di Lavore, Mario Rom\'an
https://arxiv.org/abs/2510.01131 https://arxiv.org/pdf/2510.01131

Accepting Normalization via Markov Magmoids
Normalization of probability distributions is not a distributive law. We introduce distributive sesquilaws to abstract normalization and, from their axioms, we derive some of its properties. In particular, normalized channels form a unital magmoid with an action of the category of subdistributions; its possibilistic analogue is the action from may-must relations into the category of relations. We argue that the magmoid of normalized channels is the stochastic analog of the category of relations.

Replaced article(s) found for cs.LO. https://arxiv.org/list/cs.LO/new
[1/1]:
- Order in Partial Markov Categories
Elena Di Lavore, Mario Rom\'an, Pawe{\l} Soboci\'nski, M\'ark Sz\'eles

Absorbing Markov Chain-Based Analysis of Age of Information in Discrete-Time Dual-Queue Systems
Yifan Feng, Nail Akar, Zhengchuan Chen, Mehul Motani
https://arxiv.org/abs/2509.23360

Absorbing Markov Chain-Based Analysis of Age of Information in Discrete-Time Dual-Queue Systems
Status update systems require the timely collection of sensing information for which deploying multiple sensors/servers to obtain diversity gains is considered as a promising solution. In this work, we construct an absorbing Markov chain (AMC) to exactly model Age of Information (AoI) in a discretetime dual-queue (DTDQ) status update system with generate at will (GAW) status updates, discrete phase-type (DPH-type) distributed service times and transmission freezing. Specifically, transmission i…

Markov Combinations of Discrete Statistical Models
Orlando Marigliano, Eva Riccomagno
https://arxiv.org/abs/2509.18983 https://arxiv.org/pdf/2509.18983

Mean-Field Games with Constraints
Anran Hu, Zijiu Lyu
https://arxiv.org/abs/2510.11843 https://arxiv.org/pdf/2510.11843

Mean-Field Games with Constraints
This paper introduces a framework of Constrained Mean-Field Games (CMFGs), where each agent solves a constrained Markov decision process (CMDP). This formulation captures scenarios in which agents' strategies are subject to feasibility, safety, or regulatory restrictions, thereby extending the scope of classical mean field game (MFG) models. We first establish the existence of CMFG equilibria under a strict feasibility assumption, and we further show uniqueness under a classical monotonicity co…

Empirical PAC-Bayes bounds for Markov chains
Vahe Karagulyan, Pierre Alquier
https://arxiv.org/abs/2509.20985 https://arxiv.org/pdf/2509.20985

Empirical PAC-Bayes bounds for Markov chains
The core of generalization theory was developed for independent observations. Some PAC and PAC-Bayes bounds are available for data that exhibit a temporal dependence. However, there are constants in these bounds that depend on properties of the data-generating process: mixing coefficients, mixing time, spectral gap... Such constants are unknown in practice. In this paper, we prove a new PAC-Bayes bound for Markov chains. This bound depends on a quantity called the pseudo-spectral gap. The main …

Hidden Markov Model Decoding for LDPC Codes
Jan C Olivier, Etienne Barnard
https://arxiv.org/abs/2509.21872 https://arxiv.org/pdf/2509.21872

Hidden Markov Model Decoding for LDPC Codes
The paper proposes an iterative Hidden Markov Model (HMM) for decoding a Low Density Parity Check (LDPC) code. It is demonstrated that a first-order HMM provides a natural framework for the decoder. The HMM is time-homogeneous with a fixed transition matrix and is based on a random walk through the encoded frame bits. Each hidden state contains a pair of two encoded bits, and parity checks are naturally incorporated into the observation model. The paper shows that by implementing a forward-back…

Self-supervised diffusion model fine-tuning for costate initialization using Markov chain Monte Carlo
Jannik Graebner, Ryne Beeson
https://arxiv.org/abs/2510.02527 https://

Self-supervised diffusion model fine-tuning for costate initialization using Markov chain Monte Carlo
Global search and optimization of long-duration, low-thrust spacecraft trajectories with the indirect method is challenging due to a complex solution space and the difficulty of generating good initial guesses for the costate variables. This is particularly true in multibody environments. Given data that reveals a partial Pareto optimal front, it is desirable to find a flexible manner in which the Pareto front can be completed and fronts for related trajectory problems can be found. In this wor…

A coupling-based approach to f-divergences diagnostics for Markov chain Monte Carlo
Adrien Corenflos, Hai-Dang Dau
https://arxiv.org/abs/2510.07559 https://

A coupling-based approach to f-divergences diagnostics for Markov chain Monte Carlo
A long-standing gap exists between the theoretical analysis of Markov chain Monte Carlo convergence, which is often based on statistical divergences, and the diagnostics used in practice. We introduce the first general convergence diagnostics for Markov chain Monte Carlo based on any f-divergence, allowing users to directly monitor, among others, the Kullback--Leibler and the $χ^2$ divergences as well as the Hellinger and the total variation distances. Our first key contribution is a coupling-…

MCMC: Bridging Rendering, Optimization and Generative AI
Gurprit Singh, Wenzel Jakob
https://arxiv.org/abs/2510.09078 https://arxiv.org/pdf/2510.09078

MCMC: Bridging Rendering, Optimization and Generative AI
Generative artificial intelligence (AI) has made unprecedented advances in vision language models over the past two years. During the generative process, new samples (images) are generated from an unknown high-dimensional distribution. Markov Chain Monte Carlo (MCMC) methods are particularly effective in drawing samples from such complex, high-dimensional distributions. This makes MCMC methods an integral component for models like EBMs, ensuring accurate sample generation. Gradient-based opti…

A Structural Theory of Quantum Metastability: Markov Properties and Area Laws
Thiago Bergamaschi, Chi-Fang Chen, Umesh Vazirani
https://arxiv.org/abs/2510.08538 https://

A Structural Theory of Quantum Metastability: Markov Properties and Area Laws
Statistical mechanics assumes that a quantum many-body system at low temperature can be effectively described by its Gibbs state. However, many complex quantum systems exist only as metastable states of dissipative open system dynamics, which appear stable and robust yet deviate substantially from true thermal equilibrium. In this work, we model metastable states as approximate stationary states of a quasi-local, (KMS)-detailed-balanced master equation representing Markovian system-bath interac…

Replaced article(s) found for math.GT. https://arxiv.org/list/math.GT/new
[1/1]:
- Bonded braids and the Markov theorem
Paolo Cavicchioli, Bo\v{s}tjan Gabrov\v{s}ek, Matic Simoni\v{c}
…

Markov Modeling for Licensed and Unlicensed Band Allocation in Underlay and Overlay D2D
Po-Heng Chou, Yen-Ting Liu, Wei-Chang Chen, Walid Saad
https://arxiv.org/abs/2509.23218 h…

Markov Modeling for Licensed and Unlicensed Band Allocation in Underlay and Overlay D2D
In this paper, a novel analytical model for resource allocation is proposed for a device-to-device (D2D) assisted cellular network. The proposed model can be applied to underlay and overlay D2D systems for sharing licensed bands and offloading cellular traffic. The developed model also takes into account the problem of unlicensed band sharing with Wi-Fi systems. In the proposed model, a global system state reflects the interaction among D2D, conventional cellular, and Wi-Fi packets. Under the s…

Statistical properties of Markov shifts (part I)
Yeor Hafouta
https://arxiv.org/abs/2510.07757 https://arxiv.org/pdf/2510.07757

Statistical properties of Markov shifts (part I)
We prove central limit theorems, Berry-Esseen type theorems, almost sure invariance principles, large deviations and Livsic type regularity for partial sums of the form $S_n=\sum_{j=0}^{n-1}f_j(...,X_{j-1},X_j,X_{j+1},...)$, where $(X_j)$ is an inhomogeneous Markov chain satisfying some mixing assumptions and $f_j$ is a sequence of sufficiently regular functions. Even though the case of non-stationary chains and time dependent functions $f_j$ is more challenging, our results seem to be new alre…

Analysis of approximate linear programming solution to Markov decision problem with log barrier function
Donghwan Lee, Hyukjun Yang, Bum Geun Park
https://arxiv.org/abs/2509.19800

Analysis of approximate linear programming solution to Markov decision problem with log barrier function
There are two primary approaches to solving Markov decision problems (MDPs): dynamic programming based on the Bellman equation and linear programming (LP). Dynamic programming methods are the most widely used and form the foundation of both classical and modern reinforcement learning (RL). By contrast, LP-based methods have been less commonly employed, although they have recently gained attention in contexts such as offline RL. The relative underuse of the LP-based methods stems from the fact t…

MARCOS: Deep Thinking by Markov Chain of Continuous Thoughts
Jiayu Liu, Zhenya Huang, Anya Sims, Enhong Chen, Yee Whye Teh, Ning Miao
https://arxiv.org/abs/2509.25020 https://…

MARCOS: Deep Thinking by Markov Chain of Continuous Thoughts
The current paradigm for reasoning in large language models (LLMs) involves models "thinking out loud" via a sequence of tokens, known as chain-of-thought (CoT). This approach, while effective, has several significant drawbacks. Firstly, inference requires autoregressive generation of often thousands of CoT tokens, which is slow and computationally expensive. Secondly, it constrains reasoning to the discrete space of tokens, creating an information bottleneck across reasoning steps. Thirdly, it…

Unending Sequential Auctions
Amir Ban
https://arxiv.org/abs/2510.08742 https://arxiv.org/pdf/2510.08742

Unending Sequential Auctions
Sequential auctions for identical items with unit-demand, private-value buyers are common and often occur periodically without end, as new bidders replace departing ones. We model bidder uncertainty by introducing a probability that a bidder must exit the auction in each period. Treating the sequential auction as a Markov process, we demonstrate the existence of a unique steady state. In the absence of uncertainty, the steady state resembles a posted-price mechanism: bidders with values above…

Markov Decision Processing Networks
Sanidhay Bhambay, Thirupathaiah Vasantam, Neil Walton
https://arxiv.org/abs/2509.24541 https://arxiv.org/pdf/2509.24541…

Markov Decision Processing Networks
We introduce Markov Decision Processing Networks (MDPNs) as a multiclass queueing network model where service is a controlled, finite-state Markov process. The model exhibits a decision-dependent service process where actions taken influence future service availability. Viewed as a two-sided queueing model, this captures settings such as assemble-to-order systems, ride-hailing platforms, cross-skilled call centers, and quantum switches. We first characterize the capacity region of MDPNs. Unli…

Constraints on inelastic dark matter from the CDEX-1B experiment
Y. F. Liang, L. T. Yang, Q. Yue, K. J. Kang, Y. J. Li, H. P. An, Greeshma C., J. P. Chang, H. Chen, Y. H. Chen, J. P. Cheng, J. Y. Cui, W. H. Dai, Z. Deng, Y. X. Dong, C. H. Fang, H. Gong, Q. J. Guo, T. Guo, X. Y. Guo, L. He, J. R. He, H. X. Huang, T. C. Huang, S. Karmakar, Y. S. Lan, H. B. Li, H. Y. Li, J. M. Li, J. Li, M. C. Li, Q. Y. Li, R. M. J. Li, X. Q. Li, Y. L. Li, B. Liao, F. K. Lin, S. T. Lin, J. X. Liu, R. Z. L…

Constraints on inelastic dark matter from the CDEX-1B experiment
We present limits on spin-independent inelastic WIMP-nucleus scattering using the 737.1 kg $\cdot$ day dataset from the CDEX-1B experiment. Expected nuclear recoil spectra for various inelastic WIMP masses $m_χ$ and mass splittings $δ$ are calculated under the standard halo model. An accurate background model of CDEX-1B is constructed by simulating all major background sources. The model parameters are then determined through maximum likelihood estimation and Markov Chain Monte Carlo fitting.…

On the decoherence of Majorana zero modes mediated by gapless fermions
Sauri Bhattacharyya, Marco Grilli, Bernard van Heck
https://arxiv.org/abs/2510.07893 https://

On the decoherence of Majorana zero modes mediated by gapless fermions
We study the decoherence of a collection of Majorana zero modes weakly coupled to a gapless reservoir of non-interacting fermions. Using the Born-Markov approximation, we derive a Lindblad master equation for the dissipative dynamics of the Majorana zero modes. Due to the long-range coupling between Majorana zero modes mediated by the gapless reservoir, the Lindblad jump operators are non-local linear combinations of the Majorana operators. We show that, as a consequence, the dissipative dynami…

Oracle-based Uniform Sampling from Convex Bodies
Thanh Dang, Jiaming Liang
https://arxiv.org/abs/2510.02983 https://arxiv.org/pdf/2510.02983

Oracle-based Uniform Sampling from Convex Bodies
We propose new Markov chain Monte Carlo algorithms to sample a uniform distribution on a convex body $K$. Our algorithms are based on the Alternating Sampling Framework/proximal sampler, which uses Gibbs sampling on an augmented distribution and assumes access to the so-called restricted Gaussian oracle (RGO). The key contribution of this work is the efficient implementation of RGO for uniform sampling on $K$ via rejection sampling and access to either a projection oracle or a separation oracle…

Learning in Stackelberg Markov Games
Jun He, Andrew L. Liu, Yihsu Chen
https://arxiv.org/abs/2509.16296 https://arxiv.org/pdf/2509.16296

Learning in Stackelberg Markov Games
Designing socially optimal policies in multi-agent environments is a fundamental challenge in both economics and artificial intelligence. This paper studies a general framework for learning Stackelberg equilibria in dynamic and uncertain environments, where a single leader interacts with a population of adaptive followers. Motivated by pressing real-world challenges such as equitable electricity tariff design for consumers with distributed energy resources (such as rooftop solar and energy stor…

Learning the Influence Graph of a Markov Process that Randomly Resets to Past
Sudharsan Senthil, Avhishek Chatterjee
https://arxiv.org/abs/2509.16129 https://

Learning the Influence Graph of a Markov Process that Randomly Resets to Past
Learning the influence graph G of a high-dimensional Markov process is a challenging problem. Prior work has addressed this task when the process has finite memory. However, the more general regime in which the system probabilistically "jumps back in time" - so that the state at t+1 depends on a sample from a distant past t-d - remains unexplored. The process with probabilistic resets can be modeled as a Markov process with memory, but estimations become computationally expensive. To tackle thi…

Constraints on the interacting holographic dark energy models: implications from background and perturbations data
N. Nazari Pooya
https://arxiv.org/abs/2510.08875 https://

Constraints on the interacting holographic dark energy models: implications from background and perturbations data
In this study, we employ a two-step method to analyze models of holographic dark energy (HDE) and interacting holographic dark energy (IHDE), incorporating three distinct dark energy (DE)-dark matter (DM) interaction terms. First, using the latest background dataset, we conduct a Markov chain Monte Carlo (MCMC) analysis to constrain the free parameters of the models. Then, we assess the models against each other using the key background parameters and compare them to the $Λ$CDM standard model.…

SAGE:State-Aware Guided End-to-End Policy for Multi-Stage Sequential Tasks via Hidden Markov Decision Process
BinXu Wu, TengFei Zhang, Chen Yang, JiaHao Wen, HaoCheng Li, JingTian Ma, Zhen Chen, JingYuan Wang
https://arxiv.org/abs/2509.19853

SAGE:State-Aware Guided End-to-End Policy for Multi-Stage Sequential Tasks via Hidden Markov Decision Process
Multi-stage sequential (MSS) robotic manipulation tasks are prevalent and crucial in robotics. They often involve state ambiguity, where visually similar observations correspond to different actions. We present SAGE, a state-aware guided imitation learning framework that models tasks as a Hidden Markov Decision Process (HMDP) to explicitly capture latent task stages and resolve ambiguity. We instantiate the HMDP with a state transition network that infers hidden states, and a state-aware action…

An Orthogonal Learner for Individualized Outcomes in Markov Decision Processes
Emil Javurek, Valentyn Melnychuk, Jonas Schweisthal, Konstantin Hess, Dennis Frauen, Stefan Feuerriegel
https://arxiv.org/abs/2509.26429

An Orthogonal Learner for Individualized Outcomes in Markov Decision Processes
Predicting individualized potential outcomes in sequential decision-making is central for optimizing therapeutic decisions in personalized medicine (e.g., which dosing sequence to give to a cancer patient). However, predicting potential outcomes over long horizons is notoriously difficult. Existing methods that break the curse of the horizon typically lack strong theoretical guarantees such as orthogonality and quasi-oracle efficiency. In this paper, we revisit the problem of predicting individ…

A Transience Criterion for Uniformly Bounded Markov Chains with Asymptotically Zero Mean Drift
Dan Andrei Tudor
https://arxiv.org/abs/2510.03937 https://ar…

A Transience Criterion for Uniformly Bounded Markov Chains with Asymptotically Zero Mean Drift
In this paper, we give an overview of mean drift conditions for the state-space classification of discrete-time Markov Chains and we present a new transience criterion for uniformly bounded Markov Chains with asymptotically zero drift. The criterion does not need a condition on the second-moment drifts and can be applied to certain chains for which other criteria fail.

Efficient Prior Sensitivity and Tipping-point Analysis for Medical Research: Revisiting Sampling Importance Resampling
Tomohiro Ohigashi, Shonosuke Sugasawa
https://arxiv.org/abs/2510.10034

Efficient Prior Sensitivity and Tipping-point Analysis for Medical Research: Revisiting Sampling Importance Resampling
Bayesian methods have received increasing attention in medical research, where sensitivity analysis of prior distributions is essential. Such analyses typically require the evaluation of the posterior distribution of a parameter under multiple alternative prior settings. When the posterior distribution of the parameter of interest cannot be derived analytically, the standard approach is to re-fit the Markov chain Monte Carlo (MCMC) algorithm for each setting, which incurs substantial computatio…

Real-Time Markov Modeling for Single-Photon LiDAR: $1000 \times$ Acceleration and Convergence Analysis
Weijian Zhang, Hashan K. Weerasooriya, Prateek Chennuri, Stanley H. Chan
https://arxiv.org/abs/2509.20500

Real-Time Markov Modeling for Single-Photon LiDAR: $1000 \times$ Acceleration and Convergence Analysis
Asynchronous single-photon LiDAR (SP-LiDAR) is an important imaging modality for high-quality 3D applications and navigation, but the modeling of the timestamp distributions of a SP-LiDAR in the presence of dead time remains a very challenging open problem. Prior works have shown that timestamps form a discrete-time Markov chain, whose stationary distribution can be computed as the leading left eigenvector of a large transition matrix. However, constructing this matrix is known to be computatio…

Pseudo-MDPs: A Novel Framework for Efficiently Optimizing Last Revealer Seed Manipulations in Blockchains
Maxime Reynouard
https://arxiv.org/abs/2510.07080 https://

Pseudo-MDPs: A Novel Framework for Efficiently Optimizing Last Revealer Seed Manipulations in Blockchains
This study tackles the computational challenges of solving Markov Decision Processes (MDPs) for a restricted class of problems. It is motivated by the Last Revealer Attack (LRA), which undermines fairness in some Proof-of-Stake (PoS) blockchains such as Ethereum (\$400B market capitalization). We introduce pseudo-MDPs (pMDPs) a framework that naturally models such problems and propose two distinct problem reductions to standard MDPs. One problem reduction provides a novel, counter-intuitive per…

ClauseLens: Clause-Grounded, CVaR-Constrained Reinforcement Learning for Trustworthy Reinsurance Pricing
Stella C. Dong, James R. Finlay
https://arxiv.org/abs/2510.08429 https:/…

ClauseLens: Clause-Grounded, CVaR-Constrained Reinforcement Learning for Trustworthy Reinsurance Pricing
Reinsurance treaty pricing must satisfy stringent regulatory standards, yet current quoting practices remain opaque and difficult to audit. We introduce ClauseLens, a clause-grounded reinforcement learning framework that produces transparent, regulation-compliant, and risk-aware treaty quotes. ClauseLens models the quoting task as a Risk-Aware Constrained Markov Decision Process (RA-CMDP). Statutory and policy clauses are retrieved from legal and underwriting corpora, embedded into the agent'…

Detecting disease progression from animal movement using hidden Markov models
Dongmin Kim, Th\'eo Michelot, Katherine Mertes, Jared A. Stabach, John Fieberg
https://arxiv.org/abs/2509.21132

Detecting disease progression from animal movement using hidden Markov models
Understanding disease dynamics is crucial for managing wildlife populations and assessing spillover risk to domestic animals and humans, but infection data on free-ranging animals are difficult to obtain. Because pathogen and parasite infections can alter host movement, infection status may be inferred from animal trajectories. We present a hidden Markov model (HMM) framework that links observed movement behaviors to unobserved infection states, consistent with epidemiological compartmental mod…

Ergodic Properties of Quantum Markov Semigroups
Nicolas Mousset, Nina H. Amini
https://arxiv.org/abs/2509.20133 https://arxiv.org/pdf/2509.20133

Ergodic Properties of Quantum Markov Semigroups
In this paper, we study the ergodic theorem for infinite-dimensional quantum Markov semigroups, originally introduced by Frigerio and Verri in 1982, and its latest version developed by Carbone and Girotti in 2021. We provide a sufficient condition that ensures exponential convergence towards the positive recurrent subspace, a well-known result for irreducible quantum Markov semigroups in finite-dimensional Hilbert spaces. Several illustrative examples are presented to demonstrate the applicatio…

Tensor Network Markov Chain Monte Carlo: Efficient Sampling of Three-Dimensional Spin Glasses and Beyond
Tao Chen, Jing Liu, Youjin Deng, Pan Zhang
https://arxiv.org/abs/2509.23945

Tensor Network Markov Chain Monte Carlo: Efficient Sampling of Three-Dimensional Spin Glasses and Beyond
Sampling the three-dimensional (3D) spin glass -- i.e., generating equilibrium configurations of a 3D lattice with quenched random couplings -- is widely regarded as one of the central and long-standing open problems in statistical physics. The rugged energy landscape, pronounced critical slowing down, and intrinsic ergodicity breaking render standard Monte Carlo methods severely inefficient, particularly for large systems at low temperatures. In this work, we introduce the Tensor Network Marko…

A Markov Chain Arising from the Hopf Square Map on a Non-cocommutative Quantum Group
Donovan Snyder
https://arxiv.org/abs/2510.05298 https://arxiv.org/pdf/…

A Markov Chain Arising from the Hopf Square Map on a Non-cocommutative Quantum Group
Expanding upon the rich history of algebraic techniques in probability, we show the existence of and construct a Markov chain using the Hopf square map on a quantum group that is both non-commutative and non-cocommutative. This extends the work of Diaconis, Pang, and Ram to other Hopf algebras. The new, one-dimensional chain requires different analytical approaches. In this case we use standard martingale theory to prove the existence of a phase transition and prove bounds on the expected growt…

Verifying Memoryless Sequential Decision-making of Large Language Models
Dennis Gross, Helge Spieker, Arnaud Gotlieb
https://arxiv.org/abs/2510.06756 https://

Verifying Memoryless Sequential Decision-making of Large Language Models
We introduce a tool for rigorous and automated verification of large language model (LLM)- based policies in memoryless sequential decision-making tasks. Given a Markov decision process (MDP) representing the sequential decision-making task, an LLM policy, and a safety requirement expressed as a PCTL formula, our approach incrementally constructs only the reachable portion of the MDP guided by the LLM's chosen actions. Each state is encoded as a natural language prompt, the LLM's response is pa…

Crosslisted article(s) found for math.ST. https://arxiv.org/list/math.ST/new
[1/1]:
- A coupling-based approach to f-divergences diagnostics for Markov chain Monte Carlo
Adrien Corenflos, Hai-Dang Dau

HOMC: A MATLAB Package for Higher Order Markov Chains
Jianhong Xu
https://arxiv.org/abs/2510.02664 https://arxiv.org/pdf/2510.02664

HOMC: A MATLAB Package for Higher Order Markov Chains
We present a MATLAB package, which is the first of its kind, for Higher Order Markov Chains (HOMC). It can be used to easily compute all important quantities in our recent works relevant to higher order Markov chains, such as the $k$-step transition tensor, limiting probability distribution, ever-reaching probability tensor, and mean first passage time tensor. It can also be used to check whether a higher order chain is ergodic or regular, to construct the transition matrix of the associated re…

Q-Learning with Fine-Grained Gap-Dependent Regret
Haochen Zhang, Zhong Zheng, Lingzhou Xue
https://arxiv.org/abs/2510.06647 https://arxiv.org/pdf/2510.0664…

Q-Learning with Fine-Grained Gap-Dependent Regret
We study fine-grained gap-dependent regret bounds for model-free reinforcement learning in episodic tabular Markov Decision Processes. Existing model-free algorithms achieve minimax worst-case regret, but their gap-dependent bounds remain coarse and fail to fully capture the structure of suboptimality gaps. We address this limitation by establishing fine-grained gap-dependent regret bounds for both UCB-based and non-UCB-based algorithms. In the UCB-based setting, we develop a novel analytical f…

Reinforcement Learning from Probabilistic Forecasts for Safe Decision-Making via Conditional Value-at-Risk Planning
Michal Koren, Or Peretz, Tai Dinh, Philip S. Yu
https://arxiv.org/abs/2510.08226

Reinforcement Learning from Probabilistic Forecasts for Safe Decision-Making via Conditional Value-at-Risk Planning
Sequential decisions in volatile, high-stakes settings require more than maximizing expected return; they require principled uncertainty management. This paper presents the Uncertainty-Aware Markov Decision Process (UAMDP), a unified framework that couples Bayesian forecasting, posterior-sampling reinforcement learning, and planning under a conditional value-at-risk (CVaR) constraint. In a closed loop, the agent updates its beliefs over latent dynamics, samples plausible futures via Thompson sa…

Stability Analysis of An Integrated Multistage Stochastic Programming and Markov Decision Process Problem
Zhiyao Yang, Zhiping Chen, Huifu Xu
https://arxiv.org/abs/2509.22194 ht…

Stability Analysis of An Integrated Multistage Stochastic Programming and Markov Decision Process Problem
In this paper, we consider an integrated MSP-MDP framework which captures features of Markov decision process (MDP) and multistage stochastic programming (MSP). The integrated framework allows one to study a dynamic decision-making process that involves both transition of system states and dynamic change of the stochastic environment affected respectively by potential endogenous uncertainties and exogenous uncertainties. The integrated model differs from classical MDP models by taking into acco…

Virtual Quantum Markov Chain of four-qubit systems
Zhixing Chen, Lin Chen
https://arxiv.org/abs/2509.18803 https://arxiv.org/pdf/2509.18803

Virtual Quantum Markov Chain of four-qubit systems
We extend the framework of virtual quantum Markov chains (VQMCs) from tripartite systems to the four-qubit setting. Structural criteria such as the kernel-inclusion condition are analyzed, showing that they are necessary but not sufficient for the existence of a valid recovery map. By explicit examples, we demonstrate that the four-qubit W state admits a recovery channel and thus qualifies as a VQMC, while the GHZ state does not. We further provide semidefinite programming (SDP) formulations to…

Bayesian inference for spatio-temporal hidden Markov models using the exchange algorithm
Daniele Tancini, Riccardo Rastelli, Francesco Bartolucci
https://arxiv.org/abs/2509.15164

Bayesian inference for spatio-temporal hidden Markov models using the exchange algorithm
Spatio-temporal hidden Markov models are extremely difficult to estimate because their latent joint distributions are available only in trivial cases. In the estimation phase, these latent distributions are usually substituted with pseudo-distributions, which could affect the estimation results, in particular in the presence of strong dependence between the latent variables. In this work, we propose a spatio-temporal hidden Markov model where the latent process is an extension of the autologist…

A pedestrian's approach to large deviations in semi-Markov processes with an application to entropy production
Alexander M. Maier, Jonas H. Fritz, Udo Seifert
https://arxiv.org/abs/2509.15077

A pedestrian's approach to large deviations in semi-Markov processes with an application to entropy production
Semi-Markov processes play an important role in the effective description of partially accessible systems in stochastic thermodynamics. They occur, for instance, in coarse-graining procedures such as state lumping and when analyzing waiting times between few visible Markovian events. The finite-time measurement of any coarse-grained observable in a stochastic system depends on the specific realization of the underlying trajectory. Moreover, the fluctuations of such observables are encoded in th…

Stein's method, Markov processes, and linear eigenvalue statistics of random matrices
David Grzybowski, Mark Meckes
https://arxiv.org/abs/2509.25451 https://

Stein's method, Markov processes, and linear eigenvalue statistics of random matrices
We show how the infinitesimal exchangeable pairs approach to Stein's method combines naturally with the theory of Markov semigroups. We present a multivariate normal approximation theorem for functions of a random variable invariant with respect to a Markov semigroup. This theorem provides a Wasserstein distance bound in terms of quantities related to the infinitesimal generator of the semigroup. As an application, we deduce a rate of convergence for Johansson's celebrated theorem on linear eig…

Replaced article(s) found for cs.AI. https://arxiv.org/list/cs.AI/new
[1/5]:
- Improved Monte Carlo Planning via Causal Disentanglement for Structurally-Decomposed Markov Decis...
Larkin Liu, Shiqi Liu, Yinruo Hua, Matej Jusup

Replaced article(s) found for stat.CO. https://arxiv.org/list/stat.CO/new
[1/1]:
- A General Framework for Importance Sampling with Markov Random Walks
Cheng-Der Fuh, Yanwei Jia, Steven Kou

Wavefunction Flows: Efficient Quantum Simulation of Continuous Flow Models
David Layden, Ryan Sweke, Vojt\v{e}ch Havl\'i\v{c}ek, Anirban Chowdhury, Kirill Neklyudov
https://arxiv.org/abs/2510.08462

Wavefunction Flows: Efficient Quantum Simulation of Continuous Flow Models
Flow models are a cornerstone of modern machine learning. They are generative models that progressively transform probability distributions according to learned dynamics. Specifically, they learn a continuous-time Markov process that efficiently maps samples from a simple source distribution into samples from a complex target distribution. We show that these models are naturally related to the Schrödinger equation, for an unusual Hamiltonian on continuous variables. Moreover, we prove that the…

Uniform ergodicity of geodesic slice sampling
Mareike Hasenpflug
https://arxiv.org/abs/2510.06748 https://arxiv.org/pdf/2510.06748

Uniform ergodicity of geodesic slice sampling
Geodesic slice sampling, introduced in Durmus et al., 2024, is a slice sampling based Markov chain Monte Carlo method for approximate sampling from distributions on Riemannian manifolds. We prove that it is uniformly ergodic for distributions with compact support that have a bounded density with respect to the Riemannian measure. The constants in our convergence bound are available explicitly, and we investigate their dependence on the hyperparameters of the geodesic slice sampler, the target d…

Causal Abstractions, Categorically Unified
Markus Englberger, Devendra Singh Dhami
https://arxiv.org/abs/2510.05033 https://arxiv.org/pdf/2510.05033…

Causal Abstractions, Categorically Unified
We present a categorical framework for relating causal models that represent the same system at different levels of abstraction. We define a causal abstraction as natural transformations between appropriate Markov functors, which concisely consolidate desirable properties a causal abstraction should exhibit. Our approach unifies and generalizes previously considered causal abstractions, and we obtain categorical proofs and generalizations of existing results on causal abstractions. Using string…

Edit-Based Flow Matching for Temporal Point Processes
David L\"udke, Marten Lienen, Marcel Kollovieh, Stephan G\"unnemann
https://arxiv.org/abs/2510.06050 https://

Edit-Based Flow Matching for Temporal Point Processes
Temporal point processes (TPPs) are a fundamental tool for modeling event sequences in continuous time, but most existing approaches rely on autoregressive parameterizations that are limited by their sequential sampling. Recent non-autoregressive, diffusion-style models mitigate these issues by jointly interpolating between noise and data through event insertions and deletions in a discrete Markov chain. In this work, we generalize this perspective and introduce an Edit Flow process for TPPs th…

Extending finite mixture models with skew-normal distributions and hidden Markov models for time series
Andrea Nigri, Marco Forti, Han Lin Shang
https://arxiv.org/abs/2509.19920

Extending finite mixture models with skew-normal distributions and hidden Markov models for time series
We introduce an extension of finite mixture models by incorporating skew-normal distributions within a Hidden Markov Model framework. By assuming a constant transition probability matrix and allowing emission distributions to vary according to hidden states, the proposed model effectively captures dynamic dependencies between variables. Through the estimation of state-specific parameters, including location, scale, and skewness, the proposed model enables the detection of structural changes, su…

Quantum Replica Exchange
Zherui Chen, Joao Basso, Zhiyan Ding, Lin Lin
https://arxiv.org/abs/2510.07291 https://arxiv.org/pdf/2510.07291

Quantum Replica Exchange
The presence of energy barriers in the state space of a physical system can lead to exponentially slow convergence for sampling algorithms like Markov chain Monte Carlo (MCMC). In the classical setting, replica exchange (or parallel tempering) is a powerful heuristic to accelerate mixing in these scenarios. In the quantum realm, preparing Gibbs states of Hamiltonians faces a similar challenge, where bottlenecks can dramatically increase the mixing time of quantum dynamical semigroups. In this w…

Inference on Gaussian mixture models with dependent labels
Seunghyun Lee, Rajarshi Mukherjee, Sumit Mukherjee
https://arxiv.org/abs/2510.06501 https://arxi…

Inference on Gaussian mixture models with dependent labels
Gaussian mixture models are widely used to model data generated from multiple latent sources. Despite its popularity, most theoretical research assumes that the labels are either independent and identically distributed, or follows a Markov chain. It remains unclear how the fundamental limits of estimation change under more complex dependence. In this paper, we address this question for the spherical two-component Gaussian mixture model. We first show that for labels with an arbitrary dependence…

Q-Learning with Shift-Aware Upper Confidence Bound in Non-Stationary Reinforcement Learning
Ha Manh Bui, Felix Parker, Kimia Ghobadi, Anqi Liu
https://arxiv.org/abs/2510.03181 h…

Q-Learning with Shift-Aware Upper Confidence Bound in Non-Stationary Reinforcement Learning
We study the Non-Stationary Reinforcement Learning (RL) under distribution shifts in both finite-horizon episodic and infinite-horizon discounted Markov Decision Processes (MDPs). In the finite-horizon case, the transition functions may suddenly change at a particular episode. In the infinite-horizon setting, such changes can occur at an arbitrary time step during the agent's interaction with the environment. While the Q-learning Upper Confidence Bound algorithm (QUCB) can discover a proper pol…

Quantizer Design for Finite Model Approximations, Model Learning, and Quantized Q-Learning for MDPs with Unbounded Spaces
Osman Bicer, Ali D. Kara, Serdar Yuksel
https://arxiv.org/abs/2510.04355

Quantizer Design for Finite Model Approximations, Model Learning, and Quantized Q-Learning for MDPs with Unbounded Spaces
In this paper, for Markov decision processes (MDPs) with unbounded state spaces we present refined upper bounds presented in [Kara et. al. JMLR'23] on finite model approximation errors via optimizing the quantizers used for finite model approximations. We also consider implications on quantizer design for quantized Q-learning and empirical model learning, and the performance of policies obtained via Q-learning where the quantized state is treated as the state itself. We highlight the distinctio…

Efficient tensor-network simulations of weakly-measured quantum circuits
Darren Pereira, Leonardo Banchi
https://arxiv.org/abs/2510.07211 https://arxiv.org…

Efficient tensor-network simulations of weakly-measured quantum circuits
We present a tensor-network-based method for simulating a weakly-measured quantum circuit. In particular, we use a Markov chain to efficiently sample measurements and contract the tensor network, propagating their effect forward along the spatial direction. Applications of our algorithm include validating quantum computers (capable of mid-circuit measurements) in regimes of easy classical simulability, and studying generative-machine-learning applications, where sampling from complex stochastic…

Environmental Risk Assessment via Nonhomogeneous Hidden Semi-Markov Models with Penalized Vector Auto-Regression
Marco Mingione, Pierfrancesco Alaimo Di Loro, Francesco Lagona, Antonello Maruotti
https://arxiv.org/abs/2509.14387

Environmental Risk Assessment via Nonhomogeneous Hidden Semi-Markov Models with Penalized Vector Auto-Regression
Motivated by the study of pollution trends in the city of Bergen, we introduce a flexible statistical framework for modeling multivariate air pollution data via a nonhomogeneous Hidden Semi-Markov Vector Auto-Regression. The hidden process captures unobserved environmental conditions, while the vector autoregressive structure accounts for temporal autocorrelation and cross-pollutant dependencies. The model further allows time-varying environmental conditions to influence both the average levels…

Optimizing Fairness in Production Planning: A Human-Centric Approach to Machine and Workforce Allocation
Alexander Nasuta, Alessandro Cisi, Sylwia Olbrych, Gustavo Vieira, Rui Fernandes, Lucas Paletta, Marlene Mayr, Rishyank Chevuri, Robert Woitsch, Hans Aoyang Zhou, Anas Abdelrazeq, Robert H. Schmitt
https://arxiv.org/abs/2510.01094

Optimizing Fairness in Production Planning: A Human-Centric Approach to Machine and Workforce Allocation
This work presents a two-layer, human-centric production planning framework designed to optimize both operational efficiency and workforce fairness in industrial manufacturing. The first layer formulates the Order-Line allocation as a Constraint Programming (CP) problem, generating high-utilization production schedules that respect machine capacities, processing times, and due dates. The second layer models Worker-Line allocation as a Markov Decision Process (MDP), integrating human factors suc…

Learning Robust Penetration-Testing Policies under Partial Observability: A systematic evaluation
Raphael Simon, Pieter Libin, Wim Mees
https://arxiv.org/abs/2509.20008 https://…

Learning Robust Penetration-Testing Policies under Partial Observability: A systematic evaluation
Penetration testing, the simulation of cyberattacks to identify security vulnerabilities, presents a sequential decision-making problem well-suited for reinforcement learning (RL) automation. Like many applications of RL to real-world problems, partial observability presents a major challenge, as it invalidates the Markov property present in Markov Decision Processes (MDPs). Partially Observable MDPs require history aggregation or belief state estimation to learn successful policies. We investi…

Density convergence on Markov diffusion chaos via Stein's method
Thanh Dang, Yaozhong Hu
https://arxiv.org/abs/2509.18045 https://arxiv.org/pdf/2509.18…

Density convergence on Markov diffusion chaos via Stein's method
We study the difference between the probability density of a random variable $F$ on Markov diffusion chaos and the probability density of a general target distribution $Z$. In the special case where $F$ is a chaotic random variables and $Z$ is a Pearson target, we extend our study to the $k$-th derivatives of the densities for all $k\in \mathbb{N}$. In particular, we obtain four moment theorems for the convergence of the $k$-th derivatives of the densities of $F$ to the corresponding $k$-th der…

A Unified Deep Reinforcement Learning Approach for Close Enough Traveling Salesman Problem
Mingfeng Fan, Jiaqi Cheng, Yaoxin Wu, Yifeng Zhang, Yibin Yang, Guohua Wu, Guillaume Sartoretti
https://arxiv.org/abs/2510.03065

A Unified Deep Reinforcement Learning Approach for Close Enough Traveling Salesman Problem
In recent years, deep reinforcement learning (DRL) has gained traction for solving the NP-hard traveling salesman problem (TSP). However, limited attention has been given to the close-enough TSP (CETSP), primarily due to the challenge introduced by its neighborhood-based visitation criterion, wherein a node is considered visited if the agent enters a compact neighborhood around it. In this work, we formulate a Markov decision process (MDP) for CETSP using a discretization scheme and propose a n…

Optimizing Fairness in Production Planning: A Human-Centric Approach to Machine and Workforce Allocation
Alexander Nasuta, Alessandro Cisi, Sylwia Olbrych, Gustavo Vieira, Rui Fernandes, Lucas Paletta, Marlene Mayr, Rishyank Chevuri, Robert Woitsch, Hans Aoyang Zhou, Anas Abdelrazeq, Robert H. Schmitt
https://arxiv.org/abs/2510.01094

Optimizing Fairness in Production Planning: A Human-Centric Approach to Machine and Workforce Allocation
This work presents a two-layer, human-centric production planning framework designed to optimize both operational efficiency and workforce fairness in industrial manufacturing. The first layer formulates the Order-Line allocation as a Constraint Programming (CP) problem, generating high-utilization production schedules that respect machine capacities, processing times, and due dates. The second layer models Worker-Line allocation as a Markov Decision Process (MDP), integrating human factors suc…

Ergodicity and regularity properties of ODEs with semi-Markov switching
Tobias Hurth, Edouard Strickler
https://arxiv.org/abs/2509.25936 https://arxiv.org/…

Ergodicity and regularity properties of ODEs with semi-Markov switching
This paper is devoted to the study of a stochastic process obtained by random switching between a finite collection of vector fields. Such processes have recently been the focus of much attention in the case where the switching times are exponentially distributed, i.e., Markovian switching. In this contribution, we admit any distribution on $\mathbb{R}_+$ as a law for the switching times. We show that whenever this law is not singular with respect to the Lebesgue measure, the stochastic process…

Bayesian inference for velocity-jump models for movement
Paul G. Blackwell
https://arxiv.org/abs/2509.21226 https://arxiv.org/pdf/2509.21226

Bayesian inference for velocity-jump models for movement
The velocity-jump model is a specific type of piecewise deterministic Markov process in which an individual's velocity is constant except at times that form the events of some point process. It represents an interpretable continuous-time version of the discrete-time `step and turn' models widely used in analysing wildlife telemetry. In this paper, I derive a reversible jump Markov chain Monte Carlo algorithm to carry out exact Bayesian inference for velocity-jump models by reconstructing the tr…

Mixtures, Markov bridges and the matrix product ansatz
Davide Gabrielli, Federica Iacovissi
https://arxiv.org/abs/2509.16455 https://arxiv.org/pdf/2509.164…

Mixtures, Markov bridges and the matrix product ansatz
We give a probabilistic characterization of the set of measures that can be represented by the matrix product ansatz. By suitably enlarging the state space, we show that a probability measure can be described in terms of non negative matrices by the {\it Matrix Product Ansatz}, if and only if it can be written as a mixture of inhomogeneous product measures where the mixing law is a Markov bridge. We give a constructive procedure to identify such probabilistic features. We illustrate the result …

Reinforcement Learning with Action-Triggered Observations
Alexander Ryabchenko, Wenlong Mou
https://arxiv.org/abs/2510.02149 https://arxiv.org/pdf/2510.021…

Reinforcement Learning with Action-Triggered Observations
We study reinforcement learning problems where state observations are stochastically triggered by actions, a constraint common in many real-world applications. This framework is formulated as Action-Triggered Sporadically Traceable Markov Decision Processes (ATST-MDPs), where each action has a specified probability of triggering a state observation. We derive tailored Bellman optimality equations for this framework and introduce the action-sequence learning paradigm in which agents commit to ex…

Mixing properties of some Markov chains models in random environments
Attila Lovas, Lionel Truquet
https://arxiv.org/abs/2509.16118 https://arxiv.org/pdf/2…

Mixing properties of some Markov chains models in random environments
Markov chains in random environments (MCREs) have recently attracted renewed interest, as these processes naturally arise in many applications, such as econometrics and machine learning. Although specific asymptotic results, such as the law of the large numbers and central limit theorems, have been obtained for some of these models, their annealed dependence properties, such as strong mixing properties, are not well understood in general. We derive strong mixing properties for a wide range of M…

Multi-Actor Multi-Critic Deep Deterministic Reinforcement Learning with a Novel Q-Ensemble Method
Andy Wu, Chun-Cheng Lin, Rung-Tzuo Liaw, Yuehua Huang, Chihjung Kuo, Chia Tong Weng
https://arxiv.org/abs/2510.01083

Multi-Actor Multi-Critic Deep Deterministic Reinforcement Learning with a Novel Q-Ensemble Method
Reinforcement learning has gathered much attention in recent years due to its rapid development and rich applications, especially on control systems and robotics. When tackling real-world applications with reinforcement learning method, the corresponded Markov decision process may have huge discrete or even continuous state/action space. Deep reinforcement learning has been studied for handling these issues through deep learning for years, and one promising branch is the actor-critic architectu…

Limitations of strong coupling in non-Markovian quantum thermometry
Qing-Shou Tan, Yang Liu, Xulin Liu, Hao Chen, Xing Xiao, Wei Wu
https://arxiv.org/abs/2510.01596 https://

Limitations of strong coupling in non-Markovian quantum thermometry
We investigate quantum thermometry using a single-qubit probe embedded in a non-Markovian environment, employing the numerically exact hierarchical equations of motion (HEOM) to overcome the limitations of Born-Markov approximations. Through a systematic analysis of the dynamical and steady-state behavior of the quantum signal-to-noise ratio (QSNR) for temperature estimation, we identify several key findings that challenge the conventional expectation that strong coupling necessarily enhances t…

GLASS Flows: Transition Sampling for Alignment of Flow and Diffusion Models
Peter Holderrieth, Uriel Singer, Tommi Jaakkola, Ricky T. Q. Chen, Yaron Lipman, Brian Karrer
https://arxiv.org/abs/2509.25170

GLASS Flows: Transition Sampling for Alignment of Flow and Diffusion Models
The performance of flow matching and diffusion models can be greatly improved at inference time using reward alignment algorithms, yet efficiency remains a major limitation. While several algorithms were proposed, we demonstrate that a common bottleneck is the sampling method these algorithms rely on: many algorithms require to sample Markov transitions via SDE sampling, which is significantly less efficient and often less performant than ODE sampling. To remove this bottleneck, we introduce GL…