Article
Operations Research & Management Science
Hector Jasso-Fuentes, Raquiel R. Lopez-Martinez, J. Adolfo Minjarez-Sosa
Summary: This paper addresses a class of discrete-time Markov decision processes with cost constraints, and proves the existence of optimal control policies and characterizes them based on certain optimality criteria by solving a new problem on a space of occupation measures and a convex program.
Article
Operations Research & Management Science
Fang Chen, Xianping Guo, Zhong-Wei Liao
Summary: In this paper, the authors consider the optimal stopping problems on semi-Markov processes and establish the existence and algorithm of optimal stopping times by utilizing the equivalence between optimal stopping problems on semi-Markov processes and a special class of semi-Markov decision processes. They also provide an explicit construction of semi-Markov decision processes and show that the optimal and e-optimal stopping time can be characterized by hitting time of special sets.
JOURNAL OF OPTIMIZATION THEORY AND APPLICATIONS
(2022)
Article
Economics
Minsoo Jeong
Summary: This paper presents a novel approach to model financial time series that captures both persistency and long term stationarity. The provided statistical theory and empirical evidence support the existence and characteristic behavior of such series in real financial data.
ECONOMIC MODELLING
(2022)
Article
Automation & Control Systems
Xian Chen, Qingda Wei
Summary: In this paper, we investigate the risk-sensitive average optimality in discrete-time Markov decision processes with denumerable states and unbounded costs. By utilizing an approximation method, we derive the multiplicative Poisson equation under suitable ergodicity conditions. Furthermore, we establish the existence of a unique solution to the risk-sensitive average cost optimality equation and provide an equivalent characterization of the set of all optimal stationary policies. Finally, we introduce the policy iteration algorithm and demonstrate its convergence.
SIAM JOURNAL ON CONTROL AND OPTIMIZATION
(2023)
Article
Operations Research & Management Science
Qingda Wei, Xian Chen
Summary: In this study, nonzero-sum games for continuous-time jump processes with unbounded transition rates under expected average payoff criterion are considered. An approximating sequence of stochastic game models with extended state space is introduced to achieve uniform exponential ergodicity. Additionally, the existence of a stationary almost Markov Nash equilibrium is proven by introducing auxiliary static game models.
OPERATIONS RESEARCH LETTERS
(2021)
Article
Operations Research & Management Science
Wenzhao Zhang, Xiaolong Zou
Summary: This paper studies nonzero-sum continuous-time constrained average stochastic games with independent state processes. By introducing average occupation measures, the existence of constrained Nash equilibria is established, and it is shown that each stationary Nash equilibrium corresponds to a global minimizer of a certain mathematical program.
Article
Statistics & Probability
Xin Guo, Yonghui Huang
Summary: This paper discusses risk-sensitive average optimization for denumerable continuous-time Markov decision processes, deriving principles and proving the existence of solutions. It also demonstrates that optimal policies for finite states can approximate those for infinitely countable states.
JOURNAL OF APPLIED PROBABILITY
(2021)
Article
Engineering, Electrical & Electronic
Daniel Chen, Alexander G. Strang, Andrew W. Eckford, Peter J. Thomas
Summary: This paper presents a continuous-time formulation of the sum-product algorithm for inferring the conditional probabilities of hidden states in a system. The algorithm, based on finite, discrete-time observations, explicitly solves for the conditional probability of occupying any state given the transition rates and observations within a finite time window.
IEEE TRANSACTIONS ON SIGNAL PROCESSING
(2022)
Article
Statistics & Probability
Xin Guo, Aiko Kurushima, Alexey Piunovskiy, Yi Zhang
Summary: This study investigates a gradual-impulse control problem of continuous-time Markov decision processes and demonstrates the existence of a deterministic stationary optimal policy under natural conditions, allowing multiple simultaneous impulses, randomized selection of impulses with random effects, and accumulation of jumps. The problem is simplified to an equivalent simple discrete-time Markov decision process, where the action space is the union of gradual and impulsive actions.
ADVANCES IN APPLIED PROBABILITY
(2021)
Article
Mathematics, Applied
Nicole Baeuerle
Summary: This article investigates mean-field control problems in discrete time, including discounted reward, infinite time horizon, and compact state and action space. The existence of optimal policies is proven, and the limiting mean-field problem is derived as the number of individuals approaches infinity. Furthermore, the average reward problem is considered, and it is shown that the optimal policy in this mean-field limit is e-optimal for the discounted problem when the number of individuals is large and the discount factor is close to one. This result is significant for obtaining an average reward optimal policy in problems where the reward depends only on the distribution of individuals, by first computing an optimal measure from a static optimization problem and then achieving it with Markov Chain Monte Carlo methods. Two applications are provided: congestion avoidance on a graph and optimal positioning on a market place, which are explicitly solved.
APPLIED MATHEMATICS AND OPTIMIZATION
(2023)
Article
Management
Xingyu Bai, Xin Chen, Alexander L. Stolyar
Summary: This paper investigates a partially observable lost-sales inventory system and proves the existence of a stationary optimal policy for average cost minimization using the vanishing discount factor approach. The key contribution of this study is a method to verify the uniform boundedness of the relative discounted value function, a crucial condition in the vanishing discount factor approach. Additionally, a valid policy is constructed to "copy" the actions of another policy for a process with a different initial state.
OPERATIONS RESEARCH
(2022)
Article
Engineering, Chemical
A. Deser, J. Kuhne
Summary: This article discusses the stochastic nature of charging in aerosol particles, utilizing the framework of continuous time Markov processes to analyze the principles of charging and introducing a novel numerical method for calculating the time evolution of charging processes. Additionally, the application of ergodicity is used to determine stationary charge distributions in the case of bipolar charging in finite state-space Markov processes.
JOURNAL OF AEROSOL SCIENCE
(2021)
Article
Statistics & Probability
Peng Liao, Zhengling Qi, Runzhe Wan, Predrag Klasnja, Susan A. Murphy
Summary: This study focuses on the batch (off-line) policy learning problem in the infinite horizon Markov decision process and proposes a doubly robust estimator to estimate the average reward. Moreover, an optimization algorithm is developed to compute the optimal policy in a parameterized stochastic policy class.
ANNALS OF STATISTICS
(2022)
Article
Physics, Multidisciplinary
Andreas Dechant
Summary: In this study, we investigate the problem of minimizing the entropy production for a physical process described by Markov jump dynamics. We find that, without any additional constraints, a given time-evolution can be realized with arbitrarily small entropy production at the expense of diverging activity. However, when the activity is fixed, the dynamics that minimizes the entropy production is driven by conservative forces. Moreover, we express the value of the minimum entropy production in terms of the graph-distance based Wasserstein distance between the initial and final configuration, which introduces a new type of speed limit relating dissipation, the average number of transitions, and the Wasserstein distance. We also demonstrate our findings using simple state networks, a time-dependent pump, and spin flips in the Ising model.
JOURNAL OF PHYSICS A-MATHEMATICAL AND THEORETICAL
(2022)
Article
Computer Science, Interdisciplinary Applications
Jiawen Hu, Ancha Xu, Bo Li, Haitao Liao
Summary: This study focuses on the case where system degradation and environmental condition evolution are governed by Markov processes. It proposes an inspection/maintenance policy, determines long-run average cost based on semi-regenerative properties, and minimizes cost by jointly determining key parameters.
COMPUTERS & INDUSTRIAL ENGINEERING
(2021)