We study the problem of online learning in a class of Markov decision processes known as linearly solvable MDPs. In the stationary version of this problem, a learner interacts with its environment by directly controlling the state transitions, attempting to balance a fixed state-dependent cost and a certain smooth cost penalizing extreme control inputs. In the current paper, we consider an onli...

This paper presents an abstraction-refinement framework for Segala’s probabilistic automata (PA), a slight variant of Markov decision processes. We use Condon and Ladner’s two-player probabilistic game automata extended with possible and required transitions — as in Larsen and Thomsen’s modal transition systems — as abstract models. The key idea is to refine player-one and player-two states sep...

In this paper, we present a Distributionally Robust Markov Decision Process (DRMDP) approach for addressing the dynamic epidemic control problem. The Susceptible-Exposed-Infectious-Recovered (SEIR) model is widely used to represent stochastic spread of infectious diseases, such as COVID-19. While Processes (MDP) offers mathematical framework identifying optimal actions, vaccination and transmis...

  A novel optimal single machine replacement policy using a single as well as a two-stage decision making process is proposed based on the quality of items produced. In a stage of this policy, if the number of defective items in a sample of produced items is more than an upper threshold, the machine is replaced. However, the machine is not replaced if the number of defective items is less than ...

We consider Laplacians for directed graphs and examine their eigenvalues. We introduce a notion of a circulation in a directed graph and its connection with the Rayleigh quotient. We then define a Cheeger constant and establish the Cheeger inequality for directed graphs. These relations can be used to deal with various problems that often arise in the study of non-reversible Markov chains inclu...

Graph transformation systems (GTS) have been proposed for high-level stochastic modelling of dynamic systems and networks. The resulting systems can be described as semi-Markov processes with graphs as states and transformations as transitions. The operational semantics of such processes can be explored through stochastic simulation. In this paper, we develop the basic theory of stochastic grap...

We introduce two-level discounted games played by two players on a perfect-information stochastic game graph. The upper level game is a discounted game and the lower level game is an undiscounted reachability game. Two-level games model hierarchical and sequential decision making under uncertainty across different time scales. We show the existence of pure memoryless optimal strategies for both...

Related DatabasesWeb of Science You must be logged in with an active subscription to view this.Article DataHistorySubmitted: 20 April 2020Accepted: 03 February 2021Published online: 29 2021KeywordsMarkov decision process, partial observation, long-run average payoffAMS Subject Headings90C39, 90C40, 37A50, 60J20Publication DataISSN (print): 0363-0129ISSN (online): 1095-7138Publisher: Society for...

We analyze Markov chains for generating a random k-coloring of a random graph Gn,d/n. When the average degree d is constant, a random graph has maximum degree log n/ log log n, with high probability. We efficiently generate a random k-coloring when k = Ω(log log n/ log log log n), i.e., with many fewer colors than the maximum degree. Previous results hold for a more general class of graphs, but...