We address the problem of computing an optimal value function for Markov decision processes. Since finding this function quickly and accurately requires substantial computation effort, techniques that accelerate fundamental algorithms have been a main focus of research. Among them prioritization solvers suggest solutions to the problem of ordering backup operations. Prioritization techniques fo...

This paper presents a novel framework for studying partially observable Markov decision processes (POMDPs) with finite state, action, observation sets, and discounted rewards. The new framework is solely based on future-reward vectors associated with future policies, which is more parsimonious than the traditional framework based on belief vectors. It reveals the connection between the POMDP pr...

This paper proposes, applies and evaluates a new technique for texture classification in digital images. The work describes, as far as possible in a quantitative way, the concept of texture in digital images. Furthermore, we developed an innovative model that allows classifying and characterizing texture in digital images, to be used as a useful tool in noninvasive inspection of visual surfaces...

For a discrete-time finite-state Markov chain, we develop an adaptive importance sampling scheme to estimate the expected total cost before hitting a set of terminal states. This scheme updates the change of measure at every transition using constant or decreasing step-size stochastic approximation. The updates are shown to concentrate asymptotically in a neighborhood of the desired zero varian...

In this paper, the ambiguity of nite state irreducible Markov chain trajectories is reminded and is obtained for two state Markov chain. I give an applicable example of this concept in President election

A popular technique for studying random properties of a combinatorial set is to design a Markov chain Monte Carlo algorithm. For many problems there are natural Markov chains connecting the set of allowable conngurations which are based on local moves, or \Glauber dynamics." Typically these single site update algorithms are diicult to analyze, so often the Markov chain is modiied to update seve...

indicates the categories of a chosen cycle or could refer to additional attributes of the patients like age or sex. RESULTS: State probability and survival curves merely show specific aggregates of the data while classic Markov trace visualizations with for example bubble diagrams do not visualize data in a sense that would facilitate a detection of proportions and trends. Applying Parallel Set...

Fast response times and the satisfaction of response time quantile targets are important performance criteria for almost all transaction processing and computer-communication systems. We present a distributed uniformizationbased technique for obtaining response time densities from very large unstructured Markov models. Our method utilises hypergraph partitioning to minimise inter-processor comm...

Open queueing networks with Markovian arrival processes and phase type service times are considered. A technique is proposed for the speciication of the models. Using such a speciication, a Markov chain can be generated automatically. Furthermore, this description can be used immediately for simulation. An approximate disaggregation{aggregation technique for nite state Markov chains is applied....

We shortly review the uncoupling-coupling method, a Markov chain Monte Carlo based approach to compute statistical properties of systems like medium-sized biomolecules. This technique has recently been proposed for the efficient computation of biomolecular conformations. One crucial step of UC is the decomposition of reversible nearly uncoupled Markov chains into rapidly mixing subchains. We sh...