Mark Kac introduced a method for calculating the distribution of the integral Av= ∫ T 0 v(Xt) dt for a function v of a Markov process (Xt; t¿0) and a suitable random time T , which yields the Feynman–Kac formula for the moment-generating function of Av. We review Kac’s method, with emphasis on an aspect often overlooked. This is Kac’s formula for moments of Av, which may be stated as follows. F...

Abstract We prove that the scaling limit of nearest-neighbour senile reinforced random walk is Brownian Motion when the time T spent on the first edge has finite mean. We show that under suitable conditions, when T has heavy tails the scaling limit is the so-called fractional kinetics process, a random time-change of Brownian motion. The proof uses the standard tools of time-change and invarian...

Let (Xt)t≥0 be a continuous-time irreducible Markov chain on a finite statespace E, let v be a map v : E → R\{0} and let (φt)t≥0 be an additive functional defined by φt = ∫ t 0 v(Xs)ds. We consider the cases where the process (φt)t≥0 is oscillating and where (φt)t≥0 has a negative drift. In each of the cases we condition the process (Xt, φt)t≥0 on the event that (φt)t≥0 stays non-negative until...

A convergence analysis is presented for additive Schwarz iterations when applied to consistent singular systems of equations of the form Ax = b. The theory applies to singular M -matrices with one-dimensional null space and is applicable in particular to systems representing ergodic Markov chains, and to certain discretizations of partial differential equations. Additive Schwarz can be seen as ...

We show that the method of Kipnis and Varadhan (Comm. Math. Phys. 104 (1986) 1) to construct a martingale approximation to an additive functional of a stationary ergodic Markov process via the resolvent is universal in the sense that a martingale approximation exists if and only if the resolvent representation converges. A sufficient condition for the existence of a martingale approximation is ...

Many large Markov decision processes (MDPs) can be represented compactly using a structured representation such as a dynamic Bayesian network. Unfortunately, the compact representation does not help standard MDP algorithms, because the value function for the MDP does not retain the structure of the process description. We argue that in many such MDPs, structure is approximately retained. That i...

The Restricted Additive Schwarz (RAS) method is adapted to the problem of computing the stationary probability distribution vector of large, sparse, irreducible stochastic matrices. Inexact and two-level variants are also considered, as well as acceleration by Krylov subspace methods. The convergence properties are analyzed and extensive numerical experiments aimed at assessing the effect of va...

Joint source–channel coding for stationary memoryless and Gauss–Markov sources and binary Markov channels is considered. The channel is an additive-noise channel where the noise process is an M th-order Markov chain. Two joint source-channel coding schemes are considered. The first is a channel-optimized vector quantizer—optimized for both source and channel. The second scheme consists of a sca...

In this study we model the daily rainfall occurrence using Markov Chain Analogue Yearmodel (MCAYM) and the intensity or amount of daily rainfall using three different probability distributions; gamma, exponential and mixed exponential distributions. Combining the occurrence and intensity model we obtain Markov Chain Analogue Year gamma model (MCAYGM), Markov Chain Analogue Year exponentia...

In this paper, a novel architecture, which integrates the recurrent neural network (RNN) based compensation process and the hidden Markov model (HMM) based speech recognition process into a unified framework, is proposed. The RNN is employed to estimate the additive bias, which represents the telephone channel effect, in the cepstral domain. Compensation of telephone channel effects is implemen...