We consider a reinforcement learning (RL) setting in which the agent interacts with a sequence of episodic MDPs. At the start of each episode the agent has access to some side-information or context that determines the dynamics of the MDP for that episode. Our setting is motivated by applications in healthcare where baseline measurements of a patient at the start of a treatment episode form the...

Learning temporal dependencies between variables over continuous time is an important and challenging task. Continuous-time Bayesian networks effectively model such processes but are limited by the number of conditional intensity matrices, which grows exponentially in the number of parents per variable. We develop a partition-based representation using regression trees and forests whose paramet...

We present update formulas that allow us to express the stationary distribution of a continuous-time Markov process with denumerable state space having generator matrix Q∗ through a continuous-time Markov process with generator matrix Q. Under suitable stability conditions, numerical approximations can be derived from the update formulas, and we show that the algorithms converge at a geometric ...

This week we take the limit ∆t → 0. The limit is a process Xt that is defined for all t in some range, such as t ∈ [0, T ]. The process takes place in continuous time. This week, Xt is a continuous function of t. The process has continuous sample paths. It is natural to suppose that the limit of a Markov process is a continuous time Markov process. The limits we obtain this week will be either ...