Distributed Optimization in Adaptive Networks: Appendix

1 Markov Decision Processes Consider a Markov chain (w(k), a(k)) defined for k = 0, 1,. .. and with w(k) ∈ W, a(k) in A, where W and A are finite sets representing the system state space and the action space, respectively. The transition probabilities are defined by the function P θ (w , a , w, a) = Pr w(k + 1) = w, a(k + 1) = a| w(k) = w , a(k) = a. Here, θ ∈ R N is a vector of policy parameters. We will make the following assumption regarding the dynamics. Assumption 1.1. For all θ, the Markov chain (w(k)) is ergodic (aperiodic, irre-ducible). While the system is in state w ∈ W and action a ∈ A is applied, a reward r(w, a) is accrued. We will use the shorthand r(k) = r(w(k), a(k)). Given Assumption 1.1, we can define the long term average reward by λ(θ) = lim K→∞