Successive Approximation Methods in Undiscounted Stochastic Games

T HIS PAPER considers two-person, zero-sum stochastic games with finite state space = {1, -, N} and in each state i E 2, two finite sets K(i) and L(i) of actions available to player 1 and 2, respectively. The state of the system is observed at equidistant epochs. When the system is observed to be in state i, the two players choose an action, or a randomization of actions out of K(i) and L(i), respectively. When the actions k C K(i), 1 E L(i) are chosen in state i, thenP*" ' 0 denotes the probability that state j is the next state to be observed ( PPki = 1) and qi is the one-step expected reward earned by player 1 from player 2. If the payoffs are discounted at the interest rate r > 0, the stochastic game is called the r-discounted game. The existence of a value and of stationary optimal policies in the r-discounted game goes essentially back to Shapley [22]; in addition it is easily verified that value-iteration converges to the value of the game, in view of the value-iteration operator being a contraction mapping on EN, the N-dimensional Euclidean space. In the undiscounted version of the game, i.e., when the long run average return per unit time is the criterion to be considered, one or both players may fail to have optimal stationary policies, as follows from an example in Gillette [11]. Both for this model and for the case of more general state and action spaces, recurrency conditions with respect to the transition probability matrices (tpm's) associated with the stationary policies have been obtained under which the existence of a stationary pair of equilibrium policies (AEP) is guaranteed (see Federgruen [7], Hoffman and Karp [13], Rogers [18], Sobel [23], and Stern [24]). So far, very little attention has been paid to the actual computation of both the asymptotic average value g * and of a solution v * to the average