Zero - Sum Games with Vector - Valued Payoffs

In this lecture we formulate and prove the celebrated approachability theorem of Blackwell, which extends von Neumann's minimax theorem to zero-sum games with vector-valued payoffs [1]. (The proof here is based on the presentation in [2]; a similar presentation was given by Foster and Vohra [3].) This theorem is powerful in its own right, but also has significant implications for regret minimization; as we will see in the next lecture, the algorithmic insight behind Blackwell's theorem can be used to easily develop both external and internal regret minimizing algorithms. We first define two-player zero-sum games with vector-valued payoffs. Each player i has an action space A i (assumed to be finite). In a vector-valued game, the payoff to player 1 when the action pair (a 1 , a 2) is played is Π(a 1 , a 2) ∈ R K , for some finite K; that is, the payoff to player 1 is a vector. Similarly, the payoff to player 2 is −Π(a 1 , a 2). We use similar notation as earlier lectures: i.e., we let Π(s 1 , s 2) denote the expected payoff to player 1 when each player i uses mixed action s i ∈ ∆(A i). We will typically view s i as a vector in R A i , with s i (a i) equal to the probability that player i places on a i. The game is played repeatedly by the players. We use s t i to denote the mixed action chosen by player i at time t, and we let a t i denote the actual action played by player i at time t. We let h T = (a 0 ,. .. , a T −1) denote the history of the actual play up to time T. We assume that the payoffs all lie in the unit ball (with respect to the standard Euclidean norm): Π(a 1 , a 2) ≤ 1 for all a 1 , a 2. Since action spaces are finite, this just amounts to a rescaling of payoffs for analytical simplicity. 2 Approachability We first develop approachability in the scalar payoff setting. We then generalize to halfspaces in the vector-valued payoff setting, and finally state Blackwell's theorem for approachability of general convex sets. We first develop the notion of approachability in the one-dimensional (i.e., scalar payoff) setting, where K = 1. In this case players 1 …