Improving the Pareto UCB1 Algorithm on the Multi-Objective Multi-Armed Bandit

In this work, we introduce a straightforward approach for bounding the regret of Multi-Objective Multi-Armed Bandit (MO-MAB) heuristics extended from standard bandit algorithms. The proposed methodology allows us to easily build upon the regret analysis of the heuristics in the standard bandit setting. Using our approach, we improve the Pareto UCB1 algorithm, that is the multi-objective extension of the seminal UCB1, by performing a tighter regret analysis. The resulting Pareto UCB1* also has the advantage of being empirically usable without any approximation. 1 Multi-Objective Multi-Armed Bandit The Multi-Objective Multi-Armed Bandit (MO-MAB) setting [1] is described by a set of arms K associated with a set of random variables vectors {xk,t|t ≥ 1} for each k inK. LetN be the number of objectives. Vector xk,t = [xk,t,1, . . . , xk,t,N ] indicates the random outcome of the k-th arm in its t-th trial, where xk,t,i ∈ R. We consider the stochastic setting where all xk,t associated with k are independent and identically distributed according to some unknown distribution with unknown expectation vector μk = [μk,1, . . . , μk,N ]. Given two arms a and b, a is said to dominate, or Pareto-dominate, b (denoted a b) if μa,i ≥ μb,i for every objective i. The dominance is strict (denoted a b) if μa,i > μb,i for every objective i. Finally, the two arms are incomparable (denoted a ‖ b) if a b and b a. The set of optimal arms contains all the non-dominated arms such thatK∗ = {k ∈ K|@k′ ∈ K,μk′ μk}. The Pareto front P , also referred to as the Pareto-optimal set, contains the expectations of the optimal arms such that P = {μk∗ ∀k∗ ∈ K∗}. In this work, we consider the setting where all optimal arms are considered equivalent, that is we are not biased in playing any of them more than others (in K∗). The problem can be formulated as a game where a player sequentially selects arms inK and observes rewards according to the played arms. Let k(t) denote the arm played at episode t and the reward r(t) = xk(t),t. The goal is to simultaneously maximize the reward over time for all objectives. Therefore, we want to play as much as possible any optimal arm in K∗. Let nk(t) denote the number of times arm k has been played of to time t − 1. The performance is measured with the expected regret metric denoted as E[R(T )] = ∑ k∈K E[nk(T )]∆k, (1) where T is the number of episodes performed up to now and ∆k corresponds to the regret of playing arm k instead of an optimal arm (in K∗). A typical approach for adapting standard bandit heuristics to the MO-MAB setting relies on the concept of Pareto-dominance. Instead of playing the arm that maximizes the expected regret, one ∗{audrey.durand.2, charles.bordet.1}@ulaval.ca †[email protected]