Stochastic Contextual Bandits with Known Reward Functions

Many sequential decision-making problems in communication networks such as power allocation in energy harvesting communications, mobile computational offloading, and dynamic channel selection can be modeled as contextual bandit problems which are natural extensions of the well-known multi-armed bandit problem. In these problems, each resource allocation or selection decision can make use of available side-information such as harvested power, specifications of the jobs to be offloaded, or the number of packets to be transmitted. In contextual bandit problems, at each step of a sequence of trials, an agent observes the side information or context, pulls one arm and receives the reward for that arm. We consider the stochastic formulation where the context-reward tuples are independently drawn from an unknown distribution in each trial. The goal is to design strategies for minimizing the expected cumulative regret or reward loss compared to a distribution-aware genie. We analyze a setting where the reward is a known non-linear function of the context and the chosen arm’s current state which is the case with many networking applications. This knowledge of the reward function enables us to exploit the obtained reward information to learn about rewards for other possible contexts. We first consider the case of discrete and finite context-spaces and propose DCB( ), an algorithm that yields regret which scales logarithmically in time and linearly in the number of arms that are not optimal for any context. This is in contrast with existing algorithms where the regret scales linearly in the total number of arms. Also, the storage requirements of DCB( ) do not grow with time. DCB( ) is an extension of the UCB1 policy for the standard multi-armed bandits to contextual bandits using sophisticated proof techniques for regret analysis. We then study continuous context-spaces with Lipschitz reward functions and propose CCB( , δ), an algorithm that uses DCB( ) as a subroutine. CCB( , δ) reveals a novel regret-storage trade-off that is parametrized by δ. Tuning δ to the time horizon allows us to obtain sub-linear regret bounds, while requiring sub-linear storage. Joint learning for all the contexts results in regret bounds that are unachievable by any existing contextual bandit algorithm for continuous context-spaces. Similar performance bounds are also shown to hold for unknown horizon case by employing a doubling trick.