Associative Reinforcement Learning using Linear Probabilistic Concepts

We consider the problem of maximizing the total number of successes while learning about a probability function determining the likelihood of a success. In particular, we consider the case in which the probability function is represented by a linear function of the attribute vector associated with each action/choice. In the scenario we consider, learning proceeds in trials and in each trial, the algorithm is given a number of alternatives to choose from, each having an attribute vector associated with it, and for the alternative it selects it gets either a success or a failure with probability determined by applying a xed but unknown linear success probability function to the attribute vector. Our algorithms consist of a learning method like the Widrow-Ho rule and a probabilistic selection strategy which work together to resolve the so-called exploration-exploitation tradeo . We analyze the performance of these methods by proving bounds on the worst-case regret, or how many less successes they expect to get as compared to the ideal (but unrealistic) strategy that knows the target probability function. Our analysis shows that the worst-case (expected) regret for our methods is almost optimal: the upper bounds grow with the number m of trials and the number n of alternatives like O(m 3=4 n 1=2 ) and O(m 4=5 n 2=5 ), and the lower bound is