Cognitive Science Honors Thesis A Computational Account of Sensory Prediction Error Gating in Reinforcement Learning Models

A successful return in tennis requires a tennis player first to determine where best to place her return and then to correctly execute her swing. If she makes an errant return, she now faces a credit assignment problem: Should this negative outcome be attributed to poor shot selection or to an error in motor execution? McDougle et al. propose a solution to this problem when the source of the error is the motor system. They posit that motor errors are communicated to the decision-making system whereby they gate learning in order to prevent the undesired negative reinforcement of the chosen action. This gating hypothesis was motivated by recent anatomical evidence that the cerebellum — a crucial node in a network widely thought to process motor execution errors — sends direct subcortical projections to the basal ganglia — a crucial node in a network widely thought to drive reinforcement learning and decision-making. In McDougle et al.’s gating model, motor execution errors scale learning rates in a temporal difference (TD) reinforcement learning model of decision-making. However, the most prominent attempts to link the basal ganglia to reinforcement learning models have instead suggested that actor-critic (AC) models may be more appropriate models of basal ganglia anatomy. In the present study, we investigate the gating hypothesis from the perspective of AC models. We find that AC models can account for McDougle et al.’s behavioral results, but that gating is necessary for them to do so. Additionally, we find that simultaneous gating of the actor and critic is the only AC gating model that can account for subject data, but that AC models do not account for subject data better than TD models. We conclude with a discussion of the biological plausibility of the proposed gating mechanism from the perspective of the AC gating model.