Information Theory 4.1 Entropy and Mutual Information

Neural encoding and decoding focus on the question: " What does the response of a neuron tell us about a stimulus ". In this chapter we consider a related but different question: " How much does the neural response tell us about a stimulus ". The techniques of information theory allow us to answer this question in a quantitative manner. Furthermore, we can use them to ask what forms of neural response are optimal for conveying information about natural stimuli. Information theory is a general framework for quantifying the ability of a coding scheme to convey information. It is assumed that the code involves a number of symbols, and the quantities we consider in this chapter, the entropy and mutual information, depend on the frequencies with which these symbols, or combinations of them, are used. Entropy is a measure of the theoretical capacity of a code to convey information. Mutual information measures how much of that capacity is actually used when the code is applied to describe a particular set of data. In neuroscience applications, the symbols we consider are neuronal responses , and the data sets they describe are stimulus characteristics. In the most complete analyses, which are considered at the end of the chapter, the neuronal response is characterized by a list of action potential firing times. The symbols being analyzed in this case are sequences of action potentials. Computing the entropy and mutual information for spike sequences can be difficult because the frequency of occurrence of many different spike sequences must be determined. This typically requires a large amount of data. For this reason, many information theory analyses use simplified descriptions of the response of a neuron that reduce the number of possible 'symbols' (i.e. responses) that need to be considered. We discuss cases in which the symbols consist of responses with different numbers of action