The Rényi redundancy of generalized Huffman codes

If optimality is measured by average codeword length, Huffman's algorithm gives optimal codes, and the redundancy can be measured as the difference between the average codeword length and Shannon's entropy. If the objective function is replaced by an exponentially weighted average, then a simple modification of Huffman's algorithm gives optimal codes. The redundancy can now be measured as the difference between this new average and Renyi's generalization of Shannon's entropy. By decreasing some of the codeword lengths in a Shannon code, the upper bound on the redundancy given in the standard proof of the noiseless source coding theorem is improved. The lower bound is improved by randomizing between codeword lengths, allowing linear programming techniques to be used on an integer programming problem. These bounds are shown to be asymptotically equal, providing a new proof of Kricevski's results on the redundancy of Huffman codes. These results are generalized to the Renyi case and are related to Gallager's bound on the redundancy of Huffman codes. PREVIOUS WORK I N 1961, Renyi [12] proposed that the Shannon entropy could be generalized to s+1 ( m ) H.( p) =-s-log i~l p}l 0, which approaches the Shannon entropy ass~ o+. In 1965, Campbell [1] showed that just as the Shannon entropy is a lower bound on the average codeword length of a uniquely decodable code, the Renyi entropy is a lower bound on the exponentially weighted average codeword length ~log ( i~l P;2sl;)' s > 0.