If optimality is measured by average codeword length, Huffman's algorithm gives optimal codes, and the redundancy can be measured as the difference between the average codeword length and Shannon's entropy. If the objective function is replaced by an exponentially weighted average, then a simple modification of Huffman's algorithm gives optimal codes. The redundancy can now be measured as the d...

The Linux pseudorandom number generator (PRNG) is a PRNG with entropy inputs which is widely used in many security related applications and protocols. This PRNG is written as an open source code which is subject to regular changes. It was last analyzed in the work of Gutterman et al. in 2006 [GPR06] but since then no new analysis has been made available, while in the meantime several changes ha...

The article discusses excess entropy defined as mutual information between the past and future of a stationary process. The central result is an ergodic decomposition: Excess entropy is the sum of self-information of shift-invariant σ-field and the average of excess entropies for the ergodic components of the process. The result is derived using generalized conditional mutual information for fi...

We offer two noiseless codes for blocks of integers Xn = (X1, . . . , Xn). We provide explicit bounds on the relative redundancy that are valid for any distribution F in the class of memoryless sources with a possibly infinite alphabet whose marginal distribution is monotone. Specifically we show that the expected code length L(Xn) of our first universal code is dominated by a linear function o...

We use the \conservation of entropy" 1] to derive a simple formula for the redundancy of a large class of variable-to-variable length codes on discrete, memoryless sources. We obtain new asymptotic upper bounds on the redundancy of the \Tunstall-Huuman" code and the \Tunstall-Shannon-Fano" code. For some sources we provide the best existing upper bound for the smallest achievable asymptotic red...

To investigate the fundamental causes of bloat, six artificial random binary tree search spaces are presented. Fitness is given by program syntax (the genetic programming genotype). GP populations are evolved on both random problems and problems with “building blocks”. These are compared to problems with explicit ineffective code (introns, junk code, inviable code). Our results suggest the entr...

It is proven that for every random variable with a countably infinite set of outcomes and finite entropy there exists an optimal prefix code which can be constructed from Huffman codes for truncated versions of the random variable, and that the average lengths of any sequence of Huffman codes for the truncated versions converge to that of the optimal code. Also, it is shown that every optimal i...