Rate-Distortion Bounds for High-Resolution Vector Quantization via Gibbs's Inequality

Gibbs’s inequality states that the differential entropy of a random variable with probability density function (pdf) f is less than or equal to its cross entropy with any other pdf g defined on the same alphabet, i.e., h(X) ≤ −E[log g(X)]. Using this inequality with a cleverly chosen g, we derive a lower bound on the smallest output entropy that can be achieved by quantizing a d-dimensional source with given expected rth-power distortion. Specialized to the one-dimensional case, and in the limit of vanishing distortion, this lower bound converges to the output entropy achieved by a uniform quantizer, thereby recovering the result by Gish and Pierce that uniform quantizers are asymptotically optimal as the allowed distortion tends to zero. Our lower bound holds for any d-dimensional memoryless source that has a pdf and whose differential entropy and Rényi information dimension are finite. In contrast to Gish and Pierce, we do not require any additional constraints on the continuity or decay of the source pdf.