Practical and Effective Re-Pair Compression

Re-Pair is an efficient grammar compressor that operates by recursively replacing high-frequency character pairs with new grammar symbols. The most space-efficient linear-time algorithm computing Re-Pair uses (1 + )n+ √ n words on top of the re-writable text (of length n and stored in n words), for any constant > 0; in practice however, this solution uses complex sub-procedures preventing it from being practical. In this paper, we present an implementation of the abovementioned result making use of more practical solutions; our tool further improves the working space to (1.5 + )n words (text included), for some small constant . As a second contribution, we focus on compact representations of the output grammar. The lower bound for storing a grammar with d rules is log(d!) + 2d ≈ d log d + 0.557d bits, and the most efficient encoding algorithm in the literature uses at most d log d + 2d bits and runs in O(d1.5) time. We describe a linear-time heuristic maximizing the compressibility of the output Re-Pair grammar. On real datasets, our grammar encoding uses—on average—only 2.8% more bits than the informationtheoretic minimum. In half of the tested cases, our compressor improves the output size of 7-Zip with maximum compression rate turned on. 1998 ACM Subject Classification E.4 Coding and Information Theory, E.1 Data Structures, F.2.2 Nonnumerical Algorithms and Problems