Block Sorting Text Compression — Final Report

A recent development in text compression is a “block sorting” algorithm which permutes the input text according to a special sort procedure and then processes the permuted text with Move-to-Front and a final statistical compressor. The technique combines good speed with excellent compression performance. This report investigates the block sorting compression algorithm, in particular trying to understand its operation and limitations. Various approaches are investigated in an attempt to improve the compression with block sorting, most of which involve a hierarchy of coding models to allow fast adaptation to local contexts. The best technique involves a new “structured” coding model, especially designed for compressing data with skew symbol distributions. Block sorting compression is found to be related to work by Shannon in 1951 on the prediction of English text. The work confirms block-sorting as a good text compression technique, with a compression approaching that of the currently best compressors while being much faster than other compressors of comparable performance. Preface This is third report of a series on block sorting text compression (previous members Technical Reports 111, 120, Refs 10 & 11). A shorter version was presented as a paper at ACSC’96 (ref [12]). This present report was prepared as a comprehensive account of my experience with block sorting compression, including several interesting curiosities, but grew too large for publication as a paper. Important material will be extracted to form probably two journal papers, but this text remains as the comprehensive and coherent picture of the work. While it largely replaces the two earlier reports, those do contain some useful material which is omitted from here. For example Tech Report 111 includes extensive logs of the compression process and output which could be useful for people interested in the details of the operation. 1 The report is available by anonymous FTP from ftp.cs.auckland.ac.nz /out/peter-f/TechRep130.ps