Enhanced SATD-based cost function for mode selection of H.264/AVC intra coding

In H.264/AVC, the concept of rate-distortion optimization (RDO) mode decision has proven to be an important coding tool. However, this mode decision method makes the encoding process extremely complex, especially in the computation of the rate-distortion cost function, which includes the computations of the sum of squared difference (SSD) between the original and reconstructed image blocks and context-based entropy coding of the block. In this paper, we propose an enhanced low-complexity cost function for H.264/AVC intra 4 × 4 and 8 × 8 mode selections. The enhanced cost function uses sum of absolute Hadamardtransformed differences (SATD) and mean absolute deviation of the residual block to estimate distortion part of the cost function. A threshold-based large coefficients count is also used for estimating the bit-rate part. The proposed method improves the rate-distortion performance of conventional fast cost functions while maintaining low complexity requirement. As a result, the encoding process can be significantly accelerated with the use of the proposed cost function. Simulation results confirm that the proposed method reduces about 83.74 % computation of intra encoding of JM 12.4 encoder with negligible rate-distortion performance degradation. M. G. Sarwer (B) · Q. M. J. Wu Department of Electrical and Computer Engineering, University of Windsor, 401 Sunset Avenue, Windsor, ON, Canada e-mail: [email protected] Q. M. J. Wu e-mail: [email protected] M. G. Sarwer · X.-P. Zhang Department of Electrical and Computer Engineering, Ryerson University, Toronto, ON, Canada X.-P. Zhang e-mail: [email protected]