Multi-frame Super Resolution for Improving Vehicle Licence Plate Recognition

License plate recognition (LPR) by digital image processing, which is widely used in traffic monitor and control, is one of the most important goals in Intelligent Transportation System (ITS). In real ITS, the resolution of input images are not very high since technology challenges and cost of high resolution cameras. However, when the license plate image is taken at low resolution, the license plate cannot be readable; hence, the recognition algorithm could not work well. There are many reasons resulting in the degradation of captured license plate images, such as downsampling, blurring, warping, noising, and distance of car from camera. Many researchers try to enhance the quality of input images by image restoration algorithms to improve the LPR final accuracy. Recently, super-resolution (SR) techniques are widely used to construct a high-resolution (HR) image from several observed low-resolution (LR) images, thereby removing the degradations caused by the imaging of a low resolution camera. As mentioned, in real ITS, the resolution of input image is not high, but there are successive frames from a target, therefore multi-frame SR methods can be used to overcome the ITS resolution challenges. In this paper, an SR technique based on POCS (Projection onto Convex Sets) is used to reconstruct an HR license plate image from a set of registered LR images. The normalized convolution (NC) framework is used in POCS, in which the local signal is approximated through a projection onto a subspace. However, the window function of adaptive NC is adapted to local linear structures. This results in more samples of the same modality being fused for the reconstruction, which in turn reduces diffusion across discontinuities, that is very important factor in improving LPR accuracy. The first step in multi-frame SR is image registration which is necessary to improve quality of the reconstructed HR image, especially in LPR when the quality of the reconstructed edges of characters is very important. For simplicity, it is often supposed simple motions (usually translation) between successive frames in multi-frame SR, but changes in scale, rotation and translation in license plate successive images may happened. It means that the registration is one of the main challenges in SR used for LPR. This paper proposes use of a two-step image matching algorithm to improve the quality of registration stage. In the first step, Fourier-Mellin image matching is used for registration which overcomes the scale and rotation challenge, but the accuracy of registration is not suitable. After matching of the successive input images by Fourier-Mellin algorithm, the Keren or Vandewalle image matching is used to improve the quality of final registration. For real LR images, Fourier-Mellin plus Keren shows higher performance while for simulated LR images, Fourier-Mellin plus Vandewalle shows higher performance. In order to compare the results of two proposed SR algorithms for LPR application with the other methods, we prepare three real datasets of successive frames for Persian LPR, the first and the second one are captured HR and LR successive frames, respectively, while the third one is a downsampled LR version of HR frames. The output HR image of all compared methods is feed to a demo version of a Persian LPR software (www.farsiocr.ir), and the accuracy of each character and the accuracy each license are reported. Five SR methods are compared including: cubic interpolation, ASDS-AR (Adaptive Sparse Domain Selection and Adaptive Regularization), standard POCS, our first and second proposed SR method which both of them firstly use Fourier-Mellin registration, while the first one uses Keren, and the second one uses Vandewalle image matching for a fine registration. Moreover, to present the effectiveness of using SR methods before LPR, the LR images are also directly feed to LPR software. The results represent when the length of license is less than 50 pixels, using SR methods before LPR improves the recognition accuracy. Moreover, when the license plate length is less 35 pixels, SR methods could not improve the performances. Our investigations show that for LR downsampled images from HR ones, our proposed SR method with Fourier-Mellin plus Keren registration reaches to the highest performance, while for real LR images, which are captured by a low resolution camera, our proposed SR method with Fourier-Mellin plus Vandewalle registration reaches to the highest performance. On the other hand, since some Persian numerical characters, like 2 (2) and 3 (3) are very similar to each other, all of the compared methods may confuse between them in LPR step, therefore, the accuracy per license of all compared methods are not high. Among all previous compared methods, for LR images with length between 35 to 50 pixels, the standard PCOS shows the best results, while our proposed SR methods improve the accuracy per character around 25%, with respect to PCOS method.