Video-based License Plate Reader

This paper proposes an algorithm that automatically reads license plate from videos that has taken from personal devices. Unlike commonly used license plate reader in toll payment or road control system, video from personal devices usually has more variation in viewing angle, resolution, shutter speed and so on. This work read single frame from video first, then extract the license plate by detecting high-density vertical edge areas and filtered by color and boundary features. Then the preprocessing step automatically correct the rotation of the image, remove the background noise and segment the character portion of the license plate. Lastly, this algorithm recognizes plate with template matching and filters the peaks by peak intensity and coordinates. Based on these, videos with multiple frames can improve the overall accuracy and reliability. Keywords— Edge detection; Hough transform; Morphological image processing; template matching Automatic license plate reader, also known as Automatic number plate recognition (ANPR), was first invented in 1976 in at the Police Scientific Development Branch in the UK and are widely used among police forces worldwide today as it is essential for numerous real-life applications, such as traffic control, automatic toll collection and road traffic monitoring[1]. These systems usually require special hardware: one or more road-rule enforcement cameras, a single camera with assistant IR illumination, or closed-circuit television are often used. On the software side, ANPR typically detect the location of the license plate in an image first, pre-process the plate image by orientation and sizing correction, and segment the useful information portion of the license plate, finally it uses optical character recognition (OCR) to read the plate. Additional processing to check characters and positions to classify license plate and link license plate information to some database are often needed depending on the applications. The variation of the plate types and environmental differences increases the complexity of both the detection and recognitions part [2]. The plate variation include: 1) plate location including both plate image locates at different position of an image, and also the numerical information locates at different place of the license plate; 2) language and character font, special characters, total number of characters; 3) Color, includes both character and background color and pattern; 4) special plate such as for disabled people and special characters and so on. Fig. 1 shows that license plates in California has different color, location, fond from the plate from Colorado; even within California, the background/foreground color, background pattern, special characters, and special-use plates all exist. Also variety of plate frames that are commonly used on the car can also cause additional noise to the recognition results. Fig. 1. Variety of license plates. There are also imaging variations when pictures or videos are taken, these includes illumination (day, night, additional illumination beam etc.), similar pattern as background noise, camera resolution, camera shutter speed, motion blur (both from camera and from car), viewing angle, number of cameras and so on. The accuracy and reliability of the ANPR system is always a key part of the algorithm, thus much effort has been made to increase the ALPR reliability and accurate nowadays [3-4]. Recently, ALPR also gets interest beyond the police forces as personal cameras are more available and the computing hardware gets cheaper and cheaper. My project here is to detect and recognize the California license plate by analyzing videos from personal devices, such as cell phones, camera carried by hobby drones, which usually has low resolution or speed limit. In this case, it introduces more complexity and challenges (Fig. 2) as the personal devices usually does not have fixed viewing angle or field of view as the road cameras, they may suffer from the low resolution, slower speed and limited computing power, all results in complexity of the algorithm and longer computing time. Fig 2. Variety of viewing angle, extreme illumination condition and resolution. I. METHOD AND RESULTS From the video data, first a single frame image is read form the video. It goes through 4 algorithm steps to reads the plates, as shown in Fig.3. Fig 3. Algorithm flow chart (top) and Single frame example read from a video (bottom). A. License Plate Extraction First step is to detect the location of a license plate and extract the sub-image that includes the license plate for further processing steps. This removes most of the background noise and enables faster processing speed. This stage influences the accuracy of the recognition algorithm. There have been many different approaches used in previous studies to extract license plate and filter out false results. Four most commonly ones are: 1) Boundery and Edge information [5] 2) Texture Features[6] 3) Color features[7] 4) Character features [8] Since the license plate normally has a rectangular shape and a fixed aspect ratio, we can use edge detection to finding all rectangles and filter it with the known aspect ratio. However, due to the viewing angle, the license plate may shows a perspective effect or distortion from camera, occlusion or frames may all change the ratio; bumpers in the car increase the noise of finding rectangles, so many literatures recommend using vertical edges to find the license plate [3, 9]. Most license plates involve rapid characters and background change, also known as the texture feature of the license plate. This feature results in high edge density areas. Scan-line technique [10], sliding concentric window [11], adaptive boosting(AdaBoost) combining with Haar-like feature [6] and other techniques have been used to detect license plate accordingly. For many countries, there are specific colors for backgrounds and foregrounds are allowed in the license plate, in that case, color feature can be used for plate extraction. However, defining the pixel color using the RGB value is very difficult, especially in different illumination conditions. Many techniques would convert the RGB color to hue, lightness, and saturation (HLS) or other color models to improve robustness. For California, there is variation in plate color, making it more difficult to use color feature. Lastly, character feature can also be used to detect license plate. In this case, we can directly read the license characters, especially for some countries where there are special limited characters exist on the license plates that can be used to detect features. Scale-invariant feature transform (SIFT) can be used to detect license plate [13], and maximally stable extremal regions (MSER) can also be used to assist this process [14]. However, these feature detection are time consuming and often requires high quality images (focus and resolution). In my project I combined the first 3 methods to detect the license plates. I first use Sobel filter for vertical edge detection of the image frame, as shown in Fig 4. Fig. 4 Map of the vertical edges from Sobel filter. Then I apply Gaussian filter to blur and find the highdensity area in the edge map. The California license plates often involve high contrast of colors between characters and the background, and the texture feature also has rapid color change. Here I take this advantage and applied the same filtering for all 3 RGB channels, and filter out single color edges, such as that from rear light or environmental background, as shown in Fig. 5. Fig. 5: High density vertical-edge regions (top) and selected regions after color feature filtering (bottom); Then I use region labeling to label all possible regions. Odd shape regions are filtered out by a threshold of the ratio of region area versus rectangular bounding box area. Lastly I use a fixed aspect ratio of 2:1 with a small variation to select possible regions of license plates, as shown in Fig 6. Fig 6. Filter regions from boundary feature in license plate extraction step. SIFT with character template is also used and evaluated in the extraction step. However, I found it has very high requirement on the image quality as it capture matched feature with high resolution license plate, but no results at low resolution ones. Also SIFT take much longer time. The final algorithm doesn’t include SIFT part. Detected region is highlighted in a yellow box in the original image as part of the final results with labeling of the detected license plate number (Fig 7). Fig. 7. Detected license plate highlighted in original image B. License Plate autocorrection After extract the license plate, I applied a preprocessing algorithm to autocorrect the sub-image in order to remove noise, segment the character region, and enhance the signal. Given the fact that personal devices may take videos/photos with random angle, the first pre-processing is to correct the rotation of the plate. Map of veritical edges from Sobel filter Area filter by vertical-edge density in all 3 color channels