An Airborne Wide Angle Camera System for Automatic near Real-time Road Traffic Monitoring

In a society that relies on individual mobility day to day sufficient methods for area-wide traffic monitoring and guidance are requested. Therefore, large-area traffic monitoring with high spatial and temporal resolution becomes necessary in order to exploit the existing road capacities sufficiently. Moreover, daily commuters are interested in predictions of their travel times to work and back, being able to plan their daily business or to change to public transportation systems in case of extensive traffic congestion. Furthermore, mass events and natural disasters require fast and sufficient traffic guidance, for the public and especially for actionand relief forces. In this paper, we present a remote sensing approach for automatic traffic monitoring based on an airborne wide angle digital camera system, namely the “3K-Camera”, which was recently developed at the German Aerospace Center (DLR). This wide angle frame sensor system consists mainly of three non-metric off-the-shelf Canon EOS 1Ds Mark II digital cameras (one nadir, and two side looking) being able to take 16.7 megapixel color images with a frame rate of up to 3 fps. This makes the camera type well suited for traffic monitoring applications. With an oblique angle of 35° for the side looking cameras the coverage of the camera system is up to 8 km /1.5 km across track/in flight direction (at a flight height of 3000 m with a ground sampling distance of 45 cm). With the digital images of this sensor being time synchronized to a GPS/IMU unit, direct georeferencing of the obtained aerial images is performed. In order to perform near real-time processing of images to derive traffic information five Industrial PCs are onboard the aircraft as well as a downlink antenna for sending traffic data extracted from aerial images directly to the ground. With a ground station receiving the traffic data from the aircraft and sending it to an Internet Traffic Portal we are able to distribute actual traffic data to security organizations and authorities in near real-time. For traffic monitoring, the camera system is in a recording mode called “burst mode”. In this mode, the camera takes a series of four or five exposures with a frame rate of 3 fps, and then it pauses for several seconds. During this pause, the aircraft moves significantly over ground. Then, with an overlap of about 10 % to 20 % to the first exposure “burst”, the second exposure sequence is started. Continuing this periodical shift between exposure sequences and brakes, we are able to perform an area-wide traffic monitoring without producing an overwhelming amount of data. However, this special image recording mode requires new methods and algorithms for automated traffic monitoring that are quite different from methods used for vehicle tracking based on continuous image time series with high frame rates or even video sequences. Our strategy for traffic monitoring from this exposures obtained in “burst mode” is to start with a road extraction limiting the search area for vehicles and then to perform a vehicle detection only in the first image of an image sequence. Then detected vehicles are tracked over the following images within one image “burst”. Inputs of the processing chain for automatic traffic monitoring are georeferenced optical images from the 3K-Camera. The processing chain is divided in roadside detection, roadside identification, vehicle detection, and vehicle tracking. Road detection is performed in order to constrain the search area for vehicles to the road network. For this road detection, a priori information of the rough location of the roads in terms of GIS road axes obtained from a road database is used. This information limits the search area for the roadsides. For this purpose, a buffer region around each road database segment is taken from the georeferenced image. For further road extraction, two different techniques can be used alternatively. In the first method, roadside features are found by using an edge detector based on ISEF filtering, selecting the steepest edge, which is normally the edge between the tarry roads and the vegetation. The second method extracts roads by searching the roadside markings on dynamical threshold images produced from the original images by using a line detector. The first method works quite well in rural areas, with a clear edge between road and vegetation, whereas the second method has its advantages in more urban scenarios. Then in a next processing step, detected potential roadside edges or lines are connected, corrected for gaps and smoothed for bumps by a roadside identification algorithm. For the roadside identification, the detected edges or lines are compared with the road database segments considering direction and road width. The area within the detected roadsides is input of the following vehicle detection. For the vehicle detection, features are extracted by a Canny edge detector. An edge histogram with all detected edges is produced and thresholds are adapted in order to separate edges due to vehicles from edges obtained from the road structure. Open shapes of potential vehicle edges are closed. Then, vehicles are extracted based on the shape of the edges closed in the previous processing step. For that the area, the orientation, the distance to the next feature, and the quality of the underlying road detection is examined. By applying vehicle detections to an image sequence, as delivered by the 3KCamera system, we are able to track vehicle movements in consecutive images. Vehicle tracking by template matching calculating the normalized corresponding author cross correlation is performed on consecutive image pairs and is based on the vehicle detection in the first image of an image sequence. For each vehicle detected in the first image, a template image is produced. Then, a search area for each detected vehicle is defined in the second image of the exposure sequence. Within this search area the normalized cross correlation between the template image and the second image is calculated for all 3 channels of the RGB images. The calculated correlation value gives a score for a possible hit. With this method we are able to track vehicles over a whole image sequence (burst), whereas a special evaluation algorithm that validates matches in velocity space reduces false alarms in vehicle detections and mismatched vehicles in template matching. We obtain vehicle detections with a correctness of up to 79 % at a completeness of 68 % on motorways at a flight height of 1000 m. In complex scenes like a city ring road we can proof that car detection delivers respectable results with a completeness of 65 % and a correctness of 75 %. However, with the evaluation of matches in tracking algorithms we can enhance correctness to a value of about 90%. With computing times of less than 1 minute per square kilometer for a traffic analysis on motorways and main roads we are able to provide traffic data at high actuality independent from any stationed infrastructure. This makes this system well suited for deployments in case of disasters and mass events on demand. Permanent operations in metropolitan areas may be possible on future unmanned aircrafts with our approach.