Real-Time Omnidirectional and Panoramic Stereo

Traditional stereo systems have a small field of view which limits their usefulness for certain applications. By imaging curved mirrors, the field of view of conventional cameras can be enhanced. In this paper, we present the design of a compact panoramic stereo camera which uses parabolic mirrors and is capable of producing 360 panoramic depth maps at the rate of several frames per second. Video surveillance, autonomous navigation, and site modeling are some of the applications which will benefit from such a sensor. 1 Motivation and Background Work Automated video surveillance systems need the ability to detect and track movements over a large 3-D space. Similarly, autonomous navigation systems must perceive and avoid obstacles which may not be in the field of view of conventional cameras. Site modeling, the recovery of 3-D structure of a large scene, is another application which typically requires range data over a large field of view. In these applications, multiple cameras, moving parts or active sensors are often used to compensate for the small field of view of traditional stereo systems. We have designed a stereo system which uses a combination of lenses and curved mirrors (known as a catadioptric system) to capture a wide field of view without moving parts. Previously, several researchers have used panoramic images to compute depth maps. Ishiguro et al. [1992] used panoramic images obtained from a rotating camera to compute depth. Similar rotating systems have been proposed by [Murray, 1995], [McMillan and Bishop, 1995], [Benosman et al., 1996] and [Kang and Szeliski, 1997]. The disadvantage of such systems is that they require mechanical scanning and are thus not practical for real-time applications such as surveillance and navigation. Panoramic images can be acquired in real-time by imaging curved, mirrored surfaces. One of the first uses of This work was supported in part by DARPA’s VSAM Image Understanding Program, under ONR contract N00014-97-1-0553. mirrors for stereo was in [Nayar, 1988], where Nayar suggested a wide field of view stereo system consisting of a conventional camera pointed at two specular spheres. A similar system using two convex mirrors, one placed on top of the other, was proposed by Southwell et al. [Southwell et al., 1996]. However, in both these systems the projection of the scene produced by the curved mirrors is not from a single viewpoint. Violation of the “single viewpoint assumption” implies that the pinhole camera model can not be used, thus making calibration and correspondence a more difficult task. In [Nayar and Baker, 1997], Nayar and Baker derived the class of catadioptric cameras which do produce a single view point. Later, Nene and Nayar [1998] presented several different stereo configurations using planar, parabolic, elliptic, and hyperbolic mirrors which ensure a single viewpoint. A catadioptric stereo system using hyperbolic mirrors was implemented by Chaen et al. [1997]. In this paper, we present the design of a compact panoramic stereo sensor using parabolic mirrors, which is capable of producing panoramic depth maps in realtime. 2 Panoramic Stereo Camera Figure 1 is a diagram of our compact panoramic stereo sensor, which is composed of two omnidirectional cameras, aligned vertically (coaxial). Each omnidirectional camera consists of a conventional camera, telecentric optics, and a parabolic mirror. Telecentric optics approximates orthographic projection which, together with a parabolic mirror, form a wide-angle imaging system with a single center of projection located at the focal point of the parabola [Nayar, 1997]. The reasons for aligning the sensors vertically are twofold. First, by placing one sensor above the other the singularities (where depth cannot be computed near the epipoles) occur where the cameras occlude themselves. Second, the panoramic images have corresponding, parallel epipolar lines, thus lending themselves to real-time stereo processing. A single camera could be used by placing one smaller