Notes on Motion Estimation 1 3d Velocity and Image Velocity

There are a great variety of applications that depend on analyzing the motion in image sequences. These include motion detection for surveillance, image sequence data compression (MPEG), image understanding (motion-based segmentation, depth/structure from motion), obstacle avoidance, image registration and compositing. The rst step in processing image sequences is typically image velocity estimation. The result is called the optical ow eld, a collection of two-dimensional velocity vectors, one for each small region (potentially, one for each pixel) of the image. Image velocities can be measured using correlation or block-matching (for example, see Anandan, 1989) in which each small patch of the image at one time is compared with nearby patches in the next frame. Feature extraction and matching is another way to measure the ow eld (for reviews of feature tracking methods see Barron, 1984 or Aggarwal and Nandhakumar, 1988). Gradient-based algorithms are a third approach to measuring ow elds (for example, This handout concentrates on the lter-based and gradient-based methods. Emphasis is placed on the importance of multiscale, coarse-tone , reenement of the velocity estimates. Motion occurs in many applications. Here we concentrate on natural image sequences of 3d scenes in which objects and the camera may be moving. In particular we are interested in measuring the optical ow that arises due to the relative motion between the object and the camera. To begin, consider a point on the surface of an object. We will represent 3d surface points as position vectors ~ X = (X; Y; Z) T relative to a viewer-centered coordinate frame as depicted in Fig. 1. When the camera moves, or when the object moves, this point moves along a 3d path ~ X(t) = (X(t); Y (t); Z(t)) T , relative to our viewer-centered coordinate frame. The instantaneous 3d velocity of the point is the derivative of this path with respect to time: ~ V = d ~ X(t) dt = dX dt ; dY dt ; dZ dt ! T (1) We are interested in the image locations to which the 3d point projects as a function of time. Under perspective projection, the point ~ X projects to the image point (x; y) T given 1