Computing Parametric Motion Fields

Normal Flow Let~ı and ~ be the unit vectors in the x and y directions, respectively; δ~r =~ıδx+~δy is the projected displacement field at the point ~r = x~ı + y~. If we choose a unit direction vector ~nr = nx~ı + ny~ at the image point ~r and call it the normal direction, then the normal displacement field at ~r is δ~rn = (δ~r ·~nr)~nr = (nxδx+nyδy)~nr. The normal direction ~nr can be chosen in various ways; the usual choice is the direction of the image intensity gradient ~nr = ∇I/‖∇I‖. Note that the normal displacement field along an edge is orthogonal to the edge direction. Thus, if at time t we observe an edge element at position ~r, the apparent position of that edge element at time t + ∆t will be ~r + ∆tδ~rn. This is a consequence of the well-known aperture problem. We base our method of estimating the normal displacement field on this observation. For an image frame (say collected at time t) we find edges using an implementation of the Canny edge detector. For each edge element, say at ~r, we resample the image locally to obtain a small window with its rows parallel to the image gradient direction ~nr = ∇I/‖∇I‖. For the next image frame (collected at time t0 + ∆t) we create a larger window, typically twice as large as the maximum expected value of the magnitude of the normal displacement field. We then slide the first (smaller) window along the second (larger) window and compute the difference between the image intensities. The zero of the resulting function is at distance un from the origin of the second window; note that the image gradient in the second window at the positions close to un must be positive. Our estimate of the normal displacement field is then −un, and we call it the normal flow. A real color image sequence used in our experiments is shown in Fig. 1. The corresponding normal flow is shown in Fig. 2.