Tracking millions of humans in crowded space in crowded spaces

As Aristotle noted, “man is by nature a social animal”. We do not live in isolation. On a daily basis, thousands of individuals walk in terminals, malls or city centers. They consciously or unconsciously interact with each other. They make decisions on where to go, and how to get to their destination. Their mobility is often influenced by their surrounding. Understanding human social dynamics plays a central role in the design of safer and smarter spaces. It enables the development of ambient intelligence, i.e., spaces that are sensitive and responsive to human behavior. For instance, many sites such as train terminals were constructed several years ago to serve an estimated tra c demand. However, this estimated demand is greatly exceeded by forecasted tra c within a span of one decade. Sensing how individuals move through these large spaces provides insights needed to modify the space or design new ones to accommodate increased tra c. This enables reduced congestion and smooth flow of people. In this chapter, we present the computer vision techniques behind understanding the behavior of more than hundred million individuals in crowded urban spaces. We cover the full spectrum of an intelligent system that detects and tracks humans in high density crowds using a camera network. To the best of our knowledge, we have deployed one of the largest networks of cameras (more than hundred cameras per site) to capture the trajectories of pedestrians in crowded train terminals over the course of two years. At any given time, up to a thousand pedestrians need to be tracked simultaneously (see Fig. 1). The captured dataset is publicly available to enable various research communities, from psychology to computer vision, to dive into a large-scale analysis of human mobility in crowded environments. In the remaining of the chapter, we will share all the technical details that lead to successfully analyze millions of individuals. While computer vision has made great progress in detecting humans in isolation [1, 2, 3, 4], tracking people in high density crowds is very challenging. Individuals highly occlude each other and their motion behavior is not independent. We present detailed insights on how to address these challenges with sparsity promoting priors, and discrete combinatorial optimization that models social interactions. Understanding the behavior of pedestrians using a network of cameras is comprised of the following three steps: (i) Human detection in 3D space, (ii) Tracklet generation, and