I/O Efficient Implementation of MapReduce

MapReduce is a programming model and an associated implementation used by Google for processing their massive data sets. It has a simple yet powerful interface that is amenable to a broad variety of problems. Since 2003, when the MapReduce framework was first created, more than ten thousand distinct programs have been implemented under this model. A large number of MapReduce tasks are now running on Googles clusters at any minute, processing huge amounts of data and gathering lots of useful information. For instance, in the single month of September 2007, more than 2 million MapReduce jobs have been completed, processing over 400,000 TB of input data [2]. The success of MapReduce stems from the fact that it is very easy for the programmer to write a simple program and run it effi-ciently on a thousand machines, greatly improving the engineers productivity. In this project, (since we dont have a thousand machines,) we will study the problem of how to implement the MapReduce interface efficiently on a single machine. Since the data size could be much larger than memory, I/O-efficient techniques that you have learned from class will be useful (and required!).