Benchmarking and Performance studies of MapReduce / Hadoop Framework on Blue Waters Supercomputer

MapReduce is an emerging and widely used programming model for large-scale data parallel applications that require to process large amount of raw data. There are several implementations of MapReduce framework, among which Apache Hadoop is the most commonly used and open source implementaion. These frameworks are rarely deployed on supercomputers as massive as Blue Waters. We want to evaluate how such massive HPC resource can help solving large-scale data analytics, datamining problems using MapReduce / Hadoop framework. In this paper we present our studies and detailed performance analysis of MapReduce / Hadoop framework on Blue Waters Supercomputer. We have used standard popular MapReduce benchmark suite that represents wide range of MapReduce applications with various computation and data densities. Also, we are planning to use Intel HiBench Haddop Benchamrk Suite in future. We identify few factors that significantly affect the performance of MapReduce / Hadoop and shed light on few alternatives that can improve the overall performance of MapReduce techniques on the system. The results we have obtained strengthen our belief in possibility of using massive specialized superrcomputers to tackle big data problems. We demonstrate the initial performance of the MapReduce / Hadoop framework with encoraging results and we are confident that the massive traditional High Perfromance Computaing resource can be useful in tackling the big-data research challneges and in solving large-scale data analytics, data-mining problems.