Efficient Matrix Multiplication in Hadoop

In a typical MapReduce job, each map task processing one piece of the input file. If two input matrices are stored in separate HDFS files, one map task would not be able to access the two input matrices at the same time. To deal with this problem, we propose a efficient matrix multiplication in Hadoop. For dense matrices, we use plain row major order to store the matrices on HDFS; For sparse matrices, we use the row-major-like strategy. So, a mapper can get the rows and columns by only scannig through a consecutive part of a file. We modify the Hadoop MapReduce input format, add two file paths to the two input matrices and store the input matrices in row major order. With the new file split structure, all data are distributed properly to the mappers. Finally, we propose a user feedback method to avoid the overheads of starting multiple map waves. A number of comparative experiments are conducted, the result show that our method observably improve the performance of dense matrix multiplication in MapReduce.