Distributed Storage of Large-Scale Multidimensional Electroencephalogram Data Using Hadoop and HBase

Huge volumes of data are being accumulated from a variety of sources in engineering and scientific disciplines; this has been referred to as the “Data Avalanche”. Cloud computing infrastructures (such as Amazon Elastic Compute Cloud (EC2)) are specifically designed to combine high compute performance with high performance network capability to meet the needs of data-intensive science. Reliable, scalable, distributed computing is used extensively on the cloud. Apache Hadoop is one such open-source project that provides a distributed file system to create multiple replicas of data blocks and distribute them on compute nodes throughout a cluster to enable reliable and rapid computations. Column-oriented databases built on Hadoop (such as HBase) along with MapReduce programming paradigm allows development of large scale distributed computing applications with ease. In this chapter, benchmarking results on a small in-house Hadoop cluster composed of 29 nodes each with 8-core processors is presented along with a case-study on distributed storage of electroencephalogram (EEG) data. Our results indicate that Haimonti Dutta Center for Computational Learning Systems (CCLS), Columbia University, New York 10115.email: [email protected] Alex Kamil School of General Studies, Columbia University, New York, NY 10027.e-mail: [email protected] Manoj Pooleery Center for Computational Learning Systems (CCLS), Columbia University, New York 10115. e-mail: [email protected] Simha Sethumadhavan Computer Architecture Laboratory, Department of Computer Science, Columbia University, New York, 10115. e-mail: [email protected] John Demme Department of Computer Science, Columbia University, New York, 10115. e-mail: [email protected] ∗ Corresponding Author