DiskReduce: RAIDing the Cloud

Data-Intensive Scalable Computing (DISC) file systems such as HDFS employs replication for reliability, typically delivering users with only about a third of the storage capacity of the raw disks. In this project, we investigate DiskReduce, a framework for integrating RAID into these replicated storage systems to lower storage capacity overhead, for example, from 200% to 25% when triplicated data is dynamically replaced with 8+2 RAID 6 encoding. We gathered usage data from large HDFS DISC systems and find that DISC files are huge relative to traditional and HPC file systems, but because DISC blocks are also huge, perfile RAID wastes significant capacity. We chose to encode blocks across files. We also studied the implication of reading RAIDed data to MapReduce job performance. We measured read performance benefits from replication that will be lost with erasure encoding. We find that triplicated files can be read at higher bandwidth than single-copy files as expected, but this advantage is perhaps smaller than expected, and is absent in many cases.