A Dynamic Deduplication Approach for Big Data Storage

As data is increasing every day, so it is very challenging task to manage storage devices for this explosive growth of digital data. Data reduction has become very crucial problem. Deduplication approach plays a vital role to remove redundancy in large scale cluster computing storage. As a result, deduplication provides better storage utilization by eliminating redundant copies of data and saving only one copy of data in storage devices. In this paper we propose a dynamic deduplication approach with higher efficiency, in which the data files are distributed across multiple tightly coupled computers shared by multiple users. As an effective data elimination approach it exploits data redundancy. Data deduplication first divides large data objects into smaller parts called chunks and represent them by their unique hash values using MD5 or SHA1 to identify duplicate data. The experimental results with real big data show that represented Deduplication approach improves DER (data elimination ratio) and gains storage space. KeywordsDeduplication, Whole file chunking, Content based chunking, Fixed size chunking, Deduplication ratio, Deduplication gain