SolidStream: NGS Encoding Utility with High Speed and High Efficiency Considering Transmission Bandwidth of Cloud Environments

Cloud computing implemented in conjunction with next generation communication technologies has brought about many changes in the field that utilizes enormous computing power. Especially in the field of biomedical sciences, the computing environment allows for quick analysis of next-generation sequencing (NGS) data by providing flexible and nearly unlimited computing resources. In cases where data analysis requires huge computing power, large amounts of data should transmit from a local cluster to the cloud via the next generation communication network. However, the limited bandwidth in the network can cause slow transmission speeds and connection delays. These limitations may be a serious obstacle for efficient data analysis. In order to resolve the obstacle, we propose SolidStream, a method that improves the transmission of NGS data to the cloud. The proposed SolidStream adopts a strategy of simultaneously encoding and transmitting NGS data. In SolidStream, NGS data is encoded in blocks, and each block is linked immediately to the transmission stream. Furthermore, SolidStream manages the encode stream and the transmission bandwidth using a linear buffer to improve the throughput. We evaluate the performance of SolidSteam for NGS data encoding and transfer against that of existing algorithms. When compared to gzip compression, SolidStream reduces the time needed for compression and transmission of NGS data to the cloud by a factor of 4. When compared to an NGS compression method, SolidStream reduces the time by a factor of 10. The results of the evaluation indicate that SolidStream enables efficient analysis NGS data in the cloud.