Exploiting Remote Memory in InfiniBand Clusters using a High Performance Network Block Device (HPBD)

Traditionally, remote memory accesses in cluster systems are very expensive operations, which perform 20-100 times slower than local memory accesses. Modern RDMA capable networks such as InfiniBand and Quadrics provide low latency of a few microseconds and high bandwidth of up to 10 Gbps. This has made remote memory much closer to the local memory system. Using remote idle memory to enhance local memory hierarchy thus becomes an attractive choice, especially for data intensive applications in cluster environment. In this paper, we take the challenge to design a remote paging system for remote memory utilization in InfiniBand clusters. We present the design and implementation of a high performance networking block device (HPBD), which serves as a swap device of kernel Virtual Memory (VM) system for efficient page transfer to/from remote memory servers. Our experiments show that using HPBD, quick sort performs only 1.45 times slower than local memory system, and up to 21 times faster than local disk. And our design is completely transparent to user applications. To the best of our knowledge, it is the first work of a remote pager design using InfiniBand for remote memory utilization.