Defending against Heap Overflow by Using Randomization in Nested Virtual Clusters

Heap based buffer overflows are a dangerous class of vulnerability. One countermeasure is randomizing the location of heap memory blocks. Existing techniques segregate the address space into clusters, each of which is used exclusively for one block size. This approach requires a large amount of address space reservation, and results in lower location randomization for larger blocks. In this paper, we explore the possibility of using a cluster for 2 or more block sizes. We show that a naive implementation fails because attackers can easily predict the relative location of 2 blocks with 50% probability. To overcome this problem, we design a novel allocator algorithm based on virtual clusters. When the cluster size is large, the randomization of larger blocks improves by 25% compared to existing techniques while the size of the reserved area required decreases by 37.5%.