A Fast, Minimal Memory, Consistent Hash Algorithm

We present jump consistent hash, a fast, minimal memory, consistent hash algorithm that can be expressed in about 5 lines of code. In comparison to the algorithm of Karger et al., jump consistent hash requires no storage, is faster, and does a better job of evenly dividing the key space among the buckets and of evenly dividing the workload when the number of buckets changes. Its main limitation is that the buckets must be numbered sequentially, which makes it more suitable for data storage applications than for distributed web caching. Introduction Karger et al. [ 1 ] introduced the concept of consistent hashing and gave an algorithm to implement it. Consistent hashing specifies a distribution of data among servers in such a way that servers can be added or removed without having to totally reorganize the data. It was originally proposed for web caching on the Internet, in order to address the problem that clients may not be aware of the entire set of cache servers. Since then, consistent hashing has also seen wide use in data storage applications. Here, it addresses the problem of splitting data into a set of shards, where each shard is typically managed by a single server (or a small set of replicas). As the total amount of data changes, we may want to increase or decrease the number of shards. This requires moving data in order to split the data evenly among the new set of shards, and we would like to move as little data as possible while doing so. Assume, for example, that data consisting of key­value pairs is to be split into 10 shards. A simple way to split the data is to compute a hash, h(key), of each key, and store the corresponding key­value pair in shard number h(key) mod 10. But if the amount of data grows, and now needs 12 shards to hold it, the simple approach would now assign each key to shard h(key) mod 12, which is probably not the same as h(key) mod 10; the data would need to be completely rearranged among the shards. But it is only necessary to move 1/6 of the data stored in the 10 shards in order to end up with the data balanced among 12 shards. Consistent hashing provides this. Our jump consistent hash function takes a key and a number of buckets (i.e., shards), and returns one …