Scalable Reliable SD Erlang Design

This technical report presents the design of Scalable Distributed (SD) Erlang: a set of language-level changes that aims to enable Distributed Erlang to scale for server applications on commodity hardware with at most 105 cores. We cover a number of aspects, specifically anticipated architecture, anticipated failures, scalable data structures, and scalable computation. Other two components that guided us in the design of SD Erlang are design principles and typical Erlang applications. The design principles summarise the type of modifications we aim to allow Erlang scalability. Erlang exemplars help us to identify the main Erlang scalability issues and hypothetically validate the SD Erlang design. We start by giving an overview of hardware architectures to get an idea of the type of typical hardware architectures we may expect in the next 4-5 years. We expect it to be a NUMA architecture where cores are grouped in modules of 32-64 cores, and each host has 4-6 such modules. A server will consist of ≈ 100 hosts, and a cloud will consist of 2-5 servers (Chapter 3). Then we analyse failures that may happen during SD Erlang program execution. The anticipated failures have an impact on the design decisions we take for SD Erlang (Chapter 4). To scale a language must provide scalable in-memory data structures, scalable persistent data structures, and a scalable computation model. We analyse ETS tables as the main Erlang in-memory data structures. The results show that ETS tables may have scalability problems, however, it is too soon to draw any conclusions as there were no yet real experiments with SD Erlang. As ETS tables are a part of Erlang Virtual Machine (VM) they are out of scope of the current research, and will be dealt by other partners of the RELEASE project (Section 6.1). For the persistent data structures we analyse Mnesia, CoachDB, Casandra, and Riak. The analysis of the database properties shows that such DataBase Management Systems (DBMSs) as Riak and Casandra will be able to provide essential scalability (Section 6.2). SD Erlang’s scalable computation model has two parts. The first part answers the question ‘How to scale a network of Erlang nodes?’. Currently, Distributed Erlang has a fully connected network of nodes with transitive connections. We propose to change that and add scalable groups (s groups). In s groups nodes have transitive connections with nodes of the same s group and non-transitive connections with other nodes. The idea of s groups is similar to Distributed Erlang hidden global groups. S groups differ from global groups in that nodes can belong to a multiple number of s groups or do not belong to an s group at all, and information about nodes is not global, i.e. in s groups nodes collect information about nodes of the same s group but they do not share this information with nodes of other s groups (Section 5.1). The second part of the SD Erlang design answers the question ‘How to manage a scaled number of Erlang nodes?’. As the number of Erlang nodes scale it is hard for a programmer to know the exact position of all nodes. Therefore we propose a semi-explicit placement. Currently, when a process is spawned the target node must be identified explicitly. In the semi-explicit placement the target node will be chosen by a decision making process using restrictions specified by the programmer, e.g. s groups, or types of s groups, or communicating distances on which the target node is located from the initial node (Section 5.2). Finally, we provide an overview of typical Erlang applications and summarise their requirements to the SD Erlang (Chapter 7). We also discuss Erlang VM implications and outline our future work (Chapter 8).