Cluster management system design for big data infrastructures

ION OF HETEROGENEITY YARN creates containers on each machine based on the total memory and the number of CPU cores. If there are two machines with different memory size, then they will have different numbers of containers. In other words, unlike Hadoop, YARN takes resource heterogeneity into account, in the case of memory. However, YARN still does not consider heterogeneity in other resource characteristics, such as CPU speed, IO and network bandwidth. For example, let’s assume that two machines have the same memory and the same number of cores, but the CPU speeds might differ significantly. In that case, running the same number of containers on both the machines might not be optimal. FIXED CONTAINER SIZE YARN creates containers of a fixed size, and the size is configured by administrators while initiating the clusters. In the current implementation of YARN, the container size cannot be changed while running applications. Each container runs one task, therefore, a resource provided by one container can only be used by one container. However, tasks belonging to different applications may have different resource requirements. Creating 2.6. YARN LIMITATIONS