The Design and Implementation of a Fault-Tolerant Cluster Manager

Cluster management middleware schedules tasks on a cluster, controls access to shared resources, provides for task submission and monitoring, and coordinates the cluster’s fault tolerance mechanisms. Thus, reliable continuous operation of the management middleware is a prerequisite to the reliable operation of the cluster. Hence, the management middleware should tolerate a wide class of faults with minimal interruptions to management operations. This paper describes design considerations and implementation details of cluster mangement middleware for high performance computing in space, where fault rates are significantly higher than for earth-bound systems. We describe key detection, recovery, and reconfiguration mechanisms for different components of the system. The system is based on centralized decision making. Unlike other systems, the decision making capability is protected by active replication and the ability to restore the decision maker to full operational and fault tolerance capabilities following node failure. The management middleware is used to provide the application tasks with an out-of-band signaling capability that can be a key building block for application-level fault tolerance mechanisms. The middleware described has been implemented as part of the UCLA FaultTolerant Cluster Testbed (FTCT) project. Based on measurements of this implementation, we present preliminary evaluation of the overheads incurred by the management middleware.