Hybrid Warm Water Direct Cooling Solution Implementation in CS300-LC

In this white paper, the warm water direct liquid cooling solution implementation in the CCS cluster solution is described. First, literature review of a range of direct liquid cooling solutions implemented in various competing vendors and their pros and cons are briefly presented. Design choices of datacenter cooling water distribution mechanisms (single loop vs primary/secondary loops), types of cooling surfaces (sandwiched single milled metal plate vs individual heat sinks) and types of cooling fluids are reviewed. Next, business and technical constraints of the CCS cluster solution with respect to warm water cooling solution are reviewed. The bulk of the paper explains the implementation details of the hybrid direct liquid cooling solution within the CCS constraints of adhering to open standards and sensitivity to system TCO. The CS300-LC direct liquid cooling architecture implements a dual loop system isolating the high pressure datacenter loop from a completely sealed secondary IT loop via in-rack heat exchanger. Deionized water-glycol mixture is circulated to blade sub-racks through chassis manifolds and through drip free quick connectors to individual blades. Redundant low-power pumps circulate secondary fluid through CPU, Memory and accelerator heat sinks at very low pressure and flow rates. A remotely accessible PLC device monitors and manages the cooling system at the rack level. System design is such that compute blades are hot swappable and CPU, memory and individual pumps are field replaceable. In the hybrid cooling system, sub-rack level cooling fans run at lower speeds to extract residual heat thereby substantially reducing the fan power and noise. In the Cray CCS Lab, a prototype CS300-LC system is instrumented to measure temperatures, flow rates of liquid and air at inlets and outlets. Depending on the system configuration (CPU, memory and accelerators in a node), 60% to 85% of the heat is removed to warm water for a wide range of datacenter inlet water temperatures, and the rest of the heat is rejected to air or indirect cooling systems, hence the term hybrid. In addition, in collaboration between Cray Mississippi State University (MSU), additional detailed measurements will be conducted in a customer datacenter environment to understand energy efficiency and PUE. Additional details of resiliency, remote monitoring and management of the hybrid cooling system are described. The chassis level leak detection mechanism is described. How the CS300-LC cooling system reacts to rapidly changing application workloads and hence cooling loads is reported. The paper is concluded by exploring potential future work of making CS300-LC close to 100% warm water cooled and enhancements to Software tools needed for fine grained, tightly coupled management of the cooling system. Keywords-CS300-LC, direct coolimg, warm water, Xeon Phi, MIC, performance metrics