Cooling the world’s largest supercomputers

Direct liquid cooling is a highly energy-efficient solution for cooling supercomputers capable of performing millions or billions of billions of calculations per second. Here are two examples in the United States and France.
Supercomputers, capable of processing more than 1015 (1 million billion) operations per second are subject to competition for ever-increasing power, especially between China, the United States, Europe and Japan. Indeed, without these calculation tools, whole swathes of contemporary science are beyond the reach of researchers: simulating the future climate (or tomorrow's weather), designing a spacecraft or an aircraft without going through dozens of expensive tests, studying the thermodynamics of a nuclear reactor or exploring the vastness of data collected on molecular biology and genetics, using the data streams sent by Earth observation satellites etc.(1)

The problem that arises is the cooling of these ever more powerful supercomputers. Here are two examples of these new supercomputers in the United States and France and the solutions adopted for cooling them.

In France, Atos announced in November 2019 a four-year 42 million-euro contract with French national meteorological service, Météo-France, to supply two supercomputers based on its latest BullSequana XH2000 technology. The new systems will multiply Météo-France’s computing power by more than 5, compared to its current solution, thus enabling it to achieve several scientific breakthroughs in weather forecast. It will be able to improve forecasts and announce further in advance unexpected small-scale, high-impact meteorological phenomena (such as intense rainfall, gales and the risk of hail) and refine its studies on the impact of climate change. Each new supercomputer is capable of processing more than 10 million billion operations per second and is amongst the most powerful meteorological supercomputers in the world. (2)

The BullSequana XH2000 boasts optimum energy-efficiency with its highly efficient water-cooled patented Direct Liquid Cooling solution, which minimizes energy consumption by using warm water up to 40°C. (2) (3)

The first supercomputer will be installed at the Centre National de Calcul de Météo-France in Toulouse from November and made available for aptitude and regular service verification phases from January 2020. The second will be installed at Espace Clément Ader (Toulouse Montaudran) from May 2020 and will enter the test phase from July 2020. These two computers will be used in the second half of 2020 for operational weather forecasting and research in atmospheric, ocean and climate sciences. (2)

In the USA, the Department of Energy announced in August 2019 that Cray (a Hewlett Packard Enterprise Company) will build the National Nuclear Security Administration (NNSA) first exascale supercomputer, “El Capitan”. To be hosted at Lawrence Livermore National Laboratory in California, El Capitan will have a peak performance of more than 1.5 exaflops (1.5×1018 or 1.5 billion billion calculations per second) and an anticipated delivery in late 2023. The total contract award is valued at USD 600 million. Featuring advanced capabilities for modeling, simulation and artificial intelligence, El Capitan is projected to run national nuclear security applications. (4)

This supercomputer will draw between 25 and 40 megawatts of power to keep the hardware humming along. That’s two to four times what the largest systems are using today. As a result of the huge thermal loads that will need to be dealt with, it will use direct liquid cooling of the board componentry to draw off excess heat.

The cooling infrastructure wrapped around these exascale supercomputers will be supplied by Motivair Cooling Solutions, which develops since 2015 Exascale-Class Coolant Distribution Units (CDUs) that supply cool water in an isolated loop to the computer nodes. The CDU’s heat exchanger transfers the heat removed from the system to a secondary loop connected to the building’s water supply, which can be a chiller, a cooling tower, or even a natural water source. (5) 

(1) https://www.lemonde.fr/blog/huet/2019/08/21/les-usa-relancent-la-course-aux-super-ordinateurs/ 
(2) https://insidehpc.com/2019/11/meteo-france-to-boost-weather-forecasting-capabilities-with-atos-supercomputer/ 
(3) https://systematic-paris-region.org/fr/actualite/meteo-france-evolue-avec-bullx-un-supercalculateur-issu-de-la-technologie-de-bull/ 
(4) https://insidehpc.com/2019/08/cray-to-build-el-capitan-exascale-supercomputer-at-llnl/ 
(5) https://www.nextplatform.com/2019/11/26/exascale-density-pushes-the-boundaries-of-cooling/