# Anticipating Solar Flares

Hugh Hudson  SSL, UC Berkeley, CA USA  School of Physics and Astronomy,
University of Glasgow, UK  Western Kentucky University, Bowling Green KY USA

##  1 Introduction

sec:intro

Solar flares have precursor signatures of several types, which may appear in
coronal and chromospheric observations. Perhaps the most remarkable of these
consist of the pre-flare activations of filaments, which may then erupt; for
example, the Skylab astronauts famously used real-time monitoring of H$\alpha$
images to identify the beginnings of eruptive flares, now known to identify
with coronal mass ejections (CMEs). In soft X-rays, the precursor activity may
show up as a characteristic slow increase ramping up to the “impulsive phase”
characterizing strongly non-thermal phenomena such as particle acceleration
and the energization of CMEs (Kane and Anderson, 1970) and thus the full
development of flare emissions. The subject of pre-flare activity, and of soft
X-ray precursor, has an extensive literature, which this article does not
attempt to review. We do note Panos, Kleint, and Zbinden (2023), who have used
machine learning to follow several observables, and have thereby recognized
clear chromospheric patterns doubtless related to the GOES coronal phenomenon
described below as HOPE (“Hot Onset Precursor Event”).

Recently Hudson et al. (2021) have examined the soft X-ray precursors, finding
them to have characteristic properties and suggesting universality for the
process. Further work has strengthened this conclusion (da Silva et al., 2023;
Battaglia et al., 2023). The HOPE phenomenon therefore has the potential to
become a tool for a flare alert on few-minute time scales. In this paper we
describe a “flare anticipation index” (FAI) based on the standard GOES soft
X-ray observations, with the objective of anticipating flare occurrence far
enough ahead in time to enable campaign-style observational programs a
sufficient warning for observations aimed at impulsive-phase physics. We find
the FAI to be extremely reliable, even for events too weak to be classified in
the standard ABCMX spectrum of NOAA flare reports. This finding strongly
confirms the case for universality of the HOPE phenomenon, even though we do
not yet understand the physics behind it.

In practical terms, the GOES soft X-rays provide a convenient basis for an
FAI, given the vast database available and the near-real-time (latency of a
few minutes) data currently provided by NOAA. This GOES-based approach, though
successfully anticipating soft X-ray events well below NOAA’s C-class, could
lead to other FAI methods that may have better still better sensitivity). Note
that flare anticipation is not the same thing as flare forecasting. The HOPE
appears to be just the earliest recognizable feature of a flare, for which
actual prediction remains a difficult problem.

This article studies the GOES FAI based on a single 3-day sample of the real-
time NOAA X-ray database, which have one-minute cadence (2024-01-02T12:00
through 2024-01-05T12:00). Such a quick sample suffices to justify operational
deployment of this very simple algorithm for assisting with observing
campaigns aimed at flare/CME origins. The Appendix describes how observers can
implement an FAI alert. The rest of this paper explains more about how it
works, but the main purpose here is just to show how to use it.

##  2 Identifying the soft X-ray HOPE

sec:hope

The X-class flare SOL2022-04-20 illustrates the HOPE pattern extremely well.
Figure fig:fai_plot shows the time development in the GOES/XRS soft 1-8 /AA/
channel. The ramp-up at the outset is the HOPE phase, as labeled; the
impulsive phase (labeled HXR) shows when hard X-rays and other nonthermal
effects happen, including ablation (“evaporation”); this merges into arcade
development and draining, and eventually simple cooling and associated field
shrinkage. This particular event displays these phases with unusual clarity.

Figure 1.: The soft X-ray developmental phases of the X-class flare
SOL2022-04-20, showing the prominent HOPE preflare increase. See the text and
Figure fig:jakimiec for further explanation. fig:fai˙plot

We explain the time development with a [T, EM] diagnostic diagram related to
that described by Jakiemiec et al. (1986), who phrased it in terms of theory
as [T, $\sqrt{n_{e}}$] rather than directly in terms of the observables.
Figure fig:jakimiec shows a clear example of such a diagram, along with a
sketch relating its features to the flare development. The [T, EM] variables
come from a direct isothermal fit, via SolarSoft tools (Freeland and Handy,
1998), to the two-channel flux residuals above estimated background levels.
The newly recognized feature underpinning the FAI development is the initial
horizontal branch, during which emission measure grows steadily, sometimes
punctuated by microflares, while the temperature of the increasing mass
remains roughly constant. The analysis in this article confirms the
universality of the soft X-ray horizontal branch as a requirement for flare
development.

Figure 2.: Left, a representative [EM, T] diagnostic displaying the hot onset
effect. Right, descriptions of the directions followed by the diagnostic
point. The striking and useful characteristic feature of the HOPE phenonenon
is the “horizontal branch,” during which the emission measure grows steadily
while the temperature remains approximately constant. These flare data for
SOL2022-04-20 (X2.2) show the constituent parts of the diagnostic diagram
exceptionally clearly, but this pattern is normally present. fig:jakimiec

In the explanatory sketch (right panel of Figure fig:jakimiec) the red arrows
show the result of energy input into coronal plasma. The impulsive phase,
which in the example shown begins at $EM\approx 0.02\times 10^{49}$ cm-3,
initiates the ablation of large amounts of new hot material as “evaporation.”
This motion of the correlation point describes a clockwise loop of the [T,EM]
trajectory. In contrast to this, the initial point already exhibits a
temperature well above that of the quiet Sun or an active region. This does
not represent “heating” in the sense of temperature increase.

##  3 The GOES-based Flare Anticipation Index (FAI)

sec:fai

We can use the appearance of the initial horizontal branch of the [EM,T]
diagram by screening on the values of the timewise motion of the isothermal
fits, [dEM/dT, T]. A first guess at such an FAI used timewise differencing on
the GOES near-realtime data. The FAI algorithm requires 5 parameters, and
Table 1 lists first-guess default values. The unit EM49 for the volumetric
emission measure is the SolarSoft standard value of 1049 cm-3.

tab:fai_params  Parameter Default Significance Integration time 1 min Set by
GOES quicklook data Difference time $\Delta t$ 5 min Initial guess EM
increment 0.005 EM49 Explored in this article Temperature range [7,14] MK
Explored in this article FAI duration 3 min Not explored in this article

Table 1.: GOES FAI Parameters

For a randomly chosen six-hour interval of GOES real-time data, the FAI
algorithm – as operated with the default parameters – generated Figure
fig:fai_realtime. The flare anticipation worked extremely well, with 100% true
positives (all flags preceded GOES 1-8 Å maxima) and at times of a few minutes
prior to the flares’ impulsive phases. Note one small anomaly, however; in the
precursor to the major event (SOL2024-02-25T17:22, M2.1) the EM increased
monotonically as the horizontal branch of the [EM,T] diagnostic evolved, but
with some irregularity. The gaps in the flag sequence in the major flare in
this example turned out to result from the default setting of the high-
temperature limit; adjusting it upwards resolved this issue. The description
in the Appendix suggests [6,20] MK.

Figure 3.: A quick check of the default parameters for a six-hour stretch of
real-time GOES data. The red vertical lines show minutes for which the flag
was set for these values. As can be seen, there were no significant false
positives but some apparent false negatives (see text). The FAI run with a
lower incremental EM parameter readily detects the weak A-class events missed
at the default setting.  fig:fai˙realtime

This article studies a 3-day quicklook data set in detail: 2024-01-02T12:00
through 2024-01-05T12:00. During this time interval NOAA reported 20 flares,
ranging from class C1.2 to M3.8, a period with a relatively high soft X-ray
background level. We list these in Table 2 along with the correspondingt FAI
“anticipation times” (GOES 1-8 Å peak minus time of first FAI flag, with
values 13.4 $\pm\ 6.0$ min).

tab:GOES_times  Date Start Peak End Class Anticipation (minutes) 2-JAN-24
13:42 13:45 13:49 C1.2 6 3-JAN-24 02:54 02:59 03:05 C1.2 6 3-JAN-24 10:00
10:10 10:14 C1.8 16 3-JAN-24 10:14 10:18 10:24 C3.0 8 3-JAN-24 13:29 13:34
13:44 C1.3 8 3-JAN-24 14:42 14:57 15:15 C1.8 24 3-JAN-24 16:13 16:21 16:28
C1.3 13 3-JAN-24 16:49 16:56 17:08 C1.5 10 4-JAN-24 00:13 00:25 00:51 C2.1 15
4-JAN-24 01:08 01:16 01:22 M1.1 11 4-JAN-24 01:22 01:55 02:12 M3.8 27 4-JAN-24
07:19 07:28 07:48 C1.5 11 4-JAN-24 08:55 09:06 09:16 C1.7 14 4-JAN-24 09:16
09:36 09:42 C3.0 15 4-JAN-24 10:20 10:30 10:35 C2.2 12 4-JAN-24 17:20 17:31
17:39 C3.3 12 5-JAN-24 00:32 00:52 01:08 C3.2 15 5-JAN-24 02:43 02:51 02:56
C1.7 9 5-JAN-24 04:18 04:25 04:30 C1.6 11 5-JAN-24 07:55 08:09 08:19 C3.7 24

Table 2.: GOES Time Comparisons

The default values for the five parameters listed in Table 1 worked well for
the test interval. We have explored adjusting the EM increment and show the
resulting event counts in Figure fig:on_stats (left panel). At the default
value the flare numbers greatly exceed the SolarSoft count of 20, which
corresponds to an EM increment of about $0.03\times 10^{49}$ cm-3. Reducing
the FAI threshold for EM increment below its default value reliably returns
many more events than the NOAA classification recognizes.

Figure 4.: Left, numbers of events as a function of the EM increment
parameter. The dotted line shows the SolarSoft flare count for the interval,
and the vertical dashed line shows the default value for EM increment. Right,
a section of time series showing the FAI failure for the energetically
A8-class event SOL2024-01-04T00:39 (see text), plotted on a linear scale with
C1 units (10-6 W/m2). This plot used an EM increment of 0.001 EM49; the solid
lines show the first of each set of consecutive flags. fig:on˙stats

The physical interpretation of the EM increment is the growth of emission
measure per five minutes, basically a rate of change. Some flares,
particularly slow and/or powerful, meet this criterion for many consecutive
1-minute samples. This indicates a monotonic increase in total EM at roughly
constant temperature. The counts in Figure fig:on_stats correspond to the
first sample of any such grouping, thus marking the earliest anticipation time
for a given flare.

The right panel of Figure fig:on_stats illustrates how the FAI may fail for
the weakest events, showing particularly a tiny C2-class peak (background-
subtracted to an A8 level), SOL2024-01-04, not forewarned even at an emission-
measure criterion of $EM_{49}=0.001$ (or lower). This reflects the practical
limit of the FAI algorithm, considering the digital steps of the GOES
telemetry and the sensors’ background fluctuations as well as the background
level.

##  4 Parameter dependences (FAI)

sec:parms

Do the FAI parameters not only anticipate flare occurrence, but also correlate
with its peak flux or other properties? This short sample does not yield
decisive results here, but the decades of archival GOES data will allow future
studies with much greater precision. For the 3-day quicklook data set
discussed in this article there is already a hint of peak-flux prediction
(Figure fig:correlations).

Figure 5.: Correlations between FAI parameters (for trigger EM upper panels,
and Te lower panels) and observables (left panel for flare peak flux, and
right panel for anticipation time). fig:correlations

The M-class flares appear in the upper right corner of the upper left panel of
Figure fig:correlations, consistent with a predictive capability for flare
magnitude. This correlation is well borne out anecdotally by the author in
reference to high M-class and X-class flares not studied in detail here. Also
anecdotally, there may be an onset temperature correlation for major events,
in that the default temperature range [7,14] MK needed to be increased to
[6,20] MK to set FAI flags for the X-class flare SOL2024-02-09. The recipe
detailed in the Appendix uses [6,20] MK for the temperature range.

A future more complete study will need to map probabilities between the set of
five model parameters and additional items of practical interest: how long
must we wait for the flare, and what will its duration be? We already have
established a correlation between emission-measure increment and flare
magnitude.

##  5 Conclusions

sec:concl

This article has briefly described a flare anticipation index (FAI) based on
the standard GOES two-channel solar soft X-ray fluxes. The purpose here is a
practical one, aiming at helping solar observers to predict imminent flare
occurrence. The sample studied here is a 3-day realtime dataset, and the FAI
worked extremely well, with 100% true positives and no false negatives at the
default parameter set. Campaign-style observations benefiting from a few
minutes’ warning of flare occurrence can therefore use the algorithm as-is.
The limitations of the FAI include the latency of the GOES realtime data,
about 4 min, and the whole-Sun nature of the data. A different database
solving these problems will doubtless enable a successor for the present FAI;
ideally the new data would consist of soft X-ray imaging observations.

This data set hints at a predictive capability of the FAI: greater emission-
measure increments in the HOPE phase correspond to more energetic flares. As
noted, this predictive capability may also extend to the HOPE Te values, but a
systematic study of archival data will be needed to quantify this. The GOES
FAI used here runs on IDL under SolarSoft (Freeland and Handy, 1998) and the
working script can be obtained from the<EMAIL_ADDRESS>
Finally, a brief comment on the significance of the uniform FAI success. The
steady growth of emission measure prior to a flare can now be recognized as a
universal property of solar flares on all magnitude scales. This slow
development itself constitutes the initial unstable action of the flaring
plasma, culminating in the main flare instability itself, with the impulsive
phase, particle acceleration, evaporation, and perhaps the CME launch. We note
that for weak events at GOES A and B class, the FAI anticipates the flare
occurrence but often finds a higher onset value of GOES Te than during the
flare that eventually develops.

### Acknowledgments

The author thanks the University of Glasgow for hospitality, and Sarah
Paterson for reading the draft manuscript prior to submission. Andrew Jones
and Tom Woods have also provided valuable discussion.

##  6 Appendix: Implementation

This recipe generates a useful FAI from one-minute samples of near-real-time
GOES soft X-ray data:

  1. i)

Download near-realtime GOES data at
https://services.swpc.noaa.gov/json/goes/primary/xrays-6-hour.json

  2. ii)

Extract the two broadband data values, XRS_A and XRS_B, the fluxes the 0.5–4 Å
and 1–8 Å bands, respectively.

  3. iii)

Form running differences between the latest value and the value $\Delta t$ min
prior, to create two new timeseries.

  4. iv)

Interpret these running-difference flux values in terms of isothermal
temperature and emission measure $[T_{6},EM_{49}]$ via SolarSoft goes_tem.pro
or equivalent. These are estimates of the electron temperature in units of 106
K, and volumetric emission measure in units of 1049 cm-3.

  5. v)

Set a flare flag where $T_{a}<T_{6}<T_{b}$ and $EM_{49}>Y$, where
$[T_{a},T_{b},\mathrm{and}\ Y]$ are free parameters. Nominal values of the
parameters for $\Delta t=5$ min set at [6, 20, 0.1] detect an M-class flare
about 15 min prior to GOES maximum in XRS_B, with no false positives or false
negatives.

Adjustment of the $Y$ parameter detects flares at different magnitudes, and
the user can work out an appropriate interface for interpreting the near-real-
time FAI signature. At the time of writing, the GOES input data from the link
above have an unavoidable 4-5 min latency.

## References

  * Battaglia et al. (2023) Battaglia, A.F., Hudson, H., Warmuth, A., Collier, H., Jeffrey, N.L.S., Caspi, A., Dickson, E.C.M., Saqri, J., Purkhart, S., Veronig, A.M., Harra, L., Krucker, S.: 2023, The existence of hot X-ray onsets in solar flares. A&A 679, A139. DOI. ADS.
  * da Silva et al. (2023) da Silva, D.F., Hui, L., Simões, P.J.A., Valio, A., Costa Joaquim, E.R., Hudson, H.S., Fletcher, L., Hayes, L.A., Hannah, I.G.: 2023, Statistical analysis of the onset temperature of solar flares in 2010-2011. MNRAS. DOI. ADS.
  * Freeland and Handy (1998) Freeland, S.L., Handy, B.N.: 1998, Data Analysis with the SolarSoft System. Sol. Phys. 182, 497\. DOI. ADS.
  * Hudson et al. (2021) Hudson, H.S., Simões, P.J.A., Fletcher, L., Hayes, L.A., Hannah, I.G.: 2021, Hot X-ray onsets of solar flares. MNRAS 501, 1273\. DOI. ADS.
  * Jakiemiec et al. (1986) Jakiemiec, J., Sylwester, B., Sylwester, J., Mewe, R., Peres, G., Serio, S., Schrijver, J.: 1986, Investigation of flare heating based on X-ray observations. Advances in Space Research 6, 237\. DOI. ADS.
  * Kane and Anderson (1970) Kane, S.R., Anderson, K.A.: 1970, Spectral Characteristics of Impulsive Solar-Flare X-Rays $>$ 10 KeV. ApJ 162, 1003\. DOI. ADS.
  * Panos, Kleint, and Zbinden (2023) Panos, B., Kleint, L., Zbinden, J.: 2023, Identifying preflare spectral features using explainable artificial intelligence. A&A 671, A73. DOI. ADS.


# Comparison of Data Center Cooling Solutions

Kawsar Haghshenas, Brian Setz, Yannis Bloch, and Marco Aiello Service
Computing Department, IAAS, University of Stuttgart, Germany.
E-mail<EMAIL_ADDRESS>submitted May 2022.

# The Emergence of Immersion Cooling for (Distributed) Data Centers

Kawsar Haghshenas, Brian Setz, Yannis Bloch, and Marco Aiello Service
Computing Department, IAAS, University of Stuttgart, Germany.
E-mail<EMAIL_ADDRESS>submitted May 2022.

# Hot Air Alternatives, the Emergent Role of Immersion Cooling

Kawsar Haghshenas, Brian Setz, Yannis Bloch, and Marco Aiello Service
Computing Department, IAAS, University of Stuttgart, Germany.
E-mail<EMAIL_ADDRESS>submitted May 2022.

# A Quantitative Evaluation of Immersion Cooling for Data Centers

Kawsar Haghshenas, Brian Setz, Yannis Bloch, and Marco Aiello Service
Computing Department, IAAS, University of Stuttgart, Germany.
E-mail<EMAIL_ADDRESS>submitted May 2022.

# Enough Hot Air:
The Role of Immersion Cooling

Kawsar Haghshenas, Brian Setz, Yannis Bloch, and Marco Aiello Service
Computing Department, IAAS, University of Stuttgart, Germany.
E-mail<EMAIL_ADDRESS>submitted May 2022.

###### Abstract

Air cooling is the traditional solution to chill servers in data centers.
However, the continuous increase in global data center energy consumption
combined with the increase of the racks’ power dissipation calls for the use
of more efficient alternatives. Immersion cooling is one such alternative. In
this paper, we quantitatively examine and compare air cooling and immersion
cooling solutions. The examined characteristics include power usage efficiency
(PUE), computing and power density, cost, and maintenance overheads. A direct
comparison shows a reduction of about 50% in energy consumption and a
reduction of about two-thirds of the occupied space, by using immersion
cooling. In addition, the higher heat capacity of used liquids in immersion
cooling compared to air allows for much higher rack power densities. Moreover,
immersion cooling requires less capital and operational expenditures. However,
challenging maintenance procedures together with the increased number of IT
failures are the main downsides. By selecting immersion cooling, cloud
providers must trade-off the decrease in energy and cost and the increase in
power density with its higher maintenance and reliability concerns. Finally,
we argue that retrofitting an air-cooled data center with immersion cooling
will result in high costs and is generally not recommended.

###### Index Terms:

Data center, Air cooling, Immersion cooling, Power usage efficiency, Power
density, Maintenance.

##  1 Introduction

The scale and the number of data centers are increasing worldwide to meet the
rising demand for IT applications and cloud services. High energy consumption
(operational cost) and environmental effects are among the main concerns,
which both require a great deal of attention. Due to the fact that IT
equipment produces heat under load, they have to be artificially cooled to
ensure availability and reliability. Therefore, the cooling system is one of
the main energy consumers of a data center. Presently, air-cooling is the most
prominent technique used in data centers. Using this technique entails that
about 40% of the total energy consumption is dedicated to cooling [1].

In an air-cooled data center, cold air is circulating through the perforated
tiles up and into the front of the servers and hot air is pushed out by new
cold air coming into the servers. By this method, it is possible to cool
server racks with at most 50kW power density [2]. In 2018, only 10% of
respondents to a survey reported that the power density of some of their racks
was above 40kW [3]. However, growing power-hungry cluster workloads, such as
machine learning ones, produce higher heat densities. These workloads are run
on dozens of GPUs, each having thousands of cores that need to be supplied by
power. The GPUs increase the power density of the rack, requesting more
cooling capacity. Moreover, recent studies show that as the end of Dennard
scaling is reached [4], and transistor sizes approach their practical
limitations, new cooling solutions are needed to keep the traditional
performance improvement trend. Therefore, CPUs and GPUs with higher power
consumption are expected to be manufactured in near future [5, 6, 7], and
consequently, the power density of racks will increase as well.

To address both high energy consumption and low power density problems
associated with air cooling, researchers and cloud providers have started to
explore alternative solutions. Liquid immersion cooling is a viable one that
has attracted attention in the last decade. In immersion cooling, components
are fully immersed into a dielectric fluid that conducts heat and does not
conduct electricity, therefore, the heat of all IT components is fully removed
by liquid, which reduces the Power Usage Efficiency (PUE) of the data center.
The PUE is defined as the ratio between total energy and the energy used for
the IT equipment of a data center.

The PUE of immersion-cooled data centers is close to perfection, about
1.02-1.04 [8, 9, 10, 11, 12] which shows that these centers consume 10-50%
less energy compared to their air-cooled counterparts, for the same amount of
computational load. In addition, compared to the air, common dielectric fluids
have a much higher heat capacity. Therefore, immersion cooling allows for more
computing power in less space. While the maximum power density per rack for an
air-cooled data center is around 50 KW, immersion-cooled counterpart allows
for up to 250 kW per rack [2, 13].

Immersion cooling is certainly an efficient solution in terms of computing
efficiency and power density. In addition, the capital expenditure for
constructing a data center, keeping constant the power density, is lower with
immersion cooling compared to air cooling [14]. But why it is not deployed in
today’s typical data centers? Maintenance and reliability, and specifically
the lack of practical information on these aspects, are the primary concerns
for adapting immersion cooling technologies [15, 16, 17, 18, 19, 20].
Increased number of IT equipment failures, liquid leakage, and liquid
evaporation are the main challenges associated with its maintenance procedure,
which also impose additional operational costs. However, there are several
successful implementations of immersion cooling in data centers. For example,
one of the biggest players in cloud computing, Microsoft, has already
constructed its first immersion-cooled data center [21].

In this paper, we explore quantitatively the trade-offs between air and
immersion cooling technologies and evaluate several aspects of both methods.
For this evaluation, we refer to various references, from research studies to
practical implementations. Additionally, quantitative analyses on data center
efficiency and computing power are presented throughout the paper. This paper
will help data center operators to have a better perspective while selecting
the best solution for their specific application. Our investigations show that
immersion cooling has significant advantages specifically for high power
applications and we expect its adoption to grow. Improving the efficiency of a
big data center by a small percentage would considerably impact total energy
consumption. In addition, the migration from air to liquid cooling could
influence operators of smaller data centers in trusting immersion cooling.
However, we also argue that retrofitting an air-cooled data center with liquid
cooling will result in high costs and is generally not recommended.

The remainder of this paper is organized as follows. The basic concepts of air
cooling and immersion cooling, together with a literature review on improving
data center efficiency, are presented in Section 2. Then, Section 3 contains
an overview of several practical implementations of immersion-cooled data
centers and several companies that offer immersion cooling server systems.
Section 4 presents and analysis and comparison of the two cooling solutions
including in terms of efficiency, density, cost, and maintenance. Finally,
concluding remarks are provided in Section 5.

##  2 Background

Computing hardware consumes electricity, which is dissipated as heat due to
resistances in the electrical circuits. In turn, this heat needs to be
dispersed to ensure the correct functioning of the equipment. In this section,
we provide some background on air cooling and immersion cooling solutions for
data centers. In addition, we present a literature review (of reviews) on data
centers’ cooling solutions.

###  2.1 Air Cooling

Data center cooling was not an issue when server rooms were still small and
computing hardware did not exceed a rack power density of 5kW [22]. These
relatively low quantities of heat was removed by basic air conditioning and
server fans, which was the easiest solution at the time. Being cheap and easy
to implement, air cooling quickly became the standard solution for server
cooling and was further developed.

Figure 1: Air-based cooling techniques for data centers

It is common for server racks in a data center to stand on a raised floor. A
raised floor allows cold air to come from underneath to the front of the
servers. Cold air flows through the perforated tiles up and into the front of
each server. The fans inside the server chassis suck the cold air inside the
servers and then force it over heatsinks covering the processors and other
heat-producing hardware. After absorbing the heat of the server components,
the warm air is pushed out by new cool air coming into the server. Then,
facility fans guide the warm air into a Computer Room Air Conditioning (CRAC)
or a Computer Room Air Handling (CRAH), to be cooled back to a low temperature
to be used again. The CRAC/CRAH is responsible for cooling the air to a
desirable temperature [23]. The cooling solutions include chiller-based,
water-side economizer, and air-side economizer. A chiller uses electricity and
moderately cold water to cool water below the ambient temperature via a
compressor. Water-side economizing adds a cooling tower to lower water
temperature via evaporation and the air-side economizer uses outside air to
reduce temperature. As the next step, the cold air is forced under the raised
floor and the cycle restarts from the beginning. In air-cooled data centers,
servers need to be set up facing each other for hot/cold aisle separation.
This is done to prevent servers from drawing in the exhaust air of other
servers. Fig. 1 provides a simplified schema of the air cooling approach for
racks in data centers. The most basic forms with simple air circulation on the
left and the more advanced ones with, e.g., CRAC cooling on the right.

###  2.2 Immersion Cooling

Immersion cooling is an approach that uses liquid instead of air to remove
heat from computing hardware. In this method, components are fully immersed
into a dielectric fluid. A dielectric fluid conducts heat and does not conduct
electricity at all; instead, it acts as an insulator. Therefore, the heat of
all the components is fully removed by the liquid, eliminating the need for
air cooling and moving parts (i.e. server fans). The commonly used dielectric
fluids have a much higher heat capacity than air [24, 10]. Most liquids used
in immersion cooling are white mineral oil, engineered electric fluids, and
other oils [24, 16]. There are two types of liquid immersion cooling: single-
phase and two-phase, Fig. 2.

Figure 2: Liquid-based cooling techniques for data centers

In single-phase immersion cooling, the fluid stays in liquid form and there is
no phase change. Heat emitting components are cooled by the fluid flowing over
them, and the heated fluid is transported away. The circulation of the fluid
is driven by a pump or by natural convection. In natural convection-driven
systems, the heated fluid floats to the top of the tank because it has a
higher volume than colder fluid. It then flows, due to more fluid rising to
the top, to the side of the tank where it is cooled by a heat exchanger
connected to an external loop. The density of the cooled liquid increases
again and it sinks back to the bottom of the tank by gravity. In a pump-driven
system, the convection is driven by a pump. The pump forces the liquid through
an inlet inside the tank and out through an outlet on the opposite side. The
liquid is then cooled by flowing through a coolant-to-water heat exchanger.

In two-phase immersion cooling, the coolant changes phase whenever it gets in
contact with a heat-producing component. In order to avoid damaging the
components, the boiling point of the coolant has to be lower than the critical
temperature of the components. While evaporating on a hot component, the gas
will float to the top of the tank and will make room for the new colder liquid
coolant. A condenser is located inside the tank above the liquid. Cooling
water is flowing through the condenser to transport the heat away. The coolant
condenses there and will fall back into the tank where it can absorb heat
again [25].

###  2.3 Literature

There is a vast body of literature about increasing the efficiency of data
centers. These works mostly focus on the scheduling of the workload and its
optimization, e.g., [26, 27]. The specific aspect of cooling has also
attracted some attention and several reviews on cooling solutions for data
centers have appeared. One of the first reviews considers the thermal aspects
for air cooling in data centers [28]. Even though some authors were already
advocating a role for immersion cooling [29], the review in [28] only
considers air-based solutions. In [30], a review of data center cooling
technologies is offered, and particular attention is devoted to the
opportunity of recovering the heat. For example, district heating can be fed
from recovered hot air from a data center.

Li et al. offer a detailed thermal analysis of cooling solutions, including
several strategies based on cold plates, waste heat recovery, and heat pipes
[31]. Immersion cooling is not considered as one of the possible solutions.
Kheirabadi and Groulx also focus on the thermal aspects of server cooling [2].
They classify the solutions as air-based (CRAC, CRAH, RDHx, and SCHx) and
liquid-based (indirect: single-phase, two-phase, heat pipe; and direct: pool
boiling, spray cooling, and jet impingement). The results of the review and
technology comparison show a higher efficiency for liquid cooling-based
solutions while maintaining that air-based solutions are a valid option
especially if high efficiency is not a top requirement. The chemical and
physical effects of immersion cooling on IT equipment are investigated in
[32]. The results are particularly relevant to understand the maintenance
needs and life expectancy of the equipment. In addition, a number of
advantages of immersion cooling are identified, such as a decrease in
overheating and temperature swings, elimination of fan failures, elimination
of dust and moisture-related failures, and reduced corrosion.

A review of the literature and product descriptions is presented in [33]. The
authors select two prototypical systems, one based on air cooling and one on
immersion cooling, and provide a comparison of the two based on simulations
especially focusing on scalability aspects. The results indicate a preference
for immersion cooling. A review of thermal management in data centers is
offered in [34]. The authors recognize that open aisle air-based cooling is
the dominant one in practice, though focus on novel techniques and
technologies that have good promise in terms of energy efficiency. Kuncoro et
al. review exclusively immersion cooling solutions and compare the performance
of different types of cooling liquids [35].

Building on the current state of the art, our aim with the present work is to
provide a quantitative comparison of cooling solutions from the computational
point of view and consider not only issues of energy efficiency, but also
computational density, investment, and maintenance costs. To the best of our
knowledge, our paper is the first work that compares air and liquid cooling
for data centers quantitatively and comprehensively.

##  3 Immersion Cooling in Practice

Although air cooling is the dominant solution in data centers, several large
companies are adopting the immersion cooling technology and a number of
startups have appeared offering innovative systems. At the time of writing,
_Microsoft_ has announced its first liquid immersion-cooled data center in
Washington, USA [21]. This data center is used for cloud-based communication
platforms such as Microsoft Teams. _Alibaba_ also uses single-phase immersion
cooling (1PIC) tanks in its data centers. They have shown that using immersion
cooling reduces the total power consumption by 36% and helps to achieve a PUE
of 1.07 [36]. Another example comes from the _BitFury_ group that built a 40+
MW data center that comprises 160 tanks running with a PUE of 1.02 using two-
phase immersion cooling (2PIC) [13].

Furthermore, some companies are offering immersion-cooled server systems.
_Asperitas_ , for example, is a Dutch company located in Amsterdam that offers
complete liquid immersion cooling solutions to its customers. Their server
enclosures operate with single-phase immersion cooling and natural convection,
thus avoiding the use of any mechanical parts. The immersion-cooled servers of
_Asperitas_ are insulated; this is done to capture all heat produced by the
servers in the fluid and to allow for maximum waste heat reutilization. Each
of their AIC24 enclosures can contain up to 48 servers or 288 GPUs with a
footprint of only 60cm x 120cm [37].

An alternative interesting approach was that proposed by the Dutch company
_Nerdalize_. The idea was to offer a distributed data center by displacing
servers in the residential buildings [38]. The immersion cooled servers would
exchange heat with water that was then used for indoor heating and hot tab
water. When the energy savings of (water) heating are taken into account, the
PUE of such a system would be less than 1.0. The company deployed several
servers before bankruptcy in 2018. Interestingly, the company which restarted
_Nerdalize_ , _LeafCloud_ , decided to opt for air cooling, mostly due to its
lower maintenance costs and failure rates.

Another company offering liquid immersion cooling enclosures in various sizes
is _Submer_ [39]. All of _Submer_ ’s products use single-phase immersion
cooling and are ranging from cabinet-sized enclosures called microPod, up to
megaPod, which are set up inside shipping containers. The microPods are
capable of cooling 5kW of components even in direct sunlight which makes them
suitable for companies who want to cool their in-house equipment efficiently.
On the other hand, megaPods are targeted for higher computing powers. They can
be put in almost any place since there is only electricity and network
connection needed. For example, it would be possible to install a megaPod onto
or near a building which then supplies the building with heat. Similar to
_Nerdalize_ and _LeafCloud_ , their products are excellent for waste heat
reutilization.

##  4 Efficiency, Density, and Cost

Cloud providers are generally profit-oriented. This means that data centers
are implemented and optimized for maximum computing output using the least
amount of investment and running costs. While some operators make decisions
with sustainability in mind, the majority of them choose the most cost-
efficient solutions.

In this section, we compare air cooling and immersion cooling solutions on
several dimensions including computing efficiency, computing density, power
density, cost, and maintenance. From the perspective of a data center
operator, there is a clear link between each of these factors and the profit
margin. The efficiency of the cooling method affects each of the mentioned
factors, in turn reinforcing the importance of choosing the appropriate
cooling solution.

###  4.1 Computing Efficiency

TABLE I: PUE values reported in the literature Source |  | Reported PUE
---
air cooling
| Reported PUE
---
immersion cooling
[8] | 1.1 | <1.04
[9] |  | 1.02
[10] |  | 1.03 - 1.17
[11] |  | 1.02 - 1.03
[12] | 1.7 - 2.9 | 1.02 - 1.03
[40] | 1.12, 1.18 |
[41] | 2.2 - 2.61 |
Figure 3: PUE values based on the cooling technology

The most popular metric for measuring the energy efficiency of a data center
is the PUE [42]. PUE, originally proposed in 2006, and standardized as ISO/IEC
30134-2:2016 in 2016. The definition of PUE is given in Equation 1.

$\text{PUE}=\frac{\text{total energy}}{\text{IT energy}}\geq 1$ (1)

Since the IT energy consumption is included in the total energy, the value of
PUE will typically be greater or equal to one. Several studies report on the
PUE of data centers with specific installations of immersion and air cooling
solutions, these are shown in Table I.

For air-cooled data centers, reported PUE values range from 1.1 to 2.9 [8, 12,
41, 40]. Values close to 1.1 can only be achieved by hyper-scale data center
facilities, which are especially optimized for efficient cooling. For example,
the state-of-the-art air-cooled data centers of Google have the PUE of 1.12
[40]. This means that in these data centers, 89% of the total energy is
consumed by IT equipment. However, high efficient data centers with state-of-
the-art design make up for only a small portion of data centers worldwide
[43]. In addition, average air-cooled data centers have a much higher PUE
compared to the most efficient ones. The average PUE has been reported to be
between 2.2 and 2.61 for the data centers in Singapore, Japan, Hong Kong, and
Australia. This shows a significant difference between state-of-the-art hyper-
scale and average data centers [41].

Studies regarding immersion cooling data centers have reported consistently
better results. For these, the PUE falls in the range 1.02 to 1.04 [8, 9, 10,
11, 12]. The PUE of 1.02 seems to be the sweet spot for immersion cooling.
Table I shows how the various studies agree on the fact that PUE values around
1.02 are achievable with immersion cooling. With a PUE of 1.02, about 98% of
the energy consumed by the data center goes to the IT equipment. This is close
to the perfect efficiency. The maximum reported PUE for immersion-cooled data
centers is 1.17 [10]. This case relates to an experiment aiming at maximum
cooling capacity without efficiency in mind. For further illustration, Fig. 3
show a plot of the values from Table I with respect to the type of technology
used for cooling.

While PUE is a useful metric for evaluating the efficiency of a data center
over time, it is not equally suitable for comparing data centers with one
another. Furthermore, it does not give direct information about the computing
efficiency of the data center. In [42], we reviewed metrics for data centers
and explored refinements and alternatives to PUE. In first approximation, PUE
just offers a reasonable indicator of data center efficiency. From a computing
perspective, it is also desirable to have a value representing the computing
power in relation to energy consumption. Computing performance is
traditionally measured by running a set of operations (a benchmark) on a
device and measuring the completion time. An example of such a performance
benchmark is measuring the number of floating point operations per second.

To compare the efficiency of different systems or facilities, one measures the
computing performance of running a benchmark in the average/peak power of the
system. This results in a metric of floating-point operations per second per
watt (FLOPS/W), a metric capable of showing the performance in relation to the
energy consumption.

In a data center, in addition to the servers for running the operations, we
have non-computational equipment such as the cooling system which consumes
energy. Therefore, we consider an alternative metric for computing efficiency
as follows.

$\eta=\frac{\text{computing performance}}{\text{power}}(FLOPS/W)$ (2)

The computing performance and the power of IT equipment are relevant for the
calculation of FLOPS/W; however, they do not change the overall efficiency
while changing the cooling solution. This is because changing the cooling
system does not influence the computing performance and the IT equipment’s
power consumption. Typically, when modifying the cooling systems, the IT
energy remains constant while the total energy changes. To calculate the
improvement in FLOPS/W by changing the cooling method, the knowledge of PUE
and FLOPS/W can be combined for an improved computing efficiency metric,
presented as Equation 3.

$\eta_{\text{data center}}=\frac{\frac{b_{0}}{b_{1}}}{\text{IT
power}}.\frac{1}{\text{PUE}}$ (3)

where $b_{0}$ and $b_{1}$ stand for the number of benchmark operations and
benchmark time, respectively. The inverse of PUE represents the part of the
energy that has been used by IT equipment. For example, a data center with a
PUE of 1.5 uses two-third of its energy for IT equipment. By changing the
cooling method, the value of $b_{0}$, $b_{1}$, and IT power remain the same.
This means that overall computing efficiency depends directly on the fraction
of power used for IT equipment. Furthermore, percentage differences in FLOPS/W
can be calculated having only the PUE.

Knowing this, the expected improvement in the data center’s overall computing
efficiency by switching from air cooling to immersion cooling can be
calculated by comparing 1/PUE values. For example, when migrating from best
practice air cooling (PUE=1.12) to immersion cooling (PUE=1.02), the overall
computing efficiency is increased by 9.8%. While this is a significant
increase that reduces the operational expenditures, migrating from standard
air cooling (PUE=2) to immersion cooling (PUE=1.02) increases the computing
efficiency by 96.1%. Therefore, such a conversion would reduce the energy
consumption by half. The increase and decrease in computing efficiency when
changing cooling method is shown in Table II.

TABLE II: Efficiency improvement by switching cooling technique From To |  | Standard
---
air cooling
| State of the art
---
air cooling
| Immersion
---
cooling
Standard air cooling | - | 78.6% | 96.1%
| State of the art
---
air cooling
-44% | - | 9.8%
Immersion cooling | -49% | -8.9% | -

Let us present a numerical example based on one of the most powerful server
CPUs currently on the market, the AMD EPYC 7742. This processor runs with
225-watt maximum power consumption and achieves around 3.48 TeraFLOPS [44].
Therefore, the processor can calculate 15.5 GigaFLOPS per watt at its peak
performance:

$\eta_{\text{EPYC7742}}=\frac{\text{3.48 TFLOPS}}{\text{225W}}=15.5\text{
(GFLOPS/W)}$ (4)

The central processing unit that calculates the FLOPS, is not the only power
consuming component of a server. Therefore, to calculate the server computing
efficiency, one needs to know the proportion of power consumed by the
processor in relation to the server’s total power. The computing efficiency of
a server is calculated by:

$\eta_{\text{server}}=\eta_{\text{processor}}.p_{\text{processor}}$ (5)

where $p_{\text{processor}}$ stands for the proportion of power consumed by
the processor in relation to the total server consumption. Various works have
reported different power breakdowns for the components of a server. In
accordance with the results presented in [45], we assume that 50% of the
server’s power is consumed by its processor. From Equation 5, the computing
efficiency of the server amounts to 7.73 GFLOPS/W.

To calculate the computing efficiency of the IT equipment, we consider the
power of all IT equipment including storage and network facility. In this way,
the computing efficiency of IT equipment is calculated as:

$\eta_{\text{IT equipment}}=\eta_{\text{server}}.p_{\text{server}}$ (6)

where $p_{\text{server}}$ stands for the proportion of the server’s power in
relation to the total IT power. Indeed, we use $p_{\text{server}}$ in equation
6, to take into account the energy consumption of non-computational IT
equipment (storage and network) in calculating the computing efficiency of IT
equipment. The contribution of the servers, storage, and network facility in
the total power consumption of a data center has been reported in several
works [46, 47]. For our numerical example, we assume the average of reported
values, that is 77%. Therefore, $\eta_{\text{IT equipment}}$ is 5.95 GFLOPS/W.

Finally, the computing efficiency of the data center is calculated by:

$\eta_{\text{data center}}=\eta_{\text{IT equipment}}.\frac{1}{\text{PUE}}$
(7)

By Equation 7, the computing efficiency of a data center cooled by standard
air cooling, state-of-the-art air cooling, and immersion cooling is calculated
as 2.98, 5.32, and 5.84, respectively. The computing efficiency of an
immersion-cooled data center is almost 9% higher than a state-of-the-art air-
cooled hyperscale data center. For the average data center, the switch to
immersion cooling offers even more improvement. Operators can decrease the
power consumption by about 50% without any decrease in computing performance.

###  4.2 Computing and Power Density

TABLE III: Power densities reported in the literature Source |  | Air cooling
---
density
| Immersion cooling
---
density
| Normalized
---
to kW/l
[8] |  | 14 kW/bathtub | 0.045
[48] |  | 400 W/bathtub | 0.23
[9] |  | 250 kW/rack | 0.14
[29] |  | 4 kW/l | 4
[2] | 33-50 kW/rack |  | 0.018-0.028
[3] | 40 kW/rack |  | 0.022

Computing density is defined as the amount of computation that a system can
offer in relation to its size. The metric used to determine the computing
density of a data center is $FLOPS/m^{3}$. This metric considers everything:
from the servers to the power management system and up to the cooling
equipment. Comparing air-cooled and immersion-cooled in these terms also shows
significant differences. The required space for immersion cooling has been
reported to be about one third of traditional air cooling method [8].

The increase in computing density by immersion cooling comes from various
factors. In immersion cooling, there is no space needed in between the servers
and racks for airflow. Tubs can stand right next to each other, and the only
limitation is the accessibility of the service crew. In addition, there is
significantly less cooling equipment needed (only piping), and no raised
floors or air vents are required [8]. According to the literature, with
immersion cooling the density increases at both the rack and facility levels
[29]. Furthermore, many air-cooled data centers trade their computing density
for efficiency and reliability [40]. The cooling units of these centers are
often bigger than they need to be.

Power density is another important factor for data center operators. Power
density must be low in air-cooled data centers, otherwise, either fan speeds
need to be increased or air temperature needs to be lowered, where both
solutions lower the energy efficiency. According to the research, rack-level
power density of air-cooled data centres is about 0.018 kW/l - 0.028 kW/l [2],
while the density of immersion-cooled data centres is between 0.045 kW/l [8]
and 0.23 kW/l [48]. Even more, 4 kW/l has been reported to be possible in
immersion-cooled data centers with enough coolant flow and a compact IT design
[29, 49]. Table III presents the data gathered from various references on the
power density of air-cooled and immersion-cooled data centers. Since power
densities are presented in kW per rack for air-cooled data centers and for
immersion cooling there is no typical rack size, the values need to be
converted to kW/l. This metric is the most meaningful one when the whole
facility is considered.

As the reported numbers show, the power density with immersion cooling is
almost six times higher than air cooling. Therefore, power density can be
another strong motivation for data center operators to consider immersion
cooling for their future projects, specifically for running their high-
performance workloads. Going forward, computing is driving devices into higher
power consumption and CPUs and GPUs capable of consuming more power are
expected to be manufactured. Given this trend, air cooling can not provide
sufficient cooling capacity, while maintaining high power densities.

###  4.3 Cost

An important factor to consider when adopting immersion cooling is capital and
operational expenditures. The capital expenditures include the sealed chassis
for immersing the IT equipment, dielectric fluids, as well as pumps and
tubing. Similar to air-cooled data centers, the operational expenditures
include electricity, staff, network connection fees, as well as supporting and
maintaining the IT equipment.

According to Bunger et al.[14], for air-cooled data centers with a power
density of 10 kW per rack, the capital expenditures are $7.02 per watt. For a
liquid-cooled data center with a similar power density, the cost is reduced
slightly to $6.98 per watt. The benefit of liquid cooling is also that much
higher power densities can be achieved. Assuming a power density of 40 kW per
rack, the expenditures are further reduced to $6.02 per Watt. Furthermore,
when it comes to the operational expenditures, a reduction between 9-20% is
expected with regards to the energy cost due to the absence of fans. This is
in agreement with the results of Neudorfer et al. who state that a 5-10%
reduction of the IT energy consumption is expected due to the avoidance of
internal fans [50]. The need for fans is also absent for the entire facility,
reducing the total energy costs by an amount in the range of 15-25%. Another
point to consider is that the dielectric fluids present in the sealed chassis
can be used virtually indefinitely, assuming some form of filtration is
present.

Day et al. highlight that when building a new data center and optimizing it
for liquid cooling from the ground up, capital expenditure savings can be
achieved over air-cooled data centers [51]. On the contrary, retrofitting an
air-cooled data center with liquid cooling can result in higher costs. An
important exception is retrofitting an air-cooled data center with limited
floor space and power capacity. In this case, the increased power density
possible with liquid cooling, as well as a reduction in energy consumption,
can address both the space and power limitations with one solution.

###  4.4 System Maintenance

The operational costs of a data center include system maintenance. The cooling
method influences the components’ environmental conditions and consequently
affects the number of and time to failures and overall equipment lifetime.

The number of maintenance requests caused by failures has been reported to be
almost 6.6% higher for an immersion-cooled data center compared to a
traditional air-cooled counterpart [15]. The higher number of maintenance
requests associated with immersion cooling impose additional operating costs.
In addition, higher failure rates can degrade the components’ lifetime, but
immersion cooling can compensate for this degradation with lower junction
temperatures [16]. Besides the number of maintenance requests, the maintenance
procedure is more challenging with immersion cooling. In this case, immersed
IT equipment is removed by opening the lid and lifting the equipment out of
the tank. This can lead to liquid evaporation.

According to the work presented in [17] and [18], the maintenance overheads
and reliability concerns, as well as the leaks and spills, are the top
contributors to the low adoption of immersion cooling. Indeed, the enclosure
that the racks are immersed in must be sealed perfectly to avoid liquid
evaporation/losses. Complete enclosure sealing would mitigate the problem but
is not practical. The high number of maintenance requests, compared to the air
cooling system, cause access issues and adds operating costs related to
compensating the fluid losses [19].

The operating costs imposed by the liquid loss have been evaluated in [15].
The cost of the lost liquid divided by the cost of the IT equipment’s energy
usage has been reported to be 4.68 for a specific implementation. This number
shows that the maintenance overhead is a significant drawback associated with
immersion cooling. It can even mitigate the energy efficiency improvement of
immersion cooling. However, it should be noted that the implementation
characteristics including the liquid price and the electricity costs affect
the reported value. In [15], these values are set at 75 $\$/liter$ and 0.09
$\$/kWh$, respectively. One-phase immersion cooling is reported to have fewer
maintenance needs than two-phase immersion cooling [19].

##  5 Conclusions

Air cooling is the traditional solution for addressing the heat dissipation
problem of data centers. High energy consumption and low cooling capacity—and
consequently limited power density—are the main challenges associated with air
cooling. Immersion cooling is emerging as a novel method with many advantages
in terms of efficiency, density, and cost. In the present work, we provide a
quantitative comparison of these two approaches and overview the results
presented in the literature regarding the two alternatives. While most data
centers around the world are relying on air cooling, immersion cooling is
recognized as a potential alternative and several cloud providers have already
constructed their immersion-cooled data centers. The key findings of the
present work are listed next.

First, based on the PUEs reported in the literature, we conclude that the
immersion cooling method consumes less energy and has a higher computing
efficiency. Our analysis shows that a typical immersion-cooled data center
consumes almost 50% less energy compared to its air-cooled counterpart.
Second, immersion-cooled data centers allow for much more compact designs,
more than three times the density of their air-cooled counterparts. In liquid
immersion-cooled data centers, there is no trade-off between efficiency and
density. Conversely, air-cooled data centers can only be either-or. Immersion-
cooled data centers can be placed in ordinary spaces having lower requirements
and the need for additional equipment. Third, while research on immersion
cooling is mostly targeting efficiency, the aspect of power density should not
be overlooked. The increase of computing power in a specific volume is even
more important than the efficiency improvement. The most conservative figures
for immersion cooling are about double density in kW/l on rack-level compared
to the maximum possible in air-cooled data centers. Resources estimating 4kW/l
illustrate what is potentially possible with IT equipment optimized for liquid
immersion cooling. This high-density capability makes immersion cooling the
first-choice solution for running high-performance workloads. In addition, it
allows manufacturing the CPUs and GPUs with higher frequency and higher power
consumption. Fourth, the capital expenditure for an air-cooled data center
with a power density of 10 kW per rack is about 4% higher compared to its
immersion-cooled counterpart. In addition, a 9-20% operational expenditures
reduction is expected with immersion cooling. Fifth and final, the main
downside of immersion cooling is the challenges related to the maintenance and
reliability concerns. This is a reasonable explanation for why even with
significant advantages in efficiency and density, not many companies have
switched their cooling solution to immersion cooling, yet.

In general, immersion cooling appears to be a solution that should be strongly
considered when designing a new data center and we expect its adoption to
grow. At the same time, retrofitting any air-cooled data center with liquid
cooling does not seem convenient in most cases, if feasible at all.
Considering the reliability and maintenance challenges and uncertainties,
immersion cooling might not be useful for small and average power densities.
On the contrary, immersion cooling appears to be the best solution for high
power densities.

## Acknowledgments

The presented research is funded by the Netherlands Organisation for
Scientific Research (NWO) in the framework of the Indo-Dutch Science Industry
Collaboration programme with project NextGenSmart DC (629.002.102).

## References

  * [1] Jiacheng Ni and Xuelian Bai.  A review of air conditioning energy performance in data centers.  Renewable and sustainable energy reviews, 67:625–640, 2017.
  * [2] Ali C Kheirabadi and Dominic Groulx.  Cooling of server electronics: A design review of existing technology.  Applied Thermal Engineering, 105:622–638, 2016.
  * [3] Max Smolaks.  Power density – the real benchmark of a data centre.  https://virtusdatacentres.com/item/389-power-density-the-real-benchmark-of-a-data-centre, 2019\.
  * [4] Robert H Dennard, Fritz H Gaensslen, Hwa-Nien Yu, V Leo Rideout, Ernest Bassous, and Andre R LeBlanc.  Design of ion-implanted mosfet’s with very small physical dimensions.  IEEE Journal of Solid-State Circuits, 9(5):256–268, 1974.
  * [5] Yifan Sun, Nicolas Bohm Agostini, Shi Dong, and David Kaeli.  Summarizing cpu and gpu design trends with product data.  arXiv preprint arXiv:1911.11313, 2019.
  * [6] Yuehong Fan, Casey Winkel, Devdatta Kulkarni, and Wenbin Tian.  Analytical design methodology for liquid based cooling solution for high tdp cpus.  In 2018 17th IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (ITherm), pages 582–586. IEEE, 2018.
  * [7] Intel.  New 2nd generation intel xeon salable processor.  https://www.intel.com/content/dam/www/public/us/en/documents/guides/2nd-gen-xeon-sp-transition-guide-final.pdf, 2017.
  * [8] Morito Matsuoka, Kazuhiro Matsuda, and Hideo Kubo.  Liquid immersion cooling technology with natural convection in data center.  In 2017 IEEE 6th international conference on Cloud networking (CloudNet), pages 1–7. IEEE, 2017.
  * [9] Xudong An, Manish Arora, Wei Huang, William C Brantley, and Joseph L Greathouse.  3d numerical analysis of two-phase immersion cooling for electronic components.  In 2018 17th IEEE intersociety conference on thermal and thermomechanical phenomena in electronic systems (ITherm), pages 609–614. IEEE, 2018.
  * [10] Richard Eiland, John Fernandes, Marianna Vallejo, Dereje Agonafer, and Veerendra Mulay.  Flow rate and inlet temperature considerations for direct immersion of a single server in mineral oil.  In Fourteenth Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (ITherm), pages 706–714. IEEE, 2014.
  * [11] Sriram Chandrasekaran, Joshua Gess, and Sushil Bhavnani.  Effect of subcooling, flow rate and surface characteristics on flow boiling performance of high performance liquid cooled immersion server model.  In 2017 16th IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (ITherm), pages 905–912. IEEE, 2017.
  * [12] Jimil Manojbhai Shah.  Characterizing contamination to expand ASHRAE envelope in airside economization and thermal and reliability in immersion cooling of data centers.  PhD thesis, The University of Texas at Arlington, 2018.
  * [13] Two-phase immersion cooling a revolution in data center efficiency.  3M™ Novec™ Engineered Fluids, 2015.
  * [14] R. Bunger, W. Torell, and V. Avelar.  Capital cost analysis is of immersive liquid-cooled vs. air-cooled large data centers.  Schneider Electric White Paper 282, 2020.
  * [15] Henry Coles and Magnus Herrlin.  Immersion cooling of electronics in dod installations.  Technical Report LBNL-1005666, Berkeley National Laboratories, 2016.
  * [16] Majid Jalili, Ioannis Manousakis, Ínigo Goiri, Pulkit A Misra, Ashish Raniwala, Husam Alissa, Bharath Ramakrishnan, Phillip Tuma, Christian Belady, Marcus Fontoura, et al.  Cost-efficient overclocking in immersion-cooled datacenters.  In Proceedings of the International Symposium on Computer Architecture (ISCA’21), 2021.
  * [17] Herb Villa.  Liquid cooling vs. immersion cooling deployment.  https://blog.rittal.us/liquid-cooling-vs-immersion-cooling-deployment, 2020\.
  * [18] Alibaba.  Immersion cooling for green computing.  https://www.opencompute.org/files/Immersion-Cooling-for-Green-Computing-V1.0.pdf, 2018\.
  * [19] Dhruv Varma.  Two-phase versus single-phase immersion cooling.  https://www.grcooling.com/wp-content/uploads/2020/03/grc-blog-library-tech-comparison-\%E2\%80\%94-two-vs-single-phase-immersion-cooling.pdf, 2019\.
  * [20] Shrinath Ramdas, Pavan Rajmane, Tushar Chauhan, Abel Misrak, and Dereje Agonafer.  Impact of immersion cooling on thermo-mechanical properties of pcb’s and reliability of electronic packages.  In International Electronic Packaging Technical Conference and Exhibition, volume 59322. American Society of Mechanical Engineers, 2019.
  * [21] Sabina Weston.  Microsoft is submerging servers in boiling liquid to prevent teams outages.  https://www.itpro.co.uk/server-storage/datacentr/359129/microsoft-submerges-servers-in-boiling-liquid-toprevent-teams?amp, 2021\.
  * [22] Voices of the Industry.  Data centers feeling the heat! The history and future of data center cooling.  https://datacenterfrontier.com/history-future-data-center-cooling/, 2020\.
  * [23] Hwaiyu Geng.  Data center handbook.  John Wiley & Sons, 2014.
  * [24] Pravin A Shinde, Pratik V Bansode, Satyam Saini, Rajesh Kasukurthy, Tushar Chauhan, Jimil M Shah, and Dereje Agonafer.  Experimental analysis for optimization of thermal performance of a server in single phase immersion cooling.  In International Electronic Packaging Technical Conference and Exhibition, volume 59322. American Society of Mechanical Engineers, 2019.
  * [25] Baris Burak Kanbur, Chenlong Wu, Simiao Fan, Wei Tong, and Fei Duan.  Two-phase liquid-immersion data center cooling system: Experimental performance and thermoeconomic analysis.  International Journal of Refrigeration, 118:290–301, 2020.
  * [26] Jungmin Son, Amir Vahid Dastjerdi, Rodrigo N. Calheiros, and Rajkumar Buyya.  Sla-aware and energy-efficient dynamic overbooking in sdn-based cloud data centers.  IEEE Trans. Sustain. Comput., 2(2):76–89, 2017.
  * [27] Kawsar Haghshenas, Somayye Taheri, Maziar Goudarzi, and Siamak Mohammadi.  Infrastructure aware heterogeneous-workloads scheduling for data center energy cost minimization.  IEEE Transactions on Cloud Computing, 2020.
  * [28] Suhas V Patankar.  Airflow and cooling in a data center.  Journal of Heat transfer, 132(7), 2010.
  * [29] Phillip E Tuma.  The merits of open bath immersion cooling of datacom equipment.  In 2010 26th Annual IEEE Semiconductor Thermal Measurement and Management Symposium (SEMI-THERM), pages 123–131. IEEE, 2010.
  * [30] Khosrow Ebrahimi, Gerard F Jones, and Amy S Fleischer.  A review of data center cooling technology, operating conditions and the corresponding low-grade waste heat recovery opportunities.  Renewable and Sustainable Energy Reviews, 31:622–638, 2014.
  * [31] Zhen Li and Satish G Kandlikar.  Current status and future trends in data-center cooling technologies.  Heat Transfer Engineering, 36(6):523–538, 2015.
  * [32] Jimil M Shah, Richard Eiland, Ashwin Siddarth, and Dereje Agonafer.  Effects of mineral oil immersion cooling on it equipment reliability and reliability enhancements to data center operations.  In 2016 15th IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (ITherm), pages 316–325. IEEE, 2016.
  * [33] Bryony Watson and Vinod Kumar Venkiteswaran.  Universal cooling of data centres: A cfd analysis.  Energy Procedia, 142:2711–2720, 2017.  Proceedings of the 9th International Conference on Applied Energy.
  * [34] Chayan Nadjahi, Hasna Louahlia, and Stéphane Lemasson.  A review of thermal management and innovative cooling strategies for data center.  Sustainable Computing: Informatics and Systems, 19:14–28, 2018\.
  * [35] IW Kuncoro, NA Pambudi, MK Biddinika, I Widiastuti, M Hijriawan, and KM Wibowo.  Immersion cooling as the next technology for data center cooling: A review.  In Journal of Physics: Conference Series, volume 1402, page 044057\. IOP Publishing, 2019.
  * [36] Yangfan Zhong.  A large scale deployment experience using immersion cooling in datacenter.  Alibaba Group: Open Compute Project Summit, 2019.
  * [37] Asperitas.  Immersion cooling solutions for datacentres.  https://www.asperitas.com/, 2021.
  * [38] Yanik Ngoko, Nicolas Saintherant, Christophe Cerin, and Denis Trystram.  How future buildings could redefine distributed computing.  In 2018 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pages 1232–1240. IEEE, 2018.
  * [39] Submer.  Datacenters that make sense.  https://submer.com/, 2021.
  * [40] Rich Miller.  Inside SuperNAP 8: switch’s Tier IV data fortress.  https://www.datacenterknowledge.com/archives/2014/02/11/inside-supernap-8-switchs-tier-iv-data-fortress/, 2014\.
  * [41] Ambrose McNevin.  Apac data center survey reveals high pue figures across the region.  https://www.datacenterdynamics.com/, 2013.
  * [42] V Dinesh Reddy, Brian Setz, G Subrahmanya VRK Rao, GR Gangadharan, and Marco Aiello.  Metrics for sustainable data centers.  IEEE Transactions on Sustainable Computing, 2(3):290–303, 2017\.
  * [43] Nicola Jones.  The information factories.  Nature, 561(7722):163–6, 2018.
  * [44] Tiffany Trader.  AMD Launches Epyc Rome, First 7nm CPU.  https://www.hpcwire.com/2019/08/08/amd-launches-epyc-rome-first-7nm-cpu/, 2019.
  * [45] Sukhpal Singh Gill and Rajkumar Buyya.  A taxonomy and future directions for sustainable cloud computing: 360 degree view.  ACM Computing Surveys (CSUR), 51(5):1–33, 2018.
  * [46] Miyuru Dayarathna, Yonggang Wen, and Rui Fan.  Data center energy consumption modeling: A survey.  IEEE Communications Surveys & Tutorials, 18(1):732–794, 2015.
  * [47] Arman Shehabi, Sarah Smith, Dale Sartor, Richard Brown, Magnus Herrlin, Jonathan Koomey, Eric Masanet, Nathaniel Horner, Inês Azevedo, and William Lintner.  United states data center energy usage report.  2016\.
  * [48] Joshua Gess, Sushil Bhavnani, Bharath Ramakrishnan, R Wayne Johnson, Daniel Harris, Roy Knight, Michael Hamilton, and Charles Ellis.  Investigation and characterization of a high performance, small form factor, modular liquid immersion cooled server model.  In 2014 Semiconductor Thermal Measurement and Management Symposium (SEMI-THERM), pages 8–16. IEEE, 2014.
  * [49] Phil Tuma.  Open bath immersion cooling in data centers: A new twist on an old idea.  Electronics Cooling, page 10, 2010.
  * [50] J. Neudorfer, M. Ellsworth, D. Kulkarni, and H. Zien.  Liquid cooling technology update.  The Green Grid White Paper 70, 2016.
  * [51] T. Day, P. Lin, and R. Bunger.  Liquid cooling technologies for data centers and edge applications.  Schneider Electric White Paper 265, 2019.

|  Kawsar Haghshenas is a post-doctoral researcher in the Smart Energy Systems
(SES) laboratory at the University of Stuttgart. She received her Ph.D. degree
in Computer Science from University of Tehran in 2020, M.Sc. degree in
Computer Engineering from Sharif University of Technology in 2014, and B.Sc.
degree in Computer Engineering from K.N.Toosi University of Technology in
2012. She was visiting the Embedded Systems Laboratory (ESL) at EPFL
University from 2017 to 2018. Her research interests include system level
energy optimization, machine learning and workload scheduling in the area of
data centers and cloud computing.
---|---
|  Brian Setz is a Research Assistant and PhD candidate at the Institute of
Architecture of Application Systems (IAAS) of the University of Stuttgart,
where he is a member of the Service Computing Department. He holds a masters
degree in Computing Science from the University of Groningen. His research
interests include: Internet of Things, Green Computing, Cloud Computing, and
Energy Efficient Data Centers.
---|---
|  Yannis Bloch studied his B.Sc. degree in Computer Science at the Institute
of Architecture of Application Systems (IAAS) of the University of Stuttgart.
---|---
|  Marco Aiello Shell Marco Aiello (SIEEE) is professor of Computer Science
and Head of the Service Computing Department at the University of Stuttgart,
Germany. He is also an elected member of the European Academy of Sciences and
Arts and honorary professor of Distributed Systems at the University of
Groningen, The Netherlands, where he was a faculty member from 2006 till 2018.
He holds a PhD in Logic from the University of Amsterdam, the Habilitation in
Applied Informatics from TU Wien, and a master degree in Engineering from La
Sapienza University of Rome. His research interests are in Service Computing,
Smart Energy Systems, and Data Centers.
---|---