#  Performance of a Triple-GEM Demonstrator in $pp$ Collisions at the CMS
Detector

M. Abbas  M. Abbrescia  H. Abdalla  A. Abdelalim  S. AbuZeid  A. Agapitos  A.
Ahmad  A. Ahmed  W. Ahmed  C. Aimè  C. Aruta  I. Asghar  P. Aspell  C. Avila
J. Babbar  Y. Ban  R. Band  S. Bansal  L. Benussi  V. Bhatnagar  M. Bianco  S.
Bianco  K. Black  L. Borgonovi  O. Bouhali  A. Braghieri  S. Braibant  S.
Butalla  S. Calzaferri  M. Caponero  J. Carlson  F. Cassese  N. Cavallo  S.S.
Chauhan  S. Colafranceschi  A. Colaleo  A. Conde Garcia  M. Dalchenko  A. De
Iorio  G. De Lentdecker  D. Dell Olio  G. De Robertis  W. Dharmaratna  S.
Dildick, 11footnotetext: Now at Rice University, Houston, Texas, USA B. Dorney
R. Erbacher  F. Fabozzi  F. Fallavollita  A. Ferraro  D. Fiorina  E. Fontanesi
M. Franco  C. Galloni  P. Giacomelli  S. Gigli  J. Gilmore  M. Gola  M.
Gruchala  A. Gutierrez  R. Hadjiiska  T. Hakkarainen  J. Hauser  K. Hoepfner
M. Hohlmann  H. Hoorani  T. Huang  P. Iaydjiev  A. Irshad  A. Iorio  F. Ivone
W. Jang  J. Jaramillo  A. Juodagalvis  E. Juska  B. Kailasapathy  T. Kamon  Y.
Kang  P. Karchin  A. Kaur  H. Kaur  H. Keller  H. Kim  J. Kim  S. Kim  B. Ko
A. Kumar  S. Kumar  N. Lacalamita  J.S.H. Lee  A. Levin  Q. Li  F. Licciulli
L. Lista  K. Liyanage  F. Loddo  M. Luhach  M. Maggi  Y. Maghrbi  N. Majumdar
K. Malagalage  S. Malhotra  S. Martiradonna  C. McLean  J. Merlin  M. Misheva
G. Mocellin  L. Moureaux  A. Muhammad  S. Muhammad  S. Mukhopadhyay  M.
Naimuddin  S. Nuzzo  R. Oliveira  P. Paolucci  I.C. Park  L. Passamonti  G.
Passeggio  A. Peck  A. Pellecchia  N. Perera  L. Petre  H. Petrow  D. Piccolo
D. Pierluigi  G. Raffone  M. Rahmani  F. Ramirez  A. Ranieri  G. Rashevski  B.
Regnery  M. Ressegotti, 22footnotetext: Now at INFN Sezione di Genova, Genova,
Italy C. Riccardi  M. Rodozov  E. Romano  C. Roskas  B. Rossi  P. Rout  J. D.
Ruiz  A. Russo  A. Safonov  A. K. Sahota  D. Saltzberg  G. Saviano  A. Shah
A. Sharma  R. Sharma  T. Sheokand  M. Shopova  F. M. Simone  J. Singh  U.
Sonnadara  E. Starling  B. Stone  J. Sturdy  G. Sultanov  Z. Szillasi  D.
Teague  D. Teyssier  T. Tuuva  M. Tytgat  I. Vai  N. Vanegas  R. Venditti  P.
Verwilligen  W. Vetens  A.K. Virdi  P. Vitulo  A. Wajid  D. Wang  K. Wang
I.J. Watson,33footnotetext: Corresponding author. N. Wickramage  D.D.C.
Wickramarathna  S. Yang  U. Yang  Y. Yang  J. Yongho  I. Yoon  Z. You  I. Yu
and S. Zaleski

###### Abstract

After the Phase-2 high-luminosity upgrade to the Large Hadron Collider (LHC),
the collision rate and therefore the background rate will significantly
increase, particularly in the high $\eta$ region. To improve both the tracking
and triggering of muons, the Compact Muon Solenoid (CMS) Collaboration plans
to install triple-layer Gas Electron Multiplier (GEM) detectors in the CMS
muon endcaps. Demonstrator GEM detectors were installed in CMS during 2017 to
gain operational experience and perform a preliminary investigation of
detector performance. We present the results of triple-GEM detector
performance studies performed in situ during normal CMS and LHC operations in
2018. The distribution of cluster size and the efficiency to reconstruct high
$p_{T}$ muons in proton–proton collisions are presented as well as the
measurement of the environmental background rate to produce hits in the GEM
detector.

##  1 Introduction

The Compact Muon Solenoid (CMS) detector [1] at the Large Hadron Collider
(LHC) has performed remarkably during the running of the LHC. CMS will be
upgraded for the High Luminosity phase of the LHC (HL-LHC) [2], which will
deliver an order of magnitude increase in the instantaneous luminosity to
$5\times 10^{34}$ cm-2 s-1 and the center-of-mass energy may increase from 13
TeV to 14 TeV. The corresponding increase in the collision rate, and therefore
radiation background, will cause difficulties in maintaining the high
efficiency and reliability of the data collection triggers with the current
CMS detector configuration. In particular, final states with muons are
extremely important to the CMS physics program, providing the motivation to
the CMS Collaboration to upgrade the muon systems to cope with the high-
luminosity conditions. The high-$\eta$ region is of particular concern, as the
increased luminosity will deliver additional radiation primarily in this
forward region. Therefore, to enhance the muon trigger and track
reconstruction capabilities, large-area triple-layer Gas Electron Multiplier
(GEM) detectors [3] are being installed in the CMS muon endcaps [4].

Figure 1: Figure showing a quadrant of the $R$-$z$ cross-section of the CMS
detector with the position of the GE1/1 detector highlighted in red (left),
and an $x$-$y$ view showing the locations where the demonstrator detectors
were installed, showing the 4 chambers used in the studies in slot 1, and the
chamber with upgraded electronics that was not included in the data taking in
slot 2 (right).

The GEM detectors are designed to complement the existing Cathode Strip
Chambers (CSC). One station of GEM detectors was installed in 2019–2020 during
the LHC Long Shutdown 2. The most forward station of CSC detectors is denoted
ME1/1, and the GEM detectors were installed in front of ME1/1 and are called
GE1/1, as shown in figure 1. Two more stations of GEM detectors will be
installed during the next LHC long shutdown in preparation of the HL-LHC.
Further information on the design and geometry of the GEM detectors is
provided in the Technical Design Report [4].

The GE1/1 station at each muon endcap consists of 36 super chambers, each
covering a $\phi$ slice of $10.15^{\circ}$ and arranged in an overlapping
fashion to provide full $2\pi$ coverage in $\phi$. The super chambers
alternate in $\phi$ between a long version covering $1.55<|\eta|<2.18$ and a
short version covering $1.61<|\eta|<2.18$, as required by the mechanical
envelope of the existing endcap, while providing maximal $\eta$ coverage. A
super chamber is comprised of two layers of trapezoidal triple-GEM detectors,
each of which provides an independent measurement of particle position. Each
layer consists of eight $\eta$ partitions of 384 strips per partition. The
strips are arranged to give good resolution in the global $\phi$ direction,
and the partitioning gives a coarse measurement of the $\eta$ of a particle
traversing the detector. The 384 strips are read out by three Very Forward
ATLAS and TOTEM (VFAT) front-end readout chips, each responsible for
outputting a binary value from 128 channels indicating if the corresponding
strip was hit.

Preparation for the utilization of GE1/1 began with the insertion of
_demonstrator_ super chambers during the 2016 year end technical shutdown
(YETS) [5]. Four superchambers were installed vertically in slot 1 as shown in
figure 1, and one superchamber was installed horizontally in slot 2. The
chambers used different versions of the detector readout electronics; the
detectors in slot 1 use version 2 which incorporate VFAT2 ASICs [6], while the
detector in slot 2 was equipped with version 3 which uses VFAT3 ASICs [7]. The
integration with the CMS DAQ system was only available for the version 2
detectors. The electronics chains, DAQ, High Voltage system and detector
control system used during the slice test are further described in ref. [5].
The main goal of this GEM demonstrator detector was to gain operational
experience with running the GEM detector and to understand the performance of
the detector during proton-proton collisions.

The detector was operated at a gas gain of $\sim 1\times 10^{4}$ with a
readout charge threshold of 3 fC for eta partitions three to eight. With this
setting, a measured detection efficiency of 95% for muons is consistent with
the test beam data [8]. Eta partitions one and two were operated at higher
thresholds, 8.8 fC and 4.0 fC respectively, due to noise from a grounding
issue in the readout electronics of version 2. This noise problem is fixed in
version 3 of the electronics. The discharge rate and gain variation due to a
potential drop caused by the current load are evaluated in the TDR [4]. We did
not expect such effects in the demonstrator where chambers were operated at a
lower gain of $1\times 10^{4}$. However, irreversible channel loss due to
discharge was observed in the demonstrator chambers. The subsequent
investigation provided a strategy for mitigating the effect of these discharge
events: increasing the protection resistor of each VFAT channel and the
decoupling resistor in the chamber HV filter [9].

In this paper, we present the results of the GEM demonstrator detector’s
performance using data collected during normal CMS operations with LHC
proton–proton collisions. This includes validation of the GEM reconstruction
software chain with real data, environmental background rate studies, and muon
detection efficiency studies. Although the GEM demonstrator detectors were
installed and operational from the end of 2017, they were still being
commissioned. To avoid disrupting the CMS read-out system during stable
collisions, GEM was only included in a subset of luminosity runs. We present
results from data collected on July 8th, 2018, when the GEM detector was
incorporated into the full CMS DAQ chain, corresponding to an integrated
luminosity of 205.4 pb-1.

In addition to providing a striking visual representation of particle physics
collision data, event displays can be a useful debugging tool, particularly
for early running. Therefore, we have updated the iSpy program [10] used to
display CMS events to include the GEM demonstrator detectors. Figure 2 shows
an example event display where a GEM hit has been found along the path of a
well-reconstructed muon belonging to a $Z$ boson candidate which decays to two
muons.

Figure 2: Event display showing a reconstructed muon (the red lines) passing
through the GEM detector (the blue trapezoidal boxes) where a GEM hit has been
reconstructed along the muon path in $R$-$z$ view (bottom left) and azimuthal
view (bottom right).

##  2 Distribution of Cluster Size

The data from events passing the CMS triggering system is read out and
reconstructed offline using the CMS software, CMSSW. The raw data from the GEM
consists of one bit for each strip which is 1 when a strip fires and 0
otherwise. The data from all strips is packed together with information on the
status of the electronics and is unpacked using a map which connects the raw
bits to the corresponding strips. A strip is registered as fired if the charge
collected in the timing window (100 ns or four proton–proton bunch crossings
in the demonstrator setup) exceeds a charge threshold which is set per $\eta$
partition. As particles passing through the detector may fire several
contiguous strips, groups of adjacent strips which have all fired are
clustered into a single reconstructed object, called a GEM RecHit. This
clustering procedure also takes into account strips that are not responsive
during the run (inactive strips). The distribution of cluster size (the number
of strips clustered into each RecHit) taken in real data is shown in figure 3
for one of the demonstrator chambers. Only clusters reconstructed along the
path of a muon candidate are included in the distribution, where the muon
candidates are chosen with the selection described in section 3. The cluster
size is dependent on the gain and the position in the readout, as the wedge
geometry of the GEM chambers mean that the strips at high $\eta$ are closer
together than at low $\eta$. Each detector must pass a requirement of the gain
non-uniformity to be less than $\pm 15\%$. A clear correlation between the
muon detector efficiency and its gain non-uniformity is not seen in the
demonstrator detectors. The average of the cluster size was shown to be about
two during test beam for all $\eta$ partitions [4]. This is consistent with
the measurements shown here, except for $\eta$ partitions one and two. These
partitions were run with a higher charge threshold due to the grounding issue
mentioned earlier, which biased the cluster size to lower values, as seen in
figure 3. The clusters being readout in conjugation with reconstructed muons
and with the expected cluster sizes show the successful operation of the GEM
data pipeline from initial data acquisition, to passing through the CMS DAQ
and being saved to disk, to unpacking and reading the strip hits, and finally
clustering into RecHits.

Figure 3: The distribution of cluster size of the GEM RecHits for each of the
$\eta$ partitions in chamber 28, layer 2. The $\eta$ partitions are numbered
from 1 at lowest $\eta$, to 8 at highest $\eta$. The cluster size increases as
the $\eta$ partition number increases as the trapezoidal detector shape means
the strips are closer together at high $\eta$.

##  3 Muon Detection Efficiency

The main goal of GE1/1 is to provide triggering and redundancy to the muon
tracking system. Therefore, understanding the detection efficiency for muons
is an important result from the demonstrator detectors. To understand the in
situ response of the GEM detector to muons, we utilize well-identified muons
reconstructed with the full CMS detector, except for the GEM detectors. The
muons used in this study are required to pass the tight identification
criteria [11], and have transverse momentum greater than 20 GeV, from events
where the VFATs are reporting the same bunch crossing time as the back end
electronics. We take this collection of muons, then propagate each muon from
the final hit to the plane which contains the GE1/1 detectors. If the muon is
successfully propagated inside a GE1/1 detector, then we consider the muon as
a GEM-occupying muon. We additionally require that the muon propagation point
is not within $0.5^{\circ}$ of the edge of the detector, to exclude edge
inefficiencies due to uncertainties in the propagation. We search the $\eta$
partition to which the muon is propagated, and if a GEM RecHit is found within
5 cm of the propagation point, we consider that RecHit matched to the
reconstructed muon.

Figure 4: Detector efficiency for the demonstrator chambers as measured using
muon reconstructed in each of the $\eta$ partitions of one of the GEM
demonstrator chambers. The inactive strip fraction indicates the fraction of
strips unable to collect data during the test period.

We take the number of matched GEM RecHit and divide by the total number of
muons propagated to the GE1/1 detectors to get the muon detection efficiency.
Figure 4 shows this value for one of the demonstrator chambers, calculated
separately for each $\eta$ partition. Due to electronics issues, some VFATs
were inactive and not registering hits during the data-taking period. The
fraction of these inactive channels is indicated and should be taken into
account when interpreting the efficiencies, which is taken based on all muons
propagated to the $\eta$ partition. With this caveat in mind, we see that the
efficiency of the demonstrator is close to the requirement of 97% efficiency,
defined by the GEM TDR [4].

##  4 Environmental Background Rates

Another important result from the demonstrator detector is the measurement of
the background rate. That is, the expected rate of particles passing through
the GEM detector as a function of luminosity. This is important as it allows
us to understand the level of radiation damage. We produced measurements of
the background rate using a stream of data which was obtained from saving the
output of randomly triggered bunch crossings, giving a zero-bias dataset.

We performed several calibration steps to obtain the event hit rate. To
analyze only the active channels, we excluded inactive or noisy channels.
Inactive channels had no hits registered in the entire dataset. Noisy channels
are channels that contain more than 2% of the total number of hits registered
by their VFAT (since there are 128 strips per VFAT we expect each channel to
produce about 0.78% of the VFAT’s total hits). After the removal of the noisy
and inactive channels, the remaining channels were used to calculate the
active area of the readout by multiplying the total area of the GEM detectors
by the fraction of active channels.

Even after removing noisy channels, some luminosity blocks (corresponding to
data taking time on the order of a minute) appeared to have all the strips in
a single VFAT to fire continuously. For each VFAT, luminosity blocks where the
number of hits is more than $2\sigma$ above the average are excluded from the
event hit rate calculation. The remaining luminosity blocks define the active
time of the VFAT, which is the number of events the VFAT is declared active
times the data collection time window of 100 ns (four LHC bunch crossings).
The event hit rate is then calculated as the number of hits divided by the
active time and active area.

Figure 5:  Hit rate as a function of the instantaneous luminosity calculated
for each $\eta$ partition of chamber 28 layer 2 (left). The data points are
corrected for the effective readout area and active time as described in the
text. Hit rate as a function of the distance from the beam pipe where the
value of the data points are obtained from the linear fits to hit rate versus
instantaneous luminosity of $1.5\times 10^{34}$ cm-2 s-1 (right). The error
bars are obtained from uncertainties in the linear fit and the curve is an
exponential function.

Figure 5 shows the hit rate obtained from the procedure above. The hit rate is
also shown as a function of cylindrical radius for a fixed reference
luminosity of $1.5\times 10^{34}$ cm-2 s-1, obtained from a linear fit to the
binned plot of the rate as a function of luminosity. In data, the various
radii are obtained by using the hit rate for a single $\eta$ partition and
plotting at the average radius, and will, therefore, actually be an average
over the radial range of the $\eta$ partition. The data show reasonable
agreement with the expected background rate based on previous simulation work
[4], and are consistent with newer simulation produced after the data
collection period [12].

##  5 Summary

A demonstrator GEM detector was installed into the CMS experiment and operated
through 2017 and 2018. The goal of the detector was to gain operational
experience and do performance studies before the full ring of GEM detectors is
operational in Run 3 of the LHC. Through the second half of 2018, the detector
was operated as part of the full CMS data acquisition system, allowing for
studies to be made using the full CMS data pipeline. These studies show that
the detector has high efficiency for reconstructing hits along moun candidate
paths, chosen by the mature CMS identification algorithms. The rate of
background hits was also measured in data. The measurements performed are
consistent with expectations based on previous simulation and test beam
studies. A total of 144 production GE1/1 chambers installed at station 1 of
the endcap region will be operational with the expected performance in Run 3
of the LHC.

## Acknowledgments

We gratefully acknowledge support from FRS-FNRS (Belgium), FWO-Flanders
(Belgium), BSF-MES (Bulgaria), MOST and NSFC (China), BMBF (Germany), CSIR
(India), DAE (India), DST (India), UGC (India), INFN (Italy), NRF (Korea),
MoSTR (Sri Lanka), DOE (USA), and NSF (USA).

## References

  * [1] CMS Collaboration, _The CMS experiment at the CERN LHC_ , _JINST_ 3 (2008) S08004.
  * [2] CMS Collaboration, _Technical Proposal for the Phase-II Upgrade of the CMS Detector_ , Tech. Rep. CERN-LHCC-2015-010.
  * [3] F. Sauli, _GEM: A new concept for electron amplification in gas detectors_ , _Nucl. Instrum. Meth. A_ 386 (1997) 531–534.
  * [4] CMS Collaboration, _CMS Technical Design Report for the Muon Endcap GEM Upgrade_ , Tech. Rep. CERN-LHCC-2015-012.
  * [5] M. Abbas et al., _Detector control system for the GE1/1 slice test_ , _JINST_ 15 (2020) P05023–P05023.
  * [6] P. Aspell, _VFAT2 - Operating Manual_ , _CERN_ (2011) .
  * [7] P. Aspell, _VFAT3 - Operating Manual_ , _CERN_ (2016) .
  * [8] M. Abbas et al., _Performance of prototype GE1/1 chambers for the CMS muon spectrometer upgrade_ , _Nucl. Instrum. Meth. A_ 972 (2020) 164104.
  * [9] F. Ivone, _Discharge mitigation strategies for the CMS GE1/1 Triple-GEM detectors_ , _JINST_ 15 (2020) C05009.
  * [10] G. Alverson, G. Eulisse, T. McCauley and L. Taylor, _ispy: a powerful and lightweight event display_ , in _Journal of Physics: Conference Series_ , vol. 396, p. 022002, 2012.
  * [11] CMS Collaboration, _Performance of CMS muon reconstruction in pp collision events at $\sqrt{s}$ = 7 TeV_, _JINST_ 7 (2012) P10002–P10002.
  * [12] M. Abbas et al., _Benchmarking simulation of the radiation environment at the LHC with a triple-GEM detector in the CMS experiment_ , _arXiv:2107.03621_ (2021) .


[1, *]Mackenzie L.Blanusa [2]Carla J.López-Zurita [2, ${\dagger}$]StephanRasp

1]University of Connecticut, Avery Point Campus, 1084 Shennecossett Rd,
Groton, CT, 06340 *]Work done during an internship at ClimateAi 2]ClimateAi,
San Francisco ${\dagger}$]Now at Google Research

Mackenzie Blanusa<EMAIL_ADDRESS>
# The role of internal variability in global climate projections of extreme
events

###### Abstract

Climate projection uncertainty can be partitioned into model uncertainty,
scenario uncertainty and internal variability. Here, we investigate the
different sources of uncertainty in the projected frequencies of daily maximum
temperature and precipitation extremes, which are defined as events that
exceed the 99.97th percentile. This is done globally using initial-condition
large ensembles. For maximum temperature extremes, internal variability
dominates in the next two decades. Around the middle of the 21st century model
and scenario uncertainty become the dominant contribution in the tropics but
internal variability remains dominant in the extra-tropics. Towards the end of
the century, model and scenario uncertainty increase to near equal
contributions of $\sim$40% each globally with large regional fluctuations. For
precipitation extremes, internal variability dominates throughout the 21st
century, except for some tropical regions, for example, West Africa. In
regions where internal variability constitutes the major source of
uncertainty, the potential impact of reducing model uncertainty on the signal-
to-noise ratio of the climate projection is estimated to be small. We discuss
the caveats of the methodology used and impact of our findings for the design
of future climate models. The importance of internal variability found here
emphasizes that large ensembles are a vital tool for understanding climate
projections.

Uncertainties in climate projections arise from three different sources
(Hawkins and Sutton, 2009). First, there is model uncertainty. Climate models
have errors in their representation of dynamical and physical processes and
disagree in how they respond to external forcings, most notably to increases
in greenhouse gases. Second is scenario uncertainty, which describes the
ambiguity related to how greenhouse gas emissions will develop in the future.
Third and last, there is internal variability which is caused by the chaotic
nature intrinsic to the Earth system. To take a simple example, a decision
maker might be interested in how many extreme heat days to expect in the
coming decade. Even in a stationary climate, the outcome is uncertain, as, by
chance, some decades see many heat extremes while others only have few. How
much each of these three uncertainties contributes to the total uncertainty
depends on the parameter of interest, the geographic location and how far
ahead one is projecting.

Naturally, for decision making one would like to reduce the uncertainty of
climate projections. However, while one can hope to reduce model uncertainty
by building better climate models, internal variability sets a hard limit on
how low uncertainty can become, similar to the intrinsic limit of
predictability in weather forecasting (Zhang et al., 2019). Therefore, if
internal variability makes up a large fraction of the total variability, even
a significant model improvement would only lead to a minor reduction in total
uncertainty. Scenario uncertainty is a curious case. Often it is assumed to be
irreducible similar to internal variability, especially as commonly used
climate model scenarios do not have likelihoods attached to them. However, in
some use cases decision makers look at a single scenario at a time. One
example is mitigation, where some influence on future emissions is assumed.

To study the partitioning of variability requires separating the forced
response from the chaotic fluctuations for a given climate model. This was
first done by fitting a smooth curve to time series of annual temperature and
precipitation and estimating internal variability from the residual by Hawkins
and Sutton (2009) and Hawkins and Sutton (2011) (hereafter abbreviated HS09
and HS11, respectively). One issue with their fitting method is that it breaks
down for noisy time-series, e.g. regional precipitation extremes (Lehner et
al., 2020, hereafter referred to as L20). This is where large ensembles can
help out, in which a single model is run multiple times with varying initial
conditions (Deser et al., 2020). Using large ensembles, one can estimate the
models forced response as the ensemble mean and the internal variability from
the ensemble variance. Because of their usefulness, more and more modeling
centers are running large ensembles. A third way to estimate internal
variability is by using an observational large ensemble. These are based on
statistical models of different modes of variability (e.g. ENSO) with
parameters fitted to the observational record. Because of this constraint to
use past observations they generally are only used to provide several
potential realizations of the past rather than future warming scenarios
(McKinnon and Deser, 2018, 2021).

Regardless of the methodological details, studies largely agree on the role of
internal variability. For annual mean temperature, internal variability plays
a large role in short-term projections (up to 15 years), while the medium-term
(30-40 years) is dominated by model uncertainty. Towards the end of the
century, scenario uncertainty becomes the dominant factor in many regions of
the earth, while internal variability becomes negligible (HS09, L20, Maher et
al. (2020)). When looking at regional uncertainty, internal variability
usually plays a larger role, often making up $>50\%$ of the total variability
for most of the 21st century (Rasmussen et al., 2016). For precipitation, a
more chaotic variable, internal variability plays a larger role and can stay
dominant even through the middle of the century regionally (HS11, L20).
Further, since precipitation is less directly influenced by global warming,
scenario uncertainty makes up a smaller fraction of the total uncertainty in
most regions even at the end of the century. HS11 also introduced the use of a
signal-to-noise ratio (SNR), dividing the expected change over the total
uncertainty. For average temperature the SNR is usually quite large, while it
is much smaller for precipitation. Looking at less averaged events or even
extremes further increases the relative importance of internal variability.
Examples of this are shown in several studies. L20 (their Fig. 7) showed that
internal variability dominates for decadal mean winter precipitation in
Seattle until the end of the century and for annual mean summer temperatures
in Southern Europe until the middle of the century. Deser et al. (2020) looked
at the internal variability of extreme heat days in Dallas, Texas and found a
large inter-ensemble spread (their Fig. 3). Suarez-Gutierrez et al. (2018)
investigated the internal variability of European summer heat extremes at
given 1.5 or 2∘C warming levels and showed significant overlap in the
distribution of the two scenarios.

In this study, we aim to build upon prior research by systematically studying
the role of internal variability for daily temperature and precipitation
extremes on a global scale. Specifically, we want to answer the following
questions:

  1. 1.

What fraction of the total uncertainty is made up by internal variability as a
function of location and forecast time?

  2. 2.

What is the signal-to-noise ratio, a rough proxy for the actionability of a
climate projection?

  3. 3.

How much can one hope to reduce total uncertainty by minimizing model
uncertainty?

##  1 Methods

###  1.1 Data

The large ensembles for this study were chosen because of their availability
in the cloud. Climate projections, and large ensembles in particular, consist
of huge amounts of data. Direct cloud access using the Zarr data
format111https://zarr.readthedocs.io/en/stable/ eliminates a big data
engineering hurdle, enabling studies such as this one for groups that do not
already have the data downloaded on a local server. We acknowledge that more
large ensembles exist, for example those included in Multi-Model Large
Ensemble Archive (MMLEA) (Deser et al., 2020) and additional ones in CMIP6,
but we did not use them as they were not available in the cloud.

Our variables of interest are daily maximum temperature (°C) and daily
precipitation (mm). For all models, daily data was accessed globally and
regridded to a 2.5 x 2.5 degree grid (as in L20) using the bilinear
interpolation in the xesmf Python package
222https://doi.org/10.5281/zenodo.1134365.

####  1.1.1 Large ensembles

Five large ensembles were used in this study and are listed in Table 1. All
large ensembles except CESM-LE are sourced from the CMIP6 cloud archive on
Pangeo’s Google Cloud Storage bucket
333https://catalog.pangeo.io/browse/master/climate/cmip6_gcs/. CESM-LE, a
CMIP5-class model, is hosted on the AWS cloud
444https://doi.org/10.26024/wt24-5j82. The number of ensemble members ranges
from 10 to 58. Only models with 10 or more members were used, as models with
fewer realizations would not provide adequate sample size. Even 10 members is
small, as ensembles with less than 30 members have been found to underestimate
variability (Wood et al., 2021). We used daily data from historical (up to
2014) and SSP5-8.5 (2015-2100) simulations for all CMIP6 large ensembles. For
the CESM-LE large ensemble, we used the 20C and RCP8.5 simulations. There are
differences between the SSP5-8.5 and RCP8.5 scenarios but in the context of
this study, we deemed them negligible. Because CESM-LE is a CMIP5-class model,
the historical period only extends to 2004 while the RCP8.5 scenario starts in
2005. To make these simulations compatible with the CMIP6 models, we
concatenated the two simulation periods and subsequently split them in 2015.
This should not matter given our definition of extremes using quantiles, see
below.

Table 1: List of large ensembles used in this study Modeling Center | Model Version | Resolution (atm/oc) | Years |  | No. of members
---
(historical, future)
Forcing | Reference
MIROC | MIROC6 |  $\sim$1.4° x 1.4° / 1° | 1850-2100 | 50, 50 |  | historical,
---
ssp585
Tatebe et al. (2019)
NCAR | CESM-LE |  $\sim$1.3° x 0.9° / 1° | 1920-2100 | 40, 40 |  | 20C,
---
rcp8.5
Kay et al. (2015)
CCCma | CanESM5 |  $\sim$2.8° x 2.8° / 1° | 1850-2180 | 50, 50 |  | historical,
---
ssp585
Swart et al. (2019)
MPI | MPI-ESM1-2-LR |  $\sim$1.9° x 1.9° / 1.5° | 1850-2100 | 10, 10 |  | historical,
---
ssp585
|  Müller et al. (2018),
---
Mauritsen et al. (2019)
EC-Earth-Consortium | EC-Earth3 |  $\sim$0.7° x 0.7° / 1° | 1850-2100 | 71, 58 |  | historical,
---
ssp585
Döscher et al. (2022)

Looking at the results, we noticed that the CanESM5 model consistently showed
much larger trends in most regions compared to the other models (see Fig. 1,
Seattle, as an example). As it turns out, CanESM5 is the most extreme example
of the "hot model" problem555https://www.carbonbrief.org/guest-post-how-
climate-scientists-should-handle-hot-models/ in CMIP6 (see Supplement of
Zelinka et al. (2020)), where cloud feedbacks lead to—what are generally
considered—unrealistically high climate sensitivities in some models.
Including CanESM5 leads to significanly larger values of model variability
(Fig. S12). In the end, we decided to remove CanESM5 from our variability
computations since its climate sensitivity is far outside the range considered
plausible. A similar process is done in the latest IPCC report (Hausfather et
al., 2022). The four remaining models represent a somewhat evenly spaced
sampling of climate sensitivities according to Zelinka et al. (2020).

We also found weird behavior of the MPI-ESM1-2-LR model in West Africa, where
temperature variability drops drastically in the future projections compared
to the historical simulations, alongside a drop in mean temperature, and
precipitation variability increases (see Fig. 1, Lagos). We do not know the
cause for this. In this instance, we decided to include the MPI-ESM1-2-LR
model in our subsequent analysis for two reasons. First, the suspicious
behavior was restricted regionally; second, even when removing the model, the
variability results do not change drastically because other models also
contribute to the spread in West Africa. It is important to keep in mind
though that model variability could be erroneously inflated in West Africa.

####  1.1.2 CMIP6 single realizations

Most large ensembles are only run for a single, high-emission scenario. To
obtain an estimate of scenario uncertainty, we follow L20 and use single
realizations of CMIP6 models for four scenarios: SSP1-2.6, SSP2-4.5, SSP3-7.0,
and SSP5-8.5. Fourteen CMIP6 models were used in this study, because of their
cloud availability of daily data for all scenarios and variables required.
These models are CMCC-CM2-SR5, CMCC-ESM2, EC-Earth3, EC-Earth3-Veg-LR, GFDL-
ESM4, IITM-ESM, INM-CM4-8, INM-CM5-0, IPSL-CM6A-LR, KACE-1-0-G, MIROC6, MPI-
ESM1-2-HR, MPI-ESM1-2-LR, and NorESM2-MM.

###  1.2 Definition of extremes

Every end user has a different definition of what an extreme is. Here, we use
a quantile-based definition but acknowledge that this is not a one-size-fits-
all solution. Specifically, we choose the 0.9997th quantile computed over a
reference period, here chosen to be 1995-2014. This corresponds to the largest
daily value in a 10 year period. Note that this is different from the usual
definition for return periods which uses the yearly maximum rather than daily
values. We still use the "1-in-10-year" terminology for convenience in some
places in this paper. A comparison to using yearly maximum return periods can
be found in the Supplement (Sec. S4). Further, we investigated the sensitivity
of the key results to the quantile chosen (Fig. S1). Looking at larger
quantiles (e.g. corresponding to a "1-in-100-year" event) generally leads to
somewhat larger relative contribution of internal variability but the
differences are, in most cases, within 20% (as measured by the contribution of
model to total uncertainty). We also tested using multi-day averages rather
than daily values to account for longer-lasting extremes but found negligible
effects ($<10\%$ with rare exceptions) on the results (Fig. S2).

The quantile values are computed for each model separately, thereby performing
an implicit quantile bias correction (see Sections 3 and S2). The projections
are then thresholded to yield a boolean time series with 1’s indicating where
an extreme event occurs. Next, we aggregate the data by year. Note that
consecutive days where the extreme threshold is crossed are counted as
separate events. Coarsening the data to avoid this has a negligible effect on
the results (Fig. S3). Subsequently, we average the yearly sums over an
aggregation period, another hyper-parameter, yielding the average yearly
number of extreme events in that period. As expected, this hyper-parameter has
a large impact on the results (Fig. S4). Larger aggregation periods lead to
much smoother results and therefore less internal variability. To choose a
reasonable value, we take inspiration from our operational work but also many
previous studies, as well as the IPCC report (Arias et al., 2021) and climate
disclosure frameworks. There, the impact of climate change is often considered
on decadal time scales, which is why we chose 10 years as an aggregation
period.

The final time series $x(m,e,t,l)$ then contains the yearly average number of
extreme events in each ensemble member in a 10 year window around the given
year and is a function of model $m$, ensemble member $e$ (except for the
single-realization CMIP6 models), time $t$ and location $l$.

Figure 1: Extreme event occurrence (days) per year from 2020-2095. Top row:
daily maximum temperature for three locations; (a) Seattle, U.S.A., (b)
Montreal, Canada, (c) Lagos, Nigeria. Bottom row: daily precipitation for the
same three regions (d-f). Each data point represent the average yearly
occurrence for a 10 year period around that year.

###  1.3 Partitioning of variability

Total uncertainty $T$ is the sum of internal variability $I$, model
uncertainty $M$ and scenario uncertainty $S$:

$T(t,l)=I(t,l)+M(t,l)+S(t,l)$ (1)

We follow the methodology of L20 to compute the terms above from large
ensembles. We also compare this to the fitting method of HS09 in the
Supplement.

Internal variability for each model is computed as

$I(m,t,l)=\text{var}_{e}(x)$ (2)

where $\text{var}_{e}(x)$ is the variance computed in the model dimension. We
can then average in the model dimension, yielding an average estimate of
internal variability:

$I(t,l)=\text{mean}_{m}(I(m,t,l))$ (3)

Model uncertainty can be computed as the variance across ensemble means of
each model:

$M(t,l)=\text{var}_{m}(\text{mean}_{e}(x))$ (4)

To compute scenario uncertainty, we use the 14 single-realization CMIP6
models. Following methods described in HS09 and L20, scenario uncertainty is
calculated as the variance across multi-model means for each of the four
scenarios, $s$:

$S(t,l)=\text{var}_{s}(\text{mean}_{m}(x))$ (5)

###  1.4 Ideal Total Uncertainty

Out of the three uncertainties, model, internal, and scenario, only model
uncertainty is generally assumed to be reducible. Internal variability could
potentially be reduced in the first decade by initialization but we did not
consider this further (HS09). We can therefore define an ideal total
uncertainty, $T_{\text{ideal}}$, for a climate projection which assumes a
model uncertainty of zero.

$T_{\text{ideal}}(t,l)=I(t,l)+S(t,l)$ (6)

If $S$ is ignored, the ideal total uncertainty is simply internal variability.

By comparing total and ideal total uncertainty we can understand the potential
for improving climate projections, which will vary in space and time. We can
estimate a max percent improvement using total (1) and ideal total uncertainty
(6) estimates:

$\text{Max \% Improvement}=(1-\sqrt{T_{\text{ideal}}}/\sqrt{T})\times 100$ (7)

Here, it is important to compute the improvement in standard deviation, or
real unit, space rather than variance, or squared unit, space, since
ultimately decision makers are interested in real units.

###  1.5 Signal to Noise Ratio

The signal to noise ratio (SNR) can be taken as a very rough proxy for the
actionability of a climate projection. It was first used in this context by
HS11. The signal is defined as the model-mean change in $x$ with respect to
the reference period 1995-2014, where
$x_{\text{ref}}(l)=\text{mean}_{m,e,t_{\text{ref}}}(x)$:

$\text{Signal}(t,l)=\text{mean}_{m,e}(x)-x_{\text{ref}}(l)$ (8)

The noise is simply the total uncertainty, as in Eq. (1). Therefore, as the
name suggests, SNR is the ratio of signal to the total uncertainty:

$\text{SNR}(t,l)=\text{Signal}(t,l)/\sqrt{T(t,l)}$ (9)

We also compute an ideal SNR using $T_{\text{ideal}}$. Low SNRs imply that the
noise dominates the signal and the projections are likely not that
informative. On the other hand, large SNRs indicate more certain predictions.

##  2 Results

###  2.1 Example locations

Figure 2: Percent contribution to total uncertainty for model uncertainty,
scenario uncertainty, and internal variability from 2020-2095 for three
locations: Seattle, U.S.A. (a,d), Montreal, Canada (b,e), Lagos, Nigeria
(c,f). Top row (a-c) is for daily maximum temperature and bottom row (d-f) for
daily precipitation.

We start by investigating variability at three locations: Seattle, USA,
Montreal, Canada and Lagos, Nigeria (see top left panel of Fig. 5 for map of
locations). These locations were chosen in hindsight after seeing the global
results because they represent different kinds of behavior. First, we look at
the evolution of extreme occurrence, $x$, in the different ensembles (Fig. 1).
As per definition, each ensemble should start out with a value of around 0.1
occurrences per year in 2020. For maximum temperature, values tend to increase
significantly towards the end of the century. In Seattle, the models roughly
agree on how much extremes are expected to increase (ignoring CanESM5). In
Montreal and Lagos, on the other hand, there is large disagreement between
models, with one set of models projecting a roughly 5 times larger increase
than another set. Projected extreme precipitation occurrence shows a much
smaller signal compared to temperature, especially in Seattle and Lagos. The
already mentioned behaviors of CanESM5 and MPI-ESM1-2-LR are clearly visible
in these figures (see discussion above).

To show the different uncertainties, we show the classic partitioning plots
following HS09 and L20 (Fig. 2). In addition, we also show the absolute values
of the uncertainties as standard deviations (Fig. 3). This is important since
decision makers will be interested in the uncertainty in real physical units
rather than squared units. The first thing that stands out is that the
uncertainty estimates are relatively noisy, especially compared to previous
literature. This is a result of the small sample size of our large ensembles
but also of the noise inherent to extremes. We decided against further
smoothing the plots to not obscure the noise in the method but note that the
small-scale fluctuations are likely caused by small sample sizes and
inaccuracies in the method (see Sec. 3). One example is the uptick of $S$ for
maximum temperature in Seattle around 2030. From looking at other locations
and previous studies, one would expect the contributions to change relatively
smoothly.

Figure 3: Estimates of model uncertainty (blue), internal variability
(orange), scenario uncertainty (green), total uncertainty (black, solid), and
ideal total uncertainty (black, dashed, i.e. model uncertainty = 0) from
2020-2095 for three locations: Seattle, U.S.A. (a,d), Montreal, Canada (b,e),
Lagos, Nigeria (c,f). Top row (a-c) is for daily maximum temperature and
bottom row (d-f) for daily precipitation. Each data point represent the
average yearly uncertainty (days) for a 10 year period around that year.

For maximum temperature, we see that for Seattle internal variability $I$
dominates with >50% until the second half of the century, followed by an
increase in model $M$ and, towards the end of the century, scenario
variability $S$. In Montreal and Lagos we see a different picture. For the
first century $I$ and $M$ are roughly equally important but model variability
quickly makes up the largest contribution. $S$ increases towards the end, more
so in Lagos, but $M$ remains the dominant contribution. Looking at the
absolute variabilities in standard deviation space (Fig. 3) we immediately see
that $I$ is not constant but rather increases significantly as total
uncertainty increases. This is in contrast to previous studies of internal
variability in mean quantities (e.g. HS09, HS11 and L20), where $I$ stays
roughly constant in time. This also means that the fitting method of HS09
would be inappropriate for this context since it only computes a time-constant
value for $I$. See Supplement for further comparison.

For precipitation, in Seattle and Montreal, $I$ represents by far the largest
contribution, throughout the century. Lagos, however, shows a large
contribution of $M$ towards the end of the century, driven largely by the
large projected increase in precipitation in the MPI-ESM1-2-LR model but also
partly in the EC-Earth3 model.

Fig 3 also shows the total uncertainty and the ideal total uncertainty. One
important thing to note is that the uncertainties only add up in variance
space but not in standard deviation space. A simple example would be a case
where $I=M=1$ (ignoring $S$ for a second). In variance space the total
uncertainty would be 2 but only $\sqrt{2}\approx 1.41$ in standard deviation
space. One consequence of this is that in cases where $M$ makes up around 50%
of total uncertainty in variance space (see end of century temperature
variability in Seattle and Lagos), setting $M=0$, only reduces total
uncertainty by around 30%. In Montreal, where model uncertainty dominates,
reducing $M$ has a much larger effect. Even here, however, where model
uncertainty represents 80% of total variance at the end of the century, the
ideal total uncertainty in real unit space is still roughly 40% of the actual
total uncertainty.

Figure 4: Estimates of signal to noise ratio (SNR) from 2020-2095. Each panel
includes SNR calculated with scenario uncertainty (black, solid), SNR ideal
with scenario uncertainty (black, dashed), SNR without scenario uncertainty
(red, solid), and SNR ideal without scenario uncertainty (red, dashed). Note
the log-scale on the y-axis.

Fig. 4 shows the signal-to-noise ratio for four different cases. In black, the
SNR including $S$ with the real $M$ in solid and the ideal case $M=0$ in
dashed. In red, the same but ignoring $S$. As mentioned above, SNRs
significantly below one represent projections that are dominated by noise and,
thus, most likely not very actionable. For maximum temperature, we can see
that in Seattle the transition point SNR $=1$ is reached around 2040,
regardless of the exact SNR case. Starting around 2050, one can see that
decreasing $M$ could have a significant effect, especially when $S$ is
ignored. For Montreal and Lagos, $M$ plays a much larger role. Model
uncertainty included, SNR approaches one around 2030-2040 but then stays
relatively constant around values of one. The ideal case of $M=0$ shows much
larger SNRs from the start with values quickly going above 2 at around 2040.
In Lagos, there is also a non-negligible impact of $S$, which becomes large
towards the end of the century, thereby pushing the SNR down again.

For precipitation, SNR values are very low at the start of the century in
Seattle and Montreal, crossing SNR$=1$ right at the end of the century
(Seattle) and in the 2050s (Montreal). Since $I$ is the dominant contribution,
the four SNR cases are pretty much identical. In Lagos, once again, $M$ plays
a big role. Including $M$ in the total variability leads to very low SNR
values. Setting $M=0$, SNR reaches the transition point around the 2040,
steadily increasing afterwards.

###  2.2 Global

After looking at three exemplary locations in detail, we now generalize our
diagnostics globally. First, similar to L20, we plot the fractional
contributions of total variance for three decades (Figs. 5 and 6). Note that
we chose earlier decades compared to L20. This is motivated by a focus on
actionability and adaptation, where in our professional experience end users
tend to be more interested in the next few decades rather than the end of the
century. However, the results change smoothly in time and the qualitative
difference between the time periods is small.

Figure 5: Variance fractions of maximum temperature extremes for the three
sources of uncertainty averaged for three decades, 2020-2029, 2040-2049 and
2070-2079. Percentage numbers are area-weighted global averages for each map.
The three green dots in the top left panel mark three exemplary locations:
Seattle, U.S.A., Montreal, Canada and Lagos, Nigeria. Figure 6: As Fig. 5 but
for precipitation extremes.

For maximum temperature, $I$ is the dominant factor for most of the globe in
the first decade. $M$ tends to be larger in the ITCZ area, the tropical
continents and the storm tracks off the US East coast. There are some patches
where $S$ is the largest uncertainty, for example in the East Pacific. Note,
however that the absolute uncertainty and signal in the first decade is
relatively small still. In the third decade, a rough division can be made
between the tropics where $M$ and $S$ are important and the extra-tropics
where $I$ remains dominant. There are however, large regional differences. The
Atlantic off the US East coast and the Gulf of Mexico have large values of
model uncertainty. Larger values of $I$ are also generally found towards the
Eastern side of the larger continents with some exceptions for example the
North Eastern US and Eastern Canada, as our example location Montreal shows.
In the 2070s, $I$ plays a minor role except in some continental regions, for
example the central US. $S$ and $M$ dominate but with a complex pattern
between them that, at least somewhat, suggests some noise in the methodology.
The results are largely in agreement with the decadal mean temperature
variabilities in L20, even though some of the regional details differ.

For precipitation the results are a lot more clear cut. $I$ is by far the
largest contributor in most regions throughout the century. The only exception
is West Africa, and to a lesser extent, the Amazon and the ITCZ where $M$
plays a larger role. As already explained, part of this behavior might be
caused by some irregularities with the MPI-ESM1-2-LR model but even without
this model the general pattern persists. This is also anecdotally confirmed by
the Sahel decadal JJA rainfall results in L20. Partly, this hints at a "real"
physical, underlying cause. However, it is also well known that coarse climate
models struggle to represent the large convective systems present in these
regions (Moncrieff et al., 2017). Further, as L20 speculate, this could also
be caused by sampling different phases of the Atlantic Ocean’s decadal
variability (Yeager et al., 2018), which for decadal prediction is potentially
fixable using observation-based initialization.

Figure 7: Decade-average signal to noise ratio (SNR) for daily maximum
temperature for four cases as described in the text. Figure 8: As Fig. 7 but
for precipitation.

Next we look at the four flavors of SNR, already discussed for the three
example locations (Figs. 7 and 8). For maximum temperature, in the 1st decade
SNR values are generally below one in the extratropical continents with little
effect of setting $M$ or $S$ to zero since $I$ makes up the largest
contribution by far. Some of the tropical continents have values of SNR
slightly above one, with only the tropical maritime regions showing large SNR
values, caused by the fact that variability there is generally very low. In
the 2040s, SNR values in the continents tend to be slightly above one. Here
the SNR rises noticeable in the "ideal" scenario of no model variability,
especially over Africa and South America. In the 2070s, this trend is even
stronger, with increases from setting $M=0$ seen globally to values above 2.
The increase is even stronger if one considers the case without scenario
uncertainty.

For precipitation, we see low SNR values globally. Only in the latter half of
the century do some regions see SNR $>1$, for example eastern North America,
Alaska, Greenland, eastern Eurasia and Central Africa. The latter region is
also the only one to see a jump in SNR from setting $M=0$, a logical
consequence of the large model variability discussed above.

Figure 9: Decade-average maximum % improvement for maximum temperature and
precipitation, including and excluding $S$.

As a final metric we look at the maximum possible improvement in total
uncertainty by reducing model uncertainty to zero (Fig. 9). For maximum
temperature, values over the extra-tropical continents remain low ($<30\%$)
for the first half of the century. Only some tropical regions show larger
values. Even in the latter half of the century, most regions show a small
maximum improvement if $S$ is considered. If $S$ is ignored, potential
improvements jump and are quite high ($>60\%$), even for some extra-tropical
continental regions. As to be expected by now, the potential improvements for
precipitation are very small throughout the century, except in the
aforementioned tropical regions.

##  3 Discussion

###  3.1 Uncertainties in method

For several reasons, which will be discussed below, the results presented in
this paper, and that of similar studies, should be seen as a "ballpark
estimate" rather than an exact assessment.

####  3.1.1 Small sample size

First, sample sizes are small for nearly all variance estimates. The smallest
ensemble size is 10 members which, as seen in Fig. 1, results in a slightly
noisier ensemble mean for the MPI-ESM1-2-LR ensemble compared to the other
models, thereby picking up some internal variability as signal, similar to the
fit method of HS09. Wood et al. (2021) found that using small ensembles (less
than 30 members) also results in an underestimation of variability. Next, the
number of models is low. After removing CanESM5 because of its unrealistically
high climate sensitivity only 4 models remain. Some confidence can be gained
by the fact that using the fitting method of HS09 on a larger CMIP6 ensemble
yields qualitatively similar results, at least in most cases (Fig. S9).
Similarly, the internal variability estimates of the different large ensemble
agree in their order of magnitude (Fig. S10). However, there clearly is a lot
of noise in the method, urging caution.

####  3.1.2 The impact of scenarios

Many large ensembles are only run for the highest emission scenario (RCP8.5 or
SSP5-8.5). This scenario has been shown to be very unlikely given current
policy trajectories (Pielke et al., 2022). Using a high emission scenario
results, in most cases, in a larger signal, i.e. change over the next century,
particularly for temperature extremes. This, in turn, causes internal and
model uncertainty to increase. However, model uncertainty increases more
quickly than internal variability (Figs. S9 and S11). It is to be expected
then that repeating this study using a lower emission scenario, such as
SSP2-4.5, would lead to a lower total uncertainty and a larger relative
contribution of internal variability.

Scenario uncertainty itself is naturally also affected by the choice of
scenarios. The four scenarios selected here and similar studies, were not
chosen based on any particular likelihood attached to them. In fact, SSP1-2.6
and especially SSP5-8.5 represent the fringes of likely outcomes. A denser
sampling of intermediate scenarios would, most likely, lead to a reduction in
scenario uncertainty as computed in this framework.

####  3.1.3 Models as truth

One assumption underlying all studies using models to estimate climate
projection uncertainty is that the models are good enough to accurately
represent the different types of variabilities. This, however, is a leap of
faith, especially when it comes to extremes, where climate models have been
shown to struggle (Gervais et al., 2014), even though some research suggests
that the fractional change in extremes is still adequately captured (Martinez-
Villalobos and Neelin, 2021). Since all climate models are based on similar
principles and suffer from the same issues, such as insufficient resolution,
there is no guarantee that the spread given by the models encompasses the
"true" behavior of the atmosphere.

For internal variability, the large ensembles used in this study roughly agree
(Fig. S10). However, true variability might be lower or higher. To investigate
this one could compare the variability of historical simulations against
observations, something we did not do here. One problem is that for very rare
events the sample size in global observations is quite small.

For model uncertainty estimates to be correct, the truth would have to be
within the envelop suggested by the models. For future projections, there is
obviously no way to check whether this is true. If the true evolution was
outside of the model distribution, the model uncertainty or error would be
much larger than estimated here. There is some evidence that low-resolution
climate models underestimate the change in extremes compared to convection-
permitting models (Kendon et al., 2020). However, because of the computational
expense of such high-resolution models, these studies have, so far, only been
done regionally.

###  3.2 Role of downscaling/post-processing

In this study an implicit bias correction was done by using quantiles,
estimated for each model separately. This is analogous to using a quantile
mapping. This is certainly necessary as the quantile values vary greatly
between models, highlighting the well known regional biases in climate models.
Computing a single quantile value across all models, therefore, leads to a
drastic increase in model uncertainty (Fig. S5).

Most sophisticated applications of climate projections, however, go beyond a
simple quantile mapping to yield trustworthy results. First, there are better
suited downscaling/post-processing techniques available (Maraun and Widmann,
2018). One example is a quantile delta mapping (Cannon et al., 2015), which
better preserves the signal of climate change, compared to a simple quantile
mapping. We tested the effect of using this method on variance estimates of
our three example locations (Fig. S5) with varying effect on the relative
contributions of $I$ and $M$. In Lagos, for example, the results differ
drastically. Without further analysis though, it is difficult to explain these
differences.

Next, there is the question of model performance. It is widely recognized that
not all climate models perform equally well in different aspects. A key metric
is climate sensitivity. Some CMIP6-class models have climate sensitivities
deemed far outside the likely range as estimated by a range of methods
(Zelinka et al., 2020). Here, we performed a very simple adjustment by simply
removing CanESM5, the "hottest" CMIP6 model, from our analysis. To narrow the
spread in climate sensitivities the latest IPCC report (Arias et al., 2021)
relies heavily on emulators to merge different climate models with appropriate
likelihoods. A similar procedure would also affect model uncertainty, most
likely decreasing it.

Lastly, as already mentioned in the previous sub-section, dynamical
downscaling using convection permitting-models (e.g. Tapiador et al. (2020))
could provide more realistic estimates of internal variability. While some
multi-model regional downscaling experiments have been done (e.g. Jacob et al.
(2020)), so far and to our knowledge, no convection-permitting large ensembles
exist, which would be required to partition uncertainties.

Current uncertainty partitioning studies, including this one, lag behind the
state-of-the-art of climate model downscaling/post-processing. A logical next
step would be to test the impact of the aforementioned methods (and more) on
uncertainty partitioning.

###  3.3 The potential impact of better climate models

One motivation of this study was to better understand the impact of improving
climate models on the usefulness of their projections. As already discussed in
the previous paragraphs, there are too many caveats in the methodology to
accept the results here as the final word. However, some key findings are also
relatively insensitive to perturbations in the method so that there should be
some confidence in at least a "ballpark" estimate.

If we assume that better climate models will reduce model uncertainty , what
can be said about the changes to actionability? We can use the Signal-to-Noise
ratio as a rough (and definitely not perfect) proxy for how actionable a
forecast is. Low SNRs (below 0.5) essentially indicate that the noise obscures
any actionable signal. For extreme precipitation, this is the case for most
regions throughout the first half of the century. Since the noise is
overwhelmingly caused by internal variability, improving models has little
impact in this framework. Even towards the latter half of the century we see
noise trumping signal in many continental regions. Only in some tropical
regions, especially West Africa, is there a big potential improvement from
reducing model uncertainty.

One can argue that none of the current generation climate models are able to
produce realistic extreme precipitation at all, thereby implying a much larger
model error. However, if the SNRs suggested here are even remotely accurate,
past observations (if available for a long enough period) can provide almost
as accurate a picture of future extreme occurrences as even the best possible
next-generation climate models, for most regions.

For temperature, the expected signal is a lot more obvious than for
precipitation, so that even with current-generation climate models we see an
actionable signal for the middle of the century. Starting around the middle of
the century, model uncertainty contributes significantly to total uncertainty,
suggesting that there could be actionable reductions in total uncertainty from
reducing model error, especially if one focuses on a single scenario.

It is important to note that there are large regional differences, which means
that the potential impact of model improvements can vary greatly from one
location to another. One example are coastal regions, where our diagnostics
often show large gradients, suggesting that there is significant uncertainty
attached to our method.

In this study, we build upon the line of work started by HS09, and more
specifically the large ensemble methodology of L20, by investigating the
partitioning of climate model uncertainty into internal variability, model and
scenario uncertainty, specifically for extreme events. Previous studies have
looked at individual examples of extremes in the context of variability but
here we extend this globally.

In the introduction, we laid out three questions which we can now attempt to
answer: 1) Internal variability dominates total uncertainty in climate
projections of temperature extremes in the near-term, after which model and
scenario uncertainty become the main contributors. For precipitation extremes,
internal variability is dominant throughout the 21st century. There are
regional differences, for example a stronger role of model variability in the
tropics. 2) The signal-to-noise ratio is large for temperature in the tropics
in the near-term and then globally from 2040 onward. For precipitation, the
expected changes are lower and the internal variability is higher, so that
signal-to-noise ratios above 1 do not appear until the 2070s in most regions.
3) Improving models and thereby reducing model uncertainty has a noticeable
effect on projections of temperature extremes starting in the 2030s/40s. For
precipitation, the potential impact is much smaller except in some isolated
tropical regions.

There are several methodological uncertainties in the approach used here,
which is why this study should be seen as a rough estimate. We also used a
very general definition of extreme events to enable a global study. Our
sensitivity tests suggest that the results are qualitatively insensitive to
the exact definition but further work is certainly required if one is to
estimate the variability for specific, high-impact extremes and use cases.
Further, the use of observation data to assess the fidelity of models to
represent extremes would certainly be desirable for more detailed studies.

Because internal variability is such an important contributor to climate
projection uncertainty, we believe that considering its effect should be an
integral part of climate modeling going forward. Specifically, this means that
single-model large ensembles should become the standard for every future
climate model. Using just a single realization could yield artificial signals
that are actually part of natural fluctuations. Such arguments have already
been made in Deser et al. (2012) and more recently in Deser et al. (2020).

Finally, we would like to emphasize again the importance of easy data access
through the cloud. As data volumes increase from next-generation climate
modeling efforts, downloading and maintaining local copies of data will become
unmanageable for many users. Cloud data democratizes access for non-academic
groups (like us) and researchers from lower income countries.

The code is available at
https://github.com/MackenzieBlanusa/InternalVariability. All data is available
freely in the cloud.

MB led the data analysis and writing. CL contributed to the model post-
processing and plotting. SR conceived the project.

No competing interests.

###### Acknowledgements.

We thank V Balaji, Dave Farnham, Carlos Hoyos, Nicola Maher and R Saravanan
for their helpful comments on this project.

## References

  * Arias et al. (2021) Arias, P., Bellouin, N., Coppola, E., Jones, R., Krinner, G., Marotzke, J., Naik, V., Palmer, M., Plattner, G.-K., Rogelj, J., et al.: Climate Change 2021: The Physical Science Basis. Contribution of Working Group14 I to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change; Technical Summary, 2021.
  * Cannon et al. (2015) Cannon, A. J., Sobie, S. R., and Murdock, T. Q.: Bias Correction of GCM Precipitation by Quantile Mapping: How Well Do Methods Preserve Changes in Quantiles and Extremes?, Journal of Climate, 28, 6938–6959, 10.1175/JCLI-D-14-00754.1, publisher: American Meteorological Society Section: Journal of Climate, 2015.
  * Deser et al. (2012) Deser, C., Knutti, R., Solomon, S., and Phillips, A. S.: Communication of the role of natural variability in future North American climate, Nature Climate Change, 2, 775–779, 2012.
  * Deser et al. (2020) Deser, C., Lehner, F., Rodgers, K. B., Ault, T., Delworth, T. L., DiNezio, P. N., Fiore, A., Frankignoul, C., Fyfe, J. C., Horton, D. E., Kay, J. E., Knutti, R., Lovenduski, N. S., Marotzke, J., McKinnon, K. A., Minobe, S., Randerson, J., Screen, J. A., Simpson, I. R., and Ting, M.: Insights from Earth system model initial-condition large ensembles and future prospects, Nature Climate Change, 10, 277–286, 10.1038/s41558-020-0731-2, number: 4 Publisher: Nature Publishing Group, 2020.
  * Döscher et al. (2022) Döscher, R., Acosta, M., Alessandri, A., Anthoni, P., Arsouze, T., Bergman, T., Bernardello, R., Boussetta, S., Caron, L.-P., Carver, G., Castrillo, M., Catalano, F., Cvijanovic, I., Davini, P., Dekker, E., Doblas-Reyes, F. J., Docquier, D., Echevarria, P., Fladrich, U., Fuentes-Franco, R., Gröger, M., v. Hardenberg, J., Hieronymus, J., Karami, M. P., Keskinen, J.-P., Koenigk, T., Makkonen, R., Massonnet, F., Ménégoz, M., Miller, P. A., Moreno-Chamarro, E., Nieradzik, L., van Noije, T., Nolan, P., O’Donnell, D., Ollinaho, P., van den Oord, G., Ortega, P., Prims, O. T., Ramos, A., Reerink, T., Rousset, C., Ruprich-Robert, Y., Le Sager, P., Schmith, T., Schrödner, R., Serva, F., Sicardi, V., Sloth Madsen, M., Smith, B., Tian, T., Tourigny, E., Uotila, P., Vancoppenolle, M., Wang, S., Wårlind, D., Willén, U., Wyser, K., Yang, S., Yepes-Arbós, X., and Zhang, Q.: The EC-Earth3 Earth system model for the Coupled Model Intercomparison Project 6, Geoscientific Model Development, 15, 2973–3020, 10.5194/gmd-15-2973-2022, 2022.
  * Gervais et al. (2014) Gervais, M., Gyakum, J. R., Atallah, E., Tremblay, L. B., and Neale, R. B.: How Well Are the Distribution and Extreme Values of Daily Precipitation over North America Represented in the Community Climate System Model? A Comparison to Reanalysis, Satellite, and Gridded Station Data, Journal of Climate, 27, 5219–5239, 10.1175/JCLI-D-13-00320.1, publisher: American Meteorological Society Section: Journal of Climate, 2014.
  * Hausfather et al. (2022) Hausfather, Z., Marvel, K., Schmidt, G. A., Nielsen-Gammon, J. W., and Zelinka, M.: Climate simulations: recognize the ‘hot model’ problem, Nature, 605, 26–29, 10.1038/d41586-022-01192-2, bandiera_abtest: a Cg_type: Comment Number: 7908 Publisher: Nature Publishing Group Subject_term: Climate change, Policy, Climate sciences, 2022.
  * Hawkins and Sutton (2009) Hawkins, E. and Sutton, R.: The Potential to Narrow Uncertainty in Regional Climate Predictions, Bulletin of the American Meteorological Society, 90, 1095–1108, 10.1175/2009BAMS2607.1, publisher: American Meteorological Society Section: Bulletin of the American Meteorological Society, 2009.
  * Hawkins and Sutton (2011) Hawkins, E. and Sutton, R.: The potential to narrow uncertainty in projections of regional precipitation change, Climate Dynamics, 37, 407–418, https://doi.org/10.1007/s00382-010-0810-6, num Pages: 407-418 Place: Heidelberg, Netherlands Publisher: Springer Nature B.V., 2011.
  * Jacob et al. (2020) Jacob, D., Teichmann, C., Sobolowski, S., Katragkou, E., Anders, I., Belda, M., Benestad, R., Boberg, F., Buonomo, E., Cardoso, R. M., Casanueva, A., Christensen, O. B., Christensen, J. H., Coppola, E., De Cruz, L., Davin, E. L., Dobler, A., Domínguez, M., Fealy, R., Fernandez, J., Gaertner, M. A., García-Díez, M., Giorgi, F., Gobiet, A., Goergen, K., Gómez-Navarro, J. J., Alemán, J. J. G., Gutiérrez, C., Gutiérrez, J. M., Güttler, I., Haensler, A., Halenka, T., Jerez, S., Jiménez-Guerrero, P., Jones, R. G., Keuler, K., Kjellström, E., Knist, S., Kotlarski, S., Maraun, D., van Meijgaard, E., Mercogliano, P., Montávez, J. P., Navarra, A., Nikulin, G., de Noblet-Ducoudré, N., Panitz, H.-J., Pfeifer, S., Piazza, M., Pichelli, E., Pietikäinen, J.-P., Prein, A. F., Preuschmann, S., Rechid, D., Rockel, B., Romera, R., Sánchez, E., Sieck, K., Soares, P. M. M., Somot, S., Srnec, L., Sørland, S. L., Termonia, P., Truhetz, H., Vautard, R., Warrach-Sagi, K., and Wulfmeyer, V.: Regional climate downscaling over Europe: perspectives from the EURO-CORDEX community, Regional Environmental Change, 20, 51, 10.1007/s10113-020-01606-9, 2020.
  * Kay et al. (2015) Kay, J. E., Deser, C., Phillips, A., Mai, A., Hannay, C., Strand, G., Arblaster, J. M., Bates, S. C., Danabasoglu, G., Edwards, J., Holland, M., Kushner, P., Lamarque, J.-F., Lawrence, D., Lindsay, K., Middleton, A., Munoz, E., Neale, R., Oleson, K., Polvani, L., and Vertenstein, M.: The Community Earth System Model (CESM) Large Ensemble Project: A Community Resource for Studying Climate Change in the Presence of Internal Climate Variability, Bulletin of the American Meteorological Society, 96, 1333–1349, 10.1175/BAMS-D-13-00255.1, publisher: American Meteorological Society Section: Bulletin of the American Meteorological Society, 2015.
  * Kendon et al. (2020) Kendon, E. J., Roberts, N. M., Fosser, G., Martin, G. M., Lock, A. P., Murphy, J. M., Senior, C. A., and Tucker, S. O.: Greater Future U.K. Winter Precipitation Increase in New Convection-Permitting Scenarios, Journal of Climate, 33, 7303–7318, 10.1175/JCLI-D-20-0089.1, publisher: American Meteorological Society Section: Journal of Climate, 2020.
  * Lehner et al. (2020) Lehner, F., Deser, C., Maher, N., Marotzke, J., Fischer, E. M., Brunner, L., Knutti, R., and Hawkins, E.: Partitioning climate projection uncertainty with multiple large ensembles and CMIP5/6, Earth System Dynamics, 11, 491–508, 10.5194/esd-11-491-2020, publisher: Copernicus GmbH, 2020.
  * Maher et al. (2020) Maher, N., Lehner, F., and Marotzke, J.: Quantifying the role of internal variability in the temperature we expect to observe in the coming decades, Environmental Research Letters, 15, 054 014, 10.1088/1748-9326/ab7d02, publisher: IOP Publishing, 2020.
  * Maraun and Widmann (2018) Maraun, D. and Widmann, M.: Statistical Downscaling and Bias Correction for Climate Research, Cambridge University Press, URL https://www.ebooks.com/en-us/book/95946434/statistical-downscaling-and-bias-correction-for-climate-research/douglas-maraun/, 2018\.
  * Martinez-Villalobos and Neelin (2021) Martinez-Villalobos, C. and Neelin, J. D.: Climate models capture key features of extreme precipitation probabilities across regions, Environmental Research Letters, 16, 024 017, 10.1088/1748-9326/abd351, publisher: IOP Publishing, 2021.
  * Mauritsen et al. (2019) Mauritsen, T., Bader, J., Becker, T., Behrens, J., Bittner, M., Brokopf, R., Brovkin, V., Claussen, M., Crueger, T., Esch, M., Fast, I., Fiedler, S., Fläschner, D., Gayler, V., Giorgetta, M., Goll, D. S., Haak, H., Hagemann, S., Hedemann, C., Hohenegger, C., Ilyina, T., Jahns, T., Jimenéz-de-la Cuesta, D., Jungclaus, J., Kleinen, T., Kloster, S., Kracher, D., Kinne, S., Kleberg, D., Lasslop, G., Kornblueh, L., Marotzke, J., Matei, D., Meraner, K., Mikolajewicz, U., Modali, K., Möbis, B., Müller, W. A., Nabel, J. E. M. S., Nam, C. C. W., Notz, D., Nyawira, S.-S., Paulsen, H., Peters, K., Pincus, R., Pohlmann, H., Pongratz, J., Popp, M., Raddatz, T. J., Rast, S., Redler, R., Reick, C. H., Rohrschneider, T., Schemann, V., Schmidt, H., Schnur, R., Schulzweida, U., Six, K. D., Stein, L., Stemmler, I., Stevens, B., von Storch, J.-S., Tian, F., Voigt, A., Vrese, P., Wieners, K.-H., Wilkenskjeld, S., Winkler, A., and Roeckner, E.: Developments in the MPI-M Earth System Model version 1.2 (MPI-ESM1.2) and Its Response to Increasing CO2, Journal of Advances in Modeling Earth Systems, 11, 998–1038, 10.1029/2018MS001400, _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1029/2018MS001400, 2019.
  * McKinnon and Deser (2018) McKinnon, K. A. and Deser, C.: Internal Variability and Regional Climate Trends in an Observational Large Ensemble, Journal of Climate, 31, 6783–6802, 10.1175/JCLI-D-17-0901.1, publisher: American Meteorological Society Section: Journal of Climate, 2018.
  * McKinnon and Deser (2021) McKinnon, K. A. and Deser, C.: The Inherent Uncertainty of Precipitation Variability, Trends, and Extremes due to Internal Variability, with Implications for Western U.S. Water Resources, Journal of Climate, 34, 9605–9622, 10.1175/JCLI-D-21-0251.1, publisher: American Meteorological Society, 2021.
  * Moncrieff et al. (2017) Moncrieff, M. W., Liu, C., and Bogenschutz, P.: Simulation, Modeling, and Dynamically Based Parameterization of Organized Tropical Convection for Global Climate Models, Journal of the Atmospheric Sciences, 74, 1363–1380, 10.1175/JAS-D-16-0166.1, publisher: American Meteorological Society Section: Journal of the Atmospheric Sciences, 2017.
  * Müller et al. (2018) Müller, W. A., Jungclaus, J. H., Mauritsen, T., Baehr, J., Bittner, M., Budich, R., Bunzel, F., Esch, M., Ghosh, R., Haak, H., Ilyina, T., Kleine, T., Kornblueh, L., Li, H., Modali, K., Notz, D., Pohlmann, H., Roeckner, E., Stemmler, I., Tian, F., and Marotzke, J.: A Higher-resolution Version of the Max Planck Institute Earth System Model (MPI-ESM1.2-HR), Journal of Advances in Modeling Earth Systems, 10, 1383–1413, 10.1029/2017MS001217, _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1029/2017MS001217, 2018.
  * Pielke et al. (2022) Pielke, Jr, R., Burgess, M. G., and Ritchie, J.: Plausible 2005–2050 emissions scenarios project between 2 °C and 3 °C of warming by 2100, Environmental Research Letters, 17, 024 027, 10.1088/1748-9326/ac4ebf, publisher: IOP Publishing, 2022.
  * Rasmussen et al. (2016) Rasmussen, D. J., Meinshausen, M., and Kopp, R. E.: Probability-Weighted Ensembles of U.S. County-Level Climate Projections for Climate Risk Analysis, Journal of Applied Meteorology and Climatology, 55, 2301–2322, 10.1175/JAMC-D-15-0302.1, publisher: American Meteorological Society Section: Journal of Applied Meteorology and Climatology, 2016.
  * Suarez-Gutierrez et al. (2018) Suarez-Gutierrez, L., Li, C., Müller, W. A., and Marotzke, J.: Internal variability in European summer temperatures at 1.5\hspace0.167em°C and 2\hspace0.167em°C of global warming, Environmental Research Letters, 13, 064 026, 10.1088/1748-9326/aaba58, publisher: IOP Publishing, 2018.
  * Swart et al. (2019) Swart, N. C., Cole, J. N. S., Kharin, V. V., Lazare, M., Scinocca, J. F., Gillett, N. P., Anstey, J., Arora, V., Christian, J. R., Hanna, S., Jiao, Y., Lee, W. G., Majaess, F., Saenko, O. A., Seiler, C., Seinen, C., Shao, A., Sigmond, M., Solheim, L., von Salzen, K., Yang, D., and Winter, B.: The Canadian Earth System Model version 5 (CanESM5.0.3), Geoscientific Model Development, 12, 4823–4873, 10.5194/gmd-12-4823-2019, 2019.
  * Tapiador et al. (2020) Tapiador, F. J., Navarro, A., Moreno, R., Sánchez, J. L., and García-Ortega, E.: Regional climate models: 30 years of dynamical downscaling, Atmospheric Research, 235, 104 785, 10.1016/j.atmosres.2019.104785, 2020.
  * Tatebe et al. (2019) Tatebe, H., Ogura, T., Nitta, T., Komuro, Y., Ogochi, K., Takemura, T., Sudo, K., Sekiguchi, M., Abe, M., Saito, F., Chikira, M., Watanabe, S., Mori, M., Hirota, N., Kawatani, Y., Mochizuki, T., Yoshimura, K., Takata, K., O’ishi, R., Yamazaki, D., Suzuki, T., Kurogi, M., Kataoka, T., Watanabe, M., and Kimoto, M.: Description and basic evaluation of simulated mean state, internal variability, and climate sensitivity in MIROC6, Geoscientific Model Development, 12, 2727–2765, 10.5194/gmd-12-2727-2019, publisher: Copernicus GmbH, 2019.
  * Wood et al. (2021) Wood, R. R., Lehner, F., Pendergrass, A. G., and Schlunegger, S.: Changes in precipitation variability across time scales in multiple global climate model large ensembles, Environmental Research Letters, 16, 084 022, 10.1088/1748-9326/ac10dd, publisher: IOP Publishing, 2021.
  * Yeager et al. (2018) Yeager, S. G., Danabasoglu, G., Rosenbloom, N. A., Strand, W., Bates, S. C., Meehl, G. A., Karspeck, A. R., Lindsay, K., Long, M. C., Teng, H., and Lovenduski, N. S.: Predicting Near-Term Changes in the Earth System: A Large Ensemble of Initialized Decadal Prediction Simulations Using the Community Earth System Model, Bulletin of the American Meteorological Society, 99, 1867–1886, 10.1175/BAMS-D-17-0098.1, publisher: American Meteorological Society Section: Bulletin of the American Meteorological Society, 2018.
  * Zelinka et al. (2020) Zelinka, M. D., Myers, T. A., McCoy, D. T., Po-Chedley, S., Caldwell, P. M., Ceppi, P., Klein, S. A., and Taylor, K. E.: Causes of Higher Climate Sensitivity in CMIP6 Models, Geophysical Research Letters, 47, e2019GL085 782, 10.1029/2019GL085782, _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1029/2019GL085782, 2020.
  * Zhang et al. (2019) Zhang, F., Sun, Y. Q., Magnusson, L., Buizza, R., Lin, S.-J., Chen, J.-H., and Emanuel, K.: What Is the Predictability Limit of Midlatitude Weather?, Journal of the Atmospheric Sciences, 76, 1077–1091, 10.1175/JAS-D-18-0269.1, publisher: American Meteorological Society Section: Journal of the Atmospheric Sciences, 2019.