# B-SDA: Bird’s-eye View Social Distancing Analysis System

Zhengye Yang†, Mingfei Sun†, Hongzhe Ye†, Zihao Xiong†, Gil Zussman†, Zoran
Kostic†
$\dagger$Department of Electrical Engineering, Columbia University, United
States of America

###### Abstract

Social distancing can reduce the infection rates in respiratory pandemics such
as COVID-19. Traffic intersections are particularly suitable for monitoring
and evaluation of social distancing behavior in metropolises. We propose and
evaluate a privacy-preserving social distancing analysis system (B-SDA), which
uses bird’s-eye view video recordings of pedestrians who cross traffic
intersections. We devise algorithms for video pre-processing, object detection
and tracking which are rooted in the known computer-vision and deep learning
techniques, but modified to address the problem of detecting very small
objects/pedestrians captured by a highly elevated camera. We propose a method
for incorporating pedestrian grouping for detection of social distancing
violations. B-SDA is used to compare pedestrian behavior based on pre-pandemic
and pandemic videos in a major metropolitan area. The accomplished pedestrian
detection performance is $63.0\%$ $AP_{50}$ and the tracking performance is
$47.6\%$ MOTA. The social distancing violation rate of $15.6\%$ during the
pandemic is notably lower than $31.4\%$ pre-pandemic baseline, indicating that
pedestrians followed CDC-prescribed social distancing recommendations. The
proposed system is suitable for deployment in real-world applications.

##  I INTRODUCTION

Respiratory viruses such as COVID-19 are spread by individuals who are in
close proximity to each other for some critical period of time. Per CDC
policies, individuals should maintain a social distance of at least 6 feet to
suppress the spread of the virus [25, 12, 11]. Streets and traffic
intersections are locations where such social-distancing violations are prone
to occur. It is desirable to provide precise measurement of social distancing
in city scenarios with a possible use case of providing data that could aid
epidemiological research. Furthermore, it is important to develop this
technology with privacy preservation in mind.

Traffic intersections in metropolises are suitable for deployment of smart-
city sensors, high-speed communications and edge computing nodes, which makes
it possible to collect, process and analyze high-bandwidth data such as videos
used for monitoring the social distancing behavior. The results of this paper
are based on the experiments performed on a test-bed deployed in a major
metropolis, which contains sensor, communications and computing infrastructure
in support of privacy preserving social distancing analysis111This work was
supported in part by National Science Foundation (NSF) under grant
CNS-1827923..

In conventional traffic intersection applications, it is most common to deploy
cameras at low altitudes. This approach has several drawbacks:

  * •

The surveillance area is limited, as dictated by the directionality of low-
altitude cameras.

  * •

Visual occlusions occur frequently for objects which are positioned behind
objects closest to a camera.

  * •

Temporal tracking of pedestrians is challenging due to occlusions.

  * •

Pedestrian face recognition and license plate recognition are possible, both
of which violate personal privacy.

These issues can be resolved by using cameras installed at high elevations,
which record bird’s-eye view videos as in Fig. 1 (right). To facilitate
successful measurement of social distancing using bird’s-eye camera
recordings, two preliminary steps are required: (i) per-frame detection of
pedestrians within the scene [15, 14, 30, 18, 21, 27, 28, 29, 5]; (ii)
reliable tracking of pedestrian trajectories across video frames [23, 2, 37,
36]. The size of pedestrians in bird’s-eye view videos is a function of video
resolution, and can be smaller than $30\times 30$ pixels for $1080p$
recordings. Processing such small objects is a challenge for conventional
object-detection and tracking algorithms. For this research, we acquired a
large bird’s-eye view video dataset, and annotated it for detection of
pedestrians. We customized the object detection method YOLOv4 [5] and the
multiple object tracking method SORT [4], such that they are able to provide
the desirable inference speed and to achieve promising detection and tracking
performance for social distancing applications.

Figure 1: Left: View from a low-elevation camera; Right: bird’s-eye view from
a high-elevation camera. Figure 2: Pipeline for the B-SDA system: (i) collect
raw videos from a bird’s-eye view camera; (ii) do calibration and background
subtraction to alleviate the effect of sub-optimal sensor quality; (iii) get
pedestrian detection results; (iv) get pedestrian tracking information; (v)
analyze pedestrian movement behaviour with social distancing analysis
algorithm.

A number of social distancing analysis prototypes have been recently proposed
[9, 19, 33, 38]. They are based on open-source datasets with low-altitude
camera views, leading to potential privacy violations. Most utilize only the
detection information to analyze social distancing, which is insufficient for
acquisition of statistics such as pedestrian throughput at traffic
intersections. When using only the distance between pedestrians to assess
proximity violations, it is impossible to discriminate between ”safe” social
groups and random people in close proximity to each other. A safe social group
is defined as a collection of people assumed to reside together, such as a
family. Identifying ”safe” social groups requires group tracking, in order not
to declare the participating pedestrians as violators of social distancing.

The main contributions of this paper are:

  * •

Social distancing analysis system based on bird’s-eye view videos (Fig. 2).

  * •

Introduction of a vision-based social group identity validation algorithm, for
bird’s-eye view based social distancing analysis.

  * •

Analysis of social distancing behavior in a major metropolis, based on
privacy-preserving videos recorded before and during the COVID-19 pandemic.

##  II Background

###  II-A Object Detection

Object detection is a computer-vision task that classifies and locates
instances of semantic objects. Most state-of-the-art object detectors are
deep-learning based. Among the prominent approaches, R-CNN [15], Fast R-CNN
[14], Faster R-CNN [30] and Mask R-CNN [18] use a two-stage structure for
object detection, which separates region proposal stage and classification
stage.

By contrast, SSD [21] and YOLO methods [27, 28, 29, 5] use a single-stage
structure with higher inference speed. As stronger backbone networks are
designed and more data augmentation methods are applied, single-stage models
evolved into many variations. Recent improvements of YOLO [5, 28, 29, 39] make
it possible to approach the detection accuracy comparable to R-CNN’s accuracy
without sacrificing YOLO’s inference speed in our traffic scenario.
Considering the speed-accuracy trade-off, we chose YOLOv4 [5] as the baseline
for object detection in this work.

###  II-B Multiple Object Tracking

The task of Multiple Object Tracking (MOT) is largely partitioned into
locating multiple objects, maintaining their identities, and yielding the
individual trajectories given an input video [23].

SORT [4] is a tracking-by-detection algorithm in which the aim is to associate
detection across frames in a video sequence. It performs imperfectly when
tracking through occlusions. DeepSORT [37] algorithm replaces the association
metric with a more informed metric that combines motion and appearance
information by using a convolutional neural network. Tracktor method [2]
accomplishes multi-object tracking with the Faster R-CNN object detector only,
but it under-performs when the objects are moving fast or when the density of
objects is high.

Given the similar performances of these methods tested in our scenario, we
chose SORT [4] as the tracking module due to its simplicity and quick
inference.

###  II-C Social Distancing Analysis

The general aspects of the visual social distancing problem have been proposed
in [8]. With the sudden outbreak of the COVID pandemic, the proposals for
social distancing surveillance have been rapidly emerging.

Given the nature of the social distancing problem, the primal task is to
detect pedestrians and measure the distance between each pair. In early works,
the solutions were based on object detection with distance approximation[9,
19, 38]. Current mainstream object detection frameworks such as Faster
R-CNN[30], Mask R-CNN [18], YOLOv3 [29] and YOLOv4 [5] are widely used. To
collect more useful identity-related statistics, tracking-by-detection MOT
method has been added into the workflow by [26, 16]. Considering the accuracy-
speed tradeoff in tracking algorithms, efficient methods like SORT and
DeepSORT are widely used for social distancing problems. Group information
among crowds is further added to reduce false positives caused by naïvely
chosen distance thresholds, and several works proposed different group
detection mechanisms[31, 35].

All research noted above is based on street-level cameras and dataset [1],
which violates pedestrian privacy. Bird’s-eye view cameras provide an
alternative which achieves privacy preservation and provides a much larger
surveillance area per camera.

###  II-D Group Detection

Due to the inherent social nature of human behavior, interactions typically
happen between small subsets of people referred to as groups [34]. Hall
proposed the proxemic theory which defined distances between two people with
different intimacy levels in the North American culture [17]. Many researchers
investigated measurement metrics for group identification - Rima et
al.proposed similarity features including velocity and tracklets [6];
Francesco et al.conducted proxemics and causality for group identification
[34]; Weina et al.proposed new measurements considering trajectory and
velocity comparisons, and constructed a bottom-up hierarchical clustering
approach which starts from separate individuals and gradually build larger
groups by merging clusters with strongest intergroup closeness [13]. Mahsa et
al.proposed an end-to-end trainable framework that are able to generate
spatio-temporal representation from and extract features of each person and
learn the representation of collective activities [10].

Crowd understanding, or crowd analysis, a topic related to group detection, is
also an active research field. Ning et al.proposed an attention-injective
deformable convolutional network called ADCrowdNet, which could address the
accuracy degradation problem of highly congested noisy scenes [20]. Yuting et
al.developed a network that can handle both detection and crowded counting
without annotation with bounding boxes [22].

##  III Description of the B-SDA System for Measuring and Analysis of Social
Distancing

###  III-A Data Pre-Processing

The work described in this paper is based on bird’s-eye view video recordings.
The use of highly elevated cameras, desirable for privacy protection reasons,
results in small and potentially blurry pedestrians. Video recordings under
various lighting and weather conditions additionally impact the accuracy of
object detection and tracking. To tackle these challenges, we apply data pre-
processing methods - Weighted-Mask Background Subtraction (WMBS) and Video
Calibration (VC).

WMBS constructs the background image from videos acquired by static cameras,
computed as the mean of all $N$ frames [7].

The background image, with a weighted parameter $\alpha$, is subtracted from
the original frames, to obtain the enhanced video shown in Eq. (1):

$F_{b}(I_{r}^{(t)})=I_{r}^{(t)}-\frac{\alpha}{N}\sum_{k=1}^{N}I_{r}^{(k)}.$
(1)

$I_{b}^{(t)}=F_{b}(I_{r}^{(t)})$ represents the output image where
$I_{r}^{(t)}$ is $t$-th frame in the original video, and $\alpha$ is the
weight coefficient.

The benefits of WMBS are:

  * •

Sharpening of the boundaries between objects and the static background.

  * •

Minimization of the influence of environmental factors including brightness,
contrast and weather conditions, while allowing the detection of static
objects.

  * •

Improvement in detection performance, with low additional computational
complexity.

VC transforms original (slanted) bird’s-eye view videos into true/calibrated
bird’s-eye videos that are perpendicular to the ground. It maps a
trapezoidally distorted traffic intersection scene into a rectangular one with
a uniform scale, and is performed prior to object detection and tracking. It
facilitates accurate localization of pedestrians while presenting all objects
”as seen straight from above”. Calibration is achieved by calculating the
homography matrix $M_{ca}$ that maps $I_{b}^{(t)}$ in image coordinates to
$F_{c}(I_{b}^{(t)})$ in real world coordinates. The calibration formula is
shown in Eq. (2).

$F_{ca}(I_{b}^{(t)})=M_{ca}\cdot I_{b}^{(t)}.$ (2)

Center cropping is the final stage in calibration, which removes parts of the
original image without useful content and increases the per-pixel size of
features, to improve the detection performance. The cropped image $I^{(t)}$ is
the input for procedures that follow.

###  III-B Detection and Tracking

The goal of object detection in tracking-by-detection is to provide object
localization and classification information to a Multiple Object Tracking
(MOT) algorithm to associate object identities. The detector provides
$m^{(t)}$ proposed bounding boxes
$D^{(t)}=\\{d_{1}^{(t)},d_{2}^{(t)},...,d_{m^{(t)}}^{(t)}\\}$ from $I^{(t)}$,
and $d_{j}^{(t)}=(dclass_{j}^{(t)},bbox_{j}^{(t)},conf_{j}^{(t)})$ records the
class, location and confidence score of the $j$-th predicted box.

Once the MOT algorithm receives $D^{(t)}$, identity association is performed
to get the tracking state
$S^{(t)}=\\{s_{1}^{(t)},s_{2}^{(t)},...,s_{m^{(t)}}^{(t)}\\}$ for all
$m^{(t)}$ objects in the $t$-th frame.
$s_{j}^{(t)}=(class_{j}^{(t)},roi_{j}^{(t)},id_{j}^{(t)})$ denotes the state
of the $j$-th object in the $t$-th frame, $class_{j}^{(t)}$ is the class of
that object, $roi_{j}^{(t)}$ is the location of that object, and
$id_{j}^{(t)}$ indicates the unique ID for that object.

###  III-C Social distancing analysis

Algorithm 1 Group Validation

0: $\textrm{Object state history: }T^{(t)},\textrm{Violation list: }L^{(t)}$

For violation pair$(i,j)$ in $L^{(t)}$:

$obj_{i}^{(t)}\leftarrow T^{(t)}[i]$

$obj_{j}^{(t)}\leftarrow T^{(t)}[j]$

If Velocity Compare($obj_{i}^{(t)},obj_{(j)}^{t}$) = True

If Trajectory Compare($obj_{i}^{(t)},obj_{j}^{(t)}$) = True

$\textrm{remove }L_{t}[(i,j)]$

return $L_{t}$

The social distancing analysis system continually receives the tracking
information $S^{(t)}$ for each frame. The system keeps updating the whole
tracking state $S$ and extracts useful information for analysis from $S$ to
create the state history $T=\\{T^{(1)},T^{(2)},...,T^{(t)}\\}$. Here,
$T^{(t)}=\\{obj_{1}^{(t)},obj_{2}^{(t)},...,obj_{m^{(t)}}^{(t)}\\}$ and
$obj_{j}^{(t)}=(roi_{j}^{(t)},bp_{j}^{(t)},id_{j}^{(t)},v_{j}^{(t)},traj_{j}^{(t)})$
denote the state of $j$-th object in the $t$-th frame. $roi_{j}^{(t)}$ and
$id_{j}^{(t)}$ are received directly from $s_{j}^{(t)}$ in $S^{(t)}$ which
represents the object bounding box information and the unique ID for this
object. $bp_{j}^{(t)}$ represents the middle point of the bounding box bottom
which can be derived from $roi_{j}^{(t)}$. $traj_{j}^{(t)}$ records all
previous tracking information for this object.

The estimation of real-world distances between objects is simplified by the
bird’s-eye video calibration. The distance of six feet in our videos is
represented by approximately 35 pixels based on our ground measurement. For
original uncalibrated images, there is a significant difference in scale
distortion between X and Y axes. This difference becomes negligible after
calibration has been applied.

Next we create an upper triangular Euclidean distance matrix for all objects
in $T^{(t)}$ to find potential social distancing violation pairs
$L^{(t)}=\\{(bid^{(t)}_{11},bid^{(t)}_{12}),...,(bid^{(t)}_{p1},bid^{(t)}_{p2})\\}$,
where $\\{bid^{(t)}_{n1},bid^{(t)}_{n2}\\}$ denotes the $n$-th violation
object ID pairs of all $p$ pairs. This is the final violation criteria for
most contemporary social distancing analysis systems [9, 19, 33, 38], although
there may still be a number of pairs in $L^{(t)}$ that belong to the same
group.

In order to avoid over-counting the number of violations, we design a cascade-
condition filter shown in Algorithm 1 to validate if a social distancing
violation pair is (incorrectly) indicated, although the pedestrians belong to
the same social group. The assumption made here is that people belonging to
the same (safe) social group are going to maintain the proxemic relationship
when they cross the traffic intersection. The proxemic relationship can be
captured by pedestrian trajectory information, which can be decomposed into
three components: velocity similarity, trajectory similarity[24] and proxemic
stability.

Trajectory similarity is measured by the Euclidean distance between two
pedestrians at the same temporal location shown in Eq. (3). The first order
derivative of a trajectory can be used as velocity estimator.

$\displaystyle sim(a,b)$ $\displaystyle=\frac{\sum_{i\in X_{a}\bigcap
X_{b}}\|(a,b)\|_{2}}{|X_{a}\bigcap X_{b}|}$ (3)

The estimation of object velocity depends heavily on the correct localization
of the bounding boxes. Caused by imperfections of the object detection
algorithm, oscillations in localization could seriously disturb the estimation
of object velocities. To achieve the precise velocity comparison between two
objects, we calculate velocities with Exponentially Weighted Averages shown in
Algorithm 2. Function Get Bp computes the middle point of the bounding box
bottom in frames, and function Find collects frames where a given object is
detected. When exponentially weighted averaging is applied, velocities are
recalculated based on previous and current velocities using the update
parameter $\alpha$.

Assume that there are two pedestrians with velocity vectors $v_{1}$ and
$v_{2}$. To compare the velocity similarity, both magnitude and direction need
to be evaluated. The Cosine Distance ($D_{cos}$) Eq. (4) is a common method to
measure the similarity in vector direction.

$D_{cos}=1-\frac{v_{1}\cdot v_{2}}{\parallel v_{1}\parallel\cdot\parallel
v_{2}\parallel}.\\\ $ (4)

In order to combine the magnitude similarity with the Cosine Distance measure,
it needs to be scaled. We use the magnitude similarity ($D_{Mag}$) which is
inspired by the formula discussed by Rima et al. [6]

$D_{Mag}=\frac{|\parallel v_{1}\parallel-\parallel
v_{2}\parallel|}{\max{(\parallel v_{1}\parallel,\parallel
v_{2}\parallel)}}.\\\ $ (5)

The overall velocity distance $D(v1,v2)$ with weight parameter $\gamma$ is
described in Eq. (6)

$D(v1,v2)=\gamma\cdot D_{cos}+(1-\gamma)\cdot D_{Mag}.\\\ $ (6)

The ratio between standard deviation of distance and trajectory similarity
between two violation candidates captures the stability of the proxemic
relationship, shown in Eq. (7). The function Trajectory Compare is the ”OR”
condition filter between trajectory similarity and trajectory stability, which
aims to be suitable for different group scenarios in a traffic intersection.

$\displaystyle f_{\textrm{stab}}$ $\displaystyle=\frac{std.(a,b)}{sim(a,b)}$
(7)

Algorithm 2 Velocity update

0: $\textrm{Tracking state: }S$

For $obj$ in S:

$bp[obj]=\textrm{Get Bp}(S,obj)$

Find $[t_{0},t_{1},...,t_{n}]$ that $obj$ exists

For $t_{k}$ in $[t_{0},t_{1},...,t_{n}]$

If $k==0$:

$v^{(t_{k})}=0$

Else:

$v^{(t_{k})}=\frac{bp[obj][t_{k}]-bp[obj][t_{k-1}]}{t_{k}-t_{k-1}}$

If apply Exponentially Weighted Averages:

If $k\leq 1$:

$v^{(t_{k})}=v^{(t_{cur})}$

Else $k\leq n$:

$v^{(t_{k})}=\alpha v^{(t_{k})}+(1-\alpha)v^{(t_{k-1})}$

The function Velocity Compare calculates the velocity similarity between two
objects indicated to be in a social-distancing violation. If the velocity
similarity lies within the predefined threshold, these two objects will be
further evaluated by the function Trajectory Compare. If all comparisons pass
the test, these two objects are declared to belong to the same group, and they
are removed from the social distancing violation list $L^{(t)}$. After these
modifications, the list can be used as an index to collect the statistics.

##  IV Experiments

###  IV-A Data Acquisition

The dataset for training has been composed from two resources: (i) public
dataset called Visdrone2019 [40], and (ii) a collection of traffic
intersection videos recorded in New York City, and annotated by our group
(B-SDA dataset). Our videos have been recorded by Hikvision
(DS-2CD5585G0-IZHS) camera. The videos have been recorded at 15 frames per
second using $1920\times 1080$ resolution. The annotation statistics are shown
in Table I. For inference purposes, videos have been systematically recorded
multiple times per day from June 2020 to Febuary 2021 during the pandemic,
with recording schedule and statistics shown in Table II.

TABLE I: Annotation statistics Dataset | Number of Frames | Number of Objects
---|---|---
B-SDA train | 7.4k | 49.7k
B-SDA test | 8.1k | 203.2k
TABLE II: Recording schedule and video statistics during the pandemic Recording Schedule | Number of Frames
---|---
09:00-09:05 | 137.7k
14:00-14:05 | 140.4k
17:30-17:35 | 148.5k
22:00-22:05 | 143.1k

###  IV-B Experimental Setup

The computational setup consists of the operating system Ubuntu 18.04 running
on a cluster of 8 vCPUs, 30GB RAM, and one Tesla P100 GPU.

To reach detection accuracy appropriate for the proposed social distancing
analysis system, we customized YOLOv4 object detection algorithm in the
following ways: (i) it is using a shallower feature map ($\frac{1}{4}$ of
input image size) was chosen to detect small pedestrians; (ii) the scale
factors of the anchor boxes were determined based on experimental studies to
adjust them to the scale of a target class. In the training process of YOLOv4,
the customized YOLOv4 started with the backbone pre-trained on the Imagenet
dataset [32]. Next, it is trained with (a) VisDrone2019 dataset [40] in
$832\times 832$ resolution for 6,000 epochs, followed by (b) B-SDA dataset for
another 6,000 epochs. We used a batch size of $64$ and the learning rate of
$10^{-3}$ with a weight decay of $5\times 10^{-4}$.

For tracking, we used the SORT algorithm [4] for real-time processing without
sacrificing much in accuracy.

The complete group validation is shown in algorithm 1. We use Eq. (4), Eq. (5)
and Eq. (6) to check the velocity similarity. Parameters $\gamma$ and
$\lambda$ are set to 0.1 and 1 respectively. The threshold for velocity
similarity is set to 0.21.

For similarity measurement for trajectories, we use Eq. (3) and Eq. (7). The
social distance threshold for Eq. (3) is $35$. The trajectory stability
threshold for Eq. (7) is set to $0.25$.

###  IV-C Verification of the Group Validation Algorithm

We evaluate the effectiveness of our algorithm for validation of pedestrian
groups.

We annotate groups of pedestrians with larger bounding boxes that cover all
pedestrians within a same group, for $10k$ frames.

In each group bounding box, pedestrians who are within the social distancing
threshold (35 pixels) are the true positives in the group validation
evaluation. The corresponding metrics are shown below.

$\displaystyle\textrm{Precision}=\frac{TP}{TP+FP}$ (8)

$\displaystyle\textrm{Recall}=\frac{TP}{TP+FN}$ (9)

$\displaystyle\textrm{F1-score}=\frac{2\times\text{Precision}\times\text{Recall}}{\text{Precision}+\text{Recall}}$
(10)

TABLE III: Group Validation Performance Traj. Comp. | Vel. Comp. | Precision | Recall | F1
---|---|---|---|---
|  | 0.92 | 0.57 | 0.66
✓ |  | 0.90 | 0.99 | 0.92
✓ | ✓ | 0.86 | 0.96 | 0.88

Table III shows that our algorithm can capture accurate grouping information,
and filter out violation pairs in the same group (Traj. Comp. denotes
Trajectory Compare and Vel. Comp. denotes Velocity Compare). Different
combination of methods are experimented with. We observe that Trajectory
Comparison brings significant improvement to the Recall and to the F1 score,
whereas Velocity Comparison is harmful to the performance. We postulate that
velocity estimation introduces a large amount of noise even when equipped with
the exponentially weighted average. Therefore, we remove the function Velocity
Estimation, and use only the function Trajectory Comparison for further
analysis of social distancing.

##  V Results

###  V-A Detection and Tracking for bird’s-eye view videos

TABLE IV: YOLOv4 performance on the B-SDA Dataset with data pre-processing applied WMBS | CC | AP | mIoU | Precision | Recall
---|---|---|---|---|---
|  | 44.9 | 71.7 | 74.2 | 49.9
✓ |  | 55.1 | 69.8 | 70.9 | 62.9
| ✓ | 58.0 | 68.75 | 84.1 | 62.8
✓ | ✓ | 63.0 | 68.77 | 73.3 | 73.0

Precise object detection and tracking results are essential to our social
distancing system. Table IV shows how data pre-processing methods affect our
customized YOLOv4 model. We use WMBS for Weighted-Mask Background Subtraction
and CC for Center Cropping. With these two methods, YOLOv4 achieves the
highest AP and Recall compared to other combinations. For the traffic
surveillance scenario, considering the traffic safety, recall is more
important than precision. We choose the YOLOv4 model with both pre-processing
methods for the detector.

The overall MOT accuracy is evaluated by the CLEAR metrics [3], where MOTA is
the main performance evaluation score. The tracking performance is evaluated
on the B-SDA test dataset. The detection result is generated by YOLOv4 with
Weighted-Mask Background Subtraction and Center Cropping. For the YOLOv4-SORT
pipeline, we obtain MOTA = 47.65%, MOTP = 71.4%, MT = 60.9%, and ML = 5.8% .

The YOLOv4-SORT pipeline provides accurate object locations and identity
information across all frames, with which we can measure the social distances
between pedestrians. Furthermore, our group validation algorithm can obtain
reliable social distancing analysis.

###  V-B Social Distancing Analysis

Figure 3: Distribution of the duration of social distancing violations (raw
statistics).

Fig. 3 shows the statistical information about the time duration of violations
for all social distancing violators in 458 videos recorded during the COVID-19
pandemic in a major metropolis. For comparison purposes, we also performed the
social distancing analysis on another B-SDA video dataset which was collected
between June and July 2019, before the COVID-19 pandemic.

To simplify the evaluation and visualization of the statistics, we note that
walking pedestrians may not be able to maintain the social distance of exactly
6 feet when crossing a street, which will cause violations that are less than
one second in duration. These short violations would count pedestrians who
behave properly as social distancing violators. Therefore, in the following
analysis and figures we omit all violations which are shorter than 1 second.

The estimation of pedestrian volume (number of pedestrians) relies on the
number of trajectories. ID switches caused by imperfect tracking enlarge the
estimated number of pedestrians. To better estimate the pedestrian volume, the
real world average trajectory length can be used as the reference. We
annotated ground truth for five video segments recorded during the pandemic,
from which we calculated that the real world average pedestrian trajectory
length is $19.11$ seconds. In the 458 recordings, the average pedestrian
trajectory length extracted from tracking inference is equal to $5.63$
seconds. This decrease in the duration of trajectories, from $19$ to $5$
seconds, reflects the fact that the ID switches enlarge the perceived number
of pedestrians. To alleviate this effect, we apply (11) to estimate the actual
pedestrian volume ($EV$). $TC$ stands for the total number of trajectories,
$AT_{GT}$ for the average trajectory length from the ground truth, and
$AT_{Infer}$ stands for the average trajectory length obtained from inference
results.

$\displaystyle EV=TC\cdot\frac{AT_{\textrm{GT}}}{AT_{\textrm{Infer}}}$ (11)

Fig. 4 shows the histogram of the duration of violations, which is filtered by
kernel density estimation (KDE) to obtain the distribution shown in Fig. 5.
Over 75 percent of violations are shorter than 4 seconds.

Figure 4: The distribution of social distancing violation duration. Figure 5:
The distribution of social distancing violation duration (estimated by KDE
using Gaussian kernel with 0.08 bandwidth).

Since our pandemic videos were recorded daily on a repetitive time schedule,
we can generate probability density distributions for each of the recording
periods. These distributions smoothed by KDE are shown in Fig. 6. There is
limited difference between different time slots. To check if the behavior of
the pedestrians is affected by the time of the day, we calculate the velocity
direction for each violation pair, and we record the angular difference in
directions to better depict the violation situation. The corresponding
distributions are shown in Fig. 7, which again indicate that the time of day
does not impact the statistics.

Figure 6: Probability distribution for the social distancing violation
duration in different times of the day (estimated by KDE with Gaussian
kernel). Figure 7: Distributions of the angle between moving directions of two
pedestrians in a violation pair at different times of day (estimated by KDE
with Gaussian kernel). Figure 8: Distribution of the angle between moving
directions of two pedestrians in a violation pair (estimated by KDE with
Gaussian kernel).

Fig. 8 shows the probability distribution of the angle between moving
directions of two pedestrians in a violation pair before and during the
pandemic. We define face-to-face violation happens when the difference of
velocity direction is larger than 150 degrees. Before the pandemic, 42.3% of
violations are face-to-face. During the pandemic, the distribution clearly
indicates that pedestrians are aware of higher chances of getting infected
when violating social distancing, and are thus more cautious when walking
towards each other. They deliberately enlarge the distance between each other
or even try to change the direction, which dramatically decreases the
percentage of face-to-face violations from 42.3% to 20.7%.

Understanding the variability in violations as a function of different times
of the day is important. We use a histogram to visualize the statistics of
average per minute violations, at different times of day, in Fig. 9. The
figure shows that there are more than $0.7$ violations per minute during day
time (8:00 a.m. to 20:00 p.m.) and less than $0.1$ violations per minute
during night time (20:00 p.m. to 08:00 a.m.). Considering that people are more
likely to come into contact with each other when crowd density is high, it
makes sense that the average number of violations is higher during the day
time.

Figure 9: The number of people who violate social distancing at different
times of day.

Fig. 10 shows the comparison of the proportion of social distancing violations
between 2020 (during the pandemic) and 2019 (before the pandemic). One can
observe that if we measured the last year’s data using current social
distancing rules, 31.4% of people would be judged as rule violators. However,
using the same criteria, only 15.6% of people are judged as rule violators
during the pandemic in 2020. This means that during the pandemic people are
aware of social distancing rules, and that they deliberately keep distance
from each other. Additionally, home quarantine orders reduce the density of
pedestrians on the streets, and the reduced density provides more space for
pedestrians to deliberately perform social distancing.

Figure 10: Pedestrian behavior before and after the social distancing protocol
was announced, showing the percentages of rule following and rule breaking
occurrences. Note that all rule breaking incidents shorter than 1 second are
excluded from this graph.

###  V-C The Speed of Inference

TABLE V: Inference Speed Module | Average Frames Per Second (FPS)
---|---
Detection (YOLOv4 FP32) | 11.07
Detection (YOLOv4 FP16) | 30.90
Tracking (SORT) | 553.83
Social Distancing Analysis | 877.78
Overall System (FP32) | 10.72
Overall System (FP16) | 28.32

The inference speed of individual model components is shown in Table V. Note
that FP16 TensorRT implementation of YOLOv4 is three times faster than FP32
implementation, without observable loss in object detection accuracy between
the two quantization settings.

##  VI Conclusion

We developed B-SDA, a privacy-preserving system for measurement and analysis
of social distancing behavior in traffic intersections based on bird’s-eye
view video recordings. The system relies on customized deep-learning based
object detection and tracking techniques, modified and retrained to be able to
detect very small pedestrians. We introduced a group validation technique to
eliminate false positives in detecting social distancing violations, which
takes into account movement characteristics including speed, orientation and
trajectory to evaluate the social distancing behavior. We collected,
conditioned and annotated several hundred videos of an intersection in New
York City before and during the COVID-19 pandemic. The videos were processed
and social distancing analysis was performed. The results were presented in
form of probability functions which capture the type and density of social
distancing violations at different times of day and a variety of conditions.
Quantitative results correlate well with anticipated and visually observed
pedestrian behavior, most dramatically represented by the fact that social
distancing violations are estimated to be much less frequent during the
pandemic ($15.6\%$) when compared to pre-pandemic statistics ($31.4\%$). The
analysis indicates that the proposed technique based on bird’s-eye view videos
is a credible candidate for analysis of social distancing, which could become
a part of a real-time epidemiological tool for managing respiratory pandemics.
A particular strength of the proposed technique is the privacy-preserving
feature, due to the small of pedestrians in bird’s-eye view videos. This
provides additional motivation to innovate in the area of small object
detection and constrained trajectory tracking.

## References

  * [1] B. Benfold and I. Reid.  Stable multi-target tracking in real-time surveillance video.  In CVPR 2011, pages 3457–3464, 2011.
  * [2] P. Bergmann, T. Meinhardt, and L. Leal-Taixe.  Tracking without bells and whistles.  arXiv e-prints, page arXiv:1903.05625, Mar. 2019.
  * [3] K. Bernardin and R. Stiefelhagen.  Evaluating multiple object tracking performance: the clear mot metrics.  EURASIP Journal on Image and Video Processing, 2008:1–10, 2008\.
  * [4] A. Bewley, Z. Ge, L. Ott, F. Ramos, and B. Upcroft.  Simple online and realtime tracking.  2016 IEEE International Conference on Image Processing (ICIP), Sep 2016.
  * [5] A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. Liao.  YOLOv4: Optimal Speed and Accuracy of Object Detection.  arXiv e-prints, page arXiv:2004.10934, Apr. 2020.
  * [6] R. Chaker, Z. A. Aghbari, and I. N. Junejo.  Social network model for crowd anomaly detection and localization.  Pattern Recogn., 61(C):266–281, Jan. 2017.
  * [7] C. COW.  Averaging video frames, 2012 (accessed July 14, 2020).
  * [8] M. Cristani, A. Del Bue, V. Murino, F. Setti, and A. Vinciarelli.  The visual social distancing problem.  IEEE Access, 8:126876–126886, 2020.
  * [9] deepak.  Social-distancing-ai.  https://github.com/deepak112/Social-Distancing-AI/tree/08a9a21ccf8ced3e6ff270628cb1c9b21a55fbee, 2020\.  Online; accessed 13 July 2020.
  * [10] M. Ehsanpour, A. Abedin, F. Saleh, J. Shi, I. Reid, and H. Rezatofighi.  Joint Learning of Social Groups, Individuals Action and Sub-group Activities in Videos.  arXiv e-prints, page arXiv:2007.02632, July 2020.
  * [11] C. for Disease Control and Prevention.  Public health guidance for community-related exposure.  https://www.cdc.gov/coronavirus/2019-ncov/php/public-health-recommendations.html, 2020\.  Online; accessed 31 July 2020.
  * [12] C. for Disease Control and Prevention.  Social distancing.  https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-sick/social-distancing.html, 2020\.  Online; accessed 31 July 2020.
  * [13] W. Ge, R. T. Collins, and R. B. Ruback.  Vision-based analysis of small groups in pedestrian crowds.  IEEE Trans. Pattern Anal. Mach. Intell., 34(5):1003–1016, May 2012\.
  * [14] R. Girshick.  Fast r-cnn.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015.
  * [15] R. Girshick, J. Donahue, T. Darrell, and J. Malik.  Region-based convolutional networks for accurate object detection and segmentation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(1):142–158, 2016.
  * [16] S. Gupta, R. Kapil, G. Kanahasabai, S. S. Joshi, and A. S. Joshi.  Sd-measure: A social distancing detector.  In 2020 12th International Conference on Computational Intelligence and Communication Networks (CICN), pages 306–311. IEEE, 2020.
  * [17] E. Hall, E. Hall, and C. P. C. L. of Congress).  The Hidden Dimension.  Anchor books. Doubleday, 1966.
  * [18] K. He, G. Gkioxari, P. Dollar, and R. Girshick.  Mask r-cnn.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
  * [19] LandingAI.  Landing ai creates an ai tool to help customers monitor social distancing in the workplace.  https://landing.ai/landing-ai-creates-an-ai-tool-to-help-customers-monitor-social-distancing-in-the-workplace, 2020\.  Online; accessed 13 July 2020.
  * [20] N. Liu, Y. Long, C. Zou, Q. Niu, L. Pan, and H. Wu.  ADCrowdNet: An Attention-injective Deformable Convolutional Network for Crowd Understanding.  arXiv e-prints, page arXiv:1811.11968, Nov. 2018.
  * [21] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg.  Ssd: Single shot multibox detector.  In European conference on computer vision, pages 21–37. Springer, 2016.
  * [22] Y. Liu, M. Shi, Q. Zhao, and X. Wang.  Point in, box out: Beyond counting persons in crowds.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6462–6471, 2019.
  * [23] W. Luo, J. Xing, A. Milan, X. Zhang, W. Liu, and T. Kim.  Multiple object tracking: A literature review.  Artificial Intelligence, 293:103448, 2014.
  * [24] N. Magdy, M. A. Sakr, T. Mostafa, and K. El-Bahnasy.  Review on trajectory similarity measures.  In 2015 IEEE seventh international conference on Intelligent Computing and Information Systems (ICICIS), pages 613–619. IEEE, 2015.
  * [25] L. Matrajt and T. Leung.  Evaluating the effectiveness of social distancing interventions against covid-19.  medRxiv, 2020.
  * [26] N. S. Punn, S. K. Sonbhadra, and S. Agarwal.  Monitoring covid-19 social distancing with person detection and tracking via fine-tuned yolo v3 and deepsort techniques.  arXiv preprint arXiv:2005.01385, 2020.
  * [27] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi.  You only look once: Unified, real-time object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016.
  * [28] J. Redmon and A. Farhadi.  Yolo9000: Better, faster, stronger.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6517–6525, 2017.
  * [29] J. Redmon and A. Farhadi.  Yolov3: An incremental improvement.  Computer Vision and Pattern Recognition, 2018.
  * [30] S. Ren, K. He, R. Girshick, and J. Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6):1137–1149, 2017.
  * [31] M. Rezaei and M. Azarmi.  Deepsocial: Social distancing monitoring and infection risk assessment in covid-19 pandemic.  Applied Sciences, 10(21):7514, 2020.
  * [32] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  arXiv e-prints, page arXiv:1409.0575, Sept. 2014.
  * [33] N. Singh Punn, S. K. Sonbhadra, and S. Agarwal.  Monitoring COVID-19 social distancing with person detection and tracking via fine-tuned YOLO v3 and Deepsort techniques.  arXiv e-prints, page arXiv:2005.01385, May 2020.
  * [34] F. Solera, S. Calderara, and R. Cucchiara.  Structured learning for detection of social groups in crowd.  In 2013 10th IEEE International Conference on Advanced Video and Signal Based Surveillance, pages 7–12, 2013.
  * [35] P. Sun, G. Draughon, and J. Lynch.  An autonomous approach to measure social distances and hygienic practices during covid-19 pandemic in public open spaces.  arXiv preprint arXiv:2011.07375, 2020.
  * [36] S. Sun, N. Akhtar, H. Song, A. Mian, and M. Shah.  Deep affinity network for multiple object tracking.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(1):104–119, 2018.
  * [37] N. Wojke, A. Bewley, and D. Paulus.  Simple online and realtime tracking with a deep association metric.  In 2017 IEEE international conference on image processing (ICIP), pages 3645–3649. IEEE, 2017.
  * [38] D. Yang, E. Yurtsever, V. Renganathan, K. A. Redmill, and Ü. Özgüner.  A Vision-based Social Distancing and Critical Density Detection System for COVID-19.  arXiv e-prints, page arXiv:2007.03578, July 2020.
  * [39] Z. Zheng, P. Wang, W. Liu, J. Li, R. Ye, and D. Ren.  Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression.  arXiv e-prints, page arXiv:1911.08287, Nov. 2019.
  * [40] P. Zhu, L. Wen, D. Du, X. Bian, Q. Hu, and H. Ling.  Vision Meets Drones: Past, Present and Future.  arXiv e-prints, page arXiv:2001.06303, Jan. 2020.


# Transferability and explainability of deep learning emulators for regional
climate model projections: Perspectives for future applications

###### Abstract

Regional climate models (RCMs) are essential tools for simulating and studying
regional climate variability and change. However, their high computational
cost limits the production of comprehensive ensembles of regional climate
projections covering multiple scenarios and driving Global Climate Models
(GCMs) across regions. RCM emulators based on deep learning models have
recently been introduced as a cost-effective and promising alternative that
requires only short RCM simulations to train the models. Therefore, evaluating
their transferability to different periods, scenarios, and GCMs becomes a
pivotal and complex task in which the inherent biases of both GCMs and RCMs
play a significant role. Here we focus on this problem by considering the two
different emulation approaches proposed in the literature (PP and MOS,
following the terminology introduced in this paper). In addition to standard
evaluation techniques, we expand the analysis with methods from the field of
eXplainable Artificial Intelligence (XAI), to assess the physical consistency
of the empirical links learnt by the models. We find that both approaches are
able to emulate certain climatological properties of RCMs for different
periods and scenarios (soft transferability), but the consistency of the
emulation functions differ between approaches. Whereas PP learns robust and
physically meaningful patterns, MOS results are GCM-dependent and lack
physical consistency in some cases. Both approaches face problems when
transferring the emulation function to other GCMs, due to the existence of
GCM-dependent biases (hard transferability). This limits their applicability
to build ensembles of regional climate projections. We conclude by giving some
prospects for future applications. This work has been submitted to Artificial
Intelligence for the Earth Systems. Copyright in this Work may be transferred
without further notice.

Regional Climate Model (RCM) emulators are a cost-effective emerging approach
for generating comprehensive ensembles of regional climate projections.
Promising results have been recently obtained using deep learning models.
However, their potential to capture the regional climate dynamics and to
emulate other periods, emission scenarios, or driving Global Climate Models
(GCMs) remains an open issue that affects their practical use. This study
explores the potential of current emulation approaches incorporating new
eXplainable Artificial Intelligence (XAI) evaluation techniques to assess the
reliability and transferability of the emulators. Our findings show that the
different global and regional model biases involved in the different
approaches play a key role for transferability. Based on the results obtained
we provide some prospects on potential applications of these models in
challenging problems.

##  1 Introduction

Regional111This work has been submitted to Artificial Intelligence for the
Earth Systems. Copyright in this Work may be transferred without further
notice. Climate Models (RCMs, Giorgi (2019)) are sophisticated tools widely
used to produce high-resolution regional climate projections. They work by
numerically solving a set of physical equations representing regional
atmospheric processes and interactions with other components, such as land,
over a limited continental region. RCMs are driven at their boundaries by the
coarse output of a Global Climate Model (GCM, Phillips (1956)), a process
often referred to as dynamical downscaling. A variety of studies (Rummukainen,
2016; Soares and Cardoso, 2018; Molina et al., 2022; Cardoso and Soares, 2022)
have assessed the added value of RCMs, pointing to a better representation of
the local scale as compared to their driving GCMs. Therefore, RCM simulations
constitute a valuable line of evidence in assessing the risks and adaptation
strategies related to climate change at the regional scale (IPCC, 2022).

The COordinated Regional climate Downscaling EXperiment (CORDEX) coordinates
the generation of regional climate projections worldwide, based on multi-model
ensembles of RCM simulations spanning different sources of uncertainty
(including those arising from the driving GCM or the emission scenario, among
others Jacob et al. (2020); Diez-Sierra et al. (2022)). However, covering the
large number of potential scenario-GCM-RCM combinations is an enormous
computational challenge. As a result, the limited availability of CORDEX
simulations in some regions hinders the comprehensive assessment uncertainty
in regional climate modeling (Kendon et al., 2010; Fernández et al., 2019).
This has led the regional climate modelling community to look for alternatives
to these costly simulations, especially as they approach the km-scale.

Empirical-Statistical Downscaling (ESD) has traditionally been a cost-
effective alternative/complement to dynamical downscaling Maraun and Widmann
(2018). ESD techniques rely on observations to learn the relationship between
large-scale meteorological fields (typically from reanalysis datasets) and
local surface variables of interest, such as temperature and precipitation.
While the development of advanced machine learning techniques, such as deep
Convolutional Neural Networks (CNNs, (LeCun et al., 1995)), opens up the
potential for downscaling over large areas (Baño-Medina et al., 2021), the
lack of sufficient observed data available for training in many regions
remains as one of the main drawbacks of ESD that limits its applicability,
particularly as finer spatial resolutions are considered.

An alternative approach recently introduced to overcome this problem is
statistical RCM emulators. These models require an existing (relatively short)
RCM simulation (driven by a particular GCM) to train the emulator, which
learns the mapping between upper-air large-scale fields and surface target
variables from the RCM. Two different RCM emulation approaches have been
recently introduced in the literature, referred to as “perfect” and
“imperfect” Erlandsen et al. (2020); Boé et al. (2022); Hobeichi et al.
(2023). Both use surface high-resolution RCM variables (typically temperature
and/or precipitation) as target, or predictand, but differ in the predictors
used. The “perfect” approach uses a set of informative upscaled large-scale
variables from the same RCM and can be therefore considered a hybrid
implementation of the Perfect Prognosis (PP) downscaling approach, using
observations (in this case pseudo-observations within the RCM world) for both
predictors and predictand Maraun and Widmann (2018). This approach maximizes
the day-to-day correspondence and physical consistency between the input-
output pairs, as both come from the same model. Here, we use well-established
terminology in ESD and refer to this approach as “PP” (from perfect
prognosis). On the contrary, the so-called ‘imperfect” approach uses the
driving GCM fields as predictors, thus coping with the lack of perfect day-to-
day correspondence and model biases in the learning process. Similarly to the
previous case, using standard terminology we refer to this approach as “MOS”,
in an analogy with the MOS (Model Output Statistics) downscaling approach,
which deals with model biases during learning.

Several studies have analyzed both emulation approaches independently (Doury
et al., 2022; Hobeichi et al., 2023) or jointly (Boé et al., 2022; van der
Meer et al., 2022), based on the comparison of particular evaluation
metrics/indices between the emulated and target RCM fields. These studies show
promising results to emulate an intermediate temporal period for the same GCM-
RCM pair, particularly for the MOS approach, since the PP one inherit GCM-RCM
biases that affect the emulated fields. However, evaluating emulators is a
challenging task due to the complexity (black box nature) of the underlying
deep learning models, so it is important to be able to analyse the inner
functioning of these models (e.g., to analyze the predictor-predictand
patterns learned) to contextualise the results. Not including this type of
analysis may lead to an incomplete assessment of the emulation capabilities of
both PP and MOS approaches. Some recent studies have explored the application
of eXplainable Artificial Intelligence (XAI) in conventional statistical
downscaling tasks to e.g., assess the physical consistency of the predictor
patterns used by the models for inference (González-Abad et al., 2023; Baño-
Medina, 2020; Balmaceda-Huarte et al., 2023; Rampal et al., 2022). This new
evaluation dimension allows for a better understanding of the capabilities and
limitations of CNNs and may allow to interpret how the different GCM and RCM
biases may affect both emulation approaches.

One of the most promising aspects of emulators is that they could be applied
to complete the scenario-GCM-RCM matrix from partial simulations, ideally from
a single scenario-GCM pair. Ideally, the emulator should capture the regional
climate dynamics of the GCM and be transferable. This means that it should be
able to emulate other periods, emission scenarios or even driving GCMs than
those considered in the training phase. However, this remains an open issue
that affects the practical use of RCM emulators for climate change
applications.

In this work, we assess the transferability of PP and MOS statistical RCM
emulators based on state-of-the-art CNNs. To this aim we combine both standard
and new XAI-based evaluation techniques, which allows us to measure the
trustworthiness of the emulators while deepening into the understanding of the
transferability of each approach to other time period or emission scenario
(soft transferability), or driving GCM (hard transferability). Based on the
current state of knowledge, we conclude by giving some perspectives for future
applications of these methods in problems where they can facilitate progress
as an alternative (or complementary) to RCMs.

##  2 Data and Methods

This study focuses on the Alps region (Figure 1) to test whether the emulators
can reproduce certain fine-scale atmospheric processes well reproduced by
RCMs, such as the influence of orography or coastal temperature gradients.

Following previous work (Doury et al., 2022; Boé et al., 2022), we use
simulations from the ALADIN63 RCM over Europe at a spatial resolution of
0.11∘, provided by EURO-CORDEX (Jacob et al., 2020) for the historical
experiment (1980-2005) and RCP8.5 scenario (2006-2100), which span a wide
range of climatic conditions. These simulations are available at the Earth
System Grid Federation (ESGF) for a total of four driving GCMs: NorESM1-M
(Bentsen et al., 2013), CNRM-CM5 (Voldoire et al., 2013), MPI-ESM-LR (Müller
et al., 2018) and HadGEM2-ES (Bellouin et al., 2011). Here, we consider the
first three, to which we will hereafter refer by the names of their respective
modeling institutions for brevity: NorESM, CNRM, and MPI.

###  2.1 Predictors and predictand

As predictand, we use daily near-surface air temperature of ALADIN63 at the
original 0.11∘ horizontal resolution, driven by the corresponding GCM under
the historical and the RCP8.5 scenarios.

To establish the relationships between low-resolution predictors and high-
resolution predictand variables, we exploit the advantage of deep learning
algorithms to automatically select relevant features from high-dimensional
predictor spaces Baño-Medina (2020). Here, we use a set of daily mean
atmospheric variables typically used as predictors for downscaling purposes
(Brands et al., 2013; Gutiérrez et al., 2013, 2019; Baño-Medina et al., 2020;
Quesada-Chacón et al., 2022): geopotential height (500, 700 hPa), specific
humidity, air temperature and both zonal and meridional wind velocities at 3
different pressure levels (500, 700 and 850 hPa).

Predictor datasets (GCMs and upscaled RCM) were re-gridded by means of a first
order conservative remapping to a common spatial resolution of 1.5∘. This
leads to a 4-dimensional (time, latitude, longitude, variable) predictor field
of dimensions $N\times 10\times 13\times 14$, with $N$ being the number of
days in the training sample. In some experiments, we considered bias adjusted
GCM predictors obtained by a simple monthly adjustment of mean values
(relative to the upscaled RCM).

###  2.2 Experimental framework

We analyze and intercompare the two approaches introduced in the literature
for statistical RCM emulation, referred to as “perfect” and “imperfect” Boé et
al. (2022). Note that we refer to these approaches as “PP” and “MOS”,
respectively, based on the standard terminology used in statistical
downscaling Maraun and Widmann (2018). Figure 1 provides a schematic
illustration of these two approaches together with the Convolutional Neural
Network (CNN) model used in this paper (see Section 2.3).

Figure 1:  Schematic representation of the deep statistical emulator training
workflows for the (a) MOS and (b) PP approaches. Arrows indicate the workflow.
Details on the convolutional neural network model used are included in the
figure: numbers on top of the convolutional layers represent their size,
numbers between layers represent filter size and numbers in parenthesis
indicate the number of filters used.

In the MOS emulator approach, predictors are taken directly from the driving
GCM (Figure 1a, pink lines). The main shortcoming of this approach is the
marginal correspondence between predictors and predictand fields (see Section
3.1), since the RCM is driven by the GCM at the boundaries of the domain, but
develops its own dynamics within the domain. On the other hand, the PP
emulator approach is trained with predictors obtained by upscaling the
corresponding RCM fields (Doury et al., 2022), so both predictors and
predictand are physically consistent and have perfect day-to-day
correspondence. This is illustrated in Figure 1b (blue lines).

In this study, we quantify and compare the strength of the day-to-day
correspondence between the MOS and PP approaches. To do so, we computed the
correlation of surface temperature at each RCM grid-box with the predictor
fields for all predictor grid-boxes. Then, we selected the maximum value
obtained in the spatial field, which typically corresponds to a nearby
predictor grid-box. Correlations are computed separately for winter (DJF) and
summer (JJA), partially eliminating the correlation arising solely from the
annual cycle.

The experimental evaluation framework consists of three phases: training,
evaluation and transferability. Building upon previous work by Doury et al.
(2022), methods are trained using both the historical period (1996-2005,
historical experiment) and a far-future period (2090-2099, RCP8.5 scenario) as
depicted in Figure 1. As a result, we depart from the common practice of
training in the historical period and testing in a future period. This allows
us to separate the extrapolation capability related to the stationarity
assumption from our analysis (Hernanz et al., 2022; Doury et al., 2022), which
is actually more relevant for standard ESD methods, as they rely on
observations to project into the future (Baño-Medina et al., 2022).

In the evaluation phase, the methods trained with predictors from the GCM1
(MOS) or the upscaled GCM1-RCM (PP) are cross-validated by applying them to
the same predictor sources, but from a different mid-future period (2041-2050;
see Figure 2a,b).

In the transferability phase (Figure 2c,d), the performance of the trained
methods is assessed by applying them to new predictor sources, arising either
from the same driving GCM (here referred to as soft or intra-GCM
transferability) or from different ones (hard or inter-GCM transferability).

Note that soft transferability is a special case for PP methods (which are
trained using upscaled GCM-RCM predictors) to test their performance when
applied directly to the GCM predictors of the same driving model used in the
training phase. Large-scale discrepancies (biases) between GCM and upscaled
RCM predictors may influence the emulated fields (Doury et al., 2022). To
overcome these discrepancies, there is the possibility to bias adjust (BA) the
GCM fields using as reference the upscaled RCM variables Boé et al. (2022).
Figure 2c illustrates the alternative use of raw or BA predictors with the two
dashed lines.

In hard transferability, the model trained with GCM1-RCM predictors is applied
to a new GCM (GCM2) and, in principle, the emulator is expected to emulate the
output of GCM2-RCM; note that this is challenging due to the different biases
affecting the training and emulation phases (GCM1 and GCM2, respectively).

Figure 2:  Experimental evaluation workflows to assess the performance of RCM
emulators in (a-b) cross-validation and (c-d) transferability conditions for
both PP and MOS approaches. Hard (inter-GCM) transferability (d) assesses the
performance of RCM emulators when applied to predictors from different GCMs.
Soft (intra-GCM) transferability (c) is special case for PP methods when
applied to predictors from the same driving GCM used for the training phase;
in this case, raw or bias adjusted (relative to the upscaled RCM) predictors
from the GCM can be used. Arrow colors indicate the training approach
(consistent with Figure 1: magenta for PP and blue for MOS), using solid for
cross-validation and dashed for transferability.

###  2.3 Deep learning models

In this work we use Convolutional Neural Networks (CNNs) which are able to
automatically infer complex spatial patterns from the input atmospheric
spaces. In particular, we use the implementation proposed in Baño-Medina et
al. (2020), known as DeepESD. This model has been successfully used for
downscaling purposes in the European continent for both precipitation and
temperature fields (Baño-Medina et al., 2020, 2022). Additionally, DeepESD has
been analyzed with XAI techniques and proved able to learn plausible and
coherent predictor-predictand links (Baño-Medina, 2020). Explainability is a
key element of this study, used to understand the advantages and limitations
of the two RCM emulator approaches.

DeepESD is a CNN composed of three convolutional layers (of 50, 25 and 10
filter maps, respectively) followed by a single dense layer; the particular
configuration used is shown in Figure 1. Each convolution is followed by a set
of Rectified Linear Units (ReLU) to allow the statistical emulator to learn
complex non-linear atmospheric predictor patterns. The feature maps of the
last hidden layer are flattened to build a dense connection with the output
neurons, which correspond to the 8836 land gridpoints of ALADIN63 fields over
the Alpine domain. The input layer is a stacked 4-D (variable, lon-lat grid,
day) predictor field, which feeds the hidden structure of the CNN.

On the more technical side, we use an Adam optimizer (Kingma and Ba, 2014) to
calibrate the model, with a batch size of 100, and a learning rate of 1E-4. We
lean on a single NVIDIA Tesla V100 GPU with 32GB of memory to perform both
calibration and inference.

We perform early-stopping with a patience of 30 epochs by randomly separating
10% of the training data as our validation dataset to avoid overfitting. That
is, when the loss in the validation dataset does not decrease in the next 30
epochs, then the network stops training.

###  2.4 Explainable artificial intelligence (XAI)

Neural networks are seen as “black-boxes” due to the complex operations
occurring in their inner layers, hindering interpretability and raising
distrust in the results these models produce. To overcome this aspect, several
XAI techniques have been recently developed to gain understanding about the
underlying patterns inferred by deep neural networks. This is key to the task
explored in this study, where the community aims to develop trustworthy and
consistent RCM emulators to replace physics-based models. Particularly, we
lean on saliency maps, which are spatial representations of the relevance of
the input features to the model predictions.

The relevance of a variable is predominantly measured by computing the
gradients of the output space relative to the input space. These gradients are
back-propagated through the hidden layers of the network and visually
displayed in the form of saliency maps. Here, we follow previous work in
climate-related applications (Kondylatos et al., 2022; González-Abad et al.,
2023), and compute the saliency maps using the Integrated Gradients (IG)
algorithm (Sundararajan et al., 2017). As a result, for each particular
location in the predictand, the saliency maps have the same dimensions as the
input features, which are 4-D arrays in our case.

In alignment with earlier studies (Toms et al., 2021; Mamalakis et al., 2022)
we post-process the “raw” gradients to enable a comparison across
samples/days. To compute the saliency map of a particular day we follow five
steps:

(1) we take the absolute value since we are interested in the relevance of the
features regardless of their sign; (2) we follow previous work (Toms et al.,
2021; Mamalakis et al., 2022; González-Abad et al., 2023) and filter the
lowest values to avoid gradient shattering by using a threshold of 1.5E-3; (3)
we compute the percentage for each saliency map individually by dividing the
value of each feature by the total sum of all features; (4) we filter again
the gradients with a threshold of 1.5E-3 which now means 0.15% relevance; (5)
we compute again the percentage of each saliency map.

Finally, we aggregate the saliency maps by averaging the values over the
training period, resulting in a collection of maps, each representing a
predictor variable.

In this study, we illustrate the changing relevance patterns of the spatial
predictors in four predictand grid-boxes located over France, the Southeastern
Alps, Sardinia and Poland.

##  3 Results

###  3.1 Predictor-predictand correlation

The strength of the day-to-day correspondence between the predictors and the
target predictand (RCM near-surface temperature) is depicted by the temporal
correlation maps of Figure 3, showing an illustrative example for a set of
predictor variables and for the NorESM-driven ALADIN63 experiment (similar
results are obtained for other driving models).

Figure 3:  Pearson temporal correlation of the daily series of the NorESM-
driven ALADIN63 surface temperature (predictand) and different predictors (in
rows) from ALADIN63 (RCM-upscaled, RCM-U) and the NorESM GCM for winter (left)
and summer (right). Values represent the maximum correlations obtained for
each of the RCM predictand gridboxes (from the correlations with all predictor
gridbox values). The numbers on top of each panel indicate the maximum / mean
spatial values.

As expected, in general, upper-air, large-scale temperature fields show the
highest correlation with near-surface temperature. Moisture shows also
widespread high correlations, while winds show lower correlations and a
seasonal pattern. This is consistent with e.g. westerlies (positive ua850)
advecting relatively warm air from the ocean in winter and cool air in summer.

Apart from the overall correspondence of the different predictors with the
target variable, predictors from the GCM (second and fourth columns) show
systematically lower correlations than those from the upscaled RCM (first and
third columns). To aid in the comparison, the maximum correlation attained and
the spatial mean are shown for each panel. Moisture (hus850) in summer is the
only exception to the degradation of the predictor-predictand relationship in
the GCM. The overall pattern with a higher correlation over northeastern
Europe appears shifted to central Europe in the GCM predictors.

These results reveal that, in general, the predictors from the PP approach are
more informative for the deep emulator methods used in this work than those
from the MOS approach. However, the latter may have some advantages since the
CNN is learning directly the relationship between the driving GCM and the
target RCM variable. In this way, the different biases can be accounted for
directly, in analogy to the MOS approach for statistical downscaling Maraun
and Widmann (2018).

###  3.2 Cross-Validation

The different models are trained using historical (1996-2005) and far-future
RCP8.5 (2090-2099) simulations, considering as predictors the upscaled RCM
large-scale fields (PP approach) or the GCM fields (MOS approach) and using
the corresponding RCM near-surface temperature as predictand. For the
evaluation of the different approaches (Figure 2a,b) we consider the mid-
future RCP8.5 (2041-2050) period.

Figure 4 shows the evaluation metrics (annual and seasonal biases, the RMSE
and the interannual correlation, in rows) for emulated surface temperature
over the test mid-future period for the ALADIN63 (RCM) simulations driven by
the NorESM model (GCM). The first two columns show the cross-validation
results for the MOS and PP approaches.

The PP approach attains lower RMSE values (around $1^{\circ}$C) than the MOS
approach ($2-3^{\circ}$C). The former also reaches higher interannual
correlation, with values above 0.95 north of the main mountain ranges and
above 0.9 to the south. The MOS approach also shows a good correlation to the
north (above 0.9) but the correlation drops to 0.7 to the south of the
mountain barriers.

Annual biases are generally low but, for the MOS approach, they are the result
of an average of larger seasonal biases of opposite signs in many locations.
Note that the emulators are trained on an annual basis, including no specific
predictor for the annual cycle. The largest winter warm biases occur in the Po
Valley and are due to the differences in the mid-future test sample with
respect to the training set, which are exacerbated particularly in the MOS
approach.

Figure 4:  Evaluation metrics for emulated surface temperature over the test
mid-future RCP8.5 (2041-2050) period for the ALADIN63 (RCM) simulations driven
by the NorESM model (GCM). The first two columns show the cross-validated
(same predictor source for train and test) results for the MOS and perfect
approaches, respectively. The last two columns show the results corresponding
to soft transferability for the perfect approach, using raw (third column) and
bias adjusted (last column) GCM predictors; in this case, the predictors for
the test period are taken from the driving GCM (instead of the upscaled RCM
used for training). The labels on the top indicate the train and test
predictors (PR) and the predictands used in each case. The rows show, from top
to bottom: annual, winter (DJF) and summer (JJA) biases, RMSE and interannual
correlation.

###  3.3 Soft Transferability of PP Methods

The last two columns in Figure 4 show the results corresponding to soft
transferability of the perfect approach to the driving GCM, using raw (third
column) and bias adjusted (last column) predictors.

The results show RMSE and interannual correlation patterns remarkably similar
to the MOS approach, albeit with slightly higher RMSE and lower correlation.
Notably, the annual and seasonal biases are significantly more pronounced when
using the raw GCM fields as the test input (third column). As we show below
(see Figure 5), this occurs due to the biases between the training upscaled
RCM and their counterpart test GCM fields, as already identified in previous
studies Doury et al. (2022); Boé et al. (2022). Therefore, these biases could
be alleviated by adjusting the GCM predictor biases (GCM-BA, relative to the
upscaled RCM fields). Indeed, using GCM-BA as input (last column in Figure 4),
the emulator shows smaller biases, overall smaller than those of the MOS
approach (in the first column).

Qualitatively similar results are obtained for the emulators trained on the
other GCM-driven simulations. We illustrate this through Figure 5 and the
large-scale temperature (ta850), as is the most informative predictor in this
study (see Sec. 3.1). Note that for other variables, such as precipitation,
the interpretation of the biases would be obscured by the combined effect of
several relevant predictors on the target variable. The diagram of Figure 5
helps in better understanding the role that biases play, by representing
linear transfer functions between two ordinate axes displaying the spatial
averages of the predictor (ta850) and the predictand (tas) respectively. For
instance, in the case of the NorESM-driven simulations, we observe the bias
(b1) of the GCM (1) relative to the corresponding upscaled RCM predictor (2)
that was used to train the EMU1 model, and how this bias is reflected in the
bias of the predictand (b2) when comparing the resulting model output (3) with
the target RCM values (4). Results for CNRM5-driven simulations are also shown
in the same diagram (5-8 and b3-b4). Note that the two GCM predictors exhibit
opposite biases when compared to the corresponding RCM upscaled predictors (b1
and b3). These biases are preserved to a large extent in the resulting
predictions when comparing the emulated and actual RCM signals (b2 and b4,
respectively). The maps on the sides display the corresponding spatial biases,
reinforcing the idea that the large-scale biases in the predictors are
inherited by the emulated surface temperature. Thus, these biases are reduced
when adjusting the biases of the GCM predictors relative to the upscaled RCM
fields, as indicated by the black and gray dots on the right ‘y’ axis,
corresponding to the results emulated from bias-adjusted GCM predictors, which
are closer to the target RCM values.

Figure 5:  Soft transferability of the perfect approach for NorESM and CNRM
GCM driving simulations. The left axis shows the spatial mean values of the
large-scale temperature (the ‘ta850’ predictor) for the upscaled RCM
predictors used to train the emulators (RCM-U; 2 and 6) and their driving GCMs
(1 and 5). The right axis shows the spatial means of the near-surface mean
temperature (the ‘tas’ predictand) resulting from the emulated models using
GCM predictors (3 and 7) and the target RCM values (4 and 8). The thin dashed
lines represent simple emulator extrapolated results for the GCM predictors
(shifted EMU1 and EMU2). The different biases are indicated by b1-b4 notations
and are accompanied by their corresponding spatial map representations. The
results when bias-adjusted predictors are considered in the training phase are
represented by solid black and grey dots on the right axis.

These results illustrate the effect of model biases in emulators and show that
the PP approach is suitable for emulating RCM outputs when using bias-adjusted
predictors from the same driving GCM used for learning. This makes these
models suitable for emulating RCM results in new periods and for new
scenarios, at least when the new predictors fall into the range of variability
used for learning.

###  3.4 Hard transferability: Emulating from new driving GCMs

Hard transferability tests whether an emulator trained with a particular GCM-
driven simulation can emulate the output that the RCM would have when driven
by a different GCM. In this case, the model trained using data from GCM1-RCM
is applied to a different GCM (GCM2) with the goal of reproducing the GCM2-RCM
target (see Figure 2d).

Figure 6 shows the biases resulting from hard transfer experiments (in
columns) corresponding to the different combinations of different pairs of
train and test GCMs from the set of available models (NorESM, CNRM and MPI).
The first two rows show annual biases for the PP and MOS methods,
respectively. The other rows show the corresponding seasonal results for
winter and summer. In most of the cases, the resulting biases exhibit a
similar spatial pattern for the PP and MOS approaches, with smaller intensity
for the former. Also, the sign of the bias reflects the influence of the GCM
on the emulated fields, with opposite biases when reversing the train-test
GCMs (cf. columns 1-3, 2-5 and 4-6). This could reflect that the spatial
structure of emulator biases is a consequence of the different biases of the
GCMs used for training and prediction.

Figure 6:  Hard transferability of the perfect (EMU1) and imperfect (EMU3)
approaches (EMU3) learnt with the NorESM driving simulation when applied to a
different GCM (CNRM5).

In the case of soft transferability, the biases between the GCM and the
upscaled RCM large-scale predictors were an avoidable source of error for RCM
emulators. However, bias adjustment is not applicable for hard
transferability, since adjusting GCM2 predictors relative to GCM1 would
effectively yield a target output close to GCM1-RCM, instead of the desired
target GCM2-RCM.

Figure 7:  Hard transferability of the PP (EMU1) and MOS (EMU3) emulator
approaches learned with the NorESM driving simulation when applied to a
different GCM (CNRM5).

The key here is that the biases between the upscaled RCM and the driving GCM
can have opposite signs, as was shown in Figure 5 (b1 and b3). Thus, adjusting
the biases across GCMs can be catastrophic for the emulator. Figure 7
illustrates the effect of this hard transferability on the spatial average. As
in Figure 5, emulators are depicted as linear transfer functions between two
ordinate axes representing the spatial average ta850 (most informative
predictor) and tas (predictand). In this case, we also depict the MOS emulator
(EMU3) trained with raw NorESM GCM predictors and NorESM-RCM tas. On the
spatial average, the emulators behave close to their linear representation,
indicated in the plot by a simple shift of the emulator to the new test
predictors (thin dashed lines), preserving the slope. For instance, when
applied to the CNRM GCM, the relatively strong slope of the MOS emulator (10)
and the slighter one of the PP emulator (9), are preserved. But both positive
slopes bring the emulator predictions far from the target (8).

Adjusting for the GCM bias would only worsen the results, since the emulator
is already providing warm tas estimates using colder ta850 inputs from CNRM5.
If these inputs are adjusted to the warmer NorESM fields, the emulator would
provide even stronger warm tas biases. For the PP emulator, the true
adjustment to apply would be $b3$, that accounts for the difference between
the upscaled RCM and the new GCM fields (CNRM5) provided as test input. This
adjustment would nudge the predicted fields in the right direction. Note,
however, that this adjustment cannot be done in practice, since the aim of
this hard-transferability exercise is to avoid the RCM simulation nested into
a second GCM. And, if this simulation is available, it is always more worth to
train a new emulator (EMU2 in Figure 7) on these new data than trying to
adjust the inputs of an emulator (EMU1) trained on a different GCM.

###  3.5 Explainability

Figure 8 shows the aggregated (over the training period) saliency maps for two
illustrative locations obtained for the PP and MOS deep models for two
illustrative driving GCMs (NorESM and CNRM). The figure displays the saliency
maps for a selection of predictors: specific humidity (hus), zonal (ua) and
meridional (va) wind velocities at 850 hPa, and air surface temperature (ta)
at 700 and 850 hPa, in rows. The number of each panel indicates the total
contribution to the output (%) of each predictor variable. The saliency maps
are displayed in pairs of columns, grouping the results for the two GCMs to
facilitate the analysis of the patterns learnt when using different driving
GCMs. Columns 1-2 (3-4) and 5-6 (7-8) show the PP (MOS) results for two
illustrative locations (in the Alps and Poland), respectively.

The figure shows remarkable differences in the relevance patters resulting
from the two approaches. The PP results are very similar for the two GCMs,
exhibiting a high local character, with patterns centered on the target
location. This gives high confidence in the results, since the deep emulator
is able to extract consistent patters of influence (with a clear physical
interpretation) from different GCM-driven simulations of the same RCM.
However, MOS patterns are in general more difficult to interpret, exhibiting
non-localized or misplaced patterns that change from model to model. In this
example, the results for CNRM are in better correspondence with the patterns
learnt with the PP approach, whereas the results for NorESM are in general
more difficult to interpret, exhibiting non-localized or misplaced relevance
patterns. This could be a consequence of the smaller day-to-day-correspondence
between predictors and predictand for MOS, which could results in statistical
artifacts with no physical consistency during the optimization process. These
results highlight the importance of introducing explainability in the
evaluation of emulators, particularly for the MOS approach. The results for
the MPI model (not shown) are more similar to the CNRM than the NorESM models.

Figure 8: Aggregated (over the training period) saliency maps for winter
temperature for two illustrative locations (loc1: close to the Alps and loc2:
Poland; indicated by dots in each panel) for the PP and MOS models learnt for
the NorESM and CNRM driving GCMs. Saliency values are displayed for specific
humidity (hus), zonal (ua) and meridional (va) wind velocities at 850 hPa, and
air surface temperature (ta) at 700 and 850 hPa, in rows. The numbers of each
panel indicates the total contribution to the output (%) of each predictor
variable.

These differences are further illustrated in Figure 9 displaying the spatial
correlations of the saliency maps for humidity at 850 hPa (the second most
relevant predict, following upper-air temperatures) across different locations
(four illustrative locations over the area of study) and three GCMs (NorESM,
CNRM and MPI). The correlations for the different MOS/PP results are shown in
the upper/lower triangles. This figure shows that the inter-GCM correlations
for the PP approach are high for each of the locations, indicating similar
spatial predictor patterns (from the upscaled GCM-driven RCM fields) being
extracted by the different deep emulators (see the dashed boxes along the
diagonal on the figure) for the same location. On the other hand, the PP
results exhibit low inter-location correlations, indicating that the models
learn specific spatial predictors for different locations. This is a desired
behavior for the RCM emulators from a physical point of view.

Contrarily, correlations from the MOS approach are in general medium and low,
both across GCMs and across locations, indicating no apparent structure in the
predictor fields relevant for the different models and locations. This is
particularly relevant for the NorESM model, whereas the MOS results for CNRM
and MPI are closer to the PP results. Therefore, there is no guarantee that
RCM emulators trained under the MOS approach are able to extract meaningful
physical information from the predictors. This requires a case by case
assessment involving further research building on physical principles and
processes.

Figure 9: Correlation of saliency maps of humidity at 850 hPa for four
illustrative locations for three RCM emulators build from different driving
GCM (NorESM, CNRM and MPI). Values in the upper/lower triangle correspond to
the results for the models learnt under the MOS/PP approaches (separated by
the dashed diagonal). Dashed boxes along the diagonal show the inter-GCM
results for each of the locations.

##  4 Conclusions and Prospects

We applied RCM emulators using the two most common approaches (PP and MOS) to
a set of existing GCM-RCM simulations from the CORDEX initiative. We did so
focusing on the soft and hard transferability of these emulators, evaluating
them on the basis of physical consistency though an XAI technique (saliency
maps).

Soft transferability would allow to confidently use these emulators on new
test periods, new emission scenarios or new driving GCM members exploring
natural climate variability. We tested it with a new temporal period produced
by the same training GCM-RCM pair. Both PP, after adjusting the GCM
predictors, and MOS successfully emulate unbiased temperature fields for
summer and winter. Nevertheless, XAI shows that PP learns from predictor
patterns with a strong local dependence to the low-level temperature and
humidity that resembles the actual climate dynamics. This agrees with studies
dealing with conventional statistical downscaling (using observational data
sets), suggesting similar local links for temperature (González-Abad et al.,
2023). However, the predictability of MOS emulators leans on spatially
extended, non-local patterns. Moreover, these depend on the driving GCM used
during training. This difference between approaches is a result of the lower
correlation between the input and output training fields in the MOS approach.
Under low day-to-day correspondence, the machine learning model is more likely
to find predictability sources anywhere in the spatial domain.

Hard transferability, that is, applying the emulator to predictor fields from
unseen climates, was tested by using as input the fields from a new GCM,
different from the training one. In this case, the unique nature of the GCM-
RCM combined biases prevent the emulator from being transferred. Since the RCM
response to the predictors can differ greatly, even in sign, between driving
GCMs, the emulator cannot foresee this response in which it was not trained.
Further understanding of the nature of these biases would be needed, to
identify transferability windows of opportunity for PP-based statistical
emulators. Recent work suggests that structural differences (e.g., aerosols
representation and atmospheric physics) between the RCM and driving GCM are a
large driver of the dissimilarity between their large-scale fields, being
almost negligible when both simulations are driven with consistent external
forcing among them (Taranu et al., 2022). One such example, would be the CNRM-
CM5-driven simulation, which indeed shows a bit more local predictor saliency
patterns under MOS training than the other GCMs.

Overall, then, the PP approach learns the same robust predictor patterns for
different driving GCMs thus increasing confidence. These predictor patterns
are locally connected to the predictand, resembling more closely the
functioning of the RCM, that is ultimately the main goal. On the other hand,
MOS yields model-dependent patterns which, in some cases, are more difficult
to interpret and lack from physical consistency. This contrasts with previous
studies, that recommended the use of MOS emulators against PP ones (Boé et
al., 2022; van der Meer et al., 2022), on the basis of conventional evaluation
metrics. We found that both approaches face problems in emulating the results
for different driving GCMs, thus limiting their applicability to fill the GCM-
RCM combination matrix of regional climate projections.

The limitations for the transferability of both RCM emulation approaches to be
used on inputs from a new GCM (hard transferability) are difficult to overcome
in general. Since they arise from the different response of the RCM to the
driving GCM biases, so the way forward can be either to (1) train the emulator
under non-biased driving fields, which could be used later also to de-bias new
predictors or to (2) train the emulator with a wider variety of GCM biases.
The former could be accomplished by training with reanalysis data (so-called
perfect boundary conditions for the RCM), but this would limit the training to
the current climate, compromising the ability of the emulator to extrapolate
under future climate conditions. The latter would imply mixing different
driving GCMs in the training phase, in order to show the emulator the RCM
response to different biases. This approach might enhance the capability of
the emulator to cope with different biases, but it will likely affect its
accuracy for any particular GCM input as compared to a specific GCM-RCM
emulator as those shown in this work and elsewhere.

Emulators have also great potential for convection-permitting RCM (CP-RCM)
simulations (Coppola et al., 2020). These are very costly RCM simulations with
grid spacings below 4km, where the parameterization representing convection
can be deactivated, reducing uncertainties e.g. for precipitation. These
simulations are typically nested into the same RCM at coarser resolution (e.g.
the 12 km resolution considered in this study). In this context, the whole
emulation process is performed in the framework of the same RCM, thus avoiding
the problem of the biases arising from the driving model mismatch with the
emulated one. The emulator would learn the relationship between the RCM and
CP-RCM and provide CP-RCM-emulated fields out of inputs from RCM simulations
at coarse resolution.

Note that in this work we did not cover spatial transferability, i.e. training
the model in one region (or on a representative selection of regions) and
using it in different regions, which is also an active field of research
Bjerre et al. (2022); Ludwig et al. (2023).

###### Acknowledgements.

This work is part of IMPETUS4CHANGE, funded by the European Union’s Horizon
Europe research and innovation programme under grant agreement No 101081555.
J.M.G and J.F acknowledge support from MCIN/AEI/10.13039/501100011033, which
funded projects ATLAS (PID2019-111481RB-I00) and CORDyS
(PID2020-116595RB-I00), respectively. Both GCM and RCM data used in this study
—from the Coupled Model Intercomparison Project Phase 5 (CMIP5) and CORDEX,
respectively,— are openly available from the Earth System Grid Federation
(ESGF, https://esgf.llnl.gov) portal. The code needed to implement the DeepESD
model can be found in a Zenodo repository
(https://doi.org/10.5281/zenodo.6828303, Baño-Medina et al. (2022)) associated
to a previous publication (Baño-Medina et al., 2022).

## References

  * Balmaceda-Huarte et al. (2023) Balmaceda-Huarte, R., J. Baño-Medina, M. E. Olmo, and M. L. Bettolli, 2023: On the use of convolutional neural networks for downscaling daily temperatures over southern south america in a climate change scenario. Climate Dynamics, 1–15.
  * Baño-Medina (2020) Baño-Medina, J., 2020: Understanding deep learning decisions in statistical downscaling models. Proceedings of the 10th International Conference on Climate Informatics, 79–85.
  * Baño-Medina et al. (2022) Baño-Medina, J., R. Manzanas, E. Cimadevilla, J. Fernández, J. González-Abad, A. S. Cofiño, and J. M. Gutiérrez, 2022: Downscaling multi-model climate projection ensembles with deep learning (deepesd): contribution to cordex eur-44. Geoscientific Model Development, 15 (17), 6747–6758.
  * Baño-Medina et al. (2020) Baño-Medina, J., R. Manzanas, and J. M. Gutiérrez, 2020: Configuration and intercomparison of deep learning neural models for statistical downscaling. Geoscientific Model Development, 13 (4), 2109–2124.
  * Baño-Medina et al. (2021) Baño-Medina, J., R. Manzanas, and J. M. Gutiérrez, 2021: On the suitability of deep convolutional neural networks for continental-wide downscaling of climate change projections. Climate Dynamics, 57 (11), 2941–2951.
  * Baño-Medina et al. (2022) Baño-Medina, J., R. Manzanas, E. Cimadevilla, J. Fernández, J. González-Abad, A. S. Cofiño, and J. M. Gutiérrez, 2022: Repository supporting the results presented in the manuscript on Downscaling Multi-Model Climate Projection Ensembles with Deep Learning (DeepESD): Contribution to CORDEX EUR-44. Zenodo.
  * Bellouin et al. (2011) Bellouin, N., and Coauthors, 2011: The hadgem2 family of met office unified model climate configurations. Geoscientific Model Development, 4 (3), 723–757.
  * Bentsen et al. (2013) Bentsen, M., and Coauthors, 2013: The Norwegian Earth System Model, NorESM1-M – Part 1: Description and basic evaluation of the physical climate. Geoscientific Model Development, 6 (3), 687–720.
  * Bjerre et al. (2022) Bjerre, E., M. N. Fienen, R. Schneider, J. Koch, and A. L. Højberg, 2022: Assessing spatial transferability of a random forest metamodel for predicting drainage fraction. Journal of Hydrology, 612, 128 177, https://doi.org/10.1016/j.jhydrol.2022.128177, URL https://www.sciencedirect.com/science/article/pii/S0022169422007508.
  * Boé et al. (2022) Boé, J., A. Mass, and J. Deman, 2022: A simple hybrid statistical–dynamical downscaling method for emulating regional climate models over Western Europe. Evaluation, application, and role of added value? Climate Dynamics, 10.1007/s00382-022-06552-2, URL https://doi.org/10.1007/s00382-022-06552-2.
  * Brands et al. (2013) Brands, S., S. Herrera, J. Fernández, and J. M. Gutiérrez, 2013: How well do cmip5 earth system models simulate present climate conditions in europe and africa? Climate dynamics, 41 (3), 803–817.
  * Cardoso and Soares (2022) Cardoso, R. M., and P. M. Soares, 2022: Is there added value in the euro-cordex hindcast temperature simulations? assessing the added value using climate distributions in europe. International Journal of Climatology.
  * Coppola et al. (2020) Coppola, E., and Coauthors, 2020: A first-of-its-kind multi-model convection permitting ensemble for investigating convective phenomena over europe and the mediterranean. Climate Dynamics, 55, 3–34.
  * Diez-Sierra et al. (2022) Diez-Sierra, J., and Coauthors, 2022: The worldwide c3s cordex grand ensemble: A major contribution to assess regional climate change in the ipcc ar6 atlas. Bulletin of the American Meteorological Society, 103 (12), E2804 – E2826, https://doi.org/10.1175/BAMS-D-22-0111.1, URL https://journals.ametsoc.org/view/journals/bams/103/12/BAMS-D-22-0111.1.xml.
  * Doury et al. (2022) Doury, A., S. Somot, S. Gadat, A. Ribes, and L. Corre, 2022: Regional climate model emulator based on deep learning: concept and first evaluation of a novel hybrid downscaling approach. Climate Dynamics, 1–29.
  * Erlandsen et al. (2020) Erlandsen, H. B., K. M. Parding, R. Benestad, A. Mezghani, and M. Pontoppidan, 2020: A hybrid downscaling approach for future temperature and precipitation change. Journal of Applied Meteorology and Climatology, 59 (11), 1793–1807.
  * Fernández et al. (2019) Fernández, J., and Coauthors, 2019: Consistency of climate change projections from multiple global and regional model intercomparison projects. Climate dynamics, 52 (1), 1139–1156.
  * Giorgi (2019) Giorgi, F., 2019: Thirty Years of Regional Climate Modeling: Where Are We and Where Are We Going next? Journal of Geophysical Research: Atmospheres, 124 (11), 5696–5723, 10.1029/2018JD030094, URL https://onlinelibrary.wiley.com/doi/abs/10.1029/2018JD030094.
  * González-Abad et al. (2023) González-Abad, J., J. Baño-Medina, and J. M. Gutiérrez, 2023: Using explainability to inform statistical downscaling based on deep learning beyond standard validation approaches. arXiv preprint arXiv:2302.01771.
  * Gutiérrez et al. (2013) Gutiérrez, J. M., D. San-Martín, S. Brands, R. Manzanas, and S. Herrera, 2013: Reassessing statistical downscaling techniques for their robust application under climate change conditions. Journal of Climate, 26 (1), 171–188.
  * Gutiérrez et al. (2019) Gutiérrez, J. M., and Coauthors, 2019: An intercomparison of a large ensemble of statistical downscaling methods over europe: Results from the value perfect predictor cross-validation experiment. International journal of climatology, 39 (9), 3750–3785.
  * Hernanz et al. (2022) Hernanz, A., J. A. García-Valero, M. Domínguez, and E. Rodríguez-Camino, 2022: A critical view on the suitability of machine learning techniques to downscale climate change projections: Illustration for temperature with a toy experiment. Atmospheric Science Letters, e1087.
  * Hobeichi et al. (2023) Hobeichi, S., N. Nishant, Y. Shao, G. Abramowitz, A. Pitman, S. Sherwood, C. Bishop, and S. Green, 2023: Using machine learning to cut the cost of dynamical downscaling. Earth’s Future, 11 (3), e2022EF003 291.
  * IPCC (2022) IPCC, 2022: Climate Change 2022: Impacts, Adaptation and Vulnerability. Summary for Policymakers, Cambridge University Press, Cambridge, UK and New York, USA, 3-33 pp.
  * Jacob et al. (2020) Jacob, D., and Coauthors, 2020: Regional climate downscaling over europe: perspectives from the euro-cordex community. Regional environmental change, 20 (2), 1–20.
  * Kendon et al. (2010) Kendon, E. J., R. G. Jones, E. Kjellström, and J. M. Murphy, 2010: Using and designing gcm–rcm ensemble regional climate projections. Journal of Climate, 23 (24), 6485–6503.
  * Kingma and Ba (2014) Kingma, D. P., and J. Ba, 2014: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  * Kondylatos et al. (2022) Kondylatos, S., I. Prapas, M. Ronco, I. Papoutsis, G. Camps-Valls, M. Piles, M.-Á. Fernández-Torres, and N. Carvalhais, 2022: Wildfire danger prediction and understanding with deep learning. Geophysical Research Letters, e2022GL099368.
  * LeCun et al. (1995) LeCun, Y., Y. Bengio, and Coauthors, 1995: Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361 (10), 1995.
  * Ludwig et al. (2023) Ludwig, M., A. Moreno-Martinez, N. Hölzel, E. Pebesma, and H. Meyer, 2023: Assessing and improving the transferability of current global spatial prediction models. Global Ecology and Biogeography, 32 (3), 356–368, 10.1111/geb.13635, URL https://onlinelibrary.wiley.com/doi/abs/10.1111/geb.13635.
  * Mamalakis et al. (2022) Mamalakis, A., I. Ebert-Uphoff, and E. A. Barnes, 2022: Explainable artificial intelligence in meteorology and climate science: Model fine-tuning, calibrating trust and learning new science. International Workshop on Extending Explainable AI Beyond Deep Models and Classifiers, Springer, 315–339.
  * Maraun and Widmann (2018) Maraun, D., and M. Widmann, 2018: Statistical downscaling and bias correction for climate research. Cambridge University Press, (2018).
  * Molina et al. (2022) Molina, M. O., J. A. M. Careto, C. Gutiérrez, E. Sánchez, and P. M. M. Soares, 2022: The added value of high-resolution euro-cordex simulations to describe daily wind speed over europe. International Journal of Climatology.
  * Müller et al. (2018) Müller, W., and Coauthors, 2018: A higher-resolution version of the Max Planck Institute Earth System Model (MPI-ESM 1.2-HR). Journal of Advances in Modeling Earth Systems, 10.
  * Phillips (1956) Phillips, N. A., 1956: The general circulation of the atmosphere: A numerical experiment. Quarterly Journal of the Royal Meteorological Society, 82 (352), 123–164.
  * Quesada-Chacón et al. (2022) Quesada-Chacón, D., K. Barfus, and C. Bernhofer, 2022: Repeatable high-resolution statistical downscaling through deep learning. Geoscientific Model Development, 15 (19), 7353–7370.
  * Rampal et al. (2022) Rampal, N., P. B. Gibson, A. Sood, S. Stuart, N. C. Fauchereau, C. Brandolino, B. Noll, and T. Meyers, 2022: High-resolution downscaling with interpretable deep learning: Rainfall extremes over New Zealand. Weather and Climate Extremes, 38, 100 525, https://doi.org/10.1016/j.wace.2022.100525, URL https://www.sciencedirect.com/science/article/pii/S2212094722001049.
  * Rummukainen (2016) Rummukainen, M., 2016: Added value in regional climate modeling. Wiley Interdisciplinary Reviews: Climate Change, 7 (1), 145–159.
  * Soares and Cardoso (2018) Soares, P. M., and R. M. Cardoso, 2018: A simple method to assess the added value using high-resolution climate distributions: application to the euro-cordex daily precipitation. International Journal of Climatology, 38 (3), 1484–1498.
  * Sundararajan et al. (2017) Sundararajan, M., A. Taly, and Q. Yan, 2017: Axiomatic attribution for deep networks. International conference on machine learning, PMLR, 3319–3328.
  * Taranu et al. (2022) Taranu, I. S., S. Somot, A. Alias, J. Boé, and C. Delire, 2022: Mechanisms behind large-scale inconsistencies between regional and global climate model-based projections over europe. Climate Dynamics, 1–26.
  * Toms et al. (2021) Toms, B. A., E. A. Barnes, and J. W. Hurrell, 2021: Assessing decadal predictability in an earth-system model using explainable neural networks. Geophysical Research Letters, 48 (12), e2021GL093 842.
  * van der Meer et al. (2022) van der Meer, M., S. de Roda Husman, and S. Lhermitte, 2022: Deep learning regional climate model emulators: a comparison of two downscaling training frameworks. Authorea Preprints.
  * Voldoire et al. (2013) Voldoire, A., and Coauthors, 2013: The CNRM-CM5.1 global climate model: description and basic evaluation. Climate Dynamics, 40 (9-10), 2091–2121.