.tocmtchapter mtchaptersubsection mtappendixnone

# Diversity Boosted Learning for Domain Generalization with Large Number of
Domains

Xi Leng
School of Science and Engineering,
The Chinese University of Hong Kong (Shenzhen)
<EMAIL_ADDRESS>
&Xiaoying Tang
School of Science and Engineering,
The Chinese University of Hong Kong (Shenzhen)
<EMAIL_ADDRESS>
&Yatao Bian
Tencent AI Lab
<EMAIL_ADDRESS>
Corresponding author

###### Abstract

Machine learning algorithms minimizing the average training loss usually
suffer from poor generalization performance due to the greedy exploitation of
correlations among the training data, which are not stable under
distributional shifts. It inspires various works for domain generalization
(DG), where a series of methods, such as Causal Matching and FISH, work by
pairwise domain operations. They would need $O(n^{2})$ pairwise domain
operations with $n$ domains, where each one is often highly expensive.
Moreover, while a common objective in the DG literature is to learn invariant
representations against domain-induced spurious correlations, we highlight the
importance of mitigating spurious correlations caused by _objects_. Based on
the observation that diversity helps mitigate spurious correlations, we
propose a Diversity boosted twO-level saMplIng framework (DOMI) utilizing
Determinantal Point Processes (DPPs) to efficiently sample the most
informative ones among large number of domains. We show that DOMI helps train
robust models against spurious correlations from both domain-side and object-
side, substantially enhancing the performance of the backbone DG algorithms on
rotated MNIST, rotated Fashion MNIST, and iwildcam datasets.

##  1 Introduction

Machine learning models are typically trained to minimize the average loss on
the training set, such as empirical risk minimization (ERM). The effectiveness
of machine learning algorithms with ERM relies on the assumption that the
testing and training data are identically drawn from the same distribution,
which is known as the IID hypothesis. However, distributional shifts between
testing and training data are usually inevitable due to data selection biases
or unobserved confounders that widely exist in real-life data. Moreover, the
data distribution of the training set is likely to be imbalanced. Certain
domains may contain the majority of data samples while other domains are only
a small fraction of the training set. Under such circumstances, models trained
by minimizing average training loss are prone to sink into spurious
correlations and suffer from poor generalization performance. Here spurious
correlations refer to misleading heuristics that only work for most training
examples but can not generalize to data from other distributions that may
appear in the test set. The goal of domain generalization (DG) is to learn a
model that can generalize well to unseen data distributions after training on
more than one data distributions. Different data distributions are denoted as
different domains in DG tasks. For example, an image classifier should be able
to discriminate the objects whatever the background of the image is. While
lots of methods have been derived to efficiently achieve this goal and show
good performances, there are two main drawbacks.

Scalability. With an unprecedented amount of applicable data nowadays, many
datasets contain a tremendous amount of domains, or massive data in each
domain, or both. For instance, WILDS (Koh et al., 2021) is a curated
collection of benchmark datasets representing distribution shifts faced in the
wild. Among these datasets, some contain thousands of domains and OGB-MolPCBA
(Hu et al., 2020b) contains more than one hundred thousand. Besides WILDS,
DrugOOD (Ji et al., 2022) is an out-of-distribution dataset curator and
benchmark for AI-aided drug discovery. Datasets of DrugOOD contain hundreds to
tens of thousands of domains. In addition to raw data with abundant domains,
domain augmentation, leveraged to improve the robustness of models in DG
tasks, can also lead to a significant increase in the number of domains. For
example, HRM (Liu et al., 2021) generates heterogeneous domains to help
exclude variant features, favoring invariant learning. Under such
circumstances, training on the whole dataset in each epoch is computationally
prohibitive, especially for methods training by pairwise operations between
domains. For instance, the computational complexity of causal matching in
MatchDG (Mahajan et al., 2021) and gradient matching in FISH (Shi et al.,
2021b) is $O(n^{2})$ with $n$ training domains.

Objective. Lots of works in the DG field focus entirely on alleviating or
excluding impacts from the domain-side. As mentioned before, a good image
classifier should be able to discriminate the objects whatever the background
of the image is, and one may naturally aim to exclude the impacts from the
background while ignoring that from the objects themselves. A general
assumption in the DG field is that data in different domains share some
“stable” features to form the causal correlations. However, a large branch of
studies hold the view that the relationship between these "stable" features
and the outputs is domain-independent given certain conditions. While plenty
of algorithms have been designed to learn such "stable" features and domain-
independent correlations, we show that this objective is insufficient, and a
simple counterexample is as follows. We highlight the importance of mitigating
spurious correlations induced from the object-side for training a robust
model.

Suppose our learning task is to train a model to distinguish between cats and
lions. The composition of the training dataset is shown in Figure 1 and the
domain here refers to the background of the figures.

Figure 1: Dataset of the counterexample. Cats are mainly silver British
shorthair (body color of which is silvery white), rarely golden British
shorthair (tan), and lions are all tan. As for the background, most of the
lions are on the grassland while most of the cats are indoors.

In this example, the correlation between features corresponding to the body
color of objects and class labels is undoubtedly independent of domains.
Moreover, it helps get high accuracy in the training set by simply taking the
tan objects as lions and the white ones as cats. Unfortunately, if this
correlation is taken as the causal correlation, the model is prone to poor
performance once the distribution of cat breeds shifts in the test set.

To tackle these two issues, a sampling strategy to select the most informative
domains or data points for mitigating impacts from both domain-side and
object-side to obtain a genuinely robust model is essential. Under the setting
of large numbers of domains and domains with massive data points, we propose a
diversity boosted two-level sampling framework named DOMI. Since we will set
forth later that diversity helps mitigate spurious correlations, a sampling
scheme to select diverse domains or data points is an essential part of DOMI.
In this paper, we incorporate Determinantal Point Process (DPP) sampling into
DOMI as one choice of diversity sampling methods. DPP (Kulesza et al., 2012)
is a point process that mimics repulsive interactions between samples, and a
draw from a DPP yields diversified subsets. Extensive experiments show that
DOMI helps efficiently alleviate spurious correlations from both domain-side
and object-side, substantially enhancing the performance of the backbone DG
algorithms on rotated MNIST, rotated Fashion MNIST, and iwildcam.

#### Summary of contributions.

Our contributions can be summarized as follows:

  * 1\. To our best knowledge, this is the first paper to take impacts from the object-side into account for achieving the goal of DG.

  * 2\. We propose DOMI, a diversity boosted two-level sampling framework to select the most informative domains and data points for mitigating impacts from both domain-side and object-side.

  * 3.We show that DOMI substantially enhances the test accuracy of the backbone DG algorithms on three benchmarks.

##  2 Related Work

Domain Generalization. In DG tasks, the training data is sampled from one or
many source domains, while the test data is sampled from the new target
domains. The goal of DG is to learn a model that can generalize well to all
domains including unseen ones after training on more than one domains
(Blanchard et al., 2011; Wang et al., 2022; Zhou et al., 2021; Shen et al.,
2021). Among recent works on domain generalization, Ben-Tal et al. (2013);
Duchi et al. (2016) utilize distributionally robust optimization (DRO) to
minimize the worst-case loss over potential test distributions instead of the
average loss of the training data. Sagawa et al. (2019) propose group DRO to
train models by minimizing the worst-case loss over groups to avoid learning
models relying on spurious correlations and therefore suffering a high loss on
some groups of data. Zhai et al. (2021) further take use of distributional and
Outlier Robust Optimization (DORO) to address the problem that DRO is
sensitive to outliers and thus suffers from poor performance and severe
instability when faced with real, large-scale tasks. On the other hand,
(Arjovsky et al., 2019; Javed et al., 2020; Krueger et al., 2021; Shi et al.,
2021a; Ahuja et al., 2020a) rather leverage Invariant Risk Minimization (IRM)
to learn features inducing invariant optimal predictors over training
environments. However, Rosenfeld et al. (2020); Kamath et al. (2021); Ahuja et
al. (2020b) hold the view that works with IRM lack formal guarantees and
present analysis to demonstrate IRM fails to generalize well even when faced
with some simple data models and fundamentally does not improve over standard
ERM. Another branch of studies assume that data from different domains share
some “stable” features whose relationships with the outputs are causal
correlations and domain-independent given certain conditions (Long et al.,
2015; Hoffman et al., 2018; Zhao et al., 2018, 2019). Among this branch of
work, Li et al. (2018); Ghifary et al. (2016); Hu et al. (2020a) hold the view
that causal correlations are independent of domain conditioned on class label,
and Muandet et al. (2013) propose DICA to learn representations marginally
independent of domain.

MatchDG. Mahajan et al. (2021) state that learning representations independent
of the domain after conditioning on the class label is insufficient for
training a robust model. They propose MatchDG to learn correlations
independent of domain conditioned on objects, where objects can be seen as
clusters within classes based on similarity. To ensure the learned features
are invariant across domains, a term of the distance between each pair of
domains is added to the objective to be minimized.

FISH. Different from the two ideas mentioned above, Shi et al. (2021b) instead
propose FISH to achieve the goal of DG. FISH uses an inter-domain gradient
matching objective to learn a model with invariant gradient direction in
different domains, where the objective augments the ERM loss with an auxiliary
term that maximizes the gradient inner product between domains. By minimizing
the loss and matching the gradients simultaneously, FISH encourages the
optimization paths to be the same for all domains, favoring invariant
predictions. To match the gradients while training, FISH incorporates a term
of inner product between gradients of each pair of domains into the objective
to be maximized.

DANN. (Ganin et al., 2016) incorporates the structure named domain
discriminator to implement adversarial training based on the theory that a
good classifier for cross-domain shifts should be able to distinguish
different classes while cannot learn to identify the domain. DOMI takes use of
an inverse version of DANN denoted as invDANN to learn domain-side features
and help select the most informative domains.

DPP. Determinantal Point Process (DPP) (Kulesza et al., 2012) is a point
process that mimics repulsive interactions. Based on a similarity matrix (DPP
kernel) of samples to be selected, a draw from a DPP yields diversified
subsets. While it shows powerful performance in selecting heterogeneous data,
DPP sampling relies on an eigendecomposition of the DPP kernel, whose cubic
complexity is a huge impediment. To address this problem, Li et al. (2016)
suggest to first construct an approximate probability distribution to the true
DPP and then efficiently samples from this approximate distribution. As one
choice of diversity sampling, DPP sampling is incorporated into DOMI to help
select the most informative domains and data points, and it can be replaced
with other diversity sampling schemes.

Although MatchDG and FISH perform well in domain generalization tasks, the
matching procedure between domains means their computational complexity is
$O(n^{2})$ with $n$ training domains. When $n$ is large, it will inevitably
slow down the training process. Therefore, we must select the most informative
domains from all the training domains. Inspired by Liu et al. (2021) that
heterogeneous training domains help to learn invariant features since more
variant features can be excluded, we conduct an analysis of diversity and
spurious correlations to further state it. To employ DPP sampling to select
diverse domains, we build a DPP kernel by measuring the similarity of
descriptions of each domain. A description of one domain is derived by a set
transformer (Lee et al., 2019). After the featurizer trained by invDANN
extracts the features of part or all of data points in a domain, this set of
features is transformed to a description.

##  3 Diversity Helps Mitigate Spurious Correlations

Spurious correlations essentially result from the imbalanced data. If a
correlation is easy to be found and is held by most of the data, algorithms
minimizing the average loss like ERM may simply take this correlation as the
causal correlation. Thus when we sample diverse data, we in fact re-balance
them and help mitigate spurious correlations. We verify this observation with
a toy example and an experiment.

###  3.1 A Toy Example

For the task and dataset mentioned above (Figure 1), we further suppose our
featurizer extracts 4 features with a binary value as shown in Table 1.

Table 1: Details of the features and the label. $X_{1}$ to $X_{3}$ correspond to features of the object and $X_{4}$ corresponds to features of the domain. |  $X_{1}$ : Mane |  $X_{2}$ : Proportion of face |  $X_{3}$ : Body color |  $X_{4}$ : Background | $y$
---|---|---|---|---|---
0 | no mane | short face | white | indoors | cat
1 | have a mane | long face | tan | grassland | lion

Then $X_{1}+X_{2}\geq 1\Rightarrow y=1$ is the causal correlation since the
proportion of lions’ faces is longer than that of cats, and $X_{2}$ may be
wrongly computed to 0 for male lions because of the existence of mane.
$X_{3}=1\Rightarrow y=1$ is the Object-spurious-correlation (Abbrev. Osc) and
$X_{4}=1\Rightarrow y=1$ is the Domain-spurious-correlation (Abbrev. Dsc).
Details of our simulated dataset is shown in Appendix A.

Suppose we have to get 6 of these 12 data samples for training where 3 of 6
come from cats and another 3 are from lions. There are 4 sampling methods to
be picked: random sampling, sampling making the data more diverse on the
object features ($X_{1}$, $X_{2}$ and $X_{3}$), sampling making the data more
diverse on the domain feature ($X_{4}$), and sampling making the data more
diverse on all 4 features. For convenience, we call these four sampling
methods $S_{1}$ to $S_{4}$ and use Manhattan Distance on feature tuples to
measure the diversity of sampled data. Table 2 shows the average training
accuracy of Osc and Dsc. When the spurious correlations get lower training
accuracy, they are more likely to be excluded, favoring exploration of the
causal correlations.

$S_{1}$ preserves the imbalance of data. A data point has a larger probability
of being sampled into the batch when it appears more often in the dataset. For
base-batches sampled by $S_{1}$, both Osc and Dsc get high accuracy and are
thus likely to be wrongly treated as causal correlations.

$S_{2}$ selects data with heterogeneity on object features and data batches
sampled by $S_{2}$ get lower accuracy for Osc than base-batches, which means
$S_{2}$ reduces the probability of taking Osc as causal correlation. However,
data batches sampled by $S_{2}$ get almost the same result for Dsc.

$S_{3}$ selects data with heterogeneity on domain-feature $X_{4}$. For these
batches of data, Dsc gets lower accuracy than base-batches and is less likely
to be taken as causal correlation while Osc has a similar result.

$S_{4}$ selects data with heterogeneity on all 4 features. Compared to base-
batches, the data batches got by $S_{4}$ have lower accuracy on both spurious
correlations.

Table 2: We use each sampling method to select 30 batches of data and compute average accuracy on two kinds of spurious correlations, which are more likely to be excluded when they get lower accuracy. Sampling Method | Accuracy of Osc | Accuracy of Dsc
---|---|---
$S_{1}$ | 0.86 | 0.68
$S_{2}$ | 0.77 | 0.66
$S_{3}$ | 0.85 | 0.50
$S_{4}$ | 0.78 | 0.49

###  3.2 An Empirical Study on Diversity and Spurious Correlations

Suppose the idea holds, i.e., training on diverse data about certain features
can help exclude spurious correlations caused by these features. And now we
have two settings for the experiment. One is training models on randomly
sampled domain lists for each round and getting their test accuracy. For the
other one, with a randomly sampled domain list as the initial domain list,
every next domain list is selected by DPP based on the current model. Then the
accuracy of the later setting should show a more oscillatory trend.

Why? Suppose we have a model with good generalization ability now, that means
the model extracts the "stable" features and knows little about spurious
correlations. If we use DPP to sample domains based on this featurizer, we
just get diverse data about the "stable" features while they are still
imbalanced as to spurious features. Trained on these data, the next model is
likely to be affected by spurious correlations and show poor generalization
ability. Similarly, suppose we have a model with poor generalization ability.
In this case, the model actually learns spurious correlations and data
attained by DPP using this featurizer is diverse as to spurious features.
Trained on these data, the next model is less likely to be affected by
spurious correlations and prone to good generalization ability.

We run experiments under the two settings on the Rotated Fashion MNIST
dataset. The rotated degree of the training dataset is from 15 to 75 while
that of the test set is 0 and 90. The rotated degree is taken to be domain
labels, i.e., the training set gets 61 domains. We denote the former setting,
i.e., randomly sampling domains in each round as baseline. About DPP-line,
after a featurizer processing all the data of a single domain, we average all
outputs and take it as the description of this domain. Then we take use of
descriptions of all training domains and cosine similarity measurement to get
a similarity matrix. Finally DPP selects domains based on this similarity
matrix. We observe the test accuracy of 20 rounds in each setting and compute
the variance every time. After repeating 10 times, the average variance in the
baseline is 31.3, much lower than the other setting, 82.2, which is in line
with our expectation.

##  4 An Empirical Study on The Insufficient Objective

Still on the Rotated Fashion MNIST dataset, in this experiment we train two
models with all else being equal on two different domain lists containing five
domains each.

How do we obtain these two domain lists? First, we randomly sample a domain
list, and use DANN and invDANN (details in Section 6.1) to respectively train
a featurizer on this domain list for the same epochs. Now we get two
featurizers. To differentiate them, we call one object-featurizer and the
other domain-featurizer since the former learns representations about object
and the latter about domain. Then we use DPP to sample domains based on each
featurizer.

Now we get these two domain lists, on which we train models and observe their
sensitivity to domain shifts and test accuracy. As for sensitivity, we get
descriptions of all 61 domains and compute similarities between each pair of
them just like in DPP procedure. Then we simply compute the sum of these
similarities. A smaller summation means more dissimilar among domains and more
sensitive to domain shifts, thus depending more on Dsc. The result shows that
while the model trained on the domain list obtained by domain-featurizer gets
a larger sum (1382.0 compared to 1201.8), it still gets lower test accuracy.
This means although training on domains derived by invDANN helps mitigate Dsc,
there exists another spurious correlation, i.e., Osc, affecting the
generalization ability of models. One possible explanation for the result is:
While the two featurizers are trained for the same epochs, the task of
differentiating domains (rotation) is more straightforward than identifying
images. Thus the object-featurizer is likely to learn both Dsc and Osc because
of insufficient learning epochs while the domain-featurizer learns well and
only extracts domain features. As is shown in Section 3.2, data attained by
DPP based on object-featurizer is diverse for two kinds of spurious features,
training on which helps alleviate two kinds of spurious correlations, and thus
gets the model with better test accuracy.

##  5 Formulation of Two Main Observations

###### Proposition 1

Diverse domains of data help exclude spurious correlation.

Consider a dataset $D=\\{D^{d}\\}$ which is a mixture of data
$D^{d}=\\{(x_{i}^{d},y_{i}^{d})\\}_{i=1}^{n_{d}}$ where $d$ is one domain of
the ground set $D$, $\mathbf{x}_{i}^{d}$ and $\mathbf{y}_{i}^{d}$ are the
$i_{th}$ data and label from domain $d$ respectively, and $n_{d}$ is the
number of samples in $d$. Suppose we now have dataset $D_{k}$ consisting of
$k$ domains. On $D_{k}$, the distribution of data is $P^{k}$(X,Y). A "good"
set denoted by $C_{k}$ is a set containing "good" correlations that get high
accuracy on $D_{k}$. The set of causal correlations is $C$. $C\subseteq C_{k}$
since causal correlations can definitely get good performance but "good"
correlations for the k domains may not be held in other domains, i.e.,
spurious correlations. Our goal is to exclude as many spurious correlations as
possible.
Given another domain $d_{k+1}$ to form dataset $D_{k+1}$ together with the
former k domains. The corresponding data distribution and the "good" set are
$P^{k+1}$(X,Y) and $C_{k+1}$, respectively. If $P^{k+1}$(X,Y) is close to
$P^{k}$(X,Y), then most of the correlations in $C_{k}$ will still be "good"
for $D_{k+1}$ and thus preserved in $C_{k+1}$. Nevertheless, if $d_{k+1}$ is a
heterogeneous domain that can significantly change the distribution of data,
then the "good" set after being constrained would be obviously smaller than
the original one, i.e., $|C_{k+1}|<<|C_{k}|$, showing that diverse domains
help exclude spurious correlations and training on which helps obtain robust
models.

###### Proposition 2

Excluding domain-spurious-correlations is insufficient for learning a robust
model.

Mahajan et al. (2021) have proposed that correlations independent of domain
conditional on class $(\Phi(x)\Vbar{D}|Y)$ are not necessarily causal
correlations if $P(\dot{x}|Y)$ changes across domains. Here $\Phi(x)$ is a
featurizer to extract features and $\dot{x}$ represents the causal features.
We now further propose that the condition is still not sufficient even if
$\dot{x}$ is consistent across domains. Since a correlation incorporating
features entirely from the objects can also be a spurious correlation. Figure
2 shows a structural causal model (SCM) that describes the data-generating
process for the domain generalization task.

Figure 2: The Structural Causal Model for the data-generating process with a
node $\hat{\mathbf{x}}$ leading to object-induced spurious correlations.

In this figure, data is divided into two parts: domain-side and object-side.
$\overline{x}$ of domain-side is the reason for Dsc. For object-side, feature
is further divided into $\dot{x}$ and $\widehat{x}$ where $\widehat{x}$ is the
reason for Osc, just like the body color of objects in Section 3.1. The three
parts together make up the observed data. Thus even if we exclude all the
domain-induced spurious correlations, i.e., entirely remove the effect from
$\overline{x}$, we may still obtain object-induced spurious correlations
resulting from $\widehat{x}$.

##  6 Methods

To sample heterogeneous domains, a powerful method is DPP sampling, a point
process which mimics repulsive interactions between samples. Based on the
similarity matrix between the data points, a draw from a DPP yields
diversified subsets. Using DPP, we propose a diversity boosted two-level
sampling framework named DOMI to tackle the issue of scalability and help
train a robust model by excluding spurious correlations from both the domain-
side and object-side.

###  6.1 invDANN

We utilize invDANN to learn domain representations. In level-one-sampling of
DOMI, invDANN trains a featurizer to extract domain features and thus help
select diverse domains.

Domain-Adversarial Neural Networks (DANN) proposed by (Ganin et al., 2016) is
composed by Featurizer, Classifier and Discriminator. Featurizer extracts
features of data samples, Classifier learns to classify class labels of data
and Discriminator learns to discriminate domains. Since DANN aims to obtain a
model can not differentiate domains to ensure Featurizer captures domain-
independent features, Discriminator is connected to the Featurizer via a
gradient reversal layer that multiplies the gradient by a certain negative
constant during backpropagation. Gradient reversal ensures that the feature
distributions over the two domains are made similar, thus resulting in domain-
independent features.

Using the architecture of DANN, we let Classifier learn to classify domain
labels of data while Discriminator learns to discriminate class labels. As an
inverse version of DANN, invDANN aims to train a model which can classify
domains while can not distinguish class labels. Thus we can get Featurizer
extracting only domain-side features.

###  6.2 DOMI

Figure 3: Illustration of the sampling framework of DOMI. The solid arrow
indicates the actual sampling flow, while the dotted arrow only indicates the
difference between randomly sampling and DOMI.

Figure 3 shows the sampling procedure of DOMI. In level-one-sampling of DOMI,
we first use invDANN to train a featurizer extracting features of domains
rather than objects on a subset of domains and data. Since the featurizer aims
at domain-side features, DPP based on $L_{d}$ can select domains to help
exclude domain-induced spurious correlations according to Proposition 1. Then,
in level-two-sampling, since we do not have available labels just like domain
labels in level one, it is infeasible to utilize invDANN again to train a
featurizer. So we instead use ERM to train a featurizer. As we mentioned
before, ERM is prone to taking shortcuts and learning spurious correlations.
Moreover, since domains attained by level-one-sampling contain diverse data on
the domain-side, ERM can avert learning Dscs. Combining these two, ERM in
level two can train a featurizer extracting features of Osc. Thus batches
selected by DPP based on $L_{b}$ can help exclude object-induced spurious
correlations. Finally, we get a subset of the dataset to tackle the issue of
scalability under the setting of tremendous domains and training on which help
obtain robust models against impacts from both Osc and Dsc.

Input: Dataset $\\{(x_{i}^{d},y_{i}^{d})\\}_{i=1}^{n_{d}}$ from $|\textbf{D}|$
domains

Output: Heterogeneous sub-dataset for training

1 Level-one-sampling

2 Train $featurizer_{1}$ using invDANN on
$\\{(x_{i}^{d},y_{i}^{d})\\}_{i=1}^{n<n_{d}}$ from domain set $D$,
$|D|<|\textbf{D}|$ ;

3 for  _d in $D$_ do

4 Extract features of all data points in $d$ denoted as $feat_{d}$ by
$featurizer_{1}$ ;

5 Taking average of $feat_{d}$ as the description of $d$;

6

7Computing similarity matrix $(L_{d})$ of descriptions of all domains;

8 Obtain set of diverse domains $(\Omega)$ by DPP sampling based on $L_{d}$;

9 Level-two-sampling

10 Train $featurizer_{2}$ using ERM on dataset
$\\{(x_{i}^{d},y_{i}^{d})\\}_{i=1}^{n_{d}}$ from $\Omega$;

11 for  _$b$ in all batches of dataset_ do

12 Extract features of all data points in $b$ denoted as $feat_{b}$ by
$featurizer_{2}$;

13 Taking average of $feat_{b}$ as the description of $b$;

14

15Computing similarity matrix $(L_{b})$ of descriptions of all batches;

16 Obtain diverse batches by DPP sampling based on $L_{b}$;

Algorithm 1 Sampling Procedure of DOMI

##  7 Experiments

We have investigated the performance of DOMI with three backbone DG algorithms
on two simulated benchmarks (Rotated MNIST, Rotated Fashion MNIST) and
iwildcam, which show that DOMI can help substantially get higher test
accuracy. The settings and results are shown as follows.

###  7.1 Configurations

#### Datasets

To satisfy the setting of large number of domains, we extend the original
simulated benchmarks on MNIST and Fashion MNIST by Piratla et al. (2020) from
rotating images 15° through 75° in intervals of 15° to intervals of 1° in the
training set, i.e., 61 domains in total. And we get test accuracy on the test
set which rotates images either 0° or 90°. Moreover, while the original
datasets rotate the same images for different degrees, we extend them by
rotating different images for each domain to fit the real cases in DG tasks.
WILDS (Koh et al., 2021) is a curated collection of benchmark datasets
representing distribution shifts faced in the wild. As one dataset in WILDS,
iwildcam contains photos of wild animals and 324 different camera traps are
taken as domains.

#### Backbones

We take MatchDG (Mahajan et al., 2021), FISH (Shi et al., 2021b) and DANN
(Ganin et al., 2016) as backbone algorithms. The former two algorithms train
models by pairwise domain operations, both causal matching of MatchDG and
gradient matching of FISH lead to $O(n^{2})$ computational complexity with $n$
domains. Since they are prohibitive to be scaled to DG tasks with a large
number of domains, it’s essential to sample part of the most informative
domains. And we further incorporate DANN as one of the backbone algorithms in
that DOMI can not only efficiently select domains by its first level but can
help deal with circumstances where each domain contains massive data by the
second level.

#### Baselines

For each one of the backbone algorithms, we set the baseline as training on
domains selected by randomly sampling scheme, compared to level-one-sampling
of DOMI and complete DOMI. We sample 5 domains for training on Rotated MNIST
and Fashion MNIST and 10 domains on iwildcam as in the original experiment of
Shi et al. (2021b). We keep other factors such as model architecture and
learning rates the same for different sampling schemes. On iwildcam, we access
30 domains and all data points in each domain for the training of level-one-
sampling in invDANN. On the other two datasets, we utilize 40 domains and 750
of 2000 data points in each domain. The number of batches ($\delta$) selected
in level-two-sampling is a hyperparameter. A smaller $\delta$ help efficiently
mitigate strong Osc and speed up training, but when Osc is weak, a small
$\delta$ leads to a waste of training data. When $\delta$ equals the number of
entire batches, DOMI reduces to DOMI with only level one sampling. In the
experiment we set $\delta$ as 115 of 157 for Rotated Fashion MNIST and 135 of
157 for Rotated MNIST. On iwildcam, since the number of batches vary among
domains, we set $\delta$ as 80% of total batches in the selected domains.

#### Model selection

During training, we use a validation set to measure the model’s performance.
The test accuracy of the model is updated after an epoch if it shows better
validating performance. That is, we save the model showing the highest
validation accuracy after the training procedure, obtain its test accuracy and
report results. For iwildcam, we use the standard validation set in WILDS. For
Rotated MNIST and Fashion MNIST, data from only source domains (rotation
degree is from 15 ° to 75 °) are used to form the validation set since using
data from target domains (rotation degree is 0 ° and 90 °) for validation goes
against the motivation of generalization to unseen domains.

###  7.2 MatchDG

MatchDG is a two-phase method, and in our experiment we set 30 epochs of
training for phase 1 and 25 epochs for phase 2. We repeat the experiment of
MatchDG 20 times with random seeds, and Table 3 shows the average test
accuracy of three sampling schemes on two datasets.

While $level_{1}$ gets higher accuracy on Rotated MNIST and $level_{2}$ shows
better performance on Fashion MNIST, they all outperform $level_{0}$, i.e.,
randomly sampling.

Table 3: Average test accuracy of MatchDG. Here $level_{0}$ denotes randomly sampling, $level_{1}$ denotes level-one-sampling of DOMI and $level_{2}$ is the complete version of DOMI. | $level_{0}$ | $level_{1}$ | $level_{2}$
---|---|---|---
Rotated MNIST | 82.0 | 84.3 | 83.9
Fashion MNIST | 38.6 | 39.5 | 39.8

Moreover, during training we observed that the test accuracy first rises to
the peak and then begins to decline along with the increase of validation
accuracy. This reduction indicates the model overfits to spurious
correlations. Thus we further record the peek value of the test accuracy in
each experiment with random seeds and denote it as maximal accuracy. After
repeating the experiment about 20 times (19 times on Rotated MNIST and 22
times on Fashion MNIST), the distribution of the test accuracy and maximal
accuracy under different sampling schemes is shown in Figure 4. While the test
accuracy of $level_{0}$ scatters, that of $level_{1}$ and $level_{2}$ centers.
Moreover, the gap between test accuracy and maximal accuracy of $level_{1}$ is
smaller than that of $level_{0}$ and $level_{2}$ further shrinks the gap.
Table 3 and Figure 4 indicate that DOMI helps train a robust model with good
performance.

(a)

(b)

Figure 4: Boxplot of test accuracy and maximal accuracy among about 20 times
of repeated experiments leveraging different sampling levels on Fashion MNIST
and Rotated MNIST. Here each tiny circle represents one experiment, of which
the vertical location corresponds to the accuracy value. The horizontal line
inside each box indicates the mean value.

###  7.3 DANN

For DANN, the training epochs are set to be 50. Table 4 shows the test
accuracy of three sampling schemes on two datasets. Similar to MatchDG, while
both $level_{1}$ and $level_{2}$ outperform $level_{0}$ on two datasets,
$level_{2}$ gets relatively lower average accuracy than $level_{1}$ on Rotated
MNIST and obviously outperforms $level_{1}$ on Fashion MNIST. A rational
explanation is that Fashion MNIST in this experiment shows strong Osc which is
weak on Rotated MNIST. Thus a small $\delta$ in level-two-sampling can
efficiently alleviate the impacts from strong Osc on Fashion MNIST while it
may lead to a certain degree of waste of training data on Rotated MNIST.
Figure 5 shows the average test accuracy for each epoch with different
$\delta$ on these two datasets. On both datasets, while a smaller $\delta$
leads to slower growth in accuracy at the initial part of training, it helps
get a more robust result, shrinking the gap between maximal accuracy and test
accuracy. On Fashion MNIST, both $\delta=75$ and $\delta=115$ outperform
training on all data batches, yet $\delta=157$ gets the best result on Rotated
MNIST.

Figure 6 shows the average maximal accuracy and test accuracy under different
sampling levels of 30 experiments with random seeds. Similar to Section 7.2,
$level_{1}$ and $level_{2}$ enhance test accuracy and $level_{2}$ obviously
shrinks the gap between test accuracy and maximal accuracy, showing that it
helps get a more robust model.

Table 4:  Average accuracy on Rotated MNIST and Rotated Fashion MNIST under three sampling schemes of DANN. | $level_{0}$ | $level_{1}$ | $level_{2}$
---|---|---|---
Rotated MNIST | 76.2 | 77.2 | 76.9
Fashion MNIST | 40.9 | 41.7 | 43.4

(a)

(b)

Figure 5: Average test accuracy of 20 experiments with random seeds during 50
epochs with different $\delta$ on Fashion MNIST and Rotated MNIST of DANN.
$\delta=157$ corresponds to DOMI with only level one.

(a)

(b)

Figure 6: Average test accuracy and maximal accuracy of 30 times of
experiments under different sampling levels on Fashion MNIST and Rotated MNIST
of DANN.

###  7.4 FISH

For FISH, the training epochs are set to be 5. Each epoch contains 300
iterations and we observe test accuracy every 30 iterations. Unlike MatchDG
and DANN, fish needs to sample domains in each iteration instead of training
on one list of domains. Sampling domains in each iteration will result in
great computational overhead compared to randomly sampling. Thus we just
sample 30 domain lists containing diverse domains using level-one-sampling of
DOMI and repeatedly train the model on these domain lists(one list for one
iteration) for $level_{1}$. As for $level_{2}$, we further utilize level-two-
sampling to sample data batches of each domain in the domain lists for
training. Table 5 shows the test accuracy. $level_{2}$ further enhances
performance by $level_{1}$, and both of them apparently outperform
$level_{0}$.
Figure 7 shows average test accuracy for each epoch with different sampling
schemes on Fashion MNIST and Rotated MNIST. On both datasets, while
$level_{2}$ leads to slower growth in accuracy at the initial part of training
because of using a smaller number of batches, it keeps outperforming
$level_{1}$ and $level_{0}$ at later epochs. $level_{1}$ also shows better
performance than $level_{0}$.

Table 5:  Average test accuracy of 10 experiments with random seeds on Rotated MNIST and Rotated Fashion MNIST under three sampling schemes of FISH.  | $level_{0}$ | $level_{1}$ | $level_{2}$
---|---|---|---
Rotated MNIST | 65.2 | 66.5 | 66.6
Fashion MNIST | 33.2 | 34.5 | 35.8

(a)

(b)

Figure 7: Average test accuracy of 10 experiments with random seeds during 50
epochs with different sampling schemes of FISH. Here we slightly abuse epoch
to mean the time we obtain test accuracy.

###  7.5 Experiments on iwildcam

The data of iwildcam is extremely unbalanced, while part of the domains
contain less than 20 photos, some domains contain over 2000 ones. In the
original experiments of Shi et al. (2021b), iwlidcam is divided into batches
in each domain. FISH samples a certain number of batches from different
domains for training in each iteration. The sampling probability of one batch
in a domain is proportional to the number of batches left in this domain. This
sampling scheme is taken as $level_{0}$ here and the result of $level_{0}$ is
taken from (Shi et al., 2021b). In each iteration, $level_{1}$ samples the
most diverse batches based on DPP using invDANN, $level_{2}$ samples some
batches in the level-one-sampling and among them further selects a subset of
batches in the level-two-sampling. Under the same setting in the original
experiments, the results on iwildcam of FISH are shown in Table 6 .

Table 6:  Macro F1 score of FISH on iwildcam under three sampling schemes | $level_{0}$ | $level_{1}$ | $level_{2}$
---|---|---|---
Iwildcam | 22.0 | 22.8 | 23.4

Although DOMI gets higher Macro F1 score, it leads to a much larger
computational overhead since it needs to do sampling in each iteration.
Moreover, for DANN and MatchDG, Macro F1 of diverse domains may be
significantly lower than randomly sampled domains because of the unbalanced
data, i.e., the diverse domains may contain much fewer data compared to the
randomly sampled domains. It would be a significant future work to tackle the
issues of extremely imbalanced data and computational overhead for algorithms
that need to do sampling for multi-times.

##  8 Conclusion

Under the setting of large number of domains and domains with massive data
points, we propose a diversity boosted two-level sampling algorithm named DOMI
to help sample the most informative subset of dataset. Empirical results show
that DOMI substantially enhances the out-of-domain accuracy and gets robust
models against spurious correlations from both domain-side and object-side.

## References

  * Ahuja et al. (2020a) Ahuja, Kartik, Shanmugam, Karthikeyan, Varshney, Kush, and Dhurandhar, Amit.  Invariant risk minimization games.  In _International Conference on Machine Learning_ , pp. 145–155. PMLR, 2020a.
  * Ahuja et al. (2020b) Ahuja, Kartik, Wang, Jun, Dhurandhar, Amit, Shanmugam, Karthikeyan, and Varshney, Kush R.  Empirical or invariant risk minimization? a sample complexity perspective.  _arXiv preprint arXiv:2010.16412_ , 2020b.
  * Arjovsky et al. (2019) Arjovsky, Martin, Bottou, Léon, Gulrajani, Ishaan, and Lopez-Paz, David.  Invariant risk minimization.  _arXiv preprint arXiv:1907.02893_ , 2019.
  * Ben-Tal et al. (2013) Ben-Tal, Aharon, Den Hertog, Dick, De Waegenaere, Anja, Melenberg, Bertrand, and Rennen, Gijs.  Robust solutions of optimization problems affected by uncertain probabilities.  _Management Science_ , 59(2):341–357, 2013.
  * Blanchard et al. (2011) Blanchard, Gilles, Lee, Gyemin, and Scott, Clayton.  Generalizing from several related classification tasks to a new unlabeled sample.  _Advances in neural information processing systems_ , 24, 2011.
  * Duchi et al. (2016) Duchi, John, Glynn, Peter, and Namkoong, Hongseok.  Statistics of robust optimization: A generalized empirical likelihood approach.  _arXiv preprint arXiv:1610.03425_ , 2016.
  * Ganin et al. (2016) Ganin, Yaroslav, Ustinova, Evgeniya, Ajakan, Hana, Germain, Pascal, Larochelle, Hugo, Laviolette, Francois, Marchand, Mario, and Lempitsky, Victor.  Domain-adversarial training of neural networks.  _The journal of machine learning research_ , 17(1):2096–2030, 2016.
  * Ghifary et al. (2016) Ghifary, Muhammad, Balduzzi, David, Kleijn, W Bastiaan, and Zhang, Mengjie.  Scatter component analysis: A unified framework for domain adaptation and domain generalization.  _IEEE transactions on pattern analysis and machine intelligence_ , 39(7):1414–1430, 2016.
  * Hoffman et al. (2018) Hoffman, Judy, Tzeng, Eric, Park, Taesung, Zhu, Jun-Yan, Isola, Phillip, Saenko, Kate, Efros, Alexei, and Darrell, Trevor.  Cycada: Cycle-consistent adversarial domain adaptation.  In _International conference on machine learning_ , pp. 1989–1998. Pmlr, 2018.
  * Hu et al. (2020a) Hu, Shoubo, Zhang, Kun, Chen, Zhitang, and Chan, Laiwan.  Domain generalization via multidomain discriminant analysis.  In _Uncertainty in Artificial Intelligence_ , pp. 292–302. PMLR, 2020a.
  * Hu et al. (2020b) Hu, Weihua, Fey, Matthias, Zitnik, Marinka, Dong, Yuxiao, Ren, Hongyu, Liu, Bowen, Catasta, Michele, and Leskovec, Jure.  Open graph benchmark: Datasets for machine learning on graphs.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020b.
  * Javed et al. (2020) Javed, Khurram, White, Martha, and Bengio, Yoshua.  Learning causal models online.  _arXiv preprint arXiv:2006.07461_ , 2020.
  * Ji et al. (2022) Ji, Yuanfeng, Zhang, Lu, Wu, Jiaxiang, Wu, Bingzhe, Huang, Long-Kai, Xu, Tingyang, Rong, Yu, Li, Lanqing, Ren, Jie, Xue, Ding, et al.  Drugood: Out-of-distribution (ood) dataset curator and benchmark for ai-aided drug discovery–a focus on affinity prediction problems with noise annotations.  _arXiv preprint arXiv:2201.09637_ , 2022.
  * Kamath et al. (2021) Kamath, Pritish, Tangella, Akilesh, Sutherland, Danica, and Srebro, Nathan.  Does invariant risk minimization capture invariance?  In _International Conference on Artificial Intelligence and Statistics_ , pp. 4069–4077. PMLR, 2021.
  * Koh et al. (2021) Koh, Pang Wei, Sagawa, Shiori, Marklund, Henrik, Xie, Sang Michael, Zhang, Marvin, Balsubramani, Akshay, Hu, Weihua, Yasunaga, Michihiro, Phillips, Richard Lanas, Gao, Irena, et al.  Wilds: A benchmark of in-the-wild distribution shifts.  In _International Conference on Machine Learning_ , pp. 5637–5664. PMLR, 2021.
  * Krueger et al. (2021) Krueger, David, Caballero, Ethan, Jacobsen, Joern-Henrik, Zhang, Amy, Binas, Jonathan, Zhang, Dinghuai, Le Priol, Remi, and Courville, Aaron.  Out-of-distribution generalization via risk extrapolation (rex).  In _International Conference on Machine Learning_ , pp. 5815–5826. PMLR, 2021.
  * Kulesza et al. (2012) Kulesza, Alex, Taskar, Ben, et al.  Determinantal point processes for machine learning.  _Foundations and Trends® in Machine Learning_ , 5(2–3):123–286, 2012.
  * Lee et al. (2019) Lee, Juho, Lee, Yoonho, Kim, Jungtaek, Kosiorek, Adam, Choi, Seungjin, and Teh, Yee Whye.  Set transformer: A framework for attention-based permutation-invariant neural networks.  In _International Conference on Machine Learning_ , pp. 3744–3753. PMLR, 2019.
  * Li et al. (2016) Li, Chengtao, Jegelka, Stefanie, and Sra, Suvrit.  Efficient sampling for k-determinantal point processes.  In _Artificial Intelligence and Statistics_ , pp. 1328–1337. PMLR, 2016.
  * Li et al. (2018) Li, Ya, Gong, Mingming, Tian, Xinmei, Liu, Tongliang, and Tao, Dacheng.  Domain generalization via conditional invariant representations.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 32, 2018.
  * Liu et al. (2021) Liu, Jiashuo, Hu, Zheyuan, Cui, Peng, Li, Bo, and Shen, Zheyan.  Heterogeneous risk minimization.  In _International Conference on Machine Learning_ , pp. 6804–6814. PMLR, 2021.
  * Long et al. (2015) Long, Mingsheng, Cao, Yue, Wang, Jianmin, and Jordan, Michael.  Learning transferable features with deep adaptation networks.  In _International conference on machine learning_ , pp. 97–105. PMLR, 2015.
  * Mahajan et al. (2021) Mahajan, Divyat, Tople, Shruti, and Sharma, Amit.  Domain generalization using causal matching.  In _International Conference on Machine Learning_ , pp. 7313–7324. PMLR, 2021.
  * Muandet et al. (2013) Muandet, Krikamol, Balduzzi, David, and Schölkopf, Bernhard.  Domain generalization via invariant feature representation.  In _International Conference on Machine Learning_ , pp. 10–18. PMLR, 2013.
  * Piratla et al. (2020) Piratla, Vihari, Netrapalli, Praneeth, and Sarawagi, Sunita.  Efficient domain generalization via common-specific low-rank decomposition.  In _International Conference on Machine Learning_ , pp. 7728–7738. PMLR, 2020.
  * Rosenfeld et al. (2020) Rosenfeld, Elan, Ravikumar, Pradeep, and Risteski, Andrej.  The risks of invariant risk minimization.  _arXiv preprint arXiv:2010.05761_ , 2020.
  * Sagawa et al. (2019) Sagawa, Shiori, Koh, Pang Wei, Hashimoto, Tatsunori B, and Liang, Percy.  Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization.  _arXiv preprint arXiv:1911.08731_ , 2019.
  * Shen et al. (2021) Shen, Zheyan, Liu, Jiashuo, He, Yue, Zhang, Xingxuan, Xu, Renzhe, Yu, Han, and Cui, Peng.  Towards out-of-distribution generalization: A survey.  _arXiv preprint arXiv:2108.13624_ , 2021.
  * Shi et al. (2021a) Shi, Claudia, Veitch, Victor, and Blei, David M.  Invariant representation learning for treatment effect estimation.  In _Uncertainty in Artificial Intelligence_ , pp. 1546–1555. PMLR, 2021a.
  * Shi et al. (2021b) Shi, Yuge, Seely, Jeffrey, Torr, Philip HS, Siddharth, N, Hannun, Awni, Usunier, Nicolas, and Synnaeve, Gabriel.  Gradient matching for domain generalization.  _arXiv preprint arXiv:2104.09937_ , 2021b.
  * Wang et al. (2022) Wang, Jindong, Lan, Cuiling, Liu, Chang, Ouyang, Yidong, Qin, Tao, Lu, Wang, Chen, Yiqiang, Zeng, Wenjun, and Yu, Philip.  Generalizing to unseen domains: A survey on domain generalization.  _IEEE Transactions on Knowledge and Data Engineering_ , 2022.
  * Zhai et al. (2021) Zhai, Runtian, Dan, Chen, Kolter, Zico, and Ravikumar, Pradeep.  Doro: Distributional and outlier robust optimization.  In _International Conference on Machine Learning_ , pp. 12345–12355. PMLR, 2021.
  * Zhao et al. (2018) Zhao, Han, Zhang, Shanghang, Wu, Guanhang, Moura, José MF, Costeira, Joao P, and Gordon, Geoffrey J.  Adversarial multiple source domain adaptation.  _Advances in neural information processing systems_ , 31, 2018.
  * Zhao et al. (2019) Zhao, Han, Des Combes, Remi Tachet, Zhang, Kun, and Gordon, Geoffrey.  On learning invariant representations for domain adaptation.  In _International Conference on Machine Learning_ , pp. 7523–7532. PMLR, 2019.
  * Zhou et al. (2021) Zhou, Kaiyang, Liu, Ziwei, Qiao, Yu, Xiang, Tao, and Loy, Chen Change.  Domain generalization: A survey.  2021\.

Appendix of DOMI

.tocmtappendix mtchapternone

##  Appendix A The Simulated Dataset

Table 7: The simulated dataset of the toy example. From these 12 data points we sample 6 for training. | $D_{1}$ | $D_{2}$ | $D_{3}$ | $D_{4}$ | $D_{5}$ | $D_{6}$ | $D_{7}$ | $D_{8}$ | $D_{9}$ | $D_{10}$ | $D_{11}$ | $D_{12}$
---|---|---|---|---|---|---|---|---|---|---|---|---
$X_{1}$ | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1
$X_{2}$ | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 1 | 1
$X_{3}$ | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1
$X_{4}$ | 0 | 0 | 0 | 1 | 0 | 1 | 1 | 1 | 0 | 0 | 1 | 1
$Y$ | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 1 | 1


# Comparing the carbon costs and benefits of low-resource solar nowcasting

Ben Dixon
University College London (UK)
<EMAIL_ADDRESS>
Jacob Bieker
Open Climate Fix
<EMAIL_ADDRESS>
María Pérez-Ortiz
AI Centre, University College London (UK)
<EMAIL_ADDRESS>
Primary author

###### Abstract

Mitigating emissions in line with climate goals requires the rapid integration
of low carbon energy sources, such as solar photovoltaics (PV) into the
electricity grid. However, the energy produced from solar PV fluctuates due to
clouds obscuring the sun’s energy. Solar PV yield nowcasting is used to help
anticipate peaks and troughs in demand to support grid integration. This paper
compares multiple low-resource approaches to nowcasting solar PV yield. To do
so, we use a dataset of UK satellite imagery and solar PV energy readings over
a 1 to 4-hour time range. Our work investigates the performance of multiple
nowcasting models. The paper also estimates the carbon emissions generated and
averted by deploying models such as these, and finds that short-term PV
forecasting may have a benefit several orders of magnitude greater than its
carbon cost and that this benefit holds for small models that could be
deployable in low-resource settings around the globe.

##  1 Introduction

Solar photovoltaics (PV) is one of the safest and cleanest sources of energy,
as measured in death rates from accidents and air pollution, and greenhouse
gas emission [11]. However, the fluctuations in the energy output, or yield,
from solar PV makes it challenging for the electricity grid to rely at large
scale on solar energy. The yield of solar PV is inherently uncertain because
clouds can obscure the sun, and clouds can become thicker, thinner, and their
movement across the sky is hard to predict [2]. In order to guarantee stable
energy supply, grid operators often have to keep gas spinning-reserve online,
releasing carbon dioxide into the atmosphere [7]. Forecasting over less than
one day in advance is called ‘nowcasting’ [13] to emphasize its dynamic use.
Nowcasting solar energy input into the grid accurately could reduce the need
to keep the reserves on standby, allowing to use every kilowatt efficiently
and enhancing the grid resilience to the fluctuations of renewable energy
sources.

Since 2019, Open Climate Fix (OCF) has been working with National Grid ESO,
the electricity system operator for Great Britain, to build accurate PV
nowcasting models that can enhance the resilience of the energy grid [5]. This
paper complements the work done by OCF. Specifically, we research alternative
low-resource methods (in terms of smaller/shallower neural networks
architectures) for predicting PV yield, and analyse their performance, their
carbon footprints and aim to estimate, even if roughly, the net emissions that
could be averted by their potential use.

As more of the world incorporates solar PV into energy grids, there will be an
increased need to develop accurate solar PV forecasts. While a body of recent
work has emphasized building more computationally efficient models, most of
the machine learning work still focuses on the opposite: building larger
models with more parameters to tackle more complex tasks [8]. Recent work
poses the question of how much is the performance gain worth [8], specially in
these applications where deployment of a simple technology could already have
a big impact. The move towards less computationally intensive models has
already happened once in weather forecasting, as deep learning, despite its
competitiveness, has often lower computational requirements than numerical
weather prediction based on differential equations [12, 13].

Through our work, we study low-resource and shallower predictive models and
try to estimate the benefit that they could bring if they were deployed large-
scale for solar nowcasting111Our low-resource settings are not simulated, this
project has been done as part of a MSc thesis dissertation, with no other
computational resources.. The models, which are build on one Tesla T4 GPU
available through Google Colab, could be deployed large-scale around the world
independently of hardware limitations. The reason we are interested in
researching this low-resource setting, by studying smaller models, is because
those are the models that could be deployed to support the grid in emerging
economies, which are where the real need is for clean energy solutions, i.e.
where the energy demand is rapidly growing.

##  2 An empirical comparison of PV yield nowcasting: Experimental results

For the task at hand, we study convolutional neural networks (CNNs), Long
Short-Term Memory networks (LSTMs) and their intersection [13] (ConvLSTMs). In
theory, this comparison may seem futile as LSTMs and ConvLSTMs should
outperform CNNs, because of their use of state information. However, Bai et al
[1] argue that simpler CNNs can often outperform LSTMs. Moreover, ConvLSTMs
require a large amount of memory simultaneously available in order to process
and update the state information used in its predictions, which means they
often take longer to train, leading to more energy usage and hence higher
emissions.

#### Dataset

Solar generation data refers to the yield from specific solar panels. The
dataset222Thanks to Open Climate Fix for providing the datasets. is composed
of satellite imaging and PV readings. The satellite images are provided by
EUMETSAT, covering the northern third of the Meteosat disc every five minutes.
Open Climate Fix developed the `eumetsat_uk_hrv` dataset [6] which takes the
‘high resolution visible’ (HRV) channel for a reduced time period and
geospatial extent. The dataset was reduced in this paper to meet computing
resources available. A 64 x 64 crop was randomly chosen to focus on Devon,
selecting images taken between 05:00 and 20:00 as there is minimal PV output
outside of this window for most of the year. However, this still left many
readings in winter when the sun was below the horizon, and the PV yield was
zero, so readings when the solar altitude was below 10 degrees were also
dropped. Open Climate Fix provided a dataset of 1311 energy yield readings
from PV systems in 5 minute increments for most of 2018-2021, originally
provided by a PV operator. The size of the dataset varied slightly based on
the prediction window. For the shortest window, there were 2018 observations
of (12, 64, 64) and (12, 1) to predict a series of 12 readings. For the
longest prediction window, there were 659 blocks of the same sizes described
earlier to predict a series of 48 readings.

#### Learning task

We formulate our learning task as predicting a sequence of values representing
the future PV yield. All models take as input 12 sets of 5 min data, i.e., 1
hour of data as input, and predict forward between 1 and 4 hours. We also
consider two learning scenarios: i) learning exclusively from past PV data
only, and ii) learning from both past PV and satellite data as inputs.

#### Models compared

For each learning scenario (with and without satellite imaging), we test both
CNNs and LSTMs, our objective being comparing their prediction efficiency and
carbon footprint at the task at hand. Additionally, LSTMs encode time
relationships explicitly, and we are interested in evaluating if this
additional complexity increases the prediction performance of our models to a
significant extent (and at what potential carbon cost). For the PV only models
(CNN and LSTM), which are the simplest models we test, we take in past PV
through multiple layers of CNNs, followed by a fully connected layer. The
other PV only model has a similar structure but uses LSTMs in place of CNNs.
The second group of models take both satellite images and past PV yield as
input. The Conv3D is a CNN which uses multiple levels to try to learn abstract
features. After each convolutional layer, a MaxPooling layer is used to reduce
the dimensionality and learn the features. This is followed by two dense
layers to resize the output to predict a sequence of values. The CNN treats
the sequence as one vector. The ConvLSTM model (see 1) is similar to the
Conv3D but uses recurrent modelling, with ConvLSTMs to process the images and
LSTMs to process the PV input sequence.

Figure 1: The ConvLSTM model, using both satellite images and past PV yield.

#### Model training and dataset split

The model is trained on 22 months of data from 1/1/2020 to 7/11/2021. To avoid
autocorrelation, the datasets are drawn from each month: days 1 to 20 are for
training, days 22 to 24 for validation, and days 27 to 29 for testing. For all
models, the models are trained to optimise MSE until convergence. Bayesian
hyperparameter tuning was done: The learning rate was varied between [0.01,
0.001, 0.0001, 0.00001], and dropout rate was varied between [0.05, 0.5] in
steps of 0.05. For the CNNs and the ConvLSTMs, the kernel size was varied
between [3,5,7]. For ConvLSTMs the number of filters was varied between
[16,32,64]. For the fully connected layers the number of nodes was varied in
the set [12,24,48]. The ConvLSTM has three ConvLSTM layers applied to images,
and one LSTM layer applied to past PV. The CNN model has three convolutional
layers applied to the images, and one convolutional layer applied to past PV
readings. For the PV-only models, the LSTM model has three layers of LSTMs,
and the CNN model has three layers of CNNs.

###  2.1 Results

The task is to predict a sequence of values of variable lengths, from 1 hour
to 4h. Table 1 shows normalised MAE and RMSE by model over different
prediction windows. All models outperform the persistence baseline, often used
for physical processes. ConvLSTM outperforms other models, especially over
longer time horizons. Interestingly, an LSTM model taking PV only delivers
results competitive with a Conv3D which also uses satellite data five out of
eight times. Does time series modelling improves predictions? The recurrent
designs have a lower error metric five out of eight times, but often by less
than 0.2%. Do satellite images improve performance? Yes, on six of eight
times, and by margins approximately increasing 0.2%-1.2%. Overall, it appears
that explicit modelling of temporal information improves the performance to a
large extent. This is most significant for the comparison between ConvLSTM and
CNN, which is when the inputs include image data. Perhaps this is because
image data carries important trend information, such as motion of clouds.

|  | Baseline | PV only | Sat and PV
---|---|---|---|---
Time | Metric | Persistence | CNN | LSTM | Conv3D | ConvLSTM
60 mins | RMSE | 11.60% | 10.46% | 9.42% | 9.44% | 9.29%
MAE | 7.91% | 7.92% | 6.60% | 6.81% | 6.60%
120 mins | RMSE | 15.18% | 11.86% | 11.93% | 11.77% | 11.55%
MAE | 11.03% | 8.72% | 9.07% | 8.55% | 8.41%
180 mins | RMSE | 17.84% | 12.83% | 12.94% | 13.20% | 12.26%
MAE | 13.10% | 10.13% | 10.00% | 9.84% | 9.15%
240 mins | RMSE | 20.32% | 13.16% | 13.02% | 13.75% | 12.52%
MAE | 14.95% | 10.16% | 10.14% | 10.08% | 9.53%
Table 1: Normalised prediction error over different time horizons, with best
results highlighted in bold.

###  2.2 Carbon emissions

This section estimates the energy requirements of our models. The carbon costs
are calculated through the time taken multiplied by estimates of the carbon
intensity of time using a GPU. Each of the models was run on a Tesla T4 GPU
available through Google Colab, assumed to be in Northern Europe, the region
closest to our location, which has a carbon efficiency of 0.21 kgCO2eq/kWh.
Energy usage simulations were conducted using the Machine Learning Impact
calculator [9]. Table 2 uses the computational time required to train and make
inferences over a two-hour prediction window and compare the ConvLSTM and 3D
CNN. It is assumed that the predictions are generated for each of the 330 UK
grid supply points, and are generated every hour, from 06:00 to 20:00, and for
a whole year, giving 330 * (20-6) * 365 = 1686,300 forecasts per year.

| Conv3D | ConvLSTM
---|---|---
Time to train model (s) | 916.39 | 673.11
Time for inference, one forecast (s) | 0.16 | 2.21
Implies: Total time for year (training + 1686,300 * inferences) (hrs) | 75 | 1024
Emissions generated (tonnes $CO_{2}$-equiv) from [9] | 0.0108 | 0.152

Table 2: Estimates of emissions generated through training and deploying
Conv3D and ConvLSTM

.

The carbon reductions through increased model accuracy are harder to quantify.
In 2020, the UK energy grid was estimated to produce 51 million tonnes of
$C0_{2}$-equivalent emissions [10]. Currently, 2% of UK energy currently comes
from solar PV, which could rise to 7% by 2050 [4]. Emissions of $C0_{2}$ per
gigawatt hour of electricity over the lifecycle of solar plant is 5 tonnes
[11], where gigawatt hour is the annual electricity consumption of 150
Europeans. Based on this data, and assuming 67.22 million people in UK and a
similar electrical consumption than Europeans, a 2% of solar energy would
generate annually 45,000 tonnes of emissions, which is 2450 times less than
keeping gas-reserves online for generating the same electricity [11]. Even if
we assume that the models in Table 2 are 0.1% better at estimating solar
output than the current standard of energy grid operators (which would be a
lower bound as in our analysis these models are still much better than
persistence, often a baseline followed for physical processes), and that in
this 0.05% of cases we may be able to turn off gas-reserves, we may be looking
at a reduction of around 5500 tonnes of $C0_{2}$ annually. Given the costs of
training the two models and generating the forecasts for a year are between
than 1/10th and 1/6th of a tonne of $C0_{2}$-equivalent, the benefits far
outweigh the costs. This puts in perspective the reduction that could be
possible if we were to deploy these low-resource models at large scale.

###  2.3 Analysing predictions and activations

The task addressed by this paper was predicting a sequence of PV values, and
was evaluated by performance on rMSE and MAE. But are the sequences of
predictions realistic? Appendix A examines the predictions given by the best
model, the ConvLSTM, over a one-hour time horizon by overlaying predicted
values an hour ahead with observed values. The predicted values generally
trend up and down correctly suggesting the model has learned the dynamics of
the sequence. However, this section also shows that the model consistently
under-predicts PV yield across the year, most strongly in early afternoon in
summer. This could be caused by the model having insufficient examples of
sunny weather, long periods of low yield, and by the use of a symmetric loss
function. Further work could explore whether loss functions which more heavily
penalise under-predictions improve results.

A frequent criticism of neural networks is that it is hard to interpret how
they determine their results. Appendix B examines the activations in a simpler
single-period neural network, to understand what drives the results, using an
filter visualisation approach inspired by Chollet [3]. This analysis suggests
the model might be learning to recognise the contrast between land and sea.
These activations would be associated with higher PV readings, as high
contrast would mean a bright and clear day. But the model might not be serving
our intended purpose of learning cloud dynamics. As a result, the model might
perform worse on satellite areas with no coastline. Further work could examine
the activations of the sequence models explored in the rest of this paper.

##  3 Conclusions

This paper includes an estimate of carbon emissions for training and running
the inference of low-resource shallow neural architectures for solar
photovoltaic nowcasting for a year. We attempt to estimate as well the
emissions that would be averted by having a better integration of solar in the
energy grid, showing that even under very pessimistic assumptions there is an
important benefit in terms of total carbon footprint averted from integrating
solar nowcasting machine learning models into the grid. Our analysis focuses
on low-resource models that could be integrated into emerging economies, where
energy requirements may be growing quickly. Further analyses are needed to
understand the benefit that larger and deeper models could bring.

The paper also examined the prediction errors and activations of a similar but
simpler neural network. This suggested the model might be influenced by an
unbalanced dataset, and that an augmented dataset or a different loss function
might improve results. Analysis of the activations suggested the model might
be learning to recognise contrast between land and sea, and further research
could examine performance on satellite images with no coastline.

A more in depth analysis of the current strategies implemented by grid
operators used to meet energy demands in case of disruption to the renewable
solar supply would be needed to better put in perspective the climate change
mitigation potential of nowcasting models.

##  Appendix A Analysing the predictions

This section explores the predictions for the best model, the ConvLSTM model,
over a one-hour time horizon. It begins by comparing the predictions against
true values for three randomly chosen days in the test set.

Figure 2: Predicted and observed PV yield across three randomly chosen days in
the test set. Predictions an hour ahead are in orange and observed values in
light blue. Note the variation in y values (solar yield) across days. The
model tends to under-predict sunny and cloudy days. The day on the left is a
darker winter day, the middle day is sunny from summer, and the right day is a
cloudy day in autumn.

Interestingly, it appears that the ConvLSTM generally predicts the overall
gradient of the prediction correctly - if the prediction window is getting
brighter or darker. However, in the middle day shown above, the model
consistently predicts that the PV yield will decrease. The middle day has PV
readings up to 75% of the two-year maximum, and is much brighter than the
other two examples which peak at around 30% and 60%. This suggests that the
model does not deal well with sunnier days, and might under-predict on those
days.

The plot below shows the total relative prediction errors per mean prediction
error over the sequence, across a two-year period. Since there are 12
predicted values with a range of [0,1], the largest possible magnitude of
error is 12. The models optimise for mean square error, which is not sensitive
to the sign of the error. This plot shows the direction of the error.

Figure 3: Plot of relative prediction error over two years - mean error over
sequence. Overall the model tends to have a negative error - unpredicts PV
yield. This is more extreme in summer, with smaller errors in winter.

It appears that the model tends to have the errors with the largest magnitude
in summer, lower magnitude errors in November, December, and January. Since
the test set days are three days chosen from the end of each month, the values
appear in lines which span the test set ranges.

##  Appendix B Interpretability

A frequent criticism of neural networks is that it is hard to interpret how
they determine their results. For simplicity, a simpler neural network was
trained to infer a single PV reading from a single image. The model was
trained for 10 epochs on a randomly selected sample of 1000 satellite images,
and it achieved a test set performance of a 2% mean square error. Below are
the activations for three images taken at different times with different
characteristics.

Activations for Case A - Bright conditions, high PV

Activations for Case B - Dark conditions, some cloud

Activations for Case C - Bright conditions, thick cloud

On the first layer on the left we see bright activations for the sea and the
land, as expected since the first layer of a CNN often learns textures. The
second layer shows high activation around the coast line and the sea, and the
third layer on the right seems to be mostly highly activated around the coast
line.

From these examples it seems that this network is learning to recognise
contrast between land and sea. Intuitively, a high contrast will mean a bright
sky with no clouds. This will be when the PV is highest, and so the
activations will be associated with better predictions.

But in this case the model might not have learnt any cloud features. This
might also be true for our main models - are they also just recognising
contrast? A useful area of further research would be examining the activations
in the more complex, time series models used in the rest of the paper.

## References

  * [1] Shaojie Bai, J. Kolter and Vladlen Koltun  “An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling”  URL: https://arxiv.org/abs/1803.01271
  * [2] Olivier Boucher and David Ransall  “Clouds and Aersols”  In _Climate Change 2013: The Physical Science Basis. Contribution of Working Group I. to the 5th Assessment Report of the Intergovernmental Panel on Climate_ Cambridge University Press, 2013, pp. 571–657
  * [3] Francois Chollet  “Deep Learning with Python (Second Edition)”  Manning, 2021
  * [4] Natioal Grid ESO  “Future Energy Scenarios: July 2022”, 2022  URL: https://www.nationalgrideso.com/future-energy/future-energy-scenarios
  * [5] National Grid ESO  “The Road to Zero Carbon”, 2022  URL: https://www.nationalgrideso.com/future-energy/our-progress/road-zero-carbon/report
  * [6] Open Climate Fix  “EUMETSAT Dataset”, 2022  URL: https://huggingface.co/datasets/openclimatefix
  * [7] David Wenzhong Gao  “Chapter 2 - Applications of ESS in Renewable Energy Microgrids”  In _Energy Storage for Sustainable Microgrid_ Oxford: Academic Press, 2015, pp. 35–77  DOI: https://doi.org/10.1016/B978-0-12-803374-6.00002-0
  * [8] Peter Henderson et al.  “Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning”  In _J. Mach. Learn. Res._ 21.1 JMLR.org, 2020
  * [9] Sasha Luccioni, Victor Schmidt, Alexandre Lacoste and Thomas Dandres  “Quantifying the Carbon Emissions of Machine Learning”  In _NeurIPS 2019 Workshop on Tackling Climate Change with Machine Learning_ , 2019  URL: https://www.climatechange.ai/papers/neurips2019/22
  * [10] UK Office National Statistics  “2020 UK greenhouse gas emissions”, 2021  URL: https://www.gov.uk/government/statistics/provisional-uk-greenhouse-gas-emissions-national-statistics-2020
  * [11] Hannah Ritchie  “What are the safest and cleanest sources of energy”  In _Our World Data. https://ourworldindata. org/safest-sources-of-energy. Accessed_ 22, 2021
  * [12] Martin Schultz et al.  “Can deep learning beat numerical weather prediction?”  In _Philosophical Transactions of The Royal Society A_ 379, 2021  DOI: https://doi.org/10.1098/rsta.2020.0097
  * [13] Xingjian Shi et al.  “Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting”  In _Advances in Neural Information Processing Systems_ 28 Curran Associates, Inc., 2015  URL: https://proceedings.neurips.cc/paper/2015/file/07563a3fe3bbe7e3ba84431ad9d055af-Paper.pdf