# Self-training Guided Adversarial Domain Adaptation For Thermal Imagery

Ibrahim Batuhan Akkaya1,2 Fazil Altinel111footnotemark: 1 Ugur Halici2,3
1Research Center indicates equal contribution Aselsan Inc  Turkey
2Department of Electrical and Electronics Engineering  Middle East Technical
University  Turkey
3NOROM Neuroscience and Neurotechnology Excellency Center  Turkey
{ibakkaya<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Deep models trained on large-scale RGB image datasets have shown tremendous
success. It is important to apply such deep models to real-world problems.
However, these models suffer from a performance bottleneck under illumination
changes. Thermal IR cameras are more robust against such changes, and thus can
be very useful for the real-world problems. In order to investigate efficacy
of combining feature-rich visible spectrum and thermal image modalities, we
propose an unsupervised domain adaptation method which does not require RGB-
to-thermal image pairs. We employ large-scale RGB dataset MS-COCO as source
domain and thermal dataset FLIR ADAS as target domain to demonstrate results
of our method. Although adversarial domain adaptation methods aim to align the
distributions of source and target domains, simply aligning the distributions
cannot guarantee perfect generalization to the target domain. To this end, we
propose a self-training guided adversarial domain adaptation method to promote
generalization capabilities of adversarial domain adaptation methods. To
perform self-training, pseudo labels are assigned to the samples on the target
thermal domain to learn more generalized representations for the target
domain. Extensive experimental analyses show that our proposed method achieves
better results than the state-of-the-art adversarial domain adaptation
methods. The code and models are publicly
available.111https://github.com/avaapm/SGADA

##  1 Introduction

Figure 1: Visualization of class activation maps on a target domain image
using occlusion sensitivity [35]. Given a person image (a), our proposed model
(d) activates semantically more meaningful parts of the image compared to our
base method (c) [32], while the model trained on only source domain images (b)
misclassifies the image as bicycle and activates wrong regions. Best viewed in
color.

Recently, significant improvements have been made by using RGB images for
image classification and detection problems [9, 14, 20, 27, 28]. The state-of-
the-art methods have been trained on large-scale RGB datasets such as MS-COCO
[22], ImageNet [5], Pascal-VOC [7], etc. However, low-lighting conditions
hinder current state-of-the-art deep learning methods trained on visible
spectrum images from performing well on computer vision tasks such as image
classification, object detection, etc. Since thermal IR cameras are more
robust against these conditions, exploiting them is useful for real-world
applications. Therefore, usage of thermal IR cameras has become more common in
the tasks related to autonomous driving, military operations, security
surveillance, etc. Since such large-scale thermal datasets are not publicly
available, it still remains an important challenge to achieve same level of
performance on thermal image datasets. Therefore, exploiting complementary
information offered by visible spectrum images is a straightforward technique
to improve performance of the methods which work on thermal images for
classification and detection problems. Unfortunately, recent studies
demonstrated that performance of a deep model well-trained on visible spectrum
images may significantly drop when applying to thermal images [6, 12, 13, 16,
17].

Since deep networks are sensitive to domain shift, a deep model trained on a
large amount of labeled source domain data may fail at generalizing to
unlabeled target domain data which are not similar to source domain data. To
overcome these issues, unsupervised domain adaptation (UDA) aims to learn a
model which maps both domains into a common feature space without requiring
image pairs. Among the recent UDA methods, adversarial domain adaptation
methods have become popular [8, 23, 25, 32]. These approaches incorporate
adversarial learning as a two-player game similar to generative adversarial
networks (GANs) [10]. Adversarial domain adaptation methods utilize a domain
discriminator to distinguish source domain from target domain and a feature
extractor to learn domain invariant representations to fool the domain
discriminator. By learning domain invariant feature representations,
adversarial domain adaptation methods assume that a classifier trained on
source domain features is able to successfully classify target domain samples
as well.

In this paper, we propose an unsupervised adversarial domain adaptation method
to align source and target domain distributions as described in Section 3. We
employ Adversarial Discriminative Domain Adaptation (ADDA) [32] method as our
base method. Although ADDA and other adversarial domain adaptation methods
have achieved successful results, these methods face a major generalization
limitation. The limitation is that even though the distributions are aligned
by learning domain invariant representations with a feature extractor,
theoretically, the classifier may not work well on the target domain as shown
in [1]. Therefore, learning discriminative representations for the unlabeled
target domain is a difficult problem.

Based on the assumption of self-training, a classifiers’ own high-confidence
predictions are correct [38]. Since we assume that the predictions are mostly
correct, exploiting the samples with high confidence values and retraining the
classifier further improves the performance of the classifier. To this end,
recent adversarial domain adaptation methods proposed to use pseudo-labels
obtained from a classifier and retrain the model using the pseudo-labeled
samples [29, 34, 36]. With these in mind, in this study, we propose a self-
training guided adversarial domain adaptation (SGADA) method to overcome the
generalization problems of adversarial domain adaptation methods (Figure 2).
To perform self-training, pseudo labels obtained after warm-up phase of our
method are assigned to the samples on the target domain to learn more
generalized representations for the target domain. Pseudo-labels are assigned
if confidences of the classifier trained on source domain and the domain
discriminator reaches to threshold values for a target domain sample.

Our proposed method makes use of features obtained from visual spectrum images
to improve classification performance on thermal domain. Moreover, our method
does not need paired samples of RGB and thermal datasets. In order to train
and test our proposed method, we use large-scale RGB dataset MS-COCO [22] and
thermal imagery dataset FLIR ADAS [11]. We evaluate the proposed method
quantitatively and qualitatively. We demonstrate our methods’ success compared
to the state-of-the-art unsupervised domain adaptation methods in Section 4.
The results show that our method improves the performance of our base model
and outperforms the state-of-the-art methods. Moreover, Figure 1 depicts that
given a thermal image, our method classifies the image correctly by activating
semantically more meaningful regions compared to our base method ADDA [32] and
the model trained only on source domain data.

Effective classification for imbalanced data is an important field of research
since class imbalance exists in many real-world applications [3, 18].
Therefore, it is important to address the class imbalance problem. In our
experimental studies (Section 4), we show that class imbalanced datasets cause
UDA methods to over-classify the majority category. We show that our proposed
method achieves better results compared to the state-of-the-art UDA methods
when class imbalance exists.

Our contributions are summarized as follows:

  * •

We demonstrate the efficacy of combining visible spectrum and thermal image
modalities by using unsupervised domain adaptation without requiring RGB-to-
thermal image pairs.

  * •

We propose a self-training guided adversarial domain adaptation method for
thermal imagery. In order to learn more generalized feature representations
for target thermal domain, we employ pseudo-labels generated by the classifier
trained on RGB images and the discriminator, and train our model with these
pseudo-labels.

  * •

In order to demonstrate results of our method, we employ the large-scale RGB
dataset MS-COCO as source domain and the thermal dataset FLIR ADAS as target
domain. Extensive experimental analyses show that our proposed method
outperforms the state-of-the-art unsupervised domain adaptation methods.

##  2 Related Work

Figure 2: An overview of our proposed self-training guided adversarial domain
adaptation (SGADA) method. Pseudo-labels generated after our methods’ warm-up
phase are assigned to target thermal images. Then, the target CNN
$\mathbf{F}_{t}$ is trained using the pseudo-labels. The classifier
$\mathbf{C}$ and the source CNN $\mathbf{F}_{s}$ are reused from our base
model, and thus they are fixed. Target feature representations are learned by
updating parameters of the target CNN $\mathbf{F}_{t}$ with respect to losses
generated by the discriminator $\mathbf{D}$ and the classifier $\mathbf{C}$.
Blue boxes indicate fixed network parameters while red boxes indicate
trainable network parameters. Best viewed in color.

By using RGB images, deep neural networks have gained popularity on computer
vision tasks such as object detection, classification etc. Although
significant improvements have been accomplished by using visible spectrum
images, it still remains a critical problem to train a deep model robust to
real-world problems, e.g. low-lighting conditions. To overcome these problems,
thermal imagery has been used for object detection and classification problems
[13, 16, 31].

Some of recent studies investigated the effects of combining RGB and thermal
images to the performance on object detection problem [6, 13, 16, 17]. Since
large-scale thermal datasets are not publicly available, in this study, we
exploit complementary information offered by visible spectrum images to
improve classification performance on thermal imagery without requiring RGB-
to-thermal image pairs. We propose an unsupervised domain adaptation method in
order to investigate efficacy of combining visible spectrum and thermal image
modalities.

Numerous recent methods attempted to address domain adaptation problem.
Recently, Generative Adversarial Networks (GANs) [10] inspired the field of
domain adaptation, and thus deep adversarial domain adaptation methods have
become popular [2, 8, 23, 25, 32].

Feature-level adversarial domain adaptation methods incorporate a domain
discriminator to distinguish source and target domains while feature extractor
learns features to fool the discriminator. Ganin et al. [8] proposed a
gradient reversal layer to learn a feature extractor which generates features
that maximize domain discriminator loss while minimizing label prediction
loss. More recently, Tzeng et al. [32] proposed a method to learn a
discriminative mapping of target images to source feature space by fooling a
domain discriminator which distinguishes the encoded target images from source
samples. Many recent works employ adversarial training paradigm in their
domain adaptation procedure [23, 25]. Although feature-level adversarial
domain adaptation methods have accomplished successful empirical results,
these methods suffer from a major limitation. As shown in [1], even if a
feature extractor is well learned to generate domain invariant features,
theoretically, the classifier may not work well on the target domain.
Therefore, learning discriminative representations for the unlabeled target
domain is considered difficult.

On the other hand, pixel-level adversarial domain adaptation methods translate
source domain data into target domain data or vice versa by using image-to-
image translation [24]. Bousmalis et al. [2] proposed an approach to learn a
transformation in pixel-level from one domain to the other. Inspired by
CycleGAN [37], Hoffman et al. [15] proposed CyCADA to increase semantic
consistency of the image translation to improve the pixel-level methods. Even
though pixel-level adversarial domain adaptation studies present remarkable
results, image-to-image translation sometimes performs poorly on the datasets
which have objects with many complex structures.

To overcome the limitations of adversarial domain adaptation methods, recent
studies propose to directly deal with relationship between decision boundary
and learned feature representations [21, 30]. Saito et al. [30] introduced to
use a minimax training method to push target feature distributions away. Lee
et al. [21] proposed to exploit adversarial dropout mechanism to learn more
discriminative features by enforcing cluster assumption [4]. However, our
experimental studies show that these methods and aforementioned adversarial
domain adaptation methods have a drawback: Class imbalanced datasets [3, 18]
lead to a performance drop for these methods.

We employ a self-training guided adversarial domain adaptation method to deal
with the generalization problems of adversarial domain adaptation methods for
thermal imagery. To the best of our knowledge, there is no self-training
guided domain adaptation study in the literature of thermal image
classification. Self-training is a technique to assign pseudo-labels to
unlabeled samples using predictions of a classifier and retrain the model
including the pseudo-labeled samples. Based on the assumption of self-
training, a classifiers’ own high-confidence predictions are correct [38].
Recent adversarial domain adaptation methods proposed to use pseudo-labels
[29, 34, 36]. In our experiments, we show that our self-training guided method
performs better than previous domain adaptation methods under class imbalance
problem by learning more generalized representations for target thermal
domain.

##  3 Proposed Method

Our proposed self-training guided adversarial domain adaptation method is
illustrated in Figure 2. Before performing self-training, we extract pseudo-
labels for the target domain samples. The pseudo-label extraction mechanism is
depicted in Figure 3.

First, a feature extractor $\mathbf{F}_{s}$ and a classifier $\mathbf{C}$ are
trained on source domain using labeled source domain RGB images (Figure
3-(a)). This step is named as pre-training. After this step, the classifier
network $\mathbf{C}$ can successfully classify the source domain images by
exploiting the features which are extracted by the source Convolutional Neural
Network (CNN) $\mathbf{F}_{s}$. After the training on the source domain, we
perform the second step: the warm-up phase for pseudo-label generation (Figure
3-(b)). In this step, we fix the parameters of the feature extractor
$\mathbf{F}_{s}$ trained on the source domain. A target specific feature
extractor $\mathbf{F}_{t}$ is learned in an unsupervised manner. By performing
this step, features extracted from the source domain and the target domain are
aligned with adversarial training. Therefore, we can use the classifier
$\mathbf{C}$ trained on the source domain to classify target domain samples.
We perform aforementioned two steps by following the training process of our
base method ADDA [32]. In the last step, we fix the parameters of the feature
extractor $\mathbf{F}_{t}$ trained on the target domain, the classifier
$\mathbf{C}$ trained on the source domain and the discriminator $\mathbf{D}$.
Then, we obtain predictions from the classifier and confidences from both the
classifier and the discriminator for the target domain samples.

Once the predictions and the confidences are obtained, we utilize the
predictions to give pseudo-labels for target domain samples using the
confidences obtained from the classifier $\mathbf{C}$ and the discriminator
$\mathbf{D}$ as shown in Figure 3-(c). We use prediction of the classifier for
a target sample if classifiers’ confidence value is higher than a threshold
and domain label prediction of discriminator is close to source domain. That
is, we can use the prediction of the classifier if the discriminator
incorrectly classifies target samples. By using this pseudo-label selection
mechanism, intuitively, we select samples with feature representations which
are close to data with known labels. Next, as illustrated in Figure 2, we
train our proposed method using extracted pseudo-labels.

Figure 3: Illustration of our pseudo-label generation mechanism.

A general definition of unsupervised domain adaptation, and self-training
guided adversarial domain adaptation procedures of our proposed method are
described in Section 3.1 and 3.2, respectively.

###  3.1 Unsupervised Domain Adaptation

In the general definition of unsupervised domain adaptation (UDA) problem, we
are given $n_{s}$ labeled samples from a source domain
$\mathcal{D}_{s}=\\{(x^{s}_{i},y^{s}_{i})\\}_{i=1}^{n_{s}}$ and $n_{t}$
unlabeled samples from a target domain
$\mathcal{D}_{t}=\\{(x^{t}_{j})\\}_{j=1}^{n_{t}}$. The goal of UDA problem is
to learn a feature extractor $\mathbf{F}_{t}$ for the target domain and a
classifier $\mathbf{C}_{t}$ which correctly classifies the features. It is not
possible to perform supervised training since there is no labeled samples in
the target domain. Therefore, UDA learns to adapt the source feature extractor
$\mathbf{F}_{s}$ and the source classifier $\mathbf{C}_{s}$ to be able to use
them on target domain.

###  3.2 Self-training Guided Adversarial Domain Adaptation (SGADA)

The task of adversarial domain adaptation methods is to adversarially align
source and target domain representations. For this purpose, adversarial domain
adaptation methods propose to reduce the gap between $\mathbf{F}_{s}(x^{s})$
and $\mathbf{F}_{t}(x^{t})$. Thus, the classifier $\mathbf{C}_{s}$ trained on
source domain can be applied to the representations on target domain, and
necessity to train a separate $\mathbf{C}_{t}$ can be eliminated. As a result,
we obtain $\mathbf{C}=\mathbf{C}_{s}=\mathbf{C}_{t}$ [32]. We employ the
feature extractor $\mathbf{F}_{s}$ and the classifier $\mathbf{C}$ which are
learned during the warm-up phase. In this subsection, we elaborate our
training scheme shown in Figure 2.

We use the following loss function for the domain discriminator $\mathbf{D}$
which distinguishes source domain from target domain:

$\displaystyle\mathcal{L}_{advD}(x^{s},x^{t},\mathbf{F}_{s},\mathbf{F}_{t})=-\frac{1}{n_{s}}\sum_{i=1}^{n_{s}}\log[\mathbf{D}(\mathbf{F}_{s}(x_{i}^{s}))]$
(1)
$\displaystyle-\frac{1}{n_{t}}\sum_{i=1}^{n_{t}}\log[1-\mathbf{D}(\mathbf{F}_{t}(x_{i}^{t}))].$

Given source images $x^{s}$ and target images $x^{t}$, we update the
parameters of the domain discriminator $\mathbf{D}$ with respect to outputs of
the feature extractors $\mathbf{F}_{s}$ and $\mathbf{F}_{t}$. While updating
the parameters, we fix and reuse the source feature extractor $\mathbf{F}_{s}$
which is trained in the pre-training step of our pseudo-label generation.

We employ two loss functions to train the target CNN $\mathbf{F}_{t}$:
adversarial loss $\mathcal{L}_{advF}$ and self-training loss
$\mathcal{L}_{clsP}$. The adversarial loss is formulated as follows:

$\mathcal{L}_{advF}(x^{s},y^{s},\mathbf{D})=\frac{1}{n_{t}}\sum_{i=1}^{n_{t}}\log[\mathbf{D}(\mathbf{F}_{t}(x_{i}^{t}))].$
(2)

Note that we reuse the parameters of the source feature extractor
$\mathbf{F}_{s}$ from the previous step to initialize $\mathbf{F}_{t}$.

We exploit pseudo-labeled target domain samples to perform self-training
guided adversarial learning. After the learning of the classifier $\mathbf{C}$
and the domain discriminator $\mathbf{D}$ is completed during the warm-up
phase, we obtain predictions from the classifier and confidences for these
predictions. Given an unlabeled target domain sample, if confidence of the
classifier $\mathbf{C}$ is higher than pre-defined threshold and the domain
discriminator $\mathbf{D}$ classifies the sample as source domain, we include
the sample during our self-training guided adversarial domain adaptation step.
Also, if the domain discriminator $\mathbf{D}$ classifies the sample as target
domain with a confidence lower than a pre-defined threshold, we assign the
pseudo-label $\hat{y}^{t}$ generated by the classifier $\mathbf{C}$ to the
sample as well. By using $\hat{n}_{t}$ pseudo-labeled samples on target
domain, we aim to train a target specific feature extractor (Figure 2). We use
the new self-training loss function for our proposed method:

$\mathcal{L}_{clsP}(x^{t},\hat{y}^{t})=\frac{1}{\hat{n}^{t}}\sum_{i=1}^{\hat{n}^{t}}\ell_{ce}(\mathbf{C}(\mathbf{F}_{t}(x_{i}^{t})),\hat{y}_{i}^{t}).$
(3)

The overall objective function to train our proposed method SGADA is defined
as:

$\displaystyle\min_{\mathbf{D}}\mathcal{L}_{advD}(x^{s},x^{t},\mathbf{F}_{s},\mathbf{F}_{t})$
(4)
$\displaystyle\min_{\mathbf{F}_{t}}\mathcal{L}_{advF}(x^{s},y^{s},\mathbf{D})$
$\displaystyle+\lambda\mathcal{L}_{clsP}(x^{t},\hat{y}^{t}),$

where $\lambda$ is a trade-off parameter. We set the trade-off parameter
$\lambda$ and thresholds based on validation split (see Section 4.2 for
further details).

##  4 Experiments

We perform extensive evaluations and compare our proposed method with several
state-of-the-art unsupervised domain adaptation methods.

###  4.1 Datasets

We prepare a new RGB-to-thermal domain adaptation setting for classification
using FLIR ADAS [11] as thermal dataset and MS-COCO [22] as visible spectrum
dataset for our experimental studies.

Figure 4: The network architectures used for our experimental analyses.

FLIR ADAS [11] consists of 9,214 thermal images with bounding box annotations.
Each image has a resolution of 640 $\times$ 512 and obtained from FLIR Tau2
camera. 60% of the images are captured during daytime and the remaining 40% of
the images are captured during night. The dataset provides both visible
spectrum (RGB) images and thermal images. We consider only the thermal images
of the dataset for our experiments. We use the training and test splits as
suggested in the dataset documentation for our experiments. The objects in the
dataset are classified into four categories i.e. bicycle, car, dog, and
person. However, the dog class has very few annotations. Therefore, the dog
class is not considered in our experimental studies. We crop square images
using bounding box annotations for objects. After objects are extracted, we
resize the images to 224 $\times$ 224\. Finally, our thermal dataset consists
of 4,137 samples of bicycle, 43,734 samples of car, and 26,294 samples of
person images. Example images from FLIR ADAS dataset are shown in the second
row of Figure 5.

Figure 5: Example images from MS-COCO dataset [22] and FLIR ADAS thermal
dataset [11]. Best viewed in color.

Figure 6: The t-SNE visualization of network activations on target thermal
domain generated by source only model (a), our base method ADDA [32] (b), our
proposed method SGADA with classifier confidences only (c), and our proposed
method SGADA with classifier and discriminator confidences (d). Best viewed in
color.

Our proposed method incorporates publicly available large-scale visible
spectrum datasets to improve classification performance on thermal dataset.
Therefore, we consider using an RGB dataset which includes the same classes as
FLIR dataset [11] (bicycle, car, and person). For this purpose, we use MS-COCO
dataset [22] as our visible spectrum dataset. In the first row of Figure 5, we
show some example images from MS-COCO dataset. MS-COCO dataset contains 91
object categories (airplane, bicycle, bird, car, person, etc.). In total,
there are 123,287 images and around 886,000 bounding boxes. 118,287 of the
images are for training while 5,000 of the images are for validation split. We
apply standard training and test splits as provided in the dataset
documentation for our experiments. We use only bicycle, car, and person
classes to match with our thermal dataset. We cropped the annotated objects
with the same procedure as applied to FLIR dataset. Once objects are
extracted, we resize the images to 224 $\times$ 224\. Our visible spectrum
image dataset extracted from MS-COCO consists of 5,732 samples of bicycle,
38,453 samples of car, and 209,162 samples of person images.

###  4.2 Implementation Details

For our experiments, we used same training procedure with ADDA [32]. For a
fair comparison with other methods, we employed ResNet-50 [14] pre-trained on
ImageNet [5] as backbone for all methods. Our network architectures are given
in Figure 4. The architecture of our feature extractors (source CNN
$\mathbf{F}_{s}$ and target CNN $\mathbf{F}_{t}$) is the ResNet-50 without the
last fully connected (FC) layer. In the figure, each convolutional residual
unit is depicted with the size of filters at the top and the outputs of each
convolutional layer at the bottom. The notation k $\times$ k, n in the
convolutional layer blocks represents a filter of size k and n channel. The
number on the top of the convolutional layer blocks denotes the number of
repetition for the unit. The domain discriminator $\mathbf{D}$ consists of
three FC layers: two consecutive hidden units of 500 neurons and the
discriminator output. The classifier $\mathbf{C}$ has only one FC layer.

To train our method, we set batch size to 32. Number of epochs was set to 15
in our experiments. Parameters were updated using ADAM optimization algorithm
[19]. For pre-training of our method on source domain only, we set learning
rate to 5e-4. For adversarial adaptation step of our method, we set learning
rate as 1e-5 and discriminator learning rate as 1e-3. We used same learning
rates as used in the previous step for self-training guided adversarial
adaptation step of our method. We set $\lambda$ value as 0.7 and threshold
value as 0.87 for our method with classifier confidences only (SGADA-Cls). For
our method with classifier and discriminator confidences (SGADA-Cls+Disc), we
used same learning rates as used in the previous step, and $\lambda$ value of
0.25, classifier threshold value of 0.79, and discriminator threshold value of
0.87. We used the same experimental settings for training and testing. We
exploit classification accuracy to compare our proposed method with other
methods.

We implemented our proposed method using PyTorch framework [26].
Implementation details, models, and the code were made publicly available at
https://github.com/avaapm/SGADA.

###  4.3 Evaluation of SGADA

In our experiments, we select visual spectrum (RGB) domain as the source
domain, and thermal domain as the target domain. As a general practice in the
field of domain adaptation, we denote source only as the target domain
performance of a model trained using only source domain images, and target
only as that of a model trained on the target domain. Performances of source
only and target only models serve as baselines for the lower and upper bound
performances.

Table 1: Per-class classification performance comparison.

Method |  Bicycle |  Car |  Person |  Average
---|---|---|---|---
Source only | 69.89 | 83.89 | 86.52 | 80.10
Pixel-DA [2] | 62.53 | 89.99 | 76.73 | 76.42
DTA [21] | 75.45 | 97.65 | 92.45 | 88.52
MCD-DA [30] | 81.71 | 94.90 | 91.83 | 89.48
DANN [8] | 78.16 | 95.07 | 96.24 | 89.82
CDAN [25] | 78.16 | 97.10 | 94.82 | 90.03
ADDA [32] | 86.67 | 96.95 | 89.10 | 90.90
SGADA (ours) | 87.13 | 94.44 | 92.03 | 91.20
Target only | 87.59 | 98.78 | 96.35 | 94.24

#### Quantitative Analysis.

We compare our proposed method SGADA with several state-of-the-art
unsupervised domain adaptation methods: Unsupervised Pixel-Level Domain
Adaptation with Generative Adversarial Networks (Pixel-DA) [2], Drop to Adapt
(DTA) [21], Maximum Classifier Discrepancy for Unsupervised Domain Adaptation
(MCD-DA) [30], Domain Adversarial Neural Network (DANN) [8], Conditional
Domain Adversarial Adaptation (CDAN) [25], and Adversarial Discriminative
Domain Adaptation (ADDA) [32]. Since these methods do not consider domain
adaptation problem for thermal datasets, there exist no reported results on
their paper for our dataset. Therefore, we trained and evaluated all these
methods for our dataset.

Table 2: Ablation studies of different pseudo-label selection scenarios for
self-training guided adversarial domain adaptation.

| Bicycle | Car | Person
---|---|---|---
| Number of samples | 3702 | 38657 | 21081
Classifier confidences only | Number of selected samples | 3995 | 35494 | 20323
Number of correctly selected samples | 2901 | 34905 | 18911
Accuracy of selected samples (%) | 72.62 | 98.34 | 93.05
Discriminator confidences only | Number of selected samples | 3598 | 36024 | 3800
Number of correctly selected samples | 2874 | 35251 | 3549
Accuracy of selected samples (%) | 79.88 | 97.85 | 93.39
Classifier and discriminator confidences together | Number of selected samples | 3557 | 35123 | 3558
Number of correctly selected samples | 2873 | 34558 | 3454
Accuracy of selected samples (%) | 80.77 | 98.39 | 97.08

Table 3: Ablation experiments.

Method |  Bicycle |  Car |  Person |  Average
---|---|---|---|---
SGADA-Cls | 87.36 | 95.27 | 90.62 | 91.08
SGADA-Cls+Disc | 87.13 | 94.44 | 92.03 | 91.20

Per-class domain adaptation performances are reported in Table 1. The results
show that our proposed method which uses classifier and discriminator
confidences together for self-training outperforms the state-of-the-art
methods. Although DTA [21] and DANN [8] perform well for car and person
classes respectively, their performance for bicycle class cannot reach the top
performances since the number of samples for bicycle class is much less than
the other classes. On the other hand, our proposed method achieves more
balanced performance scores for all classes and outperforms other methods. It
is important to address this problem since datasets for real-world problems
usually include imbalanced classes [3, 18]. As shown in the table, our
proposed method achieves more balanced class-wise accuracies compared to our
base method ADDA [32], and furthermore our method increases the average
accuracy over our base method.

#### Qualitative Analysis.

We visualize the feature representations on target thermal domain with t-SNE
[33] for qualitative analysis in Figure 6. The features of source only model
on target domain can not be discriminated very well while ADDA [32]
discriminate some overlapping points in the feature space. Our proposed model
which uses only classifier confidences for self-training (SGADA-Cls) learns
more discriminative representations. As shown in the figure, our proposed
model which uses classifier and discriminator confidences together for self-
training (SGADA-Cls+Disc) further enlarges inter-class distances, especially
for car and person classes.

#### Ablation Study.

To evaluate the contributions of our proposed method, we perform ablation
studies. We examine effects of using classifier and/or discriminator
confidences for pseudo-label selection. As described in Section 3, we select
samples using our base method. Table 2 shows three cases where we select
target domain samples using only the confidences of the classifier, using only
the confidences of the discriminator, and using the confidences of both the
classifier and the discriminator. As shown in the table, if we utilize the
discriminator confidences, the number of selected samples for the person class
decreases. Moreover, when we use both discriminator and classifier confidences
to select target domain samples, accuracy of the pseudo-labels increases
significantly for all classes. This results in better separation of feature
representations as depicted in Figure 6 (c)-(d). Furthermore, since accuracy
of selected samples for all classes using classifier and discriminator
confidences is higher than the other cases, class imbalance of our proposed
method (SGADA-Cls+Disc) reduces compared to SGADA-Cls as shown in Table 3. And
thus, the overall accuracy of our proposed method surpasses SGADA-Cls,
resulting in the best overall performance.

##  5 Conclusion

In this paper, we propose a self-training guided adversarial domain adaptation
method in order to investigate the efficacy of combining visible spectrum and
thermal image modalities by using unsupervised domain adaptation. To overcome
the generalization problems of the current adversarial domain adaptation
methods, we employ pseudo-labels obtained from a classifier trained on RGB
images, and train our method with these pseudo-labels. In order to demonstrate
results of our method, we use large-scale RGB dataset MS-COCO as the source
domain and thermal dataset FLIR ADAS as the target domain. Quantitative and
qualitative results show that our proposed method achieves better results than
the state-of-the-art adversarial domain adaptation methods by learning more
generalized feature representations for target thermal domain.

## References

  * [1] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan.  A theory of learning from different domains.  Machine learning, 2010.
  * [2] Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan.  Unsupervised pixel-level domain adaptation with generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [3] Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski.  A systematic study of the class imbalance problem in convolutional neural networks.  Neural Networks, 2018.
  * [4] Olivier Chapelle and Alexander Zien.  Semi-supervised classification by low density separation.  In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), 2005.
  * [5] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  ImageNet: A Large-Scale Hierarchical Image Database.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
  * [6] Chaitanya Devaguptapu, Ninad Akolekar, Manuj M Sharma, and Vineeth N Balasubramanian.  Borrow from anywhere: Pseudo multi-modal object detection in thermal imagery.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2019.
  * [7] Mark Everingham, Luc Gool, Christopher K. Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  International Journal of Computer Vision (IJCV), 2010.
  * [8] Yaroslav Ganin and Victor Lempitsky.  Unsupervised domain adaptation by backpropagation.  In Proceedings of the International Conference on Machine Learning (ICML), 2015.
  * [9] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
  * [10] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in Neural Information Processing Systems (NeurIPS), 2014\.
  * [11] F. A. Group.  Flir thermal dataset for algorithm training.  https://www.flir.com/oem/adas/adas-dataset-form/.
  * [12] Dayan Guan, Xing Luo, Yanpeng Cao, Jiangxin Yang, Yanlong Cao, George Vosselman, and Michael Ying Yang.  Unsupervised domain adaptation for multispectral pedestrian detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2019.
  * [13] Tiantong Guo, Cong Phuoc Huynh, and Mashhour Solh.  Domain-adaptive pedestrian detection in thermal images.  In Proceedings of the IEEE International Conference on Image Processing (ICIP), 2019.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [15] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell.  Cycada: Cycle-consistent adversarial domain adaptation.  In Proceedings of the International Conference on Machine Learning (ICML), 2018.
  * [16] Soonmin Hwang, Jaesik Park, Namil Kim, Yukyung Choi, and In So Kweon.  Multispectral pedestrian detection: Benchmark dataset and baseline.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  * [17] Shu Wang Jingjing Liu, Shaoting Zhang and Dimitris Metaxas.  Multispectral deep neural networks for pedestrian detection.  In Proceedings of the British Machine Vision Conference (BMVC), 2016\.
  * [18] Justin M Johnson and Taghi M Khoshgoftaar.  Survey on deep learning with class imbalance.  Journal of Big Data, 2019.
  * [19] Diederik P Kingma and Jimmy Lei Ba.  Adam: A method for stochastic gradient descent.  In Proceedings of the International Conference on Learning Representations (ICLR), 2015.
  * [20] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Advances in Neural Information Processing Systems (NeurIPS), 2012\.
  * [21] Seungmin Lee, Dongwan Kim, Namil Kim, and Seong-Gyun Jeong.  Drop to adapt: Learning discriminative features for unsupervised domain adaptation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
  * [22] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and Larry Zitnick.  Microsoft coco: Common objects in context.  In Proceedings of the IEEE/CVF European Conference on Computer Vision (ECCV), 2014.
  * [23] Hong Liu, Mingsheng Long, Jianmin Wang, and Michael Jordan.  Transferable adversarial training: A general approach to adapting deep classifiers.  In Proceedings of the International Conference on Machine Learning (ICML), 2019.
  * [24] Ming-Yu Liu, Thomas Breuel, and Jan Kautz.  Unsupervised image-to-image translation networks.  In Advances in Neural Information Processing Systems (NeurIPS), 2017\.
  * [25] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan.  Conditional adversarial domain adaptation.  In Advances in Neural Information Processing Systems (NeurIPS), 2018\.
  * [26] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In Advances in Neural Information Processing Systems (NeurIPS), 2019\.
  * [27] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [28] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In Advances in Neural Information Processing Systems (NeurIPS), 2015\.
  * [29] Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada.  Asymmetric tri-training for unsupervised domain adaptation.  In Proceedings of the International Conference on Machine Learning (ICML), 2017.
  * [30] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada.  Maximum classifier discrepancy for unsupervised domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [31] Philip Saponaro, Scott Sorensen, Abhishek Kolagunda, and Chandra Kambhamettu.  Material classification with thermal imagery.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  * [32] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell.  Adversarial discriminative domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [33] Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of Machine Learning Research, 2008.
  * [34] Shaoan Xie, Zibin Zheng, Liang Chen, and Chuan Chen.  Learning semantic representations for unsupervised domain adaptation.  In Proceedings of the International Conference on Machine Learning (ICML), 2018.
  * [35] Matthew D. Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In Proceedings of the IEEE/CVF European Conference on Computer Vision (ECCV), 2014.
  * [36] Weichen Zhang, Wanli Ouyang, Wen Li, and Dong Xu.  Collaborative and adversarial network for unsupervised domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [37] J. Zhu, T. Park, P. Isola, and A. A. Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2017.
  * [38] Xiaojin Jerry Zhu.  Semi-supervised learning literature survey.  University of Wisconsin-Madison Department of Computer Sciences, 2005\.


# A Deep-learning Model for Fast Prediction of Vacancy Formation in Diverse
Materials

Kamal Choudhary<EMAIL_ADDRESS>[  Bobby G. Sumpter  [

###### Abstract

The presence of point defects such as vacancies plays an important role in
material design. Here, we demonstrate that a graph neural network (GNN) model
trained only on perfect materials can also be used to predict vacancy
formation energies ($E_{vac}$) of defect structures without the need for
additional training data. Such GNN-based predictions are considerably faster
than density functional theory (DFT) calculations with reasonable accuracy and
show the potential that GNNs are able to capture a functional form for energy
predictions. To test this strategy, we developed a DFT dataset of 508
$E_{vac}$ consisting of 3D elemental solids, alloys, oxides, nitrides, and 2D
monolayer materials. We analyzed and discussed the applicability of such
direct and fast predictions. We applied the model to predict 192494 $E_{vac}$
for 55723 materials in the JARVIS-DFT database.

###### keywords:

American Chemical Society, LaTeX

National Institute of Standards and Technology] Materials Science and
Engineering Division, National Institute of Standards and Technology,
Gaithersburg, 20899, MD, USA. [Theiss Research] Theiss Research, La Jolla,
92037, CA, USA. Oak Ridge National Laboratory] Center for Nanophase Materials
Sciences, Oak Ridge National Laboratory, Oak Ridge, TN 37831, USA. IR,NMR,UV

Defects play an important role in our pursuit to engineer performance of a
material. Vacancies are a type of defects which are ubiquitous and their
presence can significantly alter catalytic, electronic, optoelectronic,
electrochemical, diffusion, and neuromorphic properties1, 2, 3, 4, 5, 6, 7, 8,
9, 10. Experimentally vacancy formation energies can be determined using
positron annihilation experiments 11. Theoretically, they can be computed
using classical force-field (FF) 12, 13 and density functional theory (DFT)
14, 15, 16, 17, 2 calculations. However, such computation can be very
computationally expensive and non-generalizable for DFT and FF based
calculations respectively.

Recently, machine learning techniques have been proposed as a faster method
for predicting defect energetics, but so far they still require time-consuming
defect data generation for model training and limit the applicability and
generalizability of the defect energetic predictions 18, 19, 20, 21, 22, 23.
Especially, graph neural network based deep-learning models 24, 25, 26, 27
have become very popular for predicting materials properties and have been
used for several bulk property predictions and their applicability needs to be
tested for defect property predictions. Two key ingredients needed for
accomplishing this task are: 1) a pretrained deep-learning model that can
directly predict the total energy of perfect and defect structures, 2) a test
DFT dataset of vacancy formation energies on which the DL model could be
applied.

In this work, we demonstrate that the atomistic line graph neural network
(ALIGNN) 28 based total energy prediction model (trained on the JARVIS-DFT 29
OptB88vdW energy per atom data for perfect bulk materials 29) can be directly
used to predict vacancy formation energy of an arbitrary material with
reasonable accuracy without requiring additional training data. The
performance in terms of mean absolute error for the energy per atom model was
reported as 0.037 eV in ref. 28. Note that we do not train any machine
learning/deep learning model in this work for defects and just used the model
parameters for energy prediction that was developed and shared publicly in
ref.28.

Developing a vacancy formation energy dataset can be extremely time-consuming
and depends on several computational setup parameters such as supercell-size,
choice of k-points, considering neutral vs. charged defects and selection of
appropriate chemical potentials. For testing the strategy adopted in this
work, we generated a DFT dataset of 508 entries with charge neutral defects
using a high-throughput approach. The dataset consists of elemental solids,
oxide, alloy and 2D materials. In addition to predicting the vacancy formation
energies, we analyze the trends, strengths and limitations of such
predictions. Lastly, we used this strategy to develop a database of vacancy
formation energies for all the materials in the JARVIS-DFT database. The deep-
learning model, the DFT dataset and the workflow are made publicly available
through the JARVIS (Joint Automated Repository for Various Integrated
Simulations) infrastructure 29.

First, we discuss the generation of vacancy formation energy dataset that is
used for testing the deep-learning model. We obtained stable elemental solids,
binary alloys, oxides and 2D materials from the JARVIS-DFT dataset. We used at
least 8 $\AA$ lattice parameter constraints in x,y,z directions to build the
supercell. We removed an atom with a unique Wyckoff position to generate the
vacancy structure using the JARVIS-Tools package
(https://github.com/usnistgov/jarvis). The defect structures were then
subjected to energy minimization using Optb88vdW functional 30 and projected
augmented wave formalism 31 in Vienna Ab initio Simulation Package (VASP)
package 32, 33. Please note commercial software is identified to specify
procedures. Such identification does not imply recommendation by National
Institute of Standards and Technology (NIST). We used the converged k-point
and cut-off from the JARVIS-DFT dataset based on total energy convergence 34.
We used an energy convergence of $10^{-6}$ eV for energy convergence during
the self-consistent cycle. Currently, we have 508 entries for the vacancies
and the dataset is still growing.

For the deep-learning predictions, we used the recently developed atomistic
line graph neural network (ALIGNN) 28, which is publicly available at
https://github.com/usnistgov/alignn. ALIGNN has been used to train fast and
accurate models for more than 65 properties of solids and molecules with high
accuracy28, 25, 35, 36. In ALIGNN, a crystal structure is represented as a
graph using atomic elements as nodes and atomic bonds as edges. Each node in
the atomistic graph is assigned 9 input node features based on its atomic
species: electronegativity, group number, covalent radius, valence electrons,
first ionization energy, electron affinity, block and atomic volume. The
inter-atomic bond distances are used as edge features with radial basis
function up to 8 Å cut-off and a 12-nearest-neighbor ($N$). This atomistic
graph is then used for constructing the corresponding line graph using
interatomic bond-distances as nodes and bond-angles as edge features. ALIGNN
uses edge-gated graph convolution for updating nodes as well as edge features
using a propagation function ($f$) for layer ($l$), atom features ($h$), and
node ($i$), details of which can be found in Ref. 28:

$h_{i}^{(l+1)}=f(h_{i}^{l}{\\{h_{j}^{l}}\\}_{{}_{i}})$ (1)

Unlike many other conventional GNNs, ALIGNN uses bond-distances as well as
bond-angles to distinguish atomic structures. The ALIGNN model is implemented
in PyTorch 37 and deep graph library (DGL) 38. A model to predict energy per
atom was developed in Ref28 which will be used as an energy predictor for both
perfect and defect structures in this work.

Figure 1:  Analysis of vacancy formation energy dataset generated in this
work. a) comparison of a subset of vacancy formation energy with respect to
available experimental and previous DFT calculations from literature. Our
dataset agrees very well with previously reported values. b) Data distribution
of all the vacancy formation energy values. Most of the values are below 3 eV.

In Fig. 1 we analyzed the DFT database for vacancy formation energies
developed in this work. We used this dataset for testing purposes only.
Although there have been several studies in generating vacancy formation
energy dataset, a fully atomistic dataset consistent with bulk and vacancy
energetics information is not available to our knowledge. Hence, we generated
a DFT dataset for vacancies consisting of a wide variety of material classes
such as elemental solids, 2D materials, oxides, and metallic alloys. We
visualize the defect formation energies of materials in Fig. 1. As mentioned
above, we only considered the charge neutral vacancies within a finite 8 $\AA$
cell size with OptB88vdW functional. The vacancy formation energy was
calculated as

$E_{vacancy}=E_{defect}-E_{perfect}+\mu$ (2)

where, $E_{vacancy}$ is the vacancy formation energy, $E_{defect}$ is the
energy of the defect structure with an atom missing, $E_{perfect}$ is the
energy of the perfect structure, $\mu$ is the chemical potential used as
energy per atom of the most stable structure of an element. The chemical
potentials used in this work are provided in the supplementary information.

Figure 2:  Comparison of DFT and ALIGNN based vacancy formation energy
predictions with (Scissor-shift) and without (Original) correction. In Fig. a,
we show the performance on the entire DFT dataset generated in this work.
Fig.b,c,d show the comparisons for elemental solids, oxides and 2D monolayers
respectively.

Currently, the vacancy formation energy dataset consists of 508 entries. We
compare a subset of this dataset with available data from previous
experimental and DFT-studies 14, 39, 40, 41, 42, 43 in Fig. 1a. We find an
excellent agreement between our dataset and that from literature with a mean
absolute error (MAE) of 0.3 eV. In Fig. 1b, we show the histogram of all the
vacancy formation energy data. We find that most of the vacancy formation
energy data lie below 3 eV. Depending on the type of engineering applications,
either a high or low $E_{vac}$ could be desirable.

Figure 3:  Periodic table trend of 192494 vacancy formation energies predicted from scissor-shifted ALIGNN based predictions. We visualize the probability that compounds with vacancy of a given element have vacancy formation energy more than 2 eV. Table 1: Analysis and comparison of ALIGNN based vacancy formation energy with respect to DFT dataset. The mean absolute errors (MAE) are calculated for all the values as well as elemental solid, oxide and 2D-monolayer subsets. The corresponding number of entries (Count) are also provided. As the ALIGNN based vacancy formation energies could be underestimated we apply scissor-shift and the improvement (% $\Delta$) of predictions with (Scissor-shifted) and without scissor shift (Original) are shown.  Sets | Count | Original MAE | Scissor-shift MAE | % $\Delta$
---|---|---|---|---
All | 508 | 1.51 | 1.0 | 33.8
Elements | 117 | 1.53 | 1.2 | 21.6
Oxides | 57 | 2.3 | 1.4 | 30.4
2D-mono | 68 | 1.9 | 1.2 | 36.8
Figure 4: Figure shows ALIGNN vacancy formation energy with scissor-shift (SS)
dataset (192494 entires) against OptB88vdW based formation energy available in
the JARVIS-DFT database and color coded with corresponding OptB88vdW bandgaps.

Next, we used ALIGNN based pretrained total energy per atom model trained on
the JARVIS-DFT dataset for predicting defect energy and perfect energy
required for vacancy formation energy following Eq. 2. This model was trained
using bulk energies for 55723 solids 28. The defect structures were generated
by deleting an atom with a unique Wyckoff position without optimizing the
atomic positions of other atoms in the defect structures. We used the same
chemical potential for elements from the JARVIS-DFT as given in the supporting
information. The comparison of ALIGNN based prediction with respect to DFT
data is shown with blue dots in Fig. 2a. Interestingly, we observed that there
is a noticeable correlation between the ALIGNN direct predictions and DFT data
with a mean absolute error of 1.51 eV. However, we found that the ALIGNN based
predictions were usually underestimated. To circumvent this issue, we applied
a scissor shift by adding 1.3 eV to all the ALIGNN based predictions
represented by green dots. Using such a shift, we were able to lower the MAE
to 1.0 eV i.e., leading to an overall 33.8 % improvement. The value of 1.3 eV
was chosen such that overall MAE is minimized. We noted that the previous
reports on machine learning for vacancy formation energies resulted in mean
absolute error values of 0.40 eV and 0.67 eV in ref. 18 and ref. 21
respectively. These models were trained on specific material classes such as
GeTe system and 2D materials, while the approach used here acts as a
generalized application of the model with MAE closer to specific case studies.
Additionally, an MAE of 1.0 eV is arguably low considering the range of the
vacancy formation energies can be upto 10 eV.

To further analyze the predictions for different types of materials, we
compared the DFT and ALIGNN predictions for elemental solids, oxides and 2D
monolayers in Fig. 2b, Fig. 2c and Fig. 2d respectively. Corresponding
original ALIGNN prediction based MAE and that with scissor-shifted values are
shown in Table 1. We found that the ALIGNN based models perform well for 2D
monolayer and oxide materials compared to elemental solids and alloys. This
behavior can be explained based on the fact that GNN architectures usually
perform message passing locally and may work well for insulting materials with
fewer bonds rather than for elemental solids and alloy systems which are
usually metallic and have delocalized electronic structures leading to a
higher number of interatomic bonds.

To analyze the effect of training dataset size of energy per atom ALIGNN model
on prediction of vacancy formation energies, we trained models for 20 %, 40 %,
60 %, 80 % and 100 % of total energy per atom training data for perfect
materials resulting in mean absolute errors of 1.20 eV, 1.22 eV, 1.11 eV, 1.05
eV and 1.0 eV for vacancy formation dataset respectively. Therefore, one
possible way to further improve the model would be to include more perfect
structures that can have a varied number of bonds for various systems. As the
JARVIS-DFT dataset for perfect structures is still growing, we believe the
defect property prediction models should further improve in the future.

Now, we applied the above strategy to predict the vacancy formation energies
of all the 55723 materials in the JARVIS-DFT leading to 192494 vacancy
formation energies. In Fig. 3, we visualize the probability that compounds
with the vacancy of a given element have vacancy formation energy of more than
2 eV as a threshold value. Interestingly, we found that C,N,O,F are some of
the common elements with high vacancy formation energies. Furthermore, we plot
the ALIGNN based vacancy formation energy dataset against OptB88vdW based
formation energy available in the JARVIS-DFT database and color code with the
corresponding OptB88vdW bandgaps in Fig. 4. We also provided the histogram
distribution of the formation energy per atom and vacancy formation energy per
atom. Interestingly, we found that high vacancy formation energies were
favored by lower formation energies per atom and high electronic bandgaps. The
vacancy formation energy histogram showed a high peak around 2 eV which was
similar to that observed in Fig. 1b. Such analysis helped us understand the
entire dataset in a nutshell and can further help in understanding defect
energetics for materials design for instance finding materials with high
dopability, water splitting etc.

In summary, we have developed a diverse dataset of vacancy formation energies
using density functional theory and demonstrate that the total energy per atom
model using ALIGNN can be directly used for predicting vacancy formation
energies without the need for additional training data. The defect database
can be highly useful for materials science community as there is a strong
demand for a benchmark dataset. We have discussed the assumptions used in this
work such as excluding charge defects and using finite cell sizes. Also, our
work has shown how a state of the art model performs on unseen data without
additional data. We have applied a heuristic scissor-shift of energy that
further improves the accuracy. Using the current strategy, we have predicted
vacancy formation energies of around 55723 compounds with 192494 entries
leading to the largest dataset of defect properties, which could have been
very expensive from DFT calculations. We provide data from this work as well
as machine learning models to help accelerate the design of new materials. Our
work has proven that GNN models can not only be useful for perfect materials
but also defect systems.

K.C. thanks the National Institute of Standards and Technology for funding,
computational, and data-management resources. Contributions from K.C. were
supported by the financial assistance award 70NANB19H117 from the U.S.
Department of Commerce, National Institute of Standards and Technology. DFT
calculations were also conducted at the Center for Nanophase Materials
Sciences, a US Department of Energy Office of Science User Facility.

Experimental and previous DFT data for benchmarking, and chemical potential of
elemental solids are provided in the supporting information.

## References

  * Hinuma et al. 2018 Hinuma, Y.; Toyao, T.; Kamachi, T.; Maeno, Z.; Takakusagi, S.; Furukawa, S.; Takigawa, I.; Shimizu, K.-i. Density functional theory calculations of oxygen vacancy formation and subsequent molecular adsorption on oxide surfaces. _The Journal of Physical Chemistry C_ 2018, _122_ , 29435–29444
  * Emery et al. 2016 Emery, A. A.; Saal, J. E.; Kirklin, S.; Hegde, V. I.; Wolverton, C. High-throughput computational screening of perovskites for thermochemical water splitting applications. _Chemistry of Materials_ 2016, _28_ , 5621–5634
  * Watkins et al. 2011 Watkins, M.; Pan, D.; Wang, E. G.; Michaelides, A.; VandeVondele, J.; Slater, B. Large variation of vacancy formation energies in the surface of crystalline ice. _Nature materials_ 2011, _10_ , 794–798
  * Emery and Wolverton 2017 Emery, A. A.; Wolverton, C. High-throughput DFT calculations of formation energy, stability and oxygen vacancy formation energy of ABO3 perovskites. _Scientific data_ 2017, _4_ , 1–10
  * Yu et al. 2014 Yu, X.; Zhan, Z.; Rong, J.; Liu, Z.; Li, L.; Liu, J. Vacancy formation energy and size effects. _Chemical Physics Letters_ 2014, _600_ , 43–45
  * Shang et al. 2016 Shang, S.-L.; Zhou, B.-C.; Wang, W. Y.; Ross, A. J.; Liu, X. L.; Hu, Y.-J.; Fang, H.-Z.; Wang, Y.; Liu, Z.-K. A comprehensive first-principles study of pure elements: Vacancy formation and migration energies and self-diffusion coefficients. _Acta Materialia_ 2016, _109_ , 128–141
  * Yan et al. 2019 Yan, X.; Zhao, Q.; Chen, A. P.; Zhao, J.; Zhou, Z.; Wang, J.; Wang, H.; Zhang, L.; Li, X.; Xiao, Z., et al. Vacancy-induced synaptic behavior in 2D WS2 nanosheet–based memristor for low-power neuromorphic computing. _Small_ 2019, _15_ , 1901423
  * Parmar et al. 2018 Parmar, N. S.; Boatner, L. A.; Lynn, K. G.; Choi, J.-W. Zn vacancy formation energy and diffusion coefficient of CVT ZnO crystals in the sub-surface micron region. _Scientific reports_ 2018, _8_ , 1–8
  * Ita and Cohen 1997 Ita, J.; Cohen, R. E. Effects of pressure on diffusion and vacancy formation in MgO from nonempirical free-energy integrations. _Physical Review Letters_ 1997, _79_ , 3198
  * Huh et al. 2020 Huh, W.; Lee, D.; Lee, C.-H. Memristors based on 2D materials as an artificial synapse for neuromorphic electronics. _Advanced Materials_ 2020, _32_ , 2002092
  * McKee et al. 1972 McKee, B.; Triftshäuser, W.; Stewart, A. Vacancy-formation energies in metals from positron annihilation. _Physical Review Letters_ 1972, _28_ , 358
  * Choudhary et al. 2018 Choudhary, K.; Biacchi, A. J.; Ghosh, S.; Hale, L.; Walker, A. R. H.; Tavazza, F. High-throughput assessment of vacancy formation and surface energies of materials using classical force-fields. _Journal of Physics: Condensed Matter_ 2018, _30_ , 395901
  * Huygh et al. 2014 Huygh, S.; Bogaerts, A.; Van Duin, A. C.; Neyts, E. C. Development of a ReaxFF reactive force field for intrinsic point defects in titanium dioxide. _Computational materials science_ 2014, _95_ , 579–591
  * Medasani et al. 2015 Medasani, B.; Haranczyk, M.; Canning, A.; Asta, M. Vacancy formation energies in metals: A comparison of MetaGGA with LDA and GGA exchange–correlation functionals. _Computational Materials Science_ 2015, _101_ , 96–107
  * Ding et al. 2015 Ding, H.; Medasani, B.; Chen, W.; Persson, K. A.; Haranczyk, M.; Asta, M. PyDII: a Python framework for computing equilibrium intrinsic point defect concentrations and extrinsic solute site preferences in intermetallic compounds. _Computer Physics Communications_ 2015, _193_ , 118–123
  * Goyal et al. 2020 Goyal, A.; Gorai, P.; Anand, S.; Toberer, E. S.; Snyder, G. J.; Stevanovic, V. On the dopability of semiconductors and governing material properties. _Chemistry of Materials_ 2020, _32_ , 4467–4480
  * Goyal et al. 2017 Goyal, A.; Gorai, P.; Peng, H.; Lany, S.; Stevanović, V. A computational framework for automation of point defect calculations. _Computational Materials Science_ 2017, _130_ , 1–9
  * Cheng et al. 2020 Cheng, Y.; Zhu, L.; Wang, G.; Zhou, J.; Elliott, S. R.; Sun, Z. Vacancy formation energy and its connection with bonding environment in solid: A high-throughput calculation and machine learning study. _Computational Materials Science_ 2020, _183_ , 109803
  * Arrigoni and Madsen 2021 Arrigoni, M.; Madsen, G. K. Evolutionary computing and machine learning for discovering of low-energy defect configurations. _npj Computational Materials_ 2021, _7_ , 1–13
  * Manzoor et al. 2021 Manzoor, A.; Arora, G.; Jerome, B.; Linton, N.; Norman, B.; Aidhy, D. S. Machine Learning Based Methodology to Predict Point Defect Energies in Multi-Principal Element Alloys. _Frontiers in Materials_ 2021, _8_ , 129
  * Frey et al. 2020 Frey, N. C.; Akinwande, D.; Jariwala, D.; Shenoy, V. B. Machine learning-enabled design of point defects in 2d materials for quantum and neuromorphic information processing. _ACS nano_ 2020, _14_ , 13406–13417
  * Mannodi-Kanakkithodi et al. 2022 Mannodi-Kanakkithodi, A.; Xiang, X.; Jacoby, L.; Biegaj, R.; Dunham, S. T.; Gamelin, D. R.; Chan, M. K. Universal machine learning framework for defect predictions in zinc blende semiconductors. _Patterns_ 2022, _3_ , 100450
  * Sharma et al. 2020 Sharma, V.; Kumar, P.; Dev, P.; Pilania, G. Machine learning substitutional defect formation energies in ABO3 perovskites. _Journal of Applied Physics_ 2020, _128_ , 034902
  * Choudhary et al. 2022 Choudhary, K.; DeCost, B.; Chen, C.; Jain, A.; Tavazza, F.; Cohn, R.; Park, C. W.; Choudhary, A.; Agrawal, A.; Billinge, S. J., et al. Recent advances and applications of deep learning methods in materials science. _npj Computational Materials_ 2022, _8_ , 1–26
  * Choudhary et al. 2022 Choudhary, K.; Yildirim, T.; Siderius, D. W.; Kusne, A. G.; McDannald, A.; Ortiz-Montalvo, D. L. Graph neural network predictions of metal organic framework CO2 adsorption properties. _Computational Materials Science_ 2022, _210_ , 111388
  * Fung et al. 2021 Fung, V.; Zhang, J.; Juarez, E.; Sumpter, B. G. Benchmarking graph neural networks for materials chemistry. _npj Computational Materials_ 2021, _7_ , 1–8
  * Fung et al. 2021 Fung, V.; Hu, G.; Ganesh, P.; Sumpter, B. G. Machine learned features from density of states for accurate adsorption energy prediction. _Nature communications_ 2021, _12_ , 1–11
  * Choudhary and DeCost 2021 Choudhary, K.; DeCost, B. Atomistic Line Graph Neural Network for improved materials property predictions. _npj Computational Materials_ 2021, _7_ , 1–8
  * Choudhary et al. 2020 Choudhary, K.; Garrity, K. F.; Reid, A. C.; DeCost, B.; Biacchi, A. J.; Hight Walker, A. R.; Trautt, Z.; Hattrick-Simpers, J.; Kusne, A. G.; Centrone, A., et al. The joint automated repository for various integrated simulations (JARVIS) for data-driven materials design. _npj Computational Materials_ 2020, _6_ , 1–13
  * Klimeš et al. 2009 Klimeš, J.; Bowler, D. R.; Michaelides, A. Chemical accuracy for the van der Waals density functional. _Journal of Physics: Condensed Matter_ 2009, _22_ , 022201
  * Blöchl 1994 Blöchl, P. E. Projector augmented-wave method. _Physical review B_ 1994, _50_ , 17953
  * Kresse and Furthmüller 1996 Kresse, G.; Furthmüller, J. Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set. _Physical review B_ 1996, _54_ , 11169
  * Kresse and Furthmüller 1996 Kresse, G.; Furthmüller, J. Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set. _Computational materials science_ 1996, _6_ , 15–50
  * Choudhary and Tavazza 2019 Choudhary, K.; Tavazza, F. Convergence and machine learning predictions of Monkhorst-Pack k-points and plane-wave cut-off in high-throughput DFT calculations. _Computational materials science_ 2019, _161_ , 300–308
  * Choudhary and Garrity 2022 Choudhary, K.; Garrity, K. Designing High-Tc Superconductors with BCS-inspired Screening, Density Functional Theory and Deep-learning. _arXiv preprint arXiv:2205.00060_ 2022,
  * Kaundinya et al. 2022 Kaundinya, P. R.; Choudhary, K.; Kalidindi, S. R. Prediction of the electron density of states for crystalline compounds with Atomistic Line Graph Neural Networks (ALIGNN). _arXiv preprint arXiv:2201.08348_ 2022,
  * Paszke et al. 2019 Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. _Advances in neural information processing systems_ 2019, _32_
  * Wang et al. 2019 Wang, M.; Zheng, D.; Ye, Z.; Gan, Q.; Li, M.; Song, X.; Zhou, J.; Ma, C.; Yu, L.; Gai, Y., et al. Deep graph library: A graph-centric, highly-performant package for graph neural networks. _arXiv preprint arXiv:1909.01315_ 2019,
  * Pham and Wang 2015 Pham, H. H.; Wang, L.-W. Oxygen vacancy and hole conduction in amorphous TiO 2. _Physical Chemistry Chemical Physics_ 2015, _17_ , 541–550
  * Kumar et al. 2015 Kumar, A.; Chernatynskiy, A.; Liang, T.; Choudhary, K.; Noordhoek, M. J.; Cheng, Y.-T.; Phillpot, S. R.; Sinnott, S. B. Charge optimized many-body (COMB) potential for dynamical simulation of Ni–Al phases. _Journal of Physics: Condensed Matter_ 2015, _27_ , 336302
  * Choudhary et al. 2015 Choudhary, K.; Liang, T.; Chernatynskiy, A.; Phillpot, S. R.; Sinnott, S. B. Charge optimized many-body (COMB) potential for Al2O3 materials, interfaces, and nanostructures. _Journal of Physics: Condensed Matter_ 2015, _27_ , 305004
  * Guo et al. 2015 Guo, Y.; Liu, D.; Robertson, J. Chalcogen vacancies in monolayer transition metal dichalcogenides and Fermi level pinning at contacts. _Applied Physics Letters_ 2015, _106_ , 173106
  * Wu et al. 2020 Wu, K.; Liu, T.; Sun, R.; Song, J.; Shi, C. First-principles calculations of oxygen vacancy in CaO crystal. _The European Physical Journal D_ 2020, _74_ , 1–6