11institutetext: University of Turin, Turin, Italy
11email<EMAIL_ADDRESS>22institutetext: INRAE, UMR
TETIS, Univ. Montpellier, Montpellier, France 33institutetext: INRIA, Univ.
Montpellier, Montpellier, France
33email<EMAIL_ADDRESS>
# Towards a multimodal framework for remote sensing image change retrieval and
captioning

Roger Ferrod(✉) 11 Luigi Di Caro 11 Dino Ienco 2233

###### Abstract

Recently, there has been increasing interest in multimodal applications that
integrate text with other modalities, such as images, audio and video, to
facilitate natural language interactions with multimodal AI systems. While
applications involving standard modalities have been extensively explored,
there is still a lack of investigation into specific data modalities such as
remote sensing (RS) data. Despite the numerous potential applications of RS
data, including environmental protection, disaster monitoring and land
planning, available solutions are predominantly focused on specific tasks like
classification, captioning and retrieval. These solutions often overlook the
unique characteristics of RS data, such as its capability to systematically
provide information on the same geographical areas over time. This ability
enables continuous monitoring of changes in the underlying landscape.

To address this gap, we propose a novel foundation model for bi-temporal RS
image pairs, in the context of change detection analysis, leveraging
Contrastive Learning and the LEVIR-CC dataset for both captioning and text-
image retrieval. By jointly training a contrastive encoder and captioning
decoder, our model add text-image retrieval capabilities, in the context of
bi-temporal change detection, while maintaining captioning performances that
are comparable to the state of the art. We release the source code and
pretrained weights at: https://github.com/rogerferrod/RSICRC.

###### Keywords:

Remote Sensing bi-temporal change detection image captioning text-image
retrieval contrastive learning

##  1 Introduction

Modern Earth observation systems allow acquiring systematic information, under
the shape of satellite imagery, to monitor and characterize the evolution of
the underlying Earth surface. Among all the applications, the possibility to
detect and characterized particular changes in the land surfaces is of
paramount importance in a variety of applications such as environmental
protection, disaster monitoring and land planning [16].

More precisely, the value of such information comes from the detailed
comparison between bi-temporal remote sensing imagery and all the related
features. The majority of current change detection approaches (e.g., [2, 13])
exhibit a limited level of interaction from a user perspective. Given a pair
of images, they mainly highlight the spatial areas affected by some change
phenomena, without any additional information or any strategies to guide the
process via possible user query. To express its full potential and allow an
appropriate interaction with users, features related to changed areas must be
effectively described and searchable, making it possible to interact with the
system in natural language. For these reasons, there is a strong interest in
the community to develop models that go beyond simple Change Detection (CD)
strategies and try instead to accurately describe bi-temporal changes
occurring or retrieve a pair of images associated to a given textual prompt.

Those aims can be achieved through multimodal foundation models (e.g., [20,
24, 29, 14]) that, once pretrained on a large-scale dataset, can be used in
many down-stream tasks with remarkable performance. Foundation vision-language
models, led by CLIP [17], have already demonstrated excellent capabilities in
remote sensing applications, too ([14, 30, 7]). However, despite great success
in other domains, Vision-Language models for Remote Sensing applications still
suffer from the scarcity of large-scale datasets. In fact, while there are
plenty of works focused on specific tasks – such as classification [1],
captioning [2] or retrieval [31] – foundational models are still at their
infancy stage in the RS domain.

Motivated by the lack of Vision-Language models for Remote Sensing
applications that manage bi-temporal change detection information, here, we
propose a foundation model specifically designed for pairs of bi-temporal
remote sensing images. To the best of our knowledge, it is the first attempt
to fill this gap in the literature. Given the shortage of resources focused on
changes in RS imagery, we propose to adopt a remote sensing image change
captioning dataset (LEVIR-CC) to assess the potential of our framework
considering both bi-temporal captioning and bi-temporal text-image retrieval
tasks. By jointly training a contrastive encoder and captioning decoder, we
provide a single model that, simultaneously, allows bi-temporal text-image
retrieval capabilities, preserving captioning abilities comparable to the
state of the art.

##  2 Related works

Our work follows a line of research started recently by [13] who introduced
the task and the dataset we used. The dataset, described more in details in
Section 3.1, is accompanied by a model named RSICCformer. Another model
(Chg2Cap) was then proposed by [2] the following year, obtaining state-of-the-
art results on the same benchmark. Both models are limited to the captioning
task. Therefore, their efforts are focused on building a solid encoder
attached to a simple decoder and train them with a captioning loss. Similar
solutions, not limited to the RS domain, include [4, 6, 5, 27, 22, 21].

With foundation models, instead, a single pretrained model can be used for
many downstream tasks, with only minor fine-tuning and supervision efforts.
Such techniques are now getting more and more attention also in the (RS)
domain, with recent advances such as RemoteCLIP [14], EarthGPT [30] and RSGPT
[7]. However, such initiatives are still focused on the analysis of static
information and do not take into account the temporal dimension, i.e., the
evolution or changes that might occur between two or more remote sensing image
acquisitions.

With the aim to combine two different paradigms (retrieval and captioning)
into a single model, though, some architectural modifications are required.
For this purposed, we based our framework on CoCa [28]. In CoCa the authors
propose a strategy to train a multimodal foundation model that can perform
various tasks, such as captioning, text-image and image-text retrieval and
visual recognition. Such work, however, is limited to static, natural images,
and it cannot be easily extended to cope with bi-temporal satelletie imagery.

With regard to the retrieval task, Vision-Language contrastive models like
CLIP [17] and derivatives, such as ALBEF [11] and ALIGN [9] can be adopted.
These multimodal AI models have already contributed to ameliorate state-of-
the-art results in many downstream task. Recently, extensions of these models
for the RS domain have been proposed, like [18] and RemoteCLIP [14], achieving
state-of-the-art performance in RS image retrieval task but, unfortunately,
such models are still focused on the analysis of static remote sensing
imagery, and they cannot be employed for the analysis of bi-temporal remote
sensing imagery for downstream change detection applications.

##  3 Method

Building on top of state-of-the-art approaches, we propose a new model to
address the challenges posed by retrieving a pair of images related to
temporal changes in remote sensing images via natural language prompts. More
precisely, our goal is to provide a model to cope with both captioning and
text-image retrieval tasks at once, for the case of remote sensing data. To
achieve this objective, we integrate a SOTA image-encoder specifically
designed for image change captioning with a new decoder that enables a joint
training of the two objectives: an autoregressive captioning loss and a
contrastive loss.

Inspired by CoCa [28] we split the decoder in two parts: an unimodal module
that encodes the textual input only and a multimodal module with cross
attention that combines textual and visual embeddings. In both cases, the
decoder prohibits tokens from attending to future tokens in the sequence. More
details on the decoder are provided in section 3.3.

Differently from CoCa, thought, our model needs to deal with a pair of images,
not a single one. Having experimented with RSICCformer [13] and Chg2Cap [2] we
have decided to adopt Chg2Cap’s encoder, with only minor revisions, given its
excellent behavior exhibited on the captioning task. The encoder is
responsible for encoding the images individually, through a pretrained model,
and then combining the features into a single representation. The details
related to the encoder are supplied in section 3.2. The overall architecture
is depicted in Figure 1.

To test our model, we relied on the LEVIR-CC dataset introduced in [13], which
was the first research work to address the Remote Sensing Image Change
Captioning (RSICC) task introducing the RSICCformer architecture. Since no
datasets are available, today, for remote sensing image retrieval with pairs
of images (before change/after change), we exploited the LEVIR-CC dataset both
for captioning and retrieval, although this choice required some special
precautions discussed in section 3.1.

Figure 1: The overall architecture of our model; once the pair of images is
encoded through two siamese pretrained models, the information is processed by
a bi-temporal encoder that merges the two representations, then a single
embedding can be retrieved through attentive pooling and contrastively
compared with the corresponding textual embedding or used directly as input
for the cross-modal decoder; the decoder is splited in two parts: unimodal
layers that only encode the textual representation and multimodal layers that
generate the captions.

###  3.1 Dataset

The LEVIR-CC dataset consists of 10,077 image pairs and 5 captions each
describing the changes happening between the two images cover the same area
(Figure 2). Half of the pairs do not have any significant change, while the
other half portrays changes such as new building or road being built. Often
the description includes detailed spatial references (e.g., “on the top-left”
or “at the bottom of the scene”) that identify the changes. Although the
dataset is vast and has over 50k sentences, the possible changes that could
occur are limited by the fact that the acquired images are taken almost in the
same geographical zone (Texas, USA), within a similar urban context.
Therefore, while it is naturally possible to exploit the data for the
captioning task, adapting the dataset for image retrieval is more challenging.
Given a textual query, indeed, more images that match the description could be
retrieved. Within a contrastive learning framework, those examples are
considered false negatives and could make the training stage challenging.

Figure 2: Examples of items taken from the LEVIR-CC dataset, where each image
pair (before/after) is accompanied by 5 human annotated captions (only one is
shown here).

###  3.2 Encoder

The encoder follows the same architecture proposed by [2], with a pretrained
siamese network to encode the image pair, a hierarchical attention mechanism
to combine the features and a residual block with cosine mask. Whereas the
original Chg2Cap model was proposed with a resnet-101 backbone – and tested on
other models pretrained on ImageNet – we included in our experiments networks
especially pretrained on remote sensing images. If our model is set to fine-
tune the encoder’s backbone, only the weights from the last two convolutional
layers of the ResNet architecture, or the transformer layers of ViT, are
updated.

After a linear projection head used to bring the feature dimension to the
desired size, we continue as in the original Chg2Cap model, in particular once
we have an embedding representation $F_{i}$ for each image $X_{i}$:

$\displaystyle F_{1}=net(X_{1})$ (1) $\displaystyle F_{2}=net(X_{2})$

with $F_{i}\in\mathbb{R}^{h\times w\times D}$ and $D$ the feature dimension,
we add learnable positional embeddings:

$F_{i}=F_{i}+F_{pos}$ (2)

with $F_{pos}\in\mathbb{R}^{h\times w\times D}$, and pass the extracted
features to a hierarchical self-attention block that will apply the attention
mechanism across the two images:

$I_{i},I_{2}=HSA(F_{1},F_{2})$ (3)

where $I_{i}\in\mathbb{R}^{h\times w\times D}$. When fusing the two
representations, through a concatenation over the hidden dimension, a cosine
mask is applied as follows:

$F_{fus}=[I_{1};I_{2}]+Cos(I_{i},I_{2})$ (4)

where $F_{fus}\in\mathbb{R}^{h\times w\times 2D}$, $[;]$ is the concatenation
operation and $Cos(\cdot,\cdot)$ the cosine similarity between the two
tensors. After applying a 2D convolution layer with 1x1 kernel size and
dimensionality reduction, $F_{fus}$ is processed by a residual block (1x1
conv, 3x3 conv, 1x1 conv) obtaining a final encoding ($E_{cap}$) for the image
pair:

$\displaystyle C=conv_{1\times 1}(F_{fus})$ (5) $\displaystyle
E_{cap}=ReLU(ResBlock(C)+C)$

where $C\in\mathbb{R}^{h\times w\times D}$ and $E_{cap}\in\mathbb{R}^{h\times
w\times D}$.

$E_{cap}$ can then be used directly as input for the decoder cross-attention
layer to generate a caption. For the contrastive loss used for the text-image
retrieval task, instead, a single embedding vector representing the pair of
images is derived. To this aim, we added a single query multi-head attention
layer as an attention pooling operation:

$E_{con}=MHA(E_{cap})$ (6)

with $E_{con}\in\mathbb{R}^{D}$

###  3.3 Decoder

The transformer-based decoder requires some modifications to both captioning
and contrastive learning. Following CoCa [28], we split the decoder in two
parts: the first one will encode the text and can be used directly by the
contrastive learning loss, while the second part will combine the textual
encoding with visual embeddings and produce the caption. Both apply casually-
masked attention to prevent the current token to attend to future tokens, but
only the second part will apply cross-attention, combining the textual and
visual representations. More formally: input tokens are mapped into a word
embedding using an embedding layer and added to positional embeddings:

$E=emb+pos$ (7)

where $emb:\mathbb{R}^{n}\rightarrow\mathbb{R}^{n\times D}$, $n$ the number of
input tokens, $D$ the embedding dimension and $pos$ the sinusoidal positional
encoding. Then the unimodal decoder layers follow, which will produce two
results: the self-attention outputs with shape $n\times D$ and a single
representation of the sequence taken from the last token:

$\displaystyle S_{seq}=MHA(E)$ (8) $\displaystyle S_{end}=S_{seq}[n]$

where $S_{seq}\in\mathbb{R}^{n\times D}$ and $S_{end}\in\mathbb{R}^{D}$. While
$S_{end}$ is used directly by the contrastive loss, $S_{seq}$ will pass
through other decoder layers, this time with cross-attention:

$E_{cross}=MHA(S_{seq},E_{cap})$ (9)

where $E_{cross}in\mathbb{R}^{n\times D}$, $S_{seq}$ the textual sequence
embeddings with shape $n\times D$ and $E_{cap}$ the image embeddings with
shape $h\times w\times D$. Finally, a linear layer ($LN$) is used for tokens
prediction:

$C=LN(E_{cross})$ (10)

with $LN\in\mathbb{R}^{n\times|V|}$ where $V$ is the vocabulary.

Differently from the encoder, which in part exploits pretrained models, the
decoder in trained from scratch. The particularity lies in the vocabulary
used, that is derived from the dataset itself. Both [13] and [2], indeed,
compose the vocabulary with the words appearing at least 5 times in the
dataset, resulting in 463 tokens. Any attempt to go beyond these limits
adopting a pretrained decoder with thousands of tokens will drop the
performance, making the results not comparable. Therefore, we have decided to
follow the same strategy adopted by previous works. We also add the
possibility to tie embedding weights with the decoder, resulting in shared
parameters.

###  3.4 Training

As already mentioned, this work aims to combine captioning capabilities with
text-image retrieval using only a single model. We achieve this with a
modified architecture which allows the two paradigms to coexist.

####  3.4.1 Possible issues related to False Negatives

However, moving from a single image to a pair of images (before and after a
change occurred) is not straightforward and requires a mechanism to tackle
false negatives, i.e., examples identified as negative by the contrastive
loss, which in fact should be considered as positive. This problem arises from
using a dataset originally intended for captioning only. The phenomenon is
common in many other tasks in which contrastive learning is employed,
especially in a self-supervised setting, and solutions exist to cope with this
particular issue [8, 26, 3]. Following the common literature, we implemented
False Negative Elimination (FNE) and False Negative Attraction (FNA)
strategies. With FNE, the detected false negative is simply removed from the
loss computation, while the FNA strategy considers the false negative as a
positive example and changes its label accordingly. In our work, the false
negative detection phase is done by comparing the captions similarities. If
two captions belonging to different image pairs (therefore considered negative
examples) have a cosine similarity higher than a given threshold ($\theta$),
the FNE or FNA mechanisms are activated. As shown in Figure 3, two different
captions with similar semantics can then be considered as the same caption
(FNA) or excluded from the same batch (FNE).

Figure 3: By performing contrastive learning, an anchor (image pair) is
compared with the captions inside the batch, the corresponding textual
description is considered a positive example, the others negative. If a False
Negative is detected (caption similarities higher than $\theta$), one can
exclude it from the loss computation (False Negative Elimination) or consider
it as positive (False Negative Attraction).

Since the textual decoder is trained from scratch, for the similarity
computation we rely on a third-party model specifically designed for sentence
comparison. We pre-compute the embeddings with Sentence-BERT [19] and then the
cosine similarity before applying the contrastive loss.

####  3.4.2 Contrastive Learning

To combine the visual representation and textual embeddings, we rely on the
InfoNCE loss, a popular contrastive learning loss. Visual and textual encoders
are jointly optimized by contrasting pairs of images and captions inside the
batch:

$\mathcal{L}_{con}=-\frac{1}{N}(\sum_{i}^{N}log\frac{exp(e_{i}\cdot
s_{i}/\tau)}{\sum_{j}^{N}exp(e_{i}\cdot
s_{j}/\tau)}+\sum_{i}^{N}log\frac{exp(s_{i}\cdot
e_{i}/\tau)}{\sum_{j}^{N}exp(s_{i}\cdot e_{j}/\tau)})$ (11)

where $N$ is the batch size, $\tau$ the temperature and $e_{i}$ and $s_{j}$
the (normalized) visual and textual embeddings, respectively.

More in detail, the batch is composed by $N$ triples $<e_{i},s_{i},l_{i}>$,
where $l_{i}$ is the label of the $(e,s)_{i}$ pair. We then compute all the
possible combinations of contrastive pairs and assign a positive or negative
label accordingly to $l_{i}$, i.e., when $s_{i}$ is the corresponding caption
for the $e_{i}$ image pair. At this point, we have to correct the labels by
detecting and removing false negatives. We take the pre-computed normalized
Sentence-BERT embeddings ($t_{i}$) and compute a $N\times N$ similarity matrix
in which we identify the combination of captions – within the batch – that
have a relative similarity higher than a given threshold $\theta$. Formally,
positive and negative examples are defined as:

$(e_{i},s_{j})=\begin{cases}(e_{i},s_{j})_{+}&\textit{if }l_{i}=l_{j}\\\
(e_{i},s_{j})_{-}&\textit{if }l_{i}\neq l_{j}\land t_{i}\cdot
t_{j}<\theta\end{cases}$ (12)

when using False Negative Elimination (FNE), or:

$(e_{i},s_{j})=\begin{cases}(e_{i},s_{j})_{+}&\textit{if }l_{i}=l_{j}\lor
t_{i}\cdot t_{j}\geq\theta\\\ (e_{i},s_{j})_{-}&\textit{if }l_{i}\neq
l_{j}\land t_{i}\cdot t_{j}<\theta\end{cases}$ (13)

when using False Negative Attraction (FNA).

####  3.4.3 Captioning

While the contrastive loss compares the whole representation of the image pair
with an analogous representation of the caption, the captioning task acts at a
more fine-grained scale. Indeed, the cross-attention of the decoder considers
the entire sequence of tokens and the spatial features from the images. Given
this information, the decoder will predict the sequence of tokens
autoregressively:

$\mathcal{L}_{cap}=-\sum_{i=1}^{n}logP(y_{i}|y_{<i},e)$ (14)

where $n$ in the sequence length and $P$ the probability of generating the
$y_{i}$ token given the previous words and the image encoding ($e$).

####  3.4.4 Training Objective

Finally, the model is jointly trained by combining both $\mathcal{L}_{con}$
and $\mathcal{L}_{cap}$ and weighting the relative contribution in the main
loss:

$\mathcal{L}=\mathcal{L}_{cap}+\lambda\cdot\mathcal{L}_{con}$ (15)

##  4 Experiments and Results

We trained our model on the LEVIR-CC dataset and compared its results with
other remote sensing models. Of those, only [13] and [2] deal with a pair of
images as input, yet they are focus only on the captioning task. Other remote
sensing models, on the other hand, are trained for text-image retrieval and/or
captioning, but only on single images.

###  4.1 Setup

Training and evaluation are performed on a single NVIDIA A40 GPU with 48 GB of
memory. We evaluate the model after each epoch, and we select the model that
minimizes the loss. The maximum number of epochs is set to 50 and the batch
size to 32. The AdamW optimizer is used, with $\epsilon=10^{-8}$ and linear
warm up reaching a target learning rate of $10^{-4}$. The InfoNCE temperature
$\tau$ is set to 0.01.

###  4.2 Experiments

We compared our results with the state of the art and other pretrained remote
sensing models. For those models that were only trained for single-image
ranking ([9, 14, 11]) we encoded the image pair as the difference between RGB
values, specifically: $E=|X_{after}-X_{before}|$.

We also add the possibility of including hard negatives examples in the
batches. A hard negative pair is an example in which the image and textual
encodings have a high similarity despite being negative examples, i.e., the
text is not the intended caption for that specific image. Therefore, at each
epoch we compute and update an index, sampling the negatives examples globally
(i.e., across the entire dataset). For this purpose, we rely on the FAISS
library [10] and cosine similarity.

We evaluated the captions with standard text quality metrics like BLEU score
[15] and ROUGE-L [12] as well as CIDEr [23] for image description evaluation.
The same metrics were used in SOTA models like [2] and [13]. For the text-
image retrieval task, instead, we relied on Precision (P), Recall (R) and Mean
Reciprocal Rank (MRR) over top-k results. Specifically, the metrics are
defined as follows:

$P@k=\frac{\text{{relevant items in top-k results}}}{k}$ (16)

$R@k=\frac{\text{{relevant items in top-k results}}}{\text{{relevant items}}}$
(17)

$MRR@k=\frac{1}{Q}\sum_{i=1}^{Q}\frac{1}{rank_{i}}$ (18)

where $Q$ is the number of queries, $rank_{i}$ the position of the first
relevant item in the top-k results for the i-th query. For each metric, the
relevant items are computed by taking into account the $\theta$ threshold used
for the false negative detection phase during the training. Therefore, if two
examples are merged during the training because their similarity is higher
than $\theta$, the same will happen during the evaluation phase.

###  4.3 Results

When evaluating the results, it is important to consider the uneven
distribution of the dataset. Since half of the dataset consists of examples
without changes (approximately 5,000 items), a top-5 search of an unchanged
scene would yield a low Recall score ( 0.1%), even if all five matches are
relevant.

At the same time, since for each caption we can have only one corresponding
image in the original dataset – or slightly more if we consider the
aggregation results induced by the use of the $\theta$ threshold – for most of
the queries the Precision will be near $1/k$, or $20\%$ if $k=5$.

In Table 1 we compare the baselines with our framework, without contrastive
learning, with False Negative Elimination and with False Negative Attraction.
As highlighted by the obtained results, the contrastive learning strategy
allows our model to outperform the baseline approaches. At the same time, we
maintained comparable performances on the captioning task, despite having a
smaller backbone than the one originally used in Chg2Cap (ResNet-50 instead of
ResNet-101). It is also interesting to note that the contrastive learning loss
had a positive influence also on the captioning task, with small advantages
compared to the baseline without contrastive learning. Among the two
strategies, False Negative Attraction came out as the best and confirms the
finding already reported in the literature [8].

P@5 | R@5 | MRR@5 | BLEU-1 | BLEU-4 | ROUGE-L | CIDEr | model
---|---|---|---|---|---|---|---
- | - | - | 82.84 | 60.92 | 72.72 | 130.97 | Chg2Cap
21.33 | 0.61 | 27.59 | - | - | - | - | ALIGN
41.95 | 0.71 | 52.32 | - | - | - | - | Remote-CLIP RN50
40.74 | 0.28 | 51.85 | - | - | - | - | ALBEF
30.1 | 0.52 | 16.87 | 79.55 | 55.76 | 62.15 | 117.14 | w/o contrastive learning
41.6 | 1.2 | 52.15 | 69.38 | 37.92 | 55.21 | 106.61 | FNE
52.32 | 2.85 | 53.51 | 82.34 | 59.04 | 62.99 | 120.14 | FNA
Table 1: Text-image retrieval and captioning results of SOTA captioning system
(Chg2Cap), Retrieval baselines (ALIGN, Remote-CLIP and ALBERF) and our models
(without contrastive learning, with False Negative Elimination or False
Negative Attraction); the best results are highlighted in bold, the second
best are underlined.

Before choosing the proper threshold value for the false negative detection
task, we manually annotated 150 captions randomly sampled from the test set,
by merging textual description with the same meaning. Based on these
annotations, we can then evaluate the quality of different threshold values,
as reported in Table 2 (a). In particular, with $\theta$ above $0.96$ (in
terms of cosine similarity) the false negative detection is perfectly aligned
with human judgment, before dropping sharply with lower similarity value. From
the Table 2 (b), instead, we can appreciate the difference between False
Negative Elimination and False Negative Attraction compared with a simpler
training method that does not apply a false negative detection strategy. We
confirm that the False Negative Attraction strategy exhibits the best
behavior, when the similarity threshold is set to $1.0$. With this setting, we
can achieve better ranking scores and comparable captioning performances
compared to the baseline approaches.

ACC | F1 | P | R | $\theta$
---|---|---|---|---
99.90 | 54.55 | 37.50 | 100 | 0.92
99.92 | 60.00 | 42.86 | 100 | 0.94
99.98 | 85.71 | 75.00 | 100 | 0.95
100 | 100 | 100 | 100 | 0.96
100 | 100 | 100 | 100 | 1.0
(a)

R@5 | BLEU-1 | BLEU-4 | model
---|---|---|---
2.23 | 82.64 | 59.76 | w/o FN detection
1.93 | 77.96 | 54.27 | FNE $\theta=0.96$
2.58 | 80.95 | 57.73 | FNA $\theta=0.96$
1.2 | 69.38 | 37.92 | FNE $\theta=1.0$
2.85 | 82.34 | 59.04 | FNA $\theta=1.0$
(b)

Table 2: a) evaluation of the threshold over 150 manually annotated samples
and b) impact on the model.

Although the main difference of our framework compared to previous approaches
is the decoder component, we also propose to replace the original backbone
used in Chg2Cap (a ResNet-101 pretrained on ImageNet) with a Remote Sensing
pretrained model. Given the good results obtained by Remote-CLIP (Table 1), we
have chosen to adopt its encoder as backbone for our framework. Despite the
fact that it has fewer parameters than ResNet-101, the pretraining phase on
remote sensing images proved to be beneficial, and the model was able to
outperform the original encoder pre-trained on ImageNet. In Table 3 we compare
the fine-tuned ResNet-50 backbone we adopted, with a non fine-tuned version
(i.e., with frozen weights) and the same architecture but trained on ImageNet.
Albeit a frozen version of the Remote-CLIP backbone already permits to achieve
more than reasonable results on the ranking task, we have observed that fine-
tuning even ameliorate the results, as briefly mentioned in Section 3.2. To
this end, we only fine-tune the last two layers.

R@5 | BLEU-1 | BLEU-4 | model
---|---|---|---
1.83 | 73.95 | 45.38 | ImageNet RN50 finetune
2.9 | 78.47 | 52.47 | RemoteCLIP RN50 no finetune
2.85 | 82.34 | 59.04 | RemoteCLIP RN50 finetune
Table 3: Comparison between different backbones: RN50 from RemoteCLIP
finetuned/non finetuned and the same architecture pretrained on ImageNet.

Concerning the importance of the (global) hard negatives sampling mechanism,
no clear evidence is available. Although this solution represents the best
practice in Contrastive Learning [25], it proved to be counterproductive in
our case, as shown in Table 4. Since we rely on self-supervised labels for the
ranking task rather than high-quality labeled data, the hard negative
mechanism might nullify the effect of false negative detection, thereby
hindering the final results.

R@5 | BLEU-1 | BLEU-4 | model
---|---|---|---
2.85 | 82.34 | 59.04 | w/o hard negatives
0.98 | 72.69 | 43.57 | w/ hard negatives
Table 4: Impact of the hard negative mechanism on performance.

##  5 Conclusion and Future works

In this work, we present a new multimodal foundation model designed for bi-
temporal remote sensing images, with captioning and text-image retrieval
capabilities. Building on top of state-of-the-art solutions, we adapt
pretrained visual encoders and a novel multitask decoder with the aim to
combining two paradigms into a single model. Given the absence of benchmarks
for RS retrieval of bi-temporal images, we propose to exploit a change
detection captioning dataset (LEVIR-CC) for both tasks, with some precautions
that aim to mitigate the problem of false negatives under contrastive learning
training. Although there is still room for improvement in the absolute metric
results for the retrieval task, our model maintains captioning performance
comparable to the state of the art, with the added benefit of also providing a
solution for the text-image retrieval task.

We believe that natural language prompting is crucial to facilitate the
exploration of Remote Sensing image archives by non-expert end-users. Our work
is a step towards this direction, but there is still work to be done in order
to achieve fully reliable solutions. For instance, efforts must be done to
create and curate a benchmark for bi-temporal image retrieval, which can be
shared with the community to stimulate research activities in this important
area. Another possible future research can be devoted to the design and
development of generic foundational visual language remote sensing models
capable of handling multiple tasks simultaneously, rather than relying on
single-task models that require systematic adaptation for each downstream
task.

## References

  * [1] Adegun, A.A., Viriri, S., Tapamo, J.R.: Review of deep learning methods for remote sensing satellite images classification: experimental survey and comparative analysis. Journal of Big Data 10, 1–24 (2023)
  * [2] Chang, S., Ghamisi, P.: Changes to captions: An attentive network for remote sensing change captioning. IEEE Trans. Image Process. (2023)
  * [3] Chen, T.S., Hung, W.C., Tseng, H.Y., Chien, S.Y., Yang, M.H.: Incremental false negative detection for contrastive learning. In: International Conference on Learning Representations (2022)
  * [4] Chouaf, S., Hoxha, G., Smara, Y., Melgani, F.: Captioning changes in bi-temporal remote sensing images. 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS pp. 2891–2894 (2021)
  * [5] Guo, Z., Wang, T., Laaksonen, J.T.: Clip4idc: Clip for image difference captioning. ArXiv abs/2206.00629 (2022)
  * [6] Hoxha, G., Chouaf, S., Melgani, F., Smara, Y.: Change captioning: A new paradigm for multitemporal remote sensing image analysis. IEEE Transactions on Geoscience and Remote Sensing 60, 1–14 (2022)
  * [7] Hu, Y., Yuan, J., Wen, C., Lu, X., Li, X.: Rsgpt: A remote sensing vision language model and benchmark (2023)
  * [8] Huynh, T., Kornblith, S., Walter, M.R., Maire, M., Khademi, M.: Boosting contrastive self-supervised learning with false negative cancellation. In: 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 986–996 (2022)
  * [9] Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q., Sung, Y.H., Li, Z., Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 139, pp. 4904–4916. PMLR (18–24 Jul 2021)
  * [10] Johnson, J., Douze, M., Jégou, H.: Billion-scale similarity search with GPUs. IEEE Transactions on Big Data 7(3), 535–547 (2019)
  * [11] Li, J., Selvaraju, R.R., Gotmare, A.D., Joty, S.R., Xiong, C., Hoi, S.C.H.: Align before fuse: Vision and language representation learning with momentum distillation. In: Neural Information Processing Systems (2021)
  * [12] Lin, C.Y.: ROUGE: A package for automatic evaluation of summaries. In: Text Summarization Branches Out. pp. 74–81. Association for Computational Linguistics, Barcelona, Spain (Jul 2004)
  * [13] Liu, C., Zhao, R., Chen, H., Zou, Z., Shi, Z.: Remote sensing image change captioning with dual-branch transformers: A new method and a large scale dataset. IEEE Transactions on Geoscience and Remote Sensing 60, 1–20 (2022)
  * [14] Liu, F., Chen, D., Guan, Z., Zhou, X., Zhu, J., Ye, Q., Fu, L., Zhou, J.: Remoteclip: A vision language foundation model for remote sensing (2024)
  * [15] Papineni, K., Roukos, S., Ward, T., jing Zhu, W.: Bleu: a method for automatic evaluation of machine translation. pp. 311–318 (2002)
  * [16] Qiqi Zhu, Xi Guo, Z.L., Li, D.: A review of multi-class change detection for satellite remote sensing imagery. Geo-spatial Information Science 27(1), 1–15 (2024)
  * [17] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning (2021)
  * [18] Rahhal, M.M.A., Bazi, Y., Alsharif, N.A., Bashmal, L., Alajlan, N.A., Melgani, F.: Multilanguage transformer for improved text to remote sensing image retrieval. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 15, 9115–9126 (2022)
  * [19] Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics (11 2019)
  * [20] Sun, X., Wang, P., Lu, W., Zhu, Z., Lu, X., He, Q., Li, J., Rong, X., Yang, Z., Chang, H., He, Q., Yang, G., Wang, R., Lu, J., Fu, K.: Ringmo: A remote sensing foundation model with masked image modeling. IEEE Transactions on Geoscience and Remote Sensing 61, 1–22 (2023)
  * [21] Tu, Y., Li, L., Su, L., Du, J., Lu, K., Huang, Q.: Viewpoint-adaptive representation disentanglement network for change captioning. IEEE Transactions on Image Processing 32, 2620–2635 (2023)
  * [22] Tu, Y., Li, L., Su, L., Lu, K., Huang, Q.: Neighborhood contrastive transformer for change captioning. IEEE Transactions on Multimedia 25, 9518–9529 (2023)
  * [23] Vedantam, R., Zitnick, C.L., Parikh, D.: Cider: Consensus-based image description evaluation. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4566–4575 (2015)
  * [24] Wang, D., Zhang, Q., Xu, Y., Zhang, J., Du, B., Tao, D., Zhang, L.: Advancing plain vision transformer toward remote sensing foundation model. IEEE Transactions on Geoscience and Remote Sensing 61, 1–15 (2022)
  * [25] Wang, F., Liu, H.: Understanding the behaviour of contrastive loss. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 2495–2504 (2020)
  * [26] Xu, L., Xie, H., Wang, F.L., Tao, X., Wang, W., Li, Q.: Contrastive sentence representation learning with adaptive false negative cancellation. Inf. Fusion 102(C) (feb 2024)
  * [27] Yao, L., Wang, W., Jin, Q.: Image difference captioning with pre-training and contrastive learning. In: AAAI Conference on Artificial Intelligence (2022)
  * [28] Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., Wu, Y.: Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research (2022)
  * [29] Zhang, J., Zhou, Z., Mai, G., Mu, L., Hu, M., Li, S.: Text2seg: Remote sensing image semantic segmentation via text-guided visual foundation models. ArXiv abs/2304.10597 (2023)
  * [30] Zhang, W., Cai, M., Zhang, T., Zhuang, Y., Mao, X.: Earthgpt: A universal multi-modal large language model for multi-sensor image comprehension in remote sensing domain (2024)
  * [31] Zhou, W., Guan, H., Li, Z., Shao, Z., Delavar, M.R.: Remote sensing image retrieval in the past decade: Achievements, challenges, and future directions. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 16, 1447–1473 (2023)


# Short-term forecast of EV charging stations occupancy probability using Big
Data streaming analysis

Francesca Soldan, Enea Bionda, Giuseppe Mauri, Silvia Celaschi
Ricerca sul Sistema Energetico - RSE S.p.A.
Milano, Italy
<EMAIL_ADDRESS>

###### Abstract

The widespread diffusion of electric mobility requires a contextual expansion
of the charging infrastructure. An extended collection and processing of
information regarding charging of electric vehicles may turn each electric
vehicle charging station into a valuable source of streaming data. Charging
point operators may profit from all these data for optimizing their operation
and planning activities. In such a scenario, big data and machine learning
techniques would allow valorizing real-time data coming from electric vehicle
charging stations.
This paper presents an architecture able to deal with data streams from a
charging infrastructure, with the final aim to forecast electric charging
station availability after a set amount of minutes from present time. Both
batch data regarding past charges and real-time data streams are used to train
a streaming logistic regression model, to take into account recurrent past
situations and unexpected actual events. The streaming model performs better
than a model trained only using historical data.
The results highlight the importance of constantly updating the predictive
model parameters in order to adapt to changing conditions and always provide
accurate forecasts.

_Keywords_ electric charging infrastructure $\cdot$ electric vehicle $\cdot$
big data streaming architecture $\cdot$ occupancy status forecast $\cdot$
streaming logistic regression

##  1 Introduction

The Integrated National Plan for Energy and Climate (PNIEC)[1], published in
January 2020 by the Italian Ministry of Economic Development, previews an
intensive spread of electric mobility by 2030. The aim of reaching 4 millions
battery electric vehicles (EVs) and 2 millions hybrid plug-in vehicles,
starting from a number of 70,000 circulating EVs, is really challenging. The
increase of EVs requires a contextual expansion of the charging station
network [2]. The actual number of 16,700 charging points is expected to grow
to 98,000-130,000, under the scenarios reported by Motus-E in a report
published in 2020 [3]. The research centre Ricerca sul Sistema Energetico (RSE
S.p.A.) confirms this scenario, estimating a number of 31,500 fast charging
points and 78,600 slow charging points by 2030, for a total of around 110,000
public charging points [4].
In this context, a more efficient electric charge data processing and
collection will be necessary. Each public charging station is indeed a
potential data source and the exploitation and valorization of all these data
can be useful for both charging point operators and final users. Charging
point operators could benefit for their planning activity, while final users
could receive updated information and forecasts of future occupancy status of
the charging stations.
Real-time data streams regarding charging station occupancies may be sent to a
central system, allowing their integration and processing with big data and
machine learning techniques. Possible final objectives could be the
identification of the most appropriate collocation of new charging stations,
the development of smart charging algorithms, the evaluation of the capacity
of power distribution systems to handle extra charging loads and the
assessment of the market value for the services provided by electric vehicles,
as vehicle-to-grid solutions [5]. In addition, the management of data coming
from electric vehicles and their charging stations has a crucial role for
operation and planning future Smart Grids [5].
This paper proposes a big data streaming architecture for providing short-term
forecasts of charging station occupancy probabilities. The predictive machine
learning algorithm takes into account both recurrent situations linked to the
past and actual unexpected events, in order to forecast the occupancy status
more accurately. The importance of considering a mix of historical and actual
conditions has been stressed during the Covid-19 disease 2019 pandemic: it has
introduced a multitude of disruptions to daily life, which conventional
forecasting models can not correctly predict. As regards the electric energy
sector, this problem arises in the context of electrical consumption forecasts
on the distribution grid [6]. However, mobility restrictions during lock-downs
have also impacted charging habits of EV owners, with inevitable influences on
predictions of occupancies of electric charging stations.

##  2 Data and methods

In order to retrieve the occupancy probability of an EV charging station a
classification model can be exploited. The logistic regression model is one of
the most fundamental and widely used classification methods [7] and has been
selected for a first architectural development.
However, a forecast model trained just using historical data can result in
large forecasting errors, especially in the case of unexpected events. A prime
example could be related to charging stations close to a stadium or an
exhibition center: a model trained only with historical data can provide high
occupancy probabilities just over days with yearly recurring events, while the
occupancy probability will be low in other cases.
As a consequence, the idea has been to initialize a Logistic Regression model
with historical data related to past charges and to increasingly update the
model using real-time data from the actual occupancy of EV charging stations.
In this way both recurrent situations linked to the past and actual unexpected
events can be taken into account.
The conversion of available data about past charges into continuous data
streams has allowed the development and testing of a big data streaming
architecture, potentially able to manage real-time data coming from EV
charging stations.

###  2.1 Available data

The data under consideration refer to 1,724 EV charges from a selected
charging station. They have the following characteristics:

  * •

the charges have been supplied in a period of three consecutive years;

  * •

the charge distributions within the different days of the week (Figure 1) and
the different hours of the day (Figure 2) indicate a higher supply frequency
in working days, with respect to Saturday and above all to Sunday, and in the
hours between 9 and 18;

  * •

the charge duration distributions, in minutes (Figure 3) and hours (Figure 4),
display the highest frequency of charges lasting less than one hour, with a
mean distribution value around 35-40 minutes.

Figure 1: Weekly distribution of charges. Figure 2: Distribution of charges in
each hour of day. Figure 3: Distribution of charge duration in minutes. Figure
4: Distribution of charge duration in hours.

###  2.2 Streaming architecture

The whole architecture has been implemented using Apache Spark and the
functions from its MLlib library [8]. In particular, the
StreamingLogisticRegression function included in the Spark MLlib library has
been selected; the function is natively able to update the initialized model
with the arrival of new data streams. The features chosen as model inputs are:

  * •

two cyclical variables to represent hours of the day;

  * •

two cyclical variables to represent months of the year;

  * •

a categorical variable to distinguish business days from weekends;

  * •

a categorical variable to distinguish working days from festivities;

  * •

seven categorical variables to represent different days of the week.

The year with the highest number of available data has been considered as a
training set to initialize the Streaming Logistic Regression model. Having at
disposal just static, historical data, it has been necessary to simulate
continuous data streaming in order to test the architecture. This task has
been realized with data from one of the two remaining years and using Apache
Kafka. It is important to note that the principal aim was to demonstrate the
feasibility of the streaming architecture implementation and not to select the
best forecasting model.
In Figure 5 the developed architecture is presented:

  * •

initial dataset is imported in Tableau for preliminary analysis and
visualizations;

  * •

test data have been selected and transformed into a continuous simulated data
stream, with a time resolution of one minute; data are sent into a Kafka topic
by a Kafka Producer;

  * •

training data have been used to initialize a StreamingLogisticRegression
model. Later, a Kafka Consumer reads the streaming data coming into the Kafka
topic; this data stream has a dual functionality: on the one hand it allows an
incremental update of the initialized model, on the other it is used to
extract hour and date in the next 15 minutes and to provide as output the
occupancy status forecast of the charging station, from the just updated
model;

  * •

occupancy probability of the considered charging station after 15 minutes from
the actual time is saved in another Kafka topic, written on InfluxDB and
visualized in Grafana.

Figure 5: Developed architecture in order to create a simulated continuous
data streaming, to train a StreamingLogisticRegression model and to extract
the resulting output.

##  3 Results

For all 525,601 minutes in the test set, the actual charge presence or absence
has been compared to the occupancy forecasts, performed 15 minutes before.
Figure 6 displays the model predictions and the actual occupancy status for
the week 22-29 September of the selected year. The considered models are the
Streaming Logistic Regression model and classical Logistic Regression model,
trained just using historical entries and not updated with real-time data; the
actual status is 1 if the charging station is occupied, 0 otherwise.

Figure 6: Occupancy probability forecasts obtained with the Streaming Logistic
Regression model (in yellow) and with the classical Logistic Regression model
(in green), compared to the actual occupancy status (in light-blue). The
actual status is 1 if the charging station is occupied, 0 otherwise. The week
between 22 and 29 September of the selected year is displayed.

In order to extract the occupancy status forecast from the occupancy
probability, a standard threshold of 0.5 is usually chosen: a probability
higher than 0.5 indicates a charge presence, while a probability lower than
0.5 indicates a charge absence. Again the class 1 stands for a charge
presence, the class 0 for a charge absence. From a visual analysis of the
results it appears that:

  * •

Streaming Logistic Regression model learns from historical data a modular
pattern, evident also for classical Logistic Regression model results. The
occupancy probability decreases indeed during the night hours and in Saturday
and Sunday, compared to the working days;

  * •

occupancy probabilities from the streaming model are generally lower than
those from the batch model. However, if a charging station is actually
occupied, the streaming forecasts display a higher increase. This increase is
more evident with long charges, when occupancy probabilities reach values
above 0.8;

  * •

occupancy probabilities are on average lower than 0.5; therefore, the
threshold to extract the corresponding class should be probably set lower than
the standard of 0.5.

The three indexes of precision, recall and F1-score allow a formalization of
these results. Considering the number of false positives (FP), false negatives
(FN), true positives (TP) and true negatives (TN) of the models, precision $p$
and recall $r$ are calculated as follows:

$p=\frac{TP}{TP+FP}\quad\quad;\quad\quad r=\frac{TP}{TP+FN}$ (1)

F1-score is the harmonic mean of $p$ and $r$:

$F_{1}=\frac{2}{\frac{1}{r}+\frac{1}{p}}=2\cdot\frac{p\cdot r}{p+r}$ (2)

Figures 7a and 7b show the precision/recall curve respectively for the
classical and the Streaming Logistic Regressions; the sample includes all
525,601 minutes in the test set. It is evident that the streaming model has a
higher precision for all the threshold values in the range 0.30-0.50, while
the batch model generally presents better recalls. The combination of
precision and recall in the F1-score attests more accurate predictions from
the Streaming Logistic Regression, for all the set thresholds (Figure 7c).

Figure 7: (a) Precision/recall curve for the classical Logistic Regression
model; (b) precision/recall curve for the Streaming Logistic Regression model;
(c) F1-score of the classical and Streaming Logistic Regression. Threshold
values are in the range 0.30-0.50. The sample includes all 525,601 minutes in
the test set.

Focusing on the Streaming Logistic Regression model, the threshold value
producing the best results is 0.30, while a threshold of 0.35 provides well
balanced precision and recall, with values for both the indexes between 0.63
and 0.67. However, a model with a better recall than precision will be chosen
in the case of need to forecast the highest number of charges, with the risk
of forecasting as a charge an event that will not be confirmed as an actual
charge. On the contrary, a model with a better precision than recall will be
chosen when it is necessary to forecast just correct charges, with the risk of
losing some charge predictions.

##  4 Conclusion

This paper presents a first model prototype to forecast the occupancy status
probability of EV charging stations. The developed big data streaming
architecture is based on Apache Spark, Spark Streaming and Apache Kafka. It
receives streaming data from a charging station and provides as output the
occupancy probability in the next 15 minutes. The selected forecasting model
is the Streaming Logistic Regression, initialized using historical data and
constantly updated with the arrival of real-time data streams.
The model learns from historical data a modular pattern, with a probability
decrease in the night hours and during the weekends. The real-time update of
the model results in an occupancy probability increase when a charge is
actually present. Therefore the streaming model provides better predictions
than the batch model. The occupancy status retrieval has been done by fixing
different threshold values: if the occupancy probability is higher than the
set threshold the prediction states the charging station occupancy, otherwise
it states the charging station availability. A threshold of 0.35 allows to
seek a balance between precision and recall indexes, resulting in the range
0.63-0.67 in this case.
The results highlight the necessity of a further optimization of Logistic
Regression parameters, such as the regularization parameters, the streaming
time window, the selected features and the gradient descent step. As regards
the choice of the classification model, just the Logistic Regression model has
been tested so far, but it is necessary to investigate which model is the best
appropriate for the specific use case; other examples could be the Decision
Tree Classifier, the Random Forest Classifier and the Gradient-Boosted Tree
Classifier [9].
Moreover, a web or mobile application could be developed to display on a map
the resulting occupancy probabilities in real-time for all the available
charging stations. This application will provide a tool of easier and more
immediate use, supporting an EV driver in the choice of the most probable free
charging station, close to his destination.
Finally, it could be useful to investigate the weights in the mix between
batch and real-time data and understand how to calibrate this mix. This
ability to keep historical and actual situations both into account can be
considered indeed as the main strength of the proposed architecture and will
become increasingly important for forecasting models related to various fields
in a fast and continuously changing world.

## References

  * Ministry of Economic Development, Ministry of the Environment and Protection of Natural Resources and the Sea, Ministry of Infrastructure and Transport [2019] Ministry of Economic Development, Ministry of the Environment and Protection of Natural Resources and the Sea, Ministry of Infrastructure and Transport, “Integrated National Energy and Climate Plan,” 2019,
https://www.mise.gov.it/images/stories/documenti/it_final_necp_main_en.pdf.

  * Autorità di Regolazione per Energia Reti e Ambiente [2020] Autorità di Regolazione per Energia Reti e Ambiente, “Memoria 18 febbraio 2020 41/2020/i/eel,” 2020,
https://www.arera.it/it/docs/20/041-20.htm.

  * Strategy& [2020] Strategy&, “Il futuro della mobilità elettrica: l’infrastruttura di ricarica in italia @2030,” Motus-E, Tech. Rep., 2020.
  * Ricerca sul Sistema Energetico - RSE S.p.A. [2019] Ricerca sul Sistema Energetico - RSE S.p.A., “Valutazione del fabbisogno di infrastrutture di ricarica pubbliche per auto elettriche all’anno 2030,” Unpublished confidential document, 2019.
  * Boyang et al. [2017] L. Boyang, M. C. Kisacikoglu, C. Liu, N. Singh, and M. Erol-Kantarci, “Big data analytics for electric vehicle integration in green smart cities,” _Green communications and computing networks_ , 2017.
  * Bionda et al. [2020] E. Bionda, A. Maldarella, F. Soldan, G. Paludetto, and F. Belloni, “Covid-19 and electricity demand: focus on Milan and Brescia distribution grids,” in _2020 AEIT International Annual Conference (AEIT)_ , 2020, pp. 1–6.
  * [7] M. Sluijmers, “Logistic Regression: Discover the Powerful Classification Technique,” https://towardsdatascience.com/logistic-regression-discover-the-powerful-classification-technique-d60c0ae4d3b1.
  * [8] Apache Software Foundation, “Apache Spark MLlib,” https://spark.apache.org/mllib/.
  * S. Li [2018] S. Li, “Machine Learning with PySpark and MLlib — Solving a Binary Classification Problem,” https://towardsdatascience.com/machine-learning-with-pyspark-and-mllib-solving-a-binary-classification-problem-96396065d2aa, 2018\.


∎

11institutetext:  Institute of Applied Physics, National Academy of Sciences
of Ukraine, Petropavlivska str. 58, 40000 Sumy, Ukraine
Tel.: +380-542-222-794
Fax: +380-542-223-760
11email<EMAIL_ADDRESS>
# Nanoclusters and nanoscale voids as possible sources of increasing dark
current in high-gradient vacuum breakdown

I.I. Musiienko  S.O. Lebedynskyi  R.I. Kholodov

(Received: date / Accepted: date)

###### Abstract

The potential barrier model considers an additional current that can lead to
the high-gradient breakdowns in accelerating structures is proposed. An
oscillatory resonance feature of the field emission current from a double-
layer metal system with a nanoscale coating is shown. The double potential
barrier was used for calculations of the field emission current density value.
The presence of resonant properties of the field emission current density of a
double-layer metal system with a nanometric coating is revealed. The field
emission current increasing more than 5 times (for the constant value of an
electric field strength $E~{}=~{}5~{}GV~{}/~{}m$) when considering not ideal
copper surface with presence of nanoclusters on the surface of or nanoscale
voids in the near-surface layer is observed.

###### Keywords:

field emission current densitypotential barrierthe transmission coefficient of
the potential barrierresonant tunnelingfull width at half maximum (FWHM).

###### pacs:

79.70.+q 03.65.Ge

††journal: Eur. Phys. J. D

##  1 Introduction

Field emission is a type of emission that is caused by the quantum mechanical
tunneling of electrons from the metal surface to a vacuum induced by an
electrostatic field r1 . This phenomenon is typical for strong fields with
electric field strength $E=10^{8}\div 10^{10}$ V / m. It is one of the main
factors, which leads to the dark currents occurring in the accelerating
structures and, as a consequence, the loss of electrical insulating properties
of the interelectrode gap r2 ; r3 ; r4 . The problem of regulating the
magnitude of the field emission current is relevant, in particular, to
preventing high-vacuum high-gradient breakdowns. Another important goal is to
obtain gradient-stable materials that eliminate the possibility of electric
discharge in the components of modern accelerators. The amplification of the
field emission current is also important, particularly for the operation of
field emission sources, tunnel microscopy requirements, electron holography,
vacuum nanoelectronics r5 ; r6 ; r7 ; r8 ; r9 .

Figure 1: Double potential barrier model of rectangular-triangular shape

Modeling of experimental operating conditions of the compact linear electron-
positron collider in the CLIC (Compact Linear Collider) project is carried out
at CERN (European Organization for Nuclear Research). It was showed when
applied a high-frequency electromagnetic field to the metal surfaces of
accelerating structures, which provides an acceleration rate of the order of
100 MV / m, the high-gradient high-vacuum breakdowns occurs r2 ; r3 ; r4 .
Since the breakdowns leads to a significant loss of energy and to the damaging
surface of facilities components, reducing the probability of its occurrence
in the CLIC accelerating structures is a necessary condition for the high-
quality functioning of the future collider.

Therefore, various methods of increasing stability of the accelerator modules
surface to high-gradient high-vacuum breakdown are being studied. Among the
possible methods to prevent high-gradient breakdowns are metal surface
modification by implanting argon, nitrogen, zirconium ions into the near-
surface copper layer (which is the base material of CLIC accelerating
structure disks) and an ion-plasma treatment of the substrate surface by a
metal layer with a greater work function r10 .

The effect of changing the electrode surface characteristics on the field
emission current density was experimentally investigated in r10  and
theoretically in r11 ; r12 . In the article r10 , it was experimentally
confirmed that the condition of the electrode surface significantly affects
the breakdown voltage. Plasma and ion-beam modification of the copper surface
leads to an increase in the high-vacuum breakdown voltage from 5 to 35 % and
decrease the dark current in the interelectrode gap. The result with the
greatest suppression of the high-vacuum breakdown in the experiments was
obtained for copper with the TiN coating and additional irradiation of the
surface with the argon ions energy is equal to 300 keV.

In articles r13 ; r14  an effect of a magnetic field on the field emission
current was interpreted. The breakdown voltage decrease due to an increase of
ionization by the electron impact was predicted. In r15  the possibility of
considering the relativistic correction for the field emission current density
of electrons from metal is described theoretically. It was found that in the
case of the field electron emission from pulsars the relativistic change in
the current value is about 10 % of the Fowler-Nordheim current density. In r16
it was found that the presence of an external magnetic field parallel to the
metal surface with a value of 10 T leads to a decrease in the transmission
coefficient by 10 %. In paper r17  the flash emission with a duration of 1 - 5
ns, induced by a pulsed electric field $E\geq 10^{7}$ V / m applied to a
cathode, produced of graphene-like structures, was experimentally detected. In
contrast to explosive emission r7  the new type of emission is not associated
with the appearance of plasma and destruction of local nano- and microregions
of the cathode surface.

The investigation r18  shows an effect of the Coulomb blockade phenomenon on
the current density value from nanoscale objects. In r19  the low-voltage
field emission of electrons from nanostructured carbon materials is
theoretically described due to the presence of resonant surface states,
provided that the emitter surface has two carbon phases with different
electronic properties. The possibility of increasing the emission current due
to the resonant tunnelling of electrons through the double potential barrier
by four or more orders of magnitude was predicted.

Investigations of the vacuum breakdown mechanism at the prototypes of the
future CLIC accelerator showed that the dark current values cannot be
explained only by an electric field enhancement by metal surface
irregularities r20 . The field emission measurements on a macroscopic system
needs a large – 30, 100 or more – correction factor to get current $I$ vs
voltage $U$ data, corresponding to the current density value from the Fowler-
Nordheim equation. However, an amplification factor is not enough for
corrections of 30 to 100 and more in the case of measuring the field emission
current density from a copper surface in experiments performed in the CERN DC
Spark Systems (electrodes in ultra high vacuum with a high voltage external
circuit, used to study breakdown). Thus, possible sources of dark current
amplification, which leads to high-vacuum breakdowns, are now being searched
for.

The main idea of this paper is to study the possibility of increasing the
field electron emission current in the case of two-layer metal systems by
choosing the coating thickness on the substrate. It is similar to a special
technique of an anti-reflective coating to reduce losses for the reflection of
light r21 ; r22 ; r23 . Figure 1 shows the shape of the double potential
barrier, which is considered for calculating the field electron emission
current from a two-layer metal system. Similar models of the potential barrier
were used to study the properties of quantum wells and resonant tunnelling of
the field electron emission through the double barrier r24 ; r25 ; r26 ; r27 ;
r28 .

The paper r29  presents an exact analytical theory for field emission from
dielectric coated cathode surfaces, by solving the one-dimensional (1D)
Schr$\ddot{o}$dinger equation with a double-triangular potential barrier,
which is formed by applying dc electric field to the dielectric coated cathode
surface. The quantum model in r29  is also compared with a modified Fowler-
Nordheim equation for a double-triangular barrier, showing qualitatively good
agreement.

The purpose of this study is to perform a numerical calculation of the field
electron emission current using the model of the double potential barrier of a
rectangular-triangular shape. One of the issues is to explain the analytical
relationship between the de Broglie wavelength of the tunnelling electron and
the width of the potential well in the case of the double rectangular
potential barrier. An important issue is to demonstrate the effect of resonant
tunnelling on the field emission current density due to the double potential
barrier. This theory is used to find additional sources of increasing the
field emission current from the metal surface.

##  2 Replacement of a single-stage triangular potential barrier with a
rectangular one of equal transparency

The double potential barrier for barrier structures in semiconductor layered
systems is considered in reference r24 . It is also used to evaluate the value
of the field emission current from a double-layer metal-semiconductor system
in r29 ; r30  and for estimates the current density from the near-surface
copper layer modiﬁed by the point vacancies in r12 . The proposed model makes
it possible to describe two different objects such as nanoclusters and
nanoscale voids.

If there is a nanoscale void in the surface metal layer, in Figure 1 the model
represents $h_{1}$ as the barrier width at the metal-vacuum interface, $h_{2}$
is the distance from the metal-vacuum interface to the location of the
nanoscale void in the surface metal layer, $h_{3}$ is the nanoscale void
diameter. If the nanocluster is located on the metal surface, then the barrier
parameters in Figure 1 can be explained as $h_{1}$ is the barrier width at the
metal-vacuum interface, $h_{2}$ is the nanocluster size, $h_{3}$ is the dipole
layer thickness of the metal to metal contact. Below in this study, the
authors will consider the nanoscale voids.

Figure 2: Replacement of the triangular shape of the potential barrier with a
rectangular one, provided that their transmission coefficients are equal

Note that for an authentic model of the potential barrier, it is necessary to
take into account the presence of electric field strength in the dipole layer,
which occurs upon contact of surfaces of various metals r30 . In the case of
two layers of identical metal, the dipole dielectric field is zero.

Fowler and Nordheim found the formula for the current emission density
$j_{F-N}$ based on the determination of the transmission coefficient $D_{F-N}$
as a function of the electron energy $W$ in the case of the triangular
potential barrier at the boundary from the metal to a vacuum. In this
investigation, it will be replaced the authentic triangular potential barrier
shape shown in Figure 2 by the rectangular one provided that their
transmission coefficients are equal. Thus, it is possible to significantly
simplify and speed up numerical calculations and avoid mathematical
transformations with Hankel functions of the second kind to find the
transmission coefficient of the potential barrier. Let us introduce the
notation: $U$ is the potential energy; the potential energy height
$C=\mu+\chi$; $\mu$ is the chemical potential; $\chi$ is the work function;
$F=e\cdot{E}$; $e$ is the electron charge; $E$ is the electric field strength.

The transmission coefficient of a rectangular potential barrier of width
$h_{1}$ r24 ; r31  can be written in the following analytical form in selected
convenient notations:

$D_{1}=\left(1+\frac{\eta^{2}}{4}sh^{2}(k_{2}h_{1})\right)^{-1},$ (1)

where $\eta=\frac{C}{\sqrt{W\varphi}}$; $k_{2}=k\sqrt{\varphi}$;
$k=\frac{\sqrt{2m_{e}}}{\hbar}$; $m_{e}$ is the electron mass; $\varphi=C-W$.

The expression for the transmission coefficient $D_{F-N}$ of the triangular
potential barrier, obtained by Fowler and Nordheim r1 , is necessary to find
the field electron emission current density at a range of values of the
electric field strength $E=10^{8}\div 10^{10}$ V / m and is equal to:

$D_{F-N}=\frac{4}{\eta}\exp{\left(-\frac{4k\varphi^{\frac{3}{2}}}{3F}\right)}.$
(2)

For $F>10^{10}$ eV / m it is necessary to use the numerical method of
obtaining the complex amplitudes of wave functions and the transmission
coefficient of the potential barrier because formula (2) for the above
interval is not accurate. From the equality $D_{1}=D_{F-N}$ it was obtained
the dependence of $h_{1}$ on the physical quantities $W$ and $F$ in general
form, which is correct for the above-mentioned range of values of the electric
field strength $E$:

$\displaystyle h_{1}=\frac{1}{k_{2}}\times$
$\displaystyle\times{arsh\left(\frac{2}{C}\cdot\sqrt{\frac{W}{D_{F-N}}\left(D_{F-N}W-D_{F-N}C+\varphi\right)}\right)},$
(3)

where $arsh(x)$ is an inverse hyperbolic sine. On condition $k_{2}h_{1}\gg 1$,
equating the right-hand sides of formulas (1) and (2) and not taking into
consideration the values of the factors before the exponents of the
expressions, an approximate formula was found for the effective $h_{1eff}$:

$h_{1eff}\approx\frac{2\varphi}{3F}.$ (4)

The approximate expression to formula (3) can be written as:

$h_{1}\approx\frac{2\varphi}{3F}+\frac{1}{2k_{2}}\ln{\left(\frac{4}{\eta}\right)}.$
(5)

Figure 3 shows the graphical dependences of $h_{1}$ on the value of $F$ for
the parameters of copper $C=12$ eV, $W=\mu$. For calculations, it is advisable
to take the electron energy near the energy value of the chemical potential
$\mu$, as the most probable current carriers of the field emission from the
metal surface are free electrons at the upper energy levels according to
Sommerfeld’s theory r1 ; r8 . The graphs of dependences (3) (indicated in
Figure 3 by a solid line) and (5) (a line of rings) coincide on the considered
interval of values of $F$. Formula (4) (the line is represented by asterisks)
approximately describes the analytical dependence $h_{1}(F)$.

Figure 3: Graphs of dependence $h_{1}(F)$: formula (3) is represented by a
solid line; expression (4) is shown by a line of asterisks; formula (5) is
depicted by a line of rings Figure 4: Graphic dependencies $h_{1}(W)$ Figure
5: Graphic dependences $D(W)$ in the case of single-stage potential barriers

Figure 4 demonstrates the dependences of the width $h_{1}$ of a rectangular
potential barrier on the electron energy for different values of $F$. Thus, if
the constant electric field strength increases, then the value of $h_{1}$
decreases. As a consequence, the width of the rectangular potential barrier
narrows, and for values of $F>10^{10}$ eV / m, the transmission coefficient
$D_{1}$ increases significantly.

Figure 5 represents the graphic dependences of the transmission coefficient
$D$ of single-stage potential barriers of triangular and rectangular shapes on
the electron energy for the parameters: $C=12$ eV, $\mu=7.5$ eV, $F=7.5\cdot
10^{9}$ eV / m. The quantity $h_{1}$ is determined for an arbitrary energy $W$
using expression (5). The curves of dependences are indicated by a solid line
in the case of the triangular barrier, and a line of asterisks is constructed
as a result of calculations by a numerical method, and a line of squares is
used to depict the dependencies for the rectangular barrier, respectively.
Therefore, using formula (5), it is possible to quantitatively replace the
triangular shape of the barrier with a rectangular one with a width of
$h_{1}$.

The field emission current density is determined by the expression:

$j=\int_{0}^{\infty}eDNdW,$ (6)

where $N$ is the number of electrons emit on a metal surface of unit area per
unit time with a kinetic energy $W$ normal to the surface. The quantity $N$
was evaluated by Nordheim and under the conditions of the problem is found in
the form:

$N=\frac{m_{e}}{2\pi^{2}\hbar^{3}}\left(\mu-W\right).$ (7)

The integrand of formula (6) is the differential current:

$dj/dW=eDN.$ (8)

The right-hand side of expression (8) determines the electron energy
distribution density. Figure 6 shows the graphic dependences of the
differential current $dj/dW$ on the electron energy at the parameters that
were used for Figure 5. In Figure 6 the solid line corresponds to the case of
the triangular potential barrier. The line of solid circles is used if the
rectangular barrier is considered. As can be seen from Figure 6, the plots of
dependences have coincided well for rectangular and triangular shapes of
potential barriers.

Figure 6: Differential current in the case of single-stage potential barriers:
graphs of analytical dependence (solid line) for the triangular shape of the
barrier; dependence curve (line of solid circles) in the case of a rectangular
barrier

##  3 Analysis of the properties of a double potential barrier of rectangular
shape

The idea of this model is to emit electrons from the metal surface to a vacuum
sequentially through two rectangular potential barriers due to the tunneling
effect. In the case of the selected parameters, the probability of an electron
with energy $W$ passing through a double potential barrier can be higher than
for a single-stage barrier, and as a consequence, resonant tunneling of
electrons will occur r25 .

Figure 7: The double potential barrier of rectangular shape

Figure 7 shows a simplified for computation potential barrier shape for the
metal-metal-vacuum medium with a thickness of the nanoscale metal coating,
which is used to find the analytical expression of the transmission
coefficient and transfer of the revealed mathematical regularities to the real
model of the double potential barrier of a system of two layers of metals. The
model is similar to the barrier shape to describe a quantum resonant-tunneling
diode r25 . Each of the regions means: V is a metal substrate, IV is a dipole
layer of metal-metal contact, III is a coating layer identical to the
substrate material, II is a potential barrier at the metal-vacuum interface, I
is a vacuum. Region II is a quantitative replacement of triangular barrier by
a rectangular one with equal transparency. The $x_{i}$ coordinates are the
boundaries of the specified regions. The parameters $h_{1}$, $h_{2}$, $h_{3}$
of the model are the width of regions II, III, IV, respectively. The subscript
numbering of the wave vectors $k_{i}$ coincides with the number of each
region. If $h_{1}=0$ or $h_{3}=0$, the double potential barrier of a
rectangular shape becomes single-stage.

The transmission coefficient of the double potential barrier of rectangular
shape in Figure 7 with the same potential energy $C$ in regions II and IV is
found in analytical form:

$D_{2}=\left(1+\frac{\eta^{2}}{4}sh^{2}\left(k_{2}\left(h_{3}+h_{1}\right)\right)+Z\right)^{-1},$
(9)

where
$Z=L\left(\frac{\eta^{4}}{16}L\sin^{2}{\left(k_{1}h_{2}\right)}-P+R\right)$;
$L=ch{\left(k_{2}\left(h_{3}+h_{1}\right)\right)}-ch{\left(k_{2}\left(h_{3}-h_{1}\right)\right)}$;
$P=\frac{\eta^{2}}{2}\sin^{2}{\left(k_{1}h_{2}\right)}\cdot
ch{\left(k_{2}\left(h_{3}+h_{1}\right)\right)}$;
$R=\frac{\eta^{3}\left(\varphi-W\right)}{8C}\sin{\left(2k_{1}h_{2}\right)}\cdot
sh{\left(k_{2}\left(h_{3}+h_{1}\right)\right)}$;
$k_{1}=k\sqrt{W}$; $k_{1}=k_{3}=k_{5}$; $k_{2}=k_{4}$.
In general, the transmission coefficient of a double barrier with different
heights of potential energy regions was found in r24 ; r28 . If $h_{1}=0$ or
$h_{3}=0$, then formula (9) is simplified to expression (1). Using the
condition of maximum manifestation of resonance:

$h_{1}=h_{3}=h,$ (10)

on condition $W=C/2$, which simplifies the calculation, formula (9) can be
written as:

$D_{2}=\left(1+\cos^{2}{\left(kh_{2}\sqrt{\frac{C}{2}}\right)}\cdot
sh^{2}\left(2kh\sqrt{\frac{C}{2}}\right)\right)^{-1}.$ (11)

The condition $D_{2}=1$ for the absolute transparency of a double potential
barrier of rectangular shape in the case of the expression (10) and at the
value $W=C/2$ is obtained in the form:

$h_{2}=\frac{\lambda_{B}}{4}\left(2n+1\right),~{}n~{}=~{}0,~{}1,~{}2,\ldots$
(12)

where $\lambda_{B}={2\pi\hbar}/{\sqrt{m_{e}C}}$ is the de Broglie wavelength
of an electron.

At the electron energy $W=6$ eV then $\lambda_{B}=5\cdot 10^{-10}$ m, and
using (12), $h_{2}=1.25\cdot 10^{-10}$ m. It is of importance, that the
formula (12), found under condition (10) and the value of $W~{}=~{}C~{}/~{}2$.
The maximum possible value of $D$ is also in the case of violation of the
condition (10), but less than one (not absolute transparency). Introduction
(10) and $W~{}=~{}C~{}/~{}2$ is necessary to simplify (9) and find the simple
mathematical expression (11) for transparency of the double rectangular
barrier.

Figure 8: Graphic dependences of the transmission coefficient $D_{2}$ on $W$

Applying conditions (10) and $h_{2}=\pi\sqrt{{2}/{C}}\approx$ $1.25\cdot
10^{-10}$ m, the transmission coefficient (9) can be represented as a second-
order Taylor series near the coordinate $W~{}=~{}C~{}/~{}2$:

$\displaystyle
D=1-\frac{4}{C^{2}}\left(ch{\left(h\sqrt{\frac{C}{2}}\right)}+\frac{\pi}{4}sh{\left(h\sqrt{\frac{C}{2}}\right)}-1\right)^{2}\times$
$\displaystyle\times{\left(W-\frac{C}{2}\right)^{2}}.$ (13)

If $h>10^{-10}$ m, then formula (13) is simplified to the form:

$D=1-\left(\frac{1+\pi/4}{C}\right)^{2}\exp{\left(h\sqrt{2C}\right)}\left(W-\frac{C}{2}\right)^{2}.$
(14)

Figure 8 represents the graphic dependences of the transmission coefficient
$D_{2}$ on the electron energy of the double potential barrier from Figure 7
at $C=12$ eV, $h_{1}=$ $3\cdot 10^{-10}$ m, $h_{3}=10^{-10}$ m. As can be seen
from Figure 8, the transmission coefficient maxima of the potential barrier
are observed for arbitrary $h_{1}$, $h_{3}$ and the height and width of the
graphic dependences peaks are increase with increasing energy $W$. If the
value of $h_{2}$ increases, the width of the peaks decreases.

Figure 9 illustrates the dependences of the differential current on the energy
$W$ under the conditions $C=12$ eV, $\mu=7.5$ eV, $h_{1}=h_{2}=1.5\cdot
10^{-10}$ m, $h_{3}=10^{-10}$ m in the case of a double potential barrier of
rectangular shape (the solid line) and rectangular single-stage barrier (the
line is depicted by dots). The parameters $h_{1}$, $h_{2}$, $h_{3}$ are
selected so that the resonance condition (12) is fulfilled. Thus, the width
$h_{2}$ is multiple to a quarter of the de Broglie wave $\lambda_{B}~{}/~{}4$,
which is similar to a special technique of an anti-reflective coating r21 ;
r22 ; r23 . An additional coating applied to the metal substrate not only
increases transparency but also reduces the energy scattering of field
emission electrons in the region of absolute transparency of the double
rectangular potential barrier. This conclusion is confirmed by the graphs in
Figure 9.

Figure 9: Differential current plots of the single-stage (depicted by dots)
and the double (solid line) rectangular potential barriers Figure 10: Graphic
dependences of the transmission coefficient $D_{2}$ on: a) $h_{1}$; b) $h_{2}$

In Figure 10 a), b) it is seen that if $W=6$ eV, $C=2W$, $h_{3}=1.25\cdot
10^{-10}$ m then the transmission coefficient (11) is the greatest when
$h_{2}=\lambda_{B}~{}/~{}4$. By increasing the values of $h_{1}$ and $h_{3}$,
the transmission coefficient decreases to zero, but the graphic appearance of
the dependencies is retained. From Figure 10 b) it can be concluded that there
is the repeatability of maxima and minima of $D_{2}$ according to formula
(12). These peaks were found numerically because it is difficult to
experimentally fulfill the condition for the thickness of the coating on the
surface of a metal substrate.

Figure 11 illustrates the graph of the differential current dependence on the
electron energy for the barrier model from Figure 7. It is observed the
resonance region of the aforementioned quantity near the coordinate of maximum
$W_{C}=5.1$ eV. The second peak is the background non- resonance differential
current. If there is no potential barrier for the motion of the electron in
region IV, which is indicated in Figure 7, the background non-resonance
differential current is transformed into a differential field emission
current. Figure 11 was obtained under conditions: $C=12$ eV, $\mu=7.5$ eV,
$h_{2}=h_{3}=1.5\cdot 10^{-10}$ m. The value of $h_{1}$ is determined from
expression (5) at $F=7.5\cdot 10^{9}$ eV / m.

Figure 11: Dependence on $W$ of differential current of the double rectangular
potential barrier

Therefore, for a theoretical study of the field emission phenomenon from a
metal in the presence of nanoscale voids in the surface metal layer, the model
of rectangular double potential barrier simplified for calculations is
proposed.

##  4 Simulation of the field emission current through a double barrier in the
presence of an external electric field

For verification with analytical results for the barrier in Figure 7, it is
calculated by the numerical method the value of the field emission current
density from a two-layer metal system in the case of the potential barrier
model from Figure 1. First of all, let us compare the dependences of the
transmission coefficient and the differential current on $W$ for the double
barrier, which correspond to Figure 7, and the double barrier in Figure 1.

Figure 12 and Figure 13 were obtained under the conditions: $C=12$ eV,
$\mu=7.5$ eV, $h_{2}=h_{3}=1.5\cdot 10^{-10}$ m. The value of $h_{1}$ is
determined for an arbitrary energy $W$ from expression (5) at $F=7.5\cdot
10^{9}$ eV / m. The curve depicted by dots corresponds to the dependence for
the model in Figure 7. The solid line indicates the dependence graph in the
case of the double potential barrier described in Figure 1. Therefore, a
quantitative coincidence of dependence graphs is observed at a shift that less
than $0.1$ eV in the maxima range of physical quantities. Figure 14 shows two
peaks of the dependence graphs of differential current on the electron energy
that will be at other values of $F$.

Figure 12: Graphic dependences of $D$ on energy $W$ in cases of double
rectangular (the line of dots) and double (the solid line) potential barriers
Figure 13: Graphic dependences of differential current on energy $W$ in cases
of double rectangular (line of dots) and double (solid line) potential
barriers Figure 14: Comparison of the field emission current density $j$ and
$j_{F-N}$ in the case of the double potential barrier model

Figure 14 demonstrates the dependence graphs of $j/j_{F-N}$ on $h_{2}$
obtained by the numerical method at different values of $h_{3}$. At the value
$E=5\cdot 10^{9}$ V / m for the copper characteristics: $\mu=7.5$ eV,
$\chi=4.5$ eV. Then the current $j_{F-N}$ from the ideal copper surface is
$4.2\cdot 10^{7}$ A / m2. In Figure 14 the maxima of the current density $j$
are more than 5 times greater than $j_{F-N}$ at the above parameters for
$h_{2}<10^{-9}$ m.

Properties for the model in Figure 7 will be typical for the double potential
barrier from Figure 1. Under the conditions of the selected parameters
$h_{1}$, $h_{2}$, $h_{3}$ for a system of two metal layers, where the upper
layer has a nanoscale thickness, the similar resonant tunneling r25  of
electrons will occur and one can observe a reduction of electron energy
scattering in the absolute transparency region. From Figure 14 it can be
concluded that with increasing $h_{2}$ the current density oscillations are
decreasing. But the important contribution to the dark current in accelerating
cavities and, hence, to the field emission current density is given by the
oscillations of the current even when the value of the parameter $h_{2}$ is
more than one nanometer. The greatest value of the current density dependence
on $h_{2}$ in Figure 14 is the first maximum at the point
$h_{2}\approx\lambda_{B}~{}/~{}4$, which is similar to a special technique of
an anti-reflective coating r21 ; r22 ; r23 .

##  5 Width investigation of the resonance oscillatory peak of field emission
current

Let us evaluate the full width at half maximum ($FWHM$) of differential
current $dj/dW$ in the resonance region from Figure 11. One can use the
expressions (13) and (14) that correspond to the formula (9). One can find the
analytical formula for $FWHM$, taking into account the expression (14) and on
condition $D(W_{1})~{}=~{}0.5$, in the following form:

$FWHM=\left(\frac{\sqrt{2}}{1+\frac{\pi}{4}}\right)C\exp{\left(-h\sqrt{\frac{C}{2}}\right)}.$
(15)

The next stage is to approximate the resonant region of the dependence $dj/dW$
from Figure 11 by a Gaussian function:

$y=y_{0}+A\exp{\left(-\frac{\left(W-W_{C}\right)^{2}}{2\omega^{2}}\right)}.$
(16)

Figure 15 shows the dependence graph (16), depicted by a line of triangles at
numerically found values: $W_{C}~{}=~{}5.1$ eV, $y_{0}~{}=~{}5.3\cdot 10^{8}$
A / m2 / eV, $\omega~{}=~{}0.04$ eV, $A~{}=~{}2.1\cdot 10^{9}$ A / m2 / eV. In
Figure 15 the solid line describes the resonant region of dependence from
Figure 11, obtained numerically. The formula for $FWHM$ was found, using (16),
and it is written as:

$FWHM=2\omega\sqrt{\ln{4}}.$ (17)

From formula (17) it is found that $FWHM~{}\approx~{}0.1$ eV. Thus, the
appearance of the resonant maximum region is a characteristic of electrons in
a narrow range of energies and selected parameters $h_{1}$, $h_{2}$, $h_{3}$,
when the condition (12) is fulfilled.

Figure 15: Approximation of the differential current dependence on the
electron energy by a Gaussian function in the resonance region

Let us compare the dependence graph of $FWHM$ of the two peaks, constructed by
a solid line in Figure 13, at different values of the electric field strength
$E$. From Figure 16, constructed under conditions $C=12$ eV, $\mu=7.5$ eV,
$h_{2}=h_{3}=1.5\cdot 10^{-10}$ m, it is concluded that the $FWHM$ value of
the differential current dependence on $W$ from Figure 13 in the resonant
maximum region at different $E$ is in a narrow range of energies and is equal
to $0.1$ eV. The value of full width at half maximum in the resonant region of
the differential current dependence on $W$ decreases with increasing $h_{3}$.
The $FWHM$ values for the background non-resonance differential current peak,
shown by the line with circles, were found more than 5 times greater in
comparison with the results for the resonance maximum region. The $FWHM$
values, represented by the line with squares in Figure 16, it is found by the
numerical method for the barrier model from Figure 1, are consistent with the
analytical result (17) in the case of the potential barrier form in Figure 7.

Figure 16: Graphic dependences of $FWHM$ on $E$ in the case of the double
potential barrier of rectangular-triangular shape

##  6 Summary

A potential barrier model has been proposed that takes into account additional
mechanisms of high-gradient high-vacuum breakdowns in accelerating structures.
It was experimentally confirmed in reference r20  that the field emission
current from a real metal surface is greater than the value calculated
theoretically. It is assumed that the sources of increased dark current in the
accelerating cavities and, consequently, the field emission current $j_{F-N}$
from the electrode surface are nanoclusters on the surface and nanoscale voids
in the near-surface metal layer.

As a result, it was shown that the field emission current from the metal
surface near the nano-objects has an oscillatory resonance feature. The
resonance condition (12) determines the size of the nanocluster on the metal
surface or the depth of the nanoscale void from the metal surface whose
diameter is a multiple of ${\lambda_{B}}~{}/~{}{4}$. The current takes on a
maximum value when the transmittance is 1 and provided the size of the
nanoscale void $h_{1}=h_{1eff}~{}\approx~{}2\varphi~{}/~{}(3eE)$.

The full width at half maximum ($FWHM$) of the resonance region does not
depend on the electric field strength, but one depends on the height of the
potential barrier $C$ and the size $h_{3}$. At the parameters $C~{}=~{}12$ eV,
$\mu~{}=~{}7.5$ eV, $h_{2}~{}=~{}h_{3}~{}=~{}1.5\cdot 10^{-10}$ m,
$F~{}=~{}7.5\cdot 10^{9}$ eV / m, it is found that $FWHM~{}\approx~{}0.1$ eV.

It is revealed that at $E~{}=~{}5$ GV / m the field emission current density
from copper surface increases more than 5 times in the case of double
potential barrier model under the resonance condition (12) compared to
$j_{F-N}$.

###### Acknowledgements.

The publication is based on the research provided by the grant support of the
National Academy of Sciences of Ukraine (NASU) for research
laboratories/groups of young scientists of the National Academy of Sciences of
Ukraine to research priority areas of development of science and technology in
2021 - 2022 under contract No 16 / 01 - 2021 (3). The authors thank to O.P.
Novak for helpful discussions on the paper.

##  7 Authors contributions

All the authors were involved in the preparation of the manuscript. All the
authors have read and approved the final manuscript.

## References

  * (1) R.H. Fowler, L. Nordheim, Proc. Roy. Soc. A 119, 173 (1928)
  * (2) A. Kyritsakis, M. Veske, K. Eimre, V. Zadin, F. Djurabekova, J. Phys. D: Appl. Phys. 51, 22 (2018)
  * (3) Nicholas C. Shipman,in Experimental study of DC vacuum breakdown and application to high-gradient accelerating structures for CLIC (University of Manchester, Manchester, 2014)
  * (4) N. Shipman, I. Profatilova, A.T. Perez Fontenla, W. Wuensch, S. Calatroni, CLIC - Note 1078, 1 (2017)
  * (5) N. Egorov, E Sheshin, in Field Emission Electronics (Springer Series in Advanced Microelectronics 60, Springer International Publishing AG, 2017)
  * (6) G.N. Fursey, in Field emission (Publishing House ”Lan”, St. Petersburg, 2012)
  * (7) G.A. Mesyats, in Explosive Electron Emission (Publishing House of Physical and Mathematical Literature, Moscow, 2011)
  * (8) G.G. Vladimirov, in Physical Electronics. Emission and Interaction Particles with a Solid (Publishing House ”Lan”, St. Petersburg, 2013)
  * (9) M.I. Elinson, G.F. Vasiliev, in Autoelectronic emission (Publishing House of Physical and Mathematical Literature, Moscow, 1958)
  * (10) V.A. Baturin, A.Yu. Karpenko, V.E. Storizhko, V.A. Shutko, Probl. At. Sci. Tech. 116, 4 (2018)
  * (11) I.I. Musiienko, R.I. Kholodov, Probl. At. Sci. Tech. 121, 3 (2019)
  * (12) I.I. Musiienko, R.I. Kholodov, J. Nano- Electron. Phys. 11, 3 (2019)
  * (13) S.O. Lebedynskyi, V.I. Miroshnichenko, R.I. Kholodov, V.A. Baturin, Probl. At. Sci. Tech. 98, 4 (2015)
  * (14) S. Lebedynskyi, O. Karpenko, R. Kholodov, V. Baturin, Ia. Profatilova, N. Shipman, W. Wuensch, Nucl. Instrum. Methods Phys. Res. A 908, 318 (2018)
  * (15) S.O. Lebedynskyi, O.O. Pasko, R.I. Kholodov, J. Nano- Electron. Phys. 11, 2 (2019)
  * (16) S. Lebedynskyi, R. Kholodov, Eur. Phys. J. D 73, 190 (2019)
  * (17) M.A. Polyakov, G.N. Fursey, J. Commun. Technol. Electron. 63, 239 (2018)
  * (18) O.E. Raichev, Phys. Rev. B 73, 195328 (2006)
  * (19) Al.A. Zakhidov, A.N. Obraztsov, A.P. Volkov, D.A. Lyashenko, JETP 100, 89 (2005)
  * (20) W. Wuensch, in 8th International Workshop on Mechanisms of Vacuum Arcs (MeVArc 2019), Padova, 2019, Field Emission - Modeling and Simulations, 85
  * (21) G.S. Landsberg, in Elementary textbook on physics: In three volumes. V. 3. Oscillations and waves. Optics. Atomic and Nuclear Physics (Mir Publishers, Moscow, 1989)
  * (22) J. Moghal, J. Kobler, J. Sauer, J. Best, M. Gardener, Andrew A.R. Watt, G. Wakefield, ACS Appl. Mater. Interfaces 4, 854 (2012)
  * (23) H.K. Raut, V.A. Ganesh, A.S. Nair, S. Ramakrishna, Energy Environ. Sci. 4, 3779 (2011)
  * (24) P. Harrison, in Quantum Wells, Wires, and Dots: Theoretical and Computational Physics of Semiconductor Nanostructures (John Wiley & Sons, Ltd., UK, 2005)
  * (25) A.S. Abdalla, M.H. Eisa, R. Alhathlool, O. Aldaghri, Optik 170, 314 (2018)
  * (26) Jihad M. Mohaidat, Kai Shum, R.R. Alfano, Phys. Rev. B, Condensed matter 48, 8809 (1993)
  * (27) Z. Xiao, D. Shi-sen, Z. Chun-Xi, arXiv preprint arXiv: 1210.0970v2, (2012)
  * (28) H.H. Hosack, J. Appl. Phys. 36, 1281 (1965)
  * (29) Y. Zhou, P. Zhang, Phys. Rev. Research 2, 043439 (2020)
  * (30) N.V. Egorov, L.I. Antonova, A.Yu. Antonov, J. Surf. Invest. 6, 906 (2012)
  * (31) L.D. Landau, E.M. Lifshitz, in Quantum Mechanics (Non-relativistic Theory), Vol. 3, 3rd edn. (Butterworth-Heinemann, Oxford, 1991)