# Meta-Learning for Few-Shot Land Cover Classification

Marc Rußwurm  Sherrie Wang  Marco Körner  David Lobell

###### Abstract

The representations of the Earth’s surface vary from one geographic region to
another. For instance, the appearance of urban areas differs between
continents, and seasonality influences the appearance of vegetation. To
capture the diversity within a single category, like as urban or vegetation,
requires a large model capacity and, consequently, large datasets. In this
work, we propose a different perspective and view this diversity as an
inductive transfer learning problem where few data samples from one region
allow a model to adapt to an unseen region. We evaluate the model-agnostic
meta-learning (MAML) algorithm on classification and segmentation tasks using
globally and regionally distributed datasets. We find that few-shot model
adaptation outperforms pre-training with regular gradient descent and fine-
tuning on (1) the Sen12MS dataset and (2) DeepGlobe data when the source
domain and target domain differ. This indicates that model optimization with
meta-learning may benefit tasks in the Earth sciences whose data show a high
degree of diversity from region to region, while traditional gradient-based
supervised learning remains suitable in the absence of a feature or label
shift.

††*equal contribution; $\dagger$ this work was conducted during a research
stay at the Lobell Lab and supported by a fellowship within the IFI programme
of the German Academic Exchange Service (DAAD).

##  1 Introduction

Figure 1: A principle components analysis (PCA) on VGG-16 [26] features of
cropland images from different countries. Representations of the same class
vary geographically; applying models trained on one geography to another would
violate the assumption in traditional supervised learning that train and test
distributions are equal. Model-agnostic meta learning provides a framework for
inductive transfer learning that adapts the model to a new region with few
data samples.

A growing constellation of satellites, combined with cloud computing and deep
learning, offers an objective and scalable way to monitor global issues from
deforestation and wildfires to urban development and road flooding [15, 6, 4,
24]. For many of these prediction problems, the bottleneck to making accurate
and timely predictions has shifted away from satellite imagery availability or
data processing limits and toward a lack of ground truth labels [34, 31, 25].
At the same time, these tasks share characteristics in remotely sensed
imagery—such as ground sampling distance, seasonality, and spectral
characteristics—no matter where on Earth they are taken. This raises the
question of whether prediction in label-scarce regions could be improved if
each model were to benefit from knowledge contained in all the datasets,
rather than solving the same prediction problem across different geographies
or time slices with independent models trained on small disjoint datasets.

The concept of using knowledge gained while solving one problem to aid the
solving of another is known in machine learning as transfer learning [17].
Transferring knowledge between tasks or domains is successful when the
problems are different but related [29]. We argue that the diverse nature of
representations on the Earth’s surface is a prime example of different-but-
related tasks. We illustrate this in Fig. 1 using representations of cropland
from four different countries. Croplands across the world are distinct from
each other, yet they share characteristics. Transfer learning allows models to
both adapt to each distribution individually and share knowledge across
regions: countries like Angola and Mali, for which smaller labeled datasets
are available, could then benefit from larger labeled datasets from countries
like Brazil and Poland.

Thus far, transfer learning on remote sensing data has largely focused on
fine-tuning pre-trained models and performing domain adaptation (Section 2).
In this work, we explore meta-learning, in which models not only learn from
data to perform tasks but _learn how to learn_ to perform tasks through
experiencing tasks on a variety of datasets. In particular, we use model-
agnostic meta-learning (MAML) for the problem of inductive transfer-learning,
where the generalization is induced by a few labeled examples in the target
domain [17]. A schematic of MAML is shown in Fig. 2 and the algorithm is
described in Section 3.2.

Our main contributions are (1) demonstrating that remote sensing tasks across
geographies can be restructured as one meta-learning problem and (2)
evaluating MAML for few-shot classification and segmentation of multi-spectral
and high-resolution remote sensing images; specifically, the well-cited
benchmark datasets Sen12MS and DeepGlobe.

##  2 Related Work

Figure 2: The model-agnostic meta learning (MAML) algorithm [8] finds initial
weights $\theta$ from which a model can adapt to a new geographic region
$\tau$ with few data samples.

Transfer learning can be divided into subcategories depending on the amount of
labeled data available in the source and target domains. Our work is focused
on the scenario in which ample labels exist in the source domain, but few
exist in the target domain. We summarize the related remote sensing
methodology accordingly.

In such a setting, one common transfer learning technique is pre-training a
neural network on ImageNet and fine-tuning [14] it on an application-specific
dataset. For high-resolution remotely sensed imagery, these include airplane
detection [5], high-resolution land cover classification [28], and disaster
mapping [9]. Xie et al. (2016) [32] extended this concept by swapping ImageNet
for the proxy task of night-light prediction that allowed them to estimate
poverty in African regions with a limited number of labeled poverty data
points. These approaches require a significant amount of problem design, such
as the choice of proxy datasets or model and which parameters to fine-tune,
and, thus, usually focus on a limited number of hand-selected tasks.

A second class of methods using deep learning for label-scarce tasks in remote
sensing has focused on developing novel network architectures or loss
functions to make learning more label-efficient. So far, these methods have
focused on optical [11], SAR [21], and hyperspectral image classification
[12]. While they decrease the number of labels required for any optical, SAR,
or hyperspectral task, these methods do not explicitly endeavor to transfer
knowledge from a data-rich geography to a data-poor one.

Non-deep learning methods for domain adaptation were summarized by Tuia et al.
(2016) [29] and include selecting invariant features, adapting data
distributions, and adapting classifiers via semi-supervised learning. For the
most part, such methods generalize only across small regions rather than
worldwide, while sometimes requiring a feature space in which inputs can be
modeled as a mixture of Gaussians or some other predefined distribution.

Lastly, meta-learning is beginning to be explored for remote sensing
applications. Alajaji and Alhichri (2020) [1] describe preliminary results of
MAML on few-shot UC Merced, OPTIMAL-31, and AID RS classification, though
again not with a focus on cross-geography generalization.

##  3 Meta-learning

Meta-learning [22] considers a large number of related tasks
$\tau\in\mathcal{T}=\\{\tau^{(1)},\dots,\tau^{(N)}\\}$ to arrive at a
predictive function that can perform well on unseen tasks $\tau$ after seeing
a few data samples. Even though meta-learning has been a topic in machine
learning for decades [22, 3], it has recently gained popularity for few-shot
problems [30, 27, 19] and has been re-introduced under a “model agnostic”
framework [8] with rapid developments in the field [18, 16, 2].

$p(\mathcal{D})$: distribution over data points;

$\alpha$: step size hyperparameters;

randomly initialize $\phi$;

repeat

sample ${D}\sim p(\mathcal{D})$;

evaluate $\boldsymbol{g}=\nabla\mathcal{L}(f_{\phi},{D})$;

update parameters $\phi\leftarrow\phi-\alpha\boldsymbol{g}$;

until  _convergence_ ;

Algorithm 1 Regular Gradient Descent

$p(\mathcal{T})$: distribution over tasks;

$\alpha,\beta$: step size hyperparameters;

randomly initialize $\theta$;

repeat

sample batch of tasks $\tau\sim p(\mathcal{T})$;

foreach  _$\tau_{i}\in\tau$_ do

initialize $\phi_{i}$ with $\theta$;

sample $\\{{D}_{\text{support}},{D}_{\text{query}}\\}\sim p(\tau_{i})$;

evaluate
$\boldsymbol{g}=\nabla_{\phi_{i}}\mathcal{L}_{\tau_{i}}(f_{\phi_{i}},{D}_{\text{support}})$;

adapt parameters $\phi_{i}\leftarrow\phi_{i}-\alpha\boldsymbol{g}$;

evaluate test loss $\mathcal{L}_{\tau_{i}}(f_{\phi_{i}},{D}_{\text{query}})$ ;

end foreach

update $\theta\leftarrow\theta-\beta\sum_{\tau_{i}\sim
p(\tau)}\nabla_{\theta}\mathcal{L}_{\tau_{i}}(f_{\phi_{i}},{D}_{\text{query}}^{\tau_{i}})$;

until  _convergence_ ;

Algorithm 2 Model-Agnostic Meta-Learning

###  3.1 Terminology and Definitions

Meta-learning introduces a set of terms that may be new to some readers, so we
clarify them in this section.

A task $\tau$ is comprised of a support dataset ${D}_{\text{support}}$ to
adjust the model parameters to the specific task and a query dataset
${D}_{\text{query}}$ to evaluate the performance. Each dataset is comprised of
inputs $\\{\boldsymbol{x}_{1},\boldsymbol{x}_{2},\ldots,\boldsymbol{x}_{m}\\}$
and corresponding labels $\\{y_{1},y_{2},\ldots,y_{m}\\}$ from a data
distribution. A $k$-shot, $n$-way classification task aims to distinguish
between $n$ classes and is trained on $k$ examples per class. Each task is
drawn from a distribution over tasks $\tau\sim p(\mathcal{T})$ to yield a set
of tasks $\\{\tau^{(1)},\tau^{(2)},\dots,\tau^{(N)}\\}$. The meta-learner
_learns how to learn_ by training and evaluating on the meta-training set.
Meta-learning hyperparameters are tuned on the meta-validation set. The meta-
test set measures generalization on new, unseen tasks.

###  3.2 Model-Agnostic Meta Learning (MAML)

Neural network parameters $\phi$ are usually initialized randomly and
optimized iteratively via gradient descent to perform well on a single
dataset, as shown in Algorithm 1. Model-agnostic meta-learning (MAML) extends
gradient descent by optimizing for a model initialization $\theta$ that leads
to good performance on a set of related tasks
$\\{\tau^{(1)},\tau^{(2)},\ldots,\tau^{(N)}\\}$. We contrast the regular
gradient descent with the MAML optimization algorithm in Algorithms 1 and 2.
Meta-training is divided into an inner loop and an outer loop. In the inner
loop, networks initialized with $\theta$ are updated to each task via $t$
steps of gradient descent on $D_{\text{support}}$ of each task. This results
in models with parameters $\phi_{i}$ adapted to each task $\tau^{(i)}$. The
outer loop updates $\theta$ based on the performance of $\phi_{i}$ on
$D_{\text{query}}$ of the meta-training batch. In so doing, MAML requires
second-order gradient calculations. The algorithm looks for a better $\theta$
until convergence, upon which the generalization error is computed on unseen
meta-test tasks.

##  4 Datasets

(a) The 125 regions of the Sen12MS dataset. The 25 meta-test regions have been
selected based on the hold-out set of the Data Fusion Contest 2020 [33]. The
75 meta-train and 25 meta-val have been randomly randomly partitioned.

(b) Example of a Sen12MS 2-way-2-shot task from region 87 and in the summer
season. Ways determines the number of classes per task while shot the number
of samples per class.

Figure 3: The Sen12MS dataset [23] is a public remote sensing dataset of 128
globally distributed regions and four distinct seasons. In this work, we
sample tasks (b) from the dataset that include samples from one region and
season aiming at adapting a deep learning model to one specific region. Figure
4: The DeepGlobe dataset contains high resolution RGB satellite imagery with
land cover labels segmented by humans. To repurpose DeepGlobe for meta-
learning, we (a) split the images into meta-train, meta-val, and meta-test
sets. Then (b) each image was split into 16 sub-images, 8 of which were placed
in the support set and 8 in the query set. Under such a setup, (c) we trained
models on the meta-train set to segment the queries after seeing $k$ shots
from the support.

We evaluate model-agnostic meta-learning on two public remote sensing datasets
that cover optical and radar data at medium and very high resolution.

###  4.1 Sentinel-1/2 Multi-Spectral (Sen12MS) Dataset

The _Sentinel-1/2 Multi-Spectral (Sen12MS)_ [23] dataset is a novel globally
distributed satellite image classification and segmentation dataset. It
contains $280\,662$ Sentinel 2 (optical) and Sentinel 1 (radar) tiles from 125
distinct regions at four different seasons. The optical and radar images were
resampled to $10\text{\,}\mathrm{m}$ ground sampling distance and span
$256\text{\times}256\text{\,}\mathrm{px}$ in height and width. The original
dataset uses tile-overlaps of 50%. For this work, we removed the overlap to
ensure independence of support and query datasets, which yielded $200\,306$
$128\text{\times}128\text{\,}\mathrm{px}$ tiles. We show true color examples
and principal component embeddings on VGG-16 features of four distinct regions
in Fig. 1. Each image tile is accompanied by a land cover label with a
comparatively coarse resolution of $500\text{\,}\mathrm{m}$ from the MODIS
Land Cover product MCD12Q1 V6 upsampled to $10\text{\,}\mathrm{m}$. In this
work, we use the Sen12MS dataset for classification and assign the most common
pixel-level label to the image tile. We use the simplified label-scheme of
International Geosphere Biosphere Programme (IGBP) categories [13] with 10
distinct classes, consistent with the IEEE Data Fusion Contest 2020 [33]. In
Fig. 3(a), the 125 globally distributed regions are shown separated into meta-
train, meta-validation, and meta-test sets. Each region contains between 196
and 850 tiles with a region-specific class distribution. We also show an
overview of all tiles of the region 131 (Marseille) from the summer season
true-color and labels. The individual
$128\text{\times}128\text{\,}\mathrm{px}$ tiles are randomly assigned to the
support or query partition of each region. The objective is to classify each
tile with its most frequent label class. Figure 3(b) illustrates this on an
example of a 2-shot 2-way task. In this case, task-datasets $D_{query}^{\tau}$
and $D_{support}^{\tau}$ contain $k=2$ randomly chosen tile-label pairs of
$n=2$ distinct classes chosen from the available classes in the region.

###  4.2 DeepGlobe Land Cover Segmentation Dataset

The DeepGlobe Challenge [7] was introduced at CVPR 2018 to advance state-of-
the-art satellite image analysis. Here, we used the land cover segmentation
data to explore the use of MAML on high-resolution satellite imagery.

The DeepGlobe land cover segmentation dataset is comprised of very high
resolution ($0.5\text{\,}\mathrm{m}$) DigitalGlobe Vivid+ images of dimension
$2448\text{\times}2448\text{\,}\mathrm{px}$ with three RGB channels. In total,
there are 803 training images, each with human-annotated semantic segmentation
labels covering seven land cover classes: urban, agriculture, rangeland,
forest, water, barren, and unknown. For the competition, 171 validation images
and 172 test images were also provided. However, since they do not have
corresponding labels, we did not include them in the following experiments.
Across the training images, the most common class is agriculture
($58\text{\,}\mathrm{\char 37\relax}$ of pixels), followed by forest
($11\text{\,}\mathrm{\char 37\relax}$), urban ($11\text{\,}\mathrm{\char
37\relax}$), rangeland ($8\text{\,}\mathrm{\char 37\relax}$), barren
($8\text{\,}\mathrm{\char 37\relax}$), water ($3\text{\,}\mathrm{\char
37\relax}$), and unknown ($0.05\text{\,}\mathrm{\char 37\relax}$).

We divided the DeepGlobe training set into three meta-datasets: a meta-
training set on which to train MAML, a meta-validation set on which to tune
MAML hyperparameters, and a meta-test set on which to evaluate generalization
(Fig. 4a). Ideally, we would evaluate whether meta-learned models generalize
better to new geographic regions. However, the DeepGlobe Land Cover dataset
does not tag images with latitude and longitude. In the absence of geographic
information, we split the images in two ways:

  1. 1.

At random, _i.e_. the 803 images were sampled uniformly at random into a
500-image meta-train, a 150-image meta-val, and a 153-image meta-test set.

  2. 2.

Using unsupervised clustering on features extracted from a pre-trained
network. DeepGlobe images were fed into a VGG-16 network pre-trained on
ImageNet, and for each image, a 4096-dimensional vector was extracted from the
first layer in the classifier. We used $k$-means to assign the images into 6
clusters and the 6 clusters were divided at random into the meta-train, meta-
val, and meta-test sets. The resulting datasets contained 454, 166, and 183
images, respectively.

Figure 6(a) visualizes the distributions of image features for the meta-train,
meta-val, and meta-test sets under these two splitting methodologies. The
results across the two splits will illuminate the settings under which MAML
improves upon pre-training and training from scratch.

Each image was further divided into 16 sub-images, each of dimension
$612\text{\times}612\text{\,}\mathrm{px}$ (Fig. 4b). Eight sub-images were
placed in the support set and 8 in the query set. At meta-train time, $k$
shots of $306\text{\times}306\text{\,}\mathrm{px}$ tiles were sampled from the
support set and $q$ queries were sampled from the query set. At meta-test
time, the entire query set was fed into the model as 32 tiles to compute
metrics (Fig. 4c).

Put succinctly, our DeepGlobe experiments explore whether a model can learn to
segment a large region ($1.2\text{\times}1.2\text{\,}\mathrm{km}$) of high
resolution satellite imagery after seeing only a small labeled section
($153\text{\times}153\text{\,}\mathrm{m}$) of it.

##  5 Models

Model-agnostic meta-learning is an optimization algorithm that uses gradient
descent and can be employed for any neural network architecture. In this work,
we chose two popular models for image classification and segmentation.

###  5.1 CNN Classification Model

Following other meta-learning approaches [8, 30], we used a straightforward
CNN architecture for the Sen12MS classification objective. The network
consisted of 7 stacked layers with 64 convolutional
$3\text{\times}3\text{\,}\mathrm{px}$ kernels followed by batch normalization
[10], ReLU activation function, and max-pooling of size 2. The input tensor
$\boldsymbol{X}\in\mathbb{R}^{128\times 128\times 15}$ of joint Sentinel-2 and
Sentinel-1 bands is projected to a 64-dimensional feature vector that maps to
the output vector $\boldsymbol{y}\in\mathbb{R}^{10}$ for each of the classes.

###  5.2 U-Net Segmentation Model

For the DeepGlobe segmentation task, we employed the popular U-Net [20]
architecture. It is a fully-convolutional segmentation model with skip
connections between encoder and decoder. We used four downsampling and
upsampling layers so that the input tensor is projected to a hidden
representation, which is then added to intermediate hidden states from the
encoder (skip connections) while being upsampled and convolved to an output
tensor whereupon each pixel represents one class label.

##  6 Experiments

We experimentally evaluated the classification and segmentation performance of
deep learning models with the same architecture trained with regular gradient
descent (pre-trained) Algorithm 1 and MAML Algorithm 2.

###  6.1 Sen12MS Classification

Figure 5: Classification results on Sen12MS. Regular pre-training with
gradient descent leads to good zero-shot performance, while models trained
with the model-agnostic meta learning algorithms outperform regular
pretraining and the randomly initialized baseline clearly throughout all ten
seen examples from a unseen region.

We assumed that data from the meta-train regions were readily available, but
at most ten image-label pairs per class can be seen from the meta-test
regions. This corresponds to a 4-way 10-shot classification scenario with four
randomly selected classes from one region. It reflects use-case of interest to
this work, where labeled data is available in some regions but not in others.

We trained the classification models with MAML on 4-way 2-shot datasets from
the meta-train regions. We treated each sub-dataset $D_{\text{support}}$ and
$D_{\text{query}}$ as a single batch of $N=k\cdot n=8$ samples.

Baselines. We compared the _MAML-trained_ model with a model that was pre-
trained on all available data from the meta-train regions using regular
gradient descent Algorithm 1. We _pre-trained_ this model with the same 4-way
2-shot batches as MAML but used the combined support and query task-datasets
for training. This resulted in a batch size of 16 image label pairs. Finally,
we also considered the scenario of having no additional data from meta-train
regions. Here, we initialized the model randomly without any prior training,
and train on each task’s support set from scratch; we refer to this baseline
as the _random_ model.

Evaluation. With the three initial CNN model parameterizations, _i.e_. _MAML-
trained_ , _pretrained_ , and _random_ , we evaluated the ability to adapt to
new unseen meta-test regions based on at most ten data samples. For this, we
sampled 100 4-way 10-shot task-datasets from the meta-test regions. We report
performance metrics on $D_{\text{query}}$ on all ten examples per class but
fine-tuned the models on subsets of $D_{\text{support}}$. We also varied the
number of seen samples from $D_{\text{support}}$ incrementally from zero-shot
to ten-shot. Zero-shot represents no fine-tuning and shows the performance
that can be obtained solely based on data from the meta-train regions.
Training on batches of one-shot to ten-shot provides incrementally more data
from the target region to the models. The meta-val regions were used to
determine a suitable step size $\alpha\in\\{0.001,0.0025,\dots,0.5,0.75,1\\}$
and gradient steps on the same data batch $n\in\\{1,2,5,10,50,100\\}$ for
fine-tuning the pre-trained model. We evaluated these hyperparameters via grid
search for each shot independently.

Classification Results. In Fig. 5, we report the accuracy scores for an
increasing number of shots. The zero-shot case, without any adaptation to the
particular meta-test region, shows that the regular pre-trained model
performed best with $55\text{\,}\mathrm{\char 37\relax}$ accuracy and a kappa
score of $0.47$. Without any adaptation on the target-region, MAML predictions
are low in accuracy, which highlights a distinct difference between meta-
learning and pre-training. However, when a single data sample from the meta-
test region is provided (1-shot), the MAML-trained model
($74\text{\,}\mathrm{\char 37\relax}$ accuracy, $0.68$ kappa score)
outperforms the pre-trained model ($59\text{\,}\mathrm{\char 37\relax}$
accuracy, $0.51$ kappa score) by a large margin. The pre-trained model only
shows a comparatively slight increase in accuracy ($54\text{\,}\mathrm{\char
37\relax}$ to $66\text{\,}\mathrm{\char 37\relax}$) throughout all seen
examples while the MAML-trained model scores $80\text{\,}\mathrm{\char
37\relax}$ accuracy and $0.76$ kappa score with all 10 shots.

###  6.2 DeepGlobe Land Cover Segmentation

(a) The DeepGlobe images were split into meta-datasets (left) at random, or
(right) in clusters based on a lower dimensional representation. Tile
representations extracted from a pretrained VGG-16 are plotted along their
first 2 principal components. The label distributions of each meta-dataset are
shown below.

(b) The effect of meta-train support size on segmentation results (mIoU) for
(left) randomly split meta-datasets and (right) clustered split meta-datasets.
Results are shown for 1 meta-test shot.

(c) The effect of number of adaptation shots on segmentation results. Results
are shown for a support size of 8.

Figure 6: Segmentation results on DeepGlobe, with two ways of splitting the
images into meta-datasets.

Our second experiment demonstrates the use of MAML on the DeepGlobe land cover
segmentation dataset. Each DeepGlobe image was considered its own task and we
trained a U-Net via MAML to segment the query set of an image after being
shown $k$ shots from the support set. The experiments were designed to
investigate the effect on the generalization of (1) meta-training label
quantity (number of support and query sub-images), (2) meta-test label
quantity (number of shots), and (3) distributional shift between meta-train
and meta-test sets (random split versus clustered split of meta-datasets). The
number of labeled sub-images in the support and query sets was varied to be
$m\in\left\\{1,2,4,8\right\\}$ and the number of shots used to adapt the U-Net
was in the range $k\in\left\\{1,2,3,4,5\right\\}$. Hyperparameters, such as
the number of epochs to meta-train MAML or train a model from scratch, were
selected using performance on the meta-validation set.

Baselines. Similar to the Sen12MS evaluation, we compared MAML to two
baselines: (1) a U-Net pre-trained on the meta-training set and fine-tuned on
$k$ shots in each meta-test task, and (2) randomly initialized U-Nets trained
independently from scratch on $k$ shots in each meta-test task. To make
comparisons fair, we showed the pre-trained model the same amount of data as
seen by MAML. If MAML was meta-trained on $m$ support tiles and $m$ query
tiles and adapted using $k$ shots, the baseline U-Net was pre-trained on $2m$
tiles per image and fine-tuned on $k$ shots per meta-test tile, and the
randomly initialized model was trained on $k$ shots. The U-Net architecture
was shared among MAML and both baselines.

Evaluation. The performance of all models was evaluated on the query tiles of
an unseen meta-test set of images. The location of the $k$ shots for each
meta-test image was sampled at random from its support set and fixed across
all models for direct comparison. The models were evaluated by means of pixel-
wise accuracy and the mean intersection over union (mIoU) score across the
meta-test queries. For elaboration on the formula used to compute mIoU, please
refer to the DeepGlobe publication [7].

(a) Random meta-dataset splits

(b) Clustered meta-dataset splits

Figure 7: Example segmentation predictions by MAML, a pre-trained U-Net, and
U-Nets trained from scratch.

Random Meta-Dataset Split Results. When the meta-datasets were randomly split,
the pre-trained model performed better than MAML and the randomly initialized
model. This was especially true at smaller meta-training set sizes (Fig.
6(b)). In other words, MAML requires a large set of meta-training tasks in
order to perform well on new tasks. As the number of shots seen by the meta-
learner increases, MAML catches up to the pre-trained model (Fig. 6(c)). In
these experiments, we did not observe fine-tuning of the pre-trained model to
improve its performance.

Figure 7(a) visualizes the predictions of MAML and the baselines on 1-shot
learning for two images: one where MAML performs well and one where it fails.
MAML appears heavily influenced by the choice of the 1 shot, while the pre-
trained model is biased toward predicting agriculture (the most common class).
The model trained from scratch is even more heavily influenced by the choice
of the 1 shot, as this is the only data it sees during training.

The success of pre-training can be attributed to the complete overlap of meta-
train and meta-test distributions, seen in Fig. 6(a). In the setting where
$p(X,y)$ are identical in the source domain and target domain, a model trained
on the source domain transfers perfectly to the target domain. These results
also expose MAML’s weaknesses when meta-train size is small: it is not able to
retain information about land cover types as effectively as a straightforward
supervised model.

Clustered Meta-Dataset Split Results. When the meta-datasets were split along
clusters, the meta-train and meta-test distributions overlapped less (Fig.
6(a)) but could still be considered to arise from the same data-generating
distribution. Whereas the meta-train set contains mostly agriculture pixels,
the meta-test set contains predominantly forest. Figures 6(b) and 6(c) show,
first and foremost, that this meta-test set is more difficult than the
randomly split meta-test set for all three models. However, MAML is able to
adapt to this distributional shift more successfully than the pre-trained
model. Example segmentations shown in Fig. 7(b) reveal that MAML’s flexibility
to adaptation can again be both helpful and detrimental: helpful when the 1
shot is representative of the image, but detrimental when it is not. We see
that the pre-trained model carries its bias toward agriculture into its meta-
test set predictions, whereas MAML does not appear to retain a strong enough
prior to recognize agriculture without being provided a shot containing that
class.

###  6.3 Visualization of Model Adaptation

(a) Adaptation of the MAML-trained CNN model to episodes from different
regions.

(b) 1D Loss surface on multiple query-task samples along the gradient of one
support-task

Figure 8: The adapted weights $\phi_{\tau}$ for task $\tau$ vary from region
to region (a). The loss surface along the direction of initial weights
$\theta$ to $\phi_{\tau}$ (b) is more convex and allows larger gradient step
sizes for model-agnostic meta learning compared to regular pretraining.

In the introduction and Fig. 1, we showed the regional diversity of
representations on the Earth’s surface using PCA on pre-trained VGG-16 image
features. In Section 3 and Fig. 2, we assumed that a neural network would
achieve optimal performance with a different set of weights $\phi^{\ast}$ for
each geographic region. In this experiment, we empirically confirmed this
hypothesis with two evaluations on meta-test regions of the Sen12MS dataset.
In Section 6.3.1, we visualize the adapted MAML weights for two distinct
geographies. Then in Section 6.3.2 we compare the loss surfaces of a MAML-
trained and a pre-trained model along one adaptation trajectory. The two
evaluations are meant to provide the reader with some intuition of what MAML
is doing in different regions and how this differs from pre-training.

####  6.3.1 Region-wise Adaptation

We studied the adaptation of MAML-model parameters $\theta$ trained on 2-shot
4-way tasks of Section 6.1. We sampled 1000 1-shot 4-way classification task-
datasets from the meta-test regions for the four most common classes (forests,
grassland, savanna, urban) and split these into a support and query partition
at ratio of 4:1. For each training task, we evaluated the gradient and adapted
the model using gradient descent with step size $0.75$ to new parameters for
each task $\phi^{\tau}$. We visualized this adaptation by flattening all model
parameters to a $231\,818$-dimensional vector and used PCA to map the
parameters to the first two principal components. We colored this embedding by
region and drew lines from the initial weights $\theta$ to the adapted task-
specific weights $\phi_{\tau}$ in Fig. 8. The adapted model-weights differ
characteristically between regions in embedding space, as can be seen in the
examples of Poland and South Sudan. This empirically shows that a separate set
of model parameters is optimal for these two different regions.

####  6.3.2 Loss Surface along Support Gradient

Next, we evaluated the query loss along one line from initial parameters
$\theta$ to task-adapted parameters $\phi$ with respect to four query tasks
with the MAML-trained model and the regularly pre-trained model. Here, we
selected one support-task and four query-tasks from the same region and
season. The gradient $\boldsymbol{g}$ was evaluated with respect to the
support-task and obtained different model weights $\phi_{\alpha}$ along the
gradient by $\phi_{\alpha}=\theta+\alpha\boldsymbol{g}$ with different step
sizes $\alpha_{\text{MAML}}\in[0,1]$, $\alpha_{\text{pre}}\in[0,0.15]$.
Initially, we evaluated larger stepsizes, but chose to show these intervals
which are proportional to the optimal step sizes for MAML and pretrained
model. We calculated the query loss using the model $f_{\phi_{\alpha}}$ for
each of the four test-tasks at different step sizes $\alpha$. This draws a
one-dimensional slice of the loss-surface along the gradient direction
determined by the support-task. In Fig. 8(b), we show this loss surface for
the MAML-trained model and the pre-trained classification model. Without
adaptation, at $\alpha=0$, the MAML-trained model evaluated a high loss
compared to the pre-trained model. This is consistent with the comparatively
poor zero-shot results from Fig. 5. With increasing step size, however, we
observe that the MAML loss decreased consistently while the pre-trained loss
remained on similar level or increased for larger step sizes. The MAML-trained
model achieved low loss in a large range of step sizes from $0.1$ to $1$
through all query-tasks while a narrow range of step sizes between $0$ and
$0.05$ lead to better accuracies on some tasks from the pre-trained model
initialization. In general, the loss surface of the MAML-trained model
followed a convex curve for all of the test examples while the loss surface
showed local minima from the pre-trained model initialization. This experiment
illustrates the difference between meta-learning and regular pre-training that
lead to very different model parameters. The loss surface of a meta-learned
model was more convex in the gradient direction of a novel task. Overall, the
MAML-trained model benefited, _i.e_. achieved lower test-loss, when being
adapted to samples of a new region regardless of the step size. For the pre-
trained model, it would have been beneficial not to adapt to one specific
region for query tasks 2 and 4.

##  7 Discussion and Conclusion

In this work, we evaluated the model-agnostic meta-learning (MAML) algorithm
for few-shot problems in land cover classification to adapt deep learning
models to individual regions with few data examples. Existing models use
regular gradient descent to pre-train a model on a large body of data and use
this pre-trained model as an initialization for datasets with fewer examples.
We compared these two approaches on land cover classification on the Sen12MS
dataset of optical and radar images from globally distributed regions and the
DeepGlobe dataset with very high-resolution imagery in few regions. The
results on Sen12MS in Section 6.1 demonstrate that MAML-optimization can
outperform regular gradient descent and pre-training of models when the
dataset includes a distinct regional diversity. The DeepGlobe results in
Section 6.2 illustrate the advantage MAML offers when the source domain
differs from the target domain in transfer learning but also highlight MAML’s
weaknesses in retaining prior knowledge and under-performing in ideal
(identical source and target domain) settings. In Section 6.3, we evaluated
the loss surfaces for pre-trained and MAML-trained models and showed that the
loss surface was more convex for MAML-trained models when adapting to new
unseen data.

We believe that the meta-learning framework can lead deep learning in Earth
observation to a new direction: away from finding incrementally better model
architectures for specific use-cases and toward unifying strategies that more
closely reflect the reality on the Earth’s surface. Much work remains to be
done to improve MAML performance by retaining stronger priors on land cover
classes, as well as to explore other meta-learning paradigms (_e.g_.
prototypical networks).

## References

  * [1] D. Alajaji and H. Alhichri.  Few shot scene classification in remote sensing using meta-agnostic machine.  In 2020 6th Conference on Data Science and Machine Learning Applications (CDMA), pages 77–80, 2020.
  * [2] Antreas Antoniou, Harrison Edwards, and Amos Storkey.  How to train your MAML.  arXiv preprint arXiv:1810.09502, 2018.
  * [3] Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier.  Learning a synaptic learning rule.  Université de Montréal, Département d’informatique et de recherche …, 1990.
  * [4] Basudeb Bhatta.  Analysis of urban growth and sprawl from remote sensing data.  Springer Science & Business Media, 2010.
  * [5] Zhong Chen, Ting Zhang, and Chao Ouyang.  End-to-end airplane detection using transfer learning in remote sensing images.  Remote Sensing, 10(1):139, 2018.
  * [6] Emilio Chuvieco.  Remote sensing of large wildfires: in the European Mediterranean Basin.  Springer Science & Business Media, 2012.
  * [7] Ilke Demir, Krzysztof Koperski, David Lindenbaum, Guan Pang, Jing Huang, Saikat Basu, Forest Hughes, Devis Tuia, and Ramesh Raska.  DeepGlobe 2018: A challenge to parse the earth through satellite images.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 172–17209. IEEE, 2018.
  * [8] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In International Conference on Machine Learning (ICML), pages 1126–1135, 2017.
  * [9] Ananya Gupta, Elisabeth Welburn, Simon Watson, and Hujun Yin.  Post disaster mapping with semantic change detection in satellite imagery.  In IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2019.
  * [10] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  arXiv preprint arXiv:1502.03167, 2015.
  * [11] Neal Jean, Sherrie Wang, Anshul Samar, George Azzari, David Lobell, and Stefano Ermon.  Tile2vec: Unsupervised representation learning for spatially distributed data.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3967–3974, 2019.
  * [12] B. Liu, X. Yu, A. Yu, P. Zhang, G. Wan, and R. Wang.  Deep few-shot learning for hyperspectral image classification.  IEEE Transactions on Geoscience and Remote Sensing, 57(4):2290–2304, 2019.
  * [13] Thomas R Loveland and AS Belward.  The IGBP-DIS global 1km land cover data set, discover: first results.  International Journal of Remote Sensing, 18(15):3289–3295, 1997\.
  * [14] Dimitrios Marmanis, Mihai Datcu, Thomas Esch, and Uwe Stilla.  Deep learning earth observation classification using ImageNet pretrained networks.  IEEE Geoscience and Remote Sensing Letters, 13(1):105–109, 2015\.
  * [15] Stephen D McCracken, Eduardo S Brondizio, Donald Nelson, Emilio F Moran, Andrea D Siqueira, and Carlos Rodriguez-Pedraza.  Remote sensing and GIS at farm property level: Demography and deforestation in the Brazilian Amazon.  Photogrammetric Engineering and Remote Sensing, 65:1311–1320, 1999\.
  * [16] Alex Nichol, Joshua Achiam, and John Schulman.  On first-order meta-learning algorithms.  arXiv preprint arXiv:1803.02999, 2018.
  * [17] Sinno Jialin Pan and Qiang Yang.  A survey on transfer learning.  IEEE Transactions on knowledge and data engineering, 22(10):1345–1359, 2009.
  * [18] Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine.  Meta-learning with implicit gradients.  In Advances in Neural Information Processing Systems, pages 113–124, 2019.
  * [19] Sachin Ravi and Hugo Larochelle.  Optimization as a model for few-shot learning.  In International Conference on Learning Representations (ICLR), 2017\.
  * [20] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 234–241. Springer, 2015.
  * [21] Mohammad Rostami, Soheil Kolouri, Eric Eaton, and Kyungnam Kim.  Deep transfer learning for few-shot SAR image classification.  Remote Sensing, 11(11), 2019.
  * [22] Jürgen Schmidhuber.  Evolutionary principles in self-referential learning, or on learning how to learn: The meta-meta-… hook.  Diplomarbeit, Technische Universität München, München, 1987.
  * [23] Michael Schmitt, Lloyd Haydn Hughes, Chunping Qiu, and Xiao Xiang Zhu.  SEN12MS–A curated dataset of georeferenced multi-spectral Sentinel-1/2 imagery for deep learning and data fusion.  arXiv preprint arXiv:1906.07789, 2019.
  * [24] E Schnebele, N Waters, et al.  Road assessment after flood events using non-authoritative data.  Natural Hazards and Earth System Sciences, 14(4):1007, 2014.
  * [25] Grant J Scott, Matthew R England, William A Starms, Richard A Marcum, and Curt H Davis.  Training deep convolutional neural networks for land–cover classification of high-resolution imagery.  IEEE Geoscience and Remote Sensing Letters, 14(4):549–553, 2017\.
  * [26] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [27] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  In Advances in Neural Information Processing Systems, pages 4077–4087, 2017.
  * [28] Xin-Yi Tong, Gui-Song Xia, Qikai Lu, Huanfeng Shen, Shengyang Li, Shucheng You, and Liangpei Zhang.  Land-cover classification with high-resolution remote sensing images using transferable deep models.  Remote Sensing of Environment, 237:111322, 2020.
  * [29] Devis Tuia, Claudio Persello, and Lorenzo Bruzzone.  Domain adaptation for the classification of remote sensing data: An overview of recent advances.  IEEE Geoscience and Remote Sensing Magazine, 4(2):41–57, 2016.
  * [30] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al.  Matching networks for one shot learning.  In Advances in neural information processing systems, pages 3630–3638, 2016.
  * [31] Sherrie Wang, William Chen, Sang Michael Xie, George Azzari, and David B Lobell.  Weakly supervised deep learning for segmentation of remote sensing imagery.  Remote Sensing, 12(2):207, 2020.
  * [32] Michael Xie, Neal Jean, Marshall Burke, David Lobell, and Stefano Ermon.  Transfer learning from deep features for remote sensing and poverty mapping.  In AAAI Conference on Artificial Intelligence, AAAI’16, pages 3929–3935. AAAI Press, 2016.
  * [33] Naoto Yokoya, Pedram Ghamisi, Ronny Hänsch, and Michael Schmitt.  2020 IEEE GRSS Data Fusion Contest: Global land cover mapping with weak supervision.  IEEE Geosci. Remote Sens. Mag., 2020.  in press.
  * [34] Xiao Xiang Zhu, Devis Tuia, Lichao Mou, Gui-Song Xia, Liangpei Zhang, Feng Xu, and Friedrich Fraundorfer.  Deep learning in remote sensing: A comprehensive review and list of resources.  IEEE Geoscience and Remote Sensing Magazine, 5(4):8–36, 2017.


# Deep Learning for Visual Navigation of Underwater Robots

MD Sunbeam
Department of Aerospace Engineering
Texas A&M University
College Station, TX, United States
<EMAIL_ADDRESS>
###### Abstract

This paper aims to briefly survey deep learning methods for visual navigation
of underwater robotics. The scope of this paper includes the visual perception
of underwater robotics with deep learning methods, the available visual
underwater datasets, imitation learning, and reinforcement learning methods
for navigation. Additionally, relevant works will be categorized under the
imitation learning or deep learning paradigm for underwater robots for clarity
of the training methodologies in the current landscape. Literature that uses
deep learning algorithms to process non-visual data for underwater navigation
will not be considered, except as contrasting examples.

###### Index Terms:

deep learning, underwater, imitation learning, AUV, visual navigation,
reinforcement learning

##  I Introduction

This paper will cover deep learning for underwater robotics. It is divided
into the perception of underwater robotics, underwater visual datasets,
imitation learning, and reinforcement learning for underwater environments.

Visual navigation in underwater robotics is more challenging than its land or
air counterparts because of the limitations of perception, particularly
computer vision. When light travels through the water, it is absorbed and
scattered, resulting in a wavelength-dependent attenuation that disturbs the
standard ways of handling vision [1]. An example of this problem is capturing
images of an object with different brightness and colors based on distance.

Additionally, environmental changes are more subtle underwater. Features may
change from drifting sand and turbulent current. Detecting and modeling these
changes is difficult, especially because of its stochastic nature and the
complex fluid equations that govern the dynamics. The hydrodynamics involved
is nonlinear and time-varying, and the robot is usually underactuated with not
all the states reachable [2]. Deep learning methods for visual navigation are
explored, because they offer a learning-based way that may handle these
challenges through a data-driven approach.

Because autonomous underwater vehicles (AUVs) are less common due to greater
hardware considerations, underwater datasets are rare and more difficult to
collect. Since deep learning is a data-driven method, it is imperative to have
public datasets to benchmark models.

The two main deep learning paradigms for control of an AUV through visual
navigation is imitation and reinforcement learning. Imitation learning is used
for underwater robotics for visual navigation, often mapping RGB images to
control commands like roll, pitch, yaw, and throttle. The problem formulation
is a supervised learning task where the objective is to learn a policy that
imitates the trajectories of a human or robot demonstrator. Two major
limitations of imitation learning algorithms are compounding error between
differences of expert and agent trajectory as well as the idea that a policy
derived from such a method will not outperform the expert demonstrator [4].

Reinforcement learning is the other deep learning paradigm, using a non-
differentiable objective function through rewards to train neural networks for
a desired task, in this case being underwater visual navigation. Reinforcement
learning differs from imitation learning in that an initial dataset is not
needed, and an environment must be explored to generate the data to be trained
on. Though reinforcement learning is less sample efficient than imitation
learning, the lack of dependence on an expert demonstrator allows it to yield
policies that are potentially superhuman [15].

The paper’s contribution are as follows:

  * •

Identify the works that focus on utilizing deep learning for perception in
underwater visual navigation algorithms.

  * •

Discuss the publicly available underwater datasets.

  * •

Categorize the deep learning methods used in AUVs under imitation learning or
reinforcement learning.

##  II Perception

Due to the challenges of visual perception in underwater robotics, [10] offers
a deep learning method to improve loop closure and detection for a Visual
Graph SLAM algorithm for underwater navigation. Siamese networks, a neural
network architecture that has enjoyed great success in similarity learning,
was leveraged to detect similar visual features and places.

Another work for underwater loop detection for use in a visual SLAM algorithm
is [11], but it sets itself apart from [10] by training a network through an
unsupervised learning method. Whereas Siamese networks are trained in a
supervised manner with image pairs through contrastive loss, the unsupervised
learning method detects similar images and loop closure through the clustering
of image descriptors.

Figure 1: Redrawn network architecture diagram of the Siamese network used to
detect loop-closure between two underwater images in [10].

Finally, [9] also deals with loop closure, but the role of the neural network
is more limited in scope. The neural network only selects loop candidates,
which are sent to an image matcher and geometric consistency verifier to
output the final loop detected images.

##  III Underwater Datasets

There are far fewer underwater robot datasets as compared to autonomous ground
vehicle or autonomous aerial vehicle datasets since collecting data underwater
remains difficult due to hardware constraints and needing a relatively
uncommon environment.

For deep learning methods, having a wide variety of datasets is paramount
because it allows the research community to tackle the same problems, often on
a vetted or curated dataset. In computer vision, deep learning innovation was
primarily accelerated by the existence of ImageNet [19]. In natural language
processing, the large corpus of textual data in the internet has fueled the
advances of large language models. For robotics, ground vehicle datasets like
CARLA [20] or KITTI [21] and aerial vehicle datasets like Mid-Air [22] have
paved the way for development of many deep learning based visual navigation
algorithms.

As such, [6] offers a visual dataset for AUVs navigating near the seabed. The
images were collected from three different depths: one at a few meters, the
second 270 meters, and the last at 380 meters. The purpose of this data is to
provide image and inertia-pressure samples for use in simultaneous
localization and mapping algorithms.

A common theme is leveraging generative adversarial networks and generative
models to augment real underwater datasets with generated synthetic images.
This is done in [7], [17], and [18]. A publicly available dataset, called
EUVP, of images that were taken during ocean experiments for exploration and
human-robot collaboration is also presented in [18].

Figure 2: These are some sample training images selected from the EUVP dataset
in [18].

##  IV Imitation Learning

For [3], a domain expert diver collected data to generate ”good” and ”bad”
navigation scenarios, which were later annotated with labels of yaw and pitch
to accomplish the task of exploring a coral reef while avoiding obstacles. A
convolutional neural network was trained to map the images to the control
commands. They evaluated their models through the percentage of coral covered
in the navigation task in certain reef area.

Similar to [3], [4] also uses a behavior cloning model to map RGB images to
yaw and pitch command. In this case, the task was to explore a shipwreck, and
a convolutional neural network was used. For this work, the neural network was
trained on a mixture of simulation and real-world data. The model was
evaluated through the training-validation-test split. Different test
accuracies are given for the real-world only dataset, simulation only dataset,
and mixed dataset. The primary difference between the two papers were the
difference in task and the nature of the demonstration data.

One work that is different is [5], which uses goal-conditioned imitation
learning for underwater visual navigation. The behavior cloning model learns
safe, reactive behavior for difficult terrain, and is conditioned to navigate
based on waypoints. The neural network architecture is a convolutional neural
network, much like those in [3] and [4]. Similarly, the model was evaluated
through a test set and evaluated in real-life qualitatively.

##  V Reinforcement Learning

In [8], a soft-actor critic deep reinforcement learning algorithm was used to
train a neural network. The AUV was a soft robot, and the task was to swim in
a straight line in disturbed water. A camera was used to collect RGB images,
which served as the observation space for the neural network. Before the deep
reinforcement learning algorithm was rolled out underwater, a model was
trained in a MuJoCo simulation.

Figure 3: Redrawn system diagram for the deep reinforcement learning
controller in [8].

For [15], a combination of imitation learning and reinforcement learning is
used. First, a generative adversarial imitation learning algorithm is used to
overcome the cold start problem of the initial neural network training to
learn a policy. Then, a reward function is designed and trained with proximal
policy optimization and soft-actor critic. The results are compared in a Unity
simulation. This work uses a light sensor, which is different from the other
imitation learning and reinforcement learning work as most use a RGB camera
for their visual sensor.

##  VI Imitation and Reinforcement Learning Categorization

The works where deep learning methods were used for only visual navigation are
considered and categorized. Works like [12], [14], and [16] use deep or
reinforcement learning methodologies, but fail to incorporate a visual sensor.
While there is much more work on using neural networks on inertial, pressure,
and position data from non-visual sensors, that is outside the scope of this
survey. Furthermore, the specific neural network architectures used in
underwater robot navigation will not be discussed in depth like in [13], but
are useful for understanding their applications in perception and control.

Deep Learning Paradigm Categorization
---
Reference |  Imitation Learning |  Reinforcement Learning
[3] |  ✓ |  ✗
[4] |  ✓ |  ✗
[5] |  ✓ |  ✗
[8] |  ✗ |  ✓
[15] |  ✓ |  ✓

There are far more works using imitation learning algorithms than
reinforcement learning. This can be explained by the idea that the simplest
class of imitation learning algorithms, behavior cloning, is good at dealing
with high dimensional inputs because convolutional layers can be used as a
feature extractor to reduce the dimension of the images. The behavior cloning
problem formulation is also easier, where the objective is imitating an expert
trajectory through a supervised learning setup.

The lack of many deep reinforcement learning works for visual navigation of
AUVs can be explained by the added difficulty of devising a proper reward
function on top of dealing with a noisy observation space. Moreover, the
reinforcement learning agent must do exploration before it can exploit through
an adequate policy, which introduces an overhead of creating or using a
simulation for the underwater robot as real-life exploration can be expensive
and dangerous.

##  VII Conclusion

In this paper, we have covered the deep learning methods used to improve
perception under water, which go into improving the visual navigation
algorithms that use those modules. Most of the work centers around in
detecting loop closures through similarity learning of images through Siamese
networks or using other neural network architectures to detect and select loop
closure candidates like in [9], [10], and [11]. The improvement of loop
closure in visual perception through deep learning improves the visual SLAM
algorithms that leverage them for navigation.

Next, we discussed the underwater datasets publicly available like the ones in
[6] and [7]. While there are some standard public datasets, there are far too
few and even less for a robotics application. This can be attributed to the
difficulty of creating such datasets due to hardware and environmental
constraints. A common trend is to leverage deep learning methods to augment
the available datasets, either through improving the quality of the images or
generating synthetic images like in [17] and [18]. However, this type of
augmentation seems more a stopgap rather than a permanent solution to the
scarcity of underwater robotics data.

Finally, we categorize the works that use deep learning methods to control an
AUV under the imitation (like in [3], [4], and [5]) or reinforcement learning
(like in [8] and [15]) paradigm. A pattern that becomes immediately obvious is
that there exists far more imitation learning works than deep reinforcement
learning works in the domain of visual navigation in underwater robotics.
Another trend is that there aren’t that many works where some visual sensors
are used in the context of imitation or reinforcement learning. This can
perhaps be explained by the hardware constraints in AUVs, where limited
computation and batteries onboard may make RGB cameras too energy-intensive.

From this survey, some gaps in the field of deep learning for visual
navigation of underwater robots become apparent. There needs to be more
underwater datasets collected and made publicly available, especially with
robotics applications. Even though deep learning methods are used to augment
existing underwater datasets, the best way to accelerate the pace and
scalability of neural networks is through more data. Like in computer vision
and natural language processing, curated datasets provide a standard for
research and competition, both of which push innovation within the field.

Another gap is the lack of reinforcement learning work for visual navigation.
The preference for imitation learning is reasonable given the harder
formulation of the reinforcement learning problem, but focus in this direction
may address the limitations of imitation learning, like compounding errors and
the learned policy never being better than the demonstrator. Because
simulation is an important component of reinforcement learning in the
exploration stage, attention in this area may lead to better simulators that
can address the lack of underwater datasets through synthetic data.

Overall, the field of deep learning for visual navigation of underwater robots
provides natural challenges crucial for the larger study of AI robotics. One
such challenge is the fundamental problem of acting under noisy or misleading
visual perception data. This offers opportunities for planning or navigation
under uncertainty. Moreover, breakthroughs in the domain of deep learning for
underwater visual navigation may be applied to the broader field of learning
from sparse environments. Underwater environments are often characterized by
sparsely distributed features. In imitation learning and deep reinforcement
learning, environments with few features may make it difficult for the neural
network to learn a useful policy for the desired task. Because of these
reasons, it is crucial to place more research emphasis on the problem of
visual navigation in underwater environments.

## References

  * [1] K. Köser and U. Frese, “Challenges in underwater visual navigation and slam,” AI Technology for Underwater Robots, pp. 125–135, 2019.
  * [2] L. Christensen, J. de Gea Fern´andez, M. Hildebrandt, C. E. Koch, and B. Wehbe, “Recent advances in AI for navigation and control of Underwater Robots,” Current Robotics Reports, vol. 3, no. 4, pp. 165–175, 2022.
  * [3] T. Manderson, J. C. Higuera, R. Cheng, and G. Dudek, “Vision-based autonomous underwater swimming in dense coral for combined collision avoidance and Target Selection,” 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2018.
  * [4] N. Karapetyan, J. V. Johnson, and I. Rekleitis, “Human diver-inspired visual navigation: Towards coverage path planning of shipwrecks,” Marine Technology Society Journal, vol. 55, no. 4, pp. 24–32, 2021.
  * [5] T. Manderson, J. Camilo Gamboa Higuera, S. Wapnick, J.-F. Tremblay, F. Shkurti, D. Meger, and G. Dudek, “Vision-based goal-conditioned policies for underwater navigation in the presence of obstacles,” Robotics: Science and Systems XVI, 2020.
  * [6] M. Ferrera, V. Creuze, J. Moras, and P. Trouv´e-Peloux, “AQUALOC: An underwater dataset for visual–inertial–pressure localization,” The International Journal of Robotics Research, vol. 38, no. 14, pp. 1549–1559, 2019.
  * [7] I. Polymenis, M. Haroutunian, R. Norman, and D. Trodden, “Virtual underwater datasets for autonomous inspections,” Journal of Marine Science and Engineering, vol. 10, no. 9, p. 1289, 2022.
  * [8] G. Li, J. Shintake, and M. Hayashibe, “Deep reinforcement learning framework for underwater locomotion of Soft Robot,” 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021.
  * [9] A. Burguera, F. Bonin-Font, E. G. Font, and A. M. Torres, “Combining deep learning and robust estimation for outlier-resilient underwater visual graph slam,” Journal of Marine Science and Engineering, vol. 10, no. 4, p. 511, 2022.
  * [10] A. Burguera, “Robust underwater visual graph slam using a Siamese neural network and robust image matching,” Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, 2022.
  * [11] A. Burguera and F. Bonin-Font, “An unsupervised neural network for loop detection in Underwater Visual Slam,” Journal of Intelligent & Robotic Systems, vol. 100, no. 3-4, pp. 1157–1177, 2020.
  * [12] X. Mu, B. He, X. Zhang, Y. Song, Y. Shen, and C. Feng, “End-to-end navigation for autonomous underwater vehicle with hybrid recurrent neural networks,” Ocean Engineering, vol. 194, p. 106602, 2019.
  * [13] J. Qin, M. Li, D. Li, J. Zhong, and K. Yang, “A survey on visual navigation and positioning for autonomous uuvs,” Remote Sensing, vol. 14, no. 15, p. 3794, 2022.
  * [14] N. Wang, Y. Wang, Y. Zhao, Y. Wang, and Z. Li, “Sim-to-real: Mapless navigation for usvs using Deep Reinforcement Learning,” Journal of Marine Science and Engineering, vol. 10, no. 7, p. 895, 2022.
  * [15] Y. Mao, F. Gao, Q. Zhang, and Z. Yang, “An AUV target-tracking method combining imitation learning and deep reinforcement learning,” Journal of Marine Science and Engineering, vol. 10, no. 3, p. 383, 2022.
  * [16] I. B. Saksvik, A. Alcocer, and V. Hassani, “A deep learning approach to dead-reckoning navigation for autonomous underwater vehicles with limited sensor payloads,” OCEANS 2021: San Diego – Porto, 2021.
  * [17] C. Fabbri, M. J. Islam, and J. Sattar, “Enhancing underwater imagery using generative adversarial networks,” 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018.
  * [18] M. J. Islam, Y. Xia, and J. Sattar, “Fast underwater image enhancement for improved visual perception,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 3227–3234, 2020.
  * [19] J. Deng, W. Dong, R. Socher, L. -J. Li, Kai Li and Li Fei-Fei, ”ImageNet: A large-scale hierarchical image database,” 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 2009, pp. 248-255, doi: 10.1109/CVPR.2009.5206848.
  * [20] Dosovitskiy, A., Ros, G., Codevilla, F., López, A. M. & Koltun, V. (2017). CARLA: An Open Urban Driving Simulator. CoRL (p./pp. 1-16), : PMLR.
  * [21] Geiger A, Lenz P, Stiller C, Urtasun R. Vision meets robotics: The KITTI dataset. The International Journal of Robotics Research. 2013;32(11):1231-1237. doi:10.1177/0278364913491297
  * [22] M. Fonder and M. Van Droogenbroeck, ”Mid-Air: A Multi-Modal Dataset for Extremely Low Altitude Drone Flights,” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Long Beach, CA, USA, 2019, pp. 553-562, doi: 10.1109/CVPRW.2019.00081.


# Two-Photon Interference from Silicon-Vacancy Centers in Remote Nanodiamonds

R. Waltrich  Institute for Quantum Optics, Ulm University, 89081 Ulm, Germany
M. Klotz  Institute for Quantum Optics, Ulm University, 89081 Ulm, Germany  V.
N. Agafonov  GREMAN, UMR 7347 CNRS, INSA-CVL, Tours University, 37200 TOURS,
France  A. Kubanek  Institute for Quantum Optics, Ulm University, 89081 Ulm,
Germany

###### Abstract

The generation of indistinguishable photons is a key requirement for solid-
state quantum emitters as a viable source for applications in quantum
technologies. Restricting the dimensions of the solid-state host to a size
well below the wavelength of light emitted by a defect-center enables
efficient external optical coupling, for example for hybrid integration into
photonic devices. However, stringent restrictions on the host dimensions
result in severe limitations on the spectral properties reducing the
indistinguishability of emitted photons. Here, we demonstrate two-photon
interference from two negatively-charged Silicon-Vacancy centers located in
remote nanodiamonds. The Hong-Ou-Mandel interference efficiency reaches 61%
with a coalescence time window of 0.35 ns. We furthermore show a high yield of
pairs of Silicon-Vacancy centers with indistinguishable optical transitions.
Therefore, our work opens new paths in hybrid quantum technology based on
indistinguishable single-photon emitters in nanodiamonds.

††preprint: APS/123-QED

In the emerging field of quantum technologies the distribution of entanglement
is a key ingredient, for example to establish long-distance quantum state
transfer and quantum networks [1]. One possible source of distributed
entanglement generation is two-photon interference (TPI), commonly known as
Hong-Ou-Mandel (HOM) interference [2]. A prerequisite are single photon
sources that produce indistinguishable photons. Two-photon interference has
been demonstrated with various sources of single photons, for example atomic
vapors [3], quantum dots [4], molecules [5], coupled atom-cavity systems [6]
and negatively-charged Nitrogen-Vacancy (NV-) centers in bulk-diamond [7, 8].
Group IV color centers in diamond and, in particular, the negatively-charged
Silicon-Vacancy center (SiV-) are of great interest for the generation of
indistinguishable photons, due to intrinsic spectral stability and narrow
inhomogeneous line-distribution shown for bulk diamond [9, 10]. In recent
years, there is an increasing effort to restrict the dimensions of the diamond
host to very small size well below the optical wavelength. Such nanodiamonds
(NDs) give rise to a large variety of new applications in the realm of hybrid
quantum systems. Individually optimized [11, 12] and large-scale [13, 14]
hybrid quantum photonic circuits [15, 16, 17] are constructed by means of
nanomanipulation [18] and advanced fabrication techniques, respectively.
Initially, the spectral properties of color centers in NDs were inferior
compared to bulk diamond blocking further use in quantum optics applications.
This deficiency was resolved over the past years by improved ND-production,
sample preparation and control techniques [19].
In this letter, we demonstrate two-photon interference from SiV- in remote
NDs. Together with recently demonstrated access to electron spin [20] this
work marks a further step towards applied hybrid quantum technology, such as
the realization of scalable quantum networks, based on SiV- in NDs.

(a)(b)(c) Figure 1:  a) Molecular structure of the SiV-. A silicon atom (Si)
is located between two adjacent carbon vacancies (V) along the [111] axis of
the diamonds crystal structure. b) Electronic level structure of the SiV- with
four optical transitions A, B, C, D arising from the ground and excited state
doublets ($\Delta_{GS}$, $\Delta_{ES}$) due to spin-orbit interaction. c)
Schematic of the two-photon interference experiment. Two identical photons
from two separate SiV- enter a beam splitter from two different input ports.
Constructive interference leads to maximal coalescence at the output ports.

The SiV- is a point defect in the lattice of a diamond crystal where a silicon
atom is located between two adjacent carbon vacancies as depicted in FIG. 1
a). At cryogenic temperatures four optically active transitions, resulting
from spin-orbit coupling, can be observed. We refer to them as transitions A,
B, C, D, as depicted in FIG. 1 b). To show two-photon interference, we excited
two SiV- in two remote NDs, separated by approximately 95 $\mu$m, off-
resonantly and spectrally filtered the dominant transition C. The photons from
both SiV- interfered on a 50:50 beamsplitter, as schematically shown in FIG. 1
c). In the case of two identical photons entering the beamsplitter from two
different input ports, the probability amplitudes for leaving at the same port
will interfere constructively while the ones for leaving at different output
ports interfere destructively. A second-order correlation measurement will
therefore result in antibunching with vanishing correlations at zero time
delay $g^{(2)}(\tau=0)=0$. In contrast, $g^{(2)}(\tau=0)=0.5$ is indicative of
interference of two single but distinguishable photons.

(a)(b)(c) Figure 2:  a) PL spectra of SiV 1 and SiV 2 (blue and red). The
measured wavelength of transition C is identical for both color centers. They
differ only in their ground state splitting (64 GHz and 52 GHz) and local
temperature (7.7K and 6.2K). b) PL spectrum of SiV 1 after filtering with the
etalons, leaving only transition C visible. c) Normalized PLE scans of SiV 1
and SiV 2 (red, blue) with measured respective linewidths of ($158\pm 5$) MHz
and ($177\pm 4$) MHz and a detuning of ($83\pm 6$) MHz. The plot is centered
around their common center at 406.827829 THz.

The SiV-s used for the experiment are located inside NDs with an avarage
diameter of around 30 nm. They were coated onto a diamond substrate to ensure
good thermal conductivity. The sample was investigated by off-resonant photo-
luminescence (PL) and resonant photo-luminescence-excitation (PLE)
measurements showing predominantly single SiV- and a spectral distribution of
$\approx$ 14 GHz for transition C (see Appendix B). To find a matching SiV-
pair we fixed the frequency of the scanning laser to the resonance of
transition C of one SiV- and scanned the sample laterally. This way only SiV-s
with spectral overlap were visible. The spectra of the two SiV- chosen for the
HOM measurement are shown in FIG. 2 a). Their ground-state splitting (GSS)
differs by 12 GHz, but transition C shows a good overlap. The fact that
transition C overlaps although the GSS of both emitters differs can be
explained by different combinations of axial and transverse strain of the host
crystal [21]. After filtering transition C (see Appendix A for details), only
a single line was observed for each SiV-, as exemplary shown for SiV 1 in FIG.
2 b). PLE measurements of the C transitions for both SiV-s revealed linewidths
of ($158\pm 5$) MHz and ($177\pm 4$) MHz with a detuning $\frac{\Delta}{2\pi}$
of ($83\pm 6$) MHz, as shown in FIG. 2 c). Single-photon emission from both
SiV-s was confirmed by off-resonant second-order correlation measurements
which, after a normalization, resulted in $g^{(2)}_{1}(0)=0.33\pm 0.07$ and
$g^{(2)}_{2}(0)=0.35\pm 0.08$ as depicted in FIG. 3 a) and b). We use the
notation $g^{(2)}_{i}(\tau)$ for correlation functions including background
noise, while the calligraphic $\mathfrak{g}^{(2)}(\tau)$ is used for the
modeled correlation function without background. The measured data was fitted
with

$g_{i}^{(2)}(t)=1+\frac{S_{i}^{2}}{I_{i}^{2}}\mathfrak{g}_{i}^{(2)}(t)\ ,$ (1)

where $S_{i}$ is the signal from the emitter $i$, $I_{i}=S_{i}+B_{i}$ is the
total signal including background counts $B_{i}$ and
$\mathfrak{g}_{i}^{(2)}(t)=1-(1+a)\cdot\mathrm{exp}(-|t|/\tau_{1})+a\cdot\mathrm{exp}(-|t|/\tau_{2})$
is a three level model of the correlation function [22]. From the fit we
determined the signal to noise ratio $S_{i}/B_{i}\approx 4$ for each
individual SiV- which sets a lower bound for the expected HOM dip, illustrated
by the gray area in FIG. 3 c). To measure the two-photon interference we then
off-resonantly excited both SiV-s independently and let the emitted photons
interfere on a 50:50 beamsplitter after the polarization of the photons was
matched by half-wave plates. The resulting correlation function is shown in
FIG. 3 c), where red dots show data for parallel polarization and blue
triangles show data for perpendicular polarization. The data was fitted with
[5]

$\displaystyle g_{\mathrm{HOM}}^{(2)}(\tau)=$ $\displaystyle\
c^{2}_{1}g^{(2)}_{1}(0)+c^{2}_{2}g^{(2)}_{2}(0)+2c_{1}c_{2}$ $\displaystyle\
\cdot\left(1-\eta\frac{\langle S_{1}\rangle\langle S_{2}\rangle}{\langle
I_{1}\rangle\langle
I_{2}\rangle}|g^{(1)}_{1}(\tau)||g^{(1)}_{2}(\tau)|\cos(\Delta\tau)\right),$

where $c_{i}=I_{i}/(I_{1}+I_{2})$ and
$g^{(1)}_{i}(\tau)=\exp(-\gamma_{i}|\tau|/2)$. The signal and noise for each
emitter were fixed with the previously determined signal to noise ratio of the
individual correlation measurements of both SiV-. The variable $\eta$ in front
of the interference term can be interpreted as an efficiency coefficient,
where a value of 0 means no two-photon-interference and 1 means maximum
interference. We determined a value of $\eta=0.61\pm 0.16$ for the case of
parallel polarization.

(a)(b)(c) Figure 3:  a) Normalized correlation function of SiV 1 with
$g_{1}^{(2)}(0)=0.33\pm 0.07$ and $\tau_{1}=(1.84\pm 0.29)$ ns. b) Normalized
correlation function of SiV 2 with $g_{2}^{(2)}(0)=0.35\pm 0.08$ and
$\tau_{2}=(2.02\pm 0.35)$ ns. c) Two-photon interference between SiV 1 and SiV
2 with $\eta=0.61\pm 0.16$. Parallel polarization, corresponding to
indistinguishable photons, is shown with red data points. Orthogonal
polarization, corresponding to distinguishable photons, is depicted by blue
triangles. The visibility $V_{\mathrm{HOM}}$ of the interference is shown in
green.

As an additional figure-of-merit we calculated the coalescence time window
(CTW) as described in [23]. It gives a time-window for which coalescence can
occur on the beam-splitter. By integrating over the visibility function

$\displaystyle
V_{\mathrm{HOM}}=1-g_{\parallel}^{(2)}(\tau)/g_{\perp}^{(2)}(\tau)$ (3)

we find

$\displaystyle\mathrm{CTW}=\int V_{\mathrm{HOM}}\mathrm{d}\tau=0.35\
\mathrm{ns}$ (4)

which is below the limit of 2$T_{1}$ expected for two ideal emitters with
absolutely indistinguishable photons. Here $T_{1}\approx 1.9$ ns is the
excited state lifetime. The result is limited by dephasing of the the emitters
and background noise.
We demonstrated the generation of indistinguishable single photons from SiV-
in two remote NDs with an efficiency of $\eta=0.61$, the extracted CTW yielded
0.35 ns. The interference visibility is limited by technical imperfections
such as polarization drifts or detector timing response, which can be improved
in future experiments. Also minor spectral diffusion during the measurement
can diminish the visibility. Our results establish SiV- in NDs as a viable
source for the generation of indistinguishable photons and open new
possibilities for the integration into hybrid quantum photonics. The
incorporation into photonic structures boosts the operation bandwidth and
paves the way to establish remote entanglement of distant quantum nodes in an
integrated fashion.
Furthermore, NDs much smaller than the wavelength of light, which contain
individual quantum emitters with spectrally indistinguishable transitions
offer new possibilities for the construction of cooperative quantum materials.
Spatial indistinguishability can be achieved by positioning the NDs in a
collective mode within a volume small compared to the third power of the
radiation wavelength, $V\ll\lambda^{3}$. Thereby, collective states can be
prepared in the Dicke regime [24] in a bottom-up approach by means of AFM-
based nanomanipulation. Cooperative processes such as superradiance [25] and
superabsorption [26] can be accessed with color centers in diamond. Pioneering
work demonstrated superradiance effects with ensembles of NV- center in the
optical [27, 28] and microwave [29] domain and indicated a first onset of
cavity-assisted superabsorption [30]. With the presented work, collective
states could now be engineered atom-by-atom and with altered settings from
spatially confined, interacting atoms to far distant non-interacting atoms
[31].

###### Acknowledgements.

We thank V. A. Davydov for the synthesis of the nanodiamonds. The project was
funded by the Baden-Württemberg Stiftung in the project Internationale
Spitzenforschung. A.K. acknowledges support of the BMBF/VDI in the project
QR.X and Spinning.

##  Appendix A Methods

The nanodiamonds used for the experiment where synthesized by high pressure
high temperature treatment of a mixture of Naphthalene - C10H8 and the Si
doping component Tetrakis(Trimethylsilyl)silane – C12H36Si5. The average size
is around 30nm and the NDs mostly host single SiV-. The same NDs where used in
[20].

Figure 4: Schematic of the optical setup for the two-photon interference
experiment. A continious-wave laser with 532nm was used for excitation of the
individual color centers SiV 1 and SiV 2 inside a flow-cryostat. The
fluorescence was directed through spectral filters, two consecutive etalons
and half-wave plates and interfered at a 50:50 non-polarizing beam splitter.
The photons were collected through single-mode fibers into SPCMs.

To measure TPI we used a home-built, two-channel confocal microscope as
depicted in FIG. 4. A continuous-wave 532nm laser was used to excite the
individual color centers off-resonantly. The beam was split at a 50:50
polarizing beamsplitter and a half-wave plate before the beam splitter allowed
to adjust the excitation power for each channel and thus balance the emission
from both SiV-. Two galvo-scanners were used to direct the beam onto the SiV-.
A knife edge prism divided the field of view of the confocal setup into two
independent channels. The sample was placed inside a continuous-flow cryostat
and cooled with liquid helium to around 4K. The NDs were coated onto a diamond
substrate for good thermal conductivity and reached local temperatures between
5K and 10K. Fluorescence of the SiV- in both channels was filtered with a
dichroic mirror, a 740/13 band-pass filter and two consecutive etalons. The
first etalon has a free spectaral range (FSR) of 850 GHz and a linewidth of 90
GHz, the second one has a FSR of 10 GHz and a linewidth of 1 GHz. Before both
paths were joined on a 50:50 non-polarizing beamsplitter, a half-wave plate
was used to adjust their polarization. Photons were then collected with two
single mode fibers, detected by single photon counting modules (SPCM) and
correlated with a time tagger device.

(a)(b) Figure 5: Confocal scan of a) SiV 1, and b) SiV 2. Both SiV- are
spatially isolated from other emitters and separated by 95 $\mu$m.

A confocal-scan of both SiV- used for the TPI is shown in FIG. 5. Each SiV-
was spatially isolated from other SiV- and both are spatially separated by 95
$\mu$m.

To determine the long time dynamics of the emitters and ensure a correct
normalization, we analyzed the correlation functions of both emitters and
fitted them to a three level model for the correlation function. Both SiV-
showed a small amount of bunching and a shelving time of around 25 ns.
Therefore, the correlation functions $g^{2}(\tau)$ are normalized to 1 with
the averaged value of $g^{2}(\tau)$ within $75$ ns $<$ $\tau<100$ ns as
highlighted by a grey areas in FIG. 6.

(a)(b)(c) Figure 6: Long time dynamics of the measured correlation functions
of SiV 1 and SiV 2 (a), b)) and the two-photon-interference (c)), revealing a
minor bunching behavior. The coincidence levels within the gray shaded regions
where used to normalize the correlation functions.

##  Appendix B Properties of the SiVs

The investigated SiV- show an inhomogeneous distribution with a full width at
half maximum (FWHM) of around 14 GHz for the position of the C- transition, as
shown in the histogram in FIG. 7 a). Multiple emitter show a significant
overlap when comparing their spectral position, but some of the investigated
lines show strong inhomogeneous broadening up to 1 GHz.

(a)(b)(c) Figure 7: a) Histogram of the measured positions of transition C for
different SiV-s. The distribution has a width of 13.6 GHz and is centered at
406.82031 THz. b) Zoomed view into the distribution showing multiple possible
transitions suitable for measuring two-photon interference. Emitters with
significant overlap are marked with a red bar representing a linewidth of 94
MHz. The pair marked with the red square was used to show the two-photon
interference. c) Evolution of the spectral position and linewidth for SiV 1
and SiV 2 over 15 scans, on a timescale of around 2 minutes. The center
position is represented by the lineplot, the bars represent the FWHM of the
individual scans.

When setting an upper limit of 400 MHz for the inhomogeneous linewidth, we can
still find multiple pairs and even groups of emitters which could be used for
TPI. FIG. 7 b) depicts suitable candidates for TPI with linewidths below 400
MHz. The x-axis is the transition energy, while the y-axis shows the
respective linewidth of an emitter. The width of the rectangular data points
represents the linewidth of the measured emitter. Transitions with significant
spectral overlap within the natural linewidth of 94 MHz [9] are connected with
a red line. The two SiV- used for the TPI in the manuscript are marked by the
red square. They were selected because of the combination of their spectral
overlap, stability and narrow inhomogeneous linewidth. Under resonant
excitation both SiV-s were spectrally stable over a total of 15 scans, as
shown in FIG. 7 c), where the spectral position is represented by the
connecting lines and the respective linewidth by the colored bars. Also the
overlapping region is depicted by the darker, shaded region.

## References

  * Wehner  _et al._ [2018] S. Wehner, D. Elkouss, and R. Hanson, Quantum internet: A vision for the road ahead, Science 362, 6412 (2018).
  * Hong  _et al._ [1987] C. K. Hong, Z. Y. Ou, and L. Mandel, Measurement of subpicosecond time intervals between two photons by interference, Phys. Rev. Lett. 59, 2044 (1987).
  * Chanelière  _et al._ [2007] T. Chanelière, D. N. Matsukevich, S. D. Jenkins, S.-Y. Lan, R. Zhao, T. A. B. Kennedy, and A. Kuzmich, Quantum interference of electromagnetic fields from remote quantum memories, Phys. Rev. Lett. 98, 113602 (2007).
  * Flagg  _et al._ [2010] E. B. Flagg, A. Muller, S. V. Polyakov, A. Ling, A. Migdall, and G. S. Solomon, Interference of single photons from two separate semiconductor quantum dots, Phys. Rev. Lett. 104, 137401 (2010).
  * Lettow  _et al._ [2010] R. Lettow, Y. L. A. Rezus, A. Renn, G. Zumofen, E. Ikonen, S. Götzinger, and V. Sandoghdar, Quantum interference of tunably indistinguishable photons from remote organic molecules, Phys. Rev. Lett. 104, 123605 (2010).
  * Legero  _et al._ [2004] T. Legero, T. Wilk, M. Hennrich, G. Rempe, , and A. Kuhn, Quantum beat of two single photons, Phys. Rev. Lett. 93, 070503 (2004).
  * Bernien  _et al._ [2012] H. Bernien, L. Childress, L. Robledo, M. Markham, D. Twitchen, and R. Hanson, Two-photon quantum interference from separate nitrogen vacancy centers in diamond, Phys. Rev. Lett. 108, 043604 (2012).
  * Sipahigil  _et al._ [2012] A. Sipahigil, M. L. Goldman, E. Togan, Y. Chu, M. Markham, D. J. Twitchen, A. S. Zibrov, A. Kubanek, and M. D. Lukin, Quantum interference of single photons from remote nitrogen-vacancy centers in diamond, Phys. Rev. Lett. 108, 143601 (2012).
  * Rogers  _et al._ [2014] L. J. Rogers, K. D. Jahnke, T. Teraji, L. Marseglia, C. Müller, B. Naydenov, H. Schauffert, C. Kranz, J. Isoya, L. P. McGuinness, and F. Jelezko, Multiple intrinsically identical single-photon emitters in the solid state, Nature Communications 5, 4739 (2014).
  * Sipahigil  _et al._ [2014] A. Sipahigil, K. D. Jahnke, L. J. Rogers, T. Teraji, J. Isoya, A. S. Zibrov, F. Jelezko, and M. D. Lukin, Indistinguishable photons from separated silicon-vacancy centers in diamond, Phys. Rev. Lett. 113, 113602 (2014).
  * Fehler  _et al._ [2021] K. G. Fehler, L. Antoniuk, N. Lettner, A. P. Ovvyan, R. Waltrich, N. Gruhler, V. A. Davydov, V. N. Agafonov, W. H. Pernice, and A. Kubanek, Hybrid quantum photonics based on artificial atoms placed inside one hole of a photonic crystal cavity, ACS photonics 8, 2635 (2021).
  * Bayer  _et al._ [2022] G. Bayer, R. Berghaus, S. Sachero, A. B. Filipovski, L. Antoniuk, N. Lettner, R. Waltrich, M. Klotz, P. Maier, V. Agafonov, and A. Kubanek, A quantum repeater platform based on single siv centers in diamond with cavity-assisted, all-optical spin access and fast coherent driving, ArXiv (2022).
  * Wan  _et al._ [2020] N. Wan, T. Lu, K. Chen, and et al., Large-scale integration of artificial atoms in hybrid photonic circuits, Nature 583, 226–231 (2020).
  * Schrinner  _et al._ [2020] P. P. J. Schrinner, J. Olthaus, D. E. Reiter, and C. Schuck, Integration of diamond-based quantum emitters with nanophotonic circuits, Nano Lett. 20, 8170 (2020).
  * Elshaari  _et al._ [2020] A. W. Elshaari, W. Pernice, K. Srinivasan, O. Benson, and V. Zwiller, Hybrid integrated quantum photonic circuits, Nat Photonics (2020).
  * Kubanek  _et al._ [2022] A. Kubanek, A. Ovvyan, L. Antoniuk, N. Lettner, and W. Pernice, Hybrid quantum nanophotonics—interfacing color center in nanodiamonds with si3n4-photonics, Yatsui, T. (eds) Progress in Nanophotonics 7. Topics in Applied Physics, Springer 147 (2022).
  * Sahoo  _et al._ [2022] S. Sahoo, V. Davydov, V. Agafonov, and S. I. Bogdanov, Hybrid quantum nanophotonic devices with color centers in nanodiamonds, ArXiv (2022).
  * Häußler  _et al._ [2019a] S. Häußler, L. Hartung, K. G. Fehler, L. Antoniuk, L. F. Kulikova, V. A. Davydov, V. N. Agafonov, F. Jelezko, and A. Kubanek, Preparing single siv center in nanodiamonds for external, optical coupling with access to all degrees of freedom, New Journal of Physics 21, 103047 (2019a).
  * Rogers  _et al._ [2019] L. J. Rogers, O. Wang, Y. Liu, L. Antoniuk, C. Osterkamp, V. A. Davydov, V. N. Agafonov, A. B. Filipovski, F. Jelezko, and A. Kubanek, Single $\mathrm{Si}$-${V}^{-}$ centers in low-strain nanodiamonds with bulklike spectral properties and nanomanipulation capabilities, Phys. Rev. Applied 11, 024073 (2019).
  * Klotz  _et al._ [2022] M. Klotz, K. G. Fehler, R. Waltrich, E. S. Steiger, S. Häußler, P. Reddy, L. F. Kulikova, V. A. Davydov, V. N. Agafonov, M. W. Doherty, and A. Kubanek, Prolonged orbital relaxation by locally modified phonon density of states for the ${\mathrm{si}v}^{-}$ center in nanodiamonds, Phys. Rev. Lett. 128, 153602 (2022).
  * Meesala  _et al._ [2018] S. Meesala, Y.-I. Sohn, B. Pingault, L. Shao, H. A. Atikian, J. Holzgrafe, M. Gündoğan, C. Stavrakas, A. Sipahigil, C. Chia, R. Evans, M. J. Burek, M. Zhang, L. Wu, J. L. Pacheco, J. Abraham, E. Bielejec, M. D. Lukin, M. Atatüre, and M. Lončar, Strain engineering of the silicon-vacancy center in diamond, Phys. Rev. B 97, 205444 (2018).
  * Aharonovich  _et al._ [2010] I. Aharonovich, S. Castelletto, D. A. Simpson, A. D. Greentree, and S. Prawer, Photophysics of chromium-related diamond single-photon emitters, Phys. Rev. A 81, 043813 (2010).
  * Proux  _et al._ [2015] R. Proux, M. Maragkou, E. Baudin, C. Voisin, P. Roussignol, and C. Diederichs, Measuring the photon coalescence time window in the continuous-wave regime for resonantly driven semiconductor quantum dots, Phys. Rev. Lett. 114, 067401 (2015).
  * Dicke [1954] R. H. Dicke, Coherence in spontaneous radiation processes, Phys. Rev. 93 (1954).
  * GROSS and HAROCHE [1982] M. GROSS and S. HAROCHE, Superradiance: An essay on the theory of collective spontaneous emission, PHYSICS REPORTS 93, 301—396 (1982).
  * Higgins  _et al._ [2014] K. Higgins, S. Benjamin, T. Stace, G. Milburn, B. Lovett, and E. Gauger, Superabsorption of light via quantum engineering, Nat. Comm. 5, 4705 (2014).
  * Bradac  _et al._ [2017] C. Bradac, M. Johnsson, and M. e. a. Breugel, Room-temperature spontaneous superradiance from single diamond nanocrystals, Nat. Commun. 8, 1205 (2017).
  * Gutsche  _et al._ [2022] J. Gutsche, A. Zand, M. Bültel, and A. Widera, Revealing superradiant emission in the single-to-bulk transition of quantum emitters in nanodiamond agglomerates, New J. Phys. 24, 053039 (2022).
  * Angerer  _et al._ [2018] A. Angerer, K. Streltsov, T. Astner, and et al., Superradiant emission from colour centres in diamond, Nature Phys 14, 1168–1172 (2018).
  * Häußler  _et al._ [2019b] S. Häußler, J. Benedikter, K. Bray, B. Regan, A. Dietrich, J. Twamley, I. Aharonovich, D. Hunger, and A. Kubanek, Diamond photonics platform based on silicon vacancy centers in a single-crystal diamond membrane and a fiber cavity, Phys. Rev. B 99, 165310 (2019b).
  * Bojer and von Zanthier [2022] M. Bojer and J. von Zanthier, Dicke-like superradiance of distant noninteracting atoms, PHYSICAL REVIEW A 106, 053712 (2022).