# Transfer learning with generative models for object detection on limited
datasets

M. Paiano1, S. Martina2,3, C. Giannelli1, F. Caruso2,3,4 1Department of
Mathematics and Computer Science, University of Florence, Viale Morgagni 67/a,
I-50134 Florence, Italy  2Department of Physics and Astronomy, University of
Florence, Via Sansone 1, I-50019 Sesto Fiorentino, Italy  3European Laboratory
for Non-Linear Spectroscopy (LENS), University of Florence, Via Nello Carrara
1, Sesto Fiorentino, I-50019, Italy  4Istituto Nazionale di Ottica, Consiglio
Nazionale delle Ricerche (CNR-INO), I-50019 Sesto Fiorentino, Italy
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The availability of data is limited in some fields, especially for object
detection tasks, where it is necessary to have correctly labeled bounding
boxes around each object. A notable example of such data scarcity is found in
the domain of marine biology, where it is useful to develop methods to
automatically detect submarine species for environmental monitoring. To
address this data limitation, the state-of-the-art machine learning strategies
employ two main approaches. The first involves pretraining models on existing
datasets before generalizing to the specific domain of interest. The second
strategy is to create synthetic datasets specifically tailored to the target
domain using methods like copy-paste techniques or ad-hoc simulators. The
first strategy often faces a significant domain shift, while the second
demands custom solutions crafted for the specific task. In response to these
challenges, here we propose a transfer learning framework that is valid for a
generic scenario. In this framework, generated images help to improve the
performances of an object detector in a few-real data regime. This is achieved
through a diffusion-based generative model that was pretrained on large
generic datasets, and is not trained on the task-specific domain. We validate
our approach on object detection tasks, specifically focusing on fishes in an
underwater environment, and on the more common domain of cars in an urban
setting. Our method achieves detection performance comparable to models
trained on thousands of images, using only a few hundreds of input data. Our
results pave the way for new generative AI-based protocols for machine
learning applications in various domains, for instance ranging from geophysics
to biology and medicine.

Keywords: Object Detection, Transfer Learning, Generative AI, Diffusion
Models, Deep Learning

##  1 Introduction

Machine Learning (ML) is the area of artificial intelligence that involves the
development of models that are trained on data to perform specific jobs. In
the last few years, we are witnessing a growing interest in generative models
as powerful tools in the context of ML. Generative models are capable of
estimating the unknown probability distribution underlying the training
dataset, in order to generate samples from such distribution. Some of the
research works in this field [24] include Gaussian Mixture Models, Hidden
Markov Models, Latent Dirichlet Allocation, Boltzmann Machines and, lately,
Variational Autoencoders and Generative Adversarial Networks [50]. More
recently, _diffusion_ models emerged as a new family of generative models
[59], overcoming GANs as the most adopted and powerful generative models in
image generation tasks [10]. They are designed according to three main
formulations: Denoising Diffusion Probabilistic Models, Score-based Generative
Models, or Stochastic Differential Equations. In all these approaches, the
main concept is that the images of the training dataset are progressively
perturbed with random noise in a fixed process (called _diffusion_) where all
the information is destroyed, and the images at the end of the process belong
to an uninformative distribution (usually a standard multivariate Gaussian
distribution). Afterward, the process is reversed with a parametrized ML model
that estimates the unknown original data distribution. The trained model can
then generate novel data samples belonging to an approximation of the original
distribution. The most commonly adopted DDPMs use two Markov chains for the
forward and backward training processes, respectively.

Given the late increasing amount of works on generative models for image
generation, we are interested in employing them in novel valuable scientific
applications. For instance, _object detection_ is the image processing task
that consists in the identification of object instances within images [65].
One of the traditional ML approaches is _supervised learning_ that requires
the input data to be labelled with the desired output of the model to be
trained. The model learns to extract statistical _features_ (patterns that
capture relevant information about the content of an image) and to predict
_labels_ for each input. For ML object detection, labels are in the form of
coordinates of bounding boxes around each target, along with identifications
of objects types called _classes_. This means that for training ML models, it
is desirable to have access to large amounts of correctly labelled data, but
the labelling work is burdensome, and the availability of high quality
annotated datasets is scarce, especially for crowded scenarios or specialized
contexts. In fact, data scarcity, along with high model complexity,
contributes to _overfitting_ : the model learns features that are too specific
to the training set, sacrificing generalization capabilities on the unseen
data. To tackle this challenge, various techniques revolve around the notion
of dataset _domain_ , defined as the joint probability distribution of the
feature-label space [17].

The most common strategy to addresses the problem of data scarcity in any ML
problem in general, and in computer vision tasks in particular, is _data
augmentation_ , which extract more information from the training datasets and
enhance their diversity [52]. In detail, it consists in an image manipulation
process that creates new images by applying on the existing dataset operations
such as color space transformations, scene cropping, noise injection or affine
transformations (translations, rotations, etc…). Another way to artificially
expand the size and diversity of the dataset strongly relies on domain
knowledge and consists in the construction of _synthetic data_ that mimic the
global distribution of real data instances [21]. However, such data
synthesizers are typically domain-specific, hard to implement, and they
require a deep knowledge of the real context captured by the images. Thus, the
cost of producing photorealistic images undermines the core advantage of
synthetic data, which is the availability of large labeled data sets for free.
The data augmentation techniques can be thought as an equivalent of the role
of imagination in humans, where alterations of the available dataset are
imagined [52]. In fact, imagination is shown to be important in learning tasks
for humans, and forming mental images helps in memory related tasks [41]. In
particular, for object detection tasks, there is evidence that the lack of
imagination in aphantasic subjects (people unable to create mental imagery)
have a negative impact on the performances of these tasks [42, 43].

Augmentations and synthesizers introduce variability, but data still belongs
to the existing domain. An alternative approach is _domain randomization_
[55], which change the domain fundamental distribution (e.g. by randomizing
foreground textures, lighting conditions, backgrounds and occlusions). This
forces the model to focus only on the essential features, by interpreting the
real-world data just as another variation among the synthetic data. However,
this technique still demands to be familiar with the tools necessary to
implement the simulation. An easier approach is to copy and paste real images
of objects onto background images [16], but this arises the challenge for
accurate segmentation of the objects to be copied.

The previous approaches rely on the assumption that both train and test
datasets are drawn from the same (possibly randomized) data distribution,
otherwise the model will underperform on the test set due to a phenomenon
known as _domain shift_. To counter this problem, the common strategy is
called _transfer learning_ [64]. The idea is based on the generalization
theory of transfer of training, studied within the framework of educational
psychology [35]. According to this theory, engaging with a new activity by
establishing connections with previous experiences enhances the learning
process. For instance, students that studied Latin learn related languages
(e.g., German) more easily [5]. Inspired by this concept, transfer learning in
ML aims to enhance model performances by transferring knowledge from one
domain (referred to as the _source_) to another (the _target_), thereby
reducing the need for domain-specific data in the target domain. One effective
approach involves training a model on a specific task, such as feature
extraction from a particular dataset, and subsequently applying it to extract
features in a different domain. If images from the target domain are
available, another common technique initially trains the model on the source
dataset and then conducts a _fine-tuning_ stage, i.e. a re-training of part
(or all) of the pretrained model using new data from the target domain [8].
Some examples of transfer learning include, but are not limited to, image
classification in the medical field [13, 33], detection of gravitational waves
[23] and material properties prediction [34]. For an extensive review of
transfer learning applications, one can consult Ref. [31]. Transfer learning
can be split into two main categories: _homogeneous_ , when the source and
target domains share the same features space, and _heterogeneous_ , otherwise
[12]. Within homogeneous transfer learning, a subcategory known as _domain
adaptation_ [38] addresses scenarios where the feature spaces exhibit
different distributions and aims to minimize the gap between the two domains.
An illustrative example of domain adaptation is the transfer of knowledge from
real pictures to cartoon images [63]. Despite sharing the same feature spaces,
the distributions of colors and shapes between the two domains differ.

In many scientific areas, detection tasks can also be useful for research
activities. Some examples are the counting of microorganisms [60], the
monitoring of animals in the wild [2, 1] and plants phenotyping [25]. In this
work, we focus on the problem of training object detectors on domains where
the availability of data is scarce. For instance, in the context of marine
biology, it is crucial to be able to monitor wild fishes and, in general, to
properly support environmental monitoring and protection. For research works
where this task was addressed with deep learning techniques see, e.g., Refs.
[57, 39, 9, 40]. However, the availability of data in this field is not of
great extent [14, 56, 20]. To resolve this issue, some works addressed the
data scarcity with the aid of simulators to generate synthetic data [32, 46];
other solutions have been found in the context of transfer learning from other
domains [53].

In this paper, we investigate a novel strategy that merges the fields of
generative models and object detection. While generative models have been
proven successful in generating data for training classifiers [3], their
application in the realm of object detection has been less explored. The
challenge arises when attempting to generate images suitable for object
detectors, as it necessitates the representation of targets within specific
layouts, adhering to bounding box positions crucial for supervised training.
This issue is formulated as _ Layout-to-Image (L2I)_ generation. Traditional
generative models, however, faced limitations in generating images with such
constraints. To avoid this challenge, some prior approaches employed GANs for
performing _style transfer_ , a form of domain adaptation, on the training set
[62, 29]. Nonetheless, this method relies on an existing training set, whether
synthetic or real, and does not truly generate entirely new images. The first
attempt to address the L2I problem was made by Layout2Im [61], but it is only
in recent times, notably with GLIGEN [36], that the generation of constrained
images has reached a satisfactory level of quality. It is worth emphasizing
that the existing literature has explored the problem within the realm of
generative models, without application to the object detection task.

Figure 1:  Transfer learning for object detection with generative models. We
employ a L2I pretrained model to generate images for transfer learning to an
object detector. We can filter out suboptimal generated images based on
benchmark metrics. For instance, the image along the red arrow is discarded
because the generative model has depicted many cars outside the bounding boxes
designated in the grounding instruction. With the remaining generated images,
we pretrain the object detector, followed by a fine-tuning on the real
dataset. Dashed lines indicate the data used for training the models.

We propose to leverage a L2I model to generate abundant labeled data for free,
ready for the training of an object detector. To further alleviate the
dependence on real labeled data, we intentionally choose not to train the
generation model specifically for the real target domain. In particular, we
exploit GLIGEN, which is peculiarly built upon existing models pretrained on
large-scale web data. This endows GLIGEN with the inherent knowledge derived
from other extensive and diverse datasets, facilitating the representation of
novel concepts not encountered during its training, aligning seamlessly with
our objectives. This sets GLIGEN apart from prior L2I generative models, which
were typically confined to be trained on datasets tailored to specific tasks.
Furthermore, the inherent value of this strategy lies in its potential
applicability to diverse domains without the explicit requirement of training
the generator on each individual domain.

In detail, our architecture, illustrated in Fig. 1, adopts a two-step process.
Initially, the object detector undergoes training exclusively on generated
images, followed by a subsequent fine-tuning stage using only the limited real
dataset. The rationale behind this process is to facilitate transfer learning,
enabling the model to exploit a large amount of available data in a source
domain, here the domain of generated images, to correctly detect data from the
target domain, here the domain of real images, for which the amount of
training data is limited. Moreover, we designed an optional filtering strategy
to assess the impact of the generated dataset on the object detector and to
select the most suitable images for its pretraining. To assess the
effectiveness of our approach, we evaluate it on two different detection
domains: cars and fishes in urban and underwater environments, respectively.
The reason for this choice is to compare the effectiveness of the proposed
approach on a common domain with high data availability (the former) with
another domain with data scarcity and interesting for the scientific community
(the latter).

The structure of the paper is as follows: our transfer learning approach with
pretrained generative models is detailed in Secs. 2 and 3. In Sec. 4 the
filtering strategies are introduced. Sec. 5 shows the real datasets selected
for our tests, while Sec. 6 presents some numerical results. Finally, Sec. 7
concludes the paper with some final remarks and outlooks.

##  2 Generative model

Our pipeline is designed to be sufficiently generic, provided that the
generator is of the L2I type, and it is capable to generalize to diverse
domains. At the best of our knowledge, only GLIGEN satisfies these
requirements in the current state-of-the-art.

In the context of GLIGEN, the generation process is guided by an instruction
$\textbf{y}=(\textbf{c},\textbf{e})$ that the model follows to create a scene
with the desired concepts in specific locations. The instruction comprises a
_prompt_ c, a written description of the image content, and a _grounding
instruction_
$\textbf{e}=\left[(e_{0},\textbf{l}_{0}),...,(e_{N},\textbf{l}_{N})\right]$.
Each $\textbf{l}_{i}$ represents the bounding box coordinates around the
object $e_{i}$ that should be depicted within. These objects, referred to as
_grounding entities_ , can be described either by text or by a reference real
image. In Sec. 6, we will present an ablation study comparing the description
of grounding entities through text versus real images. This comparison aims to
assess which grounding entity type produces targets best suited for detection
in the domain of interest. Regardless of their nature, both kinds of grounding
entities are encoded by a pretrained CLIP [47], which is a model trained on
vast web-scale data to predict associations between images and corresponding
texts, and vice versa. Renowned for its ability to make predictions on
concepts not explicitly encountered during training, CLIP encoding is
leveraged by GLIGEN, achieving the generalized representation abilities we are
seeking. Additionally, associating a reference image as $e_{i}$ with an image
edge as the corresponding $\textbf{l}_{i}$ facilitates a consistent style
transfer from the real to the generated domain. We leverage this feature to
achieve backgrounds that closely resemble the originals.

Throughout the remainder of our paper, the grounding instruction corresponds
to what in the realm of object detection is referred to as _ground truth_ ,
which is the list of all dataset labels (i.e. bounding boxes and classes of
every target). To automatically choose each $\textbf{l}_{i}$, we start by
collecting the real dataset’s ground truth information from a small set of
labeled sample shots. Subsequently, we conduct a statistical analysis to
estimate the means and variances of the bounding box parameters (i.e.
positions, shapes, and number per image) within the original dataset.
Following this, we randomly sample from Gaussian distributions using the
estimated means and variances, to get plausible parameters for the ground
truth to generate.

Therefore, as represented on the left part of Fig. 1, to generate a new image
we supply GLIGEN with the following inputs: a text prompt, a desired grounding
instruction, an image from the real dataset to get style transfer. The details
on the use of the generation model on example datasets is described in Sec. 5.

##  3 Object detector

Our pipeline is general enough to also accept any kind of object detector. For
our tests, we opted for the traditional Faster R-CNN [48] which is a region-
based model (which gives the letter “ _R_ ” in the name). We recall that a
Faster R-CNN is mainly composed of two parts, called _backbone_ and Region
Proposal Network (RPN).

The backbone uses Convolutional Neural Networks, models capable of extracting
relevant features, to transform an input image into two-dimensional tensors
known as _feature maps_. The RPN overlays the feature maps with a set of
bounding boxes, referred to as _anchors_. On each anchor, the network conducts
_objectness classification_ , which distinguishes generic objects from the
background. Subsequently, the anchors containing objects are submitted to a
standard neural network classifier for multiclass classification.

In practice, Faster R-CNN is often trained in a transfer learning setting. One
common approach involves pretraining the entire model on a standard detection
dataset (e.g., COCO [37]) and subsequently fine-tuning it on the specific
domain of interest. However, to avoid biases, we choose not to pretrain on
other detection datasets, in order to focus on the contribution of generated
data only. Besides, in [28] it is demonstrated that training the entire Faster
R-CNN on synthetic images leads to significantly poorer performance compared
to training on real data. Conversely, freezing a pretrained feature extractor
and exclusively training the remaining parts of the detector results in a
considerable performance improvement. Therefore, we freeze a pretrained
backbone, and we randomly initialize the RPN weights. Finally, it is worth
noting that the backbone can be pretrained on a dataset not necessarily
tailored for object detection, thus reducing the need for costly labeling. In
detail, in our analyses, we opted to use as backbone a ResNet-50 [26] —a
renowned 50 layers deep CNN—pretrained on ImageNet [49], a famous image
classification dataset which has become the de facto standard for the
pretraining of a CNN [30].

Overall, our Faster R-CNN has $41\,370\,911$ weights, of which $23\,454\,912$
belong to the backbone and are frozen. The remaining weights are trained with
Stochastic Gradient Descent (SGD) with momentum, which updates the weights
$\theta_{t}$ at time $t$, aiming to minimize a loss function $L$ with the
iterative equations:

$\displaystyle b_{0}$ $\displaystyle=0\ ,$ $\displaystyle b_{t}$
$\displaystyle=\mu b_{t-1}+\nabla_{\theta}L(\theta_{t-1})+\lambda\theta_{t-1}\
,$ $\displaystyle\theta_{t}$ $\displaystyle=\theta_{t-1}-\gamma b_{t}\ ,$

involving the so-called _momentum_ $\mu$, _weight decay_ $\lambda$, _learning
rate_ $\gamma$, and $\nabla_{\theta}L$ gradient of $L$. The loss to minimize
is formed by the contribution of different terms regarding the positioning of
the predicted boxes, the objectness classification and the multiclass
classification of each box. In our experiments, we set the value of $\mu=0.9$,
while $\lambda$ is $0.001$ during the model pretraining on generated data, and
$0.01$ for the fine-tuning on the real data. The reason for the latter is that
when the model is trained on a limited dataset size, it is more prone to
overfitting. We use higher values of $\mu$, that provide higher
_regularization_ , i.e. a penalization to reduce the model complexity.

Regarding the learning rate, we adopted two different scheduling strategies.
During the pretraining step, we retain $15\%$ of the generated data as a
_validation_ set, a portion of the dataset used to monitor the loss value
after each SGD phase over all the remaining training set, i.e. a training
_epoch_. Initially, $\gamma$ is set equal to $0.001$ and if the value of the
validation loss do not improve for 5 epochs, $\gamma$ is reduced by a factor
of $0.1$. Moreover, we adopt also an _early stopping_ strategy where the
detector is trained for a maximum of 200 epochs until the validation loss
stops decreasing for at least 10 epochs. In the end, we keep the weights of
the model at the epoch with the minor validation loss value. Conversely,
during the fine-tuning step, we adopted the common _1x scheduler_ which has a
total fixed number of 12 epochs, and $\gamma$ is initialized with a starting
value of $0.001$, before reducing it by a factor $0.1$ at epoch 7 and again at
epoch 10. This scheduler does not need a validation set to compute the loss
on, which allow us to save data repurposing the real validation set as the
test set. We opted to maintain consistent scheduling conditions—and not to
further optimize the fine-tuning—to achieve a fair comparison, focusing only
on the impact of the pretraining stage in the pipeline.

##  4 Metrics and filters

We can evaluate the usefulness of the generation process, described in Sec. 2,
through a filter. Its role is to select the best images for the pretraining of
the object detector, based on benchmark performance metrics. In the current
state of the art, the most used metric for evaluating generative models is the
Frechet Inception Distance (FID) [27]. It quantifies the similarity and
diversity of generated images compared to real data. On the other hand, in the
realm of object detection, the main metric to assess the quality of the
predictions is known as mean Average Precision (mAP), which is based on the
key concepts of precision and recall. In the following, we define how we
calculate FID, precision, recall, and the mAP, and we discuss how we use these
metrics to filter out suboptimal generated images.

###  4.1 FID score

The FID score quantifies the similarity between generated and real images,
measuring the difference between their feature distributions: $X$ extracted
from real images and $Y$ from generated images. These distributions are
respectively approximated by two multivariate Gaussian distributions with
means $\mu_{X}$ and $\mu_{Y}$, and covariances $\Sigma_{X}$ and $\Sigma_{Y}$.
The FID is thus computed111See the repository
https://github.com/mseitzer/pytorch-fid . by:

$d^{2}=||\mu_{X}-\mu_{Y}||^{2}+Tr(\Sigma_{X}+\Sigma_{Y}-2\,\sqrt{\Sigma_{X}\,\Sigma_{Y}}).$
(1)

In detail, features are extracted by an Inception-V3 model [54] pretrained on
ImageNet.

Even if FID is usually considered as the conventional metric of generative
models, its effectiveness in evaluating images used for training object
detectors has not been thoroughly explored. Although lower FID values indicate
better image quality, it remains unknown whether this directly correlates with
improved performances for pretraining an object detector. To perform this
analysis, we introduce a FID-based filter, which retains only those images
that yield a FID score below a certain threshold. To implement it, we compute
the FID score between the generated and the real datasets. Then, we
iteratively remove one image from the generated dataset, and we re-calculate
the FID with the real dataset. If this new FID is lower than the previous FID
value, then we definitively remove that image, responsible for distancing the
feature distributions between generated and real datasets. Otherwise, the
image is kept into the generated dataset.

###  4.2 Precision-Recall

To define precision and recall in the context of object detection, we leverage
the notion of _ Intersection over Union (IoU)_, which quantifies the
similarity between a ground truth and a predicted bounding box. It is
computed, as depicted in Fig. 2(a), with the ratio between the area of the two
boxes intersection and the area of their union. Its value ranges from 0 to 1:
a higher value of IoU denotes a larger overlap between the boxes as visible in
the three examples of Fig. 2(b). By setting an IoU threshold (typically at
0.5), a prediction box is classified as _True Positive_ (TP) if its IoU score
is above the considered threshold, _False Positive_ (FP) otherwise. Any
failure to detect a ground truth counts as a _False Negative_ (FN). These
values collectively define precision (${prec}$) and recall ($rec$) as

${prec}=\frac{TP}{TP+FP}\ ,\qquad{rec}=\frac{TP}{TP+FN}\ .$ (2)

The former measures the correct predictions across all detections and the
latter the amount of true targets found among all ground truths.

(a)

(b)

(c) A target is properly generated, and the filter correctly identifies it
with $\acs{iou}>0.5$ as TP, thus Precision and Recall are both 1.

(d) A target is wrongly generated on the left. The filter classifies it with
$\acs{iou}=0$ as FP, thus Precision and Recall are 0.5 and 1, respectively.

(e) The target on the right is not well generated. The filter cannot detect
it: it counts as FN, thus Precision and Recall are 1 and 0.5, respectively.

Figure 2: We calculate IoU as defined in (a) to evaluate the overlap (b)
between the ground truth (in green) and the predicted boxes (in red). This is
used to implement a Precision-Recall filter to automatically identify faithful
representations (c) or discrepancies (d, e) between the intended ground truth
for generation and the actual generated images.

We propose to employ precision and recall to assess the impact that the
generated images may have on the object detector pretraining. Indeed, the L2I
models—even the most recent ones—still have some limitations. In particular,
they may struggle to adhere to the intended geometric layouts, either drawing
targets outside the designated bounding boxes or, conversely, failing to
depict the desired objects within them. We want to analyze the generated
dataset, identifying potential inconsistencies with the ground truth, in case
they may compromise the pretraining of the object detector. We implement a
Precision-Recall based filter with another Faster R-CNN initialized with COCO
weights, possibly fine-tuned on the limited real dataset. This filtering
Faster R-CNN is then tested on the generated dataset, computing precision and
recall for each image. By setting thresholds for these metrics, we selectively
retain images in which all targets are strictly generated within the specified
bounding boxes. For instance, setting a prediction threshold at 1, ensures
that no target is erroneously generated outside the ground truth constraints,
discarding the example in Fig. 2(d). Similarly, a recall threshold at 1
preserves only those images where targets are correctly generated within the
specified bounding boxes, discarding the example in Fig. 2(e).

###  4.3 Mean Average Precision

As we mentioned before, the mAP is the principal metric of interest in the
context of object detection. In our analyses, we calculate the mAP on the test
set of our real datasets.

First, for a dataset of $N$ classes, we compute $N$ precision-recall curves,
one for each individual class. For a class $i$, this involves ordering the
$M_{i}$ predicted boxes in descending order based on a _confidence_ score,
which is the probability that the target within $M_{i}$ is classified as
belonging to class $i$. Then, we compute precision and recall over all images,
but considering only the boxes having confidence score greater than a
threshold. Such threshold is iteratively decreased, and new values of
precision and recall are calculated. Plotting the computed recall values on a
x-axis and precision on a y-axis, we get the precision-recall curve for class
$i$. The _average precision_ $AP_{i}$ is computed as follows [45]:

$\displaystyle AP_{i}$
$\displaystyle=\sum_{j=1}^{M_{i}-1}(rec_{i,j+1}-rec_{i,j})prec_{int}(rec_{i,j+1})$
$\displaystyle prec_{int}(rec_{i,j+1})$
$\displaystyle=\max\\{prec(rec_{i,k}):rec_{i,k}\geq rec_{i,j+1}\\}$

where $j=1,\dots,M_{i}$ are the ordered predicted boxes for class $i$,
$rec_{i,j}$ denotes the recall value calculated for class $i$ on the $j$
predicted box, and $prec(rec_{i,j})$ is the precision value corresponding to
$rec_{i,j}$. In other words, the $AP_{i}$ can be seen as the area under an
interpolation of the $i$-th precision-recall curve. At each point, the
interpolation considers the maximum precision value for which the recall is
greater than or equal to the recall at that specific point on the curve.
Finally, the mAP is the mean of all $AP_{i}$ over the number of classes $N$.
The mAP ranges between 0 and 1: values close to 1 mean that the precision is
high also when the recall is increasing. In this case, the number of both
false positives and false negatives is low.

##  5 Real and generated datasets

In this section, we describe the real datasets selected for our tests:
NuImages and OzFish. NuImages is representative of a common urban scenario,
and serves as a simple testing ground, allowing for initial analyses and
illustrative insights into the method’s efficacy. OzFish is one of the few
publicly available datasets in marine environment. The presence of fish
instances, particularly complex to label and detect since they are prone to
overlapping, makes OzFish a valuable benchmark for its unique challenges.
Moreover, we also describe the details of how we generate the data used for
the pretraining of the detector in the two aforementioned scenarios.

Several datasets have been published to train object detectors in urban
environments. For instance, KITTI [22] is the most widely used for autonomous
driving research. However, the diversity of its recording conditions is
relatively low. Instead, NuImages [6] captures scenes in multiple locations,
providing a broader and more varied perspective. Besides, it collects data in
both daytime and nighttime scenarios, taking into account various lighting and
weather conditions, offering a more realistic and challenging environment for
object detection models [18]. The dataset is composed by around $75\,000$
images, already divided in $60\,000$ training samples and $15\,000$ validation
samples. It covers up to 25 classes, primarily falling within the broader
vehicle and human macro categories. Without loss of generality, we focused our
analysis on the car class, aiming for a less complex scenario, in order to
give a simpler overview of our proposed method. Consequently, we narrow down
our dataset to include only those images containing instances of the car
class. Besides, to simulate a situation with limited data available, we make
the assumption of using a subset of the large training dataset. Instead, for
the test set, we use all the images originally intended as validation and that
contains at least a car instance, to achieve a more statistically meaningful
result. In summary, our selected dataset contains 4 500 training images at
most, and $9\,073$ test images.

To generate each image for the pretraining dataset in the urban scenario, we
recall that we need a prompt c and a grounding instruction
$\textbf{e}=\left[(e_{0},\textbf{l}_{0}),...,(e_{N},\textbf{l}_{N})\right]$,
whose first value $e_{0}$ is specifically the real image used for style
transfer. With $N$ cars to generate, the other grounding entities $e_{i}$, for
$i=1,\dots,N$, are described by the phrase "a car", as shown in Fig. 1. In
particular, we use 3 644 training images for the style transfer, which may
seem a large number, but it actually requires no human labeling intervention.
Determining the most effective prompt for guiding the generation process falls
within the field of study known as prompt engineering, but this is beyond the
scope of our current investigation. Instead, to evaluate the potentialities
from a non-expert perspective, we opt for the basic prompt "$N$ cars in a
urban environment, highly photorealistic", where $N$ is the desired number of
cars in a single image. It is crucial to note that although we have
demonstrated a generation focused on a single class example, our methodology
can be effortlessly extended to a multi-class scenario by appropriately
setting different class phrases for the $e_{i}$ in the grounding instruction.

For our second test scenario, in marine environment, we remark that the task
of acquiring high-quality labeled underwater images is challenged by two main
factors [58]. On one hand, there are several environment uncontrolled
variables, such as water turbidity, uneven illumination, and low contrast. On
the other, also intrinsic fish characteristics contributes to the task
difficulty, including intra-class variations, color similarity with the
background, occlusions, and arbitrary unusual swimming positions. Besides, the
rich biodiversity adds complexity to creating a universal dataset for marine
ecosystem observation, limiting model applicability across different
underwater areas. This is in contrast with the ML requirement for great
quantity of good quality training data, and existing datasets, as summarized
by [14], often fall short in addressing this requirement. For instance,
Fish4Knowledge [19] and Rockfish [11] do not tackle the object detection
challenge, mainly addressing it by applying classic post-processing cropping
methods, so that each fish results centered and can be identified with a mere
object classification approach. The DeepFish [51] dataset surpasses the
previous ones by also offering a semantic segmentation of fish instances, i.e.
the annotation of every pixel belonging to the fish, denoted as mask. A
Seagrass dataset was used in [15] to demonstrate that ML approaches result in
an increased abundance detection compared to human experts. However, all
previous dataset presents limitations, either due to poor image resolution
(Fish4Knowledge) or because they have a low number of fishes per image (1 at
most for DeepFish and RockFish, not more than 18 on Seagrass).

The OzFish [44] dataset has significantly increased the number of annotated
boxes per image, reaching up to hundreds. As stated in [56], this comes with
incorrect or missing bounding box annotations. Examples include overlapped
annotations on the same target, negligible small bounding boxes, and
mislabeling of objects like reefs and floating ropes annotated as fish.
Despite these imperfections, the data remains valuable, since the images are
more extensively labeled compared to previous datasets, and they also exhibit
greater variability in environments and fish positioning, allowing us to
address challenges related to turbidity and overlapping. In particular, OzFish
consists of around $1\,800$ images and containing around $45\,000$ bounding
box annotations. We partitioned the images using an 85% to 15% ratio,
resulting in 1 500 images for the training set and 265 images for the test
set.

(a)

(b)

(c)

Figure 3: Fish instances are positioned in identical bounding boxes using
three different methods: employing as the GLIGEN grounding entities the text
phrase "a fish" (a) or an image of a real fish, as shown in the box in the top
right corner of (b); pasting DeepFish masks onto an OzFish background (c).

For the generated dataset, we decided to explore two distinct approaches,
comparing a grounding instruction that relies on text descriptions with
another one where example images are used. In the former, each target
grounding entity is defined simply by the phrase "a fish" (an example is
visible in Fig. 3(a)); in the latter, we employ as grounding entities one
image selected from a set of $16$ close-ups of different fish species from the
training set (as in Fig. 3(b)). In both cases, one sample from a set of $128$
training pictures without fishes is employed to condition the background style
of a single image. The chosen prompt is "blurry fishes in motion, foggy
underwater professional photography". In Sec. 6.2, we will compare these two
kinds of generated dataset with a common state-of-the-art technique, where a
synthetic image is created by copying masks of the real objects and pasting
them onto a real background, as in Fig. 3(c). To achieve this, we leverage the
DeepFish dataset that provides semantic segmentations of fishes. We obtained
$207$ masks of around $10$ different species, and we pasted them on the same
$128$ images used for the style transfer in the previous approaches. We note
that, despite the diversity between OzFish and DeepFish datasets, both feature
tropical Australian species. For these three types of approaches, we defined
identical desired bounding box positions for all the fish instances to ensure
more comparable training scenarios. However, as discussed in Sec. 4, the
generation model might struggle to generate all the targets, particularly in
cases of overlap, which is a challenge not encountered by the copy-paste
method (as an example of this issue, compare the fishes in the bottom left
corner between the three images in Fig. 3). Finally, we observe that the fish
instances in these three datasets exhibit increasing levels of complexity.
Creating a fish with the first method merely requires a text phrase. The
second approach demands capturing well-centered pictures of the real targets
of interest, which is still a relatively simple task. The third one needs
semantic masks of the objects, making it the most laborious method, involving
more human intervention than the other two.

For all the aforementioned datasets, both real and generated, we exclude the
use of small bounding boxes, specifically ignoring those with an area that is
less than 0.2% of the image area. Indeed, it is well-known in the literature
that generative models struggle with generating small targets [4], a problem
that falls into another research field and is beyond the scope of this study.

To generate the datasets, we have used two computational servers: one with 88
Intel Xeon E5-2699 v4 CPU, 94 Gb of RAM and a Nvidia TITAN RTX GPU with 24 Gb
of memory; the second one with 32 AMD Ryzen 9 5950X CPU, 126 Gb of RAM and a
Nvidia GeForce RTX 3090 GPU with 24 Gb of memory. The execution time to
generate each single image is around 37 seconds on the first machine and 21
seconds on the second one.

##  6 Results and discussion

To validate our pipeline, we have implemented it in PyTorch, and we have used
the same servers specified in Sec. 5 to train several models on both NuImages
and OzFish datasets. The training times are around 0.7 seconds per iteration
on the first machine, and 0.3 seconds per iteration on the second one. In the
following, the details of our analyses are explained. In Fig. 6 we show some
examples of prediction in both scenarios.

###  6.1 NuImages—reducing the need for real data

Figure 4: Results of the object detection mAP evaluated on the same NuImages
test set, illustrating the impact of pretraining with varying numbers of
generated images (x-axis) and subsequent training with different quantities of
real images from the NuImages training set (curve color and marker). Solid
lines represent results with pretraining on filtered images, while dashed
lines depict pretraining on unfiltered images.

On NuImages, we show the effectiveness of our transfer learning approach by
evaluating the mAP performances of our object detector under different setup
conditions, all tested on the same set of 9 073 real images. We also examine
the impact of filters on the model training. The Faster R-CNN is pretrained
using varying amounts of our generated images—specifically, 750, 1 500, 3 000,
4 500, 6 000, 9 000 and 12 000. For comparison, we report the case where the
generated images are not used during pretraining. During this phase, the
previously specified quantities correspond to an 85% of the total generated
images used for each configuration, while the remaining 15% is kept for
validation and scheduling purposes. For instance, the second configuration is
trained on 1 500 images from the generated set and validated on 265 other
images. Subsequently, each of these pretrained models undergoes fine-tuning
with various amounts of real images—namely, 50, 300, 750, 1 500, and 4 500. We
also considered a model only pretrained on generated images, without any fine-
tuning on the real dataset.

In Fig. 4, the x-axis represents the number of generated images used for the
initial pretraining, while different colors and markers indicates the numbers
of real images used for the fine-tuning step. The y-axis shows the
corresponding mAP performances on the real test set for each configuration.
Besides, to evaluate the usefulness of our filter, we followed two distinct
approaches: one involving the selection of generated images through a
Precision-Recall filter (solid curves), and the other employing all generated
images, regardless of their quality (dashed curves). Specifically, for both
precision and recall, we set the filtering thresholds to 1, aiming to preserve
only the most accurate representations of the ground truth during the
generation process (this process discarded around 56% of all the images).

Several key observations can be drawn. Unsurprisingly, increasing the number
of real images during the fine-tuning results in higher mAP values. In fact,
the curves where a higher number of real images is used, consistently appear
on top of the ones with fewer real images. The first observation regarding our
approach is that the integration of generated images successfully compensates
for the scarcity of real data. For instance, 300 real images combined with 9
000 generated images exhibit performance equivalent to the full dataset of 4
500 real images. More surprisingly, a combination of 300 real images and 750
filtered generated images yields better results than using only 1500 real
images. This finding is quite remarkable, suggesting that the contribution of
the pretraining is crucial to obtain a good starting point for the fine-tuning
of the weights with the real images. This assertion is also supported by the
curve indicated with 0 real images, where favorable mAP values can be achieved
solely with generated images, even without fine-tuning on real ones. For
example, training the detector with 9 000 filtered generated images is
comparable to the performance obtained on 1 500 real images. We assume that
the good impact of the generated images can be attributed to GLIGEN being
trained also with knowledge from the urban domain.

The second observation that we can deduce is that the filter effectively
condenses the information from a multitude of generated images into few
important ones. Interestingly, the filtered images do not allow for better
performance over the unfiltered ones in the long run. In fact, the mAP
converges to similar values when pretraining with 12 000 images, both filtered
and unfiltered. We make the hypothesis that, for large numbers of generated
training images, the model autonomously learns how to ignore the imperfect
labeling occurring in the generated images. Conversely, when the size of the
training set is small, the model may overfit on these inaccurate data.

In another test, we also evaluate the influence of the FID score on the
training of the object detector. We arranged the unfiltered generated images
into three mutually exclusive subsets with varying FID scores: 47, 105, and
148. Each subset consisted of 3 000 images for pretraining, with an additional
530 images used for validation. All these configurations are then involved in
a fine-tuning step on the same 300 real images of Fig. 4. At the end, we found
that there are no significant improvements in training the detector on
datasets with different FID scores. Specifically, for all three cases we
obtained really similar test detection mAPs comparable with the results of
Fig. 4. This is an evidence that, even if the FID score serves as a valuable
metric for assessing the overall similarity between real and generated
distributions, it may not be the most suitable metric to measure the impact of
generated data in object detection tasks. Specifically, the FID calculates
distribution distances considering the images as a whole, and not particularly
regarding the quality of each single target. For example, we can
hypothetically consider two distinct detection datasets having exactly
identical images, but one with the correct labeling and the other one with a
completely erroneous one. In this scenario, the FID distance between the two
datasets would be 0, but their impact on the training of an object detector
would be completely different, catastrophic for the wrongly labelled dataset.

###  6.2 OzFish—ablation study on different pretraining datasets

Figure 5: Object detection mAP results on the OzFish real test set, for models
pretrained on different quantities of unfiltered generated images (x-axis) and
fine-tuned on varying numbers of OzFish training images (specified by the
curve color and marker). Dotted curves indicate models pretrained on synthetic
copy-paste images. Dashed and solid lines represent the use of images and text
as grounding entities, respectively. The black constant solid line reports for
reference the mAP value of a model pretrained on COCO and fine-tuned on all
the 1 500 OzFish training images.

In this section we discuss the application of our pipeline to the domain of
OzFish. We present a comparison between our method and other prominent state-
of-the-art approaches. Moreover, in this domain we decided not to focus on the
effects of the filter because we observed a fast convergence already using the
unfiltered images.

We employed the three datasets outlined in Sec. 5 and exemplified in Fig. 3.
Similar to the generated dataset used for the NuImages scenario, we maintained
an 85% of the generated images for pretraining and reserved the remaining 15%
for validation. As in Fig. 4, in Fig. 5 we plot the mAP trend for an
increasing number of pretraining generated images (x-axis) and fine-tuning
real images (curve color). For all cases, the mAP is computed on the same real
test set of 265 OzFish images.

The first insightful observation arises when comparing various image
generation strategies. We observe that each method’s curve appears completely
above the previous ones. Specifically, the copy-paste approach (dotted curve)
exhibits the highest performance, followed by the cases where grounding
entities are images (dashed) and text (solid). This is probably because the
copy-paste dataset incorporates true fishes features, which may prove more
effective than the generated ones. However, it is crucial to note that the
curves appear to be sorted by an increasing level of their generation
complexity. It is important to recall that the copy-paste approach synthesizes
images by relying on mask segmentation, a process that is inherently resource-
intensive, even more than labeling a dataset for object detection. In
contrast, our methods stand out for their simplicity, requiring only target
pictures for image grounding or even simpler text descriptions. For instance,
under the configuration with 9 000 pretraining images and 300 real images, the
mAP gap between text grounding and copy-paste methods is just 0.05, which is
quite tolerable, considering the substantial reduction in the burden on human
intervention.

To further investigate the contribution of fish features to mAP performance,
we calculate the FID values between each of these datasets and the real Ozfish
dataset, resulting in values of 70, 90, and 100 for the copy-paste, image
grounding, and text grounding approaches, respectively. Indeed, the copy-paste
method exhibits the lowest FID (70), indicating a feature distribution closest
to that of OzFish. In this scenario, fine-tuning with 50 OzFish images (blue
curve) shows a 0.1 mAP gap between copy-paste and text grounding pretraining,
suggesting that datasets with lower FID values contribute more effectively to
the object detector pretraining. This finding appears to differ from the FID
filter results discussed in Sec. 6.1. In that context, three datasets were
created filtering out images generated with the same method, using identical
text grounding entities and prompts, resulting in varying FID values (47, 105,
148). However, used during detector pretraining, these datasets yielded
comparable mAP values, implying that FID was not effective in filtering
detrimental images. This apparently contrasting result might find an
explanation by considering that the FID can be influenced by many factors
whose impact on the mAP is not necessarily equal. For instance, image
background and targets may have features that do not hold the same weight in
the task of pretraining an object detector. It is essential to note that our
detector has a frozen backbone and is exclusively trained on the RPN detection
portion, which focuses solely on features within the target’s bounding boxes
and not on the background. In the NuImages case, where target features are
generated with the same instructions, FID differences probably arise mainly
from variations in background features, irrelevant to the detection task.
Therefore, using a FID-based filter on those generated datasets may discard
features not relevant for the RPN training. Conversely, in the case of OzFish,
the three datasets inherently contain differently generated targets, resulting
in FID differences that may also be caused by variations in the targets’
features, contributing to both FID variation and RPN training performances.

Finally, in Fig. 5 we also establish a reference configuration whose weights
are initialized based on the COCO dataset as in [48]. This allows to benchmark
the performances of our generated datasets against the most prominent state-
of-the-art dataset for the pretraining of any object detector. In detail, the
reference model is pretrained on ImageNet and fine-tuned on COCO. We
additionally fine-tune it on the whole OzFish dataset, keeping also in this
case the backbone frozen as detailed in Sec. 3. Our model surprisingly
approaches the performance of the COCO-pretrained reference (black line in
Fig. 5) with only 300 OzFish examples (red curve), reducing the number of real
labeled images by almost three orders of magnitude: from 118K COCO images plus
1 500 OzFish images to only 300 OzFish images supported by 9 000 generated
ones. Besides, with this specific training setting, when using the entire
dataset of 1 500 real images, we manage to surpass the reference with only 1
500 generated images (green curve). These are insights that the generated data
is better suited for the transfer learning task in this context, compared to
the general-purpose COCO dataset. This is encouraging for the widespread
adoption of generative models in object detection tasks.

(a)

(b)

(c)

(d)

Figure 6: Side-by-side illustrations comparing examples of object detector
predictions (in red) with the corresponding testing ground truth (in green)
under our two scenarios. In the left column (a, c), Faster R-CNN models are
initialized with ImageNet and fine-tuned using a minimal dataset of 300 real
training images. On the right, models undergo a preliminary pretraining on 9
000 generated images (from the image grounding dataset in the (d) case),
followed by a fine-tuning on the same 300 real images. We can observe that our
approach shows increased performances in finding more targets and refining
redundant detection boxes, even in rainy or turbid environments.

##  7 Conclusion

We have proposed a method that leverages a generative model pretrained on
extensive and diverse datasets, to generate images for the pretraining of an
object detector and subsequent transfer learning to the real domain. We have
demonstrated remarkable results on both NuImages and OzFish datasets. We have
showed that our method allows to achieve good detection performance with just
few hundreds of real images, comparable to training the model on thousands of
real images. Furthermore, our method significantly reduces the human effort
involved in the annotation process, limiting the need for labor-intensive
labeling tasks.

During the preparation of our manuscript, an independent paper has appeared
online as preprint in Ref. [7], exploring the potential of training an object
detector using generated images. However, their generative model was trained
on a substantial amount of real data from a specific dataset, thus without
addressing the issue of limited datasets from a generic domain that we are
considering here. Besides, while via their model (GeoDiffusion) they primarily
report the FID as the standard metric to evaluate generative models, we here
also explore how FID impacts on mAP. While our findings are in their
preliminary stages and comprise hypotheses, they open new possibilities for
understanding the relationship between FID and mAP, hence providing an
intriguing direction for further exploration and investigation.

Future works may include additional ablation studies. Firstly, we can further
evaluate whether the primary factor affecting FID and mAP performances arises
from the overall quality of the generated image, including the background, or
solely from that of the targets. Secondly, we can explore the influence of
style transfer by not providing to the generative model any reference picture,
relying solely on its pretrained knowledge. Finally, a comprehensive
investigation into the effectiveness of the precision-recall filtering
strategy, and the implications of employing a flawed ground truth during the
training, demands further studies.

In conclusion, our method offers a novel and efficient solution to tackle the
data scarcity challenge in object detection, paving the way for further
exploration of generative models applications in this field. This can be
useful in specific domains where the gathering and labeling of data can be
problematic or costly, such as, among others, medicine and molecular biology,
atmospheric monitoring, material science, space imaging, and non-human-
friendly environments.

##  8 Acknowledgments

M.P. acknowledges the contribution of the MAREA project funded by the Tuscany
Region. S.M. acknowledges financial support from PNRR MUR project
PE0000023-NQSTI. C.G. acknowledges the contribution of the National Recovery
and Resilience Plan, Mission 4 Component 2 – Investment 1.4 – CN_00000013
“CENTRO NAZIONALE HPC, BIG DATA E QUANTUM COMPUTING”, spoke 6. C.G. and M.P.
are members of the INdAM research group GNCS. The INdAM-GNCS support is
gratefully acknowledged. F.C. acknowledges financial support by the European
Commission’s Horizon Europe Framework Programme under the Research and
Innovation Action GA n. 101070546-MUQUABIS, by the European Union’s Horizon
2020 research and innovation programme under FET-OPEN GA n. 828946-PATHOS, by
the European Defence Agency under the project Q-LAMPS Contract No. B PRJ-
RT-989, and by the MUR Progetti di Ricerca di Rilevante Interesse Nazionale
(PRIN) Bando 2022 - project n. 20227HSE83 (ThAI-MIA) funded by the European
Union–Next Generation EU.

## References

  * [1] Hüseyin Gökhan Akçay, Bekir Kabasakal, Duygugül Aksu, Nusret Demir, Melih Öz, and Ali Erdoğan.  Automated bird counting with deep learning for regional bird distribution mapping.  Animals, 10(7):1207, 2020.
  * [2] Carlos Arteta, Victor Lempitsky, and Andrew Zisserman.  Counting in the wild.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 483–498, Cham, 2016. Springer International Publishing.
  * [3] Victor Besnier, Himalaya Jain, Andrei Bursuc, Matthieu Cord, and Patrick Pérez.  This dataset does not exist: training models from generated images.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2020.
  * [4] Brais Bosquet, Daniel Cores, Lorenzo Seidenari, Víctor M Brea, Manuel Mucientes, and Alberto Del Bimbo.  A full data augmentation pipeline for small object detection based on generative adversarial networks.  Pattern Recognition, 133:108998, 2023.
  * [5] William A Brownell.  Theoretical aspects of learning and transfer of training.  Review of Educational Research, 6(3):281–290, 1936.
  * [6] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020.
  * [7] Kai Chen, Enze Xie, Zhe Chen, Lanqing Hong, Zhenguo Li, and Dit-Yan Yeung.  Integrating geometric control into text-to-image diffusion models for high-quality detection data generation via text prompt.  arXiv preprint arXiv:2306.04607, 2023.
  * [8] Kenneth Ward Church, Zeyu Chen, and Yanjun Ma.  Emerging trends: A gentle introduction to fine-tuning.  Natural Language Engineering, 27(6):763–778, 2021.
  * [9] Alberto Maximiliano Crescitelli, Lars Christian Gansel, and Houxiang Zhang.  NorFisk: fish image dataset from Norwegian fish farms for species recognition using deep neural networks.  Modeling, Identification and Control, 42(1):1–16, 2021.
  * [10] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah.  Diffusion models in vision: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):10850–10869, 2023.
  * [11] George Cutter, Kevin Stierhoff, and Jiaming Zeng.  Automated detection of rockfish in unconstrained underwater videos using haar cascades and a new image dataset: Labeled fishes in the wild.  In 2015 IEEE Winter Applications and Computer Vision Workshops, pages 57–62. IEEE, 2015.
  * [12] Oscar Day and Taghi M Khoshgoftaar.  A survey on heterogeneous transfer learning.  Journal of Big Data, 4:1–42, 2017.
  * [13] S Deepak and PM Ameer.  Brain tumor classification using deep cnn features via transfer learning.  Computers in biology and medicine, 111:103345, 2019.
  * [14] Ellen M Ditria, Rod M Connolly, Eric L Jinks, and Sebastian Lopez-Marcano.  Annotated video footage for automated identification and counting of fish in unconstrained seagrass habitats.  Frontiers in Marine Science, 8:629485, 2021.
  * [15] Ellen M Ditria, Sebastian Lopez-Marcano, Michael Sievers, Eric L Jinks, Christopher J Brown, and Rod M Connolly.  Automating the analysis of fish abundance using object detection: optimizing animal ecology with deep learning.  Frontiers in Marine Science, page 429, 2020.
  * [16] Debidatta Dwibedi, Ishan Misra, and Martial Hebert.  Cut, paste and learn: Surprisingly easy synthesis for instance detection.  In Proceedings of the IEEE international conference on computer vision, pages 1301–1310, 2017.
  * [17] Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R Arabnia.  A brief review of domain adaptation.  Advances in data science and information engineering: proceedings from ICDATA 2020 and IKE 2020, pages 877–894, 2021.
  * [18] Di Feng, Christian Haase-Schütz, Lars Rosenbaum, Heinz Hertlein, Claudius Glaeser, Fabian Timm, Werner Wiesbeck, and Klaus Dietmayer.  Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges.  IEEE Transactions on Intelligent Transportation Systems, 22(3):1341–1360, 2020.
  * [19] Robert B Fisher, Yun-Heh Chen-Burger, Daniela Giordano, Lynda Hardman, Fang-Pang Lin, et al.  Fish4Knowledge: collecting and analyzing massive coral reef fish video data, volume 104.  Springer, 2016.
  * [20] Marco Francescangeli, Simone Marini, Enoc Martínez, Joaquín Del Río, Daniel M Toma, Marc Nogueras, and Jacopo Aguzzi.  Image dataset for benchmarking automated fish detection and classification algorithms.  Scientific data, 10(1):5, 2023.
  * [21] Adrien Gaidon, Qiao Wang, Yohann Cabon, and Eleonora Vig.  Virtual worlds as proxy for multi-object tracking analysis.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4340–4349, 2016.
  * [22] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  The International Journal of Robotics Research, 32(11):1231–1237, 2013.
  * [23] Daniel George, Hongyu Shen, and E. A. Huerta.  Classification and unsupervised clustering of ligo data with deep transfer learning.  Phys. Rev. D, 97:101501, May 2018.
  * [24] Harshvardhan GM, Mahendra Kumar Gourisaria, Manjusha Pandey, and Siddharth Swarup Rautaray.  A comprehensive survey and analysis of generative models in machine learning.  Computer Science Review, 38:100285, 2020.
  * [25] Adrian Salazar Gomez, Erchan Aptoula, Simon Parsons, and Petra Bosilj.  Deep regression versus detection for counting in robotic phenotyping.  IEEE Robotics and Automation Letters, 6(2):2902–2907, 2021.
  * [26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [27] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in neural information processing systems, 30, 2017.
  * [28] Stefan Hinterstoisser, Vincent Lepetit, Paul Wohlhart, and Kurt Konolige.  On pre-trained image features and synthetic images for deep learning.  In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0–0, 2018.
  * [29] Sheng-Wei Huang, Che-Tsung Lin, Shu-Ping Chen, Yen-Yi Wu, Po-Hao Hsu, and Shang-Hong Lai.  Auggan: Cross domain adaptation with gan-based data augmentation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 718–731, 2018.
  * [30] Minyoung Huh, Pulkit Agrawal, and Alexei A Efros.  What makes imagenet good for transfer learning?  arXiv preprint arXiv:1608.08614, 2016.
  * [31] Mohammadreza Iman, Hamid Reza Arabnia, and Khaled Rasheed.  A review of deep transfer learning and recent advancements.  Technologies, 11(2), 2023.
  * [32] Yuko Ishiwaka, Xiao S. Zeng, Michael Lee Eastman, Sho Kakazu, Sarah Gross, Ryosuke Mizutani, and Masaki Nakada.  Foids: Bio-inspired fish simulation for generating synthetic datasets.  ACM Trans. Graph., 40(6), dec 2021.
  * [33] Aayush Jaiswal, Neha Gianchandani, Dilbag Singh, Vijay Kumar, and Manjit Kaur.  Classification of the covid-19 infected patients using densenet201 based deep transfer learning.  Journal of Biomolecular Structure and Dynamics, 39(15):5682–5689, 2021.
  * [34] Dipendra Jha, Kamal Choudhary, Francesca Tavazza, Wei-keng Liao, Alok Choudhary, Carelyn Campbell, and Ankit Agrawal.  Enhancing materials property prediction by leveraging computational and experimental data using deep transfer learning.  Nature communications, 10(1):5316, 2019.
  * [35] Charles Hubbard Judd.  Psychology of high-school subjects.  Ginn, 1915.
  * [36] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee.  Gligen: Open-set grounded text-to-image generation.  2023\.
  * [37] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [38] Feng Liu, Guangquan Zhang, and Jie Lu.  Heterogeneous domain adaptation: An unsupervised approach.  IEEE transactions on neural networks and learning systems, 31(12):5588–5602, 2020.
  * [39] Simone Marini, Emanuela Fanelli, Valerio Sbragaglia, Ernesto Azzurro, Joaquin Del Rio Fernandez, and Jacopo Aguzzi.  Tracking fish abundance by underwater image recognition.  Scientific reports, 8(1):13748, 2018.
  * [40] Daniel Marrable, Kathryn Barker, Sawitchaya Tippaya, Mathew Wyatt, Scott Bainbridge, Marcus Stowar, and Jason Larke.  Accelerating species recognition and labelling of fish from underwater video with machine-assisted deep learning.  Frontiers in Marine Science, 9, 2022.
  * [41] Hajer Mguidich, Bachir Zoudji, and Aïmen Khacharem.  Does imagination enhance learning? a systematic review and meta-analysis.  European Journal of Psychology of Education, pages 1–36, 2023.
  * [42] Merlin Monzel, Kristof Keidel, and Martin Reuter.  Imagine, and you will find–lack of attentional guidance through visual imagery in aphantasics.  Attention, Perception, & Psychophysics, 83:2486–2497, 2021.
  * [43] Merlin Monzel and Martin Reuter.  Where’s wanda? the influence of visual imagery vividness on visual search speed measured by means of hidden object pictures.  Attention, Perception, & Psychophysics, pages 1–6, 2023.
  * [44] Australian Institute of Marine Science (AIMS), University of Western Australia (UWA), and Curtin University.  Ozfish dataset - machine learning dataset for baited remote underwater video stations, 2019.  https://doi.org/10.25845/5e28f062c5097 [Accessed: 5-Feb-2024].
  * [45] R. Padilla, S. L. Netto, and E. A. B. da Silva.  A survey on performance metrics for object-detection algorithms.  In 2020 International Conference on Systems, Signals and Image Processing (IWSSIP), pages 237–242, 2020.
  * [46] Malte Pedersen, Daniel Lehotský, Ivan Nikolov, and Thomas B. Moeslund.  Brackishmot: The brackish multi-object tracking dataset.  In Rikke Gade, Michael Felsberg, and Joni-Kristian Kämäräinen, editors, Image Analysis, pages 17–33, Cham, 2023\. Springer Nature Switzerland.
  * [47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [48] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28, 2015.
  * [49] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115:211–252, 2015.
  * [50] Lars Ruthotto and Eldad Haber.  An introduction to deep generative modeling.  GAMM-Mitteilungen, 44(2):e202100008, 2021.
  * [51] Alzayat Saleh, Issam H Laradji, Dmitry A Konovalov, Michael Bradley, David Vazquez, and Marcus Sheaves.  A realistic fish-habitat dataset to evaluate algorithms for underwater visual analysis.  Scientific Reports, 10(1):14671, 2020.
  * [52] Connor Shorten and Taghi M Khoshgoftaar.  A survey on image data augmentation for deep learning.  Journal of big data, 6(1):1–48, 2019.
  * [53] Xin Sun, Junyu Shi, Lipeng Liu, Junyu Dong, Claudia Plant, Xinhua Wang, and Huiyu Zhou.  Transferring deep knowledge for object recognition in low-quality underwater videos.  Neurocomputing, 275:897–908, 2018.
  * [54] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016.
  * [55] Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, Varun Jampani, Cem Anil, Thang To, Eric Cameracci, Shaad Boochoon, and Stan Birchfield.  Training deep networks with synthetic data: Bridging the reality gap by domain randomization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018.
  * [56] Ricardo JM Veiga, Iñigo E Ochoa, Adela Belackova, Luís Bentes, João P Silva, Jorge Semião, and João MF Rodrigues.  Autonomous temporal pseudo-labeling for fish detection.  Applied Sciences, 12(12):5910, 2022.
  * [57] Sébastien Villon, David Mouillot, Marc Chaumont, Emily S. Darling, Gérard Subsol, Thomas Claverie, and Sébastien Villéger.  A deep learning method for accurate and fast identification of coral reef fishes in underwater images.  Ecological Informatics, 48:238–244, 2018.
  * [58] Ling Yang, Yeqi Liu, Huihui Yu, Xiaomin Fang, Lihua Song, Daoliang Li, and Yingyi Chen.  Computer vision models in intelligent aquaculture with emphasis on fish detection and behavior analysis: A review.  Archives of Computational Methods in Engineering, 28:2785–2816, 2021.
  * [59] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang.  Diffusion models: A comprehensive survey of methods and applications.  ACM Comput. Surv., 56(4), nov 2023.
  * [60] Jiawei Zhang, Chen Li, Md Mamunur Rahaman, Yudong Yao, Pingli Ma, Jinghua Zhang, Xin Zhao, Tao Jiang, and Marcin Grzegorzek.  A comprehensive review of image analysis methods for microorganism counting: from classical image processing to deep learning approaches.  Artificial Intelligence Review, pages 1–70, 2022.
  * [61] Bo Zhao, Lili Meng, Weidong Yin, and Leonid Sigal.  Image generation from layout.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8584–8593, 2019.
  * [62] Tengyun Zhao, Zhencai Shen, Hui Zou, Ping Zhong, and Yingyi Chen.  Unsupervised adversarial domain adaptation based on interpolation image for fish detection in aquaculture.  Computers and Electronics in Agriculture, 198:107004, 2022.
  * [63] Yi Zheng, Yifan Zhao, Mengyuan Ren, He Yan, Xiangju Lu, Junhui Liu, and Jia Li.  Cartoon face recognition: A benchmark dataset.  In Proceedings of the 28th ACM international conference on multimedia, pages 2264–2272, 2020.
  * [64] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He.  A comprehensive survey on transfer learning.  Proceedings of the IEEE, 109(1):43–76, 2021.
  * [65] Zhengxia Zou, Keyan Chen, Zhenwei Shi, Yuhong Guo, and Jieping Ye.  Object detection in 20 years: A survey.  Proceedings of the IEEE, 111(3):257–276, 2023.

  *[ML]: Machine Learning
  *[GANs]: Generative Adversarial Network
  *[DDPMs]: Denoising Diffusion Probabilistic Model
  *[L2I]: Layout-to-Image
  *[RPN]: Region Proposal Network
  *[CNN]: Convolutional Neural Network
  *[SGD]: Stochastic Gradient Descent
  *[FID]: Frechet Inception Distance
  *[mAP]: mean Average Precision
  *[IoU]: Intersection over Union
  *[mAPs]: mean Average Precision


# A packaged whispering gallery resonator device based on an optical
nanoantenna coupler

Angzhen Li  1,2 Ke Tian  2 Jibo Yu  1,2 Rashmi A. Minz  3 Jonathan M. Ward  4
Samir Mondal  3 Pengfei Wang  1,5,* and Síle Nic Chormaic2  1Key Laboratory of
In-Fiber Integrated Optics of Ministry of Education, College of Science,
Harbin Engineering University, Harbin 150001, China
2Light-Matter Interactions for Quantum Technologies Unit, Okinawa Institute of
Science and Technology Graduate University, Onna, Okinawa 904-0495, Japan
3Central Scientific Instruments Organisation (Council of Scientific and
Industrial Research, India), Chandigarh 160030, India
4Physics Department, University College Cork, Cork, Ireland
5Key Laboratory of Optoelectronic Devices and Systems of Ministry of Education
and Guangdong Province College of Optoelectronic Engineering, Shenzhen
University, Shenzhen 518060, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In this work, we present the design and fabrication of a packaged whispering
gallery mode (WGM) device based on an optical nanoantenna as the coupler and a
glass microsphere as the resonator. The microspheres were fabricated from SiO2
fiber or Er3+-doped fiber, the latter creating a WGM laser with a threshold of
93 $\mu$W at 1531 nm. The coupler-resonator WGM device is packaged in a glass
capillary. The performance of the packaged microlaser is characterized, with
lasing emission both excited in and collected from the WGM cavity via the
nanoantenna. The packaged system provides isolation from environmental
contamination, a small size, and unidirectional coupling while maintaining a
high quality (Q-) factor ($\sim$108). It opens up new possibilities for
practical applications of WGM microdevices in a variety of fields such as low
threshold lasers, filters, and sensors.

††journal: osajournal††articletype: Research Article

##  1 Introduction

Although whispering gallery mode (WGM) devices have shown great potential in
many fields [1, 2, 3, 4, 5, 6, 7, 8], they have encountered obstacles in
practical applications. The optical coupler, a tool for transferring the light
in and out of the resonators, is a very important part of the WGM system.
Among the various coupling schemes, such as prisms [9] and angle-polished
fibers [10], tapered fibers [11] have been demonstrated to be ideal because of
their high coupling efficiency and ability to achieve critical coupling [12].
They can also be used to tailor the cavity input-output relations in the WGM
resonator-waveguide system [13]. However, there are still issues with the
tapered fiber coupling scheme, especially in terms of practical applications.
First, tapered fibers are typically less than a few $\mu$m in diameter, making
them extremely fragile. Second, the resonator-coupler system can be affected
by environmental factors such as air currents, which continuously and
irregularly change the coupling state. Dust in the environment can also
significantly reduce the quality (Q)-factors and efficiency of the coupling. A
packaging scheme using two glass plates and a tube to fix the microsphere and
tapered fiber can make the microsphere-tapered fiber system move as a whole
while maintaining the Q-factor ($1.08\times 10^{8}$ at 1550 nm). However, it
does not solve the problems of vulnerability and environmental pollution [14].
Packaging resonator-tapered fiber systems with low refractive index optical
glues have been shown to go some way toward solving the problem of damage and
contamination [15]. A low-index polymer packaged microtoroid-tapered fiber
device with a Q-factor up to $2\times 10^{7}$ at 780 nm has been demonstrated
[16]. However, reports have shown that this method can degrade the Q-factors
of the system [15, 17, 16, 18]. Also, limited by the length of the tapered
fiber, the size of such resonator-tapered fiber packaged devices is often on
the centimeter scale [14, 16, 18]. Another idea to further scale down the
devices is to place the resonator in a capillary tube to form an embedded WGM
device. For example, excitation and collection of the whispering gallery modes
can be achieved using an etched thin-walled capillary [19], a tapered hollow
annular core fiber [20], embedded dual core hollow fiber [21] or a femtosecond
laser-engraved waveguide [22] as a coupler. These embedded packaging solutions
offer advantages in terms of system stability, smaller size, and immunity to
environmental interference, but the Q-factors can be seriously affected.

Our previous work [23] demonstrated coupling to WGM resonators using a
nanoantenna based on Rayleigh scattering [24]. The nanoantenna effectively
excites the cavity to form WGM resonances and maintain high Q-factors. Good
coupling efficiency compared with other coupling methods based on Rayleigh
scattering can also be achieved. In this paper, we propose and demonstrate a
packaged WGM resonator with a fiber-based nanoantenna. This small footprint
packaged device addresses environmental contamination while maintaining very
high Q-factors of the order of $10^{8}$ at 1531 nm pump. In addition, the
nanoantenna structure allows for excitation and collection of signals using a
single fiber probe, showing considerable promise in terms of practicality.
Based on this scheme, we have also fabricated and packaged an Er3+-doped WGM
laser and the related performances are studied and discussed. The
demonstration of lasing in a packaged WGM device fabricated directly from Er-
doped fiber is important and avoids the necessity of coating via solgel [25,
26] or using other methods [27, 28, 29].

##  2 Device Fabrication

There are several techniques used to make spherical WGM resonators [30, 28,
12, 31, 32]. One of the more common methods involves melting the end of a
section of optical fiber, using, for example, a CO2 laser [12] and we used
this technique for the initial microsphere fabrication. Such microspheres have
a fiber rod at the pole perpendicular to the equatorial plane where the
resonance is formed. Although this rod does not affect the formation of the
resonance at the equator, it poses difficulties for packaging and
miniaturization of the device. To overcome this, here, we instigated an
additional fabrication step in order to have a half-tapered fiber as a
manipulation rod parallel to the resonance-forming surface. Figure 1
illustrates the process. The half-tapered fiber, with a small amount of UV
curing glue on the tip, and a microsphere were fixed on two 3D stages. Under a
microscope, the two 3D stages were adjusted to bring the tip of the half-
tapered fiber into contact with the connecting point of the microsphere and
its thin fiber rod (Fig. 1(a)). Next, the contact area was irradiated with UV
light for a few seconds to cure the UV glue (Fig. 1(b)). After curing, the
microsphere was torn from its original rod by moving the 3D stage. For this to
work, the half-tapered fiber must be thicker than the microsphere’s intrinsic
fiber rod to facilitate breakage when moving the stage. Now the microsphere
has a supporting rod parallel to the resonance plane (Fig. 1(c)). In this
experiment, the fiber used to prepare both the microsphere and the half-
tapered fiber was a commercial silica fiber (1060XP, Thorlabs).

Figure 1: Mounting a microsphere to a connecting rod. (a) Approach a half-
tapered optical fiber coated with UV glue to the connection point between the
microsphere and the connecting rod. (b) Fix the half-taper at the connection
point under UV light. (c) Tear the microsphere free from the original fiber
rod by adjusting the position of the 3D stage.

To fabricate the nanoantenna for optical coupling to the microsphere, we
chemically etched photosensitive optical fiber (GF3, Nufern), using a
technique developed by Mondal et al., as reported in [33, 34]. For this
particular fiber, the fiber core etches quicker than the cladding under the
same hydrofluoric (HF) acid etching conditions. The first step in the
preparation process was to remove the coating from one end of the fiber and
cut the end face flat. It was then secured in a vertical position with a
micrometer-adjustable fiber optic holder. A small plastic vial containing
48$\%$ HF acid solution was fixed directly below the optical fiber. An oil
film on the surface of the acid solution prevented evaporation. The etching
process was divided into two steps. First, the position of the fiber holder
was adjusted so that the end face of the fiber just touched the surface of the
acid. Under the action of liquid surface tension, the acid rose up along the
outer wall of the fiber and etched the cladding and core simultaneously (Fig.
2(a)). After 30 minutes, the core inside the fiber formed a tiny tip (Fig.
2(b)). The second step involved adjusting the holder to move the fiber down
into the acid by about 20 $\mu$m (Fig. 2(c)). The elevated height of the acid
level and the difference between the etching rates of the cladding and core
materials created a capillary ring at the core-cladding boundary.

After about 10 minutes of etching, an antenna with a protruding nanoprobe was
obtained. The etched sample was cleaned with acetone to remove the oil layer
attached to the surface of the nanoantenna. A scanning electron microscope
(SEM) image of a fabricated nanoantenna is shown in Fig. 2(d). Finally, the
nanoantenna and the microsphere were inserted into a capillary using 3D stages
(Fig. 3(a)). After adjusting their relative positions under a microscope,
while monitoring the WGM spectrum, the two sides of the capillary were sealed
with UV glue and irradiated by UV light (Fig. 3(b)). Figure 3(c) shows a
prepared sample with a microsphere diameter of 160 $\mu$m packaged in a
capillary with 350 $\mu$m outer diameter and 250 $\mu$m inner diameter. The
fiber on one side of the nanoantenna acts as a waveguide for the output and
input optical signals, while the pigtail fiber on the other side, which is
connected to the microsphere, can be cut off as needed.

Figure 2: Fabrication of a nanoantenna. (a) The end face of the fiber is
brought into contact with the HF acid surface. Under the action of liquid
surface tension, HF acid rises up along the outer wall of the fiber and etches
the cladding and core. (b) After 30 minutes, the core inside the fiber forms a
tiny tip. Because the core is etched faster than the cladding, the sample has
a ring tube formed by cladding that is longer than the tip. (c) Moving the
sample down 20 $\mu$m and continue etching for 10 minutes. (d) A scanning
electron microscope (SEM) image of a fabricated nanoantenna. Figure 3:
Packaging process. (a) The microsphere and nanoantenna are inserted into a
short capillary tube and adjusted in position. (b) The two sides of the
capillary are sealed with UV glue and irradiated by UV light (c) A packaged
device. The zoomed-in image shows a packaged structure in detail.

##  3 Results and Discussion

A schematic of the experimental setup to test the performance of the packaged
devices is shown in Fig. 4. Pump light at 1531 nm was coupled to the
microsphere via the nanoantenna. A 3-port circulator was used to separate the
input and collected signals. A half-wave plate on the input beam was used for
polarization adjustment. The signal was transmitted from the output of the
circulator and observed on an oscilloscope (WaveSurfer 10, Teledyne LeCroy)
and an optical spectrum analyzer (MS9740B, Anritsu).

Figure 4: Schematic of the test system. OSC: Oscilloscope; OSA: Optical
spectrum analyzer; HWP: Halfwave plate; Circ.: Circulator.

The pump source was a tunable laser (TLB 6700, Newport) operating near 1531
nm. Through excitation via the nanoantenna, we observed a WGM spectrum
containing multiple resonances (Fig. 5 (a)). Four typical resonances are
labeled in the spectrum as A, B, C, and D, and their details are shown in Fig.
5(b)-(e), respectively. We see that A is a typical Fano resonance ($Q_{\rm
A}=1.1\times 10^{8}$) formed by a continuous spectrum corresponding to the
field reflected from the surface of the sphere and a discrete spectrum
corresponding to the scattering cavity field [35]. The modes B, C, and D
correspond to three typical Lorentz peaks or dips and their Q-factors are
$Q_{\rm B}=1.0\times 10^{8}$, $Q_{\rm C}=1.7\times 10^{8}$ and $Q_{\rm
D}=0.6\times 10^{8}$. We see that the resonator maintains high Q-factors after
packaging.

Figure 5: WGM spectrum. (a) Reflection spectrum under a 1531 nm tunable laser
scan. (b)-(e) Details of the four labeled resonances in the reflection
spectrum.

We have also prepared an active packaged device using the same method as
described for the passive device but Er3+-doped fiber (M12-980-125, Thorlabs)
to make the microspheres. When 1531 nm pump light was coupled into the
resonator via the nanoantenna, lasing behavior was observed at 1558 nm. Figure
6(a) shows the output laser spectrum at different pump powers. The
relationship between the pump power and the lasing output power is shown in
Fig. 6(b). Note that the pump power is measured at the input end of the
nanoantenna fiber. The launched pump power at lasing threshold is less than
0.85 mW and the laser power shows a linear relationship with the pump power
above threshold. The launched pump power is of the same order as that recorded
for lasing in a WGM resonator by pumping via a Rayleigh scatter, that is 0.485
mW [36]. In our experiment, the laser displayed single-mode operation without
any sign of saturation when the pump power was increased to 5.63 mW. At this
input power the back reflected pump power measured at the input to the OSA
with a power meter was 7.7 $\mu$W. Assuming the input and output coupling
strengths are equal, this equates to a coupling efficiency of 11.4 $\%$. At
threshold, the input power can then be expected to be 93 $\mu$W. Threshold
could be lowered by locking the pump laser to a WGM or optimizing the pump
polarization and tip position.

Figure 6: Laser performance of an Er3+-doped packaged microsphere device. (a)
The laser spectrum at different 1531 nm pump powers. The lasing emission is
centered on 1558 nm. (b) Output laser power at 1558 nm as a function of 1531
nm pump power light. Blue dots: experimental data; Red line: linear fit. The
lasing threshold is at around 0.85 mW.

##  4 Conclusion

In conclusion, we have designed and fabricated a packaged WGM device based on
a fiber nanoantenna as a coupler for both a passive and an active microsphere
resonator. The device has the advantages of having a small footprint and good
resistance to environmental interference and pollution whilst maintaining high
Q-factors of the order of 108 for the passive devices. Since the Rayleigh
scattering coupling mechanism does not rely on phase matching [37] this scheme
could, in principle, be used to fabricate packaged WGM devices from a range of
different materials for the resonator, such as multi-component glass,
crystals, etc. Another important point to note is that the nanoantenna can
achieve Rayleigh-induced coupling without the need to deposit a scatterer onto
the surface of the WGM resonator which is a non-trivial task if one wishes to
do so in a deterministic manner. The point-and-play feature of the nanoantenna
simplifies the coupling method significantly. In summary, this work offers a
reliable way to provide environmental isolation, increased robustness, and
portability to WGM resonators. This will pave the way to move WGM resonators
from the laboratory to the field for a variety of applications such as low
threshold lasers, filters, and sensors.

## Funding

National Key Program of the Natural Science Foundation of China (NSFC)
(61935006, 62090062); China Scholarship Council (201906680065); The 111
project (B13015) to the Harbin Engineering University; Okinawa Institute of
Science and Technology Graduate University; Science and Engineering Research
Board, Department of Science and Technology, India, under Grant
CRG/2019/001215.

## Acknowledgments

The authors acknowledge the Engineering Support Section of OIST Graduate
University.

## Disclosures

The authors declare no conflicts of interest.

## References

  * [1] A. A. Savchenkov, A. B. Matsko, V. S. Ilchenko, I. Solomatine, D. Seidel, and L. Maleki, “Tunable optical frequency comb with a crystalline whispering gallery mode resonator,” Phys. Rev. Lett. 101, 093902 (2008).
  * [2] J. Zhu, S. K. Özdemir, Y. F. Xiao, L. Li, L. He, D. R. Chen, and L. Yang, “On-chip single nanoparticle detection and sizing by mode splitting in an ultrahigh-q microresonator,” Nature Photonics 4, 122–122 (2009).
  * [3] J. U. Fürst, D. V. Strekalov, D. Elser, A. Aiello, U. L. Andersen, C. Marquardt, and G. Leuchs, “Quantum light from a whispering-gallery-mode disk resonator,” Phys. Rev. Lett. 106, 113901 (2011).
  * [4] F. Monifi, Ş. Kaya Özdemir, and L. Yang, “Tunable add-drop filter using an active whispering gallery mode microcavity,” Applied Physics Letters 103, 181103 (2013).
  * [5] Y. Yang, Y. Ooka, R. M. Thompson, J. M. Ward, and S. Nic Chormaic, “Degenerate four-wave mixing in a silica hollow bottle-like microresonator,” Opt. Lett. 41, 575–578 (2016).
  * [6] Z. Fang, S. Nic Chormaic, S. Wang, X. Wang, J. Yu, Y. Jiang, J. Qiu, and P. Wang, “Bismuth-doped glass microsphere lasers,” Photon. Res. 5, 740–744 (2017).
  * [7] J. M. Ward, Y. Yang, F. Lei, X.-C. Yu, Y.-F. Xiao, and S. Nic Chormaic, “Nanoparticle sensing beyond evanescent field interaction with a quasi-droplet microcavity,” Optica 5, 674–677 (2018).
  * [8] L. T. Hogan, E. H. Horak, J. M. Ward, K. A. Knapper, S. Nic Chormaic, and R. H. Goldsmith, “Toward real-time monitoring and control of single nanoparticle properties with a microbubble resonator spectrometer,” ACS Nano 13, 12743–12757 (2019).
  * [9] Y.-L. Pan and R. K. Chang, “Highly efficient prism coupling to whispering gallery modes of a square $\mu$ cavity,” Applied Physics Letters 82, 487–489 (2003).
  * [10] R. Dong, A. J. Q. M.D., J. Liao, and L. Yang, “Angle-polished fibers as a robust coupling tool for photonics,” in _Chemical, Biological, Radiological, Nuclear, and Explosives (CBRNE) Sensing XX,_ vol. 11010 J. A. Guicheteau and C. R. Howle, eds., International Society for Optics and Photonics (SPIE, 2019), pp. 290 – 296.
  * [11] J. C. Knight, G. Cheung, F. Jacques, and T. A. Birks, “Phase-matched excitation of whispering-gallery-mode resonances by a fiber taper,” Opt. Lett. 22, 1129–1131 (1997).
  * [12] M. Cai, O. Painter, and K. J. Vahala, “Observation of critical coupling in a fiber taper to a silica-microsphere whispering-gallery mode system,” Phys. Rev. Lett. 85, 74–77 (2000).
  * [13] F. Lei, J. M. Ward, P. Romagnoli, and S. Nic Chormaic, “Polarization-controlled cavity input-output relations,” Phys. Rev. Lett. 124, 103902 (2020).
  * [14] Y. Dong, K. Wang, and X. Jin, “Packaged microsphere-taper coupling system with a high q factor,” Appl. Opt. 54, 277–284 (2015).
  * [15] Y.-Z. Yan, C.-L. Zou, S.-B. Yan, F.-W. Sun, Z. Ji, J. Liu, Y.-G. Zhang, L. Wang, C.-Y. Xue, W.-D. Zhang, Z.-F. Han, and J.-J. Xiong, “Packaged silica microsphere-taper coupling system for robust thermal sensing application,” Opt. Express 19, 5753–5759 (2011).
  * [16] V. Kavungal, G. Farrell, Q. Wu, A. K. Mallik, and Y. Semenova, “A packaged whispering gallery mode strain sensor based on a polymer-wire cylindrical micro resonator,” Journal of Lightwave Technology 36, 1757–1765 (2017).
  * [17] P. Wang, R. Madugani, H. Zhao, W. Yang, J. M. Ward, Y. Yang, G. Farrell, G. Brambilla, and S. Nic Chormaic, “Packaged optical add-drop filter based on an optical microfiber coupler and a microsphere,” IEEE Photonics Technology Letters 28, 2277–2280 (2016).
  * [18] G. Zhao, Şahin Kaya Özdemir, T. Wang, L. Xu, E. King, G.-L. Long, and L. Yang, “Raman lasing and Fano lineshapes in a packaged fiber-coupled whispering-gallery-mode microresonator,” Science Bulletin 62, 875 – 878 (2017).
  * [19] X. Q. Bai and D. N. Wang, “Whispering-gallery-mode excitation in a microsphere by use of an etched cavity on a multimode fiber end,” Opt. Lett. 43, 5512–5515 (2018).
  * [20] J. Wang, X. Zhang, M. Yan, L. Yang, F. Hou, W. Sun, X. Zhang, L. Yuan, H. Xiao, and T. Wang, “Embedded whispering-gallery mode microsphere resonator in a tapered hollow annular core fiber,” Photon. Res. 6, 1124–1129 (2018).
  * [21] M. Zhang, W. Yang, K. Tian, J. Yu, A. Li, S. Wang, E. Lewis, G. Farrell, L. Yuan, and P. Wang, “In-fiber whispering-gallery mode microsphere resonator-based integrated device,” Opt. Lett. 43, 3961–3964 (2018).
  * [22] X. Liu, X. L. Cui, and D. N. Wang, “Integrated in-fiber coupler for a whispering-gallery mode microsphere resonator,” Opt. Lett. 45, 1467–1470 (2020).
  * [23] J. M. Ward, F. Lei, S. Vincent, P. Gupta, S. K. Mondal, J. Fick, and S. Nic Chormaic, “Excitation of whispering gallery modes with a “point-and-play,” fiber-based, optical nano-antenna,” Opt. Lett. 44, 3386–3389 (2019).
  * [24] F. Shu, X. Jiang, G. Zhao, and L. Yang, “A scatterer-assisted whispering-gallery-mode microprobe,” Nanophotonics 7, 1455–1460 (2018).
  * [25] L. Yang and K. J. Vahala, “Gain functionalization of silica microresonators,” Opt. Lett. 28, 592–594 (2003).
  * [26] Y. Yang, F. Lei, S. Kasumie, L. Xu, J. M. Ward, L. Yang, and S. Nic Chormaic, “Tunable erbium-doped microbubble laser fabricated by sol-gel coating,” Opt. Express 25, 1308–1313 (2017).
  * [27] C. Dong, Y. Xiao, Z. Han, G. Guo, X. Jiang, L. Tong, C. Gu, and H. Ming, “Low-threshold microlaser in Er:Yb phosphate glass coated microsphere,” IEEE Photonics Technology Letters 20, 342–344 (2008).
  * [28] S. Chen, T. Sun, K. Grattan, K. Annapurna, and R. Sen, “Characteristics of Er and Er–Yb–Cr doped phosphate microsphere fibre lasers,” Optics Communications 282, 3765 – 3769 (2009).
  * [29] J. Ward, Y. Yang, and S. Nic Chormaic, “Glass-on-glass fabrication of bottle-shpaed tunable microlasers and their applications,” Scientific Reports 6, 25252 (2016).
  * [30] F. Lissillour, K. A. Ameur, N. Dubreuil, G. M. Stephan, and M. Poulain, “Whispering-gallery-mode Nd-ZBLAN microlasers at 1.05 $\mu$m,” in _Infrared Glass Optical Fibers and Their Applications,_ vol. 3416 M. Saad, ed., International Society for Optics and Photonics (SPIE, 1998), pp. 150 – 156.
  * [31] J. M. Ward, Y. Wu, K. Khalfi, and S. Nic Chormaic, “Short vertical tube furnace for the fabrication of doped glass microsphere lasers,” Review of Scientific Instruments 81, 073106 (2010).
  * [32] R. Madugani, Y. Yang, J. M. Ward, J. D. Riordan, S. Coppola, V. Vespini, S. Grilli, A. Finizio, P. Ferraro, and S. Nic Chormaic, “Terahertz tuning of whispering gallery modes in a PDMS stand-alone, stretchable microsphere,” Opt. Lett. 37, 4762–4764 (2012).
  * [33] S. K. Mondal, A. Mitra, N. Singh, S. N. Sarkar, and P. Kapur, “Optical fiber nanoprobe preparation for near-field optical microscopy by chemical etching under surface tension and capillary action,” Opt. Express 17, 19470–19475 (2009).
  * [34] S. K. Mondal, A. Mitra, N. Singh, F. Shi, and P. Kapur, “Ultrafine fiber tip etched in hydrophobic polymer coated tube for near-field scanning plasmonic probe,” IEEE Photonics Technology Letters 23, 1382–1384 (2011).
  * [35] F.-J. Shu, C.-L. Zou, and F.-W. Sun, “Perpendicular coupler for whispering-gallery resonators,” Opt. Lett. 37, 3123–3125 (2012).
  * [36] J. Zhu, S. K. Özdemir, H. Yilmaz, B. Peng, M. Dong, M. Tomes, T. Carmon, and L. Yang, “Interfacing whispering-gallery microresonators and free space light with cavity enhanced rayleigh scattering,” Scientific Reports 4, 6396 (2014).
  * [37] A. B. Matsko and V. S. Ilchenko, “Optical resonators with whispering-gallery modes-part i: basics,” IEEE Journal of Selected Topics in Quantum Electronics 12, 3–14 (2006).