# Not All Labels Are Created Equal: A Story of Labeling Cost for Object
Detection

Ismail Elezi1 Zhiding Yu2 Anima Anandkumar2,3 Laura Leal-Taixé1 Jose M.
Alvarez2
1TUM 2NVIDIA 3CALTECH
Work performed while interning at NVIDIA.

# Not All Labels Are Equal:
Rationalizing The Labeling Costs for Training Object Detection

Ismail Elezi1 Zhiding Yu2 Anima Anandkumar2,3 Laura Leal-Taixé1 Jose M.
Alvarez2
1TUM 2NVIDIA 3CALTECH
Work performed while interning at NVIDIA.

###### Abstract

Deep neural networks have reached high accuracy on object detection but their
success hinges on large amounts of labeled data. To reduce the labels
dependency, various active learning strategies have been proposed, typically
based on the confidence of the detector. However, these methods are biased
towards high-performing classes and can lead to acquired datasets that are not
good representatives of the testing set data. In this work, we propose a
unified framework for active learning, that considers both the uncertainty and
the robustness of the detector, ensuring that the network performs well in all
classes. Furthermore, our method leverages auto-labeling to suppress a
potential distribution drift while boosting the performance of the model.
Experiments on PASCAL VOC07+12 and MS-COCO show that our method consistently
outperforms a wide range of active learning methods, yielding up to a $7.7\%$
improvement in mAP, or up to $82\%$ reduction in labeling cost. Code will be
released upon acceptance of the paper.

##  1 Introduction

The performance of deep object detection networks [1, 2] depends heavily on
the size of the labeled dataset. Adding more labeled data helps, yet adding
more data costs. Therefore, it is imperative to adopt active learning (AL)
strategies to select the most informative samples in the dataset for labeling,
and self and semi-supervised learning (SSL) approaches to leverage unlabeled
data whenever possible.

Figure 1: An object from a low-performing class (Pottedplant), original image
in blue, augmented version in orange. a) Because of its low-entropy,
uncertainty-based AL methods do not label the image. b) Because of its high
confidence, pseudo-label SSL methods wrongly pseudo-label the image and thus
harm the training. c) Because of its high-inconsistency, consistency-based SSL
methods cannot learn from it. d) Our method selects the image for labeling and
prevents it from getting pseudo-labeled.

Consistency-based Semi-Supervised Learning (SSL) methods for object detection
[3] train a network to minimize the inconsistency between its predictions.
However, as shown in Fig. 1, some images still give inconsistent predictions,
and thus the network does not learn from them. Auto-labeling uses self-
learning to label high confident predictions, _i.e_. pseudo-label (PL), but,
since networks are miscalibrated, they can generate wrong labels, potentially
harming the training. Moreover, by targetting high-confident predictions, they
ignore objects of low-performing classes.

When samples cannot be pseudo-labeled, an alternative is to obtain the ground
truth via manual labeling. Active learning (AL) for object detection [4, 5] is
a common approach to select the most promising samples for labeling to reduce
labeling costs. The selection is based on an acquisition function to assess
the informativeness of an image, typically computed based on the network’s
uncertainty. However, the acquisition function is only meaningful if the
network is already well-trained for the task, which is not always the case,
especially in the early AL cycles. Even if the network performs well in most
classes, significant intra-class variance can lead to a low accuracy on a
particular class, see Fig. 1a. In those cases, using the network predictions
to compute the acquisition function can lead to worse performance than random
sampling. Furthermore, we change the dataset distribution at every AL cycle by
selecting only the most uncertain (hard) samples until they no longer resemble
the test distribution.

In this work, we advocate for a holistic view of the labeling problem, that
is, a unified strategy to choose which samples to manually label and which
samples can be automatically labeled. We start from an uncertainty-based AL
framework and generalize its acquisition function by introducing the concept
of robustness, which is commonly not present on AL. If a network’s predictions
of an image and its random augmentations, i.e., horizontal flipping, are not
consistent, the image needs to be manually labeled. This simple yet effective
change allows us to select informative samples for both low and high-
performing classes. This is unlike classic SSL settings, where samples with
inconsistent predictions would neither be labeled nor pseudo-labeled, hence,
the information contained in them would not be used.

We are still left with the potential dataset distribution drift, for which we
propose to use auto-labeling in order to not increase the labeling costs. For
every active learning cycle, we use the previously trained network to mine
easy samples, i.e., samples where the network is confident about its
prediction, and use the network’s own prediction as labels. Note, that easy
samples are typically not used in AL cycles. Only by holistically thinking
about which samples to manually label and which to auto-label can we take full
advantage of the entire dataset. In summary, our contributions are the
following:

  * •

We propose a novel class-agnostic active learning score based on the
robustness of the network, using a novel inconsistency score.

  * •

We use auto-labeling to leverage the less informative samples, expanding the
labeled dataset for free.

  * •

We demonstrate the benefits of our method in two publicly available datasets:
PASCAL VOC07+12 and MS-COCO. Compared to state-of-the-art methods [6, 4, 7, 8,
5], our approach yields up to a $7.7\%$ and $7\%$ relative mAP improvement for
PASCAL-VOC and MS-COCO, respectively. Importantly, we can achieve the same
performance as the baseline but reduce up to $82\%$ of the labeling costs.

Figure 2: Overview of our method. We first train the network in a semi-
supervised manner. During active learning, for each image, we use the network
to compute the acquisition function and based on it decide if we actively
label the image, if we pseudo-label it, or if we only use it as part of the
unlabeled data for the next training cycle.

##  2 Related Work

Deep Active Learning (AL) for Object Detection. The traditional way of doing
AL is by training an ensemble of neural networks [9] and then selecting the
samples with the highest score defined by some acquisition function, i.e.,
entropy [10], or BALD [11]. Concurrent works [12, 13] explore a similar
direction by approximating the uncertainty via Monte-Carlo dropout [14]. These
approaches are compared [9], with the authors concluding that the ensemble
approach reaches higher results at the cost of more computational power.
Another Bayesian approach [15] trains a variational autoencoder (VAE) [16] on
both real and augmented samples, and then chooses to label the samples with
the highest reconstruction error. A different approach is the core-set [6]
that chooses to label a set of points such that a model trained over the
selected subset is competitive for the remaining data points. In the work of
[17] the authors tailor the method of [18] for AL. In general, these methods
are designed for classification-based AL and can not be directly applied to
object detection.

Recently, several methods have been adapted specifically for the task of
object detection [19, 20, 21, 22, 6, 7], some of which are based on the core-
set approaches where the diversity of the training examples is taken into
account. However, the state-of-the-art approaches are based on the uncertainty
[23, 4, 8, 5]. The work of [23] consists of an ensemble of object detectors
that provide bounding boxes and probabilities for each class of interest.
Then, a scoring function is used to obtain a single value representing the
informativeness of each unlabeled image. Similar to that is the work of [8]
where the authors compute the instance-based uncertainty. Another work [4]
gives an elegant solution, reaching promising results compared with other
single-model methods. The authors train a network in the task of detection
while learning to predict the final loss. In the sample acquisition stage,
samples with the highest prediction loss are considered the most interesting
ones and are chosen to be labeled. In the state-of-the-art approach [5],
authors define the aleatoric and epistemic uncertainty, in both class and
bounding box level, and use the combined score to determine the images that
need labeling. Our work is related but different from the above-mentioned
works. Similarly, we consider the uncertainty of the detector as part of the
solution. Unlike them, we find that the robustness of the detector is even
more reliable as an acquisition function, especially for the low-performing
classes. We then unify these two scores to reach high performance in the
majority of classes.

Deep Semi-Supervised Learning (SSL) for Object Detection is a deep learning
approach that combines a small amount of labeled data with a large amount of
unlabeled data during neural network training. Unlike in AL, where the
unlabeled data is used only during the acquiring stage, in SSL, the unlabeled
data are used during the training. Several methods have shown excellent
results [24, 25, 26, 27] by casting the problem of semi-supervised learning as
a regularization problem, in effect adding a new loss for the unlabeled
samples. Follow-up works significantly improved the performance of SSL in
object classification [28, 18, 29, 30, 31, 32].

While going from semi-supervised image classification to semi-supervised
object detection is challenging, some promising directions are given in recent
works [3, 33, 34] where the authors develop loss functions that minimize the
inconsistency between images and their augmented version. The methods
significantly improved the mAP score in Pascal VOC dataset [35]. Our work is
inspired by [3] but we instead develop an acquisition function that computes
the inconsistency between an image and an augmented version of it, and show
that such a score is more reliable than the uncertainty, especially for low-
performing classes. Furthermore, we add a pseudo-labeling module that labels
the easy images for free. Together with our developed AL method, it ensures
that our acquired dataset is a good representative of the original dataset, in
turn, improving the results.

##  3 Method

Let $D$ be a dataset divided into a labeled set $L$ and a pool of unlabeled
data $U$. We describe our acquisition function for AL in Sec. 3.1. This
consists of mining a subset of samples from the pool of unlabeled data $U$ and
transferring them to the labeled set $L$, incurring a labeling cost. However,
arbitrarily augmenting the set $L$ with only hard samples creates a
distribution drift in our training data. Hence, we propose to include in
training the easy samples, i.e., objects for which the network’s confidence is
high, by using pseudo-labeling (Sec. 3.2). Finally, we describe the training
procedure in Sec. 3.3. We show a high-level description of the method in Fig.
2.

Notation. Let $\bm{\Delta}$ be the object predictions for an image, and let
$\bm{\Delta}_{i}$ be its i-th object prediction. $\bm{\Delta}_{i}$ consists of
the bounding box $\bm{b}_{i}$ and $\bm{c}_{i}$ represents the probability
distribution after the softmax layer of the neural network. We denote the
$p$-th category of the distribution with $\bm{c}_{i}^{p}$. The bounding box
$\bm{b}_{i}$ consists of the displacement of the center and scale
coefficients, represented by the tuple $[\bm{\delta x},\bm{\delta
y},\bm{w},\bm{h}]$. Given a weakly augmented version of the original image,
e.g., by doing a horizontal flip, we define $\bm{\hat{\Delta}}$ to be the set
of its object predictions, and $\bm{\hat{\Delta}_{i}}$ consisting of the
bounding box $\bm{\hat{b}}_{i}$ and $\bm{\hat{c}}_{i}$, its i-th prediction.

###  3.1 Inconsistency-based AL

Most AL methods use some measure of uncertainty, e.g., the entropy, to compute
the acquisition function. A prediction that has a high entropy suggests that
the object is highly dissimilar to the images the network is trained on. Thus,
if labeled, it will provide different information to the ones we have.
However, we empirically find that using only an uncertainty-based acquisition
function is not an ideal solution, especially for images coming from low-
performing classes. As we show in the experiments, if the network’s
predictions for a class are incorrect, they are also unreliable to compute the
acquisition function.

Inconsistency-based acquisition function. To solve this issue, we propose a
robustness-based score for AL based on the consistency between an image and
its augmented version. If the predictions from an image and its augmented
version are very similar, then we say that the network is robust for that
image. On the other side, images where the network is inconsistent provide
information that has not been captured in the training process and are prime
candidates to be labeled. By focusing on robustness, the method is class-
agnostic and performs well in most classes.

To measure the robustness of the network, we first define the inconsistency
acquisition function $\mathcal{L}_{con_{C}}$. To do so, we feed the image and
an augmented version of it to the detector. In our case, we use horizontal
flip as augmentation. Given the sets of predictions for the original and
augmented image, we first need to match the predictions $\bm{\Delta}$ with
$\bm{\hat{\Delta}}$. We do so by computing their intersection over union
(IoU):

$\bm{\Delta^{\prime}}_{i}=\text{argmax}_{\bm{b}_{i}\in\\{\bm{b}\\}}IoU(\bm{b}_{i},\bm{\hat{b}}_{i}).$
(1)

For each matched pair, $\bm{\Delta}^{\prime}_{i}$ and $\bm{\hat{\Delta}_{i}}$,
we define their inconsistency as:

$\mathcal{L}_{con_{C}}(\bm{c^{\prime}}_{i},\bm{\hat{c}}_{i})=\frac{1}{2}[KL(\bm{c^{\prime}}_{i},\bm{\hat{c}}_{i})+KL(\bm{\hat{c}}_{i},\bm{c^{\prime}}_{i})],$
(2)

where KL represents the Kullback-Leibler divergence. The higher the
inconsistency, the more informative the sample is for training and therefore
potentially worth labeling.

Aggregating object scores for image selection. Given $\mathcal{L}_{con_{C}}$,
the inconsistency for each object prediction in an image, we define the
inconsistency of an image by aggregating the scores over $\bm{\Delta}$.
Specifically, we first apply non-maximum suppression over its predictions, and
then, define its inconsistency as:

$I(\bm{\Delta})=max_{i}\\{\mathcal{L}_{con_{C}}(\bm{c^{\prime}}_{i},\bm{\hat{c}}_{i})\\}.$
(3)

Similarly, we define the uncertainty of the image as:

$H(\bm{\Delta})=max_{i}\\{H(\bm{c}_{i})\\},$ (4)

where $H(\bm{c}_{i})$ represents the entropy over distribution $\bm{c}_{i}$.
The intuition behind using the maximum score instead of some other score, such
as the average, is that labeling an image that has at least one difficult
object, independently of the number of easy objects is beneficial because of
the difficult object. Considering that the inconsistency and the uncertainty
scores are on different scales, we unify them by multiplying them. For
$\bm{\Delta}$, we formulate our unified acquisition score as:

$A(\bm{\Delta})=H(\bm{\Delta})\times I(\bm{\Delta}).$ (5)

Having scored each image in $U$, we then sort all the images based on their
acquisition score and select to label the $N/T$ images with the highest score,
where $N$ corresponds to the acquisition budget, and $T$ corresponds to the
number of active learning steps. Note that we annotate every bounding box that
belongs to a selected image regardless of whether the box has a high score or
not. We repeat this procedure for $T$ active learning cycles.

###  3.2 Pseudo-labeling to prevent distribution drift

The active learning pipeline described above targets the most informative
(hard) samples, ignoring the confident samples. We argue that the network
should see some representative easy samples in order to ensure that no
distribution drift happens. At the same time, we want to avoid labeling
confident samples to not spend labeling resources. Hence, we propose to use
pseudo-labeling, where the network trained in the previous AL cycle provides
pseudo-labels for the network that is being trained on this cycle. We pseudo-
label an object if the network is confident, the confidence is above some
threshold $\tau$:

$\hat{y}_{i}^{p}=\begin{cases}1,&\text{if }p=argmax(\bm{c}_{i})\text{ and
}\bm{c}_{i}^{p}\geq{\tau}\\\ 0,&\text{otherwise.}\end{cases}$ (6)

We then use the one-hot pseudo-labels as ground truth for the training of the
current network. We note that in an image, the network might be confident for
some predicted bounding box, and not confident for the others. As a toy
example, given an image containing a cat and a dog, the network might be
confident for the cat and pseudo-label it, but not confident for the dog. If
we do not consider this, the standard loss functions will penalize the
predictions of the network for the region of the image which contains the dog.
However, because the region is unlabeled, the ground truth of that object will
be set to background, in turn giving a high loss even if the network makes
accurate prediction (as dog). In the next section, we describe how to fix this
issue.

###  3.3 Deep Object Detection Training

In this section, we describe the different losses used in our unified
framework for training the deep object detector. First, we describe the
multibox, consistency, and pseudo-labeling losses and finally the overall
training loss.

Multibox loss for labeled samples. For the labeled images, the network is
trained with the standard MultiBox loss for class predictions, and a smooth
$\mathcal{L}\textit{1}$ loss for bounding box predictions. Given the network’s
class predictions $\bm{c}$ and an indicator $\bm{y_{ij}^{p}}=\\{0,1\\}$ for
matching the $i$-th box to its corresponding $j$-th ground truth box of
category $p$, the MultiBox loss is defined as [2]:

$\mathcal{L}_{conf}(\bm{c},\bm{y})=-\sum_{i\in
Pos}\sum_{p=1}^{|\text{classes}|}\bm{y}_{ij}^{p}log(\bm{c}_{i}^{p})-\sum_{i\in
Neg}log(\bm{c}_{i}^{0}),$ (7)

where $Pos$ defines positive bounding boxes (containing objects), $Neg$
defines bounding boxes of class background.

Consistency loss for unlabeled samples. Our approach leverages the
inconsistency of the detector in the acquisition function. Intuitively, if the
detector has high inconsistency in an image, it can not learn from it in a
self-supervised manner, and the only way to learn from that image is to label
it during the AL cycle. During training, we need to guide the detector to
provide consistent predictions. To this end, we mirror the active learning
procedure and feed an image and its augmented version to the detector using
horizontal flips. After matching the predictions, as described in Eq. 1, we
use the class acquisition function, $\mathcal{L}_{con_{C}}$, as the loss
function for class inconsistency. To stabilize the training, we compute the
localization inconsistency loss as [3]:

$\begin{split}\mathcal{L}_{con_{L}}(\bm{b^{\prime}}_{i},\bm{\hat{b}}_{i})=&\frac{1}{4}(||\bm{\delta
x^{\prime}}_{i}-(-\bm{\hat{\delta
x}}_{i})||^{2}+||\bm{y0^{\prime}}_{i}-\bm{\hat{y0}}_{i}||^{2}+\\\
&||\bm{w^{\prime}}_{i}-\bm{\hat{w}}_{i}||^{2}+||\bm{h^{\prime}}_{i}-\bm{\hat{h}}_{i}||^{2}),\end{split}$
(8)

where, as we use horizontal flipping, we apply the negation on the
displacement of the center $\bm{\hat{\delta x}}_{i}$.

We compute the total consistency loss by averaging the losses from all matched
pairs of predictions:

$\mathcal{L}_{con}=\operatorname*{\mathbb{E}}[{\mathcal{L}_{con_{C}}(\bm{c^{\prime}},\bm{\hat{c}})}]+\operatorname*{\mathbb{E}}[{\mathcal{L}_{con_{L}}(\bm{b^{\prime}},\bm{\hat{b}})}].$
(9)

Pseudo-labeling loss. Our approach only pseudo-labels those objects in an
image where the detector is highly confident, leaving the rest of the image
unlabeled. Using the loss described in Eq. 7 would cause problems for those
predictions in the regions where there are no pseudo-labels as they would be
considered false positives. We thus modify the MultiBox Loss as:

$\displaystyle\begin{split}\mathcal{L}_{conf}(\bm{c},\bm{y},\bm{\hat{y}})=&-\sum_{i\in
Pos}\sum_{p=1}^{|\text{classes}|}\bm{y}_{ij}^{p}log(\bm{c}_{i}^{p})\\\
&-\sum_{i\in
Neg}log(\bm{c}_{i}^{0})-\sum_{i\in\hat{Pos}}\sum_{p=1}^{|\text{classes}|}\bm{\hat{y}}_{ij}^{p}log(\bm{c}_{i}^{p}),\end{split}$

where $\hat{y}$ and $\hat{Pos}$ represent the indicator and the positive
bounding boxes for the pseudo-labels.

Overall Training loss. Finally, to train the deep detection network, we
aggregate the multibox, $\mathcal{L}\textit{1}$ and consistency losses as:

$\mathcal{L}_{total}=\mathcal{L}_{conf}+\mathcal{L}_{con}+\mathcal{L}\textit{1},$
(10)

where $\mathcal{L}_{con}$ is used in all samples, while $\mathcal{L}_{conf}$
and $\mathcal{L}\textit{1}$, the smooth $L_{1}$ for bounding boxes, are used
in the labeled and pseudo-labeled samples.

##  4 Experiments

In this section, we demonstrate the effectiveness of our approach to improve
the performance of object detection. For all experiments, we report mean
average precision @0.5 (mAP) as main metric, and use two public datasets:
PASCAL VOC07+12 (VOC07+12) [35] and MS-COCO train2014 [36]. VOC07+12 consists
of $16,551$ images for training, and $4,952$ testing images taken from VOC07
testset. MS-COCO consists of $83K$ images for training, and valset2017
contains $5,000$ images for testing.

Following [4, 5], on VOC07+12 we start by randomly sampling $2,000$ images. On
the larger MS-COCO, we start by randomly sampling $5,000$ images. We perform
$5$ active learning cycles, and in each cycle, we choose $1,000$ images to
label. To ensure that the network does not diverge, we define each mini-batch
to have half the images labeled. We set the confidence threshold for the
pseudo-label threshold to $\tau=0.99$ based on the results of the zeroth
active learning step in VOC07+12.

For a fair comparison with [6, 4, 7, 5], we use the Single-Shot Detector 300
(SSD300) [2] based on a VGG [37] backbone for all our experiments. We train
the model for $120,000$ iterations using SGD with momentum. We set the initial
learning rate to $0.001$ and divide it by $10$ after $80,000$ and $100,000$
iterations, respectively. We use batches of size $32$ and a constant L2
regularization parameter set to $0.0005$. We use the same model,
hyperparameters, and the same public
implementation111https://github.com/amdegroot/ssd.pytorch. We train all
networks using four NVIDIA V100 GPUs. In all experiments, we train three
independent networks using the same initial split of randomly sampled images
and report the mean. We give the exact numbers of the mean and standard
deviation in the supplementary material.

Figure 3: VOC07+12. Left: Comparison to state-of-the-art active learning
methods; Middle: Comparison to the two SSL methods used in this work when they
do not use AL; Right: Ablation study on the effect of entropy, inconsistency,
unified score without pseudo-labeling, and our method. $\dagger$ denotes
ensemble method; $\ddagger$ denotes mixture of SSD.

Figure 4: MS-COCO. Left: Comparison to state-of-the-art active learning
methods; Middle: Comparison to the two SSL methods used in this work when they
do not use AL; Right: Ablation study on the effect of entropy, inconsistency,
unified score without pseudo-labeling, and our method. $\dagger$ denotes
ensemble method; $\ddagger$ denotes mixture of SSD.

###  4.1 Main results: Comparison with other methods

We compare our method with two baselines, random and entropy sampling, in
addition to five state-of-the-art single-model methods: Coreset [6], Learning
Loss [4], CDAL-AL [7], MI-ALD [8] and PM [5]. The last method uses a mixture
of SSD, thus adds extra parameters. We also compare with two multimodel
approaches, MC-dropout [12] and ensemble-based [9] active learning (consisting
of three neural networks). Finally, we compare to the consistency-based SSL
method and to a pseudo-labeling method.

We present the comparisons with AL methods for VOC07+12 in Fig. 3a. We observe
that in the first active learning cycle, our method has a relative improvement
over the random baseline by $10.5\%$, and over $8.2\%$ compared to the best
overall active learning method [8]. We see that the performance improvement of
our method is maintained in the other active learning cycles. In the last one,
where we use $7,000$ samples, $5,000$ of which are actively labeled, our
method outperforms the random baseline by $9.1\%$ and the best existing active
learning methods by more than $2.8\%$ [5]. Multi-model active learning
networks, namely, ensemble [9] or MC-dropout [12] outperform single models at
the cost of longer training and active learning time, and in the case of the
ensemble has $3$ times more training parameters. Nonetheless, our proposed
single model still reaches better results than multi-model methods,
outperforming the ensembles by $8\%$ in the first AL cycle, and $1.8\%$ in the
last cycle. In Fig. 3b. we compare the results of our method, with the two
semi-supervised learning methods. In the first AL cycle, our method
outperforms the consistency-based SSL by $5.6\%$, and the pseudo-labeling
method by $2\%$. In the last cycle, we outperform the consistency method by
$3.4\%$ and the pseudo-labeling method by more than $3\%$.

For MS-COCO, in Fig. 4a-b, we observe that in the first active learning cycle,
our method outperforms the random baseline by $5.8\%$, the best-performing AL
method by $2.7\%$ [5], the semi-supervised method by $5.4\%$, and the
ensembles by $5\%$. In the second cycle, our approach outperforms all the
other methods, including PM [5], by almost $4\%$ or more. We observe that this
difference is maintained in the other cycles, including the last active
learning cycle where our method outperforms the semi-supervised method, multi-
model methods [9, 12] and the best AL method [5] by $1.6\%$.

####  4.1.1 Ablation study.

|  |  |  |
---|---|---|---|---
|  |  |  |

Figure 5: VOC07+12. In the bar plots we show the accuracy per class using random sampling in the zeroth and last cycle. We present the results of each AL method for the three best-performing (”Train”, ”Car”, and ”Horse”) and worst-performing (”Bottle”, ”Pottedplant”, and ”Chair”) classes. |
---|---
(a) | (b)
Figure 6: MS-COCO. a) The percentage of classes where one acquisition function
outperforms another; b) The percentage of classes where our unified
acquisition function outperforms random with and without pseudo-labels.
Example: taking the entry ”unified” in the y-axis, and ”entropy” in the
x-axis, we get the value $0.69$ which means that ”unified” acquisition
function outperforms the ”entropy” acquisition function in $69\%$ of classes.

The effect of active learning and auto-labeling. We now analyze every module
of our method, in order to disentangle the contribution coming from them. In
Fig. 3c, we present the performance comparison of the semi-supervised model on
VOC07+12 under different acquisition functions (random, entropy,
inconsistency) and two instances of our unified method: with and without
pseudo-labels. We see that on the first active learning cycle, neither entropy
nor inconsistency significantly outperforms the results of random sampling.
However, we immediately see a significant effect, i.e., a relative improvement
over $0.9\%$ using entropy and $1.4\%$ using inconsistency, in the second
active learning cycle. We see that the increase in performance gets bigger in
the next AL cycle and in the fifth active learning cycle, the performance gain
from entropy is $2.3\%$ and from inconsistency is $2.4\%$.

We then show the results of our unified acquisition function. In the first
active learning cycle, we immediately see a significant improvement in
performance. While entropy ($67.24mAP$) and inconsistency ($67.39$mAP) reach
an insignificant improvement over random sampling ($67.19mAP$), our
acquisition function reaches $68.40mAP$, which is $1.5\%$ better than random
sampling. The performance improvement gets larger in the next cycles: $2\%$ in
the second cycle, $2.5\%$ in the third cycle, and a peak improvement of
$2.6\%$ in the fifth active learning cycle. In all cases, our proposed score
outperforms both active learning methods that are based on a single
acquisition function.

We further study the effect of pseudo-labeling in our framework. In Fig. 3c,
we observe that on the first active learning cycle, adding pseudo-labels comes
with an immediate boost, improving the results by $3.7\%$ compared to the
already well-performing acquisition function for semi-supervised learning
($5.3\%$ better than the semi-supervised method that uses random sampling). We
further observe that on the second cycle, it gives an improvement of $2.9\%$
compared to using only our acquisition function ($4.7\%$ better than the semi-
supervised method that uses random sampling). The Pseudo-labeling module
continues to give a boost in performance in all the following AL steps.

In Fig. 4c, we provide a similar ablation study for MS-COCO. We again observe
that the unified score outperforms both the entropy and inconsistency scores
in isolation. However, unlike in VOC07+12, we observe that using only the
entropy, the improvement is marginal over random sampling. On the other hand,
we observe that inconsistency works significantly better than random sampling
and entropy (we provide an explanation in the next section). We further
observe the effect of pseudo-labels. We see that in the first AL cycle, adding
pseudo-labels boosts the performance by $3.1\%$ and the performance boost is
maintained up to the last cycle. This is very different from the results of
pseudo-labeling alone, see Fig. 4b, where the performance gain is marginal. In
other words, pseudo-labeling in isolation does not work well. However, pseudo-
labeling complemented with the unified score reaches high results.

Acquisition functions. We now focus on analyzing the effects of aggregating
the two acquisition functions. In Fig. 5, we check the performance of every
individual class in the zeroth and the last AL cycle in VOC07+12 dataset. We
then focus on three best-performing classes (”Train”, ”Car”, and ”Horse”) and
three worst-performing classes (”Bottle”, ”Pottedplant”, and ”Chair”). A first
observation is that for the best-performing classes, entropy-based AL, on
average, tends to outperform inconsistency-based AL.

On the other hand, we see that inconsistency-based AL outperforms the entropy-
based AL by a significant margin in all three worst-performing classes. While
the entropy-based AL on average seems to only slightly outperform random
sampling, the inconsistency-based AL gives a relative performance gain of up
to $24\%$, $14\%$ and $18\%$ in classes ”Bottle”, ”Pottedplant”, and ”Chair”.
Intuitively, one can argue that this phenomenon is to be expected. The fact
that the network does a poor job on its predictions leads to its class
predictions being unreliable for any uncertainty-based AL method. At the same
time, a more general acquisition function dependent only on the robustness of
the network is better suited for low-performing classes. Finally, we show that
our acquisition function reaches the best overall results.

Because of the massive number of classes, we aggregate the results on MS-COCO
dataset. In Fig. 6a, we show the percentage of classes where one acquisition
function outperforms the other. We see that inconsistency outperforms entropy
in $62\%$ of the classes, and our unified score outperforms entropy in $60\%$
of the classes. This explains why in MS-COCO, which contains many more
challenging classes, the robustness-based acquisition scores significantly
outperform the uncertainty-based acquisition score. We provide the results for
each AL cycle in the supplementary.

Do we need SSL training? The inconsistency acquisition function computes the
robustness of the network for each image in the acquisition pool. If the
network is inconsistent in an image, despite that it was trained to minimize
the inconsistency of that image, then that image provides information that was
not captured by the SSL. We now check what happens if the network is not
trained in SSL, thus it has never seen the images in the labeling pool. In
Tab. 1a, we show the results of inconsistency acquisition function for a
network trained with and without consistency loss. As expected, the results
with the SSL loss significantly outperform the results of the fully-supervised
baseline. Interestingly, the results of inconsistency AL are not better than
those of random. Clearly, in order to be able to exploit the robustness
information in samples, the network needs to try minimizing their
inconsistency during training.

Figure 7: VOC07+12: Effect of pseudo-labels compared to AL alone for every
class.

PL performance boost per class. We now study if the pseudo-labels help only
some particular classes, or if they help in all classes. We start the analysis
on VOC07+12 dataset. In Fig. 7, we plot the performance gain coming from the
module for each class and compare it with the performance gain from AL alone,
and random sampling. In the first AL cycle, we see that pseudo-labels improve
over random sampling in all $20$ classes, with AL alone giving a negative
boost in two classes: ”Pottedplant” and ”DiningTable”. We also find out that
pseudo-labels give a boost over AL alone in all three worst-performing classes
(”Bottle”, ”Pottedplant”, and ”Chair”). We see a similar pattern in the other
cycles. In the second cycle, the pseudo-labels module improves over AL alone
in $14$ classes and gives a negative boost in only one class (”Dog”) compared
to AL alone that gives a negative boost in four classes. In the third cycle,
the pseudo-labels module improves over AL alone in $11$ classes and gives a
negative boost in two classes (compared to AL in five classes). In the fourth
cycle, the pseudo-labels module improves over AL alone in $15$ classes and
gives a negative boost in one class (compared to AL in three classes).
Finally, in the fifth AL cycle, the pseudo-labels module improves over AL
alone in $10$ classes with class ”Car” being a tie and gives a negative boost
in only one class, compared to AL in three classes. We thus conclude that by
focusing only on the hard samples, AL alone harms the performance on several
classes. However, adding pseudo-labels diminishes this effect, making the
network much more robust and thus preventing a dataset drift.

We do a similar analysis on the larger MS-COCO, by aggregating the results,
showing the results in Fig. 6b. While our unified acquisition function
outperforms the random acquisition in $60\%$ of the classes, adding pseudo-
labels increases the number of classes this happens to $72\%$, showing the
usefulness of pseudo-labels. We provide the experiment results for each AL
cycle in the supplementary material.

Ratio of pseudo-labels. We now study the effect of increasing the number of
pseudo-labels by allowing more noisy pseudo-labels. To do so, we lower the
pseudo-labeling threshold $\tau$ from $0.99$ to $0.9$ and $0.5$. We present
the results in Fig. 8a. We observe that we reach the best overall results by
using an extremely high threshold $\tau=0.99$. Decreasing $\tau$ to $0.9$ and
thus allowing more pseudo-labels harms the performance. Further decreasing it
to $0.5$, hence allowing many more pseudo-labels, actually harms the entire
training. We thus conclude that we need to be selective in the choice of
pseudo-labels.

To understand why the performance improvement of the network trained with the
pseudo-labels module diminishes in the later active learning cycles, we study
the pseudo-labels gain as a function of the pseudo-labels ratio to the entire
labels. As we show in Fig. 8b, in the first active learning cycle where the
pseudo-labels bring a maximum gain ($3.7\%$), roughly half of the labels are
pseudo-labels. With the decrease of the number of pseudo-labels, we see a
tendency for the gain to lower. Our intuition is that when the number of
pseudo-labels is high, despite them being noisy, they still help the training
process. An interesting fact is that while the total number of pseudo-labels
decreases for each AL cycle (because $1,000$ images are removed from the
labeling pool) the number of pseudo-labels for an image increases with each
cycle, starting from $0.58$ in the first cycle, to $0.81$ in the last one.
Thus, the network becomes better at selecting pseudo-labels during AL cycles.

On MS-COCO, where the number of images that can be potentially pseudo-labeled
is higher ($78K$ compared to $14,651$ in VOC0712), the performance gain from
the pseudo-labels module does not diminish. This is because the ratio of
pseudo-labels in all cycles remains high.

| Cycle | Random | No SSL | SSL
---|---|---|---
1 | 64.23 | 63.26 | 67.39
2 | 66.33 | 65.79 | 70.42
3 | 67.51 | 67.16 | 72.43
4 | 68.60 | 68.65 | 72.80
5 | 69.27 | 70.33 | 74.90

| Cycle | 0.5 | 0.9 | 0.99
---|---|---|---
1 | 80.04 | 91.13 | 96.68
2 | 84.00 | 92.21 | 96.01
3 | 86.00 | 93.32 | 95.80
4 | 87.55 | 93.41 | 95.61
5 | 90.05 | 94.64 | 95.57

(a) | (b)
Table 1: VOC07+12. a) The effect of training on SSL. b) PL correctness with $\tau$. |
---|---
(a) | (b)
Figure 8: VOC07+12. Left: Accuracy as a function of $\tau$ for selecting
pseudo-labels. Right: Accuracy improvement with respect to the pseudo-labels
ratio to the entire labels.

Pseudo-labels noise. Deep neural networks are overconfident and not well-
calibrated, thus inducing pseudo-labeling errors. We consider a pseudo-label
correct if the predicted class is the same as the ground truth and
intersection over union with the ground truth object is over $0.5$. We provide
the correctness of the pseudo-labels given by our model in Tab. 1b. We see
that setting the pseudo-labeling threshold to $0.99$ leads to $3.7\%$ pseudo-
labeling errors. This percentage remains constant over different AL cycles
suggesting the network is robust to this amount of noise.

Limitations. Our method is task-specific and limited to a set of known
categories. We did not conduct any experiment on multi-task networks or in an
open-world setting.

##  5 Conclusions

In this work, we developed a framework that reduces the labeling costs for
object detection. Our framework consists of a novel acquisition function based
on the robustness of the neural network with respect to its predictions, and
an auto-labeling scheme that prevents a potential distribution drift. In this
way, our unified model chooses to actively label the most informative samples
in the dataset, while it pseudo-labels the easiest samples. This allows us to
use the majority of the dataset in a supervised manner while reducing the
labeling costs. As we showed in the experiments, we can reduce the labeling
costs by up to $82\%$ in order to reach the same results as a fully-supervised
baseline.

## References

  * [1] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun.  Faster R-CNN: towards real-time object detection with region proposal networks.  In NeurIPS, 2015.
  * [2] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg.  Ssd: Single shot multibox detector.  In ECCV, 2016.
  * [3] Jisoo Jeong, Seungeui Lee, Jeesoo Kim, and Nojun Kwak.  Consistency-based semi-supervised learning for object detection.  In NeurIPS, 2019.
  * [4] Donggeun Yoo and In So Kweon.  Learning loss for active learning.  In CVPR, 2019.
  * [5] Jiwoong Choi, Ismail Elezi, Hyuk-Jae Lee, Clément Farabet, and Jose M. Alvarez.  Active learning for deep object detection via probabilistic modeling.  In ICCV, 2021.
  * [6] Ozan Sener and Silvio Savarese.  Active learning for convolutional neural networks: A core-set approach.  In ICLR, 2018.
  * [7] Sharat Agarwal, Himanshu Arora, Saket Anand, and Chetan Arora.  Contextual diversity for active learning.  In ECCV, 2020.
  * [8] Tianning Yuan, Fang Wan, Mengying Fu, Jianzhuang Liu, Songcen Xu, Xiangyang Ji, and Qixiang Ye.  Multiple instance active learning for object detection.  In CVPR, 2021.
  * [9] William H. Beluch, Tim Genewein, Andreas Nürnberger, and Jan M. Köhler.  The power of ensembles for active learning in image classification.  In CVPR, 2018.
  * [10] Claude E. Shannon.  A mathematical theory of communication.  Mobile Computing and Communications Review, 2001.
  * [11] Neil Houlsby, Ferenc Huszar, Zoubin Ghahramani, and Máté Lengyel.  Bayesian active learning for classification and preference learning.  CoRR, abs/1112.5745, 2011.
  * [12] Yarin Gal, Riashat Islam, and Zoubin Ghahramani.  Deep bayesian active learning with image data.  In ICML, 2017.
  * [13] Andreas Kirsch, Joost van Amersfoort, and Yarin Gal.  Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning.  In NeurIPS, 2019.
  * [14] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In ICML, 2016.
  * [15] Toan Tran, Thanh-Toan Do, Ian D. Reid, and Gustavo Carneiro.  Bayesian generative active deep learning.  In ICML, 2019.
  * [16] Diederik P. Kingma and Max Welling.  Auto-encoding variational bayes.  In ICLR, 2014.
  * [17] Mingfei Gao, Zizhao Zhang, Guo Yu, Sercan Ömer Arik, Larry S. Davis, and Tomas Pfister.  Consistency-based semi-supervised active learning: Towards minimizing labeling cost.  In ECCV, 2020.
  * [18] David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel.  Mixmatch: A holistic approach to semi-supervised learning.  In NeurIPS, 2019.
  * [19] Hamed Habibi Aghdam, Abel Gonzalez-Garcia, Antonio M. López, and Joost van de Weijer.  Active learning for deep detection neural networks.  In ICCV, 2019.
  * [20] Chieh-Chi Kao, Teng-Yok Lee, Pradeep Sen, and Ming-Yu Liu.  Localization-aware active learning for object detection.  In ACCV, 2018.
  * [21] Soumya Roy, Asim Unmesh, and Vinay P. Namboodiri.  Deep active learning for object detection.  In BMVC, 2018.
  * [22] Sai Vikas Desai, Akshay Chandra Lagandula, Wei Guo, Seishi Ninomiya, and Vineeth N. Balasubramanian.  An adaptive supervision framework for active learning in object detection.  In BMVC, 2019.
  * [23] Elmar Haussmann, Michele Fenzi, Kashyap Chitta, Jan Ivanecky, Hanson Xu, Donna Roy, Akshita Mittel, Nicolas Koumchatzky, Clement Farabet, and Jose M Alvarez.  Scalable active learning for object detection.  In IV, 2020.
  * [24] Dong-Hyun Lee.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  ICMLW, 2013.
  * [25] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  In NeurIPS, 2017.
  * [26] Samuli Laine and Timo Aila.  Temporal ensembling for semi-supervised learning.  In ICLR, 2017.
  * [27] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii.  Virtual adversarial training: A regularization method for supervised and semi-supervised learning.  tPAMI, 2019.
  * [28] Avital Oliver, Augustus Odena, Colin Raffel, Ekin Dogus Cubuk, and Ian J. Goodfellow.  Realistic evaluation of deep semi-supervised learning algorithms.  In NeurIPS, 2018.
  * [29] David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel.  Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring.  In ICLR.
  * [30] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  In NeurIPS, 2020.
  * [31] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki.  Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling.  In NeurIPS, 2021.
  * [32] David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alex Kurakin.  Adamatch: A unified approach to semi-supervised learning and domain adaptation.  CoRR, abs/2106.04732, 2021.
  * [33] Jisoo Jeong, Vikas Verma, Minsung Hyun, Juho Kannala, and Nojun Kwak.  Interpolation-based semi-supervised learning for object detection.  In CVPR, 2021.
  * [34] Yihe Tang, Weifeng Chen, Yijun Luo, and Yuting Zhang.  Humble teachers teach better students for semi-supervised object detection.  In CVPR, 2021.
  * [35] Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman.  The pascal visual object classes (VOC) challenge.  IJCV, 2010.
  * [36] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: common objects in context.  In ECCV, 2014.
  * [37] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  In ICLR, 2015.

— Supplementary material —

##  1 Exact numbers for the experiments given in the main paper

In the paper, we provide plots for the main experiments due to the limited
space. In Tables 2, 3a, 3b, 4, 5a, 5b, we summarize the exact numbers
corresponding to Figures 3a, 3b, 3c, 4a, 4b and 4c of the main paper. We
provide the mean and the standard deviation for each method and AL cycle. Each
experiment has been run three times.

Cycle | Random | Entropy | Core-Set [6] | LLAL [4] | Ensemble [9] | MC-dropout [12] | CDAL-RL [7] | MI-AOD [8] | PM [5] | Ours
---|---|---|---|---|---|---|---|---|---|---
0 | 60.82$\pm$0.2 | 61.23$\pm$0.8 | 62.36$\pm$0.5 | 60.95$\pm$0.4 | 60.82$\pm$0.2 | 60.82$\pm$0.2 | 61.45 $\pm$0.2 | 62.20 $\pm$0.2 | 61.30 $\pm$0.5 |  63.25$\pm$0.2
1 | 64.23$\pm$0.2 | 63.57$\pm$0.9 | 65.90$\pm$0.4 | 64.91$\pm$0.5 | 65.70$\pm$0.9 | 66.90$\pm$0.3 | 65.30 $\pm$0.2 | 65.60 $\pm$0.2 | 65.56 $\pm$0.3 |  70.95 $\pm$0.1
2 | 66.33$\pm$0.2 | 66.94$\pm$0.2 | 67.63$\pm$0.2 | 66.90$\pm$0.3 | 69.20$\pm$0.3 | 68.40$\pm$0.2 | 68.20 $\pm$0.3 | 69.25 $\pm$0.2 | 68.43 $\pm$0.1 |  72.88$\pm$0.1
3 | 67.51$\pm$0.2 | 68.70$\pm$0.2 | 68.88$\pm$0.5 | 69.05$\pm$0.5 | 71.50$\pm$0.2 | 70.80$\pm$0.4 | 70.30 $\pm$0.2 | 70.35 $\pm$0.2 | 70.77 $\pm$0.1 |  73.55$\pm$0.2
4 | 68.60$\pm$0.5 | 69.82$\pm$0.1 | 69.44$\pm$0.3 | 70.35$\pm$0.6 | 72.90$\pm$0.3 | 71.90$\pm$0.5 | 71.60 $\pm$0.2 | 70.80 $\pm$0.2 | 72.52 $\pm$0.1 |  74.75$\pm$0.2
5 | 69.27$\pm$0.2 | 70.18$\pm$0.3 | 70.16$\pm$0.1 | 71.49$\pm$0.7 | 74.29$\pm$0.2 | 73.81$\pm$0.0 | 72.20 $\pm$0.2 | 72.00 $\pm 0.2$ | 73.52 $\pm 0.5$ | 75.60$\pm 0.2$

Table 2: VOC07+12. Comparison to state-of-the-art active learning methods. We initially use $2,000$ randomly sampled images and, in every other cycle, we label $1,000$ extra images. Our method outperforms all the other methods, including ensembles, by a large margin. | Cycle | Random | SSL-cons. [3] | SSL-PL [24] | Ours
---|---|---|---|---
0 | 60.82$\pm$0.2 |  63.25$\pm$0.2 |  63.25$\pm$0.2 |  63.25$\pm$0.2
1 | 64.23$\pm$0.2 | 67.19 $\pm$0.1 | 69.60 $\pm$0.5 |  70.95 $\pm$0.1
2 | 66.33$\pm$0.2 | 69.44 $\pm$0.1 | 70.90 $\pm$0.5 |  72.88$\pm$0.1
3 | 67.51$\pm$0.2 | 71.13 $\pm$0.1 | 71.80 $\pm$0.1 |  73.55$\pm$0.2
4 | 68.60$\pm$0.5 | 72.18 $\pm$0.1 | 72.60 $\pm$0.2 |  74.75$\pm$0.2
5 | 69.27$\pm$0.2 | 73.10 $\pm$0.1 | 73.30 $\pm$0.2 |  75.60$\pm$0.2

| Cycle | Random | Entropy | Inconsistency | Combined | Ours
---|---|---|---|---|---
0 |  63.25$\pm$0.2 |  63.25$\pm$0.2 |  63.25$\pm$0.2 |  63.25$\pm$0.2 |  63.25$\pm$0.2
1 | 67.19 $\pm$0.1 | 67.24$\pm$0.1 | 67.39$\pm$0.9 | 68.40$\pm$0.3 |  70.95 $\pm$0.1
2 | 69.44$\pm$0.1 | 70.05$\pm$0.1 | 70.42$\pm$0.6 | 70.84$\pm$0.8 |  72.88$\pm$0.1
3 | 71.13$\pm$0.1 | 72.13$\pm$0.2 | 72.43$\pm$0.4 | 72.93$\pm$0.3 |  73.55$\pm$0.2
4 | 72.18$\pm$0.1 | 73.48$\pm$0.6 | 72.80$\pm$1.0 | 73.66$\pm$0.2 |  74.75$\pm$0.2
5 | 73.1$\pm$0.1 | 74.77$\pm$0.2 | 74.90$\pm$0.3 | 74.98$\pm$0.5 |  75.60$\pm$0.2

(a) | (b)
Table 3: VOC07+12. a) Comparison to two semi-supervised learning methods. We
initially use $2,000$ randomly sampled images and, in every other cycle, we
label $1,000$ extra images. Our method outperforms both of them by a large
margin. b) Ablation study on the effect of entropy, inconsistency, unified
score, and our method in VOC07+12. We observe that doing active learning with
either entropy or consistency outperforms the semi-supervised model, that the
unified score performs better than either of the individual scores, and that
our method reaches the best overall results.

Cycle | Random | Entropy | Core-Set [6] | Ensemble [9] | MC-dropout [12] | PM [5] | Ours
---|---|---|---|---|---|---|---
0 | 25.63$\pm$0.4 | 25.63$\pm$0.4 | 25.63$\pm$0.4 | 27.50$\pm$0.3 | 27.50$\pm$0.3 |  27.70 $\pm$0.1 | 27.50$\pm$0.3
1 | 28.40$\pm$0.1 | 28.57$\pm$0.2 | 28.10$\pm$0.5 | 28.65$\pm$0.1 | 28.70$\pm$0.2 | 29.28 $\pm$0.1 |  30.07$\pm$0.4
2 | 29.40$\pm$0.2 | 29.47$\pm$0.1 | 29.57$\pm$0.1 | 29.75$\pm$0.2 | 29.42$\pm$0.2 | 30.51$\pm$0.1 |  31.63$\pm$0.1
3 | 30.20$\pm$0.6 | 30.37$\pm$0.1 | 30.40$\pm$0.4 | 30.43$\pm$0.1 | 30.24$\pm$0.2 | 31.20 $\pm$0.1 |  32.10$\pm$0.1
4 | 31.03$\pm$0.1 | 31.17$\pm$0.2 | 31.17$\pm$0.3 | 31.20$\pm$0.2 | 31.03$\pm$0.1 | 31.86 $\pm$0.1 |  32.43$\pm$0.1
5 | 31.47$\pm$0.3 | 31.50$\pm$0.2 | 31.87$\pm$0.2 | 31.75$\pm$0.1 | 31.22$\pm$0.1 | 32.27 $\pm$0.1 |  32.80$\pm$0.0

Table 4: MS-COCO. Comparison to state-of-the-art methods. In this case, we initially use $5,000$ randomly sampled images, and, in every active learning cycle, we label $1,000$ extra images. Our method outperforms all the other methods, including ensembles, by a large margin. | Cycle | Random | SSL-cons. [3] | SSL-PL [24] | Ours
---|---|---|---|---
0 | 25.63$\pm$0.4 |  27.50$\pm$0.3 |  27.50$\pm$0.3 |  27.50$\pm$0.3
1 | 28.40$\pm$0.1 | 28.52 $\pm$0.1 | 28.70 $\pm$0.2 |  30.07$\pm$0.4
2 | 29.40$\pm$0.2 | 29.20 $\pm$0.1 | 29.42 $\pm$0.2 |  31.63$\pm$0.1
3 | 30.20$\pm$0.6 | 30.20 $\pm$0.1 | 30.24 $\pm$0.2 |  32.10$\pm$0.1
4 | 31.03$\pm$0.1 | 31.03 $\pm$0.1 | 31.03 $\pm$0.1 |  32.43$\pm$0.1
5 | 31.47$\pm$0.3 | 31.47 $\pm$0.1 | 31.22$\pm$0.1 |  32.80$\pm$0.0

| Cycle | Random | Entropy | Inconsistency | Combined | Ours
---|---|---|---|---|---
0 |  27.50$\pm$0.3 |  27.50$\pm$0.3 |  27.50$\pm$0.3 |  27.50$\pm$0.3 |  27.50$\pm$0.3
1 | 28.53$\pm$0.1 | 28.97$\pm$0.3 | 28.90$\pm$0.3 | 29.17$\pm$0.2 |  30.07$\pm$0.4
2 | 29.20$\pm$0.1 | 29.77$\pm$0.1 | 29.90$\pm$0.7 | 30.17$\pm$0.2 |  31.63 $\pm$0.1
3 | 29.87$\pm$0.1 | 29.93$\pm$0.4 | 30.85$\pm$0.4 | 30.90$\pm$0.2 |  32.10 $\pm$0.1
4 | 31.03$\pm$0.1 | 31.10$\pm$0.1 | 31.55$\pm$0.1 | 31.55$\pm$0.1 |  32.43 $\pm$0.1
5 | 31.10$\pm$0.1 | 31.40$\pm$0.1 | 31.65$\pm$0.3 | 31.50$\pm$0.1 |  32.80 $\pm$0.0

(a) | (b)
Table 5: MS-COCO. a) Comparison to two semi-supervised learning methods. We
initially use $5,000$ randomly sampled images and, in every other cycle, we
label $1,000$ extra images. Our method outperforms both of them by a large
margin. b) Ablation study on the effect of entropy, inconsistency, unified
score, and our method in MS-COCO. We observe that doing active learning with
either entropy or consistency outperforms the semi-supervised model, that the
unified score performs better than either of the individual scores, and that
our method reaches the best overall results.

##  2 Detailed results in MS-COCO

In the paper we presented aggregated results on the number of classes one
acquisition function performs another (lines 688-697), and in pseudo-label
performance boost per class (lines 777-783). We presented results by
aggregating them over the five cycles of active learning. For completeness, in
Figures 9 and 10, we provide results for each AL cycle in isolation. We see
the same trend as in the paper.

##  3 Pseudo-labels for class

In this experiment we analyze different methods for obtaining pseudo-labels.
Precisely, instead of obtaining pseudo-labels using the confidence score and a
threshold $\tau$ independently of the class, we consider the $k\%$ most
confident objects for each class and add them as pseudo-labels. In Table 6a we
present the results where we pseudo-label the top $20\%$, top $30\%$, top
$40\%$ most confident predictions for class and compare them with the results
of our method described in the main paper. We see that the methods where we
pseudo-label per class work well, but worse than our method. Thus, for both
simplicity and perormance, we choose to use our class-agnostic method.

| Cycle | Top $20\%$ | Top $30\%$ | Top $40\%$ | Ours
---|---|---|---|---
0 | 63.25$\pm$0.3 | 63.25$\pm$0.3 | 63.25$\pm$0.3 |  63.25$\pm$0.3
1 | 69.42$\pm$0.1 | 69.63$\pm$0.1 | 69.67$\pm$0.3 |  70.95$\pm$0.1
2 | 71.84$\pm$0.3 | 71.84$\pm$0.3 | 72.14$\pm$0.2 |  72.88$\pm$0.1
3 | 73.34$\pm$0.3 | 73.47$\pm$0.1 | 73.56$\pm$0.2 |  73.55$\pm$0.1
4 | 74.53$\pm$0.1 | 74.53$\pm$0.2 | 74.32$\pm$0.2 |  74.75$\pm$0.1
5 | 75.13$\pm$0.1 | 75.39$\pm$0.2 | 75.19$\pm$0.2 |  75.60$\pm$0.1

| Cycle | Random | Balanced quarter | Unified
---|---|---|---
0 | 53.66$\pm$0.2 |  63.37$\pm$0.2 | 63.25$\pm$0.2
1 | 67.39$\pm$0.4 | 68.12$\pm$0.2 |  68.40$\pm$0.3
2 | 69.90$\pm$0.5 | 70.12$\pm$0.6 |  70.84$\pm$0.8
3 | 71.38$\pm$1.2 | 71.92$\pm$0.3 |  72.93$\pm$0.3
4 | 73.53$\pm$0.4 | 73.48$\pm$0.3 |  73.66$\pm$0.2
5 | 74.30$\pm$0.4 | 73.90$\pm$0.4 |  74.98$\pm$0.5

(a) | (b)
Table 6: VOC07+12. a) The results of adding top $k\%$ most confident pseudo-
labels for class, compared to the results of our method. Top $20\%$, Top
$30\%$, Top $40\%$ represent the methods where we choose to pseudo-label the
most confident $20\%$, $30\%$ and $40\%$ pseudo-labels per class. Ours
represent our method where we pseudo-label all the objects for which the
network’s confidence is greater than $0.99$. b) Accuracy as a function of
label/unlabeled sampling strategy. Random refers to random sampling from the
entire dataset, Balanced quarter refers to having a quarter of labeled
samples; Unified refers to half of the samples being labeled. Our balanced
strategy outperforms the other two strategies. Note that in order to check
only the effect of balancing, we do not add pseudo-labels during the training.

##  4 Engineering tricks to consider

###  4.1 Non-maximum suppression

We found the effect of non-maximum suppression (NMS) to be very important in
all AL methods. Without applying NMS, active learning methods did not work
better than a random sampling method. We hypothesize that this happens because
if we do not apply NMS, the number of detected boxes is in the hundreds, so by
sheer chance, some of them might have high acquisition scores. Considering
that in a real-world scenario these boxes would be killed by NMS, we conclude
that these boxes should not be used to compute an acquisition score. Thus, for
every image, we apply NMS before proceeding with the computation of the
acquisition score.

###  4.2 Balanced mini-batches

In the main paper, for every experiment, we force that half of the samples in
a mini-batch are labeled. In this experiment, we evaluate the effect of
varying the number of labeled samples in a mini-batch. In particular, we
compare our results to having only half of the samples labeled, and a random
approach. In order to be able to quantify the effect of balancing, we do all
the experiments without adding pseudo-labels. We present the results in Table
6b. We observe that our strategy of balancing the mini-batches so they contain
an equal number of labeled and unlabeled samples, performs best by up to $1pp$
in all AL cycles except the zeroth one, when it gets outperformed by $0.12pp$
by the strategy where only a quarter of samples contain labels. We also
observe that the strategy where we do only random sampling consistently
reaches the worst results. In fact, in the zeroth AL cycle it gets
outperformed by the balanced strategies by almost $10pp$. This can be
explained by the fact that the number of labeled samples ($2,000$) is much
lower than the number of unlabeled samples ($14,651$), so in a mini-batch of
size $32$, in average, only $3.86$ samples have labels. In some mini-batches,
the number of labeled samples is $0$, and thus the loss function becomes
completely self-supervised. We observe that when the number of labeled samples
increases (by labeling other images during AL stage), the overall performance
increases, but it still lags behind the the balanced strategies.

|  |  |  |
---|---|---|---|---
(1) | (2) | (3) | (4) | (5)
Figure 9: MS-COCO. The percentage of classes where one acquisition function outperforms another. Numbers 1-5 represent the active learning cycle. Example: taking the entry ”unified” in the y-axis, and ”entropy” in the x-axis in (1), we get the value $0.65$ which means that ”unified” acquisition function outperforms the ”entropy” acquisition function in $65\%$ of classes during the first active learning cycle. |  |  |  |
---|---|---|---|---
(1) | (2) | (3) | (4) | (5)
Figure 10: MS-COCO. The percentage of classes where our unified acquisition
function outperforms random with and without pseudo-labels. Numbers 1-5
represent the active learning cycle. Example: taking the entry
”unified_with_PL” in the y-axis, and ”random” in the x-axis in (1), we get the
value $0.76$ which means that our method outperforms the random acquisition
function in $76\%$ of classes during the first active learning cycle.


# Current experimental upper bounds on spacetime diffusion

Martijn Janse  These authors share first authorship and contributed equally.
Dennis G. Uitenbroek  These authors share first authorship and contributed
equally. Loek van Everdingen  These authors share first authorship and
contributed equally. Jaimy Plugge  Bas Hensen<EMAIL_ADDRESS>Tjerk H. Oosterkamp<EMAIL_ADDRESS>Leiden Institute of
Physics, Leiden University, P.O. Box 9504, 2300 RA Leiden, The Netherlands

(February 2024)

###### Abstract

A consistent theory describing the dynamics of quantum systems interacting on
a classical spacetime was recently put forward by Oppenheim et al..[1, 2].
Quantum states may retain their coherence, at the cost of some amount of
stochasticity of the spacetime metric, characterized by a spacetime diffusion
parameter. Here, we report existing experimental upper bounds on such
spacetime diffusion, based on a review of several types of experiments with
very low force noise over a broad range of test masses from single atoms to
several kilograms. We find an upper bound at least 15 orders of magnitude
lower as compared to the initial bounds for explicit models presented by
Oppenheimn et al. The results presented here provide a path forward for future
experiments that can help evaluate classical-quantum theories.

##  I Introduction

Oppenheim et al. have pointed out that quantum mechanics can be combined with
a classical interpretation of gravity while still providing a self-consistent
framework [1, 2]. To this end, they have introduced a spacetime diffusion
parameter, $D_{2}$, which describes the fluctuations arising due to the
interplay between a quantum system and its surrounding classical gravitational
potential. Put differently, this is the rate of diffusion of the conjugate
momenta of the Newtonian potential, that is necessary to preserve any amount
of coherence of the quantum system subject to classical gravity.

Three different models were put forward in Oppenheim et al. [1] to describe
this spacetime diffusion parameter, each with their own upper and lower bound.
The ultra-local continuous model has already been ruled out in the original
paper by comparison with experiments. However, the remaining two models, i.e.
the ultra-local discrete and the non-local continuous model, are not yet
rejected (Equations 46 and 47 respectively in [1]). In the ultra-local
discrete model $D_{2}$ has an upper bound of

$\frac{l_{P}^{3}}{m_{P}}D_{2}\leq\frac{\sigma_{a}^{2}N\Delta
Tr^{4}_{N}}{m_{N}G^{2}}$ (1)

and in the non-local continuous model $D_{2}$ has an upper bound of

$l_{P}^{2}D_{2}\leq\frac{\sigma_{a}^{2}N\Delta Tr^{3}_{N}}{G^{2}}.$ (2)

Here, $l_{P}$ is the Planck length, $m_{P}$ the Planck mass, $G$ the
gravitational constant, $r_{N}$ the radius of the nucleus and $m_{N}$ its
mass. In addition to this upper bound, Oppenheim et al. also introduce a lower
bound based on decoherence experiments. We do not address this lower bound
extensively in this work. In order for the theory to be ruled out in the
current formulation, there are still 26 and 24 orders of magnitude of $D_{2}$
to be overcome in the ultra-local discrete model and the non-local continuous
model respectively.

Leaving the natural constants aside and assuming the radius and mass of the
nucleus to be constant in the experiment we consider, we find three parameters
that combine in the same way for both expressions. These are the acceleration
noise $\sigma_{a}$, the number of nuclei $N$, and the measurement time $\Delta
T$. These three parameters essentially introduce a figure of merit
$\mathrm{FOM}_{D_{2}}$, which needs to be minimized in order for the upper
bound on $D_{2}$ to be lowered for both models. The $\mathrm{FOM}_{D_{2}}$ may
be deduced from precision experiments that minimize acceleration and/or force
noise

$\mathrm{FOM}_{D_{2}}=\sigma_{a}^{2}N\Delta T=S_{a}N.\\\ $ (3)

Here, we use the spectral density $S_{a}=\sigma_{a}^{2}\Delta T$. To
facilitate the comparison with different experiments, we prefer to use the
spectral densities $S_{a}$ rather than the variance $\sigma_{a}^{2}$, such
that the measurement time $\Delta T$ drops out of the equations. Since many
experiments focus on the force noise, $S_{F}$, rather than the acceleration
noise, it is necessary to convert $S_{a}$ to $S_{F}$, using
$\sigma_{F}=m\sigma_{a}$ or $S_{F}=m^{2}S_{a}$. The mass $m=MN/N_{A}$ is
related to the number of particles $N$ through the average molar mass $M$ of
the materials used and Avogadro’s number $N_{A}$. In the case of molecules, we
multiply the number of particles $N$ by the number of nuclei per molecule.

In this work we point out that the figure of merit for modern-day experiments,
ranging from atom interferometry to the LISA Pathfinder mission, can vastly
outperform the figure of merit of the Cavendish torsion balance experiment
which Oppenheim et al. used as a first source of experimental bound on their
models [1]. For this first bound, Oppenheim et al. used $N=$e{26}\text{\,}$$,
$\sigma_{a}=$e{-7}\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}^{2}$$ and $\Delta
T=$100\text{\,}\mathrm{s}$$, leading to
$\mathrm{FOM}_{D_{2}}=$e{14}\text{\,}\mathrm{m}^{2}\mathrm{/}\mathrm{s}^{3}$$.

##  II Results

In Figure 1 we show $\mathrm{FOM}_{D_{2}}$ as a function of the test mass for
different types of experiments with low force or acceleration noise. The
underlying data is shown in Table I in Appendix A. We observe that experiments
with test masses in the range between $e{-22}\text{\,}\mathrm{k}\mathrm{g}$
and ${10}^{2}\text{\,}\mathrm{k}\mathrm{g}$ can yield a stronger upper bound
compared to the Cavendish torsion balance experiment. An experiment by Gisler
et al. [3] achieves a $\mathrm{FOM}_{D_{2}}$ that is 15 orders of magnitude
lower by using a nanowire mechanical resonator with an effective mass of
$9.3e{-15}\text{\,}\mathrm{k}\mathrm{g}$. This measurement results in the
lowest $\mathrm{FOM}_{D_{2}}$ with an absolute measurement of acceleration
noise on Earth. A slightly higher $\mathrm{FOM}_{D_{2}}$ is achieved by
Martynov et al. [4], which concerns the LIGO mirror with a test mass of
several kilograms. This is an indication that the minimal achievable
acceleration noise is largely independent of the test mass.

On top of that, we also observe an improvement of 19 orders of magnitude for
Armano et al. [5], i.e. the LISA Pathfinder mission to explore in-space
detection of gravitational waves. In this experiment a displacement is
measured by means of a laser interferometric arm. However, this concerns a
differential acceleration noise measurement and is in that sense a relative
measurement instead of an absolute measurement of the acceleration noise. This
may have implications for the applicability of these experimental results to
bound the models. Furthermore, in their derivation, Oppenheim et al. note that
the bound on $D_{2}$ will depend on the functional choice of $D_{2}(\Phi)$ on
the Newtonian potential $\Phi$, for which they assume that of a large
background potential, i.e. that of the Earth’s. However, the LISA Pathfinder
measurement is done very far away from the Earth, at its Lagrange point
$\mathrm{L_{1}}$. This may affect the validity of this $\mathrm{FOM}_{D_{2}}$
calculation.

In terms of $\mathrm{FOM}_{D_{2}}$, atom interferometry experiments outperform
all other types of experiments: Asenbaum et al. [6] improve the
$\mathrm{FOM}_{D_{2}}$ by 25 orders of magnitude in comparison to the
Cavendish experiment put forward in the original paper by Oppenheim et al. and
the $\mathrm{FOM}_{D_{2}}$ even drops below the lower bound of the ultra-local
discrete model describing the spacetime diffusion. However, the value is still
one order of magnitude larger than the limit set by the non-local continuous
model. We note that this experiment is qualitatively different from the
Cavendish torsion balance experiments in the sense that a phase shift is
measured instead of a displacement. This is a relative measurement of the
acceleration noise and as such the same considerations apply as for the
experiment by Armano et al. [5]. Lastly, although an acceleration sensitivity
is measured in the atom interferometer, we are unsure to which extent its
results are valid to calculate the upper bound. After all, these atom
interferometry experiments closely resemble the matter-wave interferometry
experiment that Oppenheim et al. use to set a lower bound based on decoherence
experiments [7]. It raises the question if one and the same experiment can put
both an upper and a lower bound on $D_{2}$.

We point out that several low-force noise measurements on Earth can still be
improved, such as magnets levitating in a superconducting well making use of
the Meissner effect, reported by Fuchs et al. [8] and Timberlake et al. [9].
These experiments measure force sensitivity by means of yet another technique,
namely Superconducting Quantum Interference Devices (SQUID) detection. This
type of experiment is currently not limited by thermal noise. Further
improvements in vibration isolation will help to close the gap to the thermal
noise floor, potentially outperforming the LIGO mirror.

Figure 1: The figure of merit $\mathrm{FOM}_{D_{2}}$ as calculated for
different kind of force sensitivity experiments. Different types of
experiments along a broad mass range were considered. Plotted experiments are
considered to be representative of their type. The actual measurement reported
for each experiment is plotted as a circle. A diamond, indicating a potential
thermal force limit, is added to experiments in which the total force noise is
at least two times larger than the thermal force noise. Markers with a colored
edge indicate that the experiment was a differential measurement and therefore
the validity of the calculated $\mathrm{FOM}_{D_{2}}$ is questioned. The
shaded areas in the lower part of the plot indicate the lower bound that was
put on $\mathrm{FOM}_{D_{2}}$ by Oppenheim et al. for respectively the ultra-
local discrete model and the non-local continuous model.

##  III Discussion

Summarizing, a broad range of experiments lay a solid foundation for a new
upper bound to the spacetime diffusion parameter $D_{2}$ that is at least 15
orders of magnitude better than the first bound provided in the original
paper. Experiments for which the applicability of the results to this upper
bound calculation are questioned, are indicated with an open marker in Figure
1. Inclusion of these experiments can improve the upper bound by up to 25
orders of magnitude, thus dropping below the bound set by the ultra-local
discrete model. Furthermore, we would like to point out that, using the
experimental results of Asenbaum et al., there is still a gap of one order of
magnitude (out of 26 orders of magnitude) that needs to be closed for the non-
local continuous model.

Additionally, it is important to realise that we have only considered an
improvement of the $\it{upper}$ bound on the spacetime diffusion parameter in
this work. Oppenheim et al. calculate the $\it{lower}$ limit on $D_{2}$, set
by coherence measurements of large-mass quantum superposition states, from
interferometry experiments showing coherent quantum interference of fullerene
molecules [7]. Although we have not elaborately reviewed experimental results
of coherence measurements, we argue that this lower bound may be raised by
examining more novel experiments. For instance, Fein et al. [10] present an
updated result of the matter-wave interferometry experiment, which Oppenheim
et al. put forward as a first lower bound, where the mass in superposition now
exceeds $25\text{\,}\mathrm{k}\mathrm{D}\mathrm{a}$ and the decoherence rate
is given by $\lambda=\tau^{-1}$ = $133\text{\,}\mathrm{H}\mathrm{z}$. Next,
Wang et al. [11] report the coherence of a 171Yb+ ion qubit with a decoherence
rate of
$\lambda=$182\text{\,}\mathrm{\SIUnitSymbolMicro}\mathrm{H}\mathrm{z}$$.
However, the superposed state in question is a spin superposition and thus not
a mass displacement superposition. Lastly, Bild et al. [12] show a cat state
of a $16\text{\,}\mathrm{\SIUnitSymbolMicro g}$ mechanical oscillator with a
decoherence rate of $\lambda\sim$1\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$$.
Closer examination of these decoherence experiments may result in new insights
on the lower bound on $D_{2}$.

In conclusion, by examining a broad range of modern-day experiments over a
large test mass range we can lower the upper bound on the spacetime diffusion
parameter $D_{2}$. If only direct measurements of acceleration sensitivity on
Earth are considered, i.e. excluding the open data points in Figure 1, the
updated bounds become

$$e{-16}\text{\,}$\geq\frac{l_{P}^{3}}{m_{P}}D_{2}\geq$e{-25}\text{\,}$$ (4)

for the ultra-local discrete model and

$$e{-24}\text{\,}$\geq l_{P}^{2}D_{2}\geq$e{-35}\text{\,}$$ (5)

for the non-local continuous model. Here we report a conservative bound. More
in-depth scrutiny of the theory put forward by Oppenheim et al. should provide
a better understanding of the applicability of the atom interferometry and
LISA Pathfinder experimental results.

In order to discuss future improvements of $\mathrm{FOM}_{D_{2}}$, the origin
of the force noise in the experiments in Figure 1 is considered. We
distinguish between thermal force noise $S_{F}^{\textrm{th}}$ and other noise
sources. The former can be expressed as

$S_{F}^{\textrm{th}}=4k_{\textrm{B}}Tm\gamma=4k_{\textrm{B}}Tm\omega_{0}/Q$
(6)

with $T$ the mode temperature and $\gamma$ the damping coefficient of the
oscillating mode of the test mass. Furthermore, the resonance frequency is
denoted as $\omega_{0}$ and the quality factor is denoted as $Q$ for the
oscillating mode. An experiment is considered to be thermally limited if the
thermal force noise $S_{F}^{\textrm{th}}$ is larger than half the measured
total force noise $S_{F}$. This distinction is made since thermal force noise
has a lower limit set by the fluctuation-dissipation theorem. The
$\mathrm{FOM}_{D_{2}}$ calculated from only the thermal force noise of an
experiment is plotted in Figure 1 if it is at most a factor of two lower than
the $\mathrm{FOM}_{D_{2}}$ calculated from the measured total force noise.
This is the case for at least four experiments for which the force or
acceleration sensitivity is measured directly. Other sources of force noise
generally originate from imperfections in the measurement setup, such as
insufficient vibration isolation.

Assuming an experiment is thermally limited, we have
$\mathrm{FOM}_{D_{2}}=4Nk_{B}T\omega_{0}/(mQ)\sim T\omega_{0}/Q$. This
relation shows that lowering the test mass $m$ is not necessary to improve the
$\mathrm{FOM}_{D_{2}}$, as also becomes clear from Figure 1. However, this
relation indicates that the parameters $T$, $\omega_{0}$ and $Q$ can be used
to lower the thermal noise floor. Lowering the temperature $T$ of an
experiment often requires drastic changes to the experimental setup, such as
carrying out the experiment inside a dilution refrigerator or using vibration
isolation systems [13]. The parameter $Q$ is related to the amount of damping
in a mechanical oscillator. Improving the $\mathrm{FOM}_{D_{2}}$ in terms of
damping would require improved fabrication of the, often nanomechanical,
oscillator. This is generally the preferred direction for improving force-
sensitivity experiments, as is reflected in the recent development of a broad
range of high-$Q$ resonators [14]. In terms of the resonance frequency
$\omega_{0}$ we observe for the reviewed experiments that the
$\mathrm{FOM}_{D_{2}}$ is slightly better for experiments with a lower
$\omega_{0}$. Lowering the resonance frequency $\omega_{0}$ of an experiment
also comes with technical challenges, as vibration isolation at ever lower
frequencies becomes increasingly difficult.

Summarizing, several of the presented experiments pave the way to further
limit the bounds they put on the spacetime diffusion parameter. We look
forward to further discussions on the validity of the upper bounds provided by
the atom interferometry and LISA Pathfinder experiments, as well as other ways
to evaluate the theory for the spacetime diffusion parameter put forward by
Oppenheim et al.

## Acknowledgments

We thank P. Logman and G. van de Stolpe for their contributions to the
manuscript. This work was supported by the European Union (ERC StG, CLOSEtoQG,
Project 101041115), the EU Horizon Europe EIC Pathfinder project QuCoM
(10032223) and the NWO grant OCENW.GROOT.2019.088.

## Appendix A: overview of experiments

For this paper we have reviewed a broad range of different experimental
platforms that aim to measure force or acceleration with ultra-high
sensitivity. These include (mechanical) resonators with high quality factors
that for instance make use of soft clamping or levitation. Other types of
experiments are more mesoscopic or macroscopic (massive), such as modern-day
torsion pendula and the latest vibration-isolated platforms employed by the
LIGO and LISA Pathfinder collaboration. Yet another type of experiment that we
have included, are atom interferometry measurements. These are qualitatively
different in the sense that they measure a phase shift instead of a
displacement, but provide nonetheless very precise acceleration measurements.

We have included measurements of force or acceleration sensitivity that are
actual experimental results (as opposed to theoretically achievable values).
These include for example measurements with SQUIDs or laser interferometry. We
have not included papers that only report experimental results of e.g.
displacement of strain sensitivity measurements (as opposed to force or
acceleration sensitivity), although we think that these measurements could
also give potentially interesting results for $\mathrm{FOM}_{D_{2}}$. In order
to find relevant papers, we thankfully consulted previous reviews on precision
force-sensitivity measurements [14, 15, 16, 17, 18].

Based on test mass magnitude, fabrication technique and measurement method, we
distinguish ten different types of experiments. They are clustered as follows:

  * •

suspended systems: (carbon) nanotubes, nanowires (e.g. suspsended SiC or Al
wires), nanobeams (e.g. GaAs or single-cryslal Si cantilevers), SiN membranes,
mesoscopic (e.g. milligram torsion pendulum) and massive (e.g. the LIGO
mirror),

  * •

levitated systems: trapped ion, optical levitation (e.g. silica nanobeads in
optical tweezers) and magnetic levitation (e.g. magnets in superconducting
wells),

  * •

free-falling systems: atom interferometry.

For every category, the two or three experiments that gave the best
$\mathrm{FOM}_{D_{2}}$ were plotted in Figure 1. To calculate
$\mathrm{FOM}_{D_{2}}$, we have looked up the element and mass of the test
mass, from which we could derive the number of nuclei. Additionally, we listed
the spectral density of the force and acceleration sensitivity. These values
are also listed in Table I. If available in the paper and applicable to the
experiment, we have also looked up the resonance frequency, quality factors,
mode temperatures and damping coefficient in order to be able to calculate the
thermal noise floor via Equation 6.

Table 1: Overview of all experiments that were considered in this paper. Note that not all experiments presented in this table are plotted in Figure 1. For each experiment the measured parameters relevant for calculating the figure of merit, $\mathrm{FOM}_{D_{2}}$, are shown. The columns in this table contain from left to right: a reference to the original paper, the type of experiment, the material of the mechanical oscillator, the test mass $m$, the number of nuclei $N$, the resonance frequency $f_{0}$, the total force noise $S_{F}$, the acceleration noise $S_{a}$ and the figure of merit $\mathrm{FOM}_{D_{2}}$ of the experiment. (†) If a value in the table was unavailable in the original paper, we report the value mentioned in the review paper. (∗) The material described is borosilicate glass. We assumed this to consist for 80$\%$ of SiO2 and for 20$\%$ of B2O3. Reference | Type | Element | m [kg] | N [-] |  $f_{0}$ [Hz] |  $\sqrt{S_{F}}$ [N/$\sqrt{$\mathrm{Hz}$}$] |  $\sqrt{S_{a}}$ [ms${}^{-2}/\sqrt{$\mathrm{Hz}$}$] |  $\mathrm{FOM}_{D_{2}}$ [$m^{2}s^{-3}$]
---|---|---|---|---|---|---|---|---
Asenbaum ’17 [6] | Atom interfer. | Rb |  1.44 $\cdot 10^{-19}$ |  1.00 $\cdot 10^{6}$ | - |  7.08 $\cdot 10^{-28}$ |  4.91 $\cdot 10^{-9}$ |  2.41 $\cdot 10^{-11}$
Biedermann ’15 [19] | Atom interfer. | Cs |  2.21 $\cdot 10^{-17}$ |  1.00 $\cdot 10^{8}$ | - |  9.09 $\cdot 10^{-25}$ |  4.12 $\cdot 10^{-8}$ |  1.70 $\cdot 10^{-7}$
Armano ’18 [5] | Massive | Au | 1.93 |  5.89 $\cdot 10^{24}$ | - |  3.35 $\cdot 10^{-15}$ |  1.74 $\cdot 10^{-15}$ |  1.78 $\cdot 10^{-5}$
Gisler ’22 [3] | Nanowire |  Si3N4 |  9.30 $\cdot 10^{-15}$ |  2.79 $\cdot 10^{11}$ |  1.41 $\cdot 10^{6}$ |  9.60 $\cdot 10^{-21}$ |  1.03 $\cdot 10^{-6}$ |  2.98 $\cdot 10^{-1}$
Seis ’22 [20] | Membrane |  Si3N4 |  1.50 $\cdot 10^{-11}$ |  4.51 $\cdot 10^{14}$ |  1.49 $\cdot 10^{6}$ |  6.50 $\cdot 10^{-19}$ |  4.33 $\cdot 10^{-8}$ |  8.46 $\cdot 10^{-1}$
Martynov ’16 [4, 15]† | Massive |  SiO2 |  4.00 $\cdot 10^{1}$ |  1.20 $\cdot 10^{27}$ |  1.00 $\cdot 10^{2}$ |  1.57 $\cdot 10^{-12}$ |  3.92 $\cdot 10^{-14}$ | 1.85
Fogliano ’21 [21] | Nanowire | SiC |  1.60 $\cdot 10^{-14}$ |  4.82 $\cdot 10^{11}$ |  1.16 $\cdot 10^{4}$ |  4.00 $\cdot 10^{-20}$ |  2.50 $\cdot 10^{-6}$ | 3.01
Fuchs ’24 [8] | Magnetic lev. |  Nd2Fe14B |  4.00 $\cdot 10^{-7}$ |  3.79 $\cdot 10^{18}$ |  2.67 $\cdot 10^{1}$ |  5.00 $\cdot 10^{-16}$ |  1.25 $\cdot 10^{-9}$ | 5.92
Hamilton ’15 [22] | Atom interfer. | Cs |  4.41 $\cdot 10^{-18}$ |  2.00 $\cdot 10^{7}$ | - |  2.60 $\cdot 10^{-21}$ |  5.89 $\cdot 10^{-4}$ | 6.93
Mamin ’01 [23] | Nanobeam | Si |  6.85 $\cdot 10^{-13}$ |  1.47 $\cdot 10^{13}$ |  4.98 $\cdot 10^{3}$ |  8.20 $\cdot 10^{-19}$ |  1.20 $\cdot 10^{-6}$ |  2.11 $\cdot 10^{1}$
Timberlake ’23 [9] | Magnetic lev. |  Nd2Fe14B |  2.30 $\cdot 10^{-8}$ |  2.18 $\cdot 10^{17}$ |  4.24 $\cdot 10^{1}$ |  3.40 $\cdot 10^{-16}$ |  1.48 $\cdot 10^{-8}$ |  4.76 $\cdot 10^{1}$
Liang ’22 [17] | Optical lev. |  SiO2 |  4.68 $\cdot 10^{-18}$ |  1.41 $\cdot 10^{8}$ |  1.75 $\cdot 10^{5}$ |  4.34 $\cdot 10^{-21}$ |  9.27 $\cdot 10^{-4}$ |  1.21 $\cdot 10^{2}$
Maiwald ’09 [24] | Trapped ion |  Mg+ |  4.04 $\cdot 10^{-26}$ | 1.00 |  1.00 $\cdot 10^{6}$ |  4.60 $\cdot 10^{-25}$ |  1.14 $\cdot 10^{1}$ |  1.30 $\cdot 10^{2}$
Norte ’16 [25] | Membrane |  Si3N4 |  2.30 $\cdot 10^{-11}$ |  6.91 $\cdot 10^{14}$ |  1.50 $\cdot 10^{5}$ |  1.00 $\cdot 10^{-17}$ |  4.35 $\cdot 10^{-7}$ |  1.31 $\cdot 10^{2}$
Monteiro ’20 [26] | Optical lev. |  SiO2 |  9.42 $\cdot 10^{-13}$ |  2.27 $\cdot 10^{14}$ |  6.30 $\cdot 10^{1}$ |  8.78 $\cdot 10^{-19}$ |  9.32 $\cdot 10^{-7}$ |  1.97 $\cdot 10^{2}$
Kampel ’17 [27, 15]† | Membrane |  Si3N4 |  1.00 $\cdot 10^{-11}$ |  3.00 $\cdot 10^{14}$ |  1.60 $\cdot 10^{6}$ |  9.81 $\cdot 10^{-18}$ |  9.81 $\cdot 10^{-7}$ |  2.89 $\cdot 10^{2}$
Héritier ’18 [28] | Nanowire | C |  4.10 $\cdot 10^{-15}$ |  2.06 $\cdot 10^{11}$ |  2.50 $\cdot 10^{4}$ |  1.88 $\cdot 10^{-19}$ |  4.59 $\cdot 10^{-5}$ |  4.33 $\cdot 10^{2}$
Tebbenjohanns ’20 [29, 17]† | Optical lev. |  SiO2 |  3.49 $\cdot 10^{-18}$ |  1.05 $\cdot 10^{8}$ |  1.46 $\cdot 10^{5}$ |  8.00 $\cdot 10^{-21}$ |  2.29 $\cdot 10^{-3}$ |  5.52 $\cdot 10^{2}$
Tebbenjohanns ’19 [30, 17]† | Optical lev. |  SiO2 |  3.49 $\cdot 10^{-18}$ |  1.05 $\cdot 10^{8}$ |  1.46 $\cdot 10^{5}$ |  1.00 $\cdot 10^{-20}$ |  2.87 $\cdot 10^{-3}$ |  8.63 $\cdot 10^{2}$
Lewandowski ’21 [31] | Magnetic lev. |  SiO2, B2O3 |  2.50 $\cdot 10^{-10}$ |  8.26 $\cdot 10^{15}$ | 1.75 |  8.83 $\cdot 10^{-17}$ |  3.53 $\cdot 10^{-7}$ |  1.03 $\cdot 10^{3}$
Teufel ’09 [32] | Nanowire | Al |  5.50 $\cdot 10^{-15}$ |  1.23 $\cdot 10^{11}$ |  1.04 $\cdot 10^{6}$ |  5.10 $\cdot 10^{-19}$ |  9.27 $\cdot 10^{-5}$ |  1.05 $\cdot 10^{3}$
Cripe ’19 [33, 15]† | Nanobeam | GaAs |  5.00 $\cdot 10^{-11}$ |  4.16 $\cdot 10^{14}$ |  8.76 $\cdot 10^{2}$ |  9.81 $\cdot 10^{-17}$ |  1.96 $\cdot 10^{-6}$ |  1.60 $\cdot 10^{3}$
Reinhardt ’16 [34] | Membrane |  Si3N4 |  4.00 $\cdot 10^{-12}$ |  1.20 $\cdot 10^{14}$ |  4.08 $\cdot 10^{4}$ |  2.00 $\cdot 10^{-17}$ |  5.00 $\cdot 10^{-6}$ |  3.00 $\cdot 10^{3}$
Gieseler ’13 [35] | Optical lev. |  SiO2 |  3.00 $\cdot 10^{-18}$ |  9.03 $\cdot 10^{7}$ |  1.25 $\cdot 10^{5}$ |  2.00 $\cdot 10^{-20}$ |  6.67 $\cdot 10^{-3}$ |  4.01 $\cdot 10^{3}$
Delić ’20 [36] | Optical lev. |  SiO2 |  2.83 $\cdot 10^{-18}$ |  8.52 $\cdot 10^{7}$ |  3.05 $\cdot 10^{5}$ |  1.94 $\cdot 10^{-20}$ |  6.87 $\cdot 10^{-3}$ |  4.02 $\cdot 10^{3}$
Corbitt ’07 [37] | Mesoscopic |  SiO2 |  1.00 $\cdot 10^{-3}$ |  3.01 $\cdot 10^{22}$ |  1.80 $\cdot 10^{3}$ |  3.95 $\cdot 10^{-13}$ |  3.95 $\cdot 10^{-10}$ |  4.70 $\cdot 10^{3}$
Westphal ’21 [38] | Mesoscopic | Au |  2.18 $\cdot 10^{-4}$ |  6.66 $\cdot 10^{20}$ |  3.59 $\cdot 10^{-3}$ |  9.07 $\cdot 10^{-13}$ |  4.16 $\cdot 10^{-9}$ |  1.15 $\cdot 10^{4}$
Rider ’18 [39] | Optical lev. |  SiO2 |  1.53 $\cdot 10^{-13}$ |  4.62 $\cdot 10^{12}$ |  2.50 $\cdot 10^{2}$ |  1.15 $\cdot 10^{-17}$ |  7.50 $\cdot 10^{-5}$ |  2.60 $\cdot 10^{4}$
Kawasaki ’20 [40] | Optical lev. |  SiO2 |  8.40 $\cdot 10^{-14}$ |  2.53 $\cdot 10^{12}$ |  3.01 $\cdot 10^{2}$ |  1.00 $\cdot 10^{-17}$ |  1.19 $\cdot 10^{-4}$ |  3.58 $\cdot 10^{4}$
Hempston ’17 [41] | Optical lev. |  SiO2 |  7.60 $\cdot 10^{-19}$ |  2.29 $\cdot 10^{7}$ |  7.20 $\cdot 10^{4}$ |  3.20 $\cdot 10^{-20}$ |  4.21 $\cdot 10^{-2}$ |  4.06 $\cdot 10^{4}$
De Bonis ’18 [16] | Nanotube | C |  8.60 $\cdot 10^{-21}$ |  4.32 $\cdot 10^{5}$ |  2.92 $\cdot 10^{7}$ |  4.30 $\cdot 10^{-21}$ |  5.00 $\cdot 10^{-1}$ |  1.08 $\cdot 10^{5}$
Hälg ’21 [42] | Membrane |  Si3N4 |  1.40 $\cdot 10^{-11}$ |  4.21 $\cdot 10^{14}$ |  1.42 $\cdot 10^{6}$ |  2.80 $\cdot 10^{-16}$ |  2.00 $\cdot 10^{-5}$ |  1.68 $\cdot 10^{5}$
Priel ’22 [43] | Optical lev. |  SiO2 |  5.85 $\cdot 10^{-13}$ |  1.76 $\cdot 10^{13}$ |  1.00 $\cdot 10^{5}$ |  1.00 $\cdot 10^{-16}$ |  1.71 $\cdot 10^{-4}$ |  5.14 $\cdot 10^{5}$
Moser ’13 [44] | Nanotube | C |  1.00 $\cdot 10^{-20}$ |  5.02 $\cdot 10^{5}$ |  4.20 $\cdot 10^{6}$ |  1.20 $\cdot 10^{-20}$ | 1.20 |  7.23 $\cdot 10^{5}$
Weber ’16 [45] | Nanotube | C |  9.60 $\cdot 10^{-18}$ |  4.82 $\cdot 10^{8}$ |  4.60 $\cdot 10^{7}$ |  3.90 $\cdot 10^{-19}$ |  4.06 $\cdot 10^{-2}$ |  7.95 $\cdot 10^{5}$
Ranjit ’16 [46] | Optical lev. |  SiO2 |  3.75 $\cdot 10^{-17}$ |  1.13 $\cdot 10^{9}$ |  2.83 $\cdot 10^{3}$ |  1.63 $\cdot 10^{-18}$ |  4.35 $\cdot 10^{-2}$ |  2.14 $\cdot 10^{6}$
Krause ’12 [47] | Membrane |  Si3N4 |  1.00 $\cdot 10^{-11}$ |  3.00 $\cdot 10^{14}$ |  2.75 $\cdot 10^{4}$ |  9.81 $\cdot 10^{-16}$ |  9.81 $\cdot 10^{-5}$ |  2.89 $\cdot 10^{6}$
Nichol ’12 [48] | Nanowire | Si |  2.66 $\cdot 10^{-17}$ |  5.72 $\cdot 10^{8}$ |  7.86 $\cdot 10^{5}$ |  1.95 $\cdot 10^{-18}$ |  7.33 $\cdot 10^{-2}$ |  3.07 $\cdot 10^{6}$
Biercuk ’10 [49] | Trapped ion |  Be+ |  1.95 $\cdot 10^{-24}$ |  1.30 $\cdot 10^{2}$ |  8.67 $\cdot 10^{5}$ |  3.90 $\cdot 10^{-22}$ |  2.00 $\cdot 10^{2}$ |  5.22 $\cdot 10^{6}$
Ranjit ’15 [50] | Optical lev. |  SiO2 |  3.75 $\cdot 10^{-14}$ |  1.13 $\cdot 10^{12}$ |  1.07 $\cdot 10^{3}$ |  2.17 $\cdot 10^{-16}$ |  5.79 $\cdot 10^{-3}$ |  3.78 $\cdot 10^{7}$
Affolter ’20 [51] | Trapped ion |  Be+ |  1.50 $\cdot 10^{-24}$ |  1.00 $\cdot 10^{2}$ |  1.58 $\cdot 10^{6}$ |  1.20 $\cdot 10^{-21}$ |  8.02 $\cdot 10^{2}$ |  6.43 $\cdot 10^{7}$
Timberlake ’19 [52] | Magnetic lev. |  Nd2Fe14B |  4.00 $\cdot 10^{-6}$ |  3.79 $\cdot 10^{19}$ |  1.94 $\cdot 10^{1}$ |  7.85 $\cdot 10^{-12}$ |  1.96 $\cdot 10^{-6}$ |  1.46 $\cdot 10^{8}$
Guzmán C. ’14 [53] | Mesoscopic |  SiO2 |  2.50 $\cdot 10^{-5}$ |  7.53 $\cdot 10^{20}$ |  1.07 $\cdot 10^{4}$ |  2.45 $\cdot 10^{-11}$ |  9.81 $\cdot 10^{-7}$ |  7.24 $\cdot 10^{8}$
Shaniv ’17 [54] | Trapped ion |  Sr+ |  1.44 $\cdot 10^{-25}$ | 1.00 |  1.13 $\cdot 10^{6}$ |  2.80 $\cdot 10^{-20}$ |  1.94 $\cdot 10^{5}$ |  3.78 $\cdot 10^{10}$
Blums ’18 [55] | Trapped ion |  Yb+ |  2.89 $\cdot 10^{-25}$ | 1.00 |  8.29 $\cdot 10^{5}$ |  3.47 $\cdot 10^{-19}$ |  1.20 $\cdot 10^{6}$ |  1.44 $\cdot 10^{12}$
Cavendish 1798 [56] | Massive | Pb | 1.00 |  1.00 $\cdot 10^{26}$ |  2.00 $\cdot 10^{-3}$ |  1.00 $\cdot 10^{-6}$ |  1.00 $\cdot 10^{-6}$ |  1.00 $\cdot 10^{14}$

## Bibliography

## References

  * Oppenheim  _et al._ [2023] J. Oppenheim, C. Sparaciari, B. Šoda, and Z. Weller-Davies, Gravitationally induced decoherence vs space-time diffusion: testing the quantum nature of gravity, Nature Communications 14, 7910 (2023).
  * Oppenheim [2023] J. Oppenheim, A postquantum theory of classical gravity, Physical Review X 13, 41040 (2023).
  * Gisler  _et al._ [2022] T. Gisler, M. Helal, D. Sabonis, U. Grob, M. Héritier, C. L. Degen, A. H. Ghadimi, and A. Eichler, Soft-clamped silicon nitride string resonators at millikelvin temperatures, Physical Review Letters 129, 1 (2022).
  * Martynov  _et al._ [2016] D. Martynov, E. D. Hall, and et al., Sensitivity of the advanced ligo detectors at the beginning of gravitational wave astronomy, Physical Review D 93, 112004 (2016).
  * Armano and et al. [2018] M. Armano and et al., Beyond the required lisa free-fall performance: New lisa pathfinder results down to 20 muhz, Physical Review Letters 120, 61101 (2018).
  * Asenbaum  _et al._ [2017] P. Asenbaum, C. Overstreet, T. Kovachy, D. D. Brown, J. M. Hogan, and M. A. Kasevich, Phase shift in an atom interferometer due to spacetime curvature across its wave function, Physical Review Letters 118, 1 (2017).
  * Gerlich  _et al._ [2011] S. Gerlich, S. Eibenberger, M. Tomandl, S. Nimmrichter, K. Hornberger, P. J. Fagan, J. Tüxen, M. Mayor, and M. Arndt, Quantum interference of large organic molecules, Nature Communications 2, 263 (2011).
  * Fuchs  _et al._ [2024] T. M. Fuchs, D. G. Uitenbroek, J. Plugge, N. V. Halteren, P. V. Soest, A. Vinante, H. Ulbricht, and T. H. Oosterkamp, Measuring gravity with milligram levitated masses, Science Advances 2949, 1 (2024).
  * Timberlake  _et al._ [2023] C. Timberlake, E. Simcox, and H. Ulbricht, Linear cooling of a levitated micromagnetic cylinder by vibration, ArXiv 2310.03880, 1 (2023).
  * Fein  _et al._ [2019] Y. Y. Fein, P. Geyer, P. Zwick, F. Kiałka, S. Pedalino, M. Mayor, S. Gerlich, and M. Arndt, Quantum superposition of molecules beyond 25 kda, Nature Physics 15, 1242 (2019).
  * Wang  _et al._ [2021] P. Wang, C. Y. Luan, M. Qiao, M. Um, J. Zhang, Y. Wang, X. Yuan, M. Gu, J. Zhang, and K. Kim, Single ion qubit with estimated coherence time exceeding one hour, Nature Communications 12, 233 (2021).
  * Bild  _et al._ [2023] M. Bild, M. Fadel, Y. Yang, U. V. Lüpke, P. Martin, A. Bruno, and Y. Chu, Schrödinger cat states of a 16-microgram mechanical oscillator, Science 380, 274 (2023).
  * Wit  _et al._ [2019] M. D. Wit, G. Welker, K. Heeck, F. M. Buters, H. J. Eerkens, G. Koning, H. V. der Meer, D. Bouwmeester, and T. H. Oosterkamp, Vibration isolation with high thermal conductance for a cryogen-free dilution refrigerator, Review of Scientific Instruments 90, 015112 (2019).
  * Eichler [2022] A. Eichler, Ultra-high-q nanomechanical resonators for force sensing, Materials for Quantum Technology 2, 043001 (2022).
  * Carney  _et al._ [2021] D. Carney, G. Krnjaic, D. C. Moore, C. A. Regal, G. Afek, S. Bhave, B. Brubaker, T. Corbitt, J. Cripe, N. Crisosto, A. Geraci, S. Ghosh, J. G. Harris, A. Hook, E. W. Kolb, J. Kunjummen, R. F. Lang, T. Li, T. Lin, Z. Liu, J. Lykken, L. Magrini, J. Manley, N. Matsumoto, A. Monte, F. Monteiro, T. Purdy, C. J. Riedel, R. Singh, S. Singh, K. Sinha, J. M. Taylor, J. Qin, D. J. Wilson, and Y. Zhao, Mechanical quantum sensing in the search for dark matter, Quantum Science and Technology 6, 024002 (2021).
  * Bonis  _et al._ [2018] S. L. D. Bonis, C. Urgell, W. Yang, C. Samanta, A. Noury, J. Vergara-Cruz, Q. Dong, Y. Jin, and A. Bachtold, Ultrasensitive displacement noise measurement of carbon nanotube mechanical resonators, Nano Letters 18, 5324 (2018).
  * Liang  _et al._ [2023] T. Liang, S. Zhu, P. He, Z. Chen, Y. Wang, C. Li, Z. Fu, X. Gao, X. Chen, N. Li, Q. Zhu, and H. Hu, Yoctonewton force detection based on optically levitated oscillator, Fundamental Research 3, 57 (2023).
  * Moore and Geraci [2021] D. C. Moore and A. A. Geraci, Searching for new physics using optically levitated sensors, Quantum Science and Technology 6, 014008 (2021).
  * Biedermann  _et al._ [2015] G. W. Biedermann, X. Wu, L. Deslauriers, S. Roy, C. Mahadeswaraswamy, and M. A. Kasevich, Testing gravity with cold-atom interferometers, Physical Review A - Atomic, Molecular, and Optical Physics 91, 033629 (2015).
  * Seis  _et al._ [2022] Y. Seis, T. Capelle, E. Langman, S. Saarinen, E. Planz, and A. Schliesser, Ground state cooling of an ultracoherent electromechanical system, Nature Communications 13, 1 (2022).
  * Fogliano  _et al._ [2021] F. Fogliano, B. Besga, A. Reigue, L. M. de Lépinay, P. Heringlake, C. Gouriou, E. Eyraud, W. Wernsdorfer, B. Pigeau, and O. Arcizet, Ultrasensitive nano-optomechanical force sensor operated at dilution temperatures, Nature Communications 12, 1 (2021).
  * Hamilton  _et al._ [2015] P. Hamilton, M. Jaffe, J. M. Brown, L. Maisenbacher, B. Estey, and H. Müller, Atom interferometry in an optical cavity, Physical Review Letters 114, 1 (2015).
  * Mamin and Rugar [2001] H. J. Mamin and D. Rugar, Sub-attonewton force detection at millikelvin temperatures, Applied Physics Letters 79, 3358 (2001).
  * Maiwald  _et al._ [2009] R. Maiwald, D. Leibfried, J. Britton, J. C. Bergquist, G. Leuchs, and D. J. Wineland, Stylus ion trap for enhanced access and sensing, Nature Physics 5, 551 (2009).
  * Norte  _et al._ [2016] R. A. Norte, J. P. Moura, and S. Gröblacher, Mechanical resonators for quantum optomechanics experiments at room temperature, Physical Review Letters 116, 147202 (2016).
  * Monteiro  _et al._ [2020] F. Monteiro, W. Li, G. Afek, C. L. Li, M. Mossman, and D. C. Moore, Force and acceleration sensing with optically levitated nanogram masses at microkelvin temperatures, Physical Review A 101, 053835 (2020).
  * Kampel  _et al._ [2017] N. S. Kampel, R. W. Peterson, R. Fischer, P. L. Yu, K. Cicak, R. W. Simmonds, K. W. Lehnert, and C. A. Regal, Improving broadband displacement detection with quantum correlations, Physical Review X 7, 021008 (2017).
  * Héritier  _et al._ [2018] M. Héritier, A. Eichler, Y. Pan, U. Grob, I. Shorubalko, M. D. Krass, Y. Tao, and C. L. Degen, Nanoladder cantilevers made from diamond and silicon, Nano Letters 18, 1814 (2018).
  * Tebbenjohanns  _et al._ [2020] F. Tebbenjohanns, M. Frimmer, V. Jain, D. Windey, and L. Novotny, Motional sideband asymmetry of a nanoparticle optically levitated in free space, Physical Review Letters 124, 013603 (2020).
  * Tebbenjohanns  _et al._ [2019] F. Tebbenjohanns, M. Frimmer, A. Militaru, V. Jain, and L. Novotny, Cold damping of an optically levitated nanoparticle to microkelvin temperatures, Physical Review Letters 122, 223601 (2019).
  * Lewandowski  _et al._ [2021] C. W. Lewandowski, T. D. Knowles, Z. B. Etienne, and B. D’Urso, High-sensitivity accelerometry with a feedback-cooled magnetically levitated microsphere, Physical Review Applied 15, 014050 (2021).
  * Teufel  _et al._ [2009] J. D. Teufel, T. Donner, M. A. Castellanos-Beltran, J. W. Harlow, and K. W. Lehnert, Nanomechanical motion measured with an imprecision below that at the standard quantum limit, Nature Nanotechnology 2009 4:12 4, 820 (2009).
  * Cripe  _et al._ [2019] J. Cripe, N. Aggarwal, R. Lanza, A. Libson, R. Singh, P. Heu, D. Follman, G. D. Cole, N. Mavalvala, and T. Corbitt, Measurement of quantum back action in the audio band at room temperature, Nature 568, 364 (2019).
  * Reinhardt  _et al._ [2016] C. Reinhardt, T. Müller, A. Bourassa, and J. C. Sankey, Ultralow-noise sin trampoline resonators for sensing and optomechanics, Physical Review X 6, 021001 (2016).
  * Gieseler  _et al._ [2013] J. Gieseler, L. Novotny, and R. Quidant, Thermal nonlinearities in a nanomechanical oscillator, Nature Physics 9, 806 (2013).
  * Delić  _et al._ [2020] U. Delić, M. Reisenbauer, K. Dare, D. Grass, V. Vuletić, N. Kiesel, and M. Aspelmeyer, Cooling of a levitated nanoparticle to the motional quantum ground state, Science 367, 892 (2020).
  * Corbitt  _et al._ [2007] T. Corbitt, Y. Chen, E. Innerhofer, H. Müller-Ebhardt, D. Ottaway, H. Rehbein, D. Sigg, S. Whitcomb, C. Wipf, and N. Mavalvala, An all-optical trap for a gram-scale mirror, Physical Review Letters 98, 150802 (2007).
  * Westphal  _et al._ [2021] T. Westphal, H. Hepach, J. Pfaff, and M. Aspelmeyer, Measurement of gravitational coupling between millimetre-sized masses, Nature 591, 225 (2021).
  * Rider  _et al._ [2018] A. D. Rider, C. P. Blakemore, G. Gratta, and D. C. Moore, Single-beam dielectric-microsphere trapping with optical heterodyne detection, Physical Review A 97, 013842 (2018).
  * Kawasaki  _et al._ [2020] A. Kawasaki, A. Fieguth, N. Priel, C. P. Blakemore, D. Martin, and G. Gratta, High sensitivity, levitated microsphere apparatus for short-distance force measurements, Review of Scientific Instruments 91, 083201 (2020).
  * Hempston  _et al._ [2017] D. Hempston, J. Vovrosh, M. Toroš, G. Winstone, M. Rashid, and H. Ulbricht, Force sensing with an optically levitated charged nanoparticle, Applied Physics Letters 111, 133111 (2017).
  * Hälg  _et al._ [2021] D. Hälg, T. Gisler, Y. Tsaturyan, L. Catalini, U. Grob, M. D. Krass, M. Héritier, H. Mattiat, A. K. Thamm, R. Schirhagl, E. C. Langman, A. Schliesser, C. L. Degen, and A. Eichler, Membrane-based scanning force microscopy, Physical Review Applied 15, L021001 (2021).
  * Priel  _et al._ [2022] N. Priel, A. Fieguth, C. P. Blakemore, E. Hough, A. Kawasaki, D. Martin, G. Venugopalan, and G. Gratta, Dipole moment background measurement and suppression for levitated charge sensors, Science Advances 8, 2361 (2022).
  * Moser  _et al._ [2013] J. Moser, J. Güttinger, A. Eichler, M. J. Esplandiu, D. E. Liu, M. I. Dykman, and A. Bachtold, Ultrasensitive force detection with a nanotube mechanical resonator, Nature Nanotechnology 8, 493 (2013).
  * Weber  _et al._ [2016] P. Weber, J. Güttinger, A. Noury, J. Vergara-Cruz, and A. Bachtold, Force sensitivity of multilayer graphene optomechanical devices, Nature Communications 7, 12496 (2016).
  * Ranjit  _et al._ [2016] G. Ranjit, M. Cunningham, K. Casey, and A. A. Geraci, Zeptonewton force sensing with nanospheres in an optical lattice, Physical Review A 93, 053801 (2016).
  * Krause  _et al._ [2012] A. G. Krause, M. Winger, T. D. Blasius, Q. Lin, and O. Painter, A high-resolution microchip optomechanical accelerometer, Nature Photonics 6, 768 (2012).
  * Nichol  _et al._ [2012] J. M. Nichol, E. R. Hemesath, L. J. Lauhon, and R. Budakian, Nanomechanical detection of nuclear magnetic resonance using a silicon nanowire oscillator, Physical Review B - Condensed Matter and Materials Physics 85, 054414 (2012).
  * Biercuk  _et al._ [2010] M. J. Biercuk, H. Uys, J. W. Britton, A. P. Vandevender, and J. J. Bollinger, Ultrasensitive detection of force and displacement using trapped ions, Nature Nanotechnology 2010 5:9 5, 646 (2010).
  * Ranjit  _et al._ [2015] G. Ranjit, D. P. Atherton, J. H. Stutz, M. Cunningham, and A. A. Geraci, Attonewton force detection using microspheres in a dual-beam optical trap in high vacuum, Physical Review A - Atomic, Molecular, and Optical Physics 91, 051805 (2015).
  * Affolter  _et al._ [2020] M. Affolter, K. A. Gilmore, J. E. Jordan, and J. J. Bollinger, Phase-coherent sensing of the center-of-mass motion of trapped-ion crystals, Physical Review A 102, 052609 (2020).
  * Timberlake  _et al._ [2019] C. Timberlake, G. Gasbarri, A. Vinante, A. Setter, and H. Ulbricht, Acceleration sensing with magnetically levitated oscillators above a superconductor, Applied Physics Letters 115, 224101 (2019).
  * Cervantes  _et al._ [2014] F. G. Cervantes, L. Kumanchik, J. Pratt, and J. M. Taylor, High sensitivity optomechanical reference accelerometer over 10 khz, Applied Physics Letters 104, 221111 (2014).
  * Shaniv and Ozeri [2017] R. Shaniv and R. Ozeri, Quantum lock-in force sensing using optical clock doppler velocimetry, Nature Communications 8, 14157 (2017).
  * Blums  _et al._ [2018] V. Blums, M. Piotrowski, M. I. Hussain, B. G. Norton, S. C. Connell, S. Gensemer, M. Lobino, and E. W. Streed, A single-atom 3d sub-attonewton force sensor, Science Advances 4, eaao4453 (2018).
  * Cavendish [1798] H. Cavendish, Experiments to determine the density of the earth, Philosophical Transactions Royal Society London 88, 469 (1798).