11institutetext:  University of Surrey, Surrey, UK 11email:
<EMAIL_ADDRESS>22institutetext:
University of Oxford, Oxford, UK
22email<EMAIL_ADDRESS>
# Bayesian Detector Combination for Object Detection with Crowdsourced
Annotations

Zhi Qin Tan 11 0000-0002-5521-6808 Olga Isupova 22 0000-0002-8506-6871 Gustavo
Carneiro 11 0000-0002-5571-6220 Xiatian Zhu 11 0000-0002-9284-2955 Yunpeng Li
11 0000-0003-4798-541X

###### Abstract

Acquiring fine-grained object detection annotations in unconstrained images is
time-consuming, expensive, and prone to noise, especially in crowdsourcing
scenarios. Most prior object detection methods assume accurate annotations; A
few recent works have studied object detection with noisy crowdsourced
annotations, with evaluation on distinct synthetic crowdsourced datasets of
varying setups under artificial assumptions. To address these algorithmic
limitations and evaluation inconsistency, we first propose a novel Bayesian
Detector Combination (BDC) framework to more effectively train object
detectors with noisy crowdsourced annotations, with the unique ability of
automatically inferring the annotators’ label qualities. Unlike previous
approaches, BDC is model-agnostic, requires no prior knowledge of the
annotators’ skill level, and seamlessly integrates with existing object
detection models. Due to the scarcity of real-world crowdsourced datasets, we
introduce large synthetic datasets by simulating varying crowdsourcing
scenarios. This allows consistent evaluation of different models at scale.
Extensive experiments on both real and synthetic crowdsourced datasets show
that BDC outperforms existing state-of-the-art methods, demonstrating its
superiority in leveraging crowdsourced data for object detection. Our code and
data are available at: https://github.com/zhiqin1998/bdc.

###### Keywords:

Bayesian Detector Combination Object detection Crowdsourced annotations

##  1 Introduction

Despite significant advancements in the development of object detection with
natural images [41, 53, 9, 14], existing methods typically assume the
availability of large, accurately annotated datasets [61, 37]. Nonetheless,
this assumption does not often hold true in practice. For example, accurate
labels are often expensive to acquire in many applications. A large number of
experts with varying professional skills can be involved to crowdsource
annotations of images sampled with replacement from a large dataset [50, 33,
8]. Usually, the annotations from multiple annotators are unified and cleaned
as the last step in annotation collection. However, disagreements arise when
annotating complex images, making it challenging even for experienced expert
annotators to achieve unanimity due to high interobserver variability [26, 54]
and thus, rendering the role of an annotation arbitrator nearly impossible
(see Figure 1). As a result, _multiple noisy annotations per image_ are
obtained. Such a setting for annotation crowdsourcing is often called the
multi-rater problem. The majority of methods on multi-rater learning have been
developed for image classification [50, 59, 20] and segmentation [24, 22, 4,
60, 21, 58], while we consider object detection here.

(a) ID: 70282 (b) ID: 310072 (c) ID: 516471 (d) ID: 326247 (e) ID: 416149 (f)
(g) (h) (i) (j) (k) (l) (m) (n) (o)

Figure 1: Examples of ambiguous cases with noisy or incorrect annotations on
(a - e) MS COCO [29], (f - j) VinDr-CXR [36] and (k - o) disaster response
dataset [19] to identify damaged and undamaged buildings with class names ‘d’
and ‘u’, respectively. (a) Mislabelling one object with two bounding boxes. (b
- d) Duplicate annotations of visually similar objects (_e.g_., car/truck,
donut/cake and fork/spoon). (d)(e) Incorrect annotation of object classes
(_e.g_., cake labelled as a bowl, only annotating one of the cake decorations
as ‘person’). (f - o) Annotations from different annotators (represented by
the different colours) of the VinDr-CXR and disaster response datasets show
significant disagreement. (Details of the datasets are given in Section 4.1.)

Several previous methods that exploit crowdsourced annotations have been
proposed to improve object detection performance [18, 27]. A common approach
estimates the ground truth from noisy annotations through majority voting [46]
which implicitly assumes that all annotators are equally accurate in their
annotations. Alternatively, in [27], the weight of each annotation is
predicted based on the annotators’ proficiency levels which are assumed to be
known a priori; however, this information is difficult to obtain in practice.
On the other hand, in [18], their proposed method is designed specifically for
Faster R-CNN (FRCNN) [41] and cannot be generalised to other object detectors.
Moreover, these methods have been developed and evaluated on distinct, private
synthetic datasets constructed using different methods. Thus, we cannot
directly compare their reported experiment results.

Figure 2: Overall architecture of the proposed BDC. The updating of the
aggregator’s parameter and the object detector’s parameters is repeated
iteratively until convergence. Best viewed in colour.

To address these algorithmic and evaluation limitations, in this work, we
first propose a model-agnostic annotation aggregation and object detector
training pipeline, named Bayesian Detector Combination (BDC). BDC aggregates
noisy bounding boxes and class labels by modelling the annotation reliability
of each crowd member for both elements (Figure 2), without requiring any
additional knowledge of the ground truth annotations or their quality.
Furthermore, due to the limited access to the ground truth of real
crowdsourced datasets, we synthesise multiple datasets simulating various
crowdsourcing scenarios for systematic evaluation of the previous works and
the proposed method. The main contributions of our work include:

  * •

We propose a model-agnostic Bayesian Detector Combination (BDC) framework that
simultaneously infers the annotation quality of each crowd member and infers
consensus bounding boxes and soft class labels from noisy, crowdsourced
annotations to improve model performance.

  * •

We implement and systematically evaluate the previous methods on real-world
and synthetic datasets with crowdsourced annotations which simulate varying
crowdsourcing scenarios.

  * •

Extensive experiments conducted demonstrate the superior performance of BDC
compared with prior alternatives in key metrics such as the average precision,
the scalability with the increasing number of annotators, and the robustness
against noisy annotators.

##  2 Related Work

##### Classification with Crowdsourced Annotations.

Learning with crowdsourced annotations for classification tasks has been
studied in various works. Sheng _et al_. [46] adopted a simple approach to
aggregate multiple labels via majority voting (MV) where the mode class is
selected as the ground truth. Several works [11, 57, 39] modelled the
sensitivity and specificity of different annotators via confusion matrices
obtained through the Expectation-Maximisation algorithm or a Bayesian
approach. Kim and Ghahramani [23] introduced a general framework for Bayesian
classifier combination (BCC) to model the relationship between the unknown
true label and the annotated class label which is extended by Venanzi _et al_.
[51] with the concept of annotator communities. Albarqouni _et al_. [3] and
Isupova _et al_. [19] extended the idea of previous works [39, 23] to
convolutional neural networks (CNN) with softmax outputs. Several recent works
use a trained classifier to rank consensus labels and incorporate the ranking
scores into model learning [15, 7]. Another research direction is to take all
noisy annotations separately as the input during the training process [55, 10,
16, 20, 43, 59, 50], but Wei _et al_. [56] concluded that label aggregation is
preferred over label separation when there are sufficient number of annotators
or the noise rate is moderate. These methods are not directly applicable to
object detection tasks since they do not involve regressing the locations of
bounding boxes.

##### Object Detection with Crowdsourced Annotations.

Hu and Meina [18] proposed Crowd R-CNN where one first pre-processes the
annotations by combining the bounding boxes using a clustering algorithm,
before aggregating the labels and learning the sensitivity of each annotator
[39] to different object classes during model training. Crowd R-CNN is
tailored for Faster R-CNN (FRCNN) [41] and is incompatible with one-stage
object detectors such as YOLOv7 [53]. Le _et al_. [27] proposed to pre-process
annotations using Weighted Boxes Fusion (WBF) [48] and introduced expert
agreement re-weighted loss (EARL) to emphasise boxes with higher annotator
agreement for model training. It requires access to each annotator’s
proficiency level to perform WBF and compute EARL, which is often challenging
to obtain for real-world crowdsourced data. Compared with the aforementioned
methods, the proposed BDC framework can incorporate any object detector in a
plug-and-play fashion without requiring any additional inputs such as
annotators’ proficiency levels. Furthermore, previous works [18, 27] evaluated
their methods on different synthetic datasets, with different numbers of
annotators and proficiency levels. For example, Hu and Meina [18] performed
evaluation on synthetic datasets derived from PASCAL VOC [13] and MS COCO
[29]; Le _et al_. [27] synthesised a MED-MNIST dataset based on MNIST [28] and
adopted the VinDr-CXR dataset [36]. Thus, their experiment results cannot be
compared directly.

##  3 Method

Problem Statement. We assume a training set $D=\\{x^{i},y^{i}\\}_{i=1}^{N}$
with pairs of input images $x^{i}$ and crowdsourced annotations
$y^{i}=\\{y^{i}_{m}\\}_{m=1}^{M^{i}}$, where $N$ is the total number of
training samples and $M^{i}$ is the number of annotations of the $i$-th image
from all annotators. Each annotation is a tuple of a bounding box, class label
and the annotator ID, $y^{i}_{m}=\\{(b^{i}_{m},c^{i}_{m},k^{i}_{m})\\}$ in
which $b^{i}_{m}\in\mathbb{R}^{4}$ contains the x-y coordinates of the box
centre as well as the width and height of the bounding box,
$c^{i}_{m}\in\\{1,...,J\\}$ represents the annotated class label of the object
and $k^{i}_{m}\in\\{1,...,K\\}$ is the annotator ID. $J$ denotes the number of
object classes while $K$ is the total number of annotators. Note that each
annotator may annotate a varying number of object instances. For brevity,
notations in the rest of this section are simplified to a single training
sample $i$. Our goal is to learn an object detector with noisy crowdsourced
annotations while inferring and leveraging annotators’ varying annotation
abilities.

###  3.1 Bayesian Detector Combination

An overview of the proposed crowdsourced annotation aggregation framework is
shown in Figure 2. At a high level, the proposed framework consists of four
components, namely: the object detector (OD) module, the annotations-
predictions matcher, the bounding box aggregator (BBA) and the class label
aggregator (CLA). Firstly, we perform a one-to-many matching of the
predictions generated by the OD to the crowdsourced annotations. The bounding
box error is computed and used to update a Gaussian-Gamma prior distribution
of the BBA for each annotator. Next, the class probability predictions by the
OD and the class label annotations are used to update a Dirichlet prior in the
CLA. The posterior distributions obtained from both aggregators are used to
aggregate the bounding boxes and class labels. Finally, we train and optimise
the OD’s parameters and aggregators’ parameters iteratively until convergence.
We present a detailed introduction of each component below.

####  3.1.1 Object Detector Module.

The object detector (OD) module with a parameter set, $\theta_{O}$, outputs a
varying number of $P$ predictions,
$\hat{y}=\\{(\hat{b}_{n},\hat{p}_{n})\\}_{n=1}^{P}$, given an input image,
$x$, where $\hat{b}_{n}\in\mathbb{R}^{4}$ and $\hat{p}_{n}\in[0,1]^{J}$ are
the predicted bounding box and class probability, respectively. We train the
model by minimising an overall loss function:

$\mathcal{L_{D}}=\frac{1}{H}\sum^{H}_{n=1}\gamma_{n}\mathcal{L}(\mathfrak{b}_{n},\rho_{n},\hat{y})\,\,,$
(1)

where $\gamma_{n}$, $\mathfrak{b}_{n}$ and $\rho_{n}$ are the loss scale
factor, the aggregated bounding box, and class probabilities obtained from the
two aggregators defined in Sections 3.1.3 and 3.1.4, respectively. $H$ is the
number of aggregated annotations. $\mathcal{L}$ is the loss function that the
base OD model adopts. Since the OD operates as a black-box module in the
proposed framework, any object detector can be used here as long as it outputs
the predicted bounding boxes and the associated class label probabilities.

####  3.1.2 Annotations-Predictions Matcher.

To update the priors of the aggregators, we first match each annotation to one
of the predictions. To find the optimal prediction, $\hat{y}_{m}^{*}$, for
each annotation, $y_{m}$, we minimise a matching cost function:

$\hat{y}_{m}^{*}=\operatorname*{arg\,min}_{\hat{y}_{n}\in\hat{y}}\mathcal{L}_{match}(\hat{y}_{n},y_{m})\,\,,$
(2)

where

$\mathcal{L}_{match}(\hat{y}_{n},y_{m})=-\hat{p}_{n(c_{m})}+\lambda_{1}\mathcal{L}_{IoU}(\hat{b}_{n},b_{m})+\lambda_{2}||\hat{b}_{n}-b_{m}||_{1}\,\,.$
(3)

$\lambda_{1},\lambda_{2}\in\mathbb{R}$ are hyperparameters and
$\mathcal{L}_{IoU}$ is the IoU loss [42]. In short, for each annotation, we
aim to find the prediction with a higher predicted probability of the
annotated class label, a higher IoU and a smaller L1 distance between the
predicted and the annotated bounding box. This is inspired by the set
prediction loss in DETR [9].

Instead of finding a one-to-one matching with a global minimum matching cost,
we are interested in a one-to-many matching that one prediction can be matched
to many crowdsourced annotations, with a local minimum matching cost, _i.e_.
finding the prediction with minimum cost for each annotation individually.
Moreover, a simple heuristic matching rule is sufficient instead of the
Hungarian algorithm in DETR which is more computationally demanding.

####  3.1.3 Bounding Box Aggregator.

In the Bounding Box Aggregator (BBA), the translation and scaling errors in
the x and y-axis of each annotator are modelled with a Gaussian distribution:

$p(\epsilon_{m}|k_{m}=k,\bm{\mu},\bm{\sigma})=\mathcal{N}({\mu}^{k},{\sigma}^{k})\,\,,$
(4)

where $\epsilon_{m}\in\mathbb{R}^{4}$ contains the translation and scaling
errors in the x and y-axis. $\bm{\mu}$ and $\bm{\sigma}$ are a collection of
Gaussian parameters with ${\mu}^{k}$ and ${\sigma}^{k}$ representing the mean
and covariance of $\epsilon_{m}$ for the annotator $k$, respectively. The
Gaussian distributions have a Gaussian-Gamma prior with hyperparameters
$\mu^{k}_{0}$, $\upsilon^{k}_{0}$ and $\beta^{k}_{0}$ [6, 35]. To update the
prior distribution, we compute the four error terms between all matched
prediction and annotation boxes for each annotator with:

$\epsilon_{m}=\Big{[}\begin{matrix}\hat{b}_{m(1)}^{*}-b_{m(1)},&&\hat{b}_{m(2)}^{*}-b_{m(2)},&&\hat{b}_{m(3)}^{*}\div
b_{m(3)},&&\hat{b}_{m(4)}^{*}\div b_{m(4)}\end{matrix}\Big{]}\,\,,$ (5)

where $\hat{b}_{m}^{*}$ and $b_{m}$ are the matched prediction and annotation
boxes, respectively. The subtraction and division operations are designed to
make the error terms translation-invariant and scale-invariant. The Gaussian-
Gamma prior is updated as follows:

$\tilde{\mu}^{k}=\frac{\mu^{k}_{0}+M^{k}\bar{\epsilon}^{k}}{M^{k}+1}\,\,,$ (6)

$\tilde{\upsilon}^{k}=\upsilon^{k}_{0}+\frac{M^{k}}{2}\,\,,$ (7)

$\tilde{\beta}^{k}=\beta^{k}_{0}+\frac{M^{k}(\bar{\epsilon}^{k}-\mu^{k}_{0})(\bar{\epsilon}^{k}-\mu^{k}_{0})^{\top}}{2(M^{k}+1)}+\frac{1}{2}\sum_{m=1}^{M}\text{1}[k_{m}=k](\epsilon_{m}-\bar{\epsilon}^{k})(\epsilon_{m}-\bar{\epsilon}^{k})^{\top}\,\,,$
(8)

where $M^{k}$ is the number of annotation-prediction matches associated with
the $k$-th annotator, $\bar{\epsilon}^{k}$ is the mean of all $\epsilon_{m}$
for the $k$-th annotator and $\text{1}[\cdot]$ is the indicator function.

The obtained posterior mean (_i.e_. $\mu^{k}=\tilde{\mu}^{k}$) is used to
correct the annotated boxes for each annotator:

$b_{m}:=(b_{m}+\begin{bmatrix}\mu^{k}_{(1)},&\mu^{k}_{(2)},&0,&0\end{bmatrix})\odot\begin{bmatrix}1,&1,&\mu^{k}_{(3)},&\mu^{k}_{(4)}\end{bmatrix}\,.$
(9)

All annotations matched to the same prediction are aggregated via a weighted
average using the posterior precision, $P^{k}=1\oslash diag(\sigma^{k})$,
where $\oslash$ denotes the element-wise division and
$\sigma^{k}=\tilde{\beta}^{k}/\tilde{\upsilon}^{k}$. The posterior precision,
$P^{k}$, of each annotator is used as weights to obtain the final boxes,
$\\{\mathfrak{b}_{n}\\}_{n=1}^{H}$, with

$\mathfrak{b}_{n}=\frac{\sum_{(b,k)\in\kappa_{n}}P^{k}\odot
b}{\sum_{(b,k)\in\kappa_{n}}P^{k}}\,\,.$ (10)

Here, $H$ indicates the number of predictions matched to at least one
annotation which is also equal to the number of aggregated annotations defined
in Section 3.1.1. $\kappa_{n}$ is the set of tuples consisting of the
annotated bounding boxes and their annotator ID that are matched to the $n$-th
prediction. Inspired by EARL [27], a loss scale factor, $\gamma_{n}$, is
computed for each aggregated box as the ratio of the number of matched
annotations to $K$ and is used to re-weight the loss function of the OD in
Equation 1. We refer the readers to Appendix 0.A for further details on the
updating procedure of the Bounding Box Aggregator.

####  3.1.4 Class Label Aggregator.

To aggregate crowdsourced class labels, we extend the Bayesian classifier
combination neural network (BCCNet) [19] to the proposed BDC framework. In the
class label aggregator (CLA), the annotated class label from each annotator,
$c_{m}$, is modelled as a multinomial distribution conditioning on the true
label of the object:

$p(c_{m}|k_{m}=k,t_{m}=j,\bm{\pi})={\pi}^{k}_{j,c_{m}}\,\,,$ (11)

where $\bm{\pi}$ is a collection of confusion matrices,
${\pi}^{k}\in\mathbb{R}^{J\times J}$ is the confusion matrix of the $k$-th
annotator and $t_{m}$ is the ground truth class label. Each row,
${\pi}^{k}_{j}$, provides the event probabilities of a multinomial
distribution for the $k$-th annotator when $t_{m}=j$ and is modelled with a
Dirichlet prior with hyperparameters
${\alpha}^{k}_{0j}=\\{\alpha^{k}_{0j,1},...,\alpha^{k}_{0j,J}\\}$ such that
$\pi^{k}_{j}|\alpha^{k}_{0j}\sim Dir(\alpha^{k}_{0j})$ [23].

The multinomial posterior distribution of the true label (_i.e_. the
aggregated class label probability) can be obtained as

$\rho_{n,j}=\exp\left(\ln\hat{p}_{n,j}+\sum_{(c,k)\in\tilde{\kappa}_{n}}\mathbb{E}_{\pi^{k}_{j}}\ln\pi^{k}_{j,c}\right)\,\,,$
(12)

where $\tilde{\kappa}_{n}$ is the set of tuples consisting of the annotated
class labels and their annotator ID that are matched to the $n$-th prediction.
$\hat{p}_{n,j}$ is the OD’s class-$j$ probability of the $n$-th prediction.
Note that the CLA outputs class probabilities (soft labels) unlike other
methods (_e.g_. MV) that output a single class (hard label). Appendix 0.B
provides the derivation for Equation 12 and a detailed discussion of the
updating steps of the Class Label Aggregator.

####  3.1.5 Training.

The optimal OD’s parameters, $\theta_{O}$, and aggregators’ parameters,
$\bm{\mu}$, $\bm{\upsilon}$, $\bm{\beta}$ and $\bm{\alpha}$, are learnt
iteratively in the BDC framework. Firstly, we initialise the parameters either
randomly or with prior knowledge of the domain. Next, we train the OD by
optimising the loss function, $\mathcal{L_{D}}$, with the aggregated
annotations obtained using the current values of the aggregators’ parameters.
After one epoch of training, we obtain the OD’s predictions using the current
value of $\theta_{O}$ to update the prior distributions of aggregators. These
two steps are repeated iteratively until the parameters converge. See Appendix
0.C for a further discussion of the training steps.

##  4 Experiments

Training details. To demonstrate the generalisability and robustness of BDC in
incorporating different object detection algorithms, three common object
detection algorithms are used in our experiments, YOLOv7 [53], FRCNN [41] with
the ResNet50 backbone and EVA [14]. During training, we randomly apply data
augmentation such as rotation, translation, zooming and horizontal flipping to
the training images and annotations. $\lambda_{1},\lambda_{2}$ in Equation 3
are set to 2 and 5, respectively, following Carion _et al_. [9]. Lastly, the
Gaussian-Gamma prior of the BBA, $\bm{\mu}_{0}$, $\bm{\upsilon}_{0}$ and
$\bm{\beta}_{0}$, are set to 0, 10 and 0.5, respectively, while the Dirichlet
prior of the CLA, $\bm{\alpha_{0}}$, is initialised to 1 and each diagonal
entry is set to 10. The complete training configuration can be found in
Appendix 0.D.

##### Comparison against other aggregation methods.

We compare the performance of BDC against multiple baselines as shown in
Tables 1, 2, 3 and 4. ‘NA’ denotes the process of training the model with all
the crowdsourced annotations as the ground truth annotations [56]. ‘MV’ stands
for majority voting where we apply clustering to the crowdsourced annotations,
then aggregate the class labels via MV and aggregate the bounding boxes as the
biggest box where the majority had annotated pixel-wise, following the
practice in [46]. We also implement Crowd R-CNN [18] and WBF-EARL [27]
designed for crowdsourced object detection. For WBF-EARL [27] which requires
the knowledge of annotators’ proficiency levels, we followed the settings in
[27] for the VinDR-CXR dataset while for the synthetic datasets, they are set
to the overall accuracy of the annotators.

###  4.1 Real Crowdsourced Datasets

##### VinDr-CXR.

We evaluate the annotation aggregation methods on VinDr-CXR [36], a thoracic
abnormalities detection dataset. VinDr-CXR consists of 14 abnormalities
classes and contains 15,000 training and 3,000 testing chest radiograph
images. The training images are annotated by 17 experienced radiologists where
at least three annotators are randomly assigned to annotate each image.
Figures 1(f), 1(g), 1(h), 1(i) and 1(j) show examples of the annotations with
high bounding box and class label disagreement by the annotators. Furthermore,
the testing images are annotated with the consensus of five expert
radiologists through a two-stage labelling process and the test set
annotations are not publicly available. The predictions are submitted to the
VinBigData Kaggle [12] to obtain mean average precision (AP) at 0.4 IoU
threshold on the private test set.

##### Disaster Response Dataset.

We further use the disaster response dataset [19] where the task is to detect
undamaged and damaged buildings on satellite images taken after hurricane
Maria in 2017. The dataset consisting of 464 satellite images of Dominica is
annotated by 13 semi-professional paid crowd members to ensure high labelling
quality. Despite this effort, labels from different crowd members vary
significantly (Figures 1(k), 1(l), 1(m), 1(n) and 1(o)). Consequently, the
dataset does not contain ground truth labels. Thus, its qualitative results
are presented in Appendix 0.E.

Table 1: AP.4 results on VinDr-CXR [36] dataset obtained from the private
score of VinBigData Kaggle competition [12]. Crowd R-CNN can only integrate
FRCNN, hence there are no results reported for the two other entries
corresponding to Crowd R-CNN.

Method | Test AP.4
---|---
YOLOv7 | FRCNN | EVA
NA | 17.4 | 17.2 | 7.8
MV | 13.9 | 16.3 | 8.2
Crowd R-CNN [18] | - | 16.7 | -
WBF-EARL [27] | 16.4 | 17.0 | 8.4
BDC (ours) | 19.2 | 17.9 | 8.9

##### Quantitative Results.

Table 1 presents the AP.4 metric obtained from the VinBigData Kaggle
competition and shows that the proposed BDC leads to higher AP.4 than other
methods across different OD models on the VinDr-CXR test set. As Crowd R-CNN
can only integrate FRCNN, there are no results reported for the two other
entries corresponding to Crowd R-CNN.

##### Qualitative Results.

We visualise the results of the aggregated annotations from each method on the
VinDr-CXR dataset in Figure 3. Since the true ground truth of the datasets is
not provided, we use ‘MV’ as the best approximation of the ground truth for
this qualitative assessment and observe that BDC produces more accurate
aggregation compared to the other methods.

(a) NA (b) MV

(c) WBF-EARL (d) Crowd R-CNN

(e) BDC (ours)

Figure 3: Comparison of aggregated labels from different methods on the VinDr-
CXR [36]. For WBF-EARL [27], the number beside the class label is the
annotators’ level of agreement while for Crowd R-CNN [18] and BDC, the number
indicates the class probability. For NA, the colours represent the different
annotators.

###  4.2 Synthetic Datasets

To alleviate the limitations in the available real-world crowdsourced datasets
including the lack of ground truth labels, we synthesise several synthetic
datasets with crowdsourced annotations to enable the systematic evaluation of
the proposed method. This makes available the ground truth annotations to
allow us to obtain detailed test evaluation metrics (e.g., AP.5, AP.75,
AP.5:.95) and verify the performance of the proposed method by evaluating the
aggregated crowdsourced annotations with the true ground truth annotations of
the training dataset (indicated as ‘Train Aggregation AP’ in Tables 2, 3 and
4). Two common object detection datasets are used to construct the synthetic
datasets: (1) Pascal Visual Object Classes (VOC) 2007 [13], a 20-class object
detection dataset which consists of 5,011 and 4,952 images for training and
testing, respectively; (2) Microsoft Common Objects in Context (COCO) 2017
[29], a large-scale 80-class object detection dataset consisting of 118,287
training and 5,000 validation images.

##### Synthetic settings.

Four synthetic settings are investigated in this work: (1) _VOC-FULL_ and (2)
_COCO-FULL_ where 10 annotators with average skill levels annotate the whole
datasets; (3) _VOC-MIX_ where the first 5 annotators are experts while the
remaining 20 annotators have an average skill level. All annotators annotate
the whole dataset since Pascal VOC is a small dataset; and (4) _COCO-MIX_
where 20 experts, 550 annotators with average skill level and 330 annotators
with poor skill level work together to annotate the MS COCO dataset.
Additionally, the last 100 are random annotators, _i.e_., the annotated class
labels and bounding boxes are drawn from a uniform distribution. Since MS COCO
is a large-scale dataset, we set that each annotator annotates a small subset
of images, _i.e_., the coverage for the expert, average, poor and random
annotators are 5%, 1%, 0.5% and 1%, respectively. Note that all annotators in
these settings are simulated using deep learning models (See Appendix 0.F for
further discussions of the synthesisation method and pseudocode). Additional
experiments where we directly used the object detectors’ predictions as
synthetic crowdsourced annotations can be found in Appendix 0.H.

Table 2: Test and train aggregation AP results for YOLOv7 model trained with
different annotation aggregation methods on synthesised and ground truth
datasets.

Dataset | Method | Test AP | Train Aggregation AP
---|---|---|---
AP.5 | AP.75 | AP.5:.95 | AP.5 | AP.75 | AP.5:.95
VOC- FULL | NA | 59.7 | 45.3 | 40.7 | 10.2 | 9.9 | 9.9
MV | 72.2 | 52.3 | 47.3 | 62.0 | 48.5 | 48.5
WBF-EARL [27] | 69.3 | 52.3 | 45.7 | 22.3 | 16.4 | 14.5
BDC (ours) | 84.9 | 63.5 | 54.8 | 95.0 | 80.6 | 66.3
VOC- MIX | NA | 46.1 | 27.6 | 27.4 | 4.4 | 4.3 | 4.3
MV | 77.1 | 61.0 | 53.7 | 75.7 | 70.6 | 68.6
WBF-EARL [27] | 61.0 | 47.2 | 40.9 | 27.9 | 24.9 | 20.0
BDC (ours) | 86.2 | 69.0 | 58.8 | 97.6 | 94.9 | 77.2
VOC | GT | 87.7 | 74.0 | 65.3 | 100 | 100 | 100
COCO- FULL | NA | 53.4 | 41.4 | 37.4 | 8.6 | 8.2 | 8.2
MV | 61.9 | 46.4 | 42.7 | 52.5 | 44.4 | 43.3
WBF-EARL [27] | 55.6 | 42.5 | 38.9 | 14.6 | 8.9 | 8.6
BDC (ours) | 65.0 | 49.6 | 44.5 | 83.6 | 66.1 | 56.9
COCO- MIX | NA | 53.4 | 41.2 | 37.5 | 8.5 | 7.9 | 8.0
MV | 60.9 | 46.4 | 42.4 | 45.8 | 42.3 | 41.1
WBF-EARL [27] | 56.2 | 42.6 | 39.1 | 12.1 | 7.0 | 7.0
BDC (ours) | 64.8 | 49.3 | 44.4 | 80.5 | 62.9 | 54.3
COCO | GT | 67.5 | 52.8 | 48.6 | 100 | 100 | 100

##### Quantitative Results.

Tables 2, 3 and 4 report the experiment results of YOLOv7, FRCNN and EVA
trained with different aggregation methods on all synthetic settings,
respectively. Additionally, we trained each model with ground truth
annotations of each dataset (denoted by ‘GT’) to obtain reference optimal
model performance. The aggregation APs of the train dataset for NA and WBF-
EARL are low because NA directly uses all the crowdsourced annotations while
WBF outputs multiple bounding boxes per class per object instance, leading to
a much lower precision (see Figure 4 for examples). We do not include
performance from Crowd R-CNN for the COCO-MIX setting as its training time
increases linearly with $K$ which means that the training time is
prohibitively high for this dataset where $K=1,000$. In comparison, the extra
computational cost introduced by the proposed BDC is small when compared to
the much heavier training cost of the OD backbone (see Section 4.4 for an
analysis of the computational cost). Tables 2, 3 and 4 demonstrate the
superior performance of BDC compared to other methods from simpler synthetic
settings to more complex synthetic crowdsourced annotations, _i.e_. from VOC-
FULL to COCO-MIX. In addition, BDC achieves the highest train aggregation AP
compared to other methods across all settings.

Table 3: Test and train aggregation AP results for FRCNN model trained with
different annotation aggregation methods on synthesised and ground truth
datasets. Crowd R-CNN has a training time that increases linearly with $K$,
hence there is no result reported for COCO-MIX dataset with $K=1,000$.

Dataset | Method | Test AP | Train Aggregation AP
---|---|---|---
AP.5 | AP.75 | AP.5:.95 | AP.5 | AP.75 | AP.5:.95
VOC- FULL | NA | 47.4 | 11.3 | 19.4 | 10.2 | 9.9 | 9.9
MV | 79.7 | 53.5 | 48.8 | 62.0 | 48.5 | 48.5
Crowd R-CNN [18] | 71.6 | 33.9 | 30.1 | 58.3 | 43.0 | 31.5
WBF-EARL [27] | 71.8 | 37.7 | 38.7 | 22.3 | 16.4 | 14.5
BDC (ours) | 83.7 | 56.3 | 51.0 | 95.0 | 80.4 | 66.3
VOC- MIX | NA | 34.1 | 4.4 | 12.2 | 4.4 | 4.3 | 4.3
MV | 82.0 | 57.1 | 51.9 | 75.7 | 70.6 | 68.6
Crowd R-CNN [18] | 70.2 | 31.2 | 29.3 | 60.6 | 44.2 | 31.0
WBF-EARL [27] | 69.8 | 36.5 | 37.4 | 27.9 | 24.9 | 20.0
BDC (ours) | 84.7 | 59.3 | 52.9 | 97.6 | 94.6 | 76.9
VOC | GT | 84.6 | 62.1 | 55.2 | 100 | 100 | 100
COCO- FULL | NA | 39.7 | 12.0 | 17.7 | 8.6 | 8.2 | 8.2
MV | 55.6 | 35.7 | 33.4 | 52.5 | 44.4 | 43.3
Crowd R-CNN [18] | 48.5 | 27.1 | 24.2 | 49.5 | 36.2 | 33.7
WBF-EARL [27] | 51.9 | 28.0 | 28.4 | 14.6 | 8.9 | 8.6
BDC (ours) | 56.6 | 35.3 | 33.5 | 83.3 | 66.0 | 56.7
COCO- MIX | NA | 43.9 | 17.5 | 21.4 | 8.5 | 7.9 | 8.0
MV | 54.8 | 34.5 | 32.9 | 45.8 | 42.3 | 41.1
Crowd R-CNN [18] | - | - | - | - | - | -
WBF-EARL [27] | 51.9 | 28.0 | 28.3 | 12.1 | 7.0 | 7.0
BDC (ours) | 56.1 | 34.6 | 33.0 | 80.3 | 62.7 | 54.1
COCO | GT | 58.0 | 38.5 | 35.7 | 100 | 100 | 100

Table 4: Test and train aggregation AP results for EVA model trained with
different annotation aggregation methods on synthesised and ground truth
datasets.

Dataset | Method | Test AP | Train Aggregation AP
---|---|---|---
AP.5 | AP.75 | AP.5:.95 | AP.5 | AP.75 | AP.5:.95
VOC- FULL | NA | 90.9 | 80.9 | 69.7 | 10.2 | 9.9 | 9.9
MV | 91.0 | 82.2 | 71.2 | 62.0 | 48.5 | 48.5
WBF-EARL [27] | 93.0 | 83.5 | 71.1 | 22.3 | 16.4 | 14.5
BDC (ours) | 94.2 | 85.6 | 71.7 | 95.3 | 81.2 | 66.7
VOC- MIX | NA | 86.6 | 65.7 | 57.5 | 4.4 | 4.3 | 4.3
MV | 91.7 | 83.0 | 72.9 | 75.7 | 70.6 | 68.6
WBF-EARL [27] | 91.3 | 82.2 | 68.7 | 27.9 | 24.9 | 20.0
BDC (ours) | 94.9 | 86.5 | 73.8 | 98.8 | 95.0 | 77.3
VOC | GT | 94.9 | 88.1 | 78.0 | 100 | 100 | 100
COCO- FULL | NA | 71.8 | 55.7 | 50.6 | 8.6 | 8.2 | 8.2
MV | 74.8 | 61.0 | 54.4 | 52.5 | 44.4 | 43.3
WBF-EARL [27] | 74.7 | 58.4 | 51.8 | 14.6 | 8.9 | 8.6
BDC (ours) | 78.0 | 62.7 | 55.1 | 83.3 | 65.1 | 55.3
COCO- MIX | NA | 73.3 | 57.2 | 51.7 | 8.5 | 7.9 | 8.0
MV | 75.0 | 60.6 | 54.2 | 45.8 | 42.3 | 41.1
WBF-EARL [27] | 75.2 | 59.3 | 52.9 | 12.1 | 7.0 | 7.0
BDC (ours) | 77.1 | 62.5 | 54.8 | 79.9 | 62.2 | 53.2
COCO | GT | 78.8 | 64.9 | 58.8 | 100 | 100 | 100

##### Qualitative Results.

Figure 4 visualises the results of the aggregated annotations from each method
on the VOC-MIX synthetic dataset. WBF performs box fusion on a per-class basis
and results in significantly more boxes compared to other methods (excluding
NA) which contributed to the low train aggregation AP and noisy labels during
model training. On the other hand, although both Crowd R-CNN and BDC output
class probabilities, the probability distribution of Crowd R-CNN is heavily
skewed which is similar to hard labels, while BDC outputs the learnt class
probabilities and can benefit from these soft labels during model training
[34]. Besides, BDC can cope with failure cases of its aggregator as shown by
the lower class probability of incorrect aggregation. Additional qualitative
results can be found in Appendix 0.I.

(a) NA (b) MV

(c) WBF-EARL (d) Crowd R-CNN

(e) BDC (ours) (f) Ground truth

Figure 4: Comparison of aggregated labels from different methods on the VOC-
MIX synthetic dataset. For WBF-EARL [27], the number beside the class label is
the annotators’ level of agreement while for Crowd R-CNN [18] and BDC, the
number indicates the class probability. For NA, the colours represent the
different annotators.

###  4.3 Scalability and Robustness of BDC

We analyse the scalability and robustness of the aggregation methods using the
Pascal VOC dataset. YOLOv7 is selected as the OD because it is lightweight and
fast to train. Figure 5(a) demonstrates the effect of varying numbers of
annotators, $K$, where all annotations are synthesised using annotators of
average skill level. Note that training YOLOv7 with the NA method results in a
GPU out-of-memory error when $K\geq 100$. When the number of annotators
increases, the performance of YOLOv7 increases for both BDC and MV while the
performance of both NA and WBF-EARL decreases when $K>50$. In addition, BDC
has an average AP.5:.95 improvement of 4.35 over MV.

We also investigate the effect of changing the percentage of noisy annotators
while setting $K$ to 25. Here, noisy annotators are defined as annotators with
a poor skill level while the other annotators are annotators with expert skill
level. As illustrated in Figure 5(b), we found that AP.5:.95 of all
aggregation methods reduce when we increase the percentage of noisy
annotators. However, MV drops at a faster rate compared to the proposed method
when the noisy annotator percentage is over $50\%$. We observe that BDC
remains robust even when 100% of the annotators have low skill levels and the
AP.5:.95 is 18.5 higher than that obtained with MV. The consistent performance
of BDC (up to 80% noise) can be attributed to the ability of BDC to correctly
capture the ability of each annotator and assign proper weights when
aggregating the annotations (See Appendix 0.J for an analysis of the posterior
distributions learnt in the aggregators).

(a) Scalability of BDC

(b) Robustness of BDC

Figure 5: (a) Effect of the number of annotators, $K$, and (b) effect of
varying percentage of noisy annotators (with $K=25$) on the test AP.5:.95 of
YOLOv7. Results for the ‘NA’ method are unavailable when $K\geq 100$ due to
GPU out-of-memory error.

###  4.4 Computational Cost

Table 5 presents the wall time of each annotation aggregation method spent to
pre-process the annotations and train the Faster R-CNN (FRCNN) [41] model for
one epoch on the VOC-MIX synthetic dataset. The wall time is computed with a
desktop workstation with an i9-10900X CPU and an RTX 3090 GPU. As shown in
Table 5, MV, Crowd R-CNN and WBF-EARL pre-process the annotations before model
training while NA and the proposed BDC require no pre-processing step. Crowd
R-CNN has the longest training time which increases linearly with the number
of annotators, $K$. On the other hand, the proposed BDC performs the
annotations-predictions matching step and the updating steps of the
aggregators’ parameters every training epoch and therefore has a slightly
longer training time compared to NA, MV and WBF-EARL. However, this increase
in training time is small when compared to the much higher training cost of
the OD module (618.6s OD training time out of 662.9s). Note that the model
inference time is unaffected by the type of annotation aggregation method.

Table 5: Annotations pre-processing time and FRCNN model training time per
epoch of different annotation aggregation methods on the VOC-MIX dataset. The
wall time is computed on a desktop workstation with an i9-10900X CPU and an
RTX 3090 GPU.

Method | Pre-processing time (s) | Training time (s)
---|---|---
NA | 0.0 | 537.0
MV | 19.4 | 505.4
Crowd R-CNN [18] | 15.6 | 1175.5
WBF-EARL [27] | 4.9 | 518.5
BDC (ours) | 0.0 | 662.9

##  5 Conclusion

We first presented a general annotation aggregation and object detection
framework, named Bayesian Detector Combination (BDC), for learning a robust
object detector from noisy crowdsourced annotations. BDC infers the consensus
bounding boxes and corresponding soft labels without the need for ground truth
annotations. BDC is model-agnostic and we demonstrated its superior
performance by integrating it with three popular object detector backbones,
namely YOLOv7, FRCNN, and EVA. Furthermore, we constructed four synthetic
datasets to simulate different crowdsourcing scenarios involving annotators of
various skill levels, enabling the systematic evaluation of the proposed BDC.
We demonstrated the improved performance from BDC on real-world datasets and
all the synthetic datasets when compared to other object detectors designed
for crowdsourced annotations. We showed that BDC can scale against varying
numbers of annotators and is robust against annotators with low annotation
reliability.

##### Acknowledgements.

Z.Q. T. acknowledges the support of an Industrial Cooperative Awards in
Science and Engineering (ICASE) studentship from the Engineering and Physical
Sciences Research Council (EPSRC) for this work. G. C. acknowledges the
support of the Engineering and Physical Sciences Research Council (EPSRC)
through grant EP/Y018036/1 and the Australian Research Council (ARC) through
grant FT190100525. The authors would like to thank the Satellite Applications
Catapult for the provision of the disaster response dataset.

## References

  * [1] Abramowitz, M., Stegun, I.A.: Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables, chap. 6, pp. 255–258. Dover, Mineola, NY, USA, 10 edn. (1972)
  * [2] Abramowitz, M., Stegun, I.A.: Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables, chap. 6, pp. 258–259. Dover, Mineola, NY, USA, 10 edn. (1972)
  * [3] Albarqouni, S., Baur, C., Achilles, F., Belagiannis, V., Demirci, S., Navab, N.: AggNet: Deep learning from crowds for mitosis detection in breast cancer histology images. IEEE Trans. Med. Imag. 35(5), 1313–1321 (May 2016)
  * [4] Baumgartner, C.F., Tezcan, K.C., Chaitanya, K., Hötker, A.M., Muehlematter, U.J., Schawkat, K., Becker, A.S., Donati, O., Konukoglu, E.: PHiSeg: Capturing uncertainty in medical image segmentation. In: Med. Imag. Comput. Comput. Assist. Interv. pp. 119–127. Shenzhen, China (2019)
  * [5] Bechtel, B., Demuzere, M., Sismanidis, P., Fenner, D., Brousse, O., Beck, C., Van Coillie, F., Conrad, O., Keramitsoglou, I., Middel, A., Mills, G., Niyogi, D., Otto, M., See, L., Verdonck, M.L.: Quality of crowdsourced data on urban morphology—the human influence experiment (HUMINEX). Urban Sci. 1(2) (May 2017)
  * [6] Bernard, J.M., Smith, A.F.M.: Bayesian Theory, chap. 3, pp. 136–138. Wiley, West Sussex, UK (1993)
  * [7] Bernhardt, M., Castro, D.C., Tanno, R., Schwaighofer, A., Tezcan, K.C., Monteiro, M., Bannur, S., Lungren, M.P., Nori, A., Glocker, B., Alvarez-Valle, J., Oktay, O.: Active label cleaning for improved dataset quality under resource constraints. Nature Commun. 13(1), 1161 (Mar 2022)
  * [8] Budd, S., Day, T., Simpson, J., Lloyd, K., Matthew, J., Skelton, E., Razavi, R., Kainz, B.: Can non-specialists provide high quality gold standard labels in challenging modalities? In: Med. Imag. Comput. Comput. Assist. Interv. pp. 251–262. Strasbourg, France (2021)
  * [9] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Eur. Conf. Comput. Vis. pp. 213–229 (2020)
  * [10] Chen, Z., Wang, H., Sun, H., Chen, P., Han, T., Liu, X., Yang, J.: Structured probabilistic end-to-end learning from crowds. In: Int. Joint Conf. Artif. Intell. pp. 1512–1518 (2020)
  * [11] Dawid, A.P., Skene, A.M.: Maximum likelihood estimation of observer error-rates using the EM algorithm. J. Roy. Statist. Soc. Ser. C (Appl. Statist.) 28(1), 20–28 (1979)
  * [12] DungNB, Nguyen, H.Q., Elliott, J., KeepLearning, Nhan, N.T., Culliton, P.: VinBigData chest x-ray abnormalities detection (2020), https://kaggle.com/competitions/vinbigdata-chest-xray-abnormalities-detection
  * [13] Everingham, M., Luc Van Gool, C.K.I.W., Winn, J., Zisserman, A.: The PASCAL Visual Object Classes Challenge 2007 (VOC2007). Int. J. Comput. Vis. 88, 303–338 (Jun 2010)
  * [14] Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., Cao, Y.: EVA: Exploring the limits of masked visual representation learning at scale. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 19358–19369. Vancouver, Canada (2023)
  * [15] Goh, H.W., Tkachenko, U., Mueller, J.: CROWDLAB: Supervised learning to infer consensus labels and quality scores for data with multiple annotators (2022), arXiv:2210.06812
  * [16] Guan, M.Y., Gulshan, V., Dai, A.M., Hinton, G.E.: Who said what: Modeling individual labelers improves classification. In: AAAI Conf. Artif. Intell. pp. 3109–3118. San Francisco, CA, USA (2017)
  * [17] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 770–778. Las Vegas, NV, USA (2016)
  * [18] Hu, Y., Meina, S.: Crowd R-CNN: An object detection model utilizing crowdsourced labels. In: Int. Conf. Vis. Image Sig. Process. pp. 1–7. Bangkok, Thailand (2020)
  * [19] Isupova, O., Li, Y., Kuzin, D., Roberts, S.J., Willis, K.J., Reece, S.: BCCNet: Bayesian classifier combination neural network. In: Neural Inf. Process. Syst. Workshop Mach. Learn. Develop. Montréal, Canada (2018)
  * [20] Jensen, M.H., Jørgensen, D.R., Jalaboi, R., Hansen, M.E., Olsen, M.A.: Improving uncertainty estimation in convolutional neural networks using inter-rater agreement. In: Med. Imag. Comput. Comput. Assist. Interv. pp. 540–548. Shenzhen, China (2019)
  * [21] Ji, W., Yu, S., Wu, J., Ma, K., Bian, C., Bi, Q., Li, J., Liu, H., Cheng, L., Zheng, Y.: Learning calibrated medical image segmentation via multi-rater agreement modeling. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 12336–12346 (2021)
  * [22] Jungo, A., Meier, R., Ermis, E., Blatti-Moreno, M., Herrmann, E., Wiest, R., Reyes, M.: On the effect of inter-observer variability for a reliable estimation of uncertainty of medical image segmentation. In: Med. Imag. Comput. Comput. Assist. Interv. pp. 682–690. Granada, Spain (2018)
  * [23] Kim, H., Ghahramani, Z.: Bayesian classifier combination. In: Int. Conf. Artif. Intell. Statist. pp. 619–627. La Palma, Canary Islands (2012)
  * [24] Kohl, S.A.A., Romera-Paredes, B., Meyer, C., Fauw, J.D., Ledsam, J.R., Maier-Hein, K.H., Eslami, S.M.A., Rezende, D.J., Ronneberger, O.: A probabilistic u-net for segmentation of ambiguous images. In: Adv. Neural Inform. Process. Syst. pp. 6965–6975. Montréal, Canada (2018)
  * [25] Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Adv. Neural Inform. Process. Syst. pp. 1097–1105. Lake Tahoe, NV, USA (2012)
  * [26] Lazarus, E., Mainiero, M.B., Schepps, B., Koelliker, S.L., Livingston, L.S.: BI-RADS lexicon for US and mammography: interobserver variability and positive predictive value. Radiology 239(2), 385–391 (Mar 2006)
  * [27] Le, K., Tran, T., Pham, H., Nguyen Trung, H., Le, T., Nguyen, H.Q.: Learning from multiple expert annotators for enhancing anomaly detection in medical image analysis. IEEE Access 11, 14105–14114 (2023)
  * [28] LeCun, Y., Cortes, C., Burges, C.: MNIST handwritten digit database (2010), http://yann.lecun.com/exdb/mnist
  * [29] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollar, P., Zitnick, L.: Microsoft COCO: Common objects in context. In: Eur. Conf. Comput. Vis. pp. 740–755. Zurich, Switzerland (2014)
  * [30] Lloyd, S.P.: Least squares quantization in PCM. IEEE Trans. Inf. Theory 28(2), 129–137 (Mar 1982)
  * [31] Loshchilov, I., Hutter, F.: SGDR: Stochastic gradient descent with warm restarts. In: Int. Conf. Learn. Represent. Toulon, France (2017)
  * [32] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: Int. Conf. Learn. Represent. New Orleans, LA, USA (2019)
  * [33] Maier-Hein, L., Mersmann, S., Kondermann, D., Stock, C., Kenngott, H.G., Sanchez, A., Wagner, M., Preukschas, A., Wekerle, A.L., Helfert, S., Bodenstedt, S., Speidel, S.: Crowdsourcing for reference correspondence generation in endoscopic images. In: Med. Imag. Comput. Comput. Assist. Interv. pp. 349–356. Boston, MA, USA (2014)
  * [34] Müller, R., Kornblith, S., Hinton, G.: When does label smoothing help? In: Adv. Neural Inform. Process. Syst. pp. 4694–4703. Vancouver, Canada (2019)
  * [35] Murphy, K.: Conjugate bayesian analysis of the gaussian distribution (2007), https://www.cs.ubc.ca/~murphyk/Papers/bayesGauss.pdf
  * [36] Nguyen, H.Q., Lam, K., Le, L.T., Pham, H.H., Tran, D.Q., Nguyen, D.B., Le, D.D., Pham, C.M., Tong, H.T.T., Dinh, D.H., Do, C.D., Doan, L.T., Nguyen, C.N., Nguyen, B.T., Nguyen, Q.V., Hoang, A.D., Phan, H.N., Nguyen, A.T., Ho, P.H., Ngo, D.T., Nguyen, N.T., Nguyen, N.T., Dao, M., Vu, V.: VinDr-CXR: An open dataset of chest x-rays with radiologist’s annotations. Sci. Data 9(1), 429 (Jul 2022)
  * [37] Nowruzi, F.E., Kapoor, P., Kolhatkar, D., Hassanat, F., Laganiere, R., Rebut, J.: How much real data do we actually need: Analyzing object detection performance using synthetic and real data. In: Int. Conf. Mach. Learn. Workshop AI Auton. Driving. Long Beach, CA, USA (2019)
  * [38] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: Pytorch: An imperative style, high-performance deep learning library. In: Adv. Neural Inform. Process. Syst. pp. 8024–8035. Vancouver, Canada (2019)
  * [39] Raykar, V.C., Yu, S., Zhao, L.H., Valadez, G.H., Florin, C., Bogoni, L., Moy, L., Raykar, C., Valadez, G.H.: Learning from crowds. J. Mach. Learn. Res. 11(43), 1297–1322 (Aug 2010)
  * [40] Redmon, J., Farhadi, A.: YOLOv3: An incremental improvement (2018), arXiv:1804.02767
  * [41] Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with region proposal networks. In: Adv. Neural Inform. Process. Syst. pp. 91–99. Montréal, Canada (2015)
  * [42] Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., Savarese, S.: Generalized intersection over union: A metric and a loss for bounding box regression. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 658–666. Long Beach, CA, USA (2019)
  * [43] Rodrigues, F., Pereira, F.C.: Deep learning from crowds. In: AAAI Conf. Artif. Intell. pp. 1611–1618. New Orleans, LA, USA (2018)
  * [44] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (Dec 2015)
  * [45] See, L., Comber, A., Salk, C., Fritz, S., van der Velde, M., Perger, C., Schill, C., McCallum, I., Kraxner, F., Obersteiner, M.: Comparing the quality of crowdsourced data contributed by expert and non-experts. PLoS One 8(7), e69958 (Jul 2013)
  * [46] Sheng, V.S., Provost, F., Ipeirotis, P.G.: Get another label? improving data quality and data mining using multiple, noisy labelers. In: ACM SIGKDD Int. Conf. Knowl. Discov. Data Min. pp. 614–622. Las Vegas, NV, USA (2008)
  * [47] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: Int. Conf. Learn. Represent. San Diego, CA, USA (2015)
  * [48] Solovyev, R., Wang, W., Gabruseva, T.: Weighted boxes fusion: Ensembling boxes from different object detection models. Image Vis. Comput. 107, 104–117 (Mar 2021)
  * [49] Song, H., Kim, M., Park, D., Shin, Y., Lee, J.G.: Learning from noisy labels with deep neural networks: A survey. IEEE Trans. Neural Netw. Learn. Syst. 34(11), 8135–8153 (Mar 2022)
  * [50] Sudre, C.H., Anson, B.G., Ingala, S., Lane, C.D., Jimenez, D., Haider, L., Varsavsky, T., Tanno, R., Smith, L., Ourselin, S., Jäger, R.H., Cardoso, M.J.: Let’s agree to disagree: Learning highly debatable multirater labelling. In: Med. Imag. Comput. Comput. Assist. Interv. pp. 665–673. Shenzhen, China (2019)
  * [51] Venanzi, M., Guiver, J., Kazai, G., Kohli, P., Shokouhi, M.: Community-based bayesian aggregation models for crowdsourcing. In: Int. Conf. World Wide Web. pp. 155–164. Seoul, Korea (2014)
  * [52] Wainwright, M.J., Jordan, M.I.: Graphical models, exponential families, and variational inference. Found. Trends® Mach. Learn. 1(1–2), 1–305 (Nov 2008)
  * [53] Wang, C.Y., Bochkovskiy, A., Liao, H.Y.M.: YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 7464–7475. Vancouver, Canada (2023)
  * [54] Watadani, T., Sakai, F., Johkoh, T., Noma, S., Akira, M., Fujimoto, K., Bankier, A.A., Lee, K.S., Müller, N.L., Song, J.W., Park, J.S., Lynch, D.A., Hansell, D.M., Remy-Jardin, M., Franquet, T., Sugiyama, Y.: Interobserver variability in the CT assessment of honeycombing in the lungs. Radiology 266(3), 936–944 (Dec 2012)
  * [55] Wei, H., Xie, R., Feng, L., Han, B., An, B.: Deep learning from multiple noisy annotators as a union. IEEE Trans. Neural Netw. Learn. Syst. pp. 1–11 (Apr 2022)
  * [56] Wei, J., Zhu, Z., Luo, T., Amid, E., Kumar, A., Liu, Y.: To aggregate or not? learning with separate noisy labels. In: ACM SIGKDD Int. Conf. Knowl. Discov. Data Min. pp. 2523–2535. Long Beach, CA, USA (2023)
  * [57] Whitehill, J., Wu, T.f., Bergsma, J., Movellan, J., Ruvolo, P.: Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In: Adv. Neural Inform. Process. Syst. pp. 2035–2043. Vancouver, Canada (2009)
  * [58] Wu, J., Fang, H., Yang, Y., Liu, Y., Gao, J., Duan, L., Yang, W., Xu, Y.: Multi-rater prism: Learning self-calibrated medical image segmentation from multiple raters (2022), arXiv:2212.00601
  * [59] Yu, S., Zhou, H.Y., Ma, K., Bian, C., Chu, C., Liu, H., Zheng, Y.: Difficulty-aware glaucoma classification with multi-rater consensus modeling. In: Med. Imag. Comput. Comput. Assist. Interv. pp. 741–750 (2020)
  * [60] Zhang, L., Tanno, R., Xu, M.C., Jin, C., Jacob, J., Ciccarelli, O., Barkhof, F., Alexander, D.C.: Disentangling human error from the ground truth in segmentation of medical images. In: Adv. Neural Inform. Process. Syst. pp. 15750–15762 (2020)
  * [61] Zhu, X., Vondrick, C., Ramanan, D., Fowlkes, C.: Do we need more training data or better models for object detection? In: Brit. Mach. Vis. Conf. Surrey, UK (2012)

Supplementary Material

This supplementary material provides the derivation of the updating steps of
the Bounding Box Aggregator and Class Label Aggregator in Appendices 0.A and
0.B, respectively, as well as the overall training steps and pseudocode of the
Bayesian Detector Combination (BDC) framework in Appendix 0.C. Moreover,
Appendix 0.D presents the training configuration of the experiments while
Appendices 0.F and 0.G further discuss the synthesisation procedure to
generate synthetic crowdsourced annotations. Besides, Appendix 0.H presents
additional experiment results for the setting where we use the predictions of
object detectors as crowdsourced annotations. Lastly, Appendices 0.E and 0.I
show additional qualitative detection results on the disaster response dataset
and the synthetic dataset, respectively, while Appendix 0.J provides an
analysis of the learnt posterior distributions of BDC on the COCO-MIX dataset.

##  Appendix 0.A Bounding Box Aggregator

We present in this section the derivation of the equations (Equations (6)-(8))
to update the posterior distribution of the error terms introduced in Equation
5. The derived error posterior mean and covariance are used to aggregate the
bounding boxes (via Equation 9 and Equation 10). These equations are used in
the training pipeline as described in Lines 7, 8, and 13 of Algorithm 1 in
Appendix 0.C.

##### Problem Setup.

Following the notations in Section 3, for brevity, notations in this section
are simplified to consider a single training sample. Let
$(y_{m},\hat{y}_{m}^{*})_{m=1}^{M}$ be the pairs of annotation-prediction
matches obtained from the Annotations-Predictions Matcher in Section 3.1.2,
where $M$ is the number of annotation-prediction matches. $y_{m}$ contains the
bounding box coordinates $b_{m}\in\mathbb{R}^{4}$ and their annotator ID
$k_{m}\in\\{1,...,K\\}$, while $\hat{y}_{m}^{*}$ contains the matched bounding
box prediction $\hat{b}_{m}^{*}\in\mathbb{R}^{4}$ from the object detector
(OD) module.

Popular object detectors, such as YOLO [40] and DETR [9], do not predict the
bounding box coordinates in the form of probability distributions. Instead,
they predict four values representing the x and y coordinates of the box
centre along with the width and height of the box. Hence, we cannot directly
perform Bayesian inference for the bounding box prediction. To address this,
we model the four error terms, _i.e_. the translation error in the x-axis, the
translation error in the y-axis, the scaling error in the x-axis and the
scaling error in the y-axis, of each annotator by multivariate Gaussian
distributions:

$p(\epsilon_{m}|k_{m}=k,\bm{\mu},\bm{\sigma})=\mathcal{N}({\mu}^{k},{\sigma}^{k})\,\,,$
(13)

where $\epsilon_{m}\in\mathbb{R}^{4}$ is a vector containing the four error
terms while $\bm{\mu}$ and $\bm{\sigma}$ are a collection of Gaussian
parameters with ${\mu}^{k}$ and ${\sigma}^{k}$ representing the mean and
covariance of $\epsilon_{m}$ for the $k$-th annotator, respectively. The four
error terms, which represent the differences between the matched prediction
and annotated bounding box from each annotator, are calculated using Equation
5 in Section 3.1.3.

In this section, the goal is to derive the posterior distribution of these
error terms shown in Equations (6)-(8).

##### Initialisation.

We impose a Gaussian-Gamma [6] prior with hyperparameters, $\mu^{k}_{0}$,
$\upsilon^{k}_{0}$ and $\beta^{k}_{0}$, for the Gaussian distributions:

$\displaystyle\mathcal{NG}({\mu}^{k},{\sigma}^{k}|\mu^{k}_{0},\upsilon^{k}_{0},\beta^{k}_{0})$
$\displaystyle\stackrel{{\scriptstyle\text{def}}}{{=}}\mathcal{N}(\mu^{k}|\mu^{k}_{0},\sigma^{k})\mathcal{G}((\sigma^{k})^{-1}|\upsilon^{k}_{0},\beta^{k}_{0})$
$\displaystyle=\frac{1}{(2\pi)^{2}}\frac{1}{\sqrt{|\sigma^{k}|}}\exp\left(-\frac{1}{2}(\mu^{k}-\mu^{k}_{0})^{\top}(\sigma^{k})^{-1}(\mu^{k}-\mu^{k}_{0})\right)$
$\displaystyle\quad\times\frac{|\beta^{k}_{0}|^{\upsilon^{k}_{0}}}{\Gamma(\upsilon^{k}_{0})}\frac{1}{|\sigma^{k}|^{\upsilon^{k}_{0}-\frac{5}{2}}}\exp\left(-tr(\beta^{k}_{0}(\sigma^{k})^{-1})\right)\,\,.$
(14)

Here, $\mathcal{NG}$ denotes the Gaussian-Gamma distribution, $\mathcal{N}$
represents the Gaussian distribution, $\mathcal{G}$ represents the Gamma
distribution, while $|\cdot|$ denotes the determinant of a matrix, $tr(\cdot)$
is the trace of a matrix, and $\Gamma$ denotes the Gamma function [1].

##### Evidence Lower Bound.

Let $\mathbf{E}$ represent the set of all the bounding box errors
$(\epsilon_{m},k_{m})$, while $\mathbf{M}_{0}$, $\mathbf{U}_{0}$ and
$\mathbf{B}_{0}$ are the Gaussian-Gamma prior hyperparameters $\mu^{k}_{0}$,
$\upsilon^{k}_{0}$ and $\beta^{k}_{0}$, respectively. The log marginal
likelihood of the observed bounding box errors $\mathbf{E}$ can be computed
as:

$\displaystyle\ln\,p(\mathbf{E}|\mathbf{M}_{0},\mathbf{U}_{0},\mathbf{B}_{0})$
$\displaystyle=\underbrace{\int
q(\bm{\mu},\bm{\sigma})\ln\frac{p(\mathbf{E},\bm{\mu},\bm{\sigma}|\mathbf{M}_{0},\mathbf{U}_{0},\mathbf{B}_{0})}{q(\bm{\mu},\bm{\sigma})}\mathop{}\\!\mathrm{d}\bm{\mu}\mathop{}\\!\mathrm{d}\bm{\sigma}\,}_{\mathcal{L}_{b}(q(\bm{\mu},\bm{\sigma}))}$
(15) $\displaystyle\quad+\underbrace{\int
q(\bm{\mu},\bm{\sigma})\ln\frac{q(\bm{\mu},\bm{\sigma})}{p(\bm{\mu},\bm{\sigma}|\mathbf{E},\mathbf{M}_{0},\mathbf{U}_{0},\mathbf{B}_{0})}\mathop{}\\!\mathrm{d}\bm{\mu}\mathop{}\\!\mathrm{d}\bm{\sigma}\,}_{KL(q(\bm{\mu},\bm{\sigma})||p(\bm{\mu},\bm{\sigma}))}\,\,,$
(16)

where $q(\bm{\mu},\bm{\sigma})$ is an arbitrary distribution of $\bm{\mu}$ and
$\bm{\sigma}$ while $KL$ is the Kullback-Leibler (KL) divergence. The joint
likelihood in Equation 15 can be rewritten as:

$p(\mathbf{E},\bm{\mu},\bm{\sigma}|\mathbf{M}_{0},\mathbf{U}_{0},\mathbf{B}_{0})=p(\mathbf{E}|\bm{\mu},\bm{\sigma})p(\bm{\mu},\bm{\sigma}|\mathbf{M}_{0},\mathbf{U}_{0},\mathbf{B}_{0})\,\,.$
(17)

Since the KL divergence is always non-negative,
$\mathcal{L}_{b}(q(\bm{\mu},\bm{\sigma}))$ is the lower bound of the marginal
likelihood, which we can rewrite as:

$\displaystyle\mathcal{L}_{b}(q(\bm{\mu},\bm{\sigma}))$ $\displaystyle=-\int
q(\bm{\mu},\bm{\sigma})\ln\frac{q(\bm{\mu},\bm{\sigma})}{p(\mathbf{E},\bm{\mu},\bm{\sigma}|\mathbf{M}_{0},\mathbf{U}_{0},\mathbf{B}_{0})}\mathop{}\\!\mathrm{d}\bm{\mu}\mathop{}\\!\mathrm{d}\bm{\sigma}$
$\displaystyle=-KL(q(\bm{\mu},\bm{\sigma})||p(\mathbf{E},\bm{\mu},\bm{\sigma}|\mathbf{M}_{0},\mathbf{U}_{0},\mathbf{B}_{0}))\,\,.$
(18)

##### Optimising $\mathcal{L}_{b}$.

We seek to maximise the lower bound (Equation 18), which is equivalent to
minimising the KL divergence between $q(\bm{\mu},\bm{\sigma})$ and
$p(\mathbf{E},\bm{\mu},\bm{\sigma}|\mathbf{M}_{0},\allowbreak\mathbf{U}_{0},\mathbf{B}_{0})$.
The KL divergence is minimised when the two distributions are equivalent:

$\displaystyle\ln\ $ $\displaystyle q(\bm{\mu},\bm{\sigma})=\ln
p(\mathbf{E},\bm{\mu},\bm{\sigma}|\mathbf{M}_{0},\mathbf{U}_{0},\mathbf{B}_{0})$
$\displaystyle\;\;\,\mathrel{\stackunder{\stackon{$=$}{$\scriptstyle
C$}}{$\scriptstyle\lx@cref{creftype~refnum}{eq:box_joint},\lx@cref{creftype~refnum}{eq:gaussgamma}$}}\sum_{k=1}^{K}\ln\frac{1}{\sqrt{|\sigma^{k}|}}\frac{1}{|\sigma^{k}|^{\upsilon^{k}_{0}+\frac{M^{k}}{2}-\frac{5}{2}}}\exp\left(-tr(\beta^{k}_{0}(\sigma^{k})^{-1})$
$\displaystyle\qquad\qquad\quad\exp\Bigg{(}-\frac{1}{2}\Big{[}(\mu^{k}-\mu^{k}_{0})^{\top}(\sigma^{k})^{-1}(\mu^{k}-\mu^{k}_{0})$
$\displaystyle\qquad\qquad\qquad+\sum_{m=1}^{M}\text{1}[k_{m}=k](\epsilon_{m}-\mu^{k})^{\top}(\sigma^{k})^{-1}(\epsilon_{m}-\mu^{k})\Big{]}\Bigg{)}\,\,,$
(19)

where $\text{1}[\cdot]$ is the indicator function,
$M^{k}:=\sum_{m=1}^{M}\text{1}[k_{m}=k]$ is the number of annotation-
prediction matches associated with the $k$-th annotator, and
$\stackrel{{\scriptstyle C}}{{=}}$ denotes equality up to an additive constant
that is independent of the function parameters.

Let us define the mean of all $\epsilon_{m}$ computed using Equation 5 for the
$k$-th annotator as follow:

$\bar{\epsilon}^{k}=\frac{1}{M^{k}}\sum_{m=1}^{M}\text{1}[k_{m}=k]\epsilon_{m}\,\,.$
(20)

We can then rewrite one term in the exponent of Equation 19 as follows:

$\displaystyle\sum_{m=1}^{M}$
$\displaystyle\text{1}[k_{m}=k](\epsilon_{m}-\mu^{k})^{\top}(\sigma^{k})^{-1}(\epsilon_{m}-\mu^{k})$
$\displaystyle=\sum_{m=1}^{M}\text{1}[k_{m}=k][(\epsilon_{m}-\bar{\epsilon}^{k})-(\mu^{k}-\bar{\epsilon}^{k})]^{\top}(\sigma^{k})^{-1}[(\epsilon_{m}-\bar{\epsilon}^{k})-(\mu^{k}-\bar{\epsilon}^{k})]$
(21)
$\displaystyle=\sum_{m=1}^{M}\text{1}[k_{m}=k](\epsilon_{m}-\bar{\epsilon}^{k})^{\top}(\sigma^{k})^{-1}(\epsilon_{m}-\bar{\epsilon}^{k})+M^{k}(\mu^{k}-\bar{\epsilon}^{k})^{\top}(\sigma^{k})^{-1}(\mu^{k}-\bar{\epsilon}^{k})\,\,.$
(22)

It can also be shown that:

$\displaystyle(\mu^{k}-\mu^{k}_{0})$
${}^{\top}(\sigma^{k})^{-1}(\mu^{k}-\mu^{k}_{0})+M^{k}(\mu^{k}-\bar{\epsilon}^{k})^{\top}(\sigma^{k})^{-1}(\mu^{k}-\bar{\epsilon}^{k})$
$\displaystyle=(M^{k}+1)\left(\mu^{k}-\frac{\mu^{k}_{0}+M^{k}\bar{\epsilon}^{k}}{M^{k}+1}\right)^{\top}(\sigma^{k})^{-1}\left(\mu^{k}-\frac{\mu^{k}_{0}+M^{k}\bar{\epsilon}^{k}}{M^{k}+1}\right)$
$\displaystyle\quad+\frac{M^{k}(\bar{\epsilon}^{k}-\mu^{k}_{0})^{\top}(\sigma^{k})^{-1}(\bar{\epsilon}^{k}-\mu^{k}_{0})}{M^{k}+1}\,\,.$
(23)

By plugging Equation 22 and Equation 23 into Equation 19, we obtain

$\displaystyle\ln\ q(\bm{\mu},\bm{\sigma})$
$\displaystyle\stackrel{{\scriptstyle
C}}{{=}}\sum_{k=1}^{K}\ln\frac{1}{\sqrt{|\sigma^{k}}|}\frac{1}{|\sigma^{k}|^{\upsilon^{k}_{0}+\frac{M^{k}}{2}-\frac{5}{2}}}\exp\left(-tr(\beta^{k}_{0}(\sigma^{k})^{-1})\right)$
$\displaystyle\qquad\
\exp\Bigg{(}-\frac{1}{2}\Bigg{[}(M^{k}+1)\left(\mu^{k}-\frac{\mu^{k}_{0}+M^{k}\bar{\epsilon}^{k}}{M^{k}+1}\right)^{\top}(\sigma^{k})^{-1}\left(\mu^{k}-\frac{\mu^{k}_{0}+M^{k}\bar{\epsilon}^{k}}{M^{k}+1}\right)$
$\displaystyle\qquad\quad+\frac{M^{k}(\bar{\epsilon}^{k}-\mu^{k}_{0})^{\top}(\sigma^{k})^{-1}(\bar{\epsilon}^{k}-\mu^{k}_{0})}{M^{k}+1}$
$\displaystyle\qquad\quad+\sum_{m=1}^{M}\text{1}[k_{m}=k](\epsilon_{m}-\bar{\epsilon}^{k})^{\top}(\sigma^{k})^{-1}(\epsilon_{m}-\bar{\epsilon}^{k})\Bigg{]}\Bigg{)}$
$\displaystyle\stackrel{{\scriptstyle
C}}{{=}}\sum_{k=1}^{K}\ln\frac{1}{\sqrt{|\sigma^{k}}|}\exp\Bigg{(}-\frac{1}{2}(M^{k}+1)\left(\mu^{k}-\frac{\mu^{k}_{0}+M^{k}\bar{\epsilon}^{k}}{M^{k}+1}\right)^{\top}(\sigma^{k})^{-1}$
$\displaystyle\qquad\qquad\qquad\qquad\left(\mu^{k}-\frac{\mu^{k}_{0}+M^{k}\bar{\epsilon}^{k}}{M^{k}+1}\right)\Bigg{)}$
$\displaystyle\qquad\times\frac{1}{|\sigma^{k}|^{\upsilon^{k}_{0}+\frac{M^{k}}{2}-\frac{5}{2}}}\exp\Bigg{(}-tr\Bigg{(}\bigg{[}\beta^{k}_{0}+\frac{M^{k}(\bar{\epsilon}^{k}-\mu^{k}_{0})(\bar{\epsilon}^{k}-\mu^{k}_{0})^{\top}}{2(M^{k}+1)}$
$\displaystyle\qquad\quad+\frac{1}{2}\sum_{m=1}^{M}\text{1}[k_{m}=k](\epsilon_{m}-\bar{\epsilon}^{k})(\epsilon_{m}-\bar{\epsilon}^{k})^{\top}\bigg{]}(\sigma^{k})^{-1}\Bigg{)}\Bigg{)}\,\,.$
(24)

We can therefore conclude that $q(\bm{\mu},\bm{\sigma})$ is the product of the
Gaussian-Gamma distributions (Equation 14) [35]:

$q(\bm{\mu},\bm{\sigma})=\prod_{k=1}^{K}\mathcal{NG}({\mu}^{k},{\sigma}^{k}|\tilde{\mu}^{k},\tilde{\upsilon}^{k},\tilde{\beta}^{k})\,\,,$
(25)

where

$\tilde{\mu}^{k}=\frac{\mu^{k}_{0}+M^{k}\bar{\epsilon}^{k}}{M^{k}+1}\,\,,$
(26) $\tilde{\upsilon}^{k}=\upsilon^{k}_{0}+\frac{M^{k}}{2}\,\,,$ (27)
$\tilde{\beta}^{k}=\beta^{k}_{0}+\frac{M^{k}(\bar{\epsilon}^{k}-\mu^{k}_{0})(\bar{\epsilon}^{k}-\mu^{k}_{0})^{\top}}{2(M^{k}+1)}+\frac{1}{2}\sum_{m=1}^{M}\text{1}[k_{m}=k](\epsilon_{m}-\bar{\epsilon}^{k})(\epsilon_{m}-\bar{\epsilon}^{k})^{\top}\,\,.$
(28)

Finally, we obtain the expected mean and covariance matrix of the updated
Gaussian-Gamma posterior as $\mu^{k}=\tilde{\mu}^{k}$ and
$\sigma^{k}=\tilde{\beta}^{k}/\tilde{\upsilon}^{k}$, respectively, for the
aggregation of the bounding boxes (via Equation 9 and Equation 10).

##  Appendix 0.B Class Label Aggregator

We present in this section the derivation of Equation (12) to update the
aggregated class labels to maximise the evidence lower bound (ELBO) of the
crowdsourced labels. We also provide the derivation to update each annotator’s
confusion matrix to maximise the ELBO via Equation 41. These updates
correspond to Lines 9 and 14, respectively, in Algorithm 1 of the training
pipeline described in Appendix 0.C and supplement the material introduced in
Section 3.1.4.

##### Problem Setup.

As in Section 3, the notations in this section are simplified to consider a
single training sample for brevity. The derivation steps follow those of the
Bayesian classifier combination in Kim and Ghahramani [23] and Isupova _et
al_. [19].

Let $\\{(c_{m},k_{m})\\}_{m=1}^{M_{n}}$ be the set of tuples consisting of the
annotated class labels $c_{m}\in\\{1,...,J\\}$ and their annotator ID
$k_{m}\in\\{1,...,K\\}$ that are matched to the $n$-th prediction
$\hat{y}_{n}$ obtained from the Annotations-Predictions Matcher in Section
3.1.2. $\hat{y}_{n}$ contains the predicted class probabilities
$\hat{p}_{n}\in[0,1]^{J}$, and $M_{n}$ denotes the number of annotations
matched to the $n$-th prediction.

In the CLA, we model the annotated class label from each annotator as a
multinomial distribution conditioning on the true label of the object:

$p(c_{m}|k_{m}=k,t_{m}=j,\bm{\pi})={\pi}^{k}_{j,c_{m}}\,\,,$ (29)

where $\bm{\pi}$ is a collection of confusion matrices,
${\pi}^{k}\in\mathbb{R}^{J\times J}$ is the confusion matrix of the $k$-th
annotator and $t_{m}$ is the ground truth class label. Each row of
$\pi_{j}^{k}$ represents the event probabilities of a multinomial distribution
for the annotated class label from the $k$-th annotator when $t_{m}=j$. The
probability of the true class label of the $n$-th object,
$p(t_{n}|x,\theta_{O})$ is modelled by the OD module with a softmax output
layer.

In this section, the goal is to learn the set of distributions for each
confusion matrix in $\bm{\pi}$ (Equations (40) and (41)) and the distribution
of the aggregated (soft) ground truth label for the $n$-th prediction
$\rho_{n,j}:=q(t_{n}=j)$ (Equation 12).

##### Initialisation.

We impose a Dirichlet prior with hyperparameters
${\alpha}^{k}_{0j}=\\{\alpha^{k}_{0j,1},\allowbreak...,\alpha^{k}_{0j,J}\\}$
such that $\pi^{k}_{j}|\alpha^{k}_{0j}\sim Dir(\alpha^{k}_{0j})$.

##### Evidence Lower Bound.

Let $\mathbf{c}$ capture all annotated class labels $(c_{m},k_{m})$,
$\mathbf{t}$ are all the true labels $t_{n}$, $\mathbf{A}_{0}$ are all the
prior hyperparameters $\alpha_{0j}^{k}$ for the Dirichlet priors. The log
marginal likelihood of the observed class labels $\mathbf{c}$ can be described
as:

$\displaystyle\ln\,p(\mathbf{c}|x,\mathbf{A}_{0},\theta_{O})$
$\displaystyle=\underbrace{\int
q(\mathbf{t},\bm{\pi})\ln\frac{p(\mathbf{c},\mathbf{t},\bm{\pi}|x,\mathbf{A}_{0},\theta_{O})}{q(\mathbf{t},\bm{\pi})}\mathop{}\\!\mathrm{d}\mathbf{t}\mathop{}\\!\mathrm{d}\bm{\pi}}_{\mathcal{L}_{c}(q,\theta_{O})}$
(30) $\displaystyle\quad+\underbrace{\int
q(\mathbf{t},\bm{\pi})\ln\frac{q(\mathbf{t},\bm{\pi})}{p(\mathbf{t},\bm{\pi}|\mathbf{c},x,\mathbf{A}_{0},\theta_{O})}\mathop{}\\!\mathrm{d}\mathbf{t}\mathop{}\\!\mathrm{d}\bm{\pi}}_{KL(q(\mathbf{t},\bm{\pi})||p(\mathbf{t},\bm{\pi}))}\,\,,$
(31)

where $q(\mathbf{t},\bm{\pi})$ is an arbitrary distribution of $\mathbf{t}$
and $\bm{\pi}$ and $\mathcal{L}_{c}(q,\theta_{O})$ is the lower bound of the
marginal likelihood.

We employ mean-field variational inference [52] and restrict our search within
factorised distributions $q(\mathbf{t},\bm{\pi})=q(\mathbf{t})q(\bm{\pi})$:

$\mathcal{L}_{c}(q,\theta_{O})=\int q(\mathbf{t})q(\bm{\pi})\ln
p(\mathbf{c},\mathbf{t},\bm{\pi}|x,\mathbf{A}_{0},\theta_{O})\mathop{}\\!\mathrm{d}\mathbf{t}\mathop{}\\!\mathrm{d}\bm{\pi}\\\
-\int q(\mathbf{t})\ln q(\mathbf{t})\mathop{}\\!\mathrm{d}\mathbf{t}-\int
q(\bm{\pi})\ln q(\bm{\pi})\mathop{}\\!\mathrm{d}\bm{\pi}\,.$ (34)

The lower bound, $\mathcal{L}_{c}(q,\theta_{O})$, depends on the joint
likelihood which can be factorised as:

$p(\mathbf{c},\mathbf{t},\bm{\pi}|x,\mathbf{A}_{0},\theta_{O})=p(\mathbf{c}|\mathbf{t},\bm{\pi})p(\mathbf{t}|x,\theta_{O})p(\bm{\pi}|\mathbf{A}_{0})\,\,.$
(35)

##### Optimising $\mathcal{L}_{c}$ with respect to $q(\bm{\pi})$.

First, re-writing the lower bound as a function of $q(\bm{\pi})$ with:

$\displaystyle\ln\tilde{p}_{\bm{\pi}}(\mathbf{c},\mathbf{t},\bm{\pi}):$
$\displaystyle=\int q(\mathbf{t})\ln
p(\mathbf{c},\mathbf{t},\bm{\pi}|x,\mathbf{A}_{0},\theta_{O})\mathop{}\\!\mathrm{d}\mathbf{t}$
$\displaystyle\stackrel{{\scriptstyle C}}{{=}}\mathbb{E}_{q(\mathbf{t})}\ln
p(\mathbf{c},\mathbf{t},\bm{\pi}|x,\mathbf{A}_{0},\theta_{O})\,\,,$ (36)

gives:

$\displaystyle\mathcal{L}_{c}(q(\bm{\pi}))$
$\displaystyle\stackrel{{\scriptstyle C}}{{=}}-\int
q(\bm{\pi})\ln\frac{q(\bm{\pi})}{\tilde{p}_{\bm{\pi}}(\mathbf{c},\mathbf{t},\bm{\pi})}\mathop{}\\!\mathrm{d}\bm{\pi}$
$\displaystyle\stackrel{{\scriptstyle
C}}{{=}}-KL(q(\bm{\pi})||\tilde{p}_{\bm{\pi}}(\mathbf{c},\mathbf{t},\bm{\pi}))\,\,.$
(37)

In other words, to maximise $\mathcal{L}_{c}(q(\bm{\pi}))$, we need to
minimise the KL divergence between $q(\bm{\pi})$ and
$\tilde{p}_{\bm{\pi}}(\mathbf{c},\mathbf{t},\bm{\pi})$ which is minimised when
the two distributions are equal:

$\displaystyle\ln\ $ $\displaystyle q(\bm{\pi})\stackrel{{\scriptstyle
C}}{{=}}\ \mathbb{E}_{q(\mathbf{t})}\ln
p(\mathbf{c},\mathbf{t},\bm{\pi}|x,\mathbf{A}_{0},\theta_{O})$
$\displaystyle\quad\mathrel{\stackunder{\stackon{$=$}{$\scriptstyle
C$}}{$\scriptstyle\lx@cref{creftype~refnum}{eq:joint}$}}\mathbb{E}_{q(\mathbf{t})}\ln
p(\mathbf{c}|\mathbf{t},\bm{\pi})p(\bm{\pi}|\mathbf{A}_{0})$ (38)
$\displaystyle\qquad\stackrel{{\scriptstyle
C}}{{=}}\sum_{k=1}^{K}\sum_{j=1}^{J}\sum_{l=1}^{J}\ln\pi^{k}_{j,l}\Bigg{(}\alpha^{k}_{0j,l}+\sum_{n=1}^{H}\sum_{(c,k^{\prime})\in\tilde{\kappa}_{n}}\text{1}[c=l]\text{1}[k^{\prime}=k]q(t_{n}=j)-1\Bigg{)}\,.$
(39)

$H$ indicates the number of predictions matched to at least one annotation,
$\tilde{\kappa}_{n}$ is the set of tuples consisting of the annotated class
labels and their annotator ID that are matched to the $n$-th prediction while
$\text{1}[\cdot]$ is the indicator function. We observe that $q(\bm{\pi})$ is
the product of the Dirichlet distributions:

$q(\bm{\pi})=\prod_{k=1}^{K}\prod_{j=1}^{J}Dir(\pi_{j}^{k}|\tilde{\alpha}_{j}^{k})\,\,,$
(40)

where the Dirichlet parameters are updated with:

$\tilde{\alpha}^{k}_{j,l}=\alpha^{k}_{0j,l}+\sum_{n=1}^{H}\sum_{(c,k^{\prime})\in\tilde{\kappa}_{n}}\text{1}[c=l]\text{1}[k^{\prime}=k]\,q(t_{n}=j)\,\,.$
(41)

##### Optimising $\mathcal{L}_{c}$ with respect to $q(\mathbf{t})$.

Next, we rewrite the lower bound as a function of $q(\mathbf{t})$ and follow
similar steps of Equations 37 and 38 to obtain the updating formula for the
distribution $q(\mathbf{t})$:

$\displaystyle\ln$ $\displaystyle\ q(\mathbf{t})\stackrel{{\scriptstyle
C}}{{=}}\ \mathbb{E}_{q(\bm{\pi})}\ln
p(\mathbf{c},\mathbf{t},\bm{\pi}|x,\mathbf{A}_{0},\theta_{O})$
$\displaystyle\quad\mathrel{\stackunder{\stackon{$=$}{$\scriptstyle
C$}}{$\scriptstyle\lx@cref{creftype~refnum}{eq:joint}$}}\mathbb{E}_{q(\bm{\pi})}\ln
p(\mathbf{c}|\mathbf{t},\bm{\pi})p(\mathbf{t}|x,\theta_{O})$
$\displaystyle\qquad\stackrel{{\scriptstyle
C}}{{=}}\sum_{n=1}^{H}\sum_{j=1}^{J}\text{1}[t_{n}=j]\left(\sum_{(c,k)\in\tilde{\kappa}_{n}}\mathbb{E}_{\pi^{k}_{j}}[\ln\pi^{k}_{j,c}]+\ln
p(t_{n}=j|x,\theta_{O})\right)\,\,.$ (42)

We derived that $q(\pi^{k}_{j})$ are the Dirichlet distributions (Equation
40). Therefore, the expected value for each annotator is given by:

$\mathbb{E}_{\pi^{k}_{j}}\ln\pi^{k}_{j,c}=\Psi(\tilde{\alpha}^{k}_{j,c})-\Psi\left(\sum^{J}_{l=1}\tilde{\alpha}^{k}_{j,l}\right)\,\,,$
(43)

where $\Psi$ denotes the digamma function [2].

Using the definition of categorical distribution (_i.e_. if $p(a)$ is a
categorical distribution, then its logarithm is given as $\ln
p(a)=\sum_{i}\text{1}[a=i]\ln p(a=i)$), we conclude that $q(t)$ is the product
of categorical distributions from Equation 42:

$q(t)=\prod_{n=1}^{H}q(t_{n})\,\,,$

such that the probability of the true label for the $n$-th object,
$\rho_{n,j}:=q(t_{n}=j)$, is:

$\rho_{n,j}=\exp\left(\ln\hat{p}_{n,j}+\sum_{(c,k)\in\tilde{\kappa}_{n}}\mathbb{E}_{\pi^{k}_{j}}\ln\pi^{k}_{j,c}\right)\,\,,$
(44)

where $\hat{p}_{n,j}:=p(t_{n}=j|x,\theta_{O})$ is the OD’s class-$j$
probability of the $n$-th prediction.

##  Appendix 0.C Training Pipeline and Pseudocode

Algorithm 1 summarises the overall training steps of the BDC framework.

Algorithm 1 Pseudocode for the Training Steps of BDC.

1:Set of input images, $x=\left\\{x^{i}\in\mathbb{R}^{I_{w}\times
I_{h}}\right\\}_{i=1}^{N}$, where $I_{w}$ and $I_{h}$ are the width and height
of the image, respectively, while $N$ is the number of training images; Set of
crowdsourced annotations,
$y=\left\\{y^{i}=\big{\\{}y^{i}_{m}=(b^{i}_{m},c^{i}_{m},k^{i}_{m})\right\\}_{m=1}^{M^{i}}\big{\\}}_{i=1}^{N}$,
where $y^{i}_{m}$ is a tuple of the annotated bounding box
$b^{i}_{m}\in\mathbb{R}^{4}$, the annotated class label
$c^{i}_{m}\in\\{1,...,J\\}$, and the annotator ID $k^{i}_{m}\in\\{1,...,K\\}$
of the $m$-th annotation in the $i$-th image, while $M^{i}$ is the number of
crowdsourced annotations in the $i$-th image; The number of training epochs,
$T\in\mathbb{Z}^{+}$; The object detector (OD) module,
$\textit{OD}_{\theta_{O}}(\cdot):\mathbb{R}^{I_{w}\times I_{h}}\to O^{P}$,
where $\theta_{O}$ is the initialised OD’s parameters,
$O=\mathbb{R}^{4}\times[0,1]^{J}$ is the predicted box coordinates and class
probabilities while $P$ is the number of predictions; The bounding box
aggregator (BBA) with the initialised parameters, $\bm{\mu}_{0}$,
$\bm{\upsilon}_{0}$ and $\bm{\beta}_{0}$; The class label aggregator (CLA)
with the initialised parameters, $\bm{\alpha}_{0}$.

2:$t\leftarrow 1$;

3:while all parameters are not converged and $t\leq T$ do

4: Initialise $\hat{y}\leftarrow\emptyset$, $\mathfrak{b}\leftarrow\emptyset$,
$\rho\leftarrow\emptyset$;

5: for $i=1$ to $N$ do

6: $\hat{y}^{i}\leftarrow$ $\textit{OD}_{\theta_{O}}(x^{i})$;$\triangleright$
Forward pass of the OD to generate predictions.

7: Match each crowdsourced annotation to one of the predictions by minimising
the matching cost (Equation 3);

8: Adjust the annotated bounding boxes (Equation 9) using the posterior mean
of the error terms in BBA;

9: $\mathfrak{b}^{i}\leftarrow$ Aggregate the bounding boxes (Equation 10)
using the posterior precision of the error terms in BBA;

10: $\rho^{i}\leftarrow$ Obtain the aggregated class probabilities (Equation
12) using the current value of the CLA’s parameters;

11: $\hat{y}:=\hat{y}\cup\hat{y}^{i}$;
$\mathfrak{b}:=\mathfrak{b}\cup\mathfrak{b}^{i}$; $\rho:=\rho\cup\rho^{i}$;

12: end for

13: Update $\theta_{O}$ by minimising the OD loss function (Equation 1) via
the gradient descent method;

14: Update $q(\bm{\mu},\bm{\sigma})$ according to Equations (26)-(28) to
maximise $\mathcal{L}_{b}$;

15: Update $q(\bm{\pi})$ according to Equation 41 to maximise
$\mathcal{L}_{c}$;

16:end while

##  Appendix 0.D Training Configuration

Table 6 shows the training settings for each object detection algorithm. Code
is implemented with PyTorch 1.13 [38] and runs on RTX Titan and 3090 GPUs.

Table 6: Training settings and configuration for YOLOv7 [53], Faster R-CNN (FRCNN) [41] and EVA [14] object detectors. Setting | YOLOv7 [53] | FRCNN [41] | EVA [14]
---|---|---|---
Pre-training data | ImageNet [44] | Merged-38M [14]
Image size | 640 | 800 | 1,024
Optimiser | SGD w/ momentum | AdamW [32]
LR | 0.001 | $5e^{-5}$
LR scheduler | Cosine decay [31] | -
Weight decay | 0.0005 | 0.1
Batch size | 32 | 8 | 8
Epochs (VOC/COCO) | 100/20 | 50/15 | 25/10

##  Appendix 0.E Results on the Disaster Response Dataset

This section presents results on the disaster response dataset [19]. As
discussed in the main paper, there is no ground truth for this dataset.
Therefore, we present only the qualitative performance of algorithms. It is
worth noting that in [19] the dataset was adjusted to be used for image
classification. However, crowd members provided bounding boxes for buildings
and labelled damages on them. Hence, these labels can be used for object
detection.

Figure 6 visualises the FRCNN object detectors’ predictions trained by
different aggregation methods on the test dataset where the original images
before and after the disaster are also presented. The images before the
disaster are not used to train the object detectors but are used to ascertain
whether an object is an undamaged building, a damaged building or not a
building. We randomly select three test images representing different degrees
of building density for a comprehensive evaluation as shown in Figure 6.
Additionally, we numbered each building of interest and provide brief
explanations for each building below.

In the first test example (left column), we observe that Building $1$ is
missing a roof (_i.e_. it is damaged) and is only correctly predicted as
damaged by FRCNNs trained with NA, MV and BDC. Furthermore, the FRCNN model
trained with MV falsely predicts a non-building object (number $2$) as an
undamaged building.

For the second test example in the central column, by comparing the images
before and after the disaster, we can see that Building $4$ and $6$ are
clearly damaged, Building $5$ is undamaged (no significant visual difference
between images before and after the disaster), and object number $3$ is not a
building. Using this observation as ground truth, the FRCNNs trained via MV
and BDC correctly predict all 4 cases while the FRCNNs learnt with other
methods did not successfully identify them.

Finally, in the third test case in the right column, it can be observed that
Buildings $7$, $9$ and $10$ are likely undamaged, as there are no notable
visual differences in the before and after images while Building $8$ is a
damaged building. We see that FRCNNs trained with NA, MV, WBF-EARL and Crowd
R-CNN correctly predict 2, 3, 1 and 0 cases, respectively. Moreover, the
FRCNNs learnt using WBF-EARL and Crowd R-CNN fail to separate Building $8$
from the building to the right. On the other hand, the FRCNN trained via BDC
correctly predict all 4 buildings. They represent some of the empirical
evidence which we observe that the FRCNN trained using BDC provides the most
accurate predictions.

(a) Before Disaster

(b) After Disaster

(c) NA

(d) MV

(e) WBF-EARL

(f) Crowd R-CNN

(g) BDC (ours)

Figure 6: (a)(b) The satellite images before and after the disaster of the
same location in the test dataset. (c - g) Predictions of FRCNN object
detectors trained via different aggregation methods. The green and red
bounding boxes represent the class ‘undamaged’ and ‘damaged’ buildings,
respectively.

##  Appendix 0.F Synthesisation of Crowdsourced Object Annotations

Denote the ground truth annotations for each training image by
$\\{\tilde{y}_{n}\\}_{n=1}^{G}$, where $\tilde{y}_{n}=\\{(s_{n},t_{n})\\}$,
$G$ is the number of ground truth instances in the image, $s_{n}$ and $t_{n}$
are the $n$-th ground truth bounding box coordinates and class label
respectively. Ground truth annotations are only used to synthesise annotations
and are not made available to the proposed aggregation method. We use deep
learning models to synthesise more realistic crowdsourced annotations with
instance-dependent noise. We present the key steps and the pseudocode for the
annotation synthesisation procedure as follows.

##### Class label synthesisation.

Firstly, we use classification networks to obtain a confusion matrix of the
ground truth and predicted classes. Images cropped using the ground truth
bounding box coordinates and their respective class labels are used to train
the classification network. Additionally, we randomly crop regions not covered
by ground truth bounding boxes in the images and treat them as the
‘background’ class to simulate false positives and false negatives. The
confusion matrix, $\mathcal{C}\in\mathbb{R}^{(J+1)\times(J+1)}$ (the $J+1$
confusion matrix size is due to the added ‘background’ class), obtained by
evaluating the classification network on the testing dataset is used as the
parameters of Dirichlet distributions so that the $k$-th annotator’s confusion
matrix, $\mathbf{C}_{k}$, can be drawn from the same $\mathcal{C}$, _i.e_.
$\mathbf{C}_{k}\sim Dir(\mathcal{C})$. Then, we use $\mathbf{C}_{k}$ to
synthesise class labels from the $k$-th annotator. We do not generate a
synthesised annotation if its synthesised class label is ‘background’ to
simulate a false negative annotation.

##### Bounding box synthesisation.

Next, to synthesise crowdsourced bounding box, $b_{m}$, we utilise a Region
Proposal Network (RPN) [41] trained on the ground truth bounding boxes to
generate proposals, $\hat{r}$, along with their objectness scores, $\hat{o}$,
for a given input image. For each ground truth box, $s_{n}$, we calculate
$\textit{IoU}(s_{n},\hat{r})\odot\hat{o}$, where $\odot$ denotes the element-
wise multiplication, and normalise the vector so that it sums to one. Using
the normalised vector as the probability vector of a categorical distribution,
a proposal is randomly selected as the annotator’s bounding box annotation to
simulate instance-dependent noise [49].

Algorithm 2 Synthesising Crowdsourced Object Annotations for the $k$-th
Annotator.

1:Confusion matrix obtained from the trained classification network,
$\mathcal{C}\in\mathbb{R}^{(J+1)\times(J+1)}$, where $J$ is the number of
classes; Set of input images, $x=\left\\{x^{i}\in\mathbb{R}^{I_{w}\times
I_{h}}\right\\}_{i=1}^{N}$, where $I_{w}$ and $I_{h}$ are the width and height
of the image, respectively, while $N$ is the number of images; Set of ground
truth annotations,
$\tilde{y}=\big{\\{}\tilde{y}^{i}=\left\\{\tilde{y}^{i}_{n}=(s^{i}_{n},t^{i}_{n})\right\\}_{n=1}^{G^{i}}\big{\\}}_{i=1}^{N}$,
where $\tilde{y}^{i}_{n}$ contains the ground truth bounding box coordinates
$s^{i}_{n}\in\mathbb{R}^{4}$ and class label $t^{i}_{n}\in\\{1,...,J\\}$ of
the $n$-th object in the $i$-th image, while $G^{i}$ is the number of ground
truth objects in the $i$-th image; Trained region proposal networks (RPN),
$\textit{RPN}(\cdot):\mathbb{R}^{I_{w}\times I_{h}}\to R^{L}$, where
$R=\mathbb{R}^{4}\times[0,1]$ is the proposal box coordinates and objectness
score while $L$ is the number of proposals; Coverage percentage of the $k$-th
annotator, $\mathfrak{c}_{k}\in[0,1]$.

2:$\mathbf{C}_{k}\sim Dir(\mathcal{C})$; $\triangleright$ $\mathcal{C}$ is
used as the parameters of Dirichlet distributions to generate
$\mathbf{C}_{k}$.

3:$I\leftarrow$ Sample($N,\mathfrak{c}_{k}$); $\triangleright$ Sample
$\mathfrak{c}_{k}N$ indices without replacement.

4:$y\leftarrow\emptyset$;

5:for $i$ in $I$ do

6: $y^{i}\leftarrow\emptyset$;

7: $\hat{r},\hat{o}\leftarrow$ RPN($x^{i}$); $\triangleright$ Forward pass of
RPN to generate proposals.

8: for $n=1$ to $G^{i}$ do

9: $c^{i}_{n}\sim Categorical(\mathbf{C}_{k(t^{i}_{n})})$;

10: if $c^{i}_{n}=J+1$ then $\triangleright$ $J+1$ refers to the augmented
‘background’ class.

11: continue $\triangleright$ Simulate a false negative annotation.

12: end if

13: $p^{i}_{n}\leftarrow$ IoU($s^{i}_{n},\hat{r}$) $\odot\ \hat{o}$;
$\triangleright$ Compute intersection over union (IoU).

14: $p^{i}_{n}:=\frac{p^{i}_{n}}{\lVert p^{i}_{n}\rVert_{1}}$;
$\triangleright$ Normalising step so that $p^{i}_{n}$ sums to one.

15: $l\sim Categorical(p^{i}_{n})$;

16: $b^{i}_{n}\leftarrow\hat{r}_{l}$; $\triangleright$ Sample a proposal.

17: $y^{i}:=y^{i}\cup\\{(b^{i}_{n},c^{i}_{n},k)\\}$;

18: end for

19: $c^{i}_{n}\sim Categorical(\mathbf{C}_{k(J+1)})$;

20: if $c^{i}_{n}\neq J+1$ then $\triangleright$ Simulate a false positive
annotation.

21: $l\sim Categorical(\frac{\hat{o}}{\lVert\hat{o}\rVert_{1}})$;

22: $b^{i}_{n}\leftarrow\hat{r}_{l}$;

23: $y^{i}:=y^{i}\cup\\{(b^{i}_{n},c^{i}_{n},k)\\}$;

24: end if

25: $y:=y\cup y^{i}$;

26:end for

27:Synthetic crowdsourced object annotations, $y$.

##### Realistic crowdsourcing simulation.

See _et al_. [45] and Bechtel _et al_. [5] demonstrated that in real-world
scenarios where data annotations are crowdsourced, the annotators often
exhibit varying levels of accuracy and consistency. We simulate this by using
different deep learning network architectures with distinct levels of
performance. Three classification network architectures are used to synthesise
crowdsourced class labels, namely AlexNet [25], VGG16 [47] and ResNet50 [17]
with an overall test accuracy of 34.65, 55.15, and 73.64 on MS COCO,
respectively. On the other hand, the region proposal networks (RPN) with
ResNet18, ResNet50 and ResNet101 backbones with respective overall average
recall of 45.04, 52.45 and 58.14 on MS COCO, are used to synthesise
crowdsourced bounding boxes. The deep learning models are trained on MS COCO
where the classification networks are trained for 5 epochs with a batch size
of 256 and a learning rate of 0.005 while the RPN are trained for 5 epochs
with a batch size of 16 and a learning rate of 0.001. The three pairs of
classification networks and RPN backbones simulate annotators of poor, average
and expert skill levels described in Section 4.2, respectively.

Furthermore, we simulate false positive annotations for each annotator by
randomly synthesising a foreground class label based on $\mathbf{C}_{k}$ and
randomly selecting a bounding box from the proposals. Lastly, to simulate the
annotation practice with a large-scale dataset, the synthesisation method has
an additional parameter: coverage, which indicates the percentage of images
seen by each annotator.

##### Pseudocode.

Algorithm 2 provides the pseudocode to synthesise crowdsourced object
annotations for a single annotator, _i.e_., the $k$-th annotator. In practice,
Algorithm 2 will be called for all $K$ annotators and their returned results
are combined to form the synthetic crowdsourced annotations.

##  Appendix 0.G Visualisation of Classification Networks and RPN Networks

To inspect the classification networks and RPN networks discussed in Appendix
0.F, we visualise in Figures 7 and 8 the confusion matrices $\mathcal{C}$ of
the different classification networks and RPN proposals from the different
backbones that are used to synthesise the COCO-MIX dataset. Small regions of
bright spots can be seen around the diagonal of the confusion matrices of the
classification networks in Figure 7. This is because the class indexes of MS
COCO are ordered by the class’s super-category and objects from classes of the
same super-category are usually visually closer to each other, leading to
higher correlations among them. For example, class ‘fork’, ‘knife’ and ‘spoon’
with class indices 43, 44 and 45 have the same super-category ‘kitchen’ and
are harder to distinguish among them compared to other classes such as ‘car’
or ‘chair’. This shows that a classification network can generate a more
realistic confusion matrix $\mathbf{C}$ compared to prior methods which do not
take into account the instance-dependent noise, as they set the diagonal of
the confusion matrices to an annotator’s proficiency level and set the off-
diagonal entries either uniformly or randomly [57, 39, 43]. Moreover, we
observe a clear distinction between the confusion matrices in Figure 7,
simulating annotators of different skills and accuracies.

Similar observations are seen for the RPN proposals. As shown in Figure 8, we
observe that easily identifiable objects, _e.g_. large television on the
right, have more proposals with larger objectness scores compared to the
harder, smaller objects. Furthermore, ResNet101 generates more accurate
proposals with overall larger objectness scores compared to the ResNet50 and
ResNet18 backbones, simulating annotators of different bounding box
accuracies.

(a) ResNet50

(b) VGG16

(c) AlexNet

(d) Uniform

Figure 7: (a) (b) (c) Confusion matrix $\mathcal{C}$ of different network
architectures evaluated on the MS COCO test set represented with logarithm-
scale heatmaps. (d) Visualisation of the confusion matrix generated by
sampling from a uniform distribution (representing the random annotators of
COCO-MIX). X-axis is the predicted class while y-axis is the ground truth
class. Each row is normalised for better visualisation.

(a) ResNet101

(b) ResNet50

(c) ResNet18

Figure 8: Top 100 proposals generated by RPN with different backbones, which
are then used to synthesise crowdsourced bounding boxes. The brighter box
indicates a larger objectness score.

##  Appendix 0.H Extensions of BDC to Object Detectors as Annotators

Although our work mainly experimented with crowdsourced datasets annotated by
real or simulated annotators, the proposed BDC can be extended to scenarios
where pre-trained object detector models are directly used as annotators. We
synthesise a synthetic dataset utilising COCO-pretrained SSD, SSDLite and
FRCNN with MobileNetV3 320 backbone provided by Torchvision. Individually, the
3 models obtain 50.3, 46.7 and 47.8 AP.5:.95 on the VOC test dataset. We
directly used their object predictions (_i.e_. predicted class labels and
bounding boxes) of the VOC training dataset as crowdsourced annotations.

Table 7 summarises the experiment results of YOLOv7, FRCNN and EVA trained
with different aggregation methods on said synthetic dataset. We observe that
the proposed BDC leads to AP.5:.95 of 55.5, 53.0 and 65.1 for YOLOv7, FRCNN
and EVA, respectively, on the test dataset which is higher than using MV
(52.5, 51.3 and 64.6 for YOLOv7, FRCNN and EVA, respectively). Moreover, BDC
consistently achieves the highest AP of the aggregated annotations for the
training dataset.

Table 7: Test and train aggregation AP results for YOLOv7, FRCNN and EVA model
trained with different annotation aggregation methods on the synthetic
crowdsourced VOC dataset created using 3 pre-trained object detectors.

Model | Method | Test AP | Train Aggregation AP
---|---|---|---
AP.5 | AP.75 | AP.5:.95 | AP.5 | AP.75 | AP.5:.95
YOLOv7 | NA | 73.8 | 53.7 | 50.9 | 20.5 | 15.1 | 13.8
MV | 78.4 | 58.6 | 52.5 | 54.2 | 38.2 | 34.8
WBF-EARL [27] | 75.8 | 59.7 | 52.7 | 54.6 | 32.2 | 32.1
BDC (ours) | 81.9 | 63.2 | 55.5 | 72.5 | 51.9 | 47.3
FRCNN | NA | 71.7 | 47.6 | 44.6 | 20.5 | 15.1 | 13.8
MV | 77.0 | 58.0 | 51.3 | 54.2 | 38.2 | 34.8
Crowd R-CNN [18] | 75.4 | 54.4 | 50.7 | 52.4 | 33.1 | 32.6
WBF-EARL [27] | 79.1 | 55.1 | 50.2 | 54.6 | 32.2 | 32.1
BDC (ours) | 81.0 | 59.5 | 53.0 | 71.3 | 48.6 | 45.5
EVA | NA | 86.3 | 74.5 | 63.1 | 20.5 | 15.1 | 13.8
MV | 87.7 | 75.6 | 64.7 | 54.2 | 38.2 | 34.8
WBF-EARL [27] | 87.5 | 75.2 | 64.4 | 54.6 | 32.2 | 32.1
BDC (ours) | 87.9 | 75.4 | 65.1 | 71.7 | 48.4 | 45.6

##  Appendix 0.I Additional Qualitative Results

Figure 9 shows additional results of the aggregated annotations from each
method on the VOC-MIX dataset. The proposed BDC outperforms other methods
across various types of objects, _e.g_. small objects, objects with
overlapping bounding boxes, _etc_.

(a) NA (b) MV

(c) WBF-EARL (d) Crowd R-CNN

(e) BDC (ours) (f) Ground truth

Figure 9: Additional comparisons of the aggregated labels from different
methods on the VOC-MIX dataset. For WBF-EARL [27], the number beside the class
label is the annotators’ level of agreement while for Crowd R-CNN [18] and
BDC, the number indicates the class probability. For NA, the colours represent
the different annotators.

##  Appendix 0.J Analysis of Posterior Distributions

The proposed BDC enables the grouping of the annotators into different
clusters of similar skill levels by analysing the learnt distributions of
bounding box errors and confusion matrices for each annotator. We perform a
K-means clustering [30] on the parameters of these distributions, _i.e_.
$\bm{\mu}$, $\bm{\upsilon}$, $\bm{\beta}$ and $\bm{\alpha}$, of the COCO-MIX
YOLOv7 experiment by setting the number of clusters to four. Figures 10 and 11
visualise the mean of each cluster’s Dirichlet posteriors $\pi^{k}$ of the CLA
and the Gaussian distribution for the x-axis translation error of BBA,
respectively. We observe that the clusters of annotators correspond to the
four groups of annotators in the COCO-MIX synthetic setting discussed in
Section 4.2, _i.e_. C1, C3, C4 and C2 correspond to expert, average, poor and
random annotators, respectively.

For example, the average diagonal values of $\pi^{k}$ for clusters C1, C3 and
C4 are similar to the overall test accuracy of the three classification
networks, _i.e_. ResNet50, VGG16 and AlexNet, used to synthesise the
crowdsourced class labels, while the heatmap of C2 is close to a uniform
distribution; The variances for the x-axis translation error of each cluster
are consistent with those of the four groups of annotators in the COCO-MIX
synthetic setting (_i.e_. variance of
$\text{C1}<\text{C3}<\text{C4}<\text{C2}$). This exemplifies the ability of
the BDC framework to correctly learn the annotators’ skill levels.

(a) C1 (0.692)

(b) C2 (0.174)

(c) C3 (0.599)

(d) C4 (0.387)

Figure 10: The mean of the Dirichlet posteriors $\pi^{k}$ of the CLA of the
COCO-MIX YOLOv7 experiment visualised as a logarithm-scale heatmap for each
cluster found using K-means clustering [30] on the aggregators’ parameters
(_i.e_. $\bm{\mu}$, $\bm{\upsilon}$, $\bm{\beta}$ and $\bm{\alpha}$). X-axis
is the annotated class while y-axis is the ground truth class. The average
diagonal value of each cluster’s $\pi^{k}$ is indicated in brackets in each
caption. Figure 11: The Gaussian distributions for the x-axis translation
error of the BBA of the COCO-MIX YOLOv7 experiment for each cluster found
using K-means clustering [30] on the aggregators’ parameters (_i.e_.
$\bm{\mu}$, $\bm{\upsilon}$, $\bm{\beta}$ and $\bm{\alpha}$).