latexText page 17 contains only floats

#  Multi-annotator Deep Learning:
A Probabilistic Framework for Classification

Marek Herde<EMAIL_ADDRESS>
Intelligent Embedded Systems
University of Kassel
Kassel, Hesse, Germany Denis Huseljic<EMAIL_ADDRESS>
Intelligent Embedded Systems
University of Kassel
Kassel, Hesse, Germany Bernhard Sick<EMAIL_ADDRESS>
Intelligent Embedded Systems
University of Kassel
Kassel, Hesse, Germany

###### Abstract

Solving complex classification tasks using deep neural networks typically
requires large amounts of annotated data. However, corresponding class labels
are noisy when provided by error-prone annotators, e.g., crowdworkers.
Training standard deep neural networks leads to subpar performances in such
multi-annotator supervised learning settings. We address this issue by
presenting a probabilistic training framework named multi-annotator deep
learning (MaDL). A downstream ground truth and an annotator performance model
are jointly trained in an end-to-end learning approach. The ground truth model
learns to predict instances’ true class labels, while the annotator
performance model infers probabilistic estimates of annotators’ performances.
A modular network architecture enables us to make varying assumptions
regarding annotators’ performances, e.g., an optional class or instance
dependency. Further, we learn annotator embeddings to estimate annotators’
densities within a latent space as proxies of their potentially correlated
annotations. Together with a weighted loss function, we improve the learning
from correlated annotation patterns. In a comprehensive evaluation, we examine
three research questions about multi-annotator supervised learning. Our
findings show MaDL’s state-of-the-art performance and robustness against many
correlated, spamming annotators.

##  1 Introduction

Supervised deep neural networks (DNNs) have achieved great success in many
classification tasks (Pouyanfar et al., 2018). In general, these DNNs require
a vast amount of annotated data for their successful employment (Algan &
Ulusoy, 2021). However, acquiring top-quality class labels as annotations is
time-intensive and/or financially expensive (Herde et al., 2021). Moreover,
the overall annotation load may exceed a single annotator’s workforce (Uma et
al., 2021). For these reasons, multiple non-expert annotators, e.g.,
crowdworkers, are often tasked with data annotation (Zhang, 2022; Gilyazev &
Turdakov, 2018). Annotators’ missing domain expertise can lead to erroneous
annotations, known as noisy labels. Further, even expert annotators cannot be
assumed to be omniscient because additional factors, such as missing
motivation, fatigue, or an annotation task’s ambiguity (Vaughan, 2018), may
decrease their performances. A popular annotation quality assurance option is
the acquisition of multiple annotations per data instance with subsequent
aggregation (Zhang et al., 2016), e.g., via majority rule. The aggregated
annotations are proxies of the ground truth (GT) labels to train DNNs.
Aggregation techniques operate exclusively on the basis of annotations. In
contrast, model-based techniques use feature or annotator information and thus
work well in low-redundancy settings, e.g., with just one annotation per
instance (Khetan et al., 2018). Through predictive models, these techniques
jointly estimate instances’ GT labels and annotators’ performances (APs) by
learning and inferring interdependencies between instances, annotators, and
their annotations. As a result, model-based techniques cannot only predict GT
labels and APs for training instances but also for test instances, i.e., they
can be applied in transductive and inductive learning settings (Vapnik, 1995).

Despite ongoing research, several challenges still need to be addressed in
multi-annotator supervised learning. To introduce these challenges, we
exemplarily look at the task of animal classification in Fig. 1. Eight
annotators have been queried to provide annotations for the image of a jaguar.
Such a query is difficult because jaguars have remarkable similarities to
other predatory cats, e.g., leopards. Accordingly, the obtained annotations
indicate a strong disagreement between the leopard and jaguar class. Simply
taking the majority vote of these annotations results in leopard as a wrongly
estimated GT label. Therefore, advanced multi-annotator supervised learning
techniques leverage annotation information from other (similar) annotated
images to estimate APs. However, producing accurate AP estimates is difficult
because one needs to learn many annotation patterns. Otherwise, the estimated
GT labels will be biased, e.g., when APs are exclusively modeled as a function
of annotators. In this case, we cannot identify annotators who are only
knowledgeable about certain classes or regions in the feature space. Another
challenge in multi-annotator supervised learning concerns potential (latent)
correlations between annotators. In our animal annotation task, we illustrate
this issue by visualizing three latent groups of similarly behaving
annotators. Although we assume that the annotators work independently of each
other, they can still share common or statistically correlated error patterns
(Chu et al., 2021). This is particularly problematic if a group of ordinary
persons strongly outvotes a much smaller group of professionals. Considering
prior information about the annotators, i.e., annotator features or metadata
(Zhang et al., 2023), can help to identify these groups. Moreover, prior
information enables a model to inductively learn performances for annotators
who have provided few or no annotations. In our example, zoological interest
could be a good indicator for this purpose. While the inductive learning of
APs for annotators poses an additional challenge to the already complex task,
its use may be beneficial for further applications, e.g., optimizing the
annotator selection in an active learning setting (Herde et al., 2021) or
training annotators to improve their own knowledge (Daniel et al., 2018).

Figure 1:  Animal classification as an illustration of a multi-annotator
supervised learning problem.

In this article, we address the above challenges by making the following
contributions: • We propose multi-annotator deep learning  (MaDL) as a
probabilistic and modular classification framework. In an end-to-end training
via a weighted maximum-likelihood approach, it learns embeddings of annotators
to account for possible correlations among them. • We specify six properties
concerning the estimation of APs and application scenarios for categorizing
related multi-annotator supervised learning techniques. • Associated with
these properties, we formulate three research questions (RQs), which we
experimentally investigate, including comparisons of MaDL to related
techniques.

The remainder of this article is structured as follows: In Section 2, we
formally introduce the problem setting of supervised learning from multiple
annotators. Subsequently, we identify central properties of multi-annotator
supervised learning techniques as a basis for categorizing related works and
pointing out their differences to MaDL in Section 3. Section 4 explains the
details of our MaDL framework. Experimental evaluations of MaDL and related
techniques are presented regarding RQs associated with the aforementioned
properties in Section 5. Finally, we conclude and give an outlook regarding
future research work in Section 6.

##  2 Problem Setting

In this section, we formalize the assumptions and objectives of multi-
annotator supervised learning for classification tasks.

Prerequisites: Without loss of generality, we represent a data instance as a
vector ${\mathbf{x}\coloneqq(x^{(1)},...,x^{(D)})^{\mathrm{T}}}$,
$D\in\mathbb{N}_{>0}$ in a $D$-dimensional real-valued input or feature space
$\Omega_{X}\coloneqq\mathbb{R}^{D}$. The $N\in\mathbb{N}_{>0}$ instances
jointly form a matrix
$\mathbf{X}\coloneqq(\mathbf{x}_{1},...,\mathbf{x}_{N})^{\mathrm{T}}$ and
originate from an unknown probability density function $\Pr(\mathbf{x})$. For
each observed instance $\mathbf{x}_{n}\sim\Pr(\mathbf{x})$, there is a GT
class label ${y_{n}\in\Omega_{Y}\coloneqq\\{1,\dots,C\\}}$. Each GT label
$y_{n}$ is assumed to be drawn from an unknown conditional distribution:
$y_{n}\sim\Pr(y\mid\mathbf{x}_{n})$. We denote the GT labels as the vector
$\mathbf{y}\coloneqq(y_{1},...,y_{N})^{\mathrm{T}}$. These GT labels are
unobserved since there is no omniscient annotator. Instead, we consider
multiple error-prone annotators. For the sake of simplicity, we represent an
annotator through individual features as a vector
$\mathbf{a}_{m}\in\Omega_{A}\coloneqq\mathbb{R}^{O},O\in\mathbb{N}_{>0}$. If
no prior annotator information is available, the annotators’ features are
defined through one-hot encoded vectors, i.e.,
$\Omega_{A}\coloneqq\\{\mathbf{e}_{1},\dots,\mathbf{e}_{M}\\}$ with
$\mathbf{a}_{m}\coloneqq\mathbf{e}_{m}$, to identify each annotator uniquely.
Otherwise, annotator features may provide information specific to the general
annotation task, e.g., the zoological interest when annotating animal images
or the years of experience in clinical practice when annotating medical data.
Together, the $M\in\mathbb{N}_{>0}$ annotators form a matrix
$\mathbf{A}\coloneqq(\mathbf{a}_{1},\dots,\mathbf{a}_{M})^{\mathrm{T}}$. We
denote the annotation assigned by annotator $\mathbf{a}_{m}$ to instance
$\mathbf{x}_{n}$ through $z_{nm}\in\Omega_{Y}\cup\\{\otimes\\}$, where
$z_{nm}=\otimes$ indicates that an annotation is unobserved, i.e., not
provided. An observed annotation is assumed to be drawn from an unknown
conditional distribution:
$z_{nm}\sim\Pr(z\mid\mathbf{x}_{n},\mathbf{a}_{m},y)$. Multiple annotations
for an instance $\mathbf{x}_{n}$ can be summarized as a vector
$\mathbf{z}_{n}\coloneqq(z_{n1},...,z_{nM})^{\mathrm{T}}$. Thereby, the set
$\mathcal{A}_{n}\coloneqq\\{m\mid m\in\\{1,\dots,M\\}\wedge
z_{nm}\in\Omega_{Y}\\}$ represents the indices of the annotators who assigned
an annotation to an instance $\mathbf{x}_{n}$. Together, the annotations of
all observed instances form the matrix
$\mathbf{Z}\coloneqq(\mathbf{z}_{1},...,\mathbf{z}_{N})^{\mathrm{T}}$. We
further assume there is a subset of annotators whose annotated instances are
sufficient to approximate the GT label distribution, i.e., together, these
annotated instances allow us to correctly differentiate between all classes.
Otherwise, supervised learning is hardly possible without explicit prior
knowledge about the distributions of GT labels and/or APs. Moreover, we expect
that the annotators independently decide on instances’ annotations and that
their APs are time-invariant.

Objectives: Given these prerequisites, the first objective is to train a
downstream GT model, which approximates the optimal GT decision function
$y_{\mathrm{GT}}:\Omega_{X}\rightarrow\Omega_{Y}$ by minimizing the expected
loss across all classes:

$\displaystyle
y_{\mathrm{GT}}(\mathbf{x})\coloneqq\operatorname*{arg\,min}_{y^{\prime}\in\Omega_{Y}}\left(\mathbb{E}_{y\mid\mathbf{x}}\left[L_{\mathrm{GT}}(y,y^{\prime})\right]\right).$
(1)

Thereby, we define the loss function
${L_{\text{GT}}:\Omega_{Y}\times\Omega_{Y}\rightarrow\\{0,1\\}}$ through the
zero-one loss:

$L_{\text{GT}}(y,y^{\prime})\coloneqq\delta(y\neq
y^{\prime})\coloneqq\begin{cases}0,\text{ if }y=y^{\prime},\\\ 1,\text{ if
}y\neq y^{\prime}.\end{cases}$ (2)

As a result, an optimal GT model for classification tasks can accurately
predict the GT labels of instances.

######  Proposition 1.

Assuming $L_{\mathrm{GT}}$ to be the zero-one loss in Eq. 2, the Bayes optimal
prediction for Eq. 1 is given by:

$\displaystyle
y_{\mathrm{GT}}(\mathbf{x})=\operatorname*{arg\,max}_{y^{\prime}\in\Omega_{Y}}\left(\Pr(y^{\prime}\mid\mathbf{x})\right).$
(3)

When learning from multiple annotators, the APs are further quantities of
interest. Therefore, the second objective is to train an AP model, which
approximates the optimal AP decision function
${y_{\mathrm{AP}}:\Omega_{X}\times\Omega_{A}\rightarrow\\{0,1\\}}$ by
minimizing the following expected loss:

$y_{\mathrm{AP}}(\mathbf{x},\mathbf{a})\coloneqq\operatorname*{arg\,min}_{y^{\prime}\in\\{0,1\\}}\left(\mathbb{E}_{y\mid\mathbf{x}}\left[\mathbb{E}_{z\mid\mathbf{x},\mathbf{a},y}\left[L_{\mathrm{AP}}\left(y^{\prime},L_{\mathrm{GT}}\left(y,z\right)\right)\right]\right]\right).$
(4)

Defining $L_{\mathrm{AP}}$ and $L_{\mathrm{GT}}$ as zero-one loss, an optimal
AP model for classification tasks can accurately predict the zero-one loss of
annotator’s class labels, i.e., whether an annotator $\mathbf{a}$ provides a
false, i.e., $y_{\mathrm{AP}}(\mathbf{x},\mathbf{a})=1$, or correct, i.e.,
$y_{\mathrm{AP}}(\mathbf{x},\mathbf{a})=0$, class label for an instance
$\mathbf{x}$.

######  Proposition 2.

Assuming both $L_{\mathrm{AP}}$ and $L_{\mathrm{GT}}$ to be the zero-one loss,
as defined in Eq. 2, the Bayes optimal prediction for Eq. 4 is given by:

$\displaystyle
y_{\mathrm{AP}}(\mathbf{x},\mathbf{a})=\delta\left(\sum_{y\in\Omega_{Y}}\Pr(y\mid\mathbf{x})\Pr(y\mid\mathbf{x},\mathbf{a},y)<0.5\right).$
(5)

We refer to Appendix A for the proofs of Proposition 1 and Proposition 2.

##  3 Related Work

This section discusses existing multi-annotator supervised learning techniques
targeting our problem setting of Section 2. Since we focus on the AP next to
the GT estimation, we restrict our discussion to techniques capable of
estimating both target types. In this context, we analyze related research
regarding three aspects, i.e., GT models, AP models, and algorithms for
training these models.

Ground truth model: The first multi-annotator supervised learning techniques
employed logistic regression models (Raykar et al., 2010; Kajino et al., 2012;
Rodrigues et al., 2013; Yan et al., 2014) for classification. Later, different
kernel-based variants of GT models, e.g., Gaussian processes, were developed
(Rodrigues et al., 2014; Long et al., 2016; Gil-Gonzalez et al., 2021).
Rodrigues et al. (2017) focused on documents and extended topic models to the
multi-annotator setting. More recently, several techniques were proposed to
train DNNs for large-scale and especially image classification tasks with
noisy annotations (Albarqouni et al., 2016; Guan et al., 2018; Khetan et al.,
2018; Rodrigues & Pereira, 2018; Yang et al., 2018; Tanno et al., 2019; Cao et
al., 2019; Platanios et al., 2020; Zhang et al., 2020; Gil-González et al.,
2021; Rühling Cachay et al., 2021; Chu et al., 2021; Li et al., 2022; Wei et
al., 2022; Gao et al., 2022). MaDL follows this line of work and also employs
a (D)NN as the GT model.

Annotator performance model: An AP model is typically seen as an auxiliary
part of the GT model since it provides AP estimates for increasing the GT
model’s performance. In this article, we reframe an AP model’s use in a more
general context because accurately assessing APs can be crucial in improving
several applications, e.g., human-in-the-loop processes (Herde et al., 2021)
or knowledge tracing (Piech et al., 2015). For this reason, we analyze
existing AP models regarding six properties, which we identified as relevant
while reviewing literature about multi-annotator supervised learning.

(P1) Class-dependent annotator performance: The simplest AP representation is
an overall accuracy value per annotator. On the one hand, AP models estimating
such accuracy values have low complexity and thus do not overfit (Rodrigues et
al., 2013; Long et al., 2016). On the other hand, they may be overly general
and cannot assess APs on more granular levels. Therefore, many other AP models
assume a dependency between APs and instances’ GT labels. Class-dependent AP
models typically estimate confusion matrices (Raykar et al., 2010; Rodrigues
et al., 2014; 2017; Khetan et al., 2018; Tanno et al., 2019; Platanios et al.,
2020; Gao et al., 2022; Li et al., 2022), which indicate annotator-specific
probabilities of mistaking one class for another, e.g., recognizing a jaguar
as a leopard. Alternatively, weights of annotation aggregation functions (Cao
et al., 2019; Rühling Cachay et al., 2021) or noise-adaption layers (Rodrigues
& Pereira, 2018; Chu et al., 2021; Wei et al., 2022) can be interpreted as
non-probabilistic versions of confusion matrices. MaDL estimates probabilistic
confusion matrices or less complex approximations, e.g., the elements on their
diagonals.

(P2) Instance-dependent annotator performance: In many real-world
applications, APs are additionally instance-dependent (Yan et al., 2014)
because instances of the same class can strongly vary in their feature values.
For example, recognizing animals in blurry images is more difficult than in
high-resolution images. Hence, several AP models estimate the probability of
obtaining a correct annotation as a function of instances and annotators
(Kajino et al., 2012; Yan et al., 2014; Guan et al., 2018; Yang et al., 2018;
Gil-Gonzalez et al., 2021; Gil-González et al., 2021). Combining instance- and
class-dependent APs results in the most complex AP models, which estimate a
confusion matrix per instance-annotator pair (Platanios et al., 2020; Zhang et
al., 2020; Rühling Cachay et al., 2021; Chu et al., 2021; Gao et al., 2022; Li
et al., 2022). MaDL also employs an AP model of this type. However, it
optionally allows dropping the instance and class dependency, which can
benefit classification tasks where each annotator provides only a few
annotations.

(P3) Annotator correlations: Although most techniques assume that annotators
do not collaborate, they can still have correlations regarding their
annotation patterns, e.g., by sharing statistically correlated error patterns
(Chu et al., 2021). Gil-Gonzalez et al. (2021) proposed a kernel-based
approach where a matrix quantifies such correlations for all pairs of
annotators. Inspired by weak supervision, Cao et al. (2019) and Rühling Cachay
et al. (2021) employ an aggregation function that takes all annotations per
instance as input to model annotator correlations. Gil-González et al. (2021)
introduce a regularized chained DNN whose weights encode correlations. Wei et
al. (2022) jointly model the annotations of all annotators as outputs and thus
take account of potential correlated mistakes. Chu et al. (2021) consider
common annotation noise through a noise adaptation layer shared across
annotators. Moreover, similar to our MaDL framework, they learn embeddings of
annotators. Going beyond, MaDL exploits these embeddings to determine
annotator correlations.

(P4) Robustness to spamming annotators: Especially on crowdsourcing platforms,
there have been several reports of workers spamming annotations (Vuurens et
al., 2011), e.g., by randomly guessing or permanently providing the same
annotation. Such spamming annotators can strongly harm the learning process.
As a result, multi-annotator supervised learning techniques are ideally robust
against these types of annotation noise. Cao et al. (2019) employ an
information-theoretic approach to separate expert annotators from possibly
correlated spamming annotators. Rühling Cachay et al. (2021) empirically
demonstrated that their weak-supervised learning technique is robust to large
numbers of randomly guessing annotators. MaDL ensures this robustness by
training via a weighted likelihood function, assigning high weights to
independent annotators whose annotation patterns have no or only slight
statistical correlations to the patterns of other annotators.

(P5) Prior annotator information: On crowdsourcing platforms, requesters may
acquire prior information about annotators (Daniel et al., 2018), e.g.,
through surveys, annotation quality tests, or publicly available profiles.
Several existing AP models leverage such information to improve learning.
Thereby, conjugate prior probability distributions, e.g., Dirichlet
distributions, represent a straightforward way of including prior estimates of
class-dependent accuracies (Raykar et al., 2010; Albarqouni et al., 2016;
Rodrigues et al., 2017). Other techniques (Platanios et al., 2020; Chu et al.,
2021), including our MaDL framework, do not directly expect prior accuracy
estimates but work with all types of prior information that can be represented
as vectors of annotator features.

(P6) Inductive learning of annotator performance: Accurate AP estimates can be
beneficial in various applications, e.g., guiding an active learning strategy
to select accurate annotators (Yang et al., 2018). For this purpose, it is
necessary that a multi-annotator supervised learning technique can inductively
infer APs for non-annotated instances. Moreover, an annotation process is
often a dynamic system where annotators leave and enter. Hence, it is highly
interesting to inductively estimate the performances of newly entered
annotators, e.g., through annotator features as used by Platanios et al. 2020
and MaDL.

Training: Several multi-annotator supervised learning techniques employ the
expectation-maximization (EM) algorithm for training (Raykar et al., 2010;
Rodrigues et al., 2013; Yan et al., 2014; Long et al., 2016; Albarqouni et
al., 2016; Guan et al., 2018; Khetan et al., 2018; Yang et al., 2018;
Platanios et al., 2020). GT labels are modeled as latent variables and
estimated during the E step, while the GT and AP models’ parameters are
optimized during the M step. The exact optimization in the M step depends on
the underlying models. Typically, a variant of gradient descent (GD), e.g.,
quasi-Newton methods, is employed, or a closed-form solution exists, e.g., for
AP models with instance-independent AP estimates. Other techniques take a
Bayesian view of the models’ parameters and therefore resort to expectation
propagation (EP) (Rodrigues et al., 2014; Long et al., 2016) or variational
inference (VI) (Rodrigues et al., 2017). As approximate inference methods are
computationally expensive and may lead to suboptimal results, several end-to-
end training algorithms have been proposed. Gil-Gonzalez et al. (2021)
introduced a localized kernel alignment-based relevance analysis that
optimizes via GD. Through a regularization term, penalizing differences
between GT and AP model parameters, Kajino et al. formulated a convex loss
function for logistic regression models. Rodrigues & Pereira (2018), Gil-
González et al. (2021), and Wei et al. (2022) jointly train the GT and AP
models by combining them into a single DNN with noise adaption layers. Chu et
al. (2021) follow a similar approach with two types of noise adaption layers:
one shared across annotators and one individual for each annotator. Gil-
González et al. (2021) employ a regularized chained DNN to estimate GT labels
and AP performances jointly. In favor of probabilistic AP estimates, Tanno et
al. (2019), Zhang et al. (2020), Li et al. (2022), and MaDL avoid noise
adaption layers but employ loss functions suited for end-to-end learning. Cao
et al. (2019) and Rühling Cachay et al. (2021) jointly learn an aggregation
function in combination with the AP and GT models.

Table 1 summarizes and completes the aforementioned discussion by categorizing
multi-annotator supervised learning techniques according to their GT model, AP
model, and training algorithm. Thereby, the AP model is characterized by the
six previously discussed properties (P1–P6). We assign ✓ if a property is
supported, ✗ if not supported, and ✦ if partially supported. More precisely, ✦
is assigned to property P5 if the technique can include prior annotator
information but needs a few adjustments and to property P6 if the technique
requires some architectural changes to learn the performances of new
annotators inductively. For property P4, a ✓ indicates that the authors have
shown that their proposed technique learns in the presence of many spamming
annotators.

Table 1: Literature categorization of multi-annotator supervised learning techniques. Reference | Ground Truth Model | Training | Annotator Performance Model
---|---|---|---
P1 | P2 | P3 | P4 | P5 | P6
Kajino et al. (2012) | Logistic Regression Model | GD | ✗ | ✓ | ✗ | ✗ | ✗ | ✗
Raykar et al. (2010) | EM & GD | ✓ | ✗ | ✗ | ✗ | ✓ | ✗
Rodrigues et al. (2013) | ✗ | ✗ | ✗ | ✗ | ✗ | ✗
Yan et al. (2014) | ✗ | ✓ | ✗ | ✗ | ✦ | ✦
Rodrigues et al. (2017) | Topic Model | VI & GD | ✓ | ✗ | ✗ | ✗ | ✓ | ✗
Rodrigues et al. (2014) | Kernel-based Model | EP | ✓ | ✗ | ✗ | ✗ | ✗ | ✗
Long et al. (2016) | EM & EP & GD | ✗ | ✗ | ✗ | ✗ | ✗ | ✗
Gil-Gonzalez et al. (2021) | GD | ✗ | ✓ | ✓ | ✗ | ✗ | ✗
Albarqouni et al. (2016) | (Deep) Neural Network | EM & GD | ✓ | ✗ | ✗ | ✗ | ✓ | ✗
Yang et al. (2018) | ✗ | ✓ | ✗ | ✗ | ✦ | ✦
Khetan et al. (2018) | ✓ | ✗ | ✗ | ✗ | ✦ | ✦
Platanios et al. (2020) | ✓ | ✓ | ✗ | ✗ | ✓ | ✓
Rodrigues & Pereira (2018) | GD | ✓ | ✗ | ✗ | ✗ | ✗ | ✗
Guan et al. (2018) | ✗ | ✓ | ✗ | ✗ | ✗ | ✗
Tanno et al. (2019) | ✓ | ✗ | ✗ | ✗ | ✦ | ✦
Cao et al. (2019) | ✓ | ✗ | ✓ | ✓ | ✗ | ✗
Zhang et al. (2020) | ✓ | ✓ | ✓ | ✗ | ✦ | ✦
Gil-González et al. (2021) | ✗ | ✓ | ✓ | ✓ | ✗ | ✗
Rühling Cachay et al. (2021) | ✓ | ✓ | ✓ | ✓ | ✗ | ✗
Chu et al. (2021) | ✓ | ✓ | ✓ | ✓ | ✓ | ✗
Li et al. (2022) | ✓ | ✓ | ✗ | ✗ | ✦ | ✦
Wei et al. (2022) | ✓ | ✗ | ✓ | ✗ | ✗ | ✗
Gao et al. (2022) | ✓ | ✓ | ✗ | ✗ | ✦ | ✦
MaDL (2023) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓

##  4 Multi-annotator Deep Learning

In this section, we present our modular probabilistic MaDL framework. We start
with a description of its underlying probabilistic model. Subsequently, we
introduce its GT and AP models’ architectures. Finally, we explain our end-to-
end training algorithm.

###  4.1 Probabilistic Model

The four nodes in Fig. 2 depict the random variables of an instance
$\mathbf{x}$, a GT label $y$, an annotator $\mathbf{a}$, and an annotation
$z$. Thereby, arrows indicate probabilistic dependencies among each other. The
random variable of an instance $\mathbf{x}$ and an annotator $\mathbf{a}$ have
no incoming arrows and thus no causal dependencies on other random variables.
In contrast, the distribution of a latent GT label $y$ depends on its
associated instance $\mathbf{x}$. For classification problems, the probability
of observing $y=c$ as GT label of an instance $\mathbf{x}$ can be modeled
through a categorical distribution:

$\displaystyle\Pr(y=c\mid\mathbf{x})\coloneqq\mathrm{Cat}(y=c\mid\bm{p}(\mathbf{x}))\coloneqq\prod_{k=1}^{C}\left(p^{(k)}(\mathbf{x})\right)^{\delta(k=c)}=p^{(c)}(\mathbf{x}),$
(6)

where
$\mathbf{p}:\Omega_{X}\rightarrow\Delta\coloneqq\\{\mathbf{p}\in[0,1]^{C}\mid\sum_{c=1}^{C}p^{(c)}=1\\}$
denotes the function outputting an instance’s true class-membership
probabilities. The outcome of an annotation process may depend on the
annotator’s features, an instance’s features, and the latent GT label. A
function $\mathbf{P}:\Omega_{X}\times\Omega_{A}\rightarrow[0,1]^{C\times C}$
outputting a row-wise normalized confusion matrix per instance-annotator pair
can capture these dependencies. The probability that an annotator $\mathbf{a}$
annotates an instance $\mathbf{x}$ of class $y=c$ with the annotation $z=k$
can then be modeled through a categorical distribution:

$\displaystyle\Pr(z=k\mid\mathbf{x},\mathbf{a},y=c)$
$\displaystyle\coloneqq\text{Cat}\left(z=k\,\middle|\,\mathbf{P}^{(c,:)}(\mathbf{x},\mathbf{a})\right)\coloneqq\prod_{l=1}^{C}\left(P^{(c,l)}(\mathbf{x},\mathbf{a})\right)^{\delta(l=k)}=P^{(c,k)}(\mathbf{x},\mathbf{a}),$
(7)

where the column vector $\mathbf{P}^{(c,:)}(\mathbf{x},\mathbf{a})\in\Delta$
corresponds to the $c$-th row of the confusion matrix
$\mathbf{P}(\mathbf{x},\mathbf{a})$.

Figure 2: Probabilistic graphical model of MaDL.

###  4.2 Model Architectures

Now, we introduce how MaDL’s GT and AP models are designed to approximate the
functions of true class-membership probabilities $\mathbf{p}$ and true
confusion matrices $\mathbf{P}$ for the respective instances and annotators.
Fig. 3 illustrates the architecture of the GT (purple) and AP (green) models
within our MaDL framework. Solid lines indicate mandatory components, while
dashed lines express optional ones.

Figure 3: Architectures of MaDL’s GT and AP models.

The GT model with parameters $\bm{\theta}$ is a (D)NN (cf. 4 in Fig. 3), which
takes an instance $\mathbf{x}$ as input to approximate its true class-
membership probabilities $\mathbf{p}(\mathbf{x})$ via
$\mathbf{\hat{p}}_{\bm{\theta}}(\mathbf{x})$. We define its decision function
in analogy to the Bayes optimal prediction in Eq. 3 through

$\displaystyle\hat{y}_{\bm{\theta}}(\mathbf{x})\coloneqq\operatorname*{arg\,max}_{y\in\Omega_{Y}}\left(\hat{p}_{\bm{\theta}}^{(y)}(\mathbf{x})\right).$
(8)

Figure 4: MaDL’s residual block combining annotator and instance embedding.

The architecture of the AP model with parameters $\bm{\omega}$ comprises
mandatory and optional components. We start by describing its most general
form, which consists of three (D)NNs and estimates annotator-, class-, and
instance-dependent APs. Annotator features $\mathbf{a}$ are propagated through
a first (D)NN (cf. 1 in Fig. 3) to learn an annotator embedding
${\mathbf{\widetilde{a}}\in\mathbb{R}^{R},R\in\mathbb{N}_{\geq 1}}$. During
training, we will use such embeddings for quantifying correlations between
annotators. Analogously, we propagate raw instance features $\mathbf{x}$ or a
representation learned by the GT model’s hidden layers through a second (D)NN
(cf. 2 in Fig. 3) for learning an instance embedding
${\mathbf{\widetilde{x}}\in\mathbb{R}^{Q},Q\in\mathbb{N}_{\geq 1}}$.
Subsequently, instance and annotator embeddings $\mathbf{\widetilde{x}}$ and
$\mathbf{\widetilde{a}}$ are combined through a third and final (D)NN (cf. 3
in Fig. 3) for approximating the true confusion matrix
$\mathbf{P}(\mathbf{x},\mathbf{a})$ via
$\mathbf{\hat{P}}_{\bm{\omega}}(\mathbf{x},\mathbf{a})$. Various architectures
for combining embeddings have already been proposed in the literature
(Fiedler, 2021). We adopt a solution from recommender systems where often
latent factors of users and items are combined (Zhang et al., 2019).
Concretely, in DNN 3, we use an outer product-based layer outputting
$\mathbf{\widetilde{o}}\in\mathbb{R}^{F},F\in\mathbb{N}_{\geq 1}$ to model the
interactions between instance and annotator embeddings (Qu et al., 2016). The
concatenation of $\mathbf{\widetilde{a}},\mathbf{\widetilde{x}}$, and
$\mathbf{\widetilde{o}}$ is propagated through a residual block (He et al.,
2016), whose architecture is visualized in Fig. 4. There, we add only the
annotator embedding $\mathbf{\widetilde{a}}$ to the learned mapping
$\mathbf{h}(\mathbf{\widetilde{a}},\mathbf{\widetilde{x}},\mathbf{\widetilde{o}})\in\mathbb{R}^{R}$.
The motivation behind this modification is that the annotator embeddings,
informing about an annotator’s individuality, are likely to be the most
influential inputs for estimating confusion matrices as APs. Empirical
investigations showed that $R=Q=F=16$ as the embedding size is a robust
default. Finally, we define the AP model’s decision function in analogy to the
Bayes optimal prediction in Eq. 5 through

$\displaystyle\hat{y}_{\bm{\theta},\bm{\omega}}(\mathbf{x},\mathbf{a})\coloneqq\delta\left(\sum_{c=1}^{C}\hat{p}_{\bm{\theta}}^{(c)}(\mathbf{x})\cdot\hat{P}_{\bm{\omega}}^{(c,c)}(\mathbf{x},\mathbf{a})<0.5\right)\coloneqq\delta\left(\underbrace{\hat{p}_{\bm{\theta},\bm{\omega}}(\mathbf{x},\mathbf{a})}_{\text{predicted
correctness probability}}<0.5\right).$ (9)

An AP model estimating a confusion matrix per instance-annotator pair can be
overly complex if there are only a few annotations per annotator or the number
of classes is high (Rodrigues et al., 2013). In such settings, ignoring the
instance features as input of the AP model may be beneficial. Alternatively,
we can constrain a confusion matrix’s degrees of freedom by reducing the
number of output neurons of the AP model. For example, we might estimate only
the diagonal elements of the confusion matrix and assume that the remaining
probability mass per row is uniformly distributed. Further, we can either
estimate each diagonal element individually (corresponding to $C$ output
neurons) or approximate them via a single scalar (corresponding to one output
neuron). Appendix G illustrates such confusion matrices with varying degrees
of freedom.

###  4.3 End-to-end Training

Given the probabilistic model and accompanying architectures of the GT and AP
models, we propose an algorithm for jointly learning their parameters. A
widespread method for training probabilistic models is to maximize the
likelihood of the observed data with respect to the model parameters. Assuming
that the joint distributions of annotations $\mathbf{Z}$ are conditionally
independent for given instances $\mathbf{X}$, we can specify the likelihood
function as follows:

$\Pr(\mathbf{Z}\mid\mathbf{X},\mathbf{A};\bm{\theta},\bm{\omega})=\prod_{n=1}^{N}\Pr(\mathbf{z}_{n}\mid\mathbf{x}_{n},\mathbf{A};\bm{\theta},\bm{\omega}).$
(10)

We further expect that the distributions of annotations $\mathbf{z}_{n}$ for a
given instance $\mathbf{x}_{n}$ are conditionally independent. Thus, we can
simplify the likelihood function:

$\displaystyle\Pr(\mathbf{Z}\mid\mathbf{X},\mathbf{A};\bm{\theta},\bm{\omega})$
$\displaystyle=\prod_{n=1}^{N}\prod_{m\in\mathcal{A}_{n}}\Pr(z_{nm}\mid\mathbf{x}_{n},\mathbf{a}_{m};\bm{\theta},\bm{\omega}).$
(11)

Leveraging our probabilistic model in Fig. 2, we can express the probability
of obtaining a certain annotation as an expectation with respect to an
instance’s (unknown) GT class label:

$\displaystyle\Pr(\mathbf{Z}\mid\mathbf{X},\mathbf{A};\bm{\theta},\bm{\omega})$
$\displaystyle=\prod_{n=1}^{N}\prod_{m\in\mathcal{A}_{n}}\mathbb{E}_{y_{n}\mid\mathbf{x}_{n};\bm{\theta}}\left[\Pr(z_{nm}\mid\mathbf{x}_{n},\mathbf{a}_{m},y_{n};\bm{\omega})\right]$
(12)
$\displaystyle=\prod_{n=1}^{N}\prod_{m\in\mathcal{A}_{n}}\left(\sum_{y_{n}=1}^{C}\Pr(y_{n}\mid\mathbf{x}_{n};\bm{\theta})\Pr(z_{nm}\mid\mathbf{x}_{n},\mathbf{a}_{m},y_{n};\bm{\omega})\right)$
(13)
$\displaystyle=\prod_{n=1}^{N}\prod_{m\in\mathcal{A}_{n}}\mathbf{e}^{\mathrm{T}}_{z_{nm}}\underbrace{\mathbf{\hat{P}}^{\mathrm{T}}_{\bm{\omega}}(\mathbf{x}_{n},\mathbf{a}_{m})\mathbf{\hat{p}}_{\bm{\theta}}(\mathbf{x}_{n})}_{\text{annotation
probabilities}},$ (14)

where $\mathbf{e}_{z_{nm}}$ denotes the one-hot encoded vector of annotation
$z_{nm}$. Taking the logarithm of this likelihood function and converting the
maximization into a minimization problem, we get

$L_{\mathbf{X},\mathbf{A},\mathbf{Z}}(\bm{\theta},\bm{\omega})\coloneqq-\sum_{n=1}^{N}\sum_{m\in\mathcal{A}_{n}}\ln\left(\mathbf{e}^{\mathrm{T}}_{z_{nm}}\mathbf{\hat{P}}^{\mathrm{T}}_{\bm{\omega}}(\mathbf{x}_{n},\mathbf{a}_{m})\mathbf{\hat{p}}_{\bm{\theta}}(\mathbf{x}_{n})\right)$
(15)

as cross-entropy loss function for learning annotation probabilities by
combining the outputs of the GT and AP models (cf. blue components in Fig. 3).
Yet, directly employing this loss function for learning may result in poor
results for two reasons.

Initialization: Reason number one has been noted by Tanno et al. (2019), who
showed that such a loss function cannot ensure the separation of the AP and GT
label distributions. This is because infinite many combinations of class-
membership probabilities and confusion matrices perfectly comply with the true
annotation probabilities, e.g., by swapping the rows of the confusion matrix
as the following example shows:

$\displaystyle\underbrace{\mathbf{P}^{\mathrm{T}}(\mathbf{x}_{n},\mathbf{a}_{m})\mathbf{p}(\mathbf{x}_{n})}_{\text{true
probabilities}}=\begin{pmatrix}1&0\\\ 0&1\end{pmatrix}\begin{pmatrix}1\\\
0\end{pmatrix}=\begin{pmatrix}1\\\ 0\end{pmatrix}=\begin{pmatrix}0&1\\\
1&0\end{pmatrix}\begin{pmatrix}0\\\
1\end{pmatrix}=\underbrace{\mathbf{\hat{P}}^{\mathrm{T}}_{\bm{\omega}}(\mathbf{x}_{n},\mathbf{a}_{m})\mathbf{\hat{p}}_{\bm{\theta}}(\mathbf{x}_{n})}_{\text{predicted
probabilities}}.$ (16)

Possible approaches aim at resolving this issue by favoring certain
combinations, e.g., diagonally dominant confusion matrices. Typically, one can
achieve this via regularization (Tanno et al., 2019; Zhang et al., 2020; Li et
al., 2022) and/or suitable initialization of the AP model’s parameters
(Rodrigues & Pereira, 2018; Wei et al., 2022). We rely on the latter approach
because it permits encoding prior knowledge about APs. Concretely, we
approximate an initial confusion matrix for any instance-annotator pair
$(\mathbf{x}_{n},\mathbf{a}_{m})$ through

$\displaystyle\mathbf{\hat{P}}_{\bm{\omega}}(\mathbf{x}_{n},\mathbf{a}_{m})\coloneqq\begin{pmatrix}\texttt{softmax}((\mathbf{v}^{\mathrm{T}}(\mathbf{x}_{n},\mathbf{a}_{m})\mathbf{W}+\mathbf{B})^{(1,:)})\\\
\vdots\\\
\texttt{softmax}((\mathbf{v}^{\mathrm{T}}(\mathbf{x}_{n},\mathbf{a}_{m})\mathbf{W}+\mathbf{B})^{(C,:)})\end{pmatrix}\approx\eta\mathbf{I}_{C}+\frac{\left(1-\eta\right)}{C-1}\left(\mathbf{1}_{C}-\mathbf{I}_{C}\right),$
(17)

where $\mathbf{I}_{C}\in\mathbb{R}^{C\times C}$ denotes an identity matrix,
$\mathbf{1}_{C}\in\mathbb{R}^{C\times C}$ an all-one matrix, and
$\eta\in(0,1)$ the prior probability of obtaining a correct annotation. For
example, in a binary classification problem, the initial confusion matrix
would approximately take the following values:

$P_{\bm{\omega}}(\mathbf{x}_{n},\mathbf{a}_{m})\approx\begin{pmatrix}\eta&1-\eta\\\
1-\eta&\eta\end{pmatrix}.$ (18)

The outputs of the softmax functions represent the confusion matrix’s rows.
Provided that the initial AP model’s last layer’s weights
$\mathbf{W}\in\mathbb{R}^{H\times C\times C},H\in\mathbb{N}_{>0}$ satisfy
$\mathbf{v}^{\mathrm{T}}(\mathbf{x}_{n},\mathbf{a}_{m})\mathbf{W}\approx\mathbf{0}_{C}\in\mathbb{R}^{C\times
C}$ for the hidden representation
$\mathbf{v}(\mathbf{x}_{n},\mathbf{a}_{m})\in\mathbb{R}^{H}$ of each instance-
annotator pair, we approximate Eq. 17 by initializing the biases
$\mathbf{B}\in\mathbb{R}^{C\times C}$ of our AP model’s output layer via

$\displaystyle\mathbf{B}\coloneqq\ln\left(\frac{\eta\cdot(C-1)}{1-\eta}\right)\mathbf{I}_{C}.$
(19)

By default, we set $\eta=0.8$ to assume trustworthy annotators a priori.
Accordingly, initial class-membership probability estimates are close to the
annotation probability estimates.

Annotator Probability Densities:

$\Pr(\mathbf{a}_{1}\mid\mathbf{A})\mathrel{\vbox{
\offinterlineskip\halign{\hfil$#$\cr\propto\cr\kern
2.0pt\cr\sim\cr\kern-2.0pt\cr}}}1$$\Pr(\mathbf{a}_{2}\mid\mathbf{A})\approx\Pr(\mathbf{a}_{3}\mid\mathbf{A})\approx\Pr(\mathbf{a}_{4}\mid\mathbf{A})\approx\Pr(\mathbf{a}_{5}\mid\mathbf{A})\mathrel{\vbox{
\offinterlineskip\halign{\hfil$#$\cr\propto\cr\kern
2.0pt\cr\sim\cr\kern-2.0pt\cr}}}4$$\Pr(\mathbf{a}_{6}\mid\mathbf{A})\approx\Pr(\mathbf{a}_{7}\mid\mathbf{A})\approx\Pr(\mathbf{a}_{8}\mid\mathbf{A})\mathrel{\vbox{
\offinterlineskip\halign{\hfil$#$\cr\propto\cr\kern
2.0pt\cr\sim\cr\kern-2.0pt\cr}}}3$

Annotator Weights:

$w(\mathbf{a}_{1})\approx\frac{8}{3}$$w(\mathbf{a}_{2})\approx
w(\mathbf{a}_{3})\approx w(\mathbf{a}_{4})\approx
w(\mathbf{a}_{5})\approx\frac{8}{4\cdot 3}$$w(\mathbf{a}_{6})\approx
w(\mathbf{a}_{7})\approx w(\mathbf{a}_{8})\approx\frac{8}{3\cdot 3}$ Figure 5:
Visualization of annotator embeddings (left) accompanied by an exemplary
calculation of annotator probability densities and annotator weights (right).

Annotator weights: Reason number two has been noted by Cao et al. (2019), who
proved that maximum-likelihood solutions fail when there are strong annotator
correlations, i.e., annotators with significant statistical correlations in
their annotation patterns. To address this issue, we explore the annotator
correlations in the latent space of the learned annotator embeddings. For this
purpose, we assume that annotators with similar embeddings share correlated
annotation patterns. Recalling our example in Fig. 1, this assumption implies
that annotators of the same latent group are located near each other. The left
plot of Fig. 5 visualizes this assumption for a two-dimensional embedding
space, where the eight annotators are arranged into three clusters as proxies
of the three latent annotator groups. We aim to extend our loss function so
that its evaluation is independent of the annotator groups’ cardinalities. For
our example, we view the three annotator groups as three independent
annotators of equal importance. To this purpose, we extend the original
likelihood function in Eq. 11 by annotator weights, such that we obtain the
weighted likelihood function:

$\displaystyle\Pr(\mathbf{Z}\mid\mathbf{X},\mathbf{A};\bm{\theta},\bm{\omega},\mathbf{w})$
$\displaystyle=\prod_{n=1}^{N}\prod_{m\in\mathcal{A}_{n}}\Pr(z_{nm}\mid\mathbf{x}_{n},\mathbf{a}_{m};\bm{\theta},\bm{\omega})^{w(\mathbf{a}_{m})},$
(20)

where
$\mathbf{w}\coloneqq(w(\mathbf{a}_{1}),\dots,w(\mathbf{a}_{M}))^{\mathrm{T}}\in\mathbb{R}_{\geq
0}^{M}$ denotes a vector of non-negative annotator weights. From a
probabilistic perspective, we can interpret such a weight $w(\mathbf{a}_{m})$
as the effective number of observations (or copies) per annotation of
annotator $\mathbf{a}_{m}$. Interpreting the annotators $\mathbf{A}$ as
samples from a continuous latent space, we define an annotator weight
$w(\mathbf{a}_{m})$ to be inversely proportional to an annotator’s
$\mathbf{a}_{m}$ probability density:

$w(\mathbf{a}_{m})\coloneqq\frac{\Pr(\mathbf{a}_{m}\mid\mathbf{A})^{-1}}{Z},Z\coloneqq
M^{-1}\left(\sum_{m=1}^{M}\Pr(\mathbf{a}_{m}\mid\mathbf{A})^{-1}\right)\text{
provided that
}\Pr(\mathbf{a}_{1}\mid\mathbf{A}),\dots,\Pr(\mathbf{a}_{M}\mid\mathbf{A})>0.$
(21)

The normalization term $Z\in\mathbb{R}_{>0}$ ensures that the number of
effective annotations remains equal to the number of annotators, i.e.,
$\sum_{m=1}^{M}w(\mathbf{a}_{m})=M$. On the right side of our example in Fig.
5, we expect that an annotator’s probability density is approximately
proportional to the cardinality of the group to which the annotator belongs.
As a result, we assign high (low) weights to annotators belonging to small
(large) groups. Inspecting the exemplary annotator weights and adding the
weights per annotator group, we observe that each group provides the same
number of effective annotations, i.e., $\nicefrac{{8}}{{3}}$. More generally,
we support our definition of the annotator weights by the following theorem,
whose proof is given in Appendix A.

######  Theorem 1.

Let there be $G\in\\{1,\dots,M\\}$ non-empty, disjoint annotator groups, which
we denote as sets of indices such that
$\mathcal{A}^{(1)}\cup\dots\cup\mathcal{A}^{(G)}=\\{1,\dots,M\\}$. Further
assume, the annotators within each group $g\in\\{1,\dots,G\\}$ share identical
annotation patterns for the observed instances, i.e., $\forall
n\in\\{1,\dots,N\\},\forall
m,l\in\mathcal{A}^{(g)}:z_{nm}=z_{nl}\wedge\Pr(z_{nm}\mid\mathbf{x}_{n},\mathbf{a}_{m})=\Pr(z_{nl}\mid\mathbf{x}_{n},\mathbf{a}_{l}),$
$(\dagger)$ and the annotators’ probability densities are proportional to
their respective groups’ cardinalities, i.e., $\forall
m\in\\{1,\dots,M\\}:\Pr(\mathbf{a}_{m}\mid\mathbf{A})\propto\sum_{g=1}^{G}\delta(m\in\mathcal{A}^{(g)})|\mathcal{A}^{(g)}|.$
$(\star)$ Then, the true weighted log-likelihood function for all $M$
annotators reduces to the log-likelihood for $G$ annotators:
$\sum_{n=1}^{N}\sum_{m=1}^{M}w(\mathbf{a}_{m})\ln\left(\Pr(z_{nm}\mid\mathbf{x}_{n},\mathbf{a}_{m})\right)\propto\sum_{n=1}^{N}\sum_{g=1}^{G}\ln\left(\Pr(z_{n{m_{g}}}\mid\mathbf{x}_{n},\mathbf{a}_{m_{g}})\right),$
where $m_{g}\in\mathcal{A}^{(g)}$ represents the index of an arbitrary
annotator of the $g$-th annotator group.

Intuitively, Theorem 1 confirms that each group $\mathcal{A}^{(g)}$,
independent of its cardinality $|\mathcal{A}^{(g)}|$, equally contributes to
the weighted log-likelihood function. This way, we avoid any bias toward a
large group of highly correlated annotators during learning. Typically, the
assumptions $(\dagger)$ ‣ 1 and $(\star)$ ‣ 1 of Theorem 1 do not hold in
practice because there are no annotator groups with identical annotation
patterns. Therefore, we estimate degrees of correlations between annotators by
computing similarities between their embeddings
$\mathbf{\widetilde{A}}\coloneqq(\mathbf{\widetilde{a}}_{1},\dots,\mathbf{\widetilde{a}}_{M})^{\mathrm{T}}$
as the basis for a nonparametric annotator probability density estimation:

$\displaystyle\Pr\left(\mathbf{a}_{m}\mid\mathbf{A}\right)\approx\Pr\left(\mathbf{\widetilde{a}}_{m}\mid\mathbf{\widetilde{A}},k_{\gamma}\right)\propto\sum_{l=1}^{M}k_{\gamma}\left(\texttt{no\\_grad}\left(\mathbf{\widetilde{a}}_{l}\right),\texttt{no\\_grad}\left(\mathbf{\widetilde{a}}_{m}\right)\right),$
(22)

where $k_{\gamma}:\mathbb{R}^{R\times R}\rightarrow\mathbb{R}_{\geq 0}$
denotes a kernel function and $\gamma\in\mathbb{R}_{>0}$ its kernel scale. The
expression $\texttt{no\\_grad}(\mathbf{\widetilde{a}}_{m})\in\mathbb{R}^{R}$
indicates that no gradient regarding the learned annotator embedding
$\mathbf{\widetilde{a}}_{m}$ is computed, which is necessary to decouple the
learning of embeddings from computing annotator weights. Otherwise, we would
learn annotator embeddings, which optimize the annotator weights instead of
reflecting the annotation patterns. Although many kernel (or similarity)
functions are conceivable, we will focus on the popular Gaussian kernel:

$\displaystyle
k_{\gamma}(\texttt{no\\_grad}\left(\mathbf{\widetilde{a}}_{m}\right),\texttt{no\\_grad}\left(\mathbf{\widetilde{a}}_{l}\right))$
$\displaystyle\propto\exp\left(-\gamma\,||\texttt{no\\_grad}\left(\mathbf{\widetilde{a}}_{m}\right)-\texttt{no\\_grad}\left(\mathbf{\widetilde{a}}_{l}\right)||_{2}^{2}\right)$
(23)

with $||\cdot||_{2}$ as Euclidean distance. Typically, the kernel scale
$\gamma$ needs to fit the observed data, i.e., annotator embeddings in our
case. Therefore, its definition a priori is challenging, such that we define
$\gamma$ as a learnable parameter subject to a prior distribution. Concretely,
we employ the gamma distribution for this purpose:

$\displaystyle\Pr\left(\gamma\mid\alpha,\beta\right)\coloneqq\mathrm{Gam}\left(\gamma\mid\alpha,\beta\right)\coloneqq\frac{\beta^{\alpha}}{\Gamma(\alpha)}\gamma^{\alpha-1}\exp\left(-\beta\gamma\right),$
(24)

where $\Gamma$ is the gamma function and
$\alpha\in\mathbb{R}_{>1},\beta\in\mathbb{R}_{>0}$ are hyperparameters. Based
on experiments, we set $\alpha=1.25,\beta=0.25$ such that the mode is
$\nicefrac{{(\alpha-1)}}{{\beta}}=1$ (defining the initial value of $\gamma$
before optimization) and the variance with
$\nicefrac{{\alpha}}{{\beta^{2}}}=20$ is high in favor of flexible learning.
As a weighted loss function, we finally get

$\displaystyle
L_{\mathbf{X},\mathbf{A},\mathbf{Z},\alpha,\beta}(\bm{\theta},\bm{\omega},\gamma)$
$\displaystyle\coloneqq-\frac{1}{|\mathbf{Z}|}\sum_{n=1}^{N}\sum_{m\in\mathcal{A}_{n}}\left(\hat{w}_{\gamma}(\mathbf{a}_{m})\ln\left(\mathbf{e}^{\mathrm{T}}_{z_{nm}}\mathbf{\hat{P}}^{\mathrm{T}}_{\bm{\omega}}(\mathbf{x}_{n},\mathbf{a}_{m})\mathbf{\hat{p}}_{\bm{\theta}}(\mathbf{x}_{n})\right)\right)-\ln\left(\mathrm{Gam}\left(\gamma\mid\alpha,\beta\right)\right),$
(25) $\displaystyle|\mathbf{Z}|$
$\displaystyle\coloneqq\sum_{n=1}^{N}\sum_{m=1}^{M}\delta(z_{nm}\in\Omega_{Y}),$
(26)

where $\hat{w}_{\gamma}(\mathbf{a}_{m})$ denotes that the annotator weights
$w(\mathbf{a}_{m})$ are estimated by learning the kernel scale $\gamma$. The
number of annotations $|\mathbf{Z}|$ is a normalization factor, which accounts
for potentially unevenly distributed annotations across mini-batches when
using stochastic GD.

Given the loss function in Eq. 25, we present the complete end-to-end training
algorithm of MaDL in Algorithm 1 and an example in Appendix B. During each
training step, we recompute the annotator weights and use them as the basis
for the weighted loss function to optimize the AP and GT models’ parameters.
After training, the optimized model parameters $(\bm{\theta},\bm{\omega})$ can
be used to make probabilistic predictions, e.g., class-membership
probabilities $\mathbf{\hat{p}}_{\bm{\theta}}(\mathbf{x})$ (cf. Fig. 3) and
annotator confusion matrix
$\mathbf{\hat{P}}_{\bm{\omega}}(\mathbf{x},\mathbf{a})$ (cf. Fig. 3), or to
decide on distinct labels, e.g., class label
$\hat{y}_{\bm{\theta}}(\mathbf{x})$ (cf. Eq. 8) and annotation error
$\hat{y}_{\bm{\theta},\bm{\omega}}(\mathbf{x},\mathbf{a})$ (cf. Eq. 9).

input: instances $\mathbf{X}$, annotators $\mathbf{A}$, annotations
$\mathbf{Z}$, number of training epochs $E$, mini-batch size $B$, initial
model parameters $(\bm{\theta},\bm{\omega})$, prior annotation accuracy
$\eta$, gamma distribution parameters $(\alpha,\beta)$;

start: initialize biases $\mathbf{B}$ of the AP model’s output layer using
$\eta$ (cf. Eq. 19);

start: initialize kernel scale
$\gamma\coloneqq\nicefrac{{(\alpha-1)}}{{\beta}}$ ;

for epoch $e\in\\{1,\dots,E\\}$ do

for sampled mini-batch
$\mathbf{\overline{X}}\coloneqq(\mathbf{x}_{i_{1}},\dots,\mathbf{x}_{i_{B}})^{\mathrm{T}},\mathbf{\overline{Z}}\coloneqq(\mathbf{z}_{i_{1}},\dots,\mathbf{z}_{i_{B}})^{\mathrm{T}}$
with $\\{i_{1},\dots,i_{B}\\}\subset\\{1,\dots,N\\}$ do

for $b\in\\{i_{1},\dots,i_{B}\\}$ do

compute class-membership probabilities
$\mathbf{\hat{p}}_{\bm{\theta}}(\mathbf{x}_{b})$ (cf. Fig. 3);

for $m\in\\{1,\dots,M\\}$ do compute confusion matrix
$\mathbf{\hat{P}}_{\bm{\omega}}(\mathbf{x}_{b},\mathbf{a}_{m})$ (cf. Fig. 3);
end

end for

for $(m,l)\in\\{1,\dots,M\\}^{2}$ do compute similarity
$k_{\gamma}(\texttt{no\\_grad}(\mathbf{\widetilde{a}}_{m}),\texttt{no\\_grad}(\mathbf{\widetilde{a}}_{l}))$
(cf. Eq. 23); end

for $m\in\\{1,\dots,M\\}$ do compute annotator weight
$w(\mathbf{a}_{m})\approx\hat{w}_{\gamma}(\mathbf{a}_{m})$ (cf. Eq. 21 and Eq.
22); end

optimize parameters $\bm{\theta},\bm{\omega},\gamma$ with reference to
$L_{\mathbf{\overline{X}},\mathbf{A},\mathbf{\overline{Z}},\alpha,\beta}(\bm{\theta},\bm{\omega},\gamma)$
(cf. Eq. 25);

end for

end for

output: optimized model parameters $(\bm{\theta},\bm{\omega})$

Algorithm 1 End-to-end training algorithm of MaDL.

##  5 Experimental Evaluation

This section investigates three RQs regarding the properties P1–P6 (cf.
Section 3) of multi-annotator supervised learning. We divide the analysis of
each RQ into four parts, which are (1) a takeaway summarizing the key
insights, (2) a setup describing the experiments, (3) a qualitative study, and
(4) a quantitative study. The qualitative studies intuitively explain our
design choices about MaDL, while the quantitative studies compare MaDL’s
performance to related techniques. Note that we analyze each RQ in the context
of a concrete evaluation scenario. Accordingly, the results provide potential
indications for an extension to related scenarios. As this section’s starting
point, we overview the general experimental setup, whose code base is publicly
available at https://www.github.com/ies-research/multi-annotator-deep-
learning.

###  5.1 Experimental Setup

We base our experimental setup on the problem setting in Section 2.
Accordingly, the goal is to evaluate the predictions of GT and AP models
trained via multi-annotator supervised learning techniques. For this purpose,
we perform experiments on several datasets with class labels provided by
error-prone annotators, with models of varying hyperparameters, and in
combination with a collection of different evaluation scores.

Datasets: We conduct experiments for the tabular and image datasets listed by
Table 2. labelme and music are actual crowdsourcing datasets, while we
simulate annotators for the other five datasets. For the labelme dataset,
Rodrigues & Pereira (2018) performed a crowdsourcing study to annotate a
subset of 1000 out of 2688 instances of eight different classes as training
data. This dataset consists of images, but due to its small training set size,
we follow the idea of Rodrigues & Pereira and transform it into a tabular
dataset by utilizing the features of a pretrained VGG-16 (Simonyan &
Zisserman, 2015) as inputs. There are class labels obtained from 59 different
annotators, and on average, about 2.5 class labels are assigned to an
instance. music is another crowdsourcing dataset, where 700 of 1000 audio
files are classified into ten music genres by 44 annotators, and on average,
about 2.9 class labels are assigned to a file. We use the features extracted
by Rodrigues et al. (2013) from the audio files for training and inference.
The artificial toy dataset with two classes and features serves to visualize
our design choices about MaDL. We generate this dataset via a Gaussian mixture
model. Frey & Slate (1991) published the letter dataset to recognize a pixel
display, represented through statistical moments and edge counts, as one of
the 26 capital letters in the alphabet for Modern English. The datasets
fmnist, cifar10, and svhn represent typical image benchmark classification
tasks, each with ten classes but different object types to recognize. Appendix
F presents a separate case study on cifar100 to investigate the outcomes on
datasets with more classes.

Table 2: Overview of datasets and associated base network architectures. Dataset | Annotators | Instances | Classes | Features | Base Network Architecture
---|---|---|---|---|---
Tabular Datasets
toy | simulated | 500 | 2 | 2 |  MLP (Rodrigues & Pereira, 2018)
letter (Frey & Slate, 1991) | simulated | 20000 | 26 | 16 |  MLP (Rodrigues & Pereira, 2018)
labelme (Rodrigues & Pereira, 2018) | real-world | 2688 | 8 | 8192 |  MLP (Rodrigues & Pereira, 2018)
music (Rodrigues et al., 2013) | real-world | 1000 | 10 | 124 |  MLP (Rodrigues & Pereira, 2018)
Image Datasets
fmnist (Xiao et al., 2017) | simulated | 70000 | 10 |  1 $\times$ 28 $\times$ 28 |  LeNet-5 (LeCun & Cortes, 1998)
cifar10 (Krizhevsky, 2009) | simulated | 60000 | 10 |  3 $\times$ 32 $\times$ 32 |  ResNet-18 (He et al., 2016)
svhn (Netzer et al., 2011) | simulated | 99289 | 10 |  3 $\times$ 32 $\times$ 32 |  ResNet-18 (He et al., 2016)

Network Architectures: Table 2 lists the base network architectures selected
to meet the datasets’ requirements. These architectures are starting points
for designing the GT and AP models, which we adjust according to the
respective multi-annotator supervised learning technique. For the tabular
datasets, we follow Rodrigues & Pereira (2018) and train a multilayer
perceptron (MLP) with a single fully connected layer of 128 neurons as a
hidden layer. A modified LeNet-5 architecture (LeCun & Cortes, 1998), a simple
convolutional neural network, serves as the basis for fmnist as a gray-scale
image dataset, while we employ a ResNet-18 (He et al., 2016) for cifar10 and
svhn as RGB image datasets. We refer to our code base for remaining details,
e.g., on the use of rectified linear units (ReLU, Glorot et al. 2011) as
activation functions.

Annotator simulation: For the datasets without real-world annotators, we adopt
simulation strategies from related work (Yan et al., 2014; Cao et al., 2019;
Rühling Cachay et al., 2021; Wei et al., 2022) and simulate annotators
according to the following five types:

fnum@@desciitemAdversarial

annotators provide false class labels on purpose. In our case, such an
annotator provides a correct class label with a probability of 0.05.

fnum@@desciitemRandomly guessing

annotators provide class labels drawn from a uniform categorical distribution.
As a result, such an annotator provides a correct class label with a
probability of $\nicefrac{{1}}{{C}}$.

fnum@@desciitemCluster-specialized

annotators’ performances considerably vary across the clusters found by the
$k$-means clustering algorithm. For images, we cluster the latent
representations of the ResNet-18 pretrained on ImageNet (Russakovsky et al.,
2015). In total, there are $k=10$ clusters. For each annotator, we randomly
define five weak and five expert clusters. An annotator provides a correct
class label with a probability of 0.95 for an expert cluster and with a
probability of 0.05 for a weak cluster.

fnum@@desciitemCommon

annotators are simulated based on the identical clustering employed for the
cluster-specialized annotators. However, their APs vary less between the
clusters. Concretely, we randomly draw a correctness probability value in the
range $[\nicefrac{{1}}{{C}},1]$ for each cluster-annotator pair.

fnum@@desciitemClass-specialized

annotators’ performances considerably vary across classes to which instances
can belong. For each annotator, we randomly define
$\lfloor\nicefrac{{C}}{{2}}\rfloor$ weak and $\lceil\nicefrac{{C}}{{2}}\rceil$
expert classes. An annotator provides a correct class label with a probability
of 0.95 for an expert class and with a probability of 0.05 for a weak class.

We simulate annotation mistakes by randomly selecting false class labels.
Table 3 lists four annotator sets (blueish rows) with varying numbers of
annotators per annotator type (first five columns) and annotation ratios (last
column). Each annotator set is associated with a concrete RQ. A copy flag
indicates that the annotators in the respective types provide identical
annotations. This way, we follow Wei et al. (2022), Cao et al. (2019), and
Rühling Cachay et al. (2021) to simulate strong correlations between
annotators. For example, the entry ”1 + 11 copies“ of the annotator set
correlated indicates twelve cluster-specialized annotators, of which one
annotator is independent, while the remaining eleven annotators share
identical annotation patterns, i.e., they are copies of each other. The
simulated annotator correlations are not directly observable because the
copied annotators likely annotate different instances. This is because of the
annotation ratios, e.g., a ratio of 0.2 indicates that each annotator provides
annotations for only $20\text{\,}\mathrm{\char 37\relax}$ of randomly chosen
instances. The annotation ratios are well below 1.0 because, in practice
(especially in crowdsourcing applications), it is unrealistic for every
annotator to annotate every instance. We refer to Appendix E presenting the
results of a case study with higher annotation ratios for cifar10.

Table 3: Simulated annotator sets for each RQ. Adversarial | Common | Cluster-specialized | Class-specialized | Random | Annotation Ratio
---|---|---|---|---|---
independent (RQ1)
1 | 6 | 2 | 1 | 0 | 0.2
correlated (RQ2)
11 copies | 6 | 1 + 11 copies | 11 copies | 0 | 0.2
random-correlated (RQ2)
1 | 6 | 2 | 1 | 90 copies | 0.2
inductive (RQ3)
10 | 60 | 20 | 10 | 0 | 0.02

Evaluation scores: Since we are interested in quantitatively assessing GT and
AP predictions, we need corresponding evaluation scores. In this context, we
interpret the prediction of APs as a binary classification problem with the AP
model predicting whether an annotator provides the correct or a false class
label for an instance. Next to categorical predictions, the GT and AP models
typically provide probabilistic outputs, which we examine regarding their
quality (Huseljic et al., 2021). We list our evaluation scores in the
following, where arrows indicate which scores need to be maximized
($\uparrow$) or minimized ($\downarrow$):

fnum@@desciiitemAccuracy

(ACC, $\uparrow$) is probably the most popular score for assessing
classification performances. For the GT estimates, it describes the fraction
of correctly classified instances, whereas it is the fraction of (potential)
annotations correctly identified as false or correct for the AP estimates:

$\displaystyle\text{GT-
ACC}(\mathbf{X},\mathbf{y},\hat{y}_{\bm{\theta}})\coloneqq\frac{1}{N}\sum_{n=1}^{N}\delta\left(y_{n}=\hat{y}_{\bm{\theta}}(\mathbf{x}_{n})\right),$
(27) $\displaystyle\text{AP-
ACC}(\mathbf{X},\mathbf{y},\mathbf{Z},\hat{y}_{\bm{\theta},\bm{\omega}})\coloneqq\frac{1}{|\mathbf{Z}|}\sum_{n=1}^{N}\sum_{m\in\mathcal{A}_{n}}\delta\left(\delta\left(y_{n}\neq
z_{nm}\right)=\hat{y}_{\bm{\theta},\bm{\omega}}(\mathbf{x}_{n},\mathbf{a}_{m})\right).$
(28)

Maximizing both scores corresponds to the Bayes optimal predictions in Eq. 3
and Eq. 5.

fnum@@desciiitemBalanced accuracy

(BAL-ACC, $\uparrow$) is a variant of ACC designed for imbalanced
classification problems (Brodersen et al., 2010). For the GT estimation, the
idea is to compute the ACC score for each class of instances separately and
then average them. Since our datasets are fairly balanced in their
distributions of class labels, we use this evaluation score only for assessing
AP estimates. We may encounter highly imbalanced binary classification
problems per annotator, where a class represents either a false or correct
annotation. For example, an adversarial annotator provides majorly false
annotations. Therefore, we extend the definition of BAL-ACC by computing the
ACC scores for each annotator-class pair separately to average them.

fnum@@desciiitemNegative log-likelihood

(NLL, $\downarrow$) is not only used as a typical loss function for training
(D)NNs but can also be used to assess the quality of probabilistic estimates:

$\displaystyle\text{GT-
NLL}(\mathbf{X},\mathbf{y},\mathbf{\hat{p}}_{\bm{\theta}})\coloneqq-\frac{1}{N}\sum_{n=1}^{N}\ln\left(\hat{p}^{(y_{n})}_{\bm{\theta}}(\mathbf{x}_{n})\right),$
(29) $\displaystyle\text{AP-
NLL}(\mathbf{X},\mathbf{y},\mathbf{Z},\hat{p}_{\bm{\theta},\bm{\omega}})\coloneqq$
$\displaystyle-\frac{1}{|\mathbf{Z}|}\sum_{n=1}^{N}\sum_{m\in\mathcal{A}_{n}}\Big{(}\delta\left(y_{n}=z_{nm}\right)\ln\left(\hat{p}_{\bm{\theta},\bm{\omega}}(\mathbf{x}_{n},\mathbf{a}_{m})\right)+\delta\left(y_{n}\neq
z_{nm}\right)\ln\left(1-\hat{p}_{\bm{\theta},\bm{\omega}}(\mathbf{x}_{n},\mathbf{a}_{m})\right)\Big{)}.$
(30)

Moreover, NLL is a proper scoring rule (Ovadia et al., 2019) such that the
best score corresponds to a perfect prediction.

fnum@@desciiitemBrier score

(BS, $\downarrow$), proposed by Brier (1950), is another proper scoring rule,
which measures the squared error between predicted probability vectors and
one-hot encoded target vectors:

$\displaystyle\text{GT-
BS}(\mathbf{X},\mathbf{y},\mathbf{\hat{p}}_{\bm{\theta}})\coloneqq\frac{1}{N}\sum_{n=1}^{N}||\mathbf{e}_{y_{n}}-\mathbf{\hat{p}}_{\bm{\theta}}(\mathbf{x}_{n})||_{2}^{2},$
(31) $\displaystyle\text{AP-
BS}(\mathbf{X},\mathbf{y},\mathbf{Z},\hat{p}_{\bm{\theta},\bm{\omega}})\coloneqq\frac{1}{|\mathbf{Z}|}\sum_{n=1}^{N}\sum_{m\in\mathcal{A}_{n}}\left(\delta\left(y_{n}=z_{nm}\right)-\hat{p}_{\bm{\theta},\bm{\omega}}(\mathbf{x}_{n},\mathbf{a}_{m})\right)^{2}.$
(32)

In the literature, there exist many further evaluation scores, particularly
for assessing probability calibration (Ovadia et al., 2019). As a
comprehensive evaluation of probabilities is beyond this article’s scope, we
focus on the aforementioned proper scoring rules. Accordingly, we have omitted
other evaluation scores, such as the expected calibration error (Naeini et
al., 2015) being a non-proper scoring rule.

Multi-annotator supervised learning techniques: By default, we train MaDL via
the weighted loss function in Eq. 25 using the hyperparameter values from
Section 4 and the most general architecture depicted by Fig. 3. Next to the
ablations as part of analyzing the three RQs, we present an ablation study on
the hyperparameters of MaDL in Appendix C and a practitioner’s guide with
concrete recommendations in Appendix G. We evaluate MaDL compared to a subset
of the related techniques presented in Section 3. This subset consists of
techniques that (1) provide probabilistic GT estimates for each instance, (2)
provide probabilistic AP estimates for each instance-annotator pair, and (3)
train a (D)NN as the GT model. Moreover, we focus on recent techniques with
varying training algorithms and properties P1–P6 (cf. Section 3). As a result,
we select crowd layer (CL, Rodrigues & Pereira, 2018), regularized estimation
of annotator confusion (REAC, Tanno et al., 2019), learning from imperfect
annotators (LIA, Platanios et al., 2020), common noise adaption layers (CoNAL,
Chu et al., 2021), and union net (UNION, Wei et al., 2022). Further, we
aggregate annotations through the majority rule as a lower baseline (LB) and
use the GT class labels as an upper baseline (UB). We adopt the architectures
of MaDL’s GT and AP models for both baselines. The GT model then trains via
the aggregated annotation (LB) or the GT class labels (UB). The AP model
trains using the aggregated annotations (LB) or the GT class labels (UB) to
optimize the annotator confusion matrices. Unless explicitly stated, no multi-
annotator supervised learning technique can access annotator features
containing prior knowledge about the annotators.

Experiment: An experiment’s run starts by splitting a dataset into train,
validation, and test sets. For music and labelme, these splits are predefined,
while for the other datasets, we randomly select $75\text{\,}\mathrm{\char
37\relax}$ of the samples for training, $5\text{\,}\mathrm{\char 37\relax}$
for validation, and $20\text{\,}\mathrm{\char 37\relax}$ for testing.
Following Rühling Cachay et al. (2021), a small validation set with GT class
labels allows a fair comparison by finding suitable hyperparameter values for
the optimizer of the respective multi-annotator supervised learning technique.
We employ the AdamW (Loshchilov & Hutter, 2019) optimizer, where the learning
rates $\\{0.01,0.005,0.001\\}$ and weight decays $\\{0.0,0.001,0.0001\\}$ are
tested. We decay learning rates via a cosine annealing schedule (Loshchilov &
Hutter, 2017) and set the optimizer’s mini-batch size to 64. For the datasets
music and labelme, we additionally perform experiments with 8 and 16 as mini-
batch sizes due to their smaller number of instances and, thus, higher
sensitivity to the mini-batch size. The number of training epochs is set to
100 for all techniques except for LIA, which we train for 200 epochs due to
its EM algorithm. After training, we select the models with the best
validation GT-ACC across the epochs. Each experiment is run five times with
different parameter initializations and data splits (except for labelme and
music). We report quantitative results as means and standard deviations over
the best five runs determined via the validation GT-ACC.

###  5.2 RQ1: Do class- and instance-dependent modeled APs improve learning?
(Properties P1, P2)

Takeaway: Estimating class- (property P1) and instance-dependent (property P2)
APs leads to superior performances of the GT and AP models. This observation
is especially true for GT models trained on datasets with real-world
annotators whose annotation patterns are unknown.

Setup: We address RQ1 by evaluating multi-annotator supervised learning
techniques with varying AP assumptions. We simulate ten annotators for the
datasets without real-world annotators according to the annotator set
independent in Table 3. Each simulated annotator provides class labels for
$20\text{\,}\mathrm{\char 37\relax}$ of randomly selected training instances.
Next to the related multi-annotator supervised learning techniques and the two
baselines, we evaluate six variants of MaDL denoted via the scheme MaDL(P1,
P2). Property P1 refers to the estimation of potential class-dependent APs.
There, we differentiate between the options class-independent (I), partially
(P) class-dependent, and fully (F) class-dependent APs. We implement them by
constraining the annotator confusion matrices’ degrees of freedom. Concretely,
class-independent refers to a confusion matrix approximated by estimating a
single scalar, partially class-dependent refers to a confusion matrix
approximated by estimating its diagonal elements, and fully class-dependent
refers to estimating each matrix element individually (cf. Appendix G).
Property P2 indicates whether the APs are estimated as a function of instances
(X) or not ($\overline{\text{X}}$). Combining the two options of the
properties P1 and P2 represents one variant. For example, MaDL(X, F) is the
default MaDL variant estimating instance- and fully class-dependent APs.

Qualitative study: Fig. 6 visualizes MaDL’s predictive behavior for the
artificial dataset toy. Thereby, each row represents the predictions of a
different MaDL variant. Since this is a binary classification problem, the
variant MaDL(X, P) is identical to MaDL(X, F), and MaDL($\overline{\text{X}}$,
P) is identical to MaDL($\overline{\text{X}}$, F). The first column visualizes
instances as circles colored according to their GT labels, plots the class-
membership probabilities predicted by the respective GT model as contours
across the feature space, and depicts the decision boundary for classification
as a black line. The last four columns show the class labels provided by four
of the ten simulated annotators. The instances’ colors indicate the class
labels provided by an annotator, their forms mark whether the class labels are
correct (circle) or false (cross) annotations, and the contours across the
feature space visualize the AP model’s predicted annotation correctness
probabilities. The GT models of the variants MaDL($\overline{\text{X}}$, F),
MaDL(X, I), and MaDL(X, F) successfully separate the instances of both
classes, whereas the GT model of MaDL($\overline{\text{X}}$, I) fails in this
task. Likely, the missing consideration of instance- and class-dependent APs
explains this observation. Further, the class-membership probabilities of the
successful MaDL variants reflect instances’ actual class labels but exhibit
the overconfident behavior typical of deterministic (D)NNs, particularly for
feature space regions without observed instances (Huseljic et al., 2021).
Investigating the estimated APs for the adversarial annotator (second column),
we see that each MaDL variant correctly predicts low APs (indicated by the
white-colored contours) across the feature space. When comparing the AP
estimates for the class-specialized annotator (fifth column), clear
differences between MaDL($\overline{\text{X}}$, I) and the other three
variants of MaDL are visible. Since MaDL($\overline{\text{X}}$, I) ignores any
class dependency regarding APs, it cannot differentiate between classes of
high and low APs. In contrast, the AP predictions of the other three variants
reflect the class structure learned by the respective GT model and thus can
separate between weak and expert classes. The performances of the cluster-
specialized and common annotator depend on the regions in the feature space.
Therefore, only the variants MaDL(X, I) and MaDL(X, F) can separate clusters
of low and high APs. For example, both variants successfully identify the two
weak clusters of the cluster-specialized annotator. Analogous to the class-
membership probabilities, the AP estimates are overconfident for feature space
regions without observed instances.

MaDL($\overline{\text{X}}$, I)

MaDL($\overline{\text{X}}$, F)

MaDL(X, I)

MaDL(X, F)

ClassificationAdversarialAnnotatorCluster-
specializedAnnotatorCommonAnnotatorClass-specializedAnnotator Figure 6:
Visualization of MaDL’s predictive behavior for the two-dimensional dataset
toy.

Quantitative study: Table 4 presents the numerical evaluation results for the
two datasets with real-world annotators. There, we only report the GT models’
test results since no annotations for the test instances are available to
assess the AP models’ test performances. Table 5 presents the GT and AP
models’ test results for the four datasets with simulated annotators. Both
tables indicate whether a technique models class-dependent (property P1)
and/or instance-dependent (property P2) APs. Generally, training with GT
labels as UB achieves the best performances, while the LB with annotations
aggregated according to the majority rule leads to the worst ones. The latter
observation confirms that leveraging AP estimates during training is
beneficial. Moreover, these AP estimates are typically meaningful,
corresponding to BAL-ACC values above 0.5. An exception is
MaDL($\overline{\text{X}}$, I) because this variant only estimates by design a
constant performance per annotator across the feature space. Comparing MaDL(X,
F) as the most general variant to related techniques, we observe that it
achieves competitive or superior results for all datasets and evaluation
scores. Next to MaDL(X, F), CoNAL often delivers better results than the
competitors. When we investigate the performances of the MaDL variants with
instance-independent APs, we find that MaDL($\overline{\text{X}}$, F) achieves
the most robust performances across all datasets. In particular, for the
datasets with real-world annotators, the ACC of the respective GT model is
superior. This observation suggests that modeling class-dependent APs
(property P1) is beneficial. We recognize a similar trend for the MaDL
variants with instance-dependent APs (property P2). Comparing each pair of
MaDL variants with X and $\overline{\text{X}}$, we observe that instance-
dependent APs often improve GT and, in particular, AP estimates. The advantage
of class- and instance-dependent APs is confirmed by CoNAL as a strong
competitor of MaDL(X, F). LIA’s inferior performance contrasts this, although
LIA estimates class- and instance-dependent APs. The difference in training
algorithms can likely explain this observation. While MaDL(X, F) and CoNAL
train via an end-to-end algorithm, LIA trains via the EM algorithm, leading to
higher runtimes and introducing additional sensitive hyperparameters, e.g.,
the number of EM iterations and training epochs per M step.

Table 4:  Results regarding RQ1 for datasets with real-world annotators:

Best

and

second best

performances are highlighted per dataset and evaluation score while excluding the performances of the UB.  Technique | P1 | P2 | Ground Truth Model | Ground Truth Model
---|---|---|---|---
ACC $\uparrow$ |  NLL $\downarrow$ |  BS $\downarrow$ |  ACC $\uparrow$ |  NLL $\downarrow$ |  BS $\downarrow$
| music | labelme
UB | ✓ | ✓ | 0.785$\pm$0.020 | 0.710$\pm$0.037 | 0.314$\pm$0.027 | 0.914$\pm$0.003 | 0.580$\pm$0.112 | 0.150$\pm$0.003
LB | ✓ | ✓ |  0.646$\pm$0.045 |  1.096$\pm$0.103 |  0.492$\pm$0.051 |  0.810$\pm$0.015 |  0.724$\pm$0.155 |  0.294$\pm$0.024
CL | ✓ | ✗ |  0.675$\pm$0.015 |  1.672$\pm$0.400 |  0.524$\pm$0.021 |  0.857$\pm$0.011 |  1.774$\pm$1.155 |  0.250$\pm$0.014
REAC | ✓ | ✗ |  0.705$\pm$0.023 |  0.893$\pm$0.081 |  0.410$\pm$0.033 |  0.843$\pm$0.006 |  0.833$\pm$0.088 |  0.254$\pm$0.006
UNION | ✓ | ✗ |  0.682$\pm$0.013 |  1.396$\pm$0.143 |  0.501$\pm$0.027 |  0.855$\pm$0.004 |  1.074$\pm$0.340 |  0.248$\pm$0.011
LIA | ✓ | ✓ |  0.658$\pm$0.023 |  1.158$\pm$0.047 |  0.498$\pm$0.020 |  0.813$\pm$0.010 |  0.976$\pm$0.234 |  0.295$\pm$0.009
CoNAL | ✓ | ✓ |  0.708$\pm$0.031 |  0.964$\pm$0.081 |  0.423$\pm$0.035 |  0.866$\pm$0.004 |  2.740$\pm$1.304 |  0.247$\pm$0.023
MaDL($\overline{\text{X}}$, I) | ✗ | ✗ |  0.718$\pm$0.010 |  0.871$\pm$0.027 |  0.394$\pm$0.009 |  0.815$\pm$0.009 |  0.616$\pm$0.125 |  0.276$\pm$0.017
MaDL($\overline{\text{X}}$, P) | ✦ | ✗ |  0.720$\pm$0.018 |  0.871$\pm$0.030 |  0.396$\pm$0.009 |  0.811$\pm$0.012 |  0.630$\pm$0.128 |  0.281$\pm$0.022
MaDL($\overline{\text{X}}$, F) | ✓ | ✗ |  0.725$\pm$0.015 |  0.977$\pm$0.064 |  0.403$\pm$0.019 |  0.859$\pm$0.007 |  1.008$\pm$0.278 |  0.240$\pm$0.014
MaDL(X, I) | ✗ | ✓ |  0.713$\pm$0.027 |  0.876$\pm$0.041 |  0.402$\pm$0.022 |  0.816$\pm$0.008 |  0.559$\pm$0.027 |  0.276$\pm$0.010
MaDL(X, P) | ✦ | ✓ |  0.714$\pm$0.014 |  0.909$\pm$0.036 |  0.398$\pm$0.013 |  0.811$\pm$0.009 |  0.771$\pm$0.160 |  0.289$\pm$0.016
MaDL(X, F) | ✓ | ✓ |  0.743$\pm$0.018 |  0.877$\pm$0.030 |  0.381$\pm$0.012 |  0.867$\pm$0.004 |  0.623$\pm$0.124 |  0.214$\pm$0.008
Table 5:  Results regarding RQ1 for datasets with simulated annotators:

Best

and

second best

performances are highlighted per dataset and evaluation score while excluding the performances of the UB.  Technique | P1 | P2 | Ground Truth Model | Annotator Performance Model
---|---|---|---|---
ACC $\uparrow$ |  NLL $\downarrow$ |  BS $\downarrow$ |  ACC $\uparrow$ |  NLL $\downarrow$ |  BS $\downarrow$ |  BAL-ACC $\uparrow$
letter (independent)
UB | ✓ | ✓ | 0.961$\pm$0.003 | 0.130$\pm$0.006 | 0.059$\pm$0.004 | 0.770$\pm$0.001 | 0.488$\pm$0.003 | 0.315$\pm$0.002 | 0.709$\pm$0.001
LB | ✓ | ✓ |  0.878$\pm$0.004 |  0.980$\pm$0.021 |  0.385$\pm$0.008 |  0.664$\pm$0.004 |  0.624$\pm$0.003 |  0.433$\pm$0.003 |  0.666$\pm$0.004
CL | ✓ | ✗ |  0.886$\pm$0.013 |  1.062$\pm$0.145 |  0.181$\pm$0.020 |  0.663$\pm$0.006 |  0.625$\pm$0.013 |  0.430$\pm$0.010 |  0.601$\pm$0.002
REAC | ✓ | ✗ |  0.936$\pm$0.005 |  0.238$\pm$0.018 |  0.097$\pm$0.007 |  0.685$\pm$0.002 |  0.560$\pm$0.001 |  0.385$\pm$0.001 |  0.604$\pm$0.002
UNION | ✓ | ✗ |  0.905$\pm$0.016 |  0.906$\pm$0.435 |  0.151$\pm$0.030 |  0.670$\pm$0.004 |  0.589$\pm$0.008 |  0.408$\pm$0.006 |  0.605$\pm$0.002
LIA | ✓ | ✓ |  0.897$\pm$0.005 |  0.778$\pm$0.052 |  0.305$\pm$0.021 |  0.669$\pm$0.004 |  0.654$\pm$0.010 |  0.447$\pm$0.004 |  0.616$\pm$0.003
CoNAL | ✓ | ✓ |  0.907$\pm$0.016 |  0.813$\pm$0.354 |  0.143$\pm$0.027 |  0.723$\pm$0.018 |  0.555$\pm$0.024 |  0.372$\pm$0.020 |  0.663$\pm$0.017
MaDL($\overline{\text{X}}$, I) | ✗ | ✗ |  0.934$\pm$0.003 |  0.269$\pm$0.035 |  0.100$\pm$0.004 |  0.607$\pm$0.001 |  0.627$\pm$0.000 |  0.444$\pm$0.000 |  0.500$\pm$0.000
MaDL($\overline{\text{X}}$, P) | ✦ | ✗ |  0.935$\pm$0.005 |  0.235$\pm$0.013 |  0.099$\pm$0.006 |  0.692$\pm$0.001 |  0.556$\pm$0.001 |  0.381$\pm$0.001 |  0.606$\pm$0.003
MaDL($\overline{\text{X}}$, F) | ✓ | ✗ |  0.933$\pm$0.005 |  0.255$\pm$0.025 |  0.100$\pm$0.005 |  0.691$\pm$0.002 |  0.556$\pm$0.001 |  0.381$\pm$0.001 |  0.606$\pm$0.002
MaDL(X, I) | ✗ | ✓ |  0.938$\pm$0.006 |  0.247$\pm$0.043 |  0.092$\pm$0.008 |  0.770$\pm$0.004 |  0.492$\pm$0.016 |  0.316$\pm$0.007 |  0.708$\pm$0.004
MaDL(X, P) | ✦ | ✓ |  0.940$\pm$0.004 |  0.242$\pm$0.045 |  0.090$\pm$0.004 |  0.770$\pm$0.006 |  0.496$\pm$0.020 |  0.316$\pm$0.009 |  0.708$\pm$0.005
MaDL(X, F) | ✓ | ✓ |  0.935$\pm$0.006 |  0.303$\pm$0.092 |  0.098$\pm$0.009 |  0.766$\pm$0.004 |  0.491$\pm$0.006 |  0.317$\pm$0.004 |  0.702$\pm$0.005
fmnist (independent)
UB | ✓ | ✓ | 0.909$\pm$0.002 | 0.246$\pm$0.005 | 0.131$\pm$0.003 | 0.756$\pm$0.001 | 0.485$\pm$0.001 | 0.321$\pm$0.001 | 0.704$\pm$0.001
LB | ✓ | ✓ |  0.883$\pm$0.001 |  0.903$\pm$0.003 |  0.385$\pm$0.001 |  0.644$\pm$0.007 |  0.645$\pm$0.005 |  0.453$\pm$0.004 |  0.585$\pm$0.007
CL | ✓ | ✗ |  0.892$\pm$0.002 |  0.312$\pm$0.008 |  0.158$\pm$0.004 |  0.674$\pm$0.002 |  0.580$\pm$0.001 |  0.402$\pm$0.001 |  0.623$\pm$0.001
REAC | ✓ | ✗ |  0.894$\pm$0.003 |  0.309$\pm$0.011 |  0.155$\pm$0.004 |  0.703$\pm$0.001 |  0.535$\pm$0.001 |  0.364$\pm$0.000 |  0.641$\pm$0.001
UNION | ✓ | ✗ |  0.893$\pm$0.002 |  0.305$\pm$0.006 |  0.155$\pm$0.003 |  0.674$\pm$0.002 |  0.570$\pm$0.002 |  0.395$\pm$0.002 |  0.622$\pm$0.001
LIA | ✓ | ✓ |  0.858$\pm$0.002 |  1.017$\pm$0.016 |  0.442$\pm$0.008 |  0.665$\pm$0.024 |  0.628$\pm$0.017 |  0.437$\pm$0.016 |  0.613$\pm$0.027
CoNAL | ✓ | ✓ |  0.894$\pm$0.004 |  0.304$\pm$0.009 |  0.155$\pm$0.004 |  0.725$\pm$0.016 |  0.521$\pm$0.018 |  0.351$\pm$0.016 |  0.679$\pm$0.018
MaDL($\overline{\text{X}}$, I) | ✗ | ✗ |  0.896$\pm$0.003 |  0.340$\pm$0.006 |  0.161$\pm$0.004 |  0.590$\pm$0.000 |  0.638$\pm$0.000 |  0.453$\pm$0.000 |  0.500$\pm$0.000
MaDL($\overline{\text{X}}$, P) | ✦ | ✗ |  0.894$\pm$0.001 |  0.307$\pm$0.003 |  0.155$\pm$0.001 |  0.705$\pm$0.001 |  0.534$\pm$0.000 |  0.363$\pm$0.000 |  0.640$\pm$0.001
MaDL($\overline{\text{X}}$, F) | ✓ | ✗ |  0.894$\pm$0.002 |  0.307$\pm$0.006 |  0.155$\pm$0.003 |  0.705$\pm$0.000 |  0.534$\pm$0.000 |  0.363$\pm$0.000 |  0.640$\pm$0.000
MaDL(X, I) | ✗ | ✓ |  0.895$\pm$0.003 |  0.291$\pm$0.005 |  0.150$\pm$0.003 |  0.752$\pm$0.004 |  0.490$\pm$0.004 |  0.325$\pm$0.003 |  0.699$\pm$0.004
MaDL(X, P) | ✦ | ✓ |  0.899$\pm$0.003 |  0.286$\pm$0.006 |  0.147$\pm$0.003 |  0.751$\pm$0.003 |  0.489$\pm$0.004 |  0.324$\pm$0.003 |  0.698$\pm$0.005
MaDL(X, F) | ✓ | ✓ |  0.896$\pm$0.002 |  0.288$\pm$0.006 |  0.148$\pm$0.003 |  0.750$\pm$0.005 |  0.491$\pm$0.005 |  0.326$\pm$0.005 |  0.697$\pm$0.006
cifar10 (independent)
UB | ✓ | ✓ | 0.933$\pm$0.002 | 0.519$\pm$0.026 | 0.118$\pm$0.004 | 0.710$\pm$0.001 | 0.547$\pm$0.001 | 0.369$\pm$0.001 | 0.658$\pm$0.001
LB | ✓ | ✓ |  0.789$\pm$0.004 |  1.081$\pm$0.031 |  0.460$\pm$0.015 |  0.575$\pm$0.021 |  0.673$\pm$0.006 |  0.481$\pm$0.006 |  0.547$\pm$0.011
CL | ✓ | ✗ |  0.833$\pm$0.003 |  0.536$\pm$0.012 |  0.242$\pm$0.004 |  0.664$\pm$0.001 |  0.604$\pm$0.002 |  0.420$\pm$0.001 |  0.613$\pm$0.001
REAC | ✓ | ✗ |  0.839$\pm$0.003 |  0.581$\pm$0.010 |  0.245$\pm$0.003 |  0.676$\pm$0.003 |  0.580$\pm$0.006 |  0.397$\pm$0.004 |  0.625$\pm$0.002
UNION | ✓ | ✗ |  0.834$\pm$0.003 |  0.595$\pm$0.022 |  0.249$\pm$0.005 |  0.668$\pm$0.001 |  0.592$\pm$0.001 |  0.410$\pm$0.001 |  0.617$\pm$0.002
LIA | ✓ | ✓ |  0.805$\pm$0.003 |  1.102$\pm$0.035 |  0.469$\pm$0.016 |  0.622$\pm$0.024 |  0.645$\pm$0.014 |  0.453$\pm$0.014 |  0.579$\pm$0.019
CoNAL | ✓ | ✓ |  0.838$\pm$0.005 |  0.530$\pm$0.021 |  0.236$\pm$0.008 |  0.668$\pm$0.001 |  0.600$\pm$0.001 |  0.416$\pm$0.001 |  0.616$\pm$0.001
MaDL($\overline{\text{X}}$, I) | ✗ | ✗ |  0.832$\pm$0.006 |  0.583$\pm$0.021 |  0.256$\pm$0.009 |  0.576$\pm$0.010 |  0.646$\pm$0.002 |  0.461$\pm$0.002 |  0.500$\pm$0.000
MaDL($\overline{\text{X}}$, P) | ✦ | ✗ |  0.844$\pm$0.004 |  0.529$\pm$0.014 |  0.231$\pm$0.004 |  0.682$\pm$0.001 |  0.568$\pm$0.001 |  0.390$\pm$0.001 |  0.630$\pm$0.002
MaDL($\overline{\text{X}}$, F) | ✓ | ✗ |  0.840$\pm$0.005 |  0.545$\pm$0.019 |  0.237$\pm$0.006 |  0.681$\pm$0.001 |  0.569$\pm$0.002 |  0.390$\pm$0.001 |  0.630$\pm$0.001
MaDL(X, I) | ✗ | ✓ |  0.843$\pm$0.005 |  0.555$\pm$0.024 |  0.236$\pm$0.008 |  0.697$\pm$0.002 |  0.559$\pm$0.005 |  0.380$\pm$0.003 |  0.646$\pm$0.002
MaDL(X, P) | ✦ | ✓ |  0.845$\pm$0.002 |  0.546$\pm$0.027 |  0.232$\pm$0.005 |  0.697$\pm$0.001 |  0.557$\pm$0.002 |  0.380$\pm$0.001 |  0.646$\pm$0.002
MaDL(X, F) | ✓ | ✓ |  0.846$\pm$0.003 |  0.521$\pm$0.014 |  0.229$\pm$0.005 |  0.697$\pm$0.002 |  0.557$\pm$0.004 |  0.379$\pm$0.002 |  0.646$\pm$0.003
svhn (independent)
UB | ✓ | ✓ | 0.965$\pm$0.000 | 0.403$\pm$0.024 | 0.064$\pm$0.001 | 0.675$\pm$0.002 | 0.567$\pm$0.001 | 0.392$\pm$0.001 | 0.590$\pm$0.004
LB | ✓ | ✓ |  0.930$\pm$0.002 |  0.811$\pm$0.030 |  0.332$\pm$0.015 |  0.581$\pm$0.021 |  0.680$\pm$0.008 |  0.487$\pm$0.008 |  0.540$\pm$0.000
CL | ✓ | ✗ |  0.944$\pm$0.001 |  0.237$\pm$0.008 |  0.085$\pm$0.002 |  0.646$\pm$0.001 |  0.598$\pm$0.001 |  0.419$\pm$0.001 |  0.546$\pm$0.001
REAC | ✓ | ✗ |  0.943$\pm$0.001 |  0.278$\pm$0.048 |  0.096$\pm$0.020 |  0.648$\pm$0.006 |  0.593$\pm$0.015 |  0.414$\pm$0.010 |  0.543$\pm$0.000
UNION | ✓ | ✗ |  0.942$\pm$0.002 |  0.250$\pm$0.005 |  0.087$\pm$0.001 |  0.646$\pm$0.001 |  0.594$\pm$0.001 |  0.416$\pm$0.000 |  0.544$\pm$0.001
LIA | ✓ | ✓ |  0.935$\pm$0.002 |  0.809$\pm$0.162 |  0.333$\pm$0.081 |  0.585$\pm$0.016 |  0.667$\pm$0.023 |  0.476$\pm$0.021 |  0.536$\pm$0.004
CoNAL | ✓ | ✓ |  0.944$\pm$0.002 |  0.246$\pm$0.012 |  0.086$\pm$0.002 |  0.688$\pm$0.036 |  0.560$\pm$0.029 |  0.384$\pm$0.026 |  0.602$\pm$0.050
MaDL($\overline{\text{X}}$, I) | ✗ | ✗ |  0.942$\pm$0.003 |  0.253$\pm$0.023 |  0.093$\pm$0.008 |  0.613$\pm$0.003 |  0.630$\pm$0.003 |  0.446$\pm$0.003 |  0.500$\pm$0.000
MaDL($\overline{\text{X}}$, P) | ✦ | ✗ |  0.940$\pm$0.002 |  0.262$\pm$0.011 |  0.091$\pm$0.003 |  0.652$\pm$0.000 |  0.585$\pm$0.000 |  0.408$\pm$0.000 |  0.544$\pm$0.000
MaDL($\overline{\text{X}}$, F) | ✓ | ✗ |  0.940$\pm$0.002 |  0.264$\pm$0.007 |  0.092$\pm$0.002 |  0.652$\pm$0.001 |  0.585$\pm$0.000 |  0.408$\pm$0.000 |  0.543$\pm$0.001
MaDL(X, I) | ✗ | ✓ |  0.944$\pm$0.003 |  0.240$\pm$0.007 |  0.085$\pm$0.003 |  0.665$\pm$0.001 |  0.575$\pm$0.001 |  0.399$\pm$0.001 |  0.565$\pm$0.001
MaDL(X, P) | ✦ | ✓ |  0.945$\pm$0.002 |  0.245$\pm$0.010 |  0.084$\pm$0.004 |  0.669$\pm$0.002 |  0.572$\pm$0.002 |  0.396$\pm$0.002 |  0.573$\pm$0.005
MaDL(X, F) | ✓ | ✓ |  0.943$\pm$0.001 |  0.254$\pm$0.013 |  0.087$\pm$0.002 |  0.668$\pm$0.003 |  0.572$\pm$0.003 |  0.396$\pm$0.003 |  0.570$\pm$0.006

###  5.3 RQ2: Does modeling correlations between (potentially spamming)
annotators improve learning? (Properties P3, P4)

Takeaway: Modeling correlations between annotators leads to better results in
scenarios with many correlated spamming annotators (property P4). Capturing
the correlations of beneficial annotators does not lead to consistently better
results (property P3). However, estimating and leveraging APs during training
becomes more critical in scenarios with correlated annotators.

Setup: We address RQ2 by evaluating multi-annotator supervised learning
techniques with and without modeling annotator correlations. We simulate two
annotator sets for each dataset without real-world annotators according to
Table 3. The first annotator set correlated consists of the same ten
annotators as in RQ1. However, we extend this set by ten additional copies of
the adversarial, the class-specialized, and one of the two cluster-specialized
annotators, so there are 40 annotators. The second annotator set random-
correlated also consists of the same ten annotators as in RQ1 but is extended
by 90 identical randomly guessing annotators. Each simulated annotator
provides class labels for $20\text{\,}\mathrm{\char 37\relax}$ of randomly
selected training instances. Next to the related multi-annotator supervised
learning techniques and the two baselines, we evaluate two variants of MaDL
denoted via the scheme MaDL(P3). Property P3 refers to the modeling of
potential annotator correlations. There, we differentiate between the variant
MaDL(W) using annotator weights via the weighted loss function (cf. Eq. 25)
and the variant MaDL($\overline{\text{W}}$) training via the loss function
without any weights (cf. Eq. 15). MaDL(W) corresponds to MaDL’s default
variant in this setup.

Qualitative study: Fig. 7 visualizes MaDL(W)’s learned annotator embeddings
and weights for the dataset letter with the two annotator sets, correlated and
random-correlated, after five training epochs. Based on MaDL(W)’s learned
kernel function, we create the two scatter plots via multi-dimensional scaling
(Kruskal, 1964) for dimensionality reduction. This way, the annotator
embeddings, originally located in an $(R=16)$-dimensional space, are
transformed into a two-dimensional space, where each circle represents one
annotator embedding. A circle’s color indicates to which annotator group the
embedding belongs. The two bar plots visualize the mean annotator weight of
the different annotator groups, again indicated by their respective color.
Analyzing the scatter plot of the annotator set correlated, we observe that
the annotator embeddings’ latent representations approximately reflect the
annotator groups’ correlations. Concretely, there are four clusters. The
center cluster corresponds to the seven independent annotators, one cluster-
specialized annotator and six common annotators. The three clusters in the
outer area represent the three groups of correlated annotators. The bar plot
confirms our goal to assign lower weights to strongly correlated annotators.
For example, the single independent cluster-specialized annotator has a weight
of 4.06, while the eleven correlated cluster-specialized annotators have a
mean weight of 0.43. We make similar observations for the annotator set
random-correlated. The scatter plot shows that the independent annotators also
form a cluster, separated from the cluster of the large group of correlated,
randomly guessing annotators. The single adversarial annotator belongs to the
cluster of randomly guessing annotators since both groups of annotators make
many annotation errors and thus have highly correlated annotation patterns.
Again, the bar plot confirms that the correlated annotators get low weights.
Moreover, these annotator weights are inversely proportional to the size of a
group of correlated annotators. For example, the 90 randomly guessing
annotators have a similar weight in sum as the single class-specialized
annotator.

correlated

Latent Dimension 2

Latent Dimension 1

Mean Annotator Weight

Annotator Group

random-correlated

Latent Dimension 2

Latent Dimension 1

Mean Annotator Weight

Annotator Group Figure 7: Visualization of MaDL(W)’s learned similarities
between annotator embeddings and associated annotator weights.

Quantitative study: Table 6 presents the GT and AP models’ test performances
for the four datasets with the annotator set correlated and Table 7 for the
annotator set random-correlated. Both tables indicate whether a technique
models correlations between annotators (property P3) and whether the authors
of a technique demonstrated its robustness against spamming annotators
(property P4). Analogous to RQ1, training with GT labels achieves the best
performances (UB), while annotation aggregation via the majority rule leads to
the worst ones (LB). The LB’s significant underperformance confirms the
importance of modeling APs in scenarios with correlated annotators. MaDL(W),
as the default MaDL variant, achieves competitive and often superior results
for all datasets and evaluation scores. In particular, for the annotator set
random-correlated, MaDL(W) outperforms the other techniques, which are
vulnerable to many randomly guessing annotators. This observation is also
confirmed when we compare MaDL(W) to MaDL($\overline{\text{W}}$). In contrast,
there is no consistent performance gain of MaDL(W) over
MaDL($\overline{\text{W}}$) for the annotator set correlated. While CoNAL is
competitive for the annotator set correlated, its performance strongly
degrades for the annotator set random-correlated. The initial E step in LIA’s
EM algorithm estimates the GT class labels via a probabilistic variant of the
majority rule. Similarly to the LB, such an estimate is less accurate for
correlated and/or spamming annotators. Besides MaDL(W), only CL and UNION
consistently outperform the LB by large margins for the annotator set random-
correlated.

Table 6:  Results regarding RQ2 for datasets with simulated annotators:

Best

and

second best

performances are highlighted per dataset and evaluation score while excluding the performances of the UB.  Technique | P3 | P4 | Ground Truth Model | Annotator Performance Model
---|---|---|---|---
ACC $\uparrow$ |  NLL $\downarrow$ |  BS $\downarrow$ |  ACC $\uparrow$ |  NLL $\downarrow$ |  BS $\downarrow$ |  BAL-ACC $\uparrow$
letter (correlated)
UB | ✗ | ✓ | 0.962$\pm$0.004 | 0.129$\pm$0.004 | 0.058$\pm$0.003 | 0.887$\pm$0.002 | 0.305$\pm$0.004 | 0.173$\pm$0.002 | 0.757$\pm$0.002
LB | ✗ | ✗ |  0.762$\pm$0.007 |  1.302$\pm$0.005 |  0.482$\pm$0.004 |  0.682$\pm$0.005 |  0.604$\pm$0.003 |  0.416$\pm$0.002 |  0.602$\pm$0.006
CL | ✗ | ✗ |  0.803$\pm$0.035 |  2.435$\pm$1.218 |  0.318$\pm$0.057 |  0.800$\pm$0.008 |  0.446$\pm$0.016 |  0.285$\pm$0.012 |  0.674$\pm$0.007
REAC | ✗ | ✗ |  0.922$\pm$0.003 |  0.288$\pm$0.065 |  0.115$\pm$0.007 |  0.815$\pm$0.001 |  0.395$\pm$0.001 |  0.249$\pm$0.001 |  0.684$\pm$0.001
UNION | ✓ | ✗ |  0.866$\pm$0.019 |  1.668$\pm$0.322 |  0.224$\pm$0.034 |  0.795$\pm$0.007 |  0.432$\pm$0.007 |  0.278$\pm$0.007 |  0.667$\pm$0.006
LIA | ✗ | ✗ |  0.823$\pm$0.005 |  1.483$\pm$0.018 |  0.569$\pm$0.007 |  0.676$\pm$0.005 |  0.629$\pm$0.004 |  0.436$\pm$0.004 |  0.575$\pm$0.004
CoNAL | ✓ | ✓ |  0.871$\pm$0.015 |  1.380$\pm$0.349 |  0.213$\pm$0.024 |  0.840$\pm$0.014 |  0.390$\pm$0.028 |  0.238$\pm$0.021 |  0.712$\pm$0.014
MaDL($\overline{\text{W}}$) | ✗ | ✗ |  0.946$\pm$0.006 |  0.293$\pm$0.082 |  0.083$\pm$0.009 |  0.883$\pm$0.002 |  0.314$\pm$0.001 |  0.178$\pm$0.002 |  0.751$\pm$0.003
MaDL(W) | ✓ | ✓ |  0.947$\pm$0.003 |  0.282$\pm$0.069 |  0.080$\pm$0.004 |  0.887$\pm$0.001 |  0.308$\pm$0.004 |  0.175$\pm$0.002 |  0.756$\pm$0.001
fmnist (correlated)
UB | ✗ | ✓ | 0.909$\pm$0.002 | 0.246$\pm$0.005 | 0.131$\pm$0.003 | 0.866$\pm$0.002 | 0.333$\pm$0.002 | 0.198$\pm$0.002 | 0.741$\pm$0.002
LB | ✗ | ✗ |  0.787$\pm$0.003 |  1.127$\pm$0.013 |  0.475$\pm$0.007 |  0.668$\pm$0.009 |  0.626$\pm$0.006 |  0.436$\pm$0.006 |  0.580$\pm$0.005
CL | ✗ | ✗ |  0.868$\pm$0.003 |  0.447$\pm$0.020 |  0.217$\pm$0.010 |  0.799$\pm$0.004 |  0.421$\pm$0.004 |  0.270$\pm$0.003 |  0.677$\pm$0.004
REAC | ✗ | ✗ |  0.873$\pm$0.004 |  0.415$\pm$0.012 |  0.196$\pm$0.006 |  0.828$\pm$0.001 |  0.382$\pm$0.001 |  0.237$\pm$0.001 |  0.697$\pm$0.001
UNION | ✓ | ✗ |  0.859$\pm$0.006 |  0.411$\pm$0.018 |  0.205$\pm$0.008 |  0.801$\pm$0.009 |  0.420$\pm$0.014 |  0.269$\pm$0.011 |  0.678$\pm$0.009
LIA | ✗ | ✗ |  0.837$\pm$0.006 |  1.277$\pm$0.008 |  0.553$\pm$0.004 |  0.685$\pm$0.002 |  0.633$\pm$0.001 |  0.441$\pm$0.001 |  0.569$\pm$0.002
CoNAL | ✓ | ✓ |  0.897$\pm$0.002 |  0.299$\pm$0.009 |  0.152$\pm$0.004 |  0.844$\pm$0.001 |  0.356$\pm$0.003 |  0.217$\pm$0.002 |  0.721$\pm$0.001
MaDL($\overline{\text{W}}$) | ✗ | ✗ |  0.904$\pm$0.002 |  0.272$\pm$0.007 |  0.139$\pm$0.003 |  0.863$\pm$0.003 |  0.337$\pm$0.004 |  0.201$\pm$0.004 |  0.737$\pm$0.004
MaDL(W) | ✓ | ✓ |  0.903$\pm$0.002 |  0.273$\pm$0.004 |  0.141$\pm$0.002 |  0.863$\pm$0.003 |  0.338$\pm$0.003 |  0.202$\pm$0.003 |  0.738$\pm$0.003
cifar10 (correlated)
UB | ✗ | ✓ | 0.933$\pm$0.002 | 0.495$\pm$0.017 | 0.118$\pm$0.003 | 0.837$\pm$0.001 | 0.384$\pm$0.001 | 0.235$\pm$0.001 | 0.711$\pm$0.001
LB | ✗ | ✗ |  0.652$\pm$0.014 |  1.309$\pm$0.016 |  0.540$\pm$0.008 |  0.602$\pm$0.011 |  0.623$\pm$0.003 |  0.436$\pm$0.003 |  0.541$\pm$0.008
CL | ✗ | ✗ |  0.850$\pm$0.007 |  0.490$\pm$0.022 |  0.224$\pm$0.011 |  0.799$\pm$0.002 |  0.439$\pm$0.004 |  0.282$\pm$0.003 |  0.674$\pm$0.002
REAC | ✗ | ✗ |  0.856$\pm$0.003 |  0.600$\pm$0.063 |  0.259$\pm$0.025 |  0.775$\pm$0.017 |  0.445$\pm$0.015 |  0.287$\pm$0.012 |  0.648$\pm$0.017
UNION | ✓ | ✗ |  0.858$\pm$0.007 |  0.499$\pm$0.024 |  0.211$\pm$0.009 |  0.800$\pm$0.003 |  0.432$\pm$0.002 |  0.276$\pm$0.002 |  0.675$\pm$0.003
LIA | ✗ | ✗ |  0.776$\pm$0.002 |  1.343$\pm$0.020 |  0.565$\pm$0.009 |  0.741$\pm$0.002 |  0.617$\pm$0.003 |  0.424$\pm$0.003 |  0.617$\pm$0.002
CoNAL | ✓ | ✓ |  0.862$\pm$0.002 |  0.473$\pm$0.005 |  0.213$\pm$0.003 |  0.800$\pm$0.001 |  0.433$\pm$0.003 |  0.277$\pm$0.002 |  0.676$\pm$0.001
MaDL($\overline{\text{W}}$) | ✗ | ✗ |  0.878$\pm$0.004 |  0.439$\pm$0.015 |  0.184$\pm$0.005 |  0.824$\pm$0.004 |  0.398$\pm$0.004 |  0.247$\pm$0.004 |  0.699$\pm$0.004
MaDL(W) | ✓ | ✓ |  0.875$\pm$0.008 |  0.434$\pm$0.020 |  0.188$\pm$0.011 |  0.823$\pm$0.002 |  0.397$\pm$0.003 |  0.248$\pm$0.002 |  0.698$\pm$0.002
svhn (correlated)
UB | ✗ | ✓ | 0.966$\pm$0.001 | 0.382$\pm$0.018 | 0.062$\pm$0.001 | 0.794$\pm$0.003 | 0.414$\pm$0.002 | 0.266$\pm$0.002 | 0.657$\pm$0.004
LB | ✗ | ✗ |  0.900$\pm$0.005 |  1.012$\pm$0.038 |  0.420$\pm$0.017 |  0.624$\pm$0.022 |  0.634$\pm$0.008 |  0.444$\pm$0.007 |  0.567$\pm$0.017
CL | ✗ | ✗ |  0.947$\pm$0.001 |  0.314$\pm$0.044 |  0.116$\pm$0.017 |  0.789$\pm$0.009 |  0.433$\pm$0.001 |  0.281$\pm$0.002 |  0.655$\pm$0.012
REAC | ✗ | ✗ |  0.946$\pm$0.002 |  0.263$\pm$0.012 |  0.097$\pm$0.005 |  0.767$\pm$0.002 |  0.431$\pm$0.001 |  0.283$\pm$0.000 |  0.620$\pm$0.003
UNION | ✓ | ✗ |  0.947$\pm$0.001 |  0.250$\pm$0.025 |  0.089$\pm$0.010 |  0.767$\pm$0.003 |  0.435$\pm$0.003 |  0.286$\pm$0.002 |  0.621$\pm$0.005
LIA | ✗ | ✗ |  0.929$\pm$0.002 |  1.123$\pm$0.023 |  0.477$\pm$0.011 |  0.716$\pm$0.013 |  0.623$\pm$0.010 |  0.431$\pm$0.010 |  0.594$\pm$0.013
CoNAL | ✓ | ✓ |  0.952$\pm$0.000 |  0.231$\pm$0.003 |  0.075$\pm$0.001 |  0.835$\pm$0.003 |  0.379$\pm$0.005 |  0.235$\pm$0.004 |  0.702$\pm$0.004
MaDL($\overline{\text{W}}$) | ✗ | ✗ |  0.950$\pm$0.002 |  0.237$\pm$0.006 |  0.078$\pm$0.003 |  0.790$\pm$0.003 |  0.416$\pm$0.002 |  0.269$\pm$0.002 |  0.652$\pm$0.002
MaDL(W) | ✓ | ✓ |  0.952$\pm$0.001 |  0.227$\pm$0.006 |  0.075$\pm$0.002 |  0.784$\pm$0.003 |  0.420$\pm$0.002 |  0.273$\pm$0.002 |  0.645$\pm$0.004
Table 7:  Results regarding RQ2 for datasets with simulated annotators:

Best

and

second best

performances are highlighted per dataset and evaluation score while excluding the performances of the UB.  Technique | P3 | P4 | Ground Truth Model | Annotator Performance Model
---|---|---|---|---
ACC $\uparrow$ |  NLL $\downarrow$ |  BS $\downarrow$ |  ACC $\uparrow$ |  NLL $\downarrow$ |  BS $\downarrow$ |  BAL-ACC $\uparrow$
letter (random-correlated)
UB | ✗ | ✓ | 0.960$\pm$0.003 | 0.131$\pm$0.006 | 0.059$\pm$0.003 | 0.937$\pm$0.002 | 0.212$\pm$0.003 | 0.104$\pm$0.002 | 0.516$\pm$0.002
LB | ✗ | ✗ |  0.056$\pm$0.009 |  3.307$\pm$0.049 |  0.965$\pm$0.004 |  0.088$\pm$0.000 |  9.950$\pm$2.090 |  1.816$\pm$0.002 |  0.500$\pm$0.000
CL | ✗ | ✗ |  0.565$\pm$0.028 |  3.519$\pm$0.455 |  0.682$\pm$0.052 |  0.925$\pm$0.000 |  0.237$\pm$0.004 |  0.124$\pm$0.002 |  0.506$\pm$0.000
REAC | ✗ | ✗ |  0.607$\pm$0.024 |  1.810$\pm$0.127 |  0.561$\pm$0.034 |  0.926$\pm$0.000 |  0.221$\pm$0.004 |  0.116$\pm$0.002 |  0.507$\pm$0.000
UNION | ✓ | ✗ |  0.615$\pm$0.034 |  3.317$\pm$0.582 |  0.625$\pm$0.065 |  0.925$\pm$0.000 |  0.232$\pm$0.004 |  0.122$\pm$0.002 |  0.506$\pm$0.000
LIA | ✗ | ✗ |  0.352$\pm$0.010 |  2.960$\pm$0.035 |  0.932$\pm$0.004 |  0.088$\pm$0.000 |  2.131$\pm$0.137 |  1.474$\pm$0.041 |  0.500$\pm$0.000
CoNAL | ✓ | ✓ |  0.581$\pm$0.015 |  2.325$\pm$0.249 |  0.599$\pm$0.027 |  0.925$\pm$0.000 |  0.236$\pm$0.002 |  0.124$\pm$0.001 |  0.507$\pm$0.000
MaDL($\overline{\text{W}}$) | ✗ | ✗ |  0.548$\pm$0.033 |  1.902$\pm$0.215 |  0.673$\pm$0.064 |  0.801$\pm$0.044 |  0.423$\pm$0.033 |  0.265$\pm$0.027 |  0.506$\pm$0.006
MaDL(W) | ✓ | ✓ |  0.932$\pm$0.003 |  0.277$\pm$0.038 |  0.101$\pm$0.005 |  0.940$\pm$0.000 |  0.204$\pm$0.003 |  0.101$\pm$0.001 |  0.519$\pm$0.001
fmnist (random-correlated)
UB | ✗ | ✓ | 0.909$\pm$0.002 | 0.246$\pm$0.005 | 0.131$\pm$0.003 | 0.888$\pm$0.000 | 0.337$\pm$0.001 | 0.191$\pm$0.000 | 0.520$\pm$0.000
LB | ✗ | ✗ |  0.172$\pm$0.019 |  2.296$\pm$0.005 |  0.899$\pm$0.001 |  0.140$\pm$0.000 |  21.865$\pm$6.169 |  1.703$\pm$0.000 |  0.500$\pm$0.000
CL | ✗ | ✗ |  0.880$\pm$0.003 |  0.462$\pm$0.169 |  0.222$\pm$0.073 |  0.880$\pm$0.003 |  0.347$\pm$0.004 |  0.200$\pm$0.003 |  0.513$\pm$0.002
REAC | ✗ | ✗ |  0.870$\pm$0.003 |  0.470$\pm$0.009 |  0.204$\pm$0.004 |  0.885$\pm$0.000 |  0.342$\pm$0.000 |  0.194$\pm$0.000 |  0.514$\pm$0.000
UNION | ✓ | ✗ |  0.884$\pm$0.002 |  0.387$\pm$0.022 |  0.182$\pm$0.007 |  0.881$\pm$0.000 |  0.345$\pm$0.000 |  0.198$\pm$0.000 |  0.514$\pm$0.000
LIA | ✗ | ✗ |  0.677$\pm$0.008 |  2.094$\pm$0.002 |  0.852$\pm$0.001 |  0.140$\pm$0.000 |  2.067$\pm$0.005 |  1.418$\pm$0.002 |  0.500$\pm$0.000
CoNAL | ✓ | ✓ |  0.858$\pm$0.012 |  0.457$\pm$0.086 |  0.219$\pm$0.031 |  0.882$\pm$0.002 |  0.344$\pm$0.002 |  0.197$\pm$0.002 |  0.516$\pm$0.001
MaDL($\overline{\text{W}}$) | ✗ | ✗ |  0.337$\pm$0.046 |  2.131$\pm$0.090 |  0.855$\pm$0.029 |  0.229$\pm$0.075 |  1.038$\pm$0.146 |  0.814$\pm$0.128 |  0.498$\pm$0.002
MaDL(W) | ✓ | ✓ |  0.896$\pm$0.002 |  0.290$\pm$0.003 |  0.150$\pm$0.002 |  0.889$\pm$0.000 |  0.337$\pm$0.000 |  0.191$\pm$0.000 |  0.520$\pm$0.000
cifar10 (random-correlated)
UB | ✗ | ✓ | 0.932$\pm$0.002 | 0.519$\pm$0.016 | 0.119$\pm$0.004 | 0.886$\pm$0.000 | 0.340$\pm$0.002 | 0.192$\pm$0.001 | 0.515$\pm$0.000
LB | ✗ | ✗ |  0.141$\pm$0.008 |  2.301$\pm$0.002 |  0.900$\pm$0.000 |  0.139$\pm$0.000 |  14.224$\pm$6.699 |  1.704$\pm$0.001 |  0.500$\pm$0.000
CL | ✗ | ✗ |  0.576$\pm$0.023 |  1.395$\pm$0.090 |  0.576$\pm$0.028 |  0.878$\pm$0.000 |  0.353$\pm$0.002 |  0.204$\pm$0.001 |  0.507$\pm$0.000
REAC | ✗ | ✗ |  0.462$\pm$0.010 |  2.093$\pm$0.062 |  0.767$\pm$0.011 |  0.875$\pm$0.001 |  0.353$\pm$0.000 |  0.204$\pm$0.000 |  0.505$\pm$0.001
UNION | ✓ | ✗ |  0.540$\pm$0.049 |  1.517$\pm$0.209 |  0.629$\pm$0.065 |  0.876$\pm$0.002 |  0.355$\pm$0.003 |  0.205$\pm$0.002 |  0.506$\pm$0.002
LIA | ✗ | ✗ |  0.211$\pm$0.014 |  2.273$\pm$0.007 |  0.894$\pm$0.001 |  0.139$\pm$0.000 |  2.096$\pm$0.007 |  1.429$\pm$0.002 |  0.500$\pm$0.000
CoNAL | ✓ | ✓ |  0.555$\pm$0.020 |  1.379$\pm$0.053 |  0.592$\pm$0.020 |  0.876$\pm$0.001 |  0.355$\pm$0.002 |  0.206$\pm$0.002 |  0.506$\pm$0.001
MaDL($\overline{\text{W}}$) | ✗ | ✗ |  0.217$\pm$0.042 |  6.992$\pm$0.386 |  1.219$\pm$0.087 |  0.872$\pm$0.001 |  0.398$\pm$0.011 |  0.229$\pm$0.009 |  0.502$\pm$0.001
MaDL(W) | ✓ | ✓ |  0.822$\pm$0.007 |  0.593$\pm$0.033 |  0.262$\pm$0.010 |  0.885$\pm$0.000 |  0.339$\pm$0.001 |  0.192$\pm$0.001 |  0.514$\pm$0.000
svhn (random-correlated)
UB | ✗ | ✓ | 0.965$\pm$0.001 | 0.399$\pm$0.017 | 0.064$\pm$0.001 | 0.877$\pm$0.000 | 0.349$\pm$0.000 | 0.201$\pm$0.000 | 0.509$\pm$0.001
LB | ✗ | ✗ |  0.190$\pm$0.000 |  2.298$\pm$0.002 |  0.899$\pm$0.000 |  0.138$\pm$0.000 |  24.019$\pm$7.802 |  1.704$\pm$0.001 |  0.500$\pm$0.000
CL | ✗ | ✗ |  0.908$\pm$0.038 |  0.398$\pm$0.226 |  0.143$\pm$0.056 |  0.873$\pm$0.001 |  0.354$\pm$0.002 |  0.205$\pm$0.001 |  0.505$\pm$0.000
REAC | ✗ | ✗ |  0.189$\pm$0.001 |  2.294$\pm$0.003 |  0.898$\pm$0.001 |  0.140$\pm$0.000 |  2.262$\pm$0.734 |  1.384$\pm$0.304 |  0.500$\pm$0.000
UNION | ✓ | ✗ |  0.881$\pm$0.104 |  0.529$\pm$0.553 |  0.179$\pm$0.154 |  0.872$\pm$0.002 |  0.356$\pm$0.008 |  0.206$\pm$0.005 |  0.505$\pm$0.000
LIA | ✗ | ✗ |  0.192$\pm$0.004 |  2.294$\pm$0.004 |  0.898$\pm$0.001 |  0.138$\pm$0.000 |  3.864$\pm$3.540 |  1.483$\pm$0.111 |  0.500$\pm$0.000
CoNAL | ✓ | ✓ |  0.231$\pm$0.048 |  2.933$\pm$0.526 |  0.956$\pm$0.072 |  0.860$\pm$0.000 |  0.414$\pm$0.008 |  0.242$\pm$0.003 |  0.500$\pm$0.000
MaDL($\overline{\text{W}}$) | ✗ | ✗ |  0.243$\pm$0.102 |  6.055$\pm$3.173 |  1.119$\pm$0.230 |  0.575$\pm$0.352 |  0.702$\pm$0.344 |  0.505$\pm$0.319 |  0.500$\pm$0.001
MaDL(W) | ✓ | ✓ |  0.940$\pm$0.002 |  0.244$\pm$0.011 |  0.091$\pm$0.003 |  0.877$\pm$0.000 |  0.349$\pm$0.000 |  0.201$\pm$0.000 |  0.508$\pm$0.000

###  5.4 RQ3: Do annotator features containing prior information about
annotators improve learning and enable inductively learning annotators’
performances? (Properties P5, P6)

Takeaway: Annotator features containing prior information about annotators
improve the learning of GT and AP models (property P5). Furthermore, we can
use these annotator features to inductively estimate the performances of
annotators unavailable during training (property P6).

Setup: We address RQ3 by evaluating multi-annotator supervised learning
techniques with and without using annotator features containing prior
information. For each dataset, we simulate 100 annotators according to the
annotator set inductive in Table 3. However, only 75 annotators provide class
labels for training. Each of them provides class labels for
$2\text{\,}\mathrm{\char 37\relax}$ of randomly selected training instances.
The lower annotation ratio is used to study the generalization across
annotators sharing similar features. The remaining 25 annotators form a test
set to assess AP predictions. We generate annotator features containing prior
information by composing information about annotator type, class-wise APs, and
cluster-wise APs. Fig. 8 provides examples for two annotators based on two
classes and four clusters. We evaluate two variants of LIA, CoNAL, and MaDL,
denoted respectively by the schemes LIA(P5), CoNAL(P5), and MaDL(P5). Property
P5 refers to a technique’s ability to consider prior information about
annotators. We differentiate between the variant with annotator features
containing prior information (A) and the one using one-hot encoded features to
separate between annotators’ identities ($\overline{\text{A}}$).
MaDL($\overline{\text{A}}$) corresponds to MaDL’s default variant in this
setup. We do not evaluate CL, UNION, and REAC since these techniques cannot
handle annotator features.

Adversarial Annotator

$\mathbf{a}_{1}$

Cluster-specialized Annotator

$\mathbf{a}_{2}$$\mathbf{a}_{1}=\begin{pmatrix}{\color[rgb]{0.1640625,0.49609375,1}\definecolor[named]{pgfstrokecolor}{rgb}{0.1640625,0.49609375,1}\text{adversarial}}\\\
{\color[rgb]{0.99609375,0.1640625,0.1640625}\definecolor[named]{pgfstrokecolor}{rgb}{0.99609375,0.1640625,0.1640625}0.04}\\\
{\color[rgb]{0.99609375,0.1640625,0.1640625}\definecolor[named]{pgfstrokecolor}{rgb}{0.99609375,0.1640625,0.1640625}0.06}\\\
{\color[rgb]{0,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.5,0.5}0.03}\\\
{\color[rgb]{0,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.5,0.5}0.07}\\\
{\color[rgb]{0,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.5,0.5}0.04}\\\
{\color[rgb]{0,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.5,0.5}0.05}\end{pmatrix}$$\mathbf{a}_{2}=\begin{pmatrix}{\color[rgb]{0.1640625,0.49609375,1}\definecolor[named]{pgfstrokecolor}{rgb}{0.1640625,0.49609375,1}\text{cluster-
specialized}}\\\
{\color[rgb]{0.99609375,0.1640625,0.1640625}\definecolor[named]{pgfstrokecolor}{rgb}{0.99609375,0.1640625,0.1640625}0.51}\\\
{\color[rgb]{0.99609375,0.1640625,0.1640625}\definecolor[named]{pgfstrokecolor}{rgb}{0.99609375,0.1640625,0.1640625}0.49}\\\
{\color[rgb]{0,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.5,0.5}0.95}\\\
{\color[rgb]{0,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.5,0.5}0.03}\\\
{\color[rgb]{0,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.5,0.5}0.95}\\\
{\color[rgb]{0,0.5,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.5,0.5}0.07}\end{pmatrix}$
Figure 8: Visualization of MaDL(A)’s inductive AP estimates for two unknown
annotators.

Qualitative study: Fig. 8 visualizes AP predictions of MaDL(A) regarding two
exemplary annotators for the dataset toy. The visualization of these AP
predictions is analogous to Fig. 6. Neither of the two annotators provides
class labels for the training, and the plotted training instances show only
potential annotations to visualize the annotation patterns. The vectors at the
right list the annotator features containing prior information for both
annotators. The colors reveal the meanings of the respective feature values.
These meanings are unknown to MaDL(A), such that its AP predictions
exclusively result from generalizing similar annotators’ features and their
annotations available during training. MaDL(A) correctly identifies the left
annotator as adversarial because it predicts low (white) AP scores across the
feature space regions close to training instances. For the right cluster-
specialized annotator, MaDL(A) accurately separates the two weak clusters
(feature space regions with predominantly crosses) with low AP estimates from
the two expert clusters (feature space regions with predominantly circles)
with high AP estimates.

Quantitative study: Table 8 presents the GT and AP models’ test performances
for the four datasets with the simulated annotator set inductive. The table
further indicates whether a technique processes prior information as annotator
features (property P5) and whether a technique can inductively estimate the
performances of annotators unavailable during the training phase (property
P6). Note that the AP results refer to the aforementioned 25 test annotators.
Hence, there are no results (marked as –) for techniques with AP models not
fulfilling property P6. For completeness, we provide the results for the 75
annotators providing class labels for training in Appendix D. As for RQ1 and
RQ2, training with GT labels leads to the best performance results (UB),
whereas learning from annotations aggregated via the majority rule mostly
results in the worst performances (LB). Inspecting the results of MaDL(A)’s GT
model compared to the other techniques, we observe competitive or partially
superior results across all four datasets. Concerning its AP model, we further
note that MaDL(A) provides meaningful AP estimates, indicated by BAL-ACC
values greater than 0.5. Comparing the GT models’ results of each pair of
variants, performance gains for LIA and MaDL demonstrate the potential
benefits of learning from annotator features containing prior annotator
information. In contrast, the GT models’ results of CoNAL(A) and
CoNAL($\overline{\text{A}}$) hardly differ.

Table 8:  Results regarding RQ3 for datasets with simulated annotators:

Best

and

second best

performances are highlighted per dataset and evaluation score while excluding the performances of the UB. The AP models’ results refer to the 25 test annotators providing no class labels for training. An entry – marks a technique whose AP model cannot make predictions for such test annotators.  Technique | P5 | P6 | Ground Truth Model | Annotator Performance Model
---|---|---|---|---
ACC $\uparrow$ |  NLL $\downarrow$ |  BS $\downarrow$ |  ACC $\uparrow$ |  NLL $\downarrow$ |  BS $\downarrow$ |  BAL-ACC $\uparrow$
letter (inductive)
UB | ✓ | ✓ | 0.962$\pm$0.002 | 0.129$\pm$0.003 | 0.058$\pm$0.002 | 0.672$\pm$0.005 | 0.745$\pm$0.047 | 0.457$\pm$0.011 | 0.612$\pm$0.005
LB | ✓ | ✓ |  0.861$\pm$0.005 |  1.090$\pm$0.017 |  0.429$\pm$0.008 |  0.569$\pm$0.008 |  0.730$\pm$0.011 |  0.522$\pm$0.007 |  0.537$\pm$0.006
LIA($\overline{\text{A}}$) | ✗ | ✗ |  0.875$\pm$0.006 |  0.901$\pm$0.060 |  0.350$\pm$0.024 | – | – | – | –
LIA(A) | ✓ | ✓ |  0.876$\pm$0.006 |  1.006$\pm$0.177 |  0.397$\pm$0.074 |  0.609$\pm$0.017 |  1.447$\pm$0.845 |  0.597$\pm$0.105 |  0.545$\pm$0.033
CoNAL($\overline{\text{A}}$) | ✗ | ✗ |  0.875$\pm$0.009 |  0.804$\pm$0.119 |  0.186$\pm$0.010 | – | – | – | –
CoNAL(A) | ✓ | ✗ |  0.874$\pm$0.007 |  0.808$\pm$0.116 |  0.186$\pm$0.011 | – | – | – | –
MaDL($\overline{\text{A}}$) | ✗ | ✗ |  0.911$\pm$0.006 |  0.334$\pm$0.026 |  0.129$\pm$0.008 | – | – | – | –
MaDL(A) | ✓ | ✓ |  0.914$\pm$0.004 |  0.303$\pm$0.009 |  0.124$\pm$0.005 |  0.668$\pm$0.007 |  0.813$\pm$0.115 |  0.471$\pm$0.015 |  0.600$\pm$0.010
fmnist (inductive)
UB | ✓ | ✓ | 0.909$\pm$0.002 | 0.246$\pm$0.005 | 0.131$\pm$0.003 | 0.730$\pm$0.008 | 0.536$\pm$0.019 | 0.357$\pm$0.010 | 0.656$\pm$0.009
LB | ✓ | ✓ |  0.881$\pm$0.002 |  0.876$\pm$0.005 |  0.370$\pm$0.002 |  0.590$\pm$0.023 |  0.681$\pm$0.005 |  0.487$\pm$0.006 |  0.537$\pm$0.010
LIA($\overline{\text{A}}$) | ✗ | ✗ |  0.852$\pm$0.003 |  1.011$\pm$0.020 |  0.436$\pm$0.010 | – | – | – | –
LIA(A) | ✓ | ✓ |  0.855$\pm$0.002 |  0.972$\pm$0.012 |  0.417$\pm$0.006 |  0.674$\pm$0.036 |  0.626$\pm$0.026 |  0.436$\pm$0.024 |  0.601$\pm$0.027
CoNAL($\overline{\text{A}}$) | ✗ | ✗ |  0.889$\pm$0.002 |  0.322$\pm$0.005 |  0.163$\pm$0.003 | – | – | – | –
CoNAL(A) | ✓ | ✗ |  0.890$\pm$0.002 |  0.323$\pm$0.011 |  0.163$\pm$0.005 | – | – | – | –
MaDL($\overline{\text{A}}$) | ✗ | ✗ |  0.895$\pm$0.002 |  0.297$\pm$0.004 |  0.152$\pm$0.002 | – | – | – | –
MaDL(A) | ✓ | ✓ |  0.893$\pm$0.004 |  0.297$\pm$0.008 |  0.153$\pm$0.004 |  0.723$\pm$0.004 |  0.538$\pm$0.003 |  0.362$\pm$0.003 |  0.649$\pm$0.005
cifar10 (inductive)
UB | ✓ | ✓ | 0.931$\pm$0.002 | 0.527$\pm$0.022 | 0.122$\pm$0.003 | 0.686$\pm$0.006 | 0.646$\pm$0.101 | 0.409$\pm$0.016 | 0.613$\pm$0.006
LB | ✓ | ✓ |  0.781$\pm$0.003 |  1.054$\pm$0.035 |  0.447$\pm$0.016 |  0.583$\pm$0.009 |  0.684$\pm$0.004 |  0.490$\pm$0.004 |  0.521$\pm$0.003
LIA($\overline{\text{A}}$) | ✗ | ✗ |  0.798$\pm$0.008 |  1.072$\pm$0.014 |  0.455$\pm$0.006 | – | – | – | –
LIA(A) | ✓ | ✓ |  0.804$\pm$0.004 |  1.056$\pm$0.022 |  0.447$\pm$0.011 |  0.607$\pm$0.020 |  0.670$\pm$0.017 |  0.477$\pm$0.016 |  0.544$\pm$0.010
CoNAL($\overline{\text{A}}$) | ✗ | ✗ |  0.835$\pm$0.002 |  0.576$\pm$0.016 |  0.245$\pm$0.005 | – | – | – | –
CoNAL(A) | ✓ | ✗ |  0.834$\pm$0.006 |  0.574$\pm$0.017 |  0.248$\pm$0.007 | – | – | – | –
MaDL($\overline{\text{A}}$) | ✗ | ✗ |  0.811$\pm$0.008 |  0.626$\pm$0.036 |  0.277$\pm$0.014 | – | – | – | –
MaDL(A) | ✓ | ✓ |  0.837$\pm$0.003 |  0.557$\pm$0.028 |  0.242$\pm$0.006 |  0.698$\pm$0.003 |  0.567$\pm$0.015 |  0.383$\pm$0.004 |  0.617$\pm$0.004
svhn (inductive)
UB | ✓ | ✓ | 0.965$\pm$0.001 | 0.393$\pm$0.015 | 0.063$\pm$0.002 | 0.613$\pm$0.004 | 0.943$\pm$0.113 | 0.511$\pm$0.015 | 0.524$\pm$0.006
LB | ✓ | ✓ |  0.927$\pm$0.002 |  0.805$\pm$0.016 |  0.328$\pm$0.009 |  0.588$\pm$0.010 |  0.704$\pm$0.007 |  0.509$\pm$0.006 |  0.511$\pm$0.007
LIA($\overline{\text{A}}$) | ✗ | ✗ |  0.929$\pm$0.003 |  0.818$\pm$0.133 |  0.336$\pm$0.068 | – | – | – | –
LIA(A) | ✓ | ✓ |  0.932$\pm$0.001 |  0.754$\pm$0.152 |  0.303$\pm$0.079 |  0.603$\pm$0.013 |  0.671$\pm$0.024 |  0.478$\pm$0.022 |  0.513$\pm$0.008
CoNAL($\overline{\text{A}}$) | ✗ | ✗ |  0.941$\pm$0.001 |  0.258$\pm$0.009 |  0.090$\pm$0.003 | – | – | – | –
CoNAL(A) | ✓ | ✗ |  0.942$\pm$0.001 |  0.260$\pm$0.012 |  0.090$\pm$0.002 | – | – | – | –
MaDL($\overline{\text{A}}$) | ✗ | ✗ |  0.928$\pm$0.002 |  0.299$\pm$0.019 |  0.109$\pm$0.005 | – | – | – | –
MaDL(A) | ✓ | ✓ |  0.935$\pm$0.001 |  0.256$\pm$0.009 |  0.098$\pm$0.002 |  0.624$\pm$0.007 |  0.632$\pm$0.013 |  0.444$\pm$0.008 |  0.521$\pm$0.006

##  6 Conclusion

In this article, we made three main contributions. (1) We started with a
formalization of the objectives in multi-annotator supervised learning.
Focusing on AP estimation, we then presented six relevant properties (cf.
P1–P6 in Section 3) for categorizing related techniques in this research area.
(2) Considering these six properties, we proposed our framework MaDL. A
modular, probabilistic design and a weighted loss function modeling annotator
correlations characterize its novelties. (3) We experimentally investigated
the six properties via three RQs. The results confirmed MaDL’s robust and
often superior performance to related multi-annotator supervised learning
techniques. The findings of this article, with a focus on AP estimation,
provide a starting point for several aspects of future research, some examples
of which are given below.

Although the annotator embeddings already contain information about the
annotation patterns concerning instances and classes, MaDL is currently
limited to computing annotator correlations on a global level, i.e., annotator
weights are not an explicit function of instance-annotator pairs. For example,
an extension in this direction may be valuable to quantify correlations in
certain regions of the feature space. Leveraging AP estimates for additional
applications, e.g., selecting the best crowdworkers to obtain high-quality
annotations during a crowdsourcing campaign (Herde et al., 2023), is also of
great value. Another neglected aspect is the study of epistemic uncertainty
(Huseljic et al., 2021). For example, the visualizations for the two-
dimensional dataset in Fig. 6 show high certainty of the GT and AP models in
feature space regions with no observed instances. However, meaningful
epistemic uncertainty estimates are essential in many (safety-critical)
applications (Hüllermeier & Waegeman, 2021) and would improve the
characterization of annotators’ knowledge. During our experiments, we showed
the potential benefit of annotator features. We had no access to a dataset
with prior information from real-world annotators, so we needed a suitable
simulation for these features. Therefore, and also noted by Zhang et al.
(2023), future research may acquire such prior information via crowdsourcing
to verify their benefit. As the concentration of annotators may fluctuate or
annotators may learn during the annotation process, taking time-varying APs
into account is another potential avenue for future research (Donmez et al.,
2010). Furthermore, there are already crowdsourcing approaches (Chang et al.,
2017) and concepts (Calma et al., 2016) supporting collaboration between
annotators. Thus, developing techniques considering or recommending such
collaborations is of practical value (Fang et al., 2012).

Finally, we limited ourselves to empirical performance results and
classification tasks with class labels as annotations. Future investigations
on theoretical performance guarantees of MaDL and the learning with different
annotation types, such as class labels with confidence scores (Berthon et al.,
2021) or partial labels (Yu et al., 2022), are apparent. Furthermore, the
extension to related supervised learning tasks, such as semantic segmentation,
sequence classification, and regression, is of interest. The goal of semantic
segmentation is to classify individual pixels (Minaee et al., 2021). A
potential approach to extend MaDL would be to implement its GT model through a
U-Net (Ronneberger et al., 2015) and feed its latent representations as input
to the AP model for estimating pixel-wise confusion matrices per annotator.
Likewise, we may adapt MaDL to be applied to sequence classification tasks,
such as named entity recognition (Li et al., 2020). Concretely, we could
implement the GT model through a BiLSTM-network with softmax outputs (Reimers
& Gurevych, 2017) and feed its latent word representations as inputs to the AP
model for estimating word-wise confusion matrices per annotator. Since both
extensions involve higher computational costs than standard classification
tasks, one may alternatively investigate the estimation of a single (pixel- or
word-independent) confusion matrix per annotator. Regression tasks expect the
prediction of continuous target variables. Therefore, the probabilistic model
of MaDL has to be adapted. For example, the GT model could estimate the mean
and variance of an instance’s target variable, while the AP model learns
annotators’ biases and variances.

#### Broader Impact Statement

Big data is a driving force behind the success of machine learning (Zhou et
al., 2017). Reducing the effort and cost required for annotating this data is
essential for its ongoing development In this context, MaDL is a possible tool
to leverage the workforce of cost-efficient but error-prone annotators. Yet,
as a central resource for data annotation, crowdsourcing can negatively impact
individuals or even entire communities. Some of these impacts include
exploiting vulnerable individuals who participate in low-wage crowdsourcing
tasks (Schlagwein et al., 2019), producing low-quality data (Daniel et al.,
2018), and outsourcing jobs (Howe, 2008). On the one hand, multi-annotator
supervised learning techniques can improve data quality and support awarding
well-performing crowdworkers. On the other hand, such a technique may
intensify the already existing competition between crowdworkers (Schlagwein et
al., 2019). It also requires tight monitoring to ensure fair assessments of
crowdworkers. Besides the benefits of annotator features containing prior
information about annotators, there are several risks. Collecting and leaking
potentially sensitive personal data about the annotators is such a significant
risk (Xia & McKernan, 2020). Thus, the annotator features must contain only
information relevant to the learning task. Further, a lack of control over
this or other processes can lead to discrimination and bias based on gender,
origin, and other factors (Goel & Faltings, 2019). For these reasons, it is
crucial to consider and address the potential risks via responsible policies
and practices when employing multi-annotator supervised learning techniques.

#### Acknowledgments

We thank Lukas Rauch for the insightful discussions and comments, which
greatly improved this article.

## References

  * Albarqouni et al. (2016) Shadi Albarqouni, Christoph Baur, Felix Achilles, Vasileios Belagiannis, Stefanie Demirci, and Nassir Navab.  Aggnet: Deep Learning from Crowds for Mitosis Detection in Breast Cancer Histology Images.  _IEEE Trans. Med. Imaging_ , 35(5):1313–1321, 2016.
  * Algan & Ulusoy (2021) Görkem Algan and Ilkay Ulusoy.  Image classification with deep learning in the presence of noisy labels: A survey.  _Knowl. Based Syst._ , 215:106771, 2021.
  * Arik & Pfister (2021) Sercan Ö Arik and Tomas Pfister.  TabNet: Attentive Interpretable Tabular Learning.  In _AAAI Conf. Artif. Intell._ , pp. 6679–6687, 2021.
  * Berthon et al. (2021) Antonin Berthon, Bo Han, Gang Niu, Tongliang Liu, and Masashi Sugiyama.  Confidence Scores Make Instance-dependent Label-noise Learning Possible.  In _Int. Conf. Machine Learn._ , pp. 825–836, Virtual Conf., 2021\.
  * Brier (1950) Glenn W Brier.  Verification of Forecasts Expressed in Terms of Probability.  _Mon. Weather Rev._ , 78(1):1–3, 1950.
  * Brodersen et al. (2010) Kay Henning Brodersen, Cheng Soon Ong, Klaas Enno Stephan, and Joachim M Buhmann.  The Balanced Accuracy and Its Posterior Distribution.  In _IEEE Int. Conf. Pattern Recognit._ , pp. 3121–3124, Istanbul, Turkey, 2010.
  * Calma et al. (2016) Adrian Calma, Jan Marco Leimeister, Paul Lukowicz, Sarah Oeste-Reiß, Tobias Reitmaier, Albrecht Schmidt, Bernhard Sick, Gerd Stumme, and Katharina Anna Zweig.  From Active Learning to Dedicated Collaborative Interactive Learning.  In _Int. Conf. Archit. Comp. Syst._ , pp. 1–8, Nuremberg, Germany, 2016.
  * Cao et al. (2019) Peng Cao, Yilun Xu, Yuqing Kong, and Yizhou Wang.  Max-MIG: an Information Theoretic Approach for Joint Learning from Crowds.  In _Int. Conf. Learn. Represent._ , New Orleans, LA, 2019.
  * Chang et al. (2017) Joseph Chee Chang, Saleema Amershi, and Ece Kamar.  Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets.  In _CHI Conf. Hum. Factors Comp. Syst._ , pp. 2334–2346, Denver, CO, 2017.
  * Chu et al. (2021) Zhendong Chu, Jing Ma, and Hongning Wang.  Learning from Crowds by Modeling Common Confusions.  In _AAAI Conf. Artif. Intell._ , pp. 5832–5840, Virtual Conf., 2021\.
  * Daniel et al. (2018) Florian Daniel, Pavel Kucherbaev, Cinzia Cappiello, Boualem Benatallah, and Mohammad Allahbakhsh.  Quality control in crowdsourcing: A survey of quality attributes, assessment techniques, and assurance actions.  _ACM Comput. Surv._ , 51(1):1–40, 2018.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _arXiv:1810.04805_ , 2018.
  * Donmez et al. (2010) Pinar Donmez, Jaime Carbonell, and Jeff Schneider.  A Probabilistic Framework to Learn from Multiple Annotators with Time-Varying Accuracy.  In _SIAM Int. Conf. Data Min._ , pp. 826–837, Columbus, OH, 2010\.
  * Fang et al. (2012) Meng Fang, Xingquan Zhu, Bin Li, Wei Ding, and Xindong Wu.  Self-Taught Active Learning from Crowds.  In _IEEE Int. Conf. Data Min._ , pp. 858–863, Brussels, Belgium, 2012.
  * Fiedler (2021) James Fiedler.  Simple Modifications to Improve Tabular Neural Networks.  _arXiv:2108.03214_ , 2021.
  * Frey & Slate (1991) Peter W. Frey and David J. Slate.  Letter recognition using Holland-style adaptive classifiers.  _Machine Learn._ , 6(2):161–182, 1991.
  * Gao et al. (2022) Zhengqi Gao, Fan-Keng Sun, Mingran Yang, Sucheng Ren, Zikai Xiong, Marc Engeler, Antonio Burazer, Linda Wildling, Luca Daniel, and Duane S. Boning.  Learning from Multiple Annotator Noisy Labels via Sample-Wise Label Fusion.  In _Eur. Conf. Comput. Vis._ , pp. 407–422, Tel Aviv, Israel, 2022\.
  * Gil-Gonzalez et al. (2021) J. Gil-Gonzalez, A. Orozco-Gutierrez, and A. Alvarez-Meza.  Learning from multiple inconsistent and dependent annotators to support classification tasks.  _Neurocomputing_ , 423:236–247, 2021.
  * Gil-González et al. (2021) Julián Gil-González, Andrés Valencia-Duque, Andrés Álvarez-Meza, Álvaro Orozco-Gutiérrez, and Andrea García-Moreno.  Regularized Chained Deep Neural Network Classifier for Multiple Annotators.  _Appl. Sci._ , 11(12):5409, 2021.
  * Gilyazev & Turdakov (2018) Ruslan A. Gilyazev and Denis Y. Turdakov.  Active Learning and Crowdsourcing: A Survey of Optimization Methods for Data Labeling.  _Program. Comput. Softw._ , 44(6):476–491, 2018\.
  * Glorot et al. (2011) Xavier Glorot, Antoine Bordes, and Yoshua Bengio.  Deep Sparse Rectifier Neural Networks.  In _Int. Conf. Artif. Intell. Stat._ , pp. 315–323, Fort Lauderdale, FL, 2011.
  * Goel & Faltings (2019) Naman Goel and Boi Faltings.  Crowdsourcing with Fairness, Diversity and Budget Constraints.  In _AAAI/ACM Conf. AI, Ethics, Soc._ , pp. 297–304, Honolulu, HI, 2019.
  * Gu et al. (2022) Keren Gu, Xander Masotto, Vandana Bachani, Balaji Lakshminarayanan, Jack Nikodem, and Dong Yin.  An instance-dependent simulation framework for learning with label noise.  _Mach. Learn._ , 2022.
  * Guan et al. (2018) Melody Y. Guan, Varun Gulshan, Andrew M. Dai, and Geoffrey E. Hinton.  Who Said What: Modeling Individual Labelers Improves Classification.  In _AAAI Conf. Artif. Intell._ , pp. 3109–3118, New Orleans, LA, 2018.
  * Gupta et al. (2014) Maya R Gupta, Samy Bengio, and Jason Weston.  Training Highly Multiclass Classifiers.  _J. Mach. Learn. Res._ , 15(1):1461–1492, 2014\.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep Residual Learning for Image Recognition.  In _Conf. Comput. Vis. Pattern Recognit._ , pp. 770–778, Las Vegas, NV, 2016.
  * He et al. (2018) Xiangnan He, Xiaoyu Du, Xiang Wang, Feng Tian, Jinhui Tang, and Tat-Seng Chua.  Outer Product-based Neural Collaborative Filtering.  In _Int. Joint Conf. Artif. Intell._ , pp. 2227–2233, Stockholm, Sweden, 2018.
  * Herde et al. (2021) Marek Herde, Denis Huseljic, Bernhard Sick, and Adrian Calma.  A Survey on Cost Types, Interaction Schemes, and Annotator Performance Models in Selection Algorithms for Active Learning in Classification.  _IEEE Access_ , 9:166970–166989, 2021.
  * Herde et al. (2023) Marek Herde, Denis Huseljic, Bernhard Sick, Ulrich Bretschneider, and Sarah Oeste-Reiß.  Who knows best? A Case Study on Intelligent Crowdworker Selection via Deep Learning.  In _Int. Workshop on Interact. Adapt. Learn. @ Eur. Conf. Mach. Learn._ , pp. 14–18, Turin, Italy, 2023.
  * Hestness et al. (2017) Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou.  Deep Learning Scaling is Predictable, Empirically.  _arXiv:1712.00409_ , 2017.
  * Hoiem et al. (2021) Derek Hoiem, Tanmay Gupta, Zhizhong Li, and Michal Shlapentokh-Rothman.  Learning Curves for Analysis of Deep Networks.  In _Int. Conf. Machine Learn._ , pp. 4287–4296, Virtual Conf., 2021\.
  * Howe (2008) Jeff Howe.  Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business, 2008.
  * Hüllermeier & Waegeman (2021) Eyke Hüllermeier and Willem Waegeman.  Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods.  _Machine Learn._ , 110:457–506, 2021.
  * Huseljic et al. (2021) Denis Huseljic, Bernhard Sick, Marek Herde, and Daniel Kottke.  Separation of Aleatoric and Epistemic Uncertainty in Deterministic Deep Neural Networks.  In _IEEE Int. Conf. Pattern Recognit._ , pp. 9172–9179, Virtual Conf., 2021.
  * Kajino et al. (2012) Hiroshi Kajino, Yuta Tsuboi, and Hisashi Kashima.  A Convex Formulation for Learning from Crowds.  In _AAAI Conf. Artif. Intell._ , pp. 73–79, Toronto, ON, 2012.
  * Khetan et al. (2018) Ashish Khetan, Zachary C. Lipton, and Animashree Anandkumar.  Learning From Noisy Singly-labeled Data.  In _Int. Conf. Learn. Represent._ , Vancouver, BC, 2018.
  * Krizhevsky (2009) Alex Krizhevsky.  Learning Multiple Layers of Features from Tiny Images.  Master’s thesis, University of Toronto, 2009.
  * Kruskal (1964) Joseph B. Kruskal.  Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis.  _Psychometrika_ , 29(1):1–27, 1964.
  * LeCun & Cortes (1998) Yann LeCun and Corinna Cortes.  The MNIST database of handwritten digits, 1998.