# CrowdTeacher: Robust Co-teaching with Noisy Answers & Sample-specific
Perturbations for Tabular Data

Mani Sotoodeh111Emory University, Atlanta GA , USA Email<EMAIL_ADDRESS>Li
Xiong  Joyce Ho

###### Abstract

Samples with ground truth labels may not always be available in numerous
domains. While learning from crowdsourcing labels has been explored, existing
models can still fail in the presence of sparse, unreliable, or differing
annotations. Co-teaching methods have shown promising improvements for
computer vision problems with noisy labels by employing two classifiers
trained on each others’ confident samples in each batch. Inspired by the idea
of separating confident and uncertain samples during the training process, we
extend it for the crowdsourcing problem. Our model, CrowdTeacher, uses the
idea that perturbation in the input space model can improve the robustness of
the classifier for noisy labels. Treating crowdsourcing annotations as a
source of noisy labeling, we perturb samples based on the certainty from the
aggregated annotations. The perturbed samples are fed to a Co-teaching
algorithm tuned to also accommodate smaller tabular data. We showcase the
boost in predictive power attained using CrowdTeacher for both synthetic and
real datasets across various label density settings. Our experiments reveal
that our proposed approach beats baselines modeling individual annotations and
then combining them, methods simultaneously learning a classifier and
inferring truth labels, and the Co-teaching algorithm with aggregated labels
through common truth inference methods.

##  1 Introduction and Background

Labeled data is essential to guarantee the success of increasingly more
complex classifiers. Unfortunately obtaining large quantities of high-quality
labels can be cost-prohibitive for several fields. For example, in the medical
domains, it may take a clinician several hours to annotate the health records
of hundreds of patients. One alternative is to gather labels using
crowdsourcing, where remotely located workers are utilized to perform the task
of labeling the data. Although these crowdworkers individually may not be as
accurate as an expert, constructing the true label from their aggregated
opinions can approximate the accuracy of an expert. However, the subjectivity
of annotators and their different qualifications introduce noise to the
labeling process. To model this noise, most studies either focus on modeling
the reliability of annotators and their correlation and reflecting it in the
label aggregation phase or combining classifier training with learning the
annotators’ trust parameters. Yet, learning through crowdsourcing-based models
can still fail in the presence of differing annotations and unreliable users
[14].

A promising direction for dealing with noisy labels for training complex
classifiers is Co-teaching [5]. Under the Co-teaching paradigm, two peer
neural networks are trained separately and specific samples are exchanged
between the networks to reduce the error of the two models and yield a more
accurate model. As a result, Co-teaching methods have shown great promise for
computer vision problems with noisy labels. Co-teaching can naturally
counteract crowdsourcing noise since it filters out noisy samples in the
beginning and only adds them at later training stages when they will be
valuable. However, Co-teaching treats each sample with the same weight. This
can cause the classifier to incorrectly learn from samples that may have fewer
annotations or diverging human labels.

To address this limitation, we propose to leverage the certainty of samples
from the label aggregation phase to inform the selection process of Co-
teaching, which has not been studied before. Our model, CrowdTeacher, uses a
perturbation scheme based on the uncertainty of the samples to improve the
robustness of the Co-teaching framework. Given the availability of samples’
uncertainty from the label aggregation step, our model uses this information
to counter the inherent noise by perturbing the input space. In addition, the
framework prioritizes the more confident samples of the classifier during the
learning process. Thus, we tackle the problem of classification with features
and crowdsourcing labels using three mechanisms:

$\bullet$

Estimation of the features’ distributions to generate synthetic data which is
then used to perturb each sample in an additive manner, proportional to its
estimated label’s uncertainty.

$\bullet$

Enhancing Co-teaching by knowledge distillation, i.e. a student-teacher model
of a simple and a complex network to accommodate smaller tabular data.

$\bullet$

Utilization of the perturbed samples as input to the above classifier to
further differentiate uncertain and certain training points based on their
loss in each epoch

Next, we formally define the problem and summarize and delineate where and how
CrowdTeacher ties into the relevant literature in crowdsourcing, data
augmentation, and learning with noisy labels.

###  1.1 Problem Definition: Classification with Crowdsourcing Annotations

In practice, there are numerous applications in which the ground truth of a
classification task is not available, or disputed. For instance in medicine,
multiple pathologists do not always necessarily agree on the malignancy status
of a tumor in an image [8], or multiple nurses do not all agree on the
presence of hospital-acquired bedsores for a patient given their charts [15].
Similarly, obtaining ground truth from experts to train reliable classifiers
can be expensive, as in the case of content filtering and regulation of posts
on social media, which are distributed among multiple non-expert annotators to
obtain some good quality labels [9]. Formally, we define learning with
crowdsourcing labels as follows:

######  Definition 1.

(Classification with Crowdsourcing Annotations) Consider a set of $R$
annotators labeling $N$ samples with $K$ possible classes. Given an answer
matrix $\textbf{A}\in\mathbb{R}^{N\times R}$ where each element $a_{nr}$
indicates the label for sample $n$ provided by annotator $r$, and the training
feature matrix $\textbf{X}_{tr}\in\mathbb{R}^{N\times M}$, the goal is to
train a classifier that accurately predicts the true labels for the test data
using only its feature matrix $\textbf{X}_{ts}$.

We use ${K}$ to denote number of classes. Simulated data from the synthesizer
used for perturbation is shown by S and the perturbed samples are denoted with
$\widetilde{\textbf{X}_{tr}}$. The set of continuous and discrete features are
shown by $F_{c}$ and $F_{d}$ respectively. Table 1 summarizes the notations
used throughout this paper.

Table 1: Summary of Notations. Symbol | Description
---|---
N | Number of Samples
R | Number of Annotators
K | Number of Classes
$\alpha$ | Perturbation Fraction
$\textbf{X}_{tr}$ | Training feature matrix
A | Answer matrix of all annotators
S | Synthetic feature matrix
$\widetilde{\textbf{X}_{tr}}$ | Perturbed training samples feature matrix
$F_{c}$ | Set of continuous features
$F_{d}$ | Set of all discrete features
$\mathbf{P}$ | Class probability matrix
${c_{i}}$ | Certainty of i-th

###  1.2 Related works

Classification with noisy answers or multiple crowdsourced labels overlaps
with three other areas: learning with crowdsourcing labels, data augmentation
and synthetic data generation for robust learning, and selective gradient
propagation. Here we summarize the three main high-level approaches for
learning with multiple annotations.

Sequential. This approach first uses a truth inference method to estimate the
ground truth for training samples. The estimated label is then used to train a
classifier.

A recent survey extensively comparing these models has shown the overall
efficiency and utility of the D&S method [13]. Our proposed model falls into
this category, however, we introduce ideas from the two other overlapping
areas to further improve the predictive performance of this basic classifier.

Simultaneous. The second perspective jointly tackles the problem of learning
classifier parameters and the estimated ground truth of the samples.
Albarqouni et al. uses the Expectation-Maximization (EM) algorithm and Maximum
a posteriori estimation to iteratively compute these two sets of parameters
until convergence [1]. Yet, this method is computationally challenging
especially for more complex classifiers.

Individual annotator’s label modeling. The last set of research works entail
learning a model for each individual labeler. Dr. Net was proposed to learn a
classifier to reproduce the labels of each annotator and is composed of two
phases, individual annotator modeling and learning labelers’ averaging weights
for the final prediction [4]. To overcome the computational challenge of
simultaneous learning and Dr. Net, multiple crowd-layer variants were
introduced to remove the computational burden of the EM loop [11], by first
estimating the ground truth of samples and then attempting to replicate the
individual annotator’s labels using a very simple neural network.
Unfortunately, such models require significant samples to properly learn a
robust classifier.

####  1.2.1 Data Augmentation and Synthetic Data Generation for Robust
Learning.

To overcome the obstacle of noisy labels or features, perturbation schemes and
data augmentations have been investigated. In computer vision, data
augmentation is done by applying operations like cropping and rotation to
combat potential mislabelled training data [2, 17, 12]. Another line of work
achieves robustness against noisy data by generating data synthesizers that
achieves the same predictive performance as using the real data. Xu et al.
have extended data augmentations to tabular data with heterogeneous feature
types using Generative Adversarial Networks and Variational Autoencoders[16].
However, such synthesizers are modeled independent of the labels or the
conflicting annotations.

####  1.2.2 Selective Gradient Propagation

To counter noisy labels and memorization effects in neural networks, the Co-
teaching algorithm adaptively changes both the number of and the set of
participating samples used in stochastic gradient descent epochs for two
differently-initialized classifiers [5]. For each epoch, Co-teaching chooses a
different number of samples with the lowest loss (as a proxy for clean data)
and updates each classifier using the clean samples of the other network. This
is in contrast to using all the samples or the clean samples of the classifier
itself that may result in memorization and early overfitting which prohibits
learning a generalizable classifier. A parallel can be drawn to similarly deal
with the inherent noisiness of aggregated crowdsourcing labels. Co-teaching
mechanism of prioritizing a smaller set of confident samples in the initial
stages of learning, and gradually incorporating more of the uncertain samples
in later epochs can be leveraged for problem of classification with
crowdsourcing labels.

##  2 Methodology

Our idea is to enhance the Co-teaching framework to account for the
uncertainty associated with the estimated truth label of the sample. We
introduce a perturbation-based scheme to the Co-teaching framework so the
trained model will be more robust to sparsity and unreliability in the
annotations. For each mini-batch update of Co-teaching, synthetic samples are
generated and used to perturb each sample _dependent_ on the uncertainty of
the estimated truth label. Thus a sample that has more certainty in the label
will be perturbed more whereas a sample that has fewer annotations is likely
to have less perturbation. The perturbed sample is then used to train the
classifier.

Input: Training Features ${\textbf{X}_{tr}}$, Answer matrix A, Perturbation
Fraction $\alpha$

Output: $Model$

Train synthesizer to create synthetic data:
${Data\\_sampler}\xleftarrow[]{}Synthesizer(\textbf{X}_{tr})$

Generate $N$ samples from resulting sampler:
$\textbf{S}\xleftarrow[]{}Data\\_sampler(N)$

Run truth inference model to get class probabilities:
$\textbf{P}\xleftarrow{}D\&S\\_Algorithm(\textbf{A})$

/* Generate perturbed samples $\widetilde{\textbf{X}_{tr}}$ */

for  _$i=1,\cdots,N$_ do

Set sample’s certainty using Eq. (1)

Sample $s_{i}$ from 10% closest samples of synthetic samples S to ${x}_{i}$
using KNN

/* Generate continuous features */

for  _$j\in F_{c}$_ do

Generate feature $\widetilde{{x}}_{ij}$ according to Eq. (2)

end for

/*Generate discrete features*/

Calculate $f^{i}_{d}$ using Eq. (3)

Sample discrete features to perturb: $F^{i}_{d_{p}}$ from $F_{d}$ such that
$|F^{i}_{d_{p}}|=f^{i}_{d}$

for  _$j\in F^{i}_{d_{p}}$_ do

Generate single feature value $\widetilde{{x}}_{ij}$ according to Eq. (4)

end for

end for

Train Co-teaching on Perturbed Samples:
$Model\xleftarrow[]{}Co\\_teaching(\widetilde{\textbf{X}_{tr}})$

Algorithm 1 CrowdTeacher.

###  2.1 Generating Synthetic Samples

To improve the robustness of the Co-teaching framework, CrowdTeacher generates
synthetic samples of the data which are then used to perturb the samples to
train the classifier. Any data synthesizer with reasonable data generation
performance can be used. For the purpose of our paper, we focus on three data
synthesizers: Conditional GAN (CTGAN) [16], TVAE [16] and Gaussian copula
[10]. CTGAN can handle mixed feature types (discrete and continuous) and has
been shown to perform competitively with other GAN-based, VAE-based, and
Bayesian network-based data synthesizer for vision benchmark datasets [10]. It
is worthwhile to note that the data synthesizer is not tied to the learning
task and can be used as a stand-alone tool.

To generate synthetic data within CrowdTeacher, the training feature matrix
$\textbf{X}_{tr}$ is fed to the synthesizer. For CTGAN synthesizer, the
discrete features $F_{d}$ are specified explicitly since they are modeled
differently compared to the continuous features $F_{c}$. Once the synthesizer
has estimated the data distribution, any number of samples can be drawn.

For CrowdTeacher, we generate the synthetic set
$\textbf{S}\in\mathbb{R}^{N\times M}$ with $N$ synthetic samples once and
assume each synthetic sample can serve as a unique perturbation source.
Although S is drawn once and is the same size as our training data to minimize
the computational footprint of our model, the synthetic set can be re-drawn at
each mini-batch of the Co-teaching framework with a larger number of samples.

###  2.2 Sample-specific Perturbations

The generated synthetic samples, S, fail to account for the uncertainty
associated with the estimated sample label as the synthetic samples are only
dependent on original training data. Thus, we introduce a mechanism to
leverage the uncertainty that arises from the truth inference method to
individually perturb each sample. For the purpose of illustration and
experimentation, we focus on the D&S algorithm [3], but note that CrowdTeacher
can be used with any robust truth inference method that quantifies the label
uncertainty for each sample. The D&S algorithm takes as an input the matrix of
annotations (A) and models annotators by a confusion matrix to capture their
chance of mistaking one class for another or correctly reporting them in
addition to the class priors. D&S outputs a matrix
$\mathbf{P}\in\mathbb{R}^{N\times K}$, where the $P_{ik}$ element denotes the
probability that sample $i$ is of class $k$. The certainty of each sample,
$c_{i}$, is then defined as the maximum probability across all the classes:

$\centering c_{i}=\max_{k\in{K}}{(P_{ik})}\quad\forall i\in N\@add@centering$
(1)

Choosing an appropriate simulated sample for perturbation. Given the data
synthesizer can generate synthetic samples that are quite different from the
original data point and can lead to more uncertainty with respect to the truth
label, we use k-nearest neighbors (KNN) to identify reasonable close samples
from $S$. For each sample, KNN is run to find the top 10% closely simulated
samples. A simulated data point, $s_{i}$, is then randomly chosen from this
top 10% and used to perturb the original point.

Perturbation. Each sample $x_{i}$ is perturbed using the simulated data point
$s_{i}$ according to the uncertainty, $c_{i}$ and a user-specified
perturbation fraction $\alpha\in[0,1]$ to obtain the perturbed sample
$\widetilde{{x}}_{i}$.

Let $s_{ij}$ represent the $j^{th}$ feature of sample $s_{i}$. If the $j^{th}$
feature is continuous, the value for the synthetic, perturbed sample
$\widetilde{{x}}_{ij}$ is a convex combination of the original and simulated
sample:

$\centering\widetilde{{x}}_{ij}=({1-{\alpha}c_{i}}){{x}_{ij}}+({{\alpha}c_{i}}){{s}_{ij}},\quad\forall
i\in N,\quad\forall j\in{{F}_{c}}\@add@centering$ (2)

For the discrete features, we use $c_{i}$ and $\alpha$ to calculate the number
of discrete features to swap. Let $|F_{d}|$ denote the number of discrete
features in the dataset, then the number of discrete features to swap for each
sample $x_{i}$, $f^{i}_{d}$ is calculated as:

$\centering f^{i}_{d}=round({\alpha}c_{i}|{F}_{d}|)\@add@centering$ (3)

Then $f^{i}_{d}$ features are randomly selected for perturbation from the
original discrete feature set and denoted as $F^{i}_{d_{p}}$. For each
feature, $j$ in this perturbation set, the feature values are replaced with
the synthetic sample value $s_{ij}$.

$\centering\widetilde{{x}}_{ij}={{s}_{ij}},\quad\forall i\in N,\quad\forall
j\in F^{i}_{d_{p}}\@add@centering$ (4)

###  2.3 Knowledge distillation-based Co-teaching for Smaller Tabular Data

To combat the large performance variations associated with running the Co-
teaching algorithm on smaller-sized tabular data, we incorporated the student-
teacher idea from knowledge distillation [6]. Thus instead of two peer
networks with the same architecture, we used one simple and one complex
network such that the number of hidden units of the simpler network is half of
the other one. Empirical results showed these modifications helped with both
the convergence of the two networks in achieving more similar evaluation
metrics and overall better performance across different synthetic datasets.

##  3 Experiments

###  3.1 Baseline Methods

The best performing methods from crowdsourcing studies (see Sec. 1.2) are
chosen as comparison models. The original Co-teaching algorithm and Co-
teaching using only uniformly perturbed input are also used to illustrate the
advantage of certainty-aware perturbation. All methods employ the same base
classifier, a neural network with one hidden layer of
$\frac{|{F}_{c}|+|{F}_{d}|}{4}$ units. Sequential methods share the same truth
inference method (D&S) and are marked with *.

  * •

Naive baseline* (Base_clf) [3]: Base classifier trained with D&S labels.

  * •

Simultaneous Expectation Maximization (S-EM) [1]: An algorithm that jointly
learns the classifier and annotators’ parameters using EM algorithm.

  * •

Dr. Net [4]: An individual annotation based model that separately learns each
annotator’s labels and their weights.

  * •

Crowdlayer (CL_MW and CL_VW) [11]: An algorithm that estimates ground truth
first and replicates each annotator’s labels via a simple final layer. This
final layer is removed at test time. The number of parameters for the last
layer determines the Crowdlayer variant. We evaluated the vector of weights
(VW) and matrix of weights (MW) variants.

  * •

Vanilla Co-teaching* (V_Coteach) [5]: The original Co-teaching algorithm
trained with D&S labels.

  * •

Co-teaching with uniform perturbation* (P_Coteach): The Co-teaching algorithm
trained on D&S labels and synthetic samples.

  * •

CrowdTeacher*: Our proposed method with the Co-teaching algorithm trained on
D&S labels and sample-specific certainty-informed perturbed samples.

We conducted our experiments using these baseline models. Since S-EM and Dr.
Net constantly performed poorly compared to the other baselines, we omitted
them from the plots for better readability. The Python implementation for all
our experiments is publicly available on GitHub.

###  3.2 Annotation Simulation

For our experiments, we set the number of annotators to be 5 ($R=5$). To
simulate the annotators’ behavior, we consider two parameters: (1) mean
reliability, or the average likelihood of the annotators to label a positive
sample correctly and (2) variability in annotators’ expertise or the
difference in their qualities. We set the distribution of samples having 1 to
5 labels as [$\tau$ , 0.55*($1-\tau$), 0.27($1-\tau$), 0.13($1-\tau$),
0.05($1-\tau$)] and vary the parameter $\tau$ for our experiments. Note that
$\tau$ determines the average number of labels per sample. Conventionally, the
Beta distribution is used to generate each annotator’s reliability. After
determining each annotator’s reliability, its labels are created by randomly
choosing (100-reliability) percent of positive cases and switching their
labels into negative 0. Flipping negative samples to positive occurs at 0.01
times this rate. Samples not assigned to specific annotators are marked with
$-1$ in the answer matrix (A). The exact parameters used for simulating
annotations in each experiment are summarized in the GitHub repository.

###  3.3 Datasets

####  3.3.1 Synthetic Datasets

To test the performance of our framework on a non-specific dataset for which
the ground truth is known, we generated synthetic data to mimic real-world
features and a range of annotator reliabilities.

_Statistical distribution families:_ Families of continuous and discrete
distributions were used to generate the synthetic data. In particular, we used
Normal, Beta, Wald, Laplace, Binomial, Multinomial, Geometric and Poisson
distributions. The corresponding distribution parameters for a feature within
each family are randomly chosen from a specified range. 5 features were chosen
from each family for a total of 40 features.

_Output:_ The ground truth labels are determined based on a polynomial
combination of feature values. Each feature’s coefficient value is chosen
randomly. To assign labels and model class balance (% of positive samples),
outputs falling in percentiles below the level of balancedness are assigned to
the positive class.

_Noise level_ : Two versions of labels are generated. Labels for a specified
percentage of samples are flipped to obtain the noisy truth used for
annotation generation. However the true labels before flipping are used for
evaluation purposes. This resembles the availability of noisy labels in
practice.

####  3.3.2 PUI Dataset

Determining whether a patient has developed a pressure ulcer injury (bedsore)
is a complex clinical decision that requires considerable nursing expertise.
Early detection of PUI is extremely useful since it is preventable with proper
care. However, even highly trained nurses do not agree on the existence or
severity of PUI cases. Training a classifier that utilizes a limited set of
annotated health records from multiple nurses can revolutionize nursing care
through use in similar clinical settings. We use the MIMIC-III dataset [7], a
publicly available dataset which holds information of patients admitted to
intensive care units (ICU) of a populated tertiary care hospital from 2001 to
2012. We identified hospital stays of individuals over 20 years old with
length of stays between 2 days and 120 days. A hospital stay was considered
positive if there was a presence of the ICD-9 diagnosis code associated with
pressure ulcer and there was a mention of PUI in the notes. A hospital stay
was negative if there was no indication of PUI in both the ICD-9 codes or the
notes.

A total of 10518 samples were identified, 31% of which are positive.

##  4 Results

Since the datasets are imbalanced, we evaluate all the models based on the
area under the precision recall curve (AUPRC). AUPRC offers a holistic picture
of CrowdTeacher’s predictive performance, independent of the classification
threshold choice. We split each dataset into 80% training & 20% test. The
AUPRCs in plots are averaged across multiple seeds. We also confirmed
CrowdTeacher performance on AUROC metric, but omit the results due to limited
space.

###  4.1 Synthetic Dataset

####  4.1.1 Sensitivity to choice of synthesizer

To analyze the effect of using different synthesizers on CrowdTeacher
performance, we compared the average gain obtained by using CrowdTeacher with
CTGAN, TVAE, and Gaussian copula synthesizers compared to using the next two
top-performing baseline methods of P_Coteach and V_Coteach, respectively shown
by circle and cross markers in Figure 1(b). Firstly, we can see that Gaussian
copula has the greatest gain among the three synthesizers. However, employing
the two other synthesizers for CrowdTeacher would still be beneficial in terms
of predictive performance in many of the sparsity settings. Given the
promising performance of Gaussian copula synthesizer, we use Gaussian copula
for all the remaining experiments.

####  4.1.2 Sensitivity to perturbation fraction ($\alpha$)

To understand the impact of the perturbation fraction, $\alpha$, we varied it
between [0.01, 0.2] and evaluated the performance of CrowdTeacher and
P_Coteach (the two perturbation-based methods). Figure 1(a) shows the average
AUPRC of P_Coteach and CrowdTeacher as $\alpha$ increases with the average
number of labels set to 2.34. It is observed that CrowdTeacher constantly
outperforms P_Coteach regardless of the chosen perturbation fraction
indicating its robustness. From the results, there is an optimal range of
$\alpha$ to achieve the greatest benefit from CrowdTeacher and that either a
very low ($\alpha\leq 0.05$) or very high ($\alpha\geq 0.2$) perturbation
fraction decreases the usefulness of CrowdTeacher but does not diminish it.
Given these results, the remainder of our experiments uses $\alpha=0.11$.

(a) Perturbation fraction.

(b) Different synthesizers.

Figure 1: CrowdTeacher Sensitivity to perturbation fraction and synthesizer
choice (in Figure 1(b) circles/crosses show gain w.r.t. P_Coteach/V_Coteach
accordingly)

####  4.1.3 Predictive Performance

Figure 2(a) shows the performance of baseline crowdsourcing and Co-teaching
variants against CrowdTeacher across various sparsity settings on the
synthetic dataset. Confirming intuition, all methods experience an increase in
AUPRC since the average number of labels per sample increases, which exposes
methods to less noisy annotation. All Co-teaching based methods (CrowdTeacher,
V_Coteach, and P_Coteach) constantly outperform both crowdlayer variants and
also Dr.Net and S-EM. The last two always performed the worst and therefore
were excluded from these plots. Even though the base classifier performance
improves with more labels, its performance gap with Co-teaching based methods
remains large in all sparsity settings. Across a wide range of label
sparsities, using CrowdTeacher results in a significant boost in AUPRC,
compared to the other two Co-teaching based methods, even with as low as only
1.68 labels per sample. Also, we can observe that V_Coteach performs worse
than P_Coteach in very sparse settings (average number of labels $<$ 2.1), but
as the number of labels increases it catches up with P_Coteach and even
surpasses it at higher densities. Another interesting observation is that
beyond an average of 2 labels per sample, all three methods reach a plateau
and only improve negligibly in response to an increased number of labels.

(a) Synthetic dataset.

(b) PUI dataset.

Figure 2: CrowdTeacher Performance on Synthetic and PUI data as average number
of labels per sample increases, averaged on 10 and 4 initializations
respectively.

###  4.2 PUI Dataset

To challenge CrowdTeacher’s performance under more chaotic distributions of
real data, we tested it on the bedsore detection task with 10k samples. Figure
2(b) shows how the performance of the chosen methods changes as the average
number of labels per sample goes up. We observed similar patterns to synthetic
dataset here too in terms of Co-teaching variants’ overall predictive
advantage over other methods, however, the gap between Co-teaching variants
and other methods is less substantial. The range of AUPRC of all models on
this dataset proves that this is a much harder learning problem, yet
CrowdTeacher is able to beat P_Coteach and V_Coteach at multiple points,
especially at lower sparsities, which are actually more practical for
obtaining labels for hospital-acquired bedsores, while at other sparsity
points it has comparable performance to these methods.

##  5 Conclusion

We proposed CrowdTeacher, a novel Co-teaching based approach that leverages
certainty of samples from truth inference algorithms to apply sample-specific
perturbations on training points, and combines it with Co-teaching algorithm
to further rectify noisy annotations and incorporate that knowledge in the
training process. Our proposed approach bridges overarching themes and ideas
from data augmentation, crowdsourcing, and learning with noisy labels and is
agnostic to the truth inference method and the synthesizer used. To illustrate
the predictive benefits of CrowdTeacher over similar methods, we conducted
experiments on both synthetic and real dataset of different scales, and our
results for both tasks (including a real-world medical classification task)
confirmed CrowdTeacher’s performance edge for learning with crowdsourced
labels. We also successfully employed Co-teaching mechanism primarily tested
on images, for tabular data. For our future work, we plan to propose new
perturbation schemes to introduce more variety for perturbations of a given
sample during training, and extend our current framework to semi-supervised
learning.

## References

  * [1] S. Albarqouni, C. Baur, F. Achilles, V. Belagiannis, S. Demirci, and N. Navab.  Aggnet: deep learning from crowds for mitosis detection in breast cancer histology images.  IEEE transactions on medical imaging, 35(5):1313–1321, 2016.
  * [2] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel.  Mixmatch: A holistic approach to semi-supervised learning.  In Advances in Neural Information Processing Systems, pages 5049–5059, 2019.
  * [3] A. P. Dawid and A. M. Skene.  Maximum likelihood estimation of observer error-rates using the em algorithm.  Applied statistics, pages 20–28, 1979.
  * [4] M. Y. Guan, V. Gulshan, A. M. Dai, and G. E. Hinton.  Who said what: Modeling individual labelers improves classification.  arXiv preprint arXiv:1703.08774, 2017.
  * [5] B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama.  Co-teaching: Robust training of deep neural networks with extremely noisy labels.  In Advances in neural information processing systems, pages 8527–8537, 2018.
  * [6] G. Hinton, O. Vinyals, and J. Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [7] A. E. Johnson, T. J. Pollard, L. Shen, H. L. Li-wei, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. A. Celi, and R. G. Mark.  Mimic-iii, a freely accessible critical care database.  Scientific data, 3:160035, 2016.
  * [8] P. Mobadersany, S. Yousefi, M. Amgad, D. A. Gutman, J. S. Barnholtz-Sloan, J. E. Velázquez Vega, D. J. Brat, and L. A. D. Cooper.  Predicting cancer outcomes from histology and genomics using convolutional networks.  Proceedings of the National Academy of Sciences, 115(13):E2970–E2979, 2018.
  * [9] V.-A. Nguyen, P. Shi, J. Ramakrishnan, U. Weinsberg, H. C. Lin, S. Metz, N. Chandra, J. Jing, and D. Kalimeris.  CLARA: Confidence of Labels and Raters, page 2542–2552.  Association for Computing Machinery, New York, NY, USA, 2020.
  * [10] N. Patki, R. Wedge, and K. Veeramachaneni.  The synthetic data vault.  In 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA), pages 399–410, Oct 2016.
  * [11] F. Rodrigues and F. Pereira.  Deep learning from crowds.  32(1), 2018.
  * [12] N. Soans, E. Asali, Y. Hong, and P. Doshi.  Sa-net: Robust state-action recognition for learning from observations.  In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 2153–2159. IEEE, 2020.
  * [13] F. Tahmasebian, L. Xiong, M. Sotoodeh, and V. Sunderam.  Crowdsourcing under data poisoning attacks: A comparative study.  In A. Singhal and J. Vaidya, editors, Data and Applications Security and Privacy XXXIV, pages 310–332, Cham, 2020. Springer International Publishing.
  * [14] F. Tahmasebian, L. Xiong, M. Sotoodeh, and V. Sunderam.  Edgeinfer: Robust truth inference under data poisoning attack.  In 2020 IEEE International Conference on Smart Data Services (SMDS), pages 45–52, 2020.
  * [15] S. M. Waugh and S. Bergquist-Beringer.  Inter-rater agreement of pressure ulcer risk and prevention measures in the national database of nursing quality indicators (ndnqi).  Research in nursing & health, 39(3):164–174, 2016.
  * [16] L. Xu, M. Skoularidou, A. Cuesta-Infante, and K. Veeramachaneni.  Modeling tabular data using conditional gan.  In Advances in Neural Information Processing Systems, pages 7335–7345, 2019.
  * [17] Z. Zhang, H. Zhang, S. O. Arik, H. Lee, and T. Pfister.  Distilling effective supervision from severe label noise.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9294–9303, 2020.