# Generating the Ground Truth:
Synthetic Data for Label Noise Research

Sjoerd de Vries1,2, Dirk Thierens1

###### Abstract

Most real-world classification tasks suffer from label noise to some extent.
Such noise in the data adversely affects the generalization error of learned
models and complicates the evaluation of noise-handling methods, as their
performance cannot be accurately measured without clean labels. In label noise
research, typically either noisy or incomplex simulated data are accepted as a
baseline, into which additional noise with known properties is injected. In
this paper, we propose SYNLABEL, a framework that aims to improve upon the
aforementioned methodologies. It allows for creating a noiseless dataset
informed by real data, by either pre-specifying or learning a function and
defining it as the ground truth function from which labels are generated.
Furthermore, by resampling a number of values for selected features in the
function domain, evaluating the function and aggregating the resulting labels,
each data point can be assigned a soft label or label distribution. Such
distributions allow for direct injection and quantification of label noise.
The generated datasets serve as a clean baseline of adjustable complexity into
which different types of noise may be introduced. We illustrate how the
framework can be applied, how it enables quantification of label noise and how
it improves over existing methodologies.

##  1 Introduction

Classification models are of great interest to the research community and
machine learning practitioners alike. When applied to real-world problems,
these models are confronted with noisy data, with noise defined as anything
that obscures the relationship between the dependent and independent variables
(Hickey 1996). Label noise can have detrimental effects on classifier
performance, model complexity, learning rates and effect size estimation
(Frénay and Verleysen 2014). Therefore, a lot of research is conducted into
prediction methods that are robust to such noise and into pre-processing steps
for filtering label noise from data (Frénay and Verleysen 2014; Song et al.
2022).

When algorithms are evaluated with regard to their ability to handle label
noise, typically, existing real-world datasets are assumed to be the ground
truth, after which artificial noise is injected into the labels (Berthon et
al. 2021; Cheng et al. 2020). Alternatively, data is entirely simulated, often
by modelling relatively simple relationships between the dependent and
independent variables (Berthon et al. 2021; Cheng et al. 2020). Lastly, a
limited number of curated datasets are publicly available for which the label
noise has been quantified by expert annotators (Xiao et al. 2015; Lee et al.
2018; Li et al. 2017).

Running experiments using any of the above types of datasets has drawbacks
associated with it: either the relationships in the data are not sufficiently
complex and therefore not realistic (simulated data), or there are no clean
labels available to evaluate on and noise has to injected into already noisy
data (real-world data), or the noise cannot be tailored, such that any method
is only tested on a very specific noise pattern (curated data). Creating a
curated dataset takes considerable effort as well. While noisy real-world data
are available in abundance, the lack of a clean evaluation set cannot be
easily overcome. In (Frénay and Verleysen 2014) it is stated, in the context
of method comparison, that the presence of label noise in the validation data
causes any estimates to be off by an unknown amount. The authors mention this
as an important open research question, one which we address in this work.

In this paper we aim to improve upon the aforementioned experimental
strategies. We present the SYNLABEL framework: Synthetic Labels As Baseline
for Experiments with Label noise. SYNLABEL facilitates the construction of
artificial tabular datasets for performance evaluation of methods dealing with
label noise. We propose to define a pre-specified or learned function as the
ground truth relationship. Then, by applying this ground truth function to any
input data contained in its domain, noiseless labels are generated. As we are
interested in testing methods on known noise, rather than finding the best
model for a specific real-world problem, this function is not required to
exactly represent the original data. The generated ground truth set can be
further transformed into a partial ground truth set for which each data point
is accompanied by a soft label: first, a number of specified variables from
the ground truth dataset are hidden. Then, by learning or specifying a
(conditional) prior distribution, resampling values from it and combining
these sampled values with fixed values for the variables that were not hidden,
a posterior distribution is generated via the ground truth relationship.
Although the prior distribution over the hidden variables is almost guaranteed
no to match the exact underlying distribution of the real-world generative
distribution over the original dataset, as before this is not necessary for
our purpose. The advantage of constructing a set with soft labels compared to
dealing with hard labels is that it allows for explicit quantification and
direct injection of label noise. Furthermore, these sets can be used for
problems from different domains that can be mapped to a label distribution
problem, e.g. learning from crowds or with confidence scores. The sets with
clean hard or soft labels serve as a starting point from which further
transformations can be applied to the data in order to generate the specific
noise of interest, allowing for any such added noise to be quantified.

In summary, the key contributions of this paper are:

  * •

The SYNLABEL framework which facilitates the generation of experimental
datasets for label noise research.

  * •

A method for constructing a ground truth dataset informed by real-world data
for the purpose of evaluation.

  * •

A method for converting hard into soft labels by resampling values for
features that are hidden from the model.

  * •

An analysis showing the advantages of using soft labels for quantification and
injection of label noise.

##  2 Related Work

Systematic experiments in label noise research require a dataset with both
clean labels as a baseline to evaluate on, as well as noisy labels. Based on
how these sets are obtained, the experiments in the label noise field can be
placed into three categories: (1) an existing dataset for which the labels
have been manually corrected is used, or artificial noise is injected into the
labels of either (2) a clean dataset which has been simulated or (3) an
existing real-world dataset.

The first type of experiment uses a curated real-world dataset for which noisy
labels have been corrected, such as Clothing1M (Xiao et al. 2015), Food-101N
(Lee et al. 2018) and WebVision (Li et al. 2017). The ground truth labels are
generally decided upon by a panel of experts. Nevertheless, even among experts
high inter-observer disagreement may occur (Veta et al. 2016).

In both the second and third category of experiments, first a baseline is
established after which noise is injected into the labels. This injected noise
is constructed based on a selected noise model. In order of decreasing
commonality and increasing complexity, the injected noise can be classified as
(Frénay and Verleysen 2014): Noisy Completely At Random (NCAR) (Angluin and
Laird 1988), Noisy At Random (NAR) (Lawrence and Schölkopf 2001) or Noisy Not
At Random (NNAR) (Jin and Ghahramani 2002; Chen et al. 2021; Xia et al. 2020;
Zhang et al. 2021; Garcia et al. 2019).

When synthetic data is constructed, the true underlying function is known by
design, e.g. data is sampled from Gaussian distributions or constructed using
rule-based generation (Hickey 1996). Clean labels are then generated from this
function, to which noise can be added. The largest downside is that these
simulated sets generally lack the complex interactions that one expects
between variables in real data.

In case of real-world data, when there are no resources available to curate
the sets, the true labels remain unknown and thus the level of natural noise
cannot be quantified (Hickey 1996). The importance of using controllable
artificial data, especially in the context of noise, was already mentioned in
(Langley 1988), as it enables systematic research into different aspects of a
domain. While the use of real-world data is the least labour intensive
experimental method, the effects of any further added label noise cannot be
separated from the inherent noise present and thereby the overall noise level
cannot be sufficiently controlled.

In summary, while different types of label noise experiments exists, they each
suffer from shortcomings. Furthermore, when using hard labels, noise cannot be
specified for any individual label, beyond the label being correct or not.

Recently, (Gu et al. 2022) proposed a framework for generating instance-
dependent label noise. Our work differs from theirs in a few ways: they
introduce a particular type of classifier based instance-dependent noise,
whereas our framework allows for any type of noise injection. Furthermore, a
dataset to be used with their work requires the labels to be clean, while we
present methods for generating such data with either clean hard or soft
labels.

##  3 The SYNLABEL Framework

We present the Synthetic Labels As Baseline for Experiments with Label noise
(SYNLABEL) framework, shown in Figure 1, which facilitates generating
synthetic tabular datasets for use in label noise experiments.

The SYNLABEL framework defines different types of datasets and transformations
between them. Each dataset consists of input variables $X$, labels $y$ and a
functional relationship between the two, $y=f(X)$. The two types of ground
truth datasets depicted in the Unobservable part of Figure 1, are generally
unobtainable for real-world problems, as their functional relationship is
defined to be exact, i.e. noiseless. In practice, an Observable dataset is
available and often the task at hand is precisely to discover a relationship
between $X$ and $y$ that generalizes well. The datasets can be further
categorized based on whether the output is a single unambiguous class, also
known as a hard label, or a discrete probability distribution over the label
space, commonly referred to as a soft label.

The user is encouraged to utilize the framework to construct a noiseless
ground truth dataset based on a known, possibly learned, functional
relationship. Any further noise applied to this dataset can be quantified
exactly for each individual data point. This allows for analysing method
performance on a specific type of noise in isolation. We stress that SYNLABEL
is not meant for optimizing models for a specific dataset, but rather for
evaluating and comparing methods in the presence of label noise. In the
following we describe the different components of the framework.

### Notation

In the SYNLABEL framework, a dataset $D$ is made up of objects $o$ denoted by
$D:o_{i}=\\{X_{i},y_{i}\\}$. The framework is meant for any deterministic
classification task:

###### Definition 1

A deterministic classification task is a task for which, given that all
required information ($X^{G}$) to the outcome $y^{G}$ is available, there is a
true deterministic function $f^{G}(X^{G})=y^{G}$.

In other words, given that we know all of the relevant information to the task
$X_{i}$, $y_{i}$ is unambiguously assigned one true class through the function
$f^{G}$ for all $y_{i}\in y^{G}$. A dataset with corresponding classification
task for which Definition 1 holds is defined as a Ground Truth ($G$) dataset
$D^{G}$:

###### Definition 2

A Ground Truth ($G$) dataset
$D^{G}:o_{i}=\\{X^{G}_{i},f^{G}(X_{i}^{G})=y_{i}^{G}\\}$ is a dataset for
which any input $X^{G}_{i}$ in the domain of $f^{G}$ is mapped to its
deterministic hard label $y^{G}_{i}$ by the true function $f^{G}$.

Note that a dataset almost never satisfies this definition unless it is
simulated. SYNLABEL offers the tools to generate such a dataset based on a
noisy dataset.

When not all features required for deterministic classification are available,
yet the true classification function is known, the dataset is referred to as a
Partial Ground Truth ($PG$) dataset $D^{PG}$, defined as:

###### Definition 3

A Partial Ground Truth ($PG$) dataset
$D^{PG}:o_{i}=\\{X^{PG}_{i},f^{G}(X^{PG}_{i})=y^{PG}_{i}\\}$ is a dataset for
which any input $X^{PG}_{i}$ in the domain of $f^{G}$ is mapped to its soft
label $y^{PG}_{i}$ by the true function $f^{G}$.

Here $X^{PG}\subset X^{G}$ contains the available data for the task, while any
unavailable features are contained in $X^{PG^{\prime}}\subset X^{G}$ such that
$X^{PG}\cup X^{PG^{\prime}}=X^{G}$. Since some of the information
($X^{PG^{\prime}}$) needed for an exact classification is missing,
$f^{G}(X^{PG})$ produces discrete label distributions or soft labels $y^{PG}$,
with quantifiable uncertainty, instead of hard labels.

In the (Partial) Ground Truth sets the input $X$ is mapped to the output $y$
by the true underlying function $f^{G}$. Such datasets with corresponding
mapping are practically not obtainable in real life, i.e. they are
unobservable.

Data that is observed in practice, $D^{O}$, has different characteristics:
noise can be present in both the observed input data $X^{O}$, as well as in
the corresponding label $y^{O}$. Often, $y^{O}$ is not measured directly and
is instead annotated by an expert or system, based on their own non-
deterministic, noisy functional relationship $f^{E}$. Such an expert may have
the same information available as is available for the classification task,
i.e. $X^{E}\subseteq X^{O}$, or additional relevant information
$X^{O^{\prime}}$ could be available: $X^{E}\subseteq(X^{O}\cup
X^{O^{\prime}})$.

An annotator, either implicitly or explicitly, assigns probabilities to the
different candidate labels corresponding to an object $o_{i}$, producing a
Observed Soft Label (OS) set with corresponding outcome $y^{OS}$. This set is
often discretized into a Observed Hard Label (OH) set with label $y^{OH}$
based on some decision function $f_{dec}$. Depending on whether the
intermediate label distribution is preserved ($OS$) or not ($OH$), the final
dataset becomes either $D^{OS}$:

###### Definition 4

An Observed Soft Label ($OS$) dataset
$D^{OS}:o_{i}=\\{X^{O}_{i},y^{OS}_{i}\\}$ is a dataset for which the input
$X^{O}$ is associated with soft labels $y^{OS}$.

or more commonly $D^{OH}$:

###### Definition 5

An Observed Hard Label ($OH$) dataset
$D^{OH}:o_{i}=\\{X^{O}_{i},y^{OH}_{i}\\}$ is a dataset for which the input
$X^{O}$ is associated with hard labels $y^{OH}$.

Note that there are no function-related requirements for these sets. While a
function $f^{O}$ may be learned from these data, it is nearly guaranteed not
to match the true functional relationship for the corresponding deterministic
classification task. An overview of the different datasets defined in
Definition 2-5 is presented in Table 1.

Figure 1: A schematic overview of the SYNLABEL framework. The white boxes
represent data, either input $X$ or labels $y$. The gray boxes represent a
type of dataset, which includes a function relating the input to the output,
although this function may not be defined in case of observable data. The
arrows represent the different transformations and functions as specified
above. $Rs$: Resampling. $f()$: a function.

##  4 Data Transformations

At the core of the SYNLABEL framework lie the different operations that enable
a user to transform the datasets from one type to another. These allow a user
to obtain both a (Partial) Ground Truth dataset for validation as well as
realistic datasets that contain the specific type of label noise for which an
experiment is to be conducted.

The different types of datasets can be transformed in two directions. The
direction in which data is most often transformed, to obtain observed sets
containing varying label noise, is down the chain: Ground Truth
$(G)\rightarrow$ Partial Ground Truth $(PG)\rightarrow$ Observed Soft Label
$(OS)\rightarrow$ Observed Hard Label $(OH)$. Transformations in this
direction ensure that the objects that are contained in each set remain
coupled, i.e. two objects $o_{i}$ in different sets following a transformation
down the chain still represent the same entity and remain linked to the
original relationship that governs the ground truth data from which they
originate. The reverse direction, up the chain, is made possible by using
learned functions. In this case, however, a distinct dataset is generated,
that is, the objects in the original set are not the same as those in the
transformed set, since a new ground truth relationship is defined. The objects
become decoupled.

Objects remain coupled following an arbitrary number of transformations both
up and down the chain only when all sets are identical, thus
$D^{G}=D^{PG}=D^{OS}=D^{OH}$. This implies that the soft labels, $y^{PG}_{i}$
and $y^{OS}_{i}$, have probability 1 for the class in $y^{OH}_{i}$ and
$y^{G}_{i}$ and probability 0 for the other classes and that the true
functional relationship $f^{G}$ required for the (Partial) Ground Truth is
known.

The transformations and functions and their corresponding types are shown in
Figure 1. A transformation can either be an identity transformation, through
which the data are not altered, or a non-identity transformation which adds
noise to the data. A function can either be a true function or some other
function, e.g. a learned function or specified decision function, denoted by
any function.

### Down the chain

In the following we describe the supported transformations down the chain.
These transitions allow the objects $o$ to remain coupled between the
different datasets.

#### From Ground Truth to Partial Ground Truth

An identity transformation exists between $X$ in $D^{G}$ and in $D^{PG}$. The
variables contained in $X^{G}$ have identical values in $D^{PG}$, but they may
be split into two sets, $X^{PG}$ and $X^{PG^{\prime}}$. Additionally, the
function $f^{G}$ describes the true relationship between $X$ and $y$:
$f^{G}(X^{G}_{i})$ is equal to the true class of object $o_{i}$ with absolute
certainty. $f^{G}(X^{PG}_{i})$ is equal to the true soft label of $o_{i}$,
given that some information contained in $X^{G}_{i}$ is now contained in
$X^{PG^{\prime}}_{i}$ and thus missing from $X^{PG}_{i}$, whereby the
classification task becomes ambiguous.

To obtain $D^{PG}$, $X^{PG^{\prime}}$ cannot simply be ignored and a function
$f^{PG}$ learned on $\\{X^{PG},y^{G}\\}$, as the resulting function would not
equal the original relationship $f^{G}$. This follows from the fact that the
variables in $X^{PG}$ would have to be irrelevant to the task and such
variables are not contained in $X^{G}$ by definition, and thus
$X^{PG^{\prime}}$ by extension. The exception is when $X^{PG^{\prime}}$ is
empty, resulting in the special case $X^{PG}=X^{G}$ and $D^{G}=D^{PG}$.
Otherwise, the missing information from $X^{PG^{\prime}}$ will cause a
different function to be learned. The labels produced by $f^{PG}$ are then
guaranteed to differ from $y^{G}$ for some objects in the domain of $f^{G}$.
Two such labels must both be the true label for such an object, which is
contradictory, hence the sets become decoupled.

In addition to the identity transformation, $X^{PG}=X^{G}$, there is a
transformation that we call feature hiding, which preserves the coupling
between the objects in both sets when $X^{PG}\neq X^{G}$ and which preserves
the truth relationship $f^{G}$. Instead of trying to learn a function from the
known features $X^{PG}$, as before, we use $f^{G}$ and the information we have
about the missing features in $X^{PG^{\prime}}$ to construct $y^{PG}$ as
follows: first we resample a number of values $j$ for the features contained
in $X^{PG^{\prime}}$ for each object $i$ in accordance with a probability
density function which was constructed based on $X^{G}$. This can be either a
conditional density function, $P(X^{PG^{\prime}}|X^{PG}$) or even
$P(X^{PG^{\prime}}|X^{PG}\cup y^{G}$), or a marginal density function,
$P(X^{PG^{\prime}})$. These multiple sampled values $X^{PG^{\prime}}_{i,j}$
are then joined with the known values $X^{PG}_{i}$ for each object $o_{i}$
after which $f^{G}$ is applied to all combinations of resampled and known
values to obtain a corresponding label. We then aggregate the obtained labels
into a soft label:

$y^{PG}_{i,c}=\sum\limits^{n}_{j=1}\frac{\mathds{1}_{c}(f^{G}(X^{PG^{\prime}}_{i,j}\cup
X^{PG}_{i}))}{n},$ (1)

which returns the probability of class $c$ for object $i$, with
$\mathds{1}_{c}$ the indicator function. As $n\rightarrow\infty$, an exact
soft label for $y^{PG}_{i}$ is obtained given the density function.

The values in $X^{G}$ could be sampled from an infinite number of
distributions. Therefore, it is impossible to determine the exact prior
distribution for $X^{PG^{\prime}}$ from which we should resample. On the other
hand, any distribution from which $X^{G}$ could possibly be sampled is a valid
choice and allows for generating soft labels that reflect the ground truth
given that specific distribution. Resampling from any valid distribution thus
allows for the creation of a new $D^{PG}$ from a $D^{G}$ by which the coupling
between objects remains intact.

Dataset type | Label | Function
---|---|---
Ground Truth ($G$) | Hard | True
Partial Ground Truth ($PG$) | Soft | True
Observed Soft Label ($OS$) | Soft | Any
Observed Hard Label ($OH$) | Hard | Any
Table 1: The different dataset types defined in SYNLABEL.

#### From Partial Ground Truth to Observed Soft Label

The following non-identity transformations from $D^{PG}$ to $D^{OS}$ serve to
produce experimental datasets tailored to a variety of different
classification tasks, with corresponding (Partial) Ground Truth labels being
available to evaluate on:

  * •

$X^{PG}$ to $X^{O}$. By using any other relationship than the identity
transformation, $X^{O}$ may be altered directly by for instance applying
Gaussian noise to (some of) the variables in $X^{PG}$. This can result in
label noise when $X^{O}$ is used to generate $y^{OS}$ further on via $X^{E}$
and $f^{E}$.

  * •

$y^{PG}$ to $y^{OS}$. If $y^{OS}$ is not established based on the variables in
$X^{O}$ and/or $X^{O^{\prime}}$, but is measured directly, noise can be
introduced directly to $y^{PG}$. This facilitates the NCAR and NAR noise
models.

  * •

$X^{PG}$ and/or $X^{PG^{\prime}}$ and possibly $y^{PG}$ to $y^{OS}$. If
$y^{OS}$ is determined based on $X^{PG}$ and/or $X^{PG^{\prime}}$ and
$y^{PG}$, noise can be generated according to the NNAR model.

  * •

$X^{O}$ to $X^{E}$. If, in contrast to the previous transformation, $y^{OS}$
is decided upon through $f^{E}$, for instance by an expert panel or system,
using information $X^{E}$ which is based on $X^{O}$, the labels $y^{OS}$ can
be manipulated by adding noise to $X^{O}$.

  * •

$X^{PG^{\prime}}$ to $X^{O^{\prime}}$ to $X^{E}$. If as in the previous
transformation $y^{OS}$ is decided upon by an expert through $f^{E}$ using
$X^{E}$, and this expert has more relevant information $X^{O^{\prime}}$
available than is contained in $X^{O}$ alone, noise can be added to $y^{OS}$
by using a non-identity transformation between $X^{PG^{\prime}}$ and
$X^{O^{\prime}}$. An example of $X^{O^{\prime}}$ would be textual descriptions
that can be used by a physician when labelling for the presence of some
disease which are not readily available to be used by a classification model.

  * •

$X^{E}$ to $y^{OS}$. If, as per the previous two transformations, $y^{OS}$ is
obtained trough $f^{E}$ based on $X^{E}$, the labels may be transformed by
adjusting the annotation function $f^{E}$.

The specific transformation used to add label noise to the data is decided
upon by the user. The transformation should be such that the resulting dataset
is suited toward the classification task for which a method is to be
validated. In case the user is interested in soft label research instead of
label noise research, the identity transformations may also be applied such
that $X^{O}$ = $X^{PG}$ and $y^{OS}$ = $y^{PG}$.

#### From Observed Soft Label to Observed Hard Label

Given $X^{O}$ and $y^{OS}$, transformation to $D^{OH}$ is straightforward. A
decision function $f_{dec}$ needs to be defined, which converts the soft
labels into hard labels. Examples of such a function would be sampling the
soft label distribution or simply selecting the class with the highest
probability, although many more decision functions are possible. Note that
this function may be stochastic, allowing for random tie breaks in case of
equal probabilities, as there exists no truth requirements for this functional
relationship.

### Back up

Transformations down the chain, i.e. $D^{G}\rightarrow D^{PG}\rightarrow
D^{OS}\rightarrow D^{OH}$, can be used to generate arbitrarily many observed
datasets from a single ground truth set by adding some specified noise. Before
this can be done, however, a ground truth dataset has to be constructed. This
can be achieved using a real-world observed dataset by utilizing a
transformation up the chain. Such transformations have to respect the
constraints posed on the different datasets per Definition 2-5, summarized in
Table 1, which generally forces the objects in the different sets to become
decoupled:

  * •

When transforming from $D^{PG}$ to $D^{G}$, a decision function $f_{dec}$ must
be applied to $y^{PG}$ to transform the soft labels into hard labels. Unless
$X^{PG^{\prime}}$ is empty and thus $X^{PG}$ = $X^{G}$, $X^{PG^{\prime}}$
contains information that is taken into account by $f^{G}$ by definition. This
information then has to result in different labels for
$f_{dec}(f^{G}(X^{PG}\cup X^{PG^{\prime}}))$ compared to
$f_{dec}(f^{G}(X^{PG}))$for some values in the domain of $f^{G}$. Due to the
missing information in $X^{PG^{\prime}}$, this transformation then decouples
the objects.

  * •

When transforming from $D^{OS}$ or $D^{OH}$ to $D^{G}$ or $D^{PG}$, the true
function $f^{G}$ for the task would need to be discovered. This is generally
impossible, as observed data is always finite and theoretically infinite
functions can be found that describe it perfectly, with no further information
being available that enables the selection of the true governing function.
Furthermore, duplicate instances in $X^{O}$ can have different hard or soft
labels, whereas $y^{G}$ is constrained to contain deterministic hard labels.

As stated before, the exception is when sets along the chain are identical and
thus noiselessly observed with a known ground truth function, which is
generally not the case.

It is possible, however, to construct a different ground truth dataset based
upon the observed dataset, such that its requirements are fulfilled. This will
cause the objects in the new dataset to become decoupled from those in the
original dataset. While this is an issue when the task of interest is to find
the best model for a specific dataset, when the aim is to create a suitable
dataset for validation of a method for a certain type of label noise this is
of no concern.

To construct a new $D^{PG}$, we have to meet Definition 3, i.e. we must obtain
both soft labels and the true relationship between $X$ and $y$. Since the
latter is generally not possible given $X^{O}$ and $y^{OS}$ or $y^{OH}$, we
propose a different approach: first a function is learned based on $X^{O}$ and
$y^{OS}$ or more commonly $y^{OH}$, i.e. $f^{O}$. We then set this function
equal to $f^{G}$, $X^{PG}$ to be $X^{O}$ and $y^{PG}$ to be $f^{G}(X^{PG})$.
In effect, we disregard the original labels and obtain new soft labels that
are generated by the application of the selected ground truth function to the
input data.

Furthermore, by imposing a deterministic decision function $f_{dec}$ upon
$f^{G}$, a new $D^{G}$ can be constructed in a similar manner from $X^{PG}$ by
setting $y^{G}=f_{dec}(f^{G}(X^{PG}))$. Note that $f_{dec}$ has to be
deterministic, so if for a binary problem $f_{dec}(f^{G}(X^{PG}))$ returns
0.5, the decision may not be randomly taken. Again, the transformation results
in a new dataset consisting of different, decoupled objects: the items in the
newly constructed $D^{G}$ are not necessarily the same as those in $D^{PG}$
and the truth function $f^{G}$ is altered as well:
$f^{G}_{new}=f_{dec}(f^{G})$. From the newly constructed $D^{G}$, any of the
transformations described in Section 4 can be applied to construct any number
of new datasets for experimentation.

The SYNLABEL framework together with all of the transformations previously
described has been implemented and made available publicly on GitHub to
encourage its use: https://github.com/sjoerd-de-vries/SYNLABEL.

##  5 Application of the Framework

In the following we demonstrate how SYNLABEL might be used in practice by the
developer of a noise-robust algorithm and highlight differences with existing
methods for label noise experimentation. We believe this to be more valuable
than a comparison of different algorithms on generated datasets - one of the
main uses of the framework - would be, as the results would speak to the
algorithmic performance instead of the quality of the framework. The dataset
used here and in Section 6 is the Keel Vehicle Silhouette set (Triguero et al.
2017), which consists of 18 features and 4 classes.

#### Constructing a Ground Truth

Figure 2: The level of label noise generated by feature hiding as measured by
the mean entropy of the resulting soft labels for different probability
density estimation methods and different numbers of features hidden. Average
over 50 runs. KDE: Kernel Density Estimation. MICE: Multivariate Imputation by
Chained Equations.

To thoroughly evaluate the performance of an algorithm, we need a number of
datasets for which the noise has been quantified. Some curated datasets are
available, for which this has been done by experts. Not all noisy instances
may have been identified, however, and the type of noise these sets contain is
fixed and might not be the of the type that interests us.

Alternatively we could generate, i.e. simulate, clean datasets from scratch
and add the exact noise we are interested in later, e.g. by sampling from
different normal distributions or constructing concentric circles. Once more
this approach is less than optimal, as the resulting datasets typically do not
capture the complexity of real-world data.

By using SYNLABEL and the transformations up the chain defined in Section 4,
we can construct a $D^{G}$ based on observed, noisy data $D^{O}$. Most
commonly, real datasets contain hard labels and as such we take
$D^{O}=D^{OH}=\\{X^{O},y^{OH}\\}$. We then transform this dataset as follows:
based on $D^{OH}$ a function $f^{O}$ is learned. We set $X^{G}$ to be $X^{O}$,
$f^{G}$ to be $f^{O}$ with a deterministic decision function $f_{dec}$ applied
to it: $f^{G}=f_{dec}(f^{O})$. Then we simply set $y^{G}=f^{G}(X^{O})$, as for
the simulated data, to construct a real-world data inspired Ground Truth
dataset.

The properties of this dataset depend on the $f^{G}$ used. If a simple linear
model is used, the resulting dataset will likely not contain the complexities
expected to be present in a real world system. On the other hand, if an
overfit neural network is used, the relationships may well be overly
complicated. As model expressivity is varied, baselines of corresponding
complexity are constructed, approximating real-world problem difficulty to
different extents.

#### Partial Ground Truth via Feature Hiding

Figure 3: Different noise measures for varying noise rates. Left: the mean
$D_{TV}$. Feature hiding was done by sampling from a marginal distribution
constructed via Kernel Density Estimation (KDE). Uniform noise (NCAR) was
added through $T_{r}$. Right: the mean entropy. Feature hiding was done by
sampling from a conditional distribution constructed using MICE. Random class-
conditional noise (NAR) was introduced by a randomly generated $T_{r}$, with
equal probabilities on the main diagonal. $T_{r}$: transition matrix. ID:
instance-dependent (NNAR). FH: feature hiding. $\Delta_{1}$: noise introduced
by FH. $\Delta_{2}$: noise introduced by applying $T_{r}$ to $D^{G}$.
$\Delta_{3}$: noise introduced by applying $T_{r}$ to $D^{PG}$.

Having generated $D^{G}$ as a baseline set for evaluation, we then generate an
additional set with soft labels. These allow for alternative, more direct ways
of quantifying and injecting label noise compared to hard labels, as shown in
Section 6.

$D^{PG}$ can be constructed by applying feature hiding, as specified by
Equation 1. First, we define which features to hide, i.e. add to
$X^{PG^{\prime}}$, and thereby which features remain in $X^{PG}$. Next, we
specify the method for constructing the prior distribution from which the data
in $X^{PG^{\prime}}$ is resampled. Several methods are implemented in SYNLABEL
by default and these can easily be extended to include custom methods. Then we
specify the number of samples drawn and apply the transformation to obtain
soft labels $y^{PG}$.

In Figure 2 we show how resampling from prior distributions for
$X^{PG^{\prime}}$ constructed via different methods results in different
levels of uncertainty in the obtained posterior distributions via Equation 1,
as measured by the Shannon entropy, for different numbers of features hidden.
As expected, sampling according to a conditional density function
$P(X^{PG^{\prime}}|X^{PG}$) or even $P(X^{PG^{\prime}}|X^{PG}\cup y^{G}$), in
this case constructed using MICE (Volker and Vink 2021), produces soft labels
with lower entropy than using a marginal density function $P(X^{PG^{\prime}})$
does.

#### Introducing Noise

Now that we have obtained both a baseline dataset with hard labels $D^{G}$ and
with soft labels $D^{PG}$, we need to introduce the specific type of label
noise we are interested in so that we can compare methods on the resulting
sets. This noise may be added via any of the transformations that have been
described in detail in Section 4. Note that such noise injection can also be
applied to a noisy real-world dataset. In this case, however, we would add
noise upon pre-existing, unspecified noise, which makes it impossible to study
the effect of the added noise in isolation, as we show in Section 6.

##  6 Quantifying Label Noise

An inherent advantage of having soft labels available is that noise can be
quantified by measures such as the Shannon entropy of the resulting
distribution, or when two distributions $P$ and $Q$ are to be compared, the
total variation distance:

$D_{TV}(P,Q)=\frac{1}{2}||P-Q||_{1}.$ (2)

The latter enables us to apply any noise generation method directly on
$D^{PG}$ to generate $D^{OS}$ and quantify the strength of the label noise by
evaluating $D_{TV}(D^{PG},D^{OS})$. As most classification tasks are concerned
with hard labels rather than soft labels, a decision function $f_{dec}$ can be
utilized to transform $y^{OS}$ into $y^{OH}$ and thereby generate $D^{OH}$.
Such a decision function could be for instance taking a sample in proportion
to the label distribution or selecting the class with the highest probability.
When a classifier has to be evaluated against some introduced noise, either
its probabilistic output can be compared directly to $y^{PG}$, or its hard
label to $y^{G}$ or $y^{PG}$, depending on whether any noise due to partly
unobserved data is of interest.

To illustrate the importance of a clean baseline and how the use of label
distributions allows both quantification of noise and direct noise injection,
we conducted experiments with the Keel Vehicle Silhouette dataset, of which
the results are shown in Figure 3. To obtain $D^{G}$ from this observed
dataset, we trained a Random Forest classifier on the original labels and set
it equal to $f^{G}$, a transformation up the chain.

On the left side the Mean Total Variation Distance is shown, either with
respect to the labels in $D^{G}$ or in case of $\Delta_{3}$ with respect to
$D^{PG}$, which measures how often a label is expected to change due to the
noise injection. We observe that there is a difference between the level of
noise introduced when applying a uniform noise matrix $T_{r}$ to a $D^{PG}$
that has been constructed via feature hiding ($FH$) from $D^{G}$, i.e.
$T_{r}(FH(D^{G}))$, and the separately added noise of
$\Delta_{1}=D^{PG}=FH(D^{G})$ and $\Delta_{3}=T_{r}(D^{PG})$,
$\Delta_{1}+\Delta_{3}$. The same applies when this noise is added to $D^{G}$
directly $\Delta_{2}=T_{r}(D^{G})$, and then added to the noise introduced by
$FH$: $\Delta_{1}+\Delta_{2}$. This illustrates how label noise applied to a
set for which the baseline noise is unknown cannot be retrospectively isolated
and properly quantified.

$T_{r}(D^{OH})$ is the result of sampling hard labels from $D^{PG}$ (100
times) and then applying the uniform flipping probability via $T^{r}$ to each
sample (100 times as well), while for $T_{r}(FH(D^{G})$ the noise matrix is
simply applied to the $y^{PG}$ directly. As desired,
$T_{r}(D^{OH})=T_{r}(FH(D^{G}))$, which shows how the direct injection of
noise into the label distribution makes repeated sampling from $D^{PG}$,
followed by repeated application of random noise functions to individual
objects, is redundant, in this case saving 10.000 repeated actions.
Furthermore, the noise level is exact instead of an estimation. In addition,
some types of noise are more naturally applied directly to a label
distribution.

Finally, $ID(T_{r}(D^{OH}))$, where ID stands for instance-dependent (NNAR),
is added to illustrate that similar results are obtained for a more complex
type of noise. The uniform $T_{r}$ from before is still used, but we applied
it twice as often to the objects with the largest ratio of distance to their
nearest neighbour of the same label to distance to a neighbour of the other
label, as in (Garcia et al. 2019).

On the right hand side the mean entropy of the label distributions is shown
and class-conditional noise is added. As entropy is not a measure between
distributions, $\Delta_{3}$ equals $T_{r}(FH(D^{G}))$, and is omitted. The
same patterns can be observed as for $D_{TV}$, demonstrating that noise added
to pre-existing, unknown noise cannot be studied in isolation.

##  7 Conclusion

In this work, we present the SYNLABEL framework which facilitates the
generation of synthetic data for label noise experiments. Standard procedure
would have the user utilize the framework to generate a ground truth dataset
inspired by a real-world dataset by learning a classification function from
the real-world data, setting it to be the ground truth and applying it to the
input data to obtain new hard labels. This dataset can be transformed into a
set with soft labels by hiding a number of the input features contained in the
domain of the selected function and resampling values from learned or pre-
specified distributions for these hidden features, evaluating the ground truth
function on the resulting data and aggregating the resulting labels. This
method called feature hiding adds measurable uncertainty into the labels,
which we show can be useful for direct injection and more thorough
quantification of label noise. These ground truth sets provide a clean
baseline to evaluate method performance on, to which any noise of interest may
be added. Conducting experiments using datasets generated by the framework
offers advantages over the three types of datasets typically used in label
noise research: the generated data are more complex than data simulated from
scratch, provide a clean baseline for evaluation which is lacking from real-
world data and allow for the noise to be controlled, in contrast to curated
data, in addition to being constructed at a low cost.

## References

  * Angluin and Laird (1988) Angluin, D.; and Laird, P. 1988.  Learning from noisy examples.  _Machine Learning_ , 2(4): 343–370.
  * Berthon et al. (2021) Berthon, A.; Han, B.; Niu, G.; Liu, T.; and Sugiyama, M. 2021.  Confidence scores make instance-dependent label-noise learning possible.  In _International Conference on Machine Learning_ , 825–836. PMLR.
  * Chen et al. (2021) Chen, P.; Ye, J.; Chen, G.; Zhao, J.; and Heng, P.-A. 2021.  Beyond class-conditional assumption: A primary attempt to combat instance-dependent label noise.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35(13), 11442–11450.
  * Cheng et al. (2020) Cheng, J.; Liu, T.; Ramamohanarao, K.; and Tao, D. 2020.  Learning with bounded instance and label-dependent label noise.  In _International Conference on Machine Learning_ , 1789–1799. PMLR.
  * Frénay and Verleysen (2014) Frénay, B.; and Verleysen, M. 2014.  Classification in the presence of label noise: a survey.  _IEEE transactions on neural networks and learning systems_ , 25(5): 845–869.
  * Garcia et al. (2019) Garcia, L. P.; Lehmann, J.; de Carvalho, A. C.; and Lorena, A. C. 2019.  New label noise injection methods for the evaluation of noise filters.  _Knowledge-Based Systems_ , 163: 693–704.
  * Gu et al. (2022) Gu, K.; Masotto, X.; Bachani, V.; Lakshminarayanan, B.; Nikodem, J.; and Yin, D. 2022.  An instance-dependent simulation framework for learning with label noise.  _Machine Learning_ , 112: 1871–1896.
  * Hickey (1996) Hickey, R. J. 1996.  Noise modelling and evaluating learning from examples.  _Artificial Intelligence_ , 82(1-2): 157–179.
  * Jin and Ghahramani (2002) Jin, R.; and Ghahramani, Z. 2002.  Learning with multiple labels.  _Advances in neural information processing systems_ , 15.
  * Langley (1988) Langley, P. 1988.  Machine learning as an experimental science.  _Machine Learning_ , 3(1): 5–8.
  * Lawrence and Schölkopf (2001) Lawrence, N.; and Schölkopf, B. 2001.  Estimating a kernel fisher discriminant in the presence of label noise.  In _18th International Conference on Machine Learning_ , 306–306. Morgan Kaufmann.
  * Lee et al. (2018) Lee, K.-H.; He, X.; Zhang, L.; and Yang, L. 2018.  Cleannet: Transfer learning for scalable image classifier training with label noise.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 5447–5456.
  * Li et al. (2017) Li, W.; Wang, L.; Li, W.; Agustsson, E.; and Gool, L. V. 2017.  WebVision Database: Visual Learning and Understanding from Web Data.  arXiv:1708.02862.
  * Song et al. (2022) Song, H.; Kim, M.; Park, D.; Shin, Y.; and Lee, J.-G. 2022.  Learning from noisy labels with deep neural networks: A survey.  _IEEE Transactions on Neural Networks and Learning Systems_.
  * Triguero et al. (2017) Triguero, I.; González, S.; Moyano, J. M.; García López, S.; Alcalá Fernández, J.; Luengo Martín, J.; Fernández Hilario, A. L.; Jesús Díaz, M. J. d.; Sánchez, L.; and Herrera Triguero, F. 2017.  KEEL 3.0: an open source software for multi-stage analysis in data mining.  _International Journal of Computational Intelligence Systems_ , 10: 1238–1249.
  * Veta et al. (2016) Veta, M.; Van Diest, P. J.; Jiwa, M.; Al-Janabi, S.; and Pluim, J. P. 2016.  Mitosis counting in breast cancer: Object-level interobserver agreement and comparison to an automatic method.  _PloS one_ , 11(8): e0161286.
  * Volker and Vink (2021) Volker, T. B.; and Vink, G. 2021.  Anonymiced Shareable Data: Using mice to Create and Analyze Multiply Imputed Synthetic Datasets.  _Psych_ , 3(4): 703–716.
  * Xia et al. (2020) Xia, X.; Liu, T.; Han, B.; Wang, N.; Gong, M.; Liu, H.; Niu, G.; Tao, D.; and Sugiyama, M. 2020.  Part-dependent label noise: Towards instance-dependent label noise.  _Advances in Neural Information Processing Systems_ , 33: 7597–7610.
  * Xiao et al. (2015) Xiao, T.; Xia, T.; Yang, Y.; Huang, C.; and Wang, X. 2015.  Learning from massive noisy labeled data for image classification.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2691–2699.
  * Zhang et al. (2021) Zhang, Y.; Zheng, S.; Wu, P.; Goswami, M.; and Chen, C. 2021.  Learning with Feature-Dependent Label Noise: A Progressive Approach.  arXiv:2103.07756.