#  Partial Label Supervision for
Agnostic Generative Noisy Label Learning

Fengbei Liu  Australian Institute for Machine Learning (AIML), University of
Adelaide  Chong Wang  Australian Institute for Machine Learning (AIML),
University of Adelaide  Yuanhong Chen  Australian Institute for Machine
Learning (AIML), University of Adelaide  Yuyuan Liu  Australian Institute for
Machine Learning (AIML), University of Adelaide  Gustavo Carneiro  Centre for
Vision, Speech and Signal Processing (CVSSP), University of Surrey

###### Abstract

Noisy label learning has been tackled with both discriminative and generative
approaches. Despite the simplicity and efficiency of discriminative methods,
generative models offer a more principled way of disentangling clean and noisy
labels and estimating the label transition matrix. However, existing
generative methods often require inferring additional latent variables through
costly generative modules or heuristic assumptions, which hinder adaptive
optimisation for different causal directions. They also assume a uniform clean
label prior, which does not reflect the sample-wise clean label distribution
and uncertainty. In this paper, we propose a novel framework for generative
noisy label learning that addresses these challenges. First, we propose a new
single-stage optimisation that directly approximates image generation by a
discriminative classifier output. This approximation significantly reduces the
computation cost of image generation, preserves the generative modelling
benefits, and enables our framework to be agnostic in regards to different
causality scenarios (i.e., image generate label or vice-versa). Second, we
introduce a new Partial Label Supervision (PLS) for noisy label learning that
accounts for both clean label coverage and uncertainty. The supervision of PLS
does not merely aim at minimising loss, but seeks to capture the underlying
sample-wise clean label distribution and uncertainty. Extensive experiments on
computer vision and natural language processing (NLP) benchmarks demonstrate
that our generative modelling achieves state-of-the-art results while
significantly reducing the computation cost. Our code is available at
https://github.com/lfb-1/GNL.

##  1 Introduction

Deep neural network (DNN) has achieved remarkable success in computer vision
[14, 24], natural language processing (NLP) [10, 63] and medical image
analysis [29, 45]. However, DNNs often require a massive amount of high-
quality annotated data for supervised training [9], which is challenging and
expensive to acquire. To alleviate such problem, some datasets have been
annotated via crowd sourcing [55], from search engines [42], or with NLP from
radiology reports [45]. Although these cheaper annotation processes enable the
construction of large-scale datasets, they also introduce noisy labels for
model training, resulting in performance degradation. Therefore, novel
learning algorithms are required to robustly train DNN models when training
sets contain noisy labels. The main challenge in noisy-label learning is that
we only observe the data, represented by random variable $X$, and respective
noisy label, denoted by variable $\tilde{Y}$, but we want to estimate the
model $p(Y|X)$, where $Y$ is the hidden clean label variable.

Most methods proposed in the field resort to two types of discriminative
learning strategies, namely: sample selection and noise transition matrix.
Sample selection [1, 13, 26] optimises the model $p_{\theta}(Y|X)$,
parameterised by $\theta$, with maximum likelihood optimisation restricted to
pseudo-clean training samples, as follows

$\displaystyle\theta^{*}=\arg\max_{\theta}\mathbb{E}_{P(X,\tilde{Y})}\left[\mathsf{clean}(X,\tilde{Y})\times
p_{\theta}(\tilde{Y}|X)\right],$ (1) $\displaystyle\text{ where
}\mathsf{clean}(X=\mathbf{x},\tilde{Y}=\tilde{\mathbf{y}})=\begin{cases}1\text{,
if }Y=\tilde{\mathbf{y}}\\\ 0\text{, otherwise}\end{cases}$

and $P(X,\tilde{Y})$ is the distribution used to generate the noisy-label and
data points for the training set. Note that
$\mathbb{E}_{P(X,\tilde{Y})}\left[\mathsf{clean}(X,\tilde{Y})\times
p_{\theta}(\tilde{Y}|X)\right]\equiv\mathbb{E}_{P(X,Y)}\left[p_{\theta}(Y|X)\right]$
if the function $\mathsf{clean}(.)$ successfully selects the clean-label
training samples. Unfortunately, $\mathsf{clean}(.)$ usually relies on the
small-loss hypothesis [2] for selecting $R\%$ of the smallest loss training
samples, which offers little guarantees of successfully selecting clean-label
samples. Approaches based on noise transition matrix [53, 6, 37] aim to
estimate a clean-label classifier and a label transition, as follows:

$\begin{split}\theta^{*}&=\arg\max_{\theta}\mathbb{E}_{P(X,\tilde{Y})}\left[\sum_{Y}p(\tilde{Y},Y|X)\right]\\\
&=\arg\max_{\theta_{1},\theta_{2}}\mathbb{E}_{P(X,\tilde{Y})}\left[\sum_{Y}p_{\theta_{1}}(\tilde{Y}|Y,X)p_{\theta_{2}}(Y|X)\right],\end{split}$
(2)

where $\theta=[\theta_{1},\theta_{2}]$, $p_{\theta_{1}}(\tilde{Y}|Y,X)$
represents a label-transition matrix, often simplified to be class-dependent
with $p_{\theta_{1}}(\tilde{Y}|Y)=p_{\theta_{1}}(\tilde{Y}|Y,X)$. Since we do
not have access to the label transition matrix, we need to estimate it from
the noisy-label training set, which is challenging because of identifiability
issues [31], making necessary the use of anchor points [37] or other types of
regularisation [6].

Figure 1: Generative noisy label learning models and their corresponding
probability functions, where the red arrow indicates the different causal
relationships between $X$ and $Y$. (a) CausalNL [61] and InstanceGM [11]
assume that $Y$ causes $X$ and optimise the joint likelihood $p(X,\tilde{Y})$.
(b) NPC [3] and DyGEN [70] assume that $X$ causes $Y$ and optimise
$p(Y|\tilde{Y},X)$ as a post-processing method. (c) Our proposed method
optimises $p(X|Y)$ or $p(Y|X)$ directly based on the causal relationship of
datasets.

On the other hand, the generative modelling approach to noisy label learning
[3, 61, 70] aims to estimate the transition matrix by assuming a causal
relationship between $X$ and $Y$, as depicted in Fig. 1. Previous methods
consider a causal generative process where $Y$ causes $X$ [61, 11] and are
trained to maximise the joint data likelihood
$p(\tilde{Y},X)=\int_{Y,Z}p(X|Y,Z)p(\tilde{Y}|Y,X)p(Y)p(Z)dYdZ$, where $Z$
represents a low-dimensional latent image representation, and $Y$ is the
latent clean label. This optimisation requires a variational distribution
$q_{\phi}(Y,Z|X)$ to maximise the evidence lower bound (ELBO):

$\begin{split}\theta_{1}^{*},\theta_{2}^{*},\phi^{*}=\arg\max_{\theta_{1},\theta_{2},\phi}\mathbb{E}_{q_{\phi}(Y,Z|X)}\left[\log\left(\frac{p_{\theta_{1}}(X|Y,Z)p_{\theta_{2}}(\tilde{Y}|Y,X)p(Y)p(Z)}{q_{\phi}(Y,Z|X)}\right)\right],\end{split}$

(3)

where $p_{\theta_{1}}(X|Y,Z)$ denotes an image generative model,
$p_{\theta_{2}}(\tilde{Y}|Y,X)$ represents the label transition model, $p(Z)$
is the latent image representation prior (commonly assumed to be a standard
normal distribution), and $p(Y)$ is the clean label prior, which is usually
assumed to be a uniform distribution. Notice from (3) that the optimisation of
the transition matrix $p_{\theta_{2}}(\tilde{Y}|Y,X)$ is constrained by the
training of $p_{\theta_{1}}(X|Y,Z)$. Such a constrained optimisation has the
potential to reduce the uncertainty of the transition matrix, which suggests
that optimising this generative model naturally improves the estimation of the
transition matrix without requiring the regularizations explored by
discriminative models [28, 54, 6, 37].

However, generative modelling faces several limitations that hinder its more
widespread utilisation. For instance, it introduces a latent variable $Z$ to
optimise $p(\tilde{Y},X)$, instead of directly optimising $p(X|Y)$ or
$p(Y|X)$. Also, the introduction of an image generation module (e.g., the
variational auto-encoder [21]) limits the framework to low-resolution images
and results in sub-optimal reconstruction generative training, which leads to
inaccurate transition matrix estimation. Even though some generative methods
do not use $Z$, and instead rely on a post-processing framework to calibrate
pre-trained classifier predictions [3, 70], they assume that high-confidence
classifications from $p(\tilde{Y}|X)$ represent reliable indicators of the
true labels $Y$, which may not be true for highly noisy datasets. Moreover,
current methods rigidly focus on a particular causal direction between $X$ and
$Y$ [3, 70, 61, 11] and cannot easily adapt to different causal directions
[59].

To address the limitations of existing generative noisy label learning, we
propose a novel single-stage generative noisy label learning framework that is
adaptable to different causal processes and directly optimises $p(X|Y)$ or
$p(Y|X)$ without relying on the latent variable $Z$. Our proposal faces three
challenges: (1) the removal of $Z$ makes $p(X|Y)$ under-constrained because
$Z$ cannot "anchor" the image generation process; (2) as shown in Fig. 1-(a),
$Z$ and $Y$ are not independent given the observation of $X$ [38], so the
removal of $Z$ implies that $X$ needs to be constrained by an informative $Y$;
and (3) how to design a single-stage training for a model that is agnostic to
different causal processes (i.e., $p(Y|X)$ or $p(X|Y)$). We denote our
contributions as follows:

  * •

Given that $p(X|Y)$ cannot be estimated due to the infinite number of
instantiations of $X$ given $Y$, we use the finite set of training samples to
instantiate $X$ and approximate $p(X|Y)$ by a discriminative classifier
output. Such an approximation avoids the estimation of $Z$, while retaining
the generative modelling benefits mentioned above.

  * •

Motivated by the decomposition of partial label learning [39], we introduce a
new partial label supervision (PLS) to dynamically approximate $Y$ with an
informative partial label prior $p(Y)$, considering both latent clean label
coverage and uncertainty. The instance-wise prior regularizes model training
by reducing $p(Y)$ to a one-hot distribution when the model has high
confidence that the training label is correct (i.e., low label uncertainty);
on the other hand, $p(Y)$ is approximated to a uniform label distribution when
the model has low confidence that the training label is clean (i.e., high
label uncertainty).

  * •

The approximations of $p(X|Y)$ and PLS constructed $p(Y)$ allow us to
formulate our single-stage optimisation based on the EM algorithm to train a
model that is agnostic to both $p(X|Y)$ and $p(Y|X)$ generation processes.

We conduct extensive experiments on both synthetic and real-world noisy label
benchmarks on computer vision and natural language processing (NLP) tasks.
Experimental results and comprehensive ablation studies show that our
framework outperforms previous generative approaches [3, 70, 61, 11] in terms
of performance, computation cost and transition matrix estimation error.
Comparisons with noise transition matrix approaches [62, 3] show that our
method produces more accurate classification and noise transition matrix
estimation without imposing any regularization to handle the identifiability
problem [61]. Regarding sample-selection methods [26, 12], our approach shows
competitive results at significantly lower computational costs.

##  2 Related Work

Sample selection is a noisy-label discriminative learning strategy, defined in
Eq. (1), which needs to handle two problems: 1) the definition of the function
$\mathsf{clean}(.)$, and 2) what to do with the samples classified as noisy.
Most definitions of $\mathsf{clean}(.)$ resort to classify small-loss samples
[2] as pseudo-clean [1, 4, 13, 16, 26, 35, 41, 48]. Other approaches select
clean samples based on K nearest neighbor classification from deep learning
feature spaces [36, 46], distance to the class-specific eigenvector from the
gram matrix eigen-decomposition using intermediate deep learning feature
spaces [19], uncertainty measures [22], or prediction consistency between
teacher and student models [17]. After sample classification, some methods
will discard the noisy-label samples for training [4, 16, 35, 41], while
others use them for semi-supervised learning [26]. The main issue with this
strategy is the hard decision strategy to classify samples as having clean or
noisy labels, as well as pseudo-labelling noisy samples with single model
predictions, without leaving much room for uncertainty in terms of possible
label candidates. These methods also tend to have relatively high run-time
training complexity because of the semi-supervised learning and the usual need
for multiple models.

Label transition model is another discriminative noisy-label learning strategy
from Eq. (2) that depends on a reliable estimation of $p(\tilde{Y}|Y,X)$ [6,
37, 53]. Forward-T [37] uses an additional classifier and anchor points from
clean-label samples to learn a class-dependent transition matrix. Part-T [53]
estimates an instance-dependent model. MEDITM [6] uses manifold regularization
for estimating the label-transition matrix. In general, the estimation of this
label transition matrix is under-constrained, leading to the identifiability
problem [31], which is addressed with strong assumptions [37], or with
additional labels per training sample [31].

Generative modelling is a noisy-label learning technique [3, 12, 61, 70] that
explores different graphical models (see Fig. 1) to estimate the image’s clean
label. Specifically, CausalNL [61] and InstanceGM [12] assume that the latent
clean label $Y$ causes $X$, and the noisy label $\tilde{Y}$ is generated from
$X$ and $Y$. The drawback of CausalNL/InstanceGM is that they optimise the
joint distribution of data and noisy labels $p(X,\tilde{Y})$ instead of
directly optimising a model associating data with the clean label, such as
$p(X|Y)$ or $p(Y|X)$. They also introduce the random variable $Z$ for image
generation, which is computationally expensive. NPC [3] and DyGEN [70] assumes
that $X$ causes $Y$ and proposes a post-processing technique for inferring
clean label $Y$. However, NPC/DyGEN assumes that high-confidence predictions
by the noisy classifier can be used as prior for $Y$, which may not be
effective in high label noise rate datasets. Also, all approaches above are
designed for a specific causal relationship and may not adapt well to datasets
with different causal relationships.

Clean label prior $p(Y)$ constrains the clean label to a set of label
candidates for a particular training sample. Such label candidates change
during the training, following two design principles: 1) increase clean label
coverage, and 2) reduce the uncertainty of the label prior. The increase of
coverage improves the chances of including the correct clean label into the
prior. Given that this may decrease the quality of the supervisory training
signal because it may spread the probability mass over too many labels, the
second design principle regularises the training by reducing the number of
label candidates in $p(Y)$. Such dynamic prior distribution may resemble Mixup
[65], label smoothing [33] or re-labeling [26] techniques that are commonly
used in label noise learning. However, these approaches do not simultaneously
follow the two design principles mentioned above. More specifically, Mixup
[65] and label smoothing [33] are effective approaches to propose soft labels
for noisy label learning, which increases label coverage, but does not
necessarily reduce label uncertainty. Re-labeling [26] replaces the training
label by a hard pseudo label, so it is very efficient, but it has limited
coverage.

Partial label learning (PLL) assumes that each image is associated with a
candidate label set defined as a partial label [43], where one of the labels
is a true positive, and the remaining labels are false positives. The goal of
PLL is to predict the single true label associated with each training sample.
PICO [44] uses contrastive learning in an EM optimisation to address PLL. CAV
[64] proposes class activation mapping to identify the true label within the
candidate set. PRODEN [34] progressively identifies the true labels from a
candidate set and updates the model parameter. Qiao et al. [39] explicitly
model the generation process of candidate labels with correct and incorrect
labels generation. The design of our partial label supervision (PLS) $p(Y)$ is
inspired by PLL, but unlike PLL, $p(Y)$ is dynamically constructed during
training since the clean label is latent. In particular, $p(Y)$ will
approximate a uniform distribution if the training label has a high
probability of being wrong. On the other hand, $p(Y)$ will reduce to a one-hot
distribution when the training label has a high probability of being correct.

##  3 Method

We denote the noisy training set as
$\mathcal{D}=\\{(\mathbf{x}_{i},\tilde{\mathbf{y}}_{i})\\}_{i=1}^{|\mathcal{D}|}$,
where $\mathbf{x}_{i}\in\mathcal{X}\subset\mathbb{R}^{H\times W\times C}$ is
the input image of size $H\times W$ with $C$ colour channels,
$\tilde{\mathbf{y}}_{i}\in\mathcal{Y}\subset\\{0,1\\}^{|\mathcal{Y}|}$ is the
observed one-hot noisy label of the $i^{th}$ sample. We also denote
$\mathbf{y}_{i}$ as the latent clean label of the $i^{th}$ sample.

Below, in Sec. 3.1, we introduce the optimisation goal of our model under
different causal relationships. In Sec. 3.2, we discuss how to approximate
generative term without the latent variable denoted by $Z$. In Sec. 3.3, we
describe how to construct the informative prior with partial label hypothesis,
and the overall details of training algorithm is presented in Sec. 3.4.

###  3.1 General optimisation goal

We first consider the causal relationship where $Y$ causes $X$ and the
optimisation goal is the maximisation of $\log p(\mathbf{x}|\mathbf{y})$,
which can be decomposed as:

$\begin{split}\log
p(\mathbf{x}|\mathbf{y})&=\log\frac{p(\tilde{\mathbf{y}},\mathbf{y},\mathbf{x})}{p(\tilde{\mathbf{y}}|\mathbf{x},\mathbf{y})p(\mathbf{y})}.\end{split}$
(4)

In Eq. (4), $p(\mathbf{y})$ represents the prior distribution of the latent
clean label. The optimisation of the log likelihood in Eq. (4) can be achieved
by introducing a variational posterior distribution
$q(\mathbf{y}|\mathbf{x})$, with:

$\begin{split}\log
p(\mathbf{x}|\mathbf{y})&=\log\frac{p(\tilde{\mathbf{y}},\mathbf{y},\mathbf{x})}{q(\mathbf{y}|\mathbf{x})}+\log\frac{q(\mathbf{y}|\mathbf{x})}{p(\tilde{\mathbf{y}}|\mathbf{x},\mathbf{y})p(\mathbf{y})},\end{split}$
(5)

which can be used to derive

$\begin{split}\mathbb{E}_{q(\mathbf{y}|\mathbf{x})}\left[\log
p(\mathbf{x}|\mathbf{y})\right]=&\mathbb{E}_{q(\mathbf{y}|\mathbf{x})}\left[\log\frac{p(\tilde{\mathbf{y}},\mathbf{y},\mathbf{x})}{q(\mathbf{y}|\mathbf{x})}\right]\\\
&+\mathsf{KL}\Big{[}q(\mathbf{y}|\mathbf{x})||p(\tilde{\mathbf{y}}|\mathbf{x},\mathbf{y})p(\mathbf{y})\Big{]},\end{split}$
(6)

where $\mathsf{KL}[.]$ denotes the Kullback Leibler divergence. Eq. (6)
connects our formulation with Expectation-Maximization (EM) algorithm [8],
where first term, represented by
$\mathbb{E}_{q(\mathbf{y}|\mathbf{x})}\left[\log\frac{p(\tilde{\mathbf{y}},\mathbf{y},\mathbf{x})}{q(\mathbf{y}|\mathbf{x})}\right]$,
is the Evidence Lower Bound (ELBO) that is decomposed as:

$\begin{split}\mathbb{E}_{q(\mathbf{y}|\mathbf{x})}\left[\log\frac{p(\tilde{\mathbf{y}},\mathbf{y},\mathbf{x})}{q(\mathbf{y}|\mathbf{x})}\right]=&\mathbb{E}_{q(\mathbf{y}|\mathbf{x})}\left[\log
p(\tilde{\mathbf{y}}|\mathbf{x},\mathbf{y})\right]\\\
&+\mathbb{E}_{q(\mathbf{y}|\mathbf{x})}\left[\log\frac{p(\mathbf{x}|\mathbf{y})p(\mathbf{y})}{q(\mathbf{y}|\mathbf{x})}\right].\\\
\end{split}$ (7)

Based on Equations (5) and (7), the expected log of $p(\mathbf{x}|\mathbf{y})$
from Eq. (6) is defined as

$\begin{split}\mathbb{E}_{q(\mathbf{y}|\mathbf{x})}&\left[\log
p(\mathbf{x}|\mathbf{y})\right]=\\\
&\underbrace{\mathbb{E}_{q(\mathbf{y}|\mathbf{x})}\left[\log
p(\tilde{\mathbf{y}}|\mathbf{x},\mathbf{y})\right]-\mathsf{KL}\left[q(\mathbf{y}|\mathbf{x})\|p(\mathbf{x}|\mathbf{y})p(\mathbf{y})\right]}_{\text{M-step}}\\\
&\underbrace{+\mathsf{KL}\left[q(\mathbf{y}|\mathbf{x})\|p(\tilde{\mathbf{y}}|\mathbf{x},\mathbf{y})p(\mathbf{y})\right]}_{\text{E-step}}.\end{split}$
(8)

According to the EM algorithm, the expected log likelihood in Eq. (8) can be
maximised by maximising the ELBO (M-step), while minimising the KL divergence
of the third term (E-step).

Considering the other type of causal relationship where $X$ causes $Y$, we can
have a similar formulation to Eq. (4). In this case, our log likelihood term
can be decomposed as:

$\log
p(\mathbf{y}|\mathbf{x})=\log\frac{p(\tilde{\mathbf{y}},\mathbf{y},\mathbf{x})}{p(\tilde{\mathbf{y}}|\mathbf{x},\mathbf{y})\color[rgb]{1,0,0}p(\mathbf{x})}\color[rgb]{0,0,0},$
(9)

where the main difference to Eq. (4) is the replacement of $p(\mathbf{y})$ by
$p(\mathbf{x})$ in the denominator. Using a variational distribution
$q(\mathbf{y}|\mathbf{x})$, we can re-write Eq. (9) as:

$\begin{split}\mathbb{E}_{q(\mathbf{y}|\mathbf{x})}\left[\log
p(\mathbf{y}|\mathbf{x})\right]=&\mathbb{E}_{q(\mathbf{y}|\mathbf{x})}\left[\log\frac{p(\tilde{\mathbf{y}},\mathbf{y},\mathbf{x})}{q(\mathbf{y}|\mathbf{x})}\right]\\\
&+\mathsf{KL}\Big{[}q(\mathbf{y}|\mathbf{x})||p(\tilde{\mathbf{y}}|\mathbf{x},\mathbf{y})\color[rgb]{1,0,0}p(\mathbf{x})\color[rgb]{0,0,0}\Big{]}.\end{split}$
(10)

Based on Eq. (9) and Eq. (10), we define the following expected log of
$p(\mathbf{y}|\mathbf{x})$:

$\begin{split}\mathbb{E}_{q(\mathbf{y}|\mathbf{x})}&\left[\log
p(\mathbf{y}|\mathbf{x})\right]=\\\
&\underbrace{\mathbb{E}_{q(\mathbf{y}|\mathbf{x})}\left[\log
p(\tilde{\mathbf{y}}|\mathbf{x},\mathbf{y})\right]-\mathsf{KL}\left[q(\mathbf{y}|\mathbf{x})\|p(\mathbf{x}|\mathbf{y})p(\mathbf{y})\right]}_{\text{M-step}}\\\
&\underbrace{+\mathsf{KL}\left[q(\mathbf{y}|\mathbf{x})\|p(\tilde{\mathbf{y}}|\mathbf{x},\mathbf{y})\color[rgb]{1,0,0}p(\mathbf{x})\color[rgb]{0,0,0}\right]}_{\text{E-step}}.\end{split}$
(11)

Hence, Eq. (8) and Eq. (11) provide the optimisation goals for different
causal relationships, where the difference between the two equations is the
presence of $p(\mathbf{x})$ or $p(\mathbf{y})$ in the E-step term.

Figure 2: Left: Description of our proposed framework, which follows Eq. (8)
and Eq. (11) for model training and Eq. (15) for Partial Label Supervision
(PLS). The term $q(\mathbf{y}|\mathbf{x})$ represents the variational
posterior in Eq. (5), and $p(\tilde{\mathbf{y}}|\mathbf{y},\mathbf{x})$
denotes the noise transition module in Eq. (7). "Coverage" and "Uncertainty"
denote the selected labels in Eq. (16) and Eq. (18). The unshaded $Y$
represents the unobserved clean label variable and $p(\mathbf{y})$ is
constructed to approximate it. Right: Examples of CIFAR10 images under 40% IDN
noise and PLS constructed partial label from Eq. (15). Note that $\mathbf{y}$
represents clean label, $\tilde{\mathbf{y}}$ denotes noisy label, $\mathbf{c}$
means label "Coverage" from Eq. (16) and $\mathbf{u}$, label "Uncertainty"
from Eq. (18).

###  3.2 Approximation of $p(\mathbf{x}|\mathbf{y})$ and $p(\mathbf{x})$

The terms $p(\mathbf{x}|\mathbf{y})$ and $p(\mathbf{x})$ are challenging to
estimate because of the infinite number of samples $X$ that can be generated
by their clean labels $Y$. One solution to mitigate such challenge is the use
of a latent image representation $Z$ to "anchor" the image generation process
[61, 12]. However, such an image generation is not relevant for the
discriminative classification that we aim to solve [3]. In addition, modelling
$Z$ becomes troublesome with large resolution input, leading to sub-optimal
reconstructions and transition matrix estimation.

Hence, we refrain from using $Z$ and define $p(\mathbf{x}|\mathbf{y})$ only in
terms of the finite number of training samples. This facilitates the direct
optimisation of $p(\mathbf{x}|\mathbf{y})$ and alleviates the problematic
training of an image generator [40]. This is achieved by the following
approximation:

$p(\mathbf{x}|\mathbf{y})\approx\frac{q(\mathbf{y}|\mathbf{x})}{\sum_{i=1}^{|\mathcal{D}|}q(\mathbf{y}|\mathbf{x}_{i})}.$
(12)

Hence, the conditional probability $p(\mathbf{x}|\mathbf{y})$ is only defined
for available training samples $\mathbf{x}$ given their latent labels
$\mathbf{y}$, so $p(\mathbf{x}|\mathbf{y})$ has dimension
$|\mathcal{D}|\times|\mathcal{Y}|$. As mentioned in [40],
$p(\mathbf{x}|\mathbf{y})$ is large when $q(\mathbf{y}|\mathbf{x})$ is also
large in comparison with $q(\mathbf{y}|\mathbf{x}_{i})$ for all training
samples $(\mathbf{x}_{i},\tilde{\mathbf{y}}_{i})\in\mathcal{D}$.

With $p(\mathbf{x}|\mathbf{y})$ approximated from the variational posterior
$q(\mathbf{y}|\mathbf{x})$, we can derive $p(\mathbf{x})$ from Bayes
marginalization:

$p(\mathbf{x})=\sum_{\mathbf{y}\in\mathcal{Y}}p(\mathbf{x}|\mathbf{y})p(\mathbf{y}).$
(13)

Notice that both approximations in Eq. (12) and Eq. (13) rely on the
discriminative classifier $q(\mathbf{y}|\mathbf{x})$. Such approximations
allow our optimisation to be more efficient than previous generative methods
[12, 61] because they do not require the use of image generative models.

###  3.3 Partial label supervision for noisy label learning

As depicted in Fig. 1(a), $Z$ and $Y$ jointly generate $X$. Based on
d-separation rule [38], $Z$ and $Y$ become dependent when $X$ is observed.
Traditional generative modelling uses a uniform $Y$ and constrains $X$ by
estimating an informative $Z$. Since we remove $Z$ from our framework, $Y$
must be informative to better constraint the generation of $X$.

In order to approximate clean label prior $p(\mathbf{y})$, previous works [3,
70] explore multi-stage training, which is computationally intensive and time
consuming. Motivated by recent study on PLL decomposition [39], we introduce
Partial Label Supervision (PLS) to construct an informative $p(\mathbf{y})$.
Unlike traditional pseudo labeling methods, which commits to a single,
definitive prediction for a noisy-label training sample and heavily rely on
model prediction confidence for pseudo label accuracy, PLS constructs
$p(\mathbf{y})$ with a set of label candidates for each sample that contains
as few labels as possible, but that also has a large probability of covering
the clean label $Y$. Therefore, PLS is designed to balance these two
objectives: maximizing label coverage while minimising label uncertainty,
which are defined by:

$\begin{split}\text{Coverage}&=\frac{1}{|\mathcal{D}|}\sum_{i=1}^{|\mathcal{D}|}\sum_{\mathbf{y}\in\mathcal{Y}}\mathbbm{1}\left[\mathbf{y}_{i}=\mathbf{y}\right]\times\mathbbm{1}\left[p_{i}(\mathbf{y})>0\right]\\\
\text{Uncertainty}&=\frac{1}{|\mathcal{D}|}\sum_{i=1}^{|\mathcal{D}|}\sum_{\mathbf{y}\in\mathcal{Y}}\mathbbm{1}\left[p_{i}(\mathbf{y})>0\right],\end{split}$
(14)

where $\mathbf{y}_{i}$ is the latent clean label of the $i^{th}$ training
sample, $p_{i}(\mathbf{y})$ denotes the prior label probability of the
$i^{th}$ training sample for the class $\mathbf{y}\in\mathcal{Y}$, and
$\mathbbm{1}(.)$ represents the indicator function.

In Eq. (14), coverage increases by approximating $p(\mathbf{y})$ to a uniform
distribution, but uncertainty is minimised when the latent clean label
$\mathbf{y}_{i}$ is assigned maximum probability. The maximisation of coverage
and minimisation of uncertainty can only be achieved by approximating the
prior $p_{i}(\mathbf{y})$ to a one-hot distribution that represents the
underlying clean label $\mathbf{y}_{i}$. To reach such a goal, we define the
clean label prior by:

$p_{i}(\mathbf{y}(j))=\frac{\tilde{\mathbf{y}}_{i}(j)+\mathbf{c}_{i}(j)+\mathbf{u}_{i}(j)}{Z},$
(15)

where $j\in\\{1,...,|\mathcal{Y}|\\}$, $\tilde{\mathbf{y}}_{i}\in\mathcal{Y}$
is the noisy label in the training set,
$\mathbf{c}_{i}\in\Delta^{|\mathcal{Y}|-1}$ denotes the candidate label that
is built to increase coverage ($\Delta^{|\mathcal{Y}|-1}$ denotes the
probability simplex), $\mathbf{u}_{i}\in\Delta^{|\mathcal{Y}|-1}$ represents
the candidate label built to decrease uncertainty, and $Z$ is a normalisation
factor to make $\sum_{\mathbf{y}\in\mathcal{Y}}p_{i}(\mathbf{y})=1$. We define
$\mathbf{c}_{i}$ and $\mathbf{u}_{i}$ below.

Since the variational posterior $q(\mathbf{y}|\mathbf{x})$ is optimized to
recover likely candidates for the clean label, we maximise coverage by
sampling from a moving average of $q(\mathbf{y}|\mathbf{x})$ for each training
sample $\mathbf{x}_{i}$ at training iteration $t$ with:

$\begin{split}\mathcal{C}_{i}^{(t)}&=\beta\times\mathcal{C}_{i}^{(t-1)}+(1-\beta)\times\bar{\mathbf{y}}_{i}^{(t)},\end{split}$
(16)

where $\beta\in[0,1]$, and $\bar{\mathbf{y}}_{i}^{(t)}$ is the softmax output
from $q(\mathbf{y}|\mathbf{x})$ that predicts the clean label from the data
input $\mathbf{x}_{i}$. For Eq. (16), $\mathcal{C}_{i}^{(t)}$ denotes the
categorical distribution of the most likely labels for the $i^{th}$ training
sample, which is used to sample the one-hot label
$\mathbf{c}_{i}\sim\mathsf{Cat}(\mathcal{C}_{i}^{(t)})$.

The minimisation of uncertainty depends on our ability to detect clean-label
and noisy-label samples. For clean samples, the label prior
$p_{i}(\mathbf{y})$ should converge to a distribution focused on few candidate
labels. For noisy samples, this label prior should be close to a uniform
distribution to keep a large number of candidate labels and to reduce the
confidence of supervision for slower fitting speed. To estimate the
probability $w_{i}\in[0,1]$ that a label is noisy, we compute the cross-
entropy loss between the variational posterior $q(\mathbf{y}|\mathbf{x})$
output $\bar{\mathbf{y}}_{i}$ and the noisy label $\tilde{\mathbf{y}}$ with:

$\ell_{i}=-\tilde{\mathbf{y}}_{i}^{\top}\log\bar{\mathbf{y}}_{i}.$ (17)

In contrast to the small-loss hypothesis explored in existing literature [26,
13], which associates noisy-label samples with the cross-entropy loss between
the label $\tilde{\mathbf{y}}$ and model output that tends to be similar to a
one-hot distribution, our variational posterior model
$q(\mathbf{y}|\mathbf{x})$ is trained to approximate the multiple PLS labels
from $p(\mathbf{y})$. As a result, $q(\mathbf{y}|\mathbf{x})$ does not merely
focus on minimising loss but seeks to capture the underlying label
distribution and uncertainty. To compute the probability $w_{i}\in[0,1]$ that
a sample contains noisy label, we use an unsupervised 2-class classification
based on the loss function in Eq. (17).

The label $\mathbf{u}_{i}$ is obtained by sampling from a uniform distribution
of all possible labels proportionally to its probability of representing a
noisy-label sample, with

$\mathbf{u}_{i}\sim\mathcal{U}\left(\mathcal{Y},\mathsf{round}(|\mathcal{Y}|\times
w_{i})\right),$ (18)

where $\mathsf{round}(|\mathcal{Y}|\times w_{i})$ represents the number of
samples to be drawn from the uniform distribution rounded up to the closest
integer.

###  3.4 Training details

We now return to the optimisation of Eq. (8) and Eq. (11), where we define the
neural network $g_{\theta}:\mathcal{X}\to\Delta^{|\mathcal{Y}|-1}$ to
represent the variational posterior $q(\mathbf{y}|\mathbf{x})$. This network
outputs the categorical distribution for the clean label in the probability
simplex space $\Delta^{|\mathcal{Y}|-1}$ given an image
$\mathbf{x}\in\mathcal{X}$. The network
$f_{\phi}:\mathcal{X}\times\Delta^{|\mathcal{Y}|-1}\to\Delta^{|\mathcal{Y}|-1}$
denotes the instance-dependent label transition module
$p(\tilde{\mathbf{y}}|\mathbf{x},\mathbf{y})$, which outputs the categorical
distribution for the noisy training label given an image and the clean label
distribution from $g_{\theta}(.)$.

The first term in the right-hand side (RHS) of Eq. (8) and Eq. (11) is
optimised with the cross-entropy loss to maximise the expected log likelihood
(i.e., the M-step in the EM algorithm):

$\mathcal{L}_{CE}(\theta,\phi,\mathcal{D})=\frac{1}{|\mathcal{D}|\times
K}\sum_{(\mathbf{x}_{i},\tilde{\mathbf{y}}_{i})\in\mathcal{D}}\sum_{j=1}^{K}\ell_{CE}(\tilde{\mathbf{y}}_{i},f_{\phi}(\mathbf{x}_{i},\hat{\mathbf{y}}_{i,j})).$
(19)

where
$\\{\hat{\mathbf{y}}_{i,j}\\}_{j=1}^{K}\sim\mathsf{Cat}(g_{\theta}(\mathbf{x}_{i}))$,
with $\mathsf{Cat}(.)$ denoting a categorical distribution. The second term in
the RHS in Eq. (8) and Eq. (11) uses the estimation of
$p(\mathbf{x}|\mathbf{y})$ from Eq. (12) to minimise the $\mathsf{KL}[.]$
divergence in M-step, as in:

$\begin{split}\mathcal{L}_{PRI}(\theta,\mathcal{D})&=\frac{1}{|\mathcal{D}|}\times\\\
&\sum_{(\mathbf{x}_{i},\tilde{\mathbf{y}}_{i})\in\mathcal{D}}\mathsf{KL}\left[g_{\theta}(\mathbf{x}_{i})\Big{\|}c_{i}\times\frac{g_{\theta}(\mathbf{x}_{i})}{\sum_{j}g_{\theta}(\mathbf{x}_{j})}\odot\mathbf{p}_{i}\right].\end{split}$
(20)

where
$\mathbf{p}_{i}=[p_{i}(j=1),...,p_{i}(j=|\mathcal{Y}|)]\in\Delta^{|\mathcal{Y}|-1}$
is the clean label prior defined in Eq. (15), $c_{i}$ is a normalisation
factor, and $\odot$ is the element-wise multiplication.

The optimisation of the last term in the RHS of Eq. (8) and Eq. (11)
represents the E-step in the EM derivation [8, 20]. This E-step consists of
minimising the $\mathsf{KL}[.]$ divergence with:

$\begin{split}\mathcal{L}_{KL}(\theta,\phi,&\mathcal{D})=\frac{1}{|\mathcal{D}|}\times\\\
&\sum_{(\mathbf{x}_{i},\tilde{\mathbf{y}}_{i})\in\mathcal{D}}\mathsf{KL}\left[g_{\theta}(\mathbf{x}_{i})\Big{\|}f_{\phi}(\mathbf{x}_{i},g_{\theta}(\mathbf{x}_{i}))\odot\mathbf{p}_{i}\right].\end{split}$
(21)

for $\log p(\mathbf{x}|\mathbf{y})$ and

$\begin{split}&\mathcal{L}_{KL}(\theta,\phi,\mathcal{D})=\frac{1}{|\mathcal{D}|}\times\\\
&\sum_{(\mathbf{x}_{i},\tilde{\mathbf{y}}_{i})\in\mathcal{D}}\mathsf{KL}\left[g_{\theta}(\mathbf{x}_{i})\Big{\|}f_{\phi}(\mathbf{x}_{i},g_{\theta}(\mathbf{x}_{i}))\odot\sum_{y=k}^{|\mathcal{Y}|}{\frac{g_{\theta}(\mathbf{x}_{i})}{\sum_{j}g_{\theta}(\mathbf{x}_{j})}\mathbf{p}_{i}}\right],\end{split}$
(22)

for $\log p(\mathbf{y}|\mathbf{x})$.

Hence the complete loss function to minimise is

$\mathcal{L}(\theta,\phi,\mathcal{D})=\mathcal{L}_{CE}(\theta,\phi,\mathcal{D})+\mathcal{L}_{PRI}(\theta,\mathcal{D})+\mathcal{L}_{KL}(\theta,\phi,\mathcal{D}).$
(23)

During testing, a test image $\mathbf{x}$ is associated with a class
categorical distribution produced by $g_{\theta}(\mathbf{x})$. The diagram of
the training and testing of our framework is presented in Fig. 2.

##  4 Experiments

We show experimental results on synthetic benchmarks with CIFAR10 and CIFAR100
datasets [23] on instance-dependent noise and NLP news classification dataset
AGNEWS [66] and 20newsgroup [25] with symmetric/asymmetric/instance-dependent
noise. We also test on real-world datasets, including real-world CIFAR10N and
CIFAR100N [51], Animal-10N [42], Red Mini-ImageNet [16], Clothing1M [55] and
Mini-Webvision [27].

| IDN CIFAR10/100 | CIFAR10/100 N | Red Mini-ImageNet | Animal-10N | Clothing1M | Mini-Webvision | AG NEWS | 20newsgroup
---|---|---|---|---|---|---|---|---
Baseline reference | kMEIDTM [6] | Real-world [52] | FaMUS [57] | Nested [5] | CausalNL [61] | UNICON [18] | DyGEN [70]
Backbone | ResNet-34 | ResNet-34 | Pre-act ResNet-18 | VGG-19BN | ResNet-501 | InceptionNetV2 | BERT2
Training epochs | 150 | 120 | 150 | 100 | 80 | 100 | 10
Batch size | 128 | 128 | 128 | 128 | 64 | 64 | 64
Learning rate | 0.02 | 0.02 | 0.02 | 0.02 | 2e-3 | 0.01 | 1e-4
Optimizer/Weight decay | SGD / 5e-4 | SGD / 5e-4 | SGD / 5e-4 | SGD / 1e-3 | SGD / 1e-3 | SGD / 1e-4 | Adam / –
LR decay at epochs | 0.1 / 100 | 0.1 / 80 | 0.1 / 100 | 0.1 / 50 | 0.1 / 40 | 0.1 / 80 | Linear decay
Data augmentation | Random Crop / Random horizontal flip | –
$\beta$ | 0.9 | 0.9
$K$ | 1 | 1

Table 1: Hyper-parameter setups and baseline references for all experiment
datasets. 1 means ImageNet pre-train. 2 means BERTbase [10] pre-train.

The CIFAR10/100 datasets contain a training set with 50K images and a testing
of 10K images of size 32 $\times$ 32 $\times$ 3, where CIFAR10 has 10 classes
and CIFAR100 has 100 classes. We follow previous works [53] and synthetically
generate instance-dependent noise (IDN) with rates in {0.2, 0.3, 0.4 ,0.5}.
The 20newsgroup dataset contains a training set with 9,051 samples and a
testing set of 7,532 samples with 20 classes. AGNEWS contains a training set
with 40K samples and a testing of 7,600 samples with 4 classes. We follow
previous work DyGEN [70] and synthetically generate
symmetric/asymmetric/instance-dependent noise with rates {0.2, 0.4}. The
CIFAR10N/CIFAR100N datasets have been proposed to study real-world annotations
for the original CIFAR10/100 images and we test our framework on {aggre,
random1, random2, random3, worse} types of noise on CIFAR10N and {noisy} on
CIFAR100N. Red Mini-ImageNet is a real-world dataset with images annotated
with the Google Cloud Data Labelling Service. This dataset has 100 classes,
each containing 600 images from ImageNet, where images are resized to 32
$\times$ 32 pixels from the original 84 $\times$ 84 pixels to enable a fair
comparison with other baselines [57]. Animal 10N is a real-world dataset
containing 10 animal species with five pairs of similar appearances (wolf and
coyote, hamster and guinea pig, etc.). The training set size is 50K and
testing size is 10K, where we follow the same set up as [5]. Clothing1M is a
real-world dataset with 100K images and 14 classes. The labels are
automatically generated from surrounding text with an estimated noise ratio of
38.5%. The dataset also contains clean training, clean validation and clean
test sets with 50K, 14K and 10K images, respectively, but we do not use the
clean training and validation sets. The clean testing is only used for
measuring model performance. Mini-Webvision is a real-world dataset that
contains 2.4 million images collected from the web. Following previous work
[18], we compare baseline methods on the first 50 classes and test on both
webvision test set and ImageNet test set.

###  4.1 Implementation details

In supplementary, Tab.I, we describe the implementation details of our method
for each dataset, including the main reference for the respective baseline. In
addition, for the Clothing1M, we sample 1000 mini-batches from the training
set, where in every mini batch we ensure that the 14 classes are evenly
sampled to form a pseudo-balanced learning problem. Also for Clothing1M, we
first resize the image to 256 $\times$ 256 and then random crop to 224
$\times$ 224 with random horizontal flipping. The warmup number of epochs is
10 for CIFAR and RedMini-ImageNet datasets, and 1 for Clothing1M.

Note that the approximation of the generative model from Eq. (12) is done
within each batch, not the entire the dataset. Also, following the discussion
by Rolf et al. [40], the minimisation of $\mathcal{L}_{PRI}(.)$ can be done
with the reversed KL:

$\begin{split}\mathcal{L}_{PRI\\_R}(\theta,\mathcal{D})&=\frac{1}{|\mathcal{D}|}\times\\\
&\sum_{(\mathbf{x}_{i},\tilde{\mathbf{y}}_{i})\in\mathcal{D}}\mathsf{KL}\left[c_{i}\times\frac{g_{\theta}(\mathbf{x}_{i})}{\sum_{j}g_{\theta}(\mathbf{x}_{j})}\odot\mathbf{p}_{i}\Big{\|}g_{\theta}(\mathbf{x}_{i})\right].\end{split}$
(24)

This reversed KL divergence also provides solutions where the model and
implied posterior are close. In fact, the KL and reversed KL losses are
equivalent when $\sum_{j}g_{\theta}(\mathbf{x}_{j})$ has a uniform
distribution over the classes in $\mathcal{Y}$ and the prior
$p_{i}(\mathbf{y})$ is uniform. We tried the optimisation using both versions
of the KL divergence (i.e., the one in Eq. (20) and Eq. (24)), with the
reversed one generally producing better results, as shown in the ablation
study in Sec. 5.3. For this reason, all experiments in Sec. 4.2 rely on the
reversed KL loss.

For most experiments, we test our approach with different optimisation goals,
$p(X|Y)$ and $p(Y|X)$, following Eq. (21) and Eq. (22), respectively. We
denote these approaches as "Ours - $p(X|Y)$" and "Ours - "$p(Y|X)$". For the
real-world datasets Animal-10N, Red Mini-ImageNet and Clothing1M, we also test
our model with the training and testing of an ensemble of two networks. Our
code is implemented on Pytorch and experiments are performed on RTX 3090.

| CIFAR10 | CIFAR100
---|---|---
Method | $20\%$ | $30\%$ | $40\%$ | $50\%$ | $20\%$ | $30\%$ | $40\%$ | $50\%$
CE | 86.93 $\pm$ 0.17 | 82.42 $\pm$ 0.44 | 76.68 $\pm$ 0.23 | 58.93 $\pm$ 1.54 | 63.94 $\pm$ 0.51 | 61.97 $\pm$ 1.16 | 58.70 $\pm$ 0.56 | 56.63 $\pm$ 0.69
DMI [56] | 89.99 $\pm$ 0.15 | 86.87 $\pm$ 0.34 | 80.74 $\pm$ 0.44 | 63.92 $\pm$ 3.92 | 64.72 $\pm$ 0.64 | 62.8 $\pm$ 1.46 | 60.24 $\pm$ 0.63 | 56.52 $\pm$ 1.18
Forward [37] | 89.62 $\pm$ 0.14 | 86.93 $\pm$ 0.15 | 80.29 $\pm$ 0.27 | 65.91 $\pm$ 1.22 | 67.23 $\pm$ 0.29 | 65.42 $\pm$ 0.63 | 62.18 $\pm$ 0.26 | 58.61 $\pm$ 0.44
CoTeaching [13] | 88.43 $\pm$ 0.08 | 86.40 $\pm$ 0.41 | 80.85 $\pm$ 0.97 | 62.63 $\pm$ 1.51 | 67.40 $\pm$ 0.44 | 64.13 $\pm$ 0.43 | 59.98 $\pm$ 0.28 | 57.48 $\pm$ 0.74
TMDNN [58] | 88.14 $\pm$ 0.66 | 84.55 $\pm$ 0.48 | 79.71 $\pm$ 0.95 | 63.33 $\pm$ 2.75 | 66.62 $\pm$ 0.85 | 64.72 $\pm$ 0.64 | 59.38 $\pm$ 0.65 | 55.68 $\pm$ 1.43
PartT [53] | 89.33 $\pm$ 0.70 | 85.33 $\pm$ 1.86 | 80.59 $\pm$ 0.41 | 64.58 $\pm$ 2.86 | 65.33 $\pm$ 0.59 | 64.56 $\pm$ 1.55 | 59.73 $\pm$ 0.76 | 56.80 $\pm$ 1.32
kMEIDTM [6] | 92.26 $\pm$ 0.25 | 90.73 $\pm$ 0.34 | 85.94 $\pm$ 0.92 | 73.77 $\pm$ 0.82 | 69.16 $\pm$ 0.16 | 66.76 $\pm$ 0.30 | 63.46 $\pm$ 0.48 | 59.18 $\pm$ 0.16
CausalNL [61] | 81.47 $\pm$ 0.32 | 80.38 $\pm$ 0.44 | 77.53 $\pm$ 0.45 | 67.39 $\pm$ 1.24 | 41.47 $\pm$ 0.43 | 40.98 $\pm$ 0.62 | 34.02 $\pm$ 0.95 | 32.13 $\pm$ 2.23
Ours - $p(Y|X)$ | 92.63 $\pm$ 0.23 | 91.39 $\pm$ 0.31 | 90.91 $\pm$ 0.43 | 89.77 $\pm$ 0.54 | 70.73 $\pm$ 0.47 | 68.46 $\pm$ 1.03 | 66.21 $\pm$ 0.42 | 61.07 $\pm$ 0.13
Ours - $p(X|Y)$ | 92.60 $\pm$ 0.44 | 91.96 $\pm$ 0.23 | 91.11 $\pm$ 0.08 | 90.57 $\pm$ 0.18 | 72.51 $\pm$ 0.39 | 70.45 $\pm$ 0.34 | 68.56 $\pm$ 0.36 | 64.00 $\pm$ 0.89

Table 2: Accuracy (%) on the test set for IDN problems on CIFAR10/100. Most
results are from kMEIDTM [6]. Experiments are repeated 5 times to compute mean
$\pm$ standard deviation. Top part shows discriminative and bottom shows
generative models. Best results are highlighted.

Dataset | AG NEWS | 20newsgroup
---|---|---
Method/Noise | 20% SN | 40%SN | 20%ASN | 40%ASN | 20%IDN | 40%IDN | 20%SN | 40%SN | 20%ASN | 40%ASN | 20%IDN | 40%IDN
Base | 82.08 | 78.17 | 81.43 | 77.15 | 85.59 | 75.86 | 78.84 | 70.81 | 74.44 | 56.18 | 77.38 | 69.81
Co-Teaching [13] | 82.99 | 78.79 | 81.96 | 78.07 | 87.85 | 76.52 | 77.66 | 69.25 | 77.51 | 67.26 | 77.45 | 73.76
JoCoR [48] | 83.82 | 81.26 | 85.88 | 77.98 | 87.04 | 79.93 | 80.92 | 73.27 | 81.01 | 69.40 | 81.57 | 74.19
CR [68] | 89.10 | 78.40 | 89.03 | 74.52 | 87.48 | 75.29 | 81.61 | 74.33 | 80.62 | 67.63 | 82.58 | 76.33
DualT [62] | 83.66 | 80.84 | 82.11 | 79.03 | 86.47 | 78.84 | 78.92 | 73.39 | 74.66 | 67.82 | 77.16 | 70.61
CausalNL [61] | 86.44 | 82.74 | 89.87 | 79.80 | 89.00 | 84.62 | 81.08 | 74.43 | 81.22 | 71.25 | 82.57 | 78.91
NPC [3] | 82.83 | 75.04 | 83.94 | 77.69 | 86.28 | 77.38 | 79.82 | 72.96 | 78.88 | 61.69 | 79.97 | 75.19
Ours - $p(Y|X)$ | 89.86 | 89.60 | 88.97 | 87.05 | 89.27 | 87.47 | 81.61 | 76.47 | 80.19 | 75.02 | 83.16 | 80.74
Ours - $p(X|Y)$ | 90.72 | 90.35 | 90.54 | 89.80 | 90.05 | 88.77 | 82.63 | 78.91 | 82.71 | 81.63 | 83.53 | 81.04
CR w/ NPC | 89.69 | 83.21 | 89.01 | 82.54 | 88.25 | 86.41 | 83.09 | 77.96 | 83.13 | 73.50 | 83.47 | 80.47
DyGEN w/ NPC | 91.42 | 89.80 | 91.37 | 90.43 | 91.41 | 88.90 | 83.82 | 79.56 | 83.63 | 81.98 | 84.07 | 81.54
Ours - $p(Y|X)$ w/ NPC | 91.28 | 90.27 | 91.81 | 90.81 | 91.26 | 89.19 | 83.73 | 79.24 | 81.38 | 82.11 | 83.53 | 81.47
Ours - $p(X|Y)$ w/ NPC | 92.34 | 91.63 | 91.87 | 91.37 | 91.78 | 89.15 | 85.15 | 81.07 | 84.65 | 84.18 | 84.58 | 82.34

Table 3: Accuracy (%) on the NLP benchmarks of news topic classification with different label noise types (SN = symmetric noise, ASN = asymmetric noise and IDN = instance dependent noise) and rates. All baselines are taken from DyGen [70]. In both tables, the bottom part shows methods that use NPC as a post-processing method. Best results are highlighted in each part of the table. | CIFAR10N | CIFAR100N
---|---|---
Method | Aggregate | Random 1 | Random 2 | Random 3 | Worst | Noisy
CE | 87.77 $\pm$ 0.38 | 85.02 $\pm$ 0.65 | 86.46 $\pm$ 1.79 | 85.16 $\pm$ 0.61 | 77.69 $\pm$ 1.55 | 55.50 $\pm$ 0.66
Forward T [37] | 88.24 $\pm$ 0.22 | 86.88 $\pm$ 0.50 | 86.14 $\pm$ 0.24 | 87.04 $\pm$ 0.35 | 79.79 $\pm$ 0.46 | 57.01 $\pm$ 1.03
T-Revision [54] | 88.52 $\pm$ 0.17 | 88.33 $\pm$ 0.32 | 87.71 $\pm$ 1.02 | 80.48 $\pm$ 1.20 | 80.48 $\pm$ 1.20 | 51.55 $\pm$ 0.31
Peer Loss [32] | 90.75 $\pm$ 0.25 | 89.06 $\pm$ 0.11 | 88.76 $\pm$ 0.19 | 88.57 $\pm$ 0.09 | 82.00 $\pm$ 0.60 | 57.59 $\pm$ 0.61
Positive-LS [33] | 91.57 $\pm$ 0.07 | 89.80 $\pm$ 0.28 | 89.35 $\pm$ 0.33 | 89.82 $\pm$ 0.14 | 82.76 $\pm$ 0.53 | 55.84 $\pm$ 0.48
F-Div [50] | 91.64 $\pm$ 0.34 | 89.70 $\pm$ 0.40 | 89.79 $\pm$ 0.12 | 89.55 $\pm$ 0.49 | 82.53 $\pm$ 0.52 | 57.10 $\pm$ 0.65
Negative-LS [49] | 91.97 $\pm$ 0.46 | 90.29 $\pm$ 0.32 | 90.37 $\pm$ 0.12 | 90.13 $\pm$ 0.19 | 82.99 $\pm$ 0.36 | 58.59 $\pm$ 0.98
CORES2 [7] | 91.23 $\pm$ 0.11 | 89.66 $\pm$ 0.32 | 89.91 $\pm$ 0.45 | 89.79 $\pm$ 0.50 | 83.60 $\pm$ 0.53 | 61.15 $\pm$ 0.73
VolMinNet [28] | 89.70 $\pm$ 0.21 | 88.30 $\pm$ 0.12 | 88.27 $\pm$ 0.09 | 88.19 $\pm$ 0.41 | 80.53 $\pm$ 0.20 | 57.80 $\pm$ 0.31
CAL [69] | 91.97 $\pm$ 0.32 | 90.93 $\pm$ 0.31 | 90.75 $\pm$ 0.30 | 90.74 $\pm$ 0.24 | 85.36 $\pm$ 0.16 | 61.73 $\pm$ 0.42
Ours - $p(Y|X)$ | 92.41 $\pm$ 0.25 | 91.04 $\pm$ 0.03 | 91.19 $\pm$ 0.30 | 91.11 $\pm$ 0.45 | 85.67 $\pm$ 0.62 | 59.03 $\pm$ 0.44
Ours - $p(X|Y)$ | 92.57 $\pm$ 0.20 | 91.97 $\pm$ 0.09 | 91.42 $\pm$ 0.06 | 91.83 $\pm$ 0.12 | 86.99 $\pm$ 0.36 | 61.54 $\pm$ 0.22
Table 4: Accuracy (%) on the test set for CIFAR10N/100N. Results are taken from [52] using methods containing a single classifier with ResNet-34. Best results are highlighted.  Method | Accuracy
---|---
CE | 79.4
SELFIE [42] | 81.8
PLC [67] | 83.4
Ours - $p(Y|X)$ | 82.4
Ours - $p(X|Y)$ | 84.7
JoCoR [48] | 82.8
Nested + Co-T [5] | 84.1
InstanceGM [12] | 84.6
Ours - $p(Y|X)$ ensemble | 84.1
Ours - $p(X|Y)$ ensemble | 85.9
Table 5: Test accuracy (%) on Animal-10N with different noise rates and baselines from InstanceGM [11]. Upper part shows single classifier comparison and bottom part displays the ensemble of classifiers results. | Noise rate
---|---
Method | 0.2 | 0.4 | 0.6 | 0.8
CE | 47.36 | 42.70 | 37.30 | 29.76
Mixup [65] | 49.10 | 46.40 | 40.58 | 33.58
Ours - $p(Y|X)$ | 54.53 | 50.23 | 45.12 | 35.78
Ours - $p(X|Y)$ | 53.34 | 49.56 | 44.08 | 34.70
DivideMix [26] | 50.96 | 46.72 | 43.14 | 34.50
MentorMix [15] | 51.02 | 47.14 | 43.80 | 33.46
FaMUS [57] | 51.42 | 48.06 | 45.10 | 35.50
Ours - $p(Y|X)$ ensemble | 58.46 | 53.38 | 48.92 | 40.68
Ours - $p(X|Y)$ ensemble | 57.56 | 52.68 | 47.12 | 39.54
Table 6: Test accuracy (%) on Red Mini-ImageNet with different noise rates and
baselines from FaMUS [57]. Upper part shows single classifier comparison and
bottom part displays the ensemble of classifiers results. Best results are
highlighted in both setups.

Method | Accuracy
---|---
CE | 68.94
Forward [37] | 69.84
PTD-R-V [53] | 71.67
ELR [30] | 72.87
kMEIDTM [6] | 73.84
CausalNL [61] | 72.24
Ours - $p(Y|X)$ (ensemble) | 73.92
Ours - $p(X|Y)$ (ensemble) | 74.35

Dataset/Methods | Webvision | ImageNet
---|---|---
Forward [37] | 61.10 | 57.36
Co-teaching [13] | 63.58 | 61.48
ELR+ [30] | 77.78 | 70.29
UNICON [18] | 77.60 | 75.29
| Ours - $p(Y|X)$
---
(ensemble)
78.72 | 75.28
| Ours - $p(X|Y)$
---
(ensemble)
76.32 | 72.08

Table 7: Left: Test accuracy (%) on the test set of Clothing1M. Results are
obtained from the respective papers. We only use the noisy training set for
training. Right: Top1 Test accuracy (%) of Mini-Webvision and ImageNet.
Results are obtained from the respective papers. Best results is highlight and
second best is underlined.

###  4.2 Experimental Results

Synthetic benchmarks. The experimental results of our method with IDN problems
on CIFAR10/100 are shown in Tab.2. Compared with the previous SOTA kMEDITM
[6], on CIFAR10, we achieve competitive performance on low noise rates and up
to 16% improvements for high noise rates. For CIFAR100, we consistently
improve 2% to 4% in all noise rates. Compared with the previous SOTA
generative model CausalNL [61], our improvement is significant for all noise
rates. The superior performance of our method indicates that our implicit
generative modelling and clean label prior construction is effective when
learning with label noise. For the two variants of our approach, i.e.,
$p(X|Y)$ and $p(Y|X)$, “Ours - $p(X|Y)$” generally shows better performance.
CIFAR10 dataset [23] is collected based on the given ten classes as a subset
of tiny images dataset. This result aligns with the causal relationship from
[60] that suggests that CIFAR10 [23] is collected based on the given ten
classes as a subset of tiny image dataset, where $Y$ causes $X$.

The experimental results with NLP benchmarks are shown in Tab. 3. Compared
with previous SOTA DyGEN [70], we achieve SOTA results on both datasets with
different noise types and noise rates. Without the NPC post-processing [3], we
consistently improve 4% to 10% for all conditions. After NPC post-processing,
we provide further improvements of 2%, compared with DyGEN. Such improvements
indicates that our method can learn with label noise on different data
modalities (images and text), and different types and rates of label noise.

Real-world benchmarks.  In Tab. 4, we show the performance of our method on
the CIFAR10N/100N benchmarks. Compared with other singlEModel baselines, our
method achieves at least 1% improvement on all noise rates on CIFAR10N, and it
has a competitive performance on CIFAR100N. Compared with CAL [69], which has
a complex multi-stage training process, our method has a much simpler end-to-
end training process. Comparing the two optimisation goals, the conclusion for
CIFAR10N/100N follows the same conclusion reached for CIFAR-IDN, where
$p(X|Y)$ tends to perform better than $p(Y|X)$.

The Red Mini-ImageNet results in Tab. 6 show that our method achieves SOTA
results for all noise rates with 2% improvements using a single model and 6%
improvements using the ensemble of two models. In particular, the improvement
is substantial compared with previous SOTA FaMUS [57] and DivideMix [26]. We
observed that $p(Y|X)$ shows superior performance on Red Mini-ImageNet, while
$p(X|Y)$ performs better on other datasets. We hypothesize that this is
because ImageNet [9], which Red Mini-ImageNet is derived from, follows a
different generation process compared with CIFAR, where images are web-crawled
first and then annotated by humans. This implies that the data distribution is
more diverse and complex than the label distribution, and thus $p(Y|X)$ can
better capture the conditional dependency. Moreover, ImageNet may contain Out-
Of-Distribution (OOD) samples that do not belong to any of the predefined
classes, which can negatively affect the optimisation of $p(X|Y)$, as it tries
to fit the data distribution regardless of the label quality.

In Tab. 7, we present results on Animal-10N (Left), Clothing1M (Middle) and
Mini-Webvision (Right). Our singlEModel result on Animal-10N achieves 1%
improvement with respect to the singlEModel SELFIE [42]. Under the setup with
two models (referred to as ’ensemble’), we achieve a 1% improvement over the
SOTA Nested Co-teaching [5]. For Clothing1M, our ensemble result shows a
competitive performance of 74.4%, which is 2% better than the previous SOTA
generative model CausalNL [61]. For Mini-Webvision, we outperform the previous
semi-supervised SOTA approach UNICON [18] by 1% in Top-1 accuracy in the
Webvision test set and present competitive performance on ImageNet test set.
For the performance differences between $p(X|Y)$ and $p(Y|X)$, we observed
that $p(X|Y)$ performs better in Animal-10N and Clothing1M datasets and
$p(Y|X)$ performs better in Mini-Webvision. Similar to CIFAR dataset,
Animal-10N and Clothing1M are collected based on the given classes. Clothing1M
dataset further discards images if the surrounding caption does not contain
the given 14 classes [55]. On the other hand, Mini-Webvision dataset is known
for containing OOD samples due to collecting images from web, which suggests
that this dataset can be better represented by $p(Y|X)$.

| CE | DivideMix [26] | CausalNL [61] | InstanceGM [12] | Ours
---|---|---|---|---|---
CIFAR | 2.1h | 7.1h | 3.3h | 30.5h | 2.3h
Clothing1M | 8h | 14h | 12h | 43h | 8.5h

Table 8: Training times of various methods on CIFAR100 with 50% IDN and
Clothing1M using the hardware listed in Sec. 4.1

Training time comparison One of the advantages of our approach is its
efficient training algorithm, particularly when compared with other generative
and discriminative methods. Tab. 8 shows the training time for competing
approaches on CIFAR100 with 50% IDN and Clothing1M using the hardware
specified in Sec. 4.1. In general, our method has a smaller training time than
competing approaches, being equivalent to the training with CE loss, around
$2\times$ faster than CausalNL [61], $3\times$ faster than DivideMix [26], and
10$\times$ faster than InstanceGM [12].

##  5 Ablation studies

###  5.1 Transition matrix estimation

Figure 3: CIFAR10 IDN 20% and 40% transition matrix MSE error and
classification accuracy. Results are taken from NPC [3] and kMEIDTM [6]. The
right-hand side y-axis shows accuracy and left-hand side y-axis displays
transition matrix MSE (x100).

In Fig. 3, we show mean square error (MSE) of transition matrix estimation and
classification accuracy on CIFAR10 IDN with 20% and 40% noise rates, compared
with SOTA transition matrix methods and generative modelling methods. Compared
with previous transition matrix methods Forward [37] and kMEIDTM [6], we
observe that our method achieves a lower transition matrix error and higher
accuracy for both noise rates. Compared with the generative methods CausalNL
[61] and NPC [3], our method achieves comparable transition matrix estimation
error, but significantly higher accuracy. This indicates the effectiveness of
our generative modelling for instance-dependent transition matrix estimation.

###  5.2 Hyper-parameter analysis

| CIFAR10-IDN
---|---
| 20% | 30% | 40% | 50%
$\beta=0.9$, $K=1$ | 92.60 | 91.96 | 91.11 | 90.57
| $\beta=0.8$ | 92.49 | 91.88 | 90.83 | 88.81
| $\beta=0.7$ | 91.55 | 90.87 | 90.62 | 88.40
| $\beta=0.5$ | 89.13 | 87.98 | 87.48 | 85.73
Hyper-parameter | $K=3$ | 92.30 | 91.83 | 90.83 | 89.75
| $\beta=0$ | 84.57 | 81.59 | 68.88 | 61.47
| $\arg\max$ | 20.19 | 18.56 | 16.09 | 15.26
Coverage | No Cov | 85.57 | 81.00 | 72.42 | 66.61
| Uniform $w$ | 90.10 | 89.66 | 86.25 | 84.28
Uncertainty | No Unc | 84.96 | 83.19 | 81.88 | 78.38
Table 9: Test accuracy (%) on CIFAR-10-IDN to study the hyper-parameters
$\beta$ in Eq. (16), $K$ in Eq. (19), and coverage and uncertainty from Eq.
(14).

In Tab. 9, we perform a hyper-parameter sensitivity test for our method on
CIFAR10-IDN, including coverage and uncertainty for prior label construction.
To test label coverage, we first examine the model performance as function of
$\beta\in\\{0.5,0.7,0.8,0.9\\}$ in Eq. (16), where the default value for
$\beta$ is $0.9$. We observe that performance does not change much for
$\beta\in\\{0.7,0.8,0.9\\}$, which indicates our model’s robustness with
respect to that hyper-parameter. For $\beta=0.5$, the performance drops
significantly, indicating that using a moving average with a relatively high
value for $\beta$ is important to avoid overfitting. We test $K\in\\{1,3\\}$
in Eq. (19) by sampling multiple times
$\\{\hat{\mathbf{y}}_{i,j}\\}_{j=1}^{K}\sim\mathsf{Cat}(g_{\theta}(\mathbf{x}_{i}))$.
We observe no significant changes to model performance with this higher value
of $K$. Therefore, we choose $K=1$ for simplicity.

We also test our model by shutting down the moving average, either by making
$\beta=0$ or by completely relying on the model’s current prediction, which is
an experiment denoted by "$\arg\max$" in Tab. 9. Note that the model shows a
major performance drop in both cases. This happens because the model overfits
to the inaccurate model predictions, biasing the training procedure.
Furthermore, $\arg\max$ has the worst performance because inaccurate model
predictions present in label prior at early training stages will lead to
confirmation bias. We also test our model without using the coverage term
$\mathbf{c}_{i}$ in Eq. (14) – this experiment is denoted as “No Cov”. In this
case, the performance of the model drops significantly for all noise rates,
compared to the default model in the first row, which indicates the importance
of having a coverage term to cover latent clean label in our prior label
construction.

Furthermore, we study the uncertainty aspect for the prior label construction.
We first experiment by setting $w_{i}$ to a uniform value (“Uniform $w$” row)
instead of the result from the unsupervised 2-class classification that
represents the probability that the sample is carrying a clean label. The
result is not competitive with the one that uses the unsupervised 2-class
classification weight, which indicates the importance of having $w$
representing a clean-label sample probability. We also test our model without
the uncertainty component $\mathbf{u}_{i}$ in Eq. (14) (see row “No Unc”).
This case shows a significant performance drop in all noise rates. These
experiments highlight the importance of uncertainty term as a regularization
on the label prior for noisy label learning.

###  5.3 Loss function analysis

We show an ablation analysis of our loss function on CIFAR10-IDN in Tab. 10.
For row#1, we use the $\mathcal{L}_{CE}(.)$ loss in Eq. (19) to train
$p(\tilde{\mathbf{y}}|\mathbf{y},\mathbf{x})$ to represent the baseline
result. For row#2, we train $q(\mathbf{y}|\mathbf{x})$ with the reverse KL
defined in Eq. (24), so we can assess the learning using partial label
directly, but without EM optimisation. Such an approach does not work well, as
evidenced by the experiments that show significant worse results in low noise
rate, and for high noise rates, training fails to converge, suggesting the
importance of our formulation and the EM optimisation. In row#3, we train
$p(\tilde{\mathbf{y}}|\mathbf{y},\mathbf{x})$ with Eq. (19), using the E-step
from Eq. (8) and $q(\mathbf{y}|\mathbf{x})$ with cross-entropy loss on soft
labels with partial label, defined by:

$\begin{split}\mathcal{L}_{CE\\_PRI}(\theta,\mathcal{D})=\frac{1}{|\mathcal{D}|}\times\sum_{(\mathbf{x}_{i},\tilde{\mathbf{y}}_{i})\in\mathcal{D}}\mathbf{p}_{i}\log
g_{\theta}(\mathbf{x}_{i}),\end{split}$ (25)

which tests if the constructed partial label can directly supervise (with
cross-entropy loss) the learning of the variational posterior. This experiment
shows that such a replacement causes a performance drop in all noise rates,
demonstrating the importance of our loss $\mathcal{L}_{PRI}$ and the
approximation of the generative model $p(\mathbf{x}|\mathbf{y})$. For row#4,
we use our method that relies on the original KL divergence in Eq. (20)
instead of the reversed KL divergence in Eq. (24).Even though results are
competitive, they are worse than the reverse KL in the next rows, which can be
justified by the argument in [47], which states that the original KL
divergence tends to converge better, but to be less robust, while the reverse
KL is more robust to label noise, but tends to have worse convergence. Another
important point is that when $\mathbf{p}_{i}$ is approximately a one-hot
vector, which can cause numerical problems in the KL divergence in Eq. (20).
In row#5, we study the contribution of Eq. (21) by training without the
E-step, which causes a performance drop for high noise rates. row#6 shows the
results from the complete objective function using Eq. (8) and Eq. (24), which
produces the best performance in all noise rates.

|  |  |  | CIFAR10-IDN
---|---|---|---|---
| $p(\tilde{\mathbf{y}}|\mathbf{y},\mathbf{x})$ | $q(\mathbf{y}|\mathbf{x})$ | E-step $KL$ | 20% | 30% | 40% | 50%
#1 | ✓ |  |  | 86.93 | 82.43 | 76.68 | 58.93
#2 |  | Eq. (24) |  | 43.19 | N/A | N/A | N/A
#3 | ✓ | Eq. (25) | ✓ | 85.96 | 82.74 | 78.34 | 73.72
#4 | ✓ | Eq. (20) | ✓ | 91.36 | 90.88 | 90.23 | 88.47
#5 | ✓ | Eq. (24) |  | 92.40 | 90.23 | 87.85 | 80.46
#6 | ✓ | Eq. (24) | ✓ | 92.60 | 91.96 | 91.11 | 90.57

Table 10: Ablation study of our proposed loss function. N/A indicates model
collapse – please see Sec. 5.3 for more details.

###  5.4 Coverage and uncertainty visualisation

(a) CIFAR10-IDN at 20% and 50% noise rates

(b) CIFAR100-IDN at 20% and 50% noise rates

(c) CIFAR10N at "Worse" and "Aggre".

Figure 4: Eq. (14) Coverage (Cov) and Uncertainty (Unc) for (a) CIFAR10-IDN
(20% and 50%), (b) CIFAR100-IDN (20% and 50%) and (c) CIFAR10N ("Worse" and
"Aggre"). Left-column graphs show clean label coverage. Middle-column graphs
show clean/noisy label sample uncertainty under the proposed PLS trained
posterior $q(\mathbf{y}|\mathbf{x})$. The right-column graph shows clean/noisy
label samples uncertainty with the training of $q(\mathbf{y}|\mathbf{x})$
without our proposed PLS. The dotted vertical line indicates the end of warmup
training.

We visualise the coverage and uncertainty of our label prior from Eq. (14) at
each training epoch for IDN CIFAR10/100 and CIFAR10N setups in Fig. 4. In all
cases, label coverage increases as training progresses, indicating that our
label prior tends to always cover the clean label. In fact, coverage reaches
nearly 100% for CIFAR10 at 20% IDN and 96% for 50% IDN. For CIFAR100 at 50%
IDN, we achieve 84% coverage, and for CIFAR10N "worse", we reach 90% coverage.
In terms of clean label uncertainty and noisy label uncertainty, we observe a
clear gap between different training samples. For clean label uncertainty, we
notice a steady reduction as training progresses for all datasets, which
indicates that our $p(\mathbf{y})$ can recover the desired one-hot label for
clean labeled samples. For noisy label samples, we notice $p(\mathbf{y})$ has
high uncertainty during training, which regularizes the training of the noisy
label samples.

In the rightmost column of Fig. 4, we plot the clean and noisy label
uncertainty by replacing the $\bar{\mathbf{y}}$ (i.e., the class probability
distribution produced by $q(\mathbf{y}|\mathbf{x})$) in Eq. (17) with
transition output $p(\tilde{\mathbf{y}}|\mathbf{y},\mathbf{x})$, which follows
traditional small-loss criterion [26]. This modification affects the
calculation of $w_{i}$ in Eq. (17) and $p(\mathbf{y})$ construction. In this
case, we observe a significantly unstable uncertainty of $p(\mathbf{y})$ for
both clean and noisy label samples. This highlights the importance of our
$q(\mathbf{y}|\mathbf{x})$ trained under PLS and its contribution in the
construction of sample-wise $p(\mathbf{y})$. The results suggest that our
label prior distribution is effective at selecting the correct clean label
while reducing the number of label candidates during training.

##  6 Conclusion

In this paper, we presented a new learning algorithm to optimise a generative
model represented by $p(X|Y)$ or $p(Y|X)$ that directly associate data and
clean labels. Our optimisation implicitly estimates $p(X|Y)$ with the
discriminative model $q(Y|X)$ [40] eliminating the inefficient generative
model training. Furthermore, we introduce an informative clean label prior,
inspired by partial-label learning [39], to cover a small number of label
candidates when the model is certain about the training label, and to cover a
large number of label candidates, otherwise. Results on synthetic and real-
world noisy-label benchmarks show that our generative method has SOTA results,
but with complexity comparable to the most efficient discriminative models
available in the literature.

A limitation of the proposed method is the missing comprehensive study of the
model $q(Y|X)$. In fact, the results shown in this paper are surprisingly
competitive given that we use fairly standard models for $q(Y|X)$ without
exploring sophisticated noisy-label learning techniques. In the future, we
will use more powerful techniques for $q(Y|X)$, including data augmentation,
combined with semi-supervised learning and noisy partial label learning.
Another issue of our model is the dataset-dependent choice of optimisation
goal (i.e., $p(Y|X)$ or $p(X|Y)$). In future work, we will explore dataset-
specific causal discovery techniques [60]. Finally, we will study more
adequate ways to approximate $p(X|Y)$ using data augmentation strategies or
using the whole training set (instead of using only mini batches) to increase
the robustness of the approximation.

## References

  * [1] E. Arazo, D. Ortego, P. Albert, N. O’Connor, and K. McGuinness.  Unsupervised label noise modeling and loss correction.  In International conference on machine learning, pages 312–321. PMLR, 2019.
  * [2] D. Arpit, S. Jastrzębski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj, A. Fischer, A. Courville, Y. Bengio, et al.  A closer look at memorization in deep networks.  In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 233–242. JMLR. org, 2017.
  * [3] H. Bae, S. Shin, B. Na, J. Jang, K. Song, and I.-C. Moon.  From noisy prediction to true label: Noisy prediction calibration via generative model, 2022.
  * [4] P. Chen, B. B. Liao, G. Chen, and S. Zhang.  Understanding and utilizing deep neural networks trained with noisy labels.  In International Conference on Machine Learning, pages 1062–1070. PMLR, 2019.
  * [5] Y. Chen and et al.  Boosting co-teaching with compression regularization for label noise.  In CVPR, pages 2688–2692, 2021.
  * [6] D. Cheng and et al.  Instance-dependent label-noise learning with manifold-regularized transition matrix estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16630–16639, 2022.
  * [7] H. Cheng, Z. Zhu, X. Li, Y. Gong, X. Sun, and Y. Liu.  Learning with instance-dependent label noise: A sample sieve approach.  In International Conference on Learning Representations, 2021.
  * [8] A. P. Dempster, N. M. Laird, and D. B. Rubin.  Maximum likelihood from incomplete data via the em algorithm.  Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1–22, 1977.
  * [9] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [10] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [11] A. Garg and et al.  Instance-dependent noisy label learning via graphical modelling.  WACV, 2022.
  * [12] A. Garg, C. Nguyen, R. Felix, T.-T. Do, and G. Carneiro.  Instance-dependent noisy label learning via graphical modelling.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2288–2298, 2023.
  * [13] B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama.  Co-teaching: Robust training of deep neural networks with extremely noisy labels.  In Advances in neural information processing systems, pages 8527–8537, 2018.
  * [14] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learningfor image recognition.  ComputerScience, 2015.
  * [15] L. Jiang, D. Huang, M. Liu, and W. Yang.  Beyond synthetic noise: Deep learning on controlled noisy labels.  In International Conference on Machine Learning, pages 4804–4815. PMLR, 2020.
  * [16] L. Jiang, Z. Zhou, T. Leung, L.-J. Li, and L. Fei-Fei.  Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels.  In International Conference on Machine Learning, pages 2304–2313. PMLR, 2018.
  * [17] T. Kaiser, L. Ehmann, C. Reinders, and B. Rosenhahn.  Blind knowledge distillation for robust image classification.  arXiv preprint arXiv:2211.11355, 2022.
  * [18] N. Karim and et al.  Unicon: Combating label noise through uniform selection and contrastive learning.  In CVPR, pages 9676–9686, 2022.
  * [19] T. Kim, J. Ko, J. Choi, S.-Y. Yun, et al.  Fine samples for learning with noisy labels.  Advances in Neural Information Processing Systems, 34:24137–24149, 2021.
  * [20] D. P. Kingma.  Variational inference & deep learning: A new synthesis.  2017\.
  * [21] D. P. Kingma and M. Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [22] J. M. Köhler, M. Autenrieth, and W. H. Beluch.  Uncertainty based detection and relabeling of noisy image labels.  In CVPR Workshops, pages 33–37, 2019.
  * [23] A. Krizhevsky, G. Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [24] A. Krizhevsky, I. Sutskever, and G. E. Hinton.  Imagenet classification with deep convolutional neural networks.  Communications of the ACM, 60(6):84–90, 2017.
  * [25] K. Lang.  Newsweeder: Learning to filter netnews.  In Machine learning proceedings 1995, pages 331–339. Elsevier, 1995\.
  * [26] J. Li and et al.  Dividemix: Learning with noisy labels as semi-supervised learning.  ICLR, 2020.
  * [27] W. Li, L. Wang, W. Li, E. Agustsson, and L. Van Gool.  Webvision database: Visual learning and understanding from web data.  arXiv preprint arXiv:1708.02862, 2017.
  * [28] X. Li, T. Liu, B. Han, G. Niu, and M. Sugiyama.  Provably end-to-end label-noise learning without anchor points.  arXiv preprint arXiv:2102.02400, 2021.
  * [29] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. Van Der Laak, B. Van Ginneken, and C. I. Sánchez.  A survey on deep learning in medical image analysis.  Medical image analysis, 42:60–88, 2017.
  * [30] S. Liu, J. Niles-Weed, N. Razavian, and C. Fernandez-Granda.  Early-learning regularization prevents memorization of noisy labels.  Advances in neural information processing systems, 33:20331–20342, 2020.
  * [31] Y. Liu, H. Cheng, and K. Zhang.  Identifiability of label noise transition matrix.  arXiv preprint arXiv:2202.02016, 2022.
  * [32] Y. Liu and H. Guo.  Peer loss functions: Learning from noisy labels without knowing noise rates.  In International Conference on Machine Learning, pages 6226–6236. PMLR, 2020.
  * [33] M. Lukasik, S. Bhojanapalli, A. K. Menon, and S. Kumar.  Does label smoothing mitigate label noise?  In International Conference on Machine Learning, 2020.
  * [34] J. Lv, M. Xu, L. Feng, G. Niu, X. Geng, and M. Sugiyama.  Progressive identification of true labels for partial-label learning.  In international conference on machine learning, pages 6500–6510. PMLR, 2020.
  * [35] E. Malach and S. Shalev-Shwartz.  Decoupling" when to update" from" how to update".  Advances in neural information processing systems, 30, 2017.
  * [36] D. Ortego, E. Arazo, P. Albert, N. E. O’Connor, and K. McGuinness.  Multi-objective interpolation training for robustness to label noise.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6606–6615, 2021.
  * [37] G. Patrini, A. Rozza, A. Krishna Menon, R. Nock, and L. Qu.  Making deep neural networks robust to label noise: A loss correction approach.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1944–1952, 2017.
  * [38] J. Pearl.  Probabilistic reasoning in intelligent systems: networks of plausible inference.  Morgan kaufmann, 1988.
  * [39] C. Qiao, N. Xu, and X. Geng.  Decompositional generation process for instance-dependent partial label learning.  In The Eleventh International Conference on Learning Representations, 2022.
  * [40] E. Rolf, N. Malkin, A. Graikos, A. Jojic, C. Robinson, and N. Jojic.  Resolving label uncertainty with implicit posterior models.  arXiv preprint arXiv:2202.14000, 2022.
  * [41] Y. Shen and S. Sanghavi.  Learning with bad training data via iterative trimmed loss minimization.  In International Conference on Machine Learning, pages 5739–5748. PMLR, 2019.
  * [42] H. Song, M. Kim, and J.-G. Lee.  Selfie: Refurbishing unclean samples for robust deep learning.  In International Conference on Machine Learning, pages 5907–5915. PMLR, 2019.
  * [43] Y. Tian, X. Yu, and S. Fu.  Partial label learning: Taxonomy, analysis and outlook.  Neural Networks, 2023.
  * [44] H. Wang, R. Xiao, Y. Li, L. Feng, G. Niu, G. Chen, and J. Zhao.  Pico+: Contrastive label disambiguation for robust partial label learning, 2022.
  * [45] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers.  Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2097–2106, 2017.
  * [46] Y. Wang, W. Liu, X. Ma, J. Bailey, H. Zha, L. Song, and S.-T. Xia.  Iterative learning with open-set noisy labels.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8688–8696, 2018.
  * [47] Y. Wang, X. Ma, Z. Chen, Y. Luo, J. Yi, and J. Bailey.  Symmetric cross entropy for robust learning with noisy labels.  In Proceedings of the IEEE International Conference on Computer Vision, pages 322–330, 2019.
  * [48] H. Wei, L. Feng, X. Chen, and B. An.  Combating noisy labels by agreement: A joint training method with co-regularization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13726–13735, 2020.
  * [49] J. Wei, H. Liu, T. Liu, G. Niu, and Y. Liu.  Understanding (generalized) label smoothing whenlearning with noisy labels.  arXiv preprint arXiv:2106.04149, 2021.
  * [50] J. Wei and Y. Liu.  When optimizing f-divergence is robust with label noise.  In International Conference on Learning Representation, 2021.
  * [51] J. Wei, Z. Zhu, H. Cheng, T. Liu, G. Niu, and Y. Liu.  Learning with noisy labels revisited: A study using real-world human annotations.  In International Conference on Learning Representations, 2021.
  * [52] J. Wei, Z. Zhu, H. Cheng, T. Liu, G. Niu, and Y. Liu.  Learning with noisy labels revisited: A study using real-world human annotations.  In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
  * [53] X. Xia, T. Liu, B. Han, N. Wang, M. Gong, H. Liu, G. Niu, D. Tao, and M. Sugiyama.  Part-dependent label noise: Towards instance-dependent label noise.  Advances in Neural Information Processing Systems, 33:7597–7610, 2020.
  * [54] X. Xia, T. Liu, N. Wang, B. Han, C. Gong, G. Niu, and M. Sugiyama.  Are anchor points really indispensable in label-noise learning?  Advances in Neural Information Processing Systems, 32, 2019.
  * [55] T. Xiao, T. Xia, Y. Yang, C. Huang, and X. Wang.  Learning from massive noisy labeled data for image classification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2691–2699, 2015.
  * [56] Y. Xu, P. Cao, Y. Kong, and Y. Wang.  L_dmi: A novel information-theoretic loss function for training deep nets robust to label noise.  In Advances in Neural Information Processing Systems, pages 6225–6236, 2019.
  * [57] Y. Xu, L. Zhu, L. Jiang, and Y. Yang.  Faster meta update strategy for noise-robust deep learning.  In CVPR, pages 144–153, 2021.
  * [58] S. Yang, E. Yang, B. Han, Y. Liu, M. Xu, G. Niu, and T. Liu.  Estimating instance-dependent bayes-label transition matrix using a deep neural network.  In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 25302–25312. PMLR, 2022.
  * [59] Y. Yao, M. Gong, Y. Du, J. Yu, B. Han, K. Zhang, and T. Liu.  Which is better for learning with noisy labels: The semi-supervised method or modeling label noise?  In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 39660–39673. PMLR, 23–29 Jul 2023.
  * [60] Y. Yao, M. Gong, Y. Du, J. Yu, B. Han, K. Zhang, and T. Liu.  Which is better for learning with noisy labels: the semi-supervised method or modeling label noise?  In International Conference on Machine Learning, pages 39660–39673. PMLR, 2023.
  * [61] Y. Yao, T. Liu, M. Gong, B. Han, G. Niu, and K. Zhang.  Instance-dependent label-noise learning under a structural causal model.  Advances in Neural Information Processing Systems, 34:4409–4420, 2021.
  * [62] Y. Yao, T. Liu, B. Han, M. Gong, J. Deng, G. Niu, and M. Sugiyama.  Dual T: Reducing estimation error for transition matrix in label-noise learning.  In Advances in Neural Information Processing Systems, volume 33, pages 7260–7271, 2020.
  * [63] T. Young, D. Hazarika, S. Poria, and E. Cambria.  Recent trends in deep learning based natural language processing.  ieee Computational intelligenCe magazine, 13(3):55–75, 2018.
  * [64] F. Zhang, L. Feng, B. Han, T. Liu, G. Niu, T. Qin, and M. Sugiyama.  Exploiting class activation value for partial-label learning.  In International Conference on Learning Representations, 2021.
  * [65] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz.  mixup: Beyond empirical risk minimization.  arXiv preprint arXiv:1710.09412, 2017.
  * [66] X. Zhang, J. Zhao, and Y. LeCun.  Character-level convolutional networks for text classification.  Advances in neural information processing systems, 28, 2015.
  * [67] Y. Zhang, S. Zheng, P. Wu, M. Goswami, and C. Chen.  Learning with feature-dependent label noise: A progressive approach.  arXiv preprint arXiv:2103.07756, 2021.
  * [68] W. Zhou and M. Chen.  Learning from noisy labels for entity-centric information extraction.  arXiv preprint arXiv:2104.08656, 2021.
  * [69] Z. Zhu, T. Liu, and Y. Liu.  A second-order approach to learning with instance-dependent label noise.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10113–10123, 2021.
  * [70] Y. Zhuang, Y. Yu, L. Kong, X. Chen, and C. Zhang.  Dygen: Learning from noisy labels via dynamics-enhanced generative modeling.  arXiv preprint arXiv:2305.19395, 2023.