#  Elucidating Noisy Data
via Uncertainty-Aware Robust Learning

Jeongeun Park<EMAIL_ADDRESS>Seungyoun Shin  Sangheum Hwang  Sungjoon
Choi<EMAIL_ADDRESS>Department of Artificial Intelligence, Korea
University, Seoul 02841, Korea  Department Computer Engineering, Dongguk
University, Seoul 04620, Korea  Department of Data Science, Seoul National
University of Science and Technology,
Seoul 01811, Korea

###### Abstract

Robust learning methods aim to learn a clean target distribution from noisy
and corrupted training data where a specific corruption pattern is often
assumed a priori. Our proposed method can not only successfully learn the
clean target distribution from a dirty dataset but also can estimate the
underlying noise pattern. To this end, we leverage a mixture-of-experts model
that can distinguish two different types of predictive uncertainty, aleatoric
and epistemic uncertainty. We show that the ability to estimate the
uncertainty plays a significant role in elucidating the corruption patterns as
these two objectives are tightly intertwined. We also present a novel
validation scheme for evaluating the performance of the corruption pattern
estimation. Our proposed method is extensively assessed in terms of both
robustness and corruption pattern estimation through a number of domains,
including computer vision and natural language processing. Code has been made
publicly available at https://github.com/jeongeun980906/Uncertainty-Aware-
Robust-Learning.

###### keywords:

Robust learning, Training With Noisy Labels, Uncertainty Estimation,
Corruption Pattern Estimation

††journal: Journal of Pattern Recognition

##  1 Introduction

In this paper, we focus on the problem of robust learning [1, 2, 3, 4, 5] with
its emphasis on elucidating the corruption patterns on the noisy training
dataset. Most existing robust learning studies [2, 5] assume that the label
corruption pattern is solely a function of class information, also known as
class-conditional noise (CCN). While this CCN assumption is simple to
formulate, it may not be useful in practice in that it is more natural to
assume for the noise pattern to be a function of input instances which is
often referred to as an instance-dependent noise (IDN) learning problem [6].

However, the original IDN learning problem is likely to be infeasible in that
it has to estimate a $C\times C$ class transition matrix per input instance.
Due to this intractability, recent work on the IDN setting focuses on a simple
binary classification problem [7, 8] or requires a small clean dataset [6]. To
mitigate this issue, we cast the IDN problem into a two-stage problem of first
partitioning the input space using uncertainty measures and then estimating
the label transition matrix per each group, which will be referred to as a
set-dependent noise (SDN) learning problem. In particular, a specific type of
predictive uncertainty, named aleatoric uncertainty, is used to partition the
input space. The clusters with high aleatoric uncertainty can be viewed as
collective outliers [9], a subset of inputs with a specific noise pattern.

We would like to stress out that robust learning and uncertainty estimation
problems are intimately related to each other as robust learning deals with
the noisy training data, which naturally give rise to predictive uncertainty.
The predictive uncertainty can be decomposed into epistemic and aleatoric
uncertainty. The former focuses on the reducible part of the uncertainty
(model uncertainty), which may come from the lack of training data. In
contrast, the latter comes from the irreducible part (data uncertainty), such
as the measurement noise. Our proposed method can estimate both types of
uncertainty in a unified framework, and it plays a significant role in
achieving robustness and estimating the SDN patterns.

To this end, we utilize a mixture-of-experts model for classification tasks
named mixture logit networks (MLN) and present an effective training method to
achieve both robustness and explainability by revealing the label corruption
process. We first present an uncertainty estimation method for the MLN that
can distinguish two different types of predictive uncertainties, epistemic
(model) uncertainty and aleatoric (data) uncertainty. Then, the estimated
uncertainty is utilized for the uncertainty-aware regularization method.
Intuitively speaking, unlike a single deterministic model (e.g., a ResNet),
using the MLN allows us to model multi-modal (and possibly noisy) target
distributions, which plays a crucial role in achieving both robustness as well
as explainability. Furthermore, we present an evaluation scheme on SDN
settings, which gives information about the collective outliers and label
noise distribution of sets.

The main contributions of this work are threefold. 1) We propose a simple yet
effective robust learning method leveraging a mixture-of-experts model on
various noise settings. 2) The proposed method can not only robustly learn
from noisy data but can also discover the set-dependent underlying noise
pattern (i.e., the noise transition matrix) as well as the two types of
predictive uncertainties (i.e., aleatoric and epistemic uncertainty) within
the dataset. 3) Finally, we present a novel evaluation scheme for validating
the set-dependent corruption pattern estimation performance.

##  2 Related Work

In the context of robust learning, the label noise patterns can be roughly
categorized into two groups, class-conditional noise (CCN) and instance-
dependent noise (IDN) settings, based on which information the label
corruptions are made. Note that the IDN setting is much more practical as it
is more natural to assume that the label noise pattern is a function of
inputs. Furthermore, the IDN setting can inherently incorporate the CCN
setting. Our proposed method can cope with both CCN and IDN settings. While
most of the robust learning literature focuses on simply estimating the clean
target distribution, a number of works have been recently made on achieving
both robustness and the ability to estimate the label noise patterns.

Co-Teaching [2] utilizes two separate networks (a teacher network and a
student network) by teaching the student network using the teacher network.
Co-Teaching+ [4] extends Co-Teaching by further leveraging the disagreement
strategy. JoCoR [5] is based on Co-Teaching+, which uses a joint loss function
of minimizing cross-entropy while maximizing the agreement between two
networks to achieve better robustness. Small loss tricks of providing small
risk for unseen clean data have been used in [10, 11, 12].

On the other hand, several works aim to achieve robustness by explicitly
estimating the label transition matrix (noise patterns). F-correction [3]
estimates the noise transition matrix and applies it to loss function
correction. Dual-T [13] incorporates a matrix factorization method to avoid
directly estimating a noisy class posterior without any anchor points (i.e.,
clean data). Total variation regularization [14] effectively regularizes the
predicted probability to be more distinguishable by restricting the total
variance distance resulting in better estimation of the noise transition
matrix.

The IDN setting is more practical than the CCN setting in that it is more
natural to assume that the annotator gets confused by ambiguous instances
leading to mislabeling. Only recently, a few papers have incorporated
confidence estimation and noise transition matrix prediction to handle
confusing instances. However, most of the work only uses the confidence
estimation to robustly learn clean target distribution in instance-dependent
noise settings. Cheng et al. [15] present confidence regularization to prevent
overfitting in multi-class classification problems. [15] is further extended
in [16] by designing a sample sieve method to get clean instances from the
noisy dataset, using confidence regularized cross-entropy loss. The confidence
regularized method gives information about the corruption of each instance but
does not provide information about label noise patterns. Another approach is
to estimate the noise transition matrix instance-wise to correct the loss
function. Part-dependent noise (i.e., PDN) was introduced by Xia et al. [17]
which approximates the transition matrix by the combination of transition
matrices for each instance. Yang et al.[18] first collect predicted clean set
to learn the noise transition matrix and then train a classifier with a
corrected loss function based on the estimated noise transition matrix.

Perhaps, the most similar setting to ours is [6] which introduced a
confidence-scored instance dependent noise setting; a label noise is given
based on prior information about confidence score by annotators. It uses both
confident estimation and noise transition matrix estimation for robust
learning. Then, the model utilizes the corrected loss function using the
confidence-based noise transition matrix. However, the proposed set-dependent
noise setting differs from the confidence-scored instance-dependent noise
setting in that the annotators can divide the set by the ambiguity measure of
each instance, making it more intuitive and straightforward. Furthermore, the
proposed method directly estimates both corruption information and the noise
distribution without the necessity of training multiple networks. The
categorization of robust learning papers is shown in Table 1

| Class-Conditional Noise (CCN) | Instance-Dependent Noise (IDN)
---|---|---
Robsut Learning | [2, 4, 5, 10, 11, 12, 1, 19] | [6, 15]
\+ Noise Transition Matrix Prediction | [20, 13] |  [17, 21], Ours

Table 1:  Categorization of robust learning papers

The mixture-of-experts models have been widely used in robust learning [22,
23, 1]. SsSMM [22] incorporates a student-teacher model similar to MentorNet
[2], but employs a finite mixture models for student networks, updating via an
EM algorithm [24] in semi-supervised manner. For robust learning for language
domains, Irie et al. [23] proposed a recurrent adaptive mixture model to
represent diverse outputs. ChoiceNet [1] utilizes a mixture density network to
model the correlated outputs where the correlation between the target and
noisy distributions are estimated in an end-to-end manner. Our proposed method
is also based on a mixture of the experts model; however, a novel uncertainty-
aware regularization method is presented.

##  3 Problem Formulation

###  3.1 Training Data Generation Process

In this paper, we focus on the classification task of finding a mapping from
an instance $\mathbf{x}$ (e.g., an RGB image) to an output $\mathbf{y}$ (e.g.,
an one-hot vector) where the input $\mathbf{x}$ and the output $\mathbf{y}$
are sampled from the input distribution $p(\mathbf{x})$ and the clean target
distribution $p(\mathbf{y}|\mathbf{x})$. We assume that some noise patterns
can be induced to both input and output where we denote $\tilde{\mathbf{x}}$
and $\tilde{\mathbf{y}}$ as the corrupted input and output, respectively. The
input noise pattern, $\tilde{\mathbf{x}}\sim
p(\tilde{\mathbf{x}},\mathbf{x})$,can be adding more blur to the instance so
that the resulting image is obscured or applying artificial manipulation to
the image (e.g., CutMix [25]).

Roughly, the output corruption process can be divided into twofold: the class
conditioned noise (CCN) and the instance dependent noise (IDN) settings. For
the CCN setting, it is assumed that the training label information is
corrupted via a single label transition matrix $T\in\mathbb{R}^{C\times C}$
where $C$ is the number of classes and
$[T]_{ij}=p(\mathbf{y}_{j}|\mathbf{y}_{i})$ is the probability of a label $i$
being shifted to a label $j$. For example, we can simply select a certain
portion of the training data and shuffle the labels uniformly randomly or
shift the labels by assigning label $1$ to $2$, label $2$ to $3$, and so
forth. In the robust learning literature, the formal and the latter are often
referred to as symmetric and asymmetric noise patterns, respectively. On the
other hand, the ICN assumes that the noise pattern is a function of an
instance (i.e., $T(\mathbf{x})\in\mathbb{R}^{C\times C}$). However, as a
single instance can only have a single target label, it would be unrealistic
to have the whole label transition matrix $T$ per instance.

Throughout this paper, a set-dependent noise (SDN) setting is utilized where
we assume that the training dataset is partitioned into subsets where each
subset contains its own label corruption matrices. This assumption is rather
more practical in that it is more natural to assume that the annotators will
be more likely to make mistakes on a specific subset consisting of hard
instances. We would like to stress that our proposed method can estimate the
label noise patterns in both CCN and SDN settings without the necessity of
additional clean data.

###  3.2 Robust Learning and Corruption Pattern Estimation

The main objectives of the proposed method are twofold: the first is to
robustly learn the underlying clean target signal out of noisy training data,
and the other is to gain the explainability of the prediction via estimating
the label corruption information as well as the predictive uncertainty.
Specifically, we disentangle the total uncertainty into aleatoric and
epistemic uncertainty similar to Kendall et al. [26] and will be explained in
the next section. Aleatoric uncertainty corresponds to the irreducible part of
the uncertainty, which is inherent in the data generation process (e.g.,
measurement noise). On the other hand, epistemic uncertainty captures the
model uncertainty, which may reduce as we have more training data.

With respect to the label corruption information, we estimate the set-
dependent noise (SDN) pattern of the training dataset without the necessity of
a clean validation dataset. Note that the SDN inherently handles the CCN as it
can simply condition the whole data. Specifically, we estimate the label
transition matrix conditioned on the subset of training of test data where the
corruption rates and the noise patterns, symmetric or asymmetric, can be
estimated from the transition matrix.

##  4 Proposed Method

Figure 1: General process of the proposed method

We present a robust learning method via a mixture-of-experts model for a
classification task named mixture logit networks (MLN) and a noise pattern
estimation method utilizing the outputs of the MLN. To fully utilize the
multiple mixtures, we further propose an uncertainty-aware regularization
method. We empirically show that this regularization method plays an
influential role in achieving both robustness and explainability. The
intuition behind leveraging the mixture model is that, when given corrupted
training data, the noise pattern will give rise to the discrepancy of the
prediction outputs, where a single deterministic model (e.g., a ResNet) often
fails to correctly capture the clean target signal. However, as a mixture
model, when adequately trained, can better capture the inconsistent output
patterns (including both clean and noisy distributions), it not only can
robustly learn the underlying target distribution but also can model the noise
patterns injected in the data generating process. The overall process of the
proposed method is illustrated in Figure 1.

The MLN architecture is illustrated in Figure 2. Suppose that the number of
mixtures is $K$, then the MLN outputs consist of mixture weights
$\\{\pi_{k}\\}_{k=1}^{K}$, logits $\\{\boldsymbol{\mu}_{k}\\}_{k=1}^{K}$ where
$\boldsymbol{\mu}_{k}\in\mathbb{R}^{C}$ and $C$ is the number of classes, and
Mixture standard deviations (Mixture STD) $\\{\sigma_{k}\\}_{k=1}^{K}$. Note
that only the uppermost layer is modified. Hence the total number of
parameters does not change significantly.

Figure 2: The proposed mixutre of logit network architecture

###  4.1 Uncertainty Estimation using the MLN

We first present ways to estimate two types of uncertainties with the MLN:
epistemic (model) uncertainty and aleatoric (data) uncertainty. We denote
$\sigma_{e}$ as epistemic uncertainty and $\sigma_{a}$ as aleatoric
uncertainty. Fist, epistemic uncertainty is computed as follows.

$\sigma_{e}^{2}=\sum_{j=1}^{K}\left(\sum_{c=1}^{C}\pi_{j}\left\|\mu^{(c)}_{j}(\mathbf{x})-\sum_{k=1}^{K}\pi_{k}\mu^{(c)}_{k}(\mathbf{x})\right\|^{2}\right)$
(1)

where $\mu^{c}_{k}$ is logit of label $c$ in $k$th mixture.

On the other hand, epistemic uncertainty ($\sigma_{e}$) indicates how much the
model is uncertain about its prediction. (1) corresponds to the weighted
average variance of each mixture’s predicted logits, which can be seen as
disagreements between $\\{\boldsymbol{\mu}_{k}\\}_{k=1}^{K}$.On the other
hand, aleatoric uncertainty ($\sigma_{a}$) is computed as follows.

$\sigma_{a}^{2}=\sum_{k=1}^{K}\pi_{k}\sigma_{k}(\mathbf{x})$ (2)

Aleatoric uncertainty captures noise inherent in observation, how much the
model is uncertain about its data. (2) indicates the weighted average of each
mixture’s predicted STD of the given input. Mixture STD
$\\{\sigma_{k}\\}_{k=1}^{K}$ denotes predicted noise by mixtures, also can be
used as attenuation factor for loss function, similar to Kendal et al. [26].

###  4.2 Mixture of the Attenuated Losses

We present a Mixture of the Attenuated Cross-Entropy (MACE) loss for
effectively training the MLN. We denote the target as $y_{i}$, which can
either be clean or noisy (i.e., $\tilde{y_{i}}$) depending on the dataset. The
proposed loss function consists of cross-entropy loss divided with the
standard deviation of each mixture (i.e., loss attenuation) and then weighted
summation of the attenuated loss for each mixture. Mixture standard deviation
$\\{\sigma_{k}\\}_{k=1}^{K}$ corresponds to the expected measurement noise of
each instance.

The MACE loss function is defined as follows:

$\mathcal{L}_{MACE}=\frac{1}{N}\sum_{i=1}^{N}\sum_{k=1}^{K}\pi_{k}(\mathbf{x}_{i})\frac{l({\boldsymbol{\mu}_{k}(\mathbf{x}_{i})},y_{i})}{\sigma_{k}(\mathbf{x}_{i})}$
(3)

where $l(\boldsymbol{\mu}_{k}(\mathbf{x}_{i}),y_{i})$ is the cross entropy
loss.

If an input is ambiguous or corrupted, it becomes more likely to make a false
prediction. Then $\sigma_{k}$ will increase to reduce the overall loss
function of the prediction. As a result, this attenuated factor prevents the
overfitting of the model to the corrupted dataset, making the proposed model
more robust.

###  4.3 Uncertainty-aware Regularization Method

Figure 3: Effect of the Regularizer On half-moon dataset

We observe that there could exist two possible problems when training only
with the original MACE loss function: simple increment of the mixture standard
deviation will minimize the loss and the insufficient usage of mixtures. In
order to resolve these issues, we propose a novel regularization method
utilizing predictive uncertainty measures. Let us first present the proposed
loss function utilizing the uncertainty measures:

$\mathcal{L}(\mathcal{D})=\mathcal{L}_{MACE}-\lambda_{1}\sigma_{e}+\lambda_{2}\sigma_{a}.$
(4)

The first problem is that $\\{\sigma_{k}\\}_{k=1}^{K}$ in (3) will simply grow
to minimize the total loss. To prevent this, we need the regularization term,
$\\{\sigma_{k}\\}_{k=1}^{K}$. This is inspired by Kendall et al. [26], where
the loss function is based on the Gaussian likelihood and is the sum of
attenuated regression loss and regularization of $\\{\sigma_{k}\\}_{k=1}^{K}$.
We present a parameter $\lambda_{2}$ as a weight.

Next, it is known that the mixture-of-experts model is prone to use only one
or few components, incapable of capturing the various data distributions.
Moreover, we observe that different mixtures are easily agreeable with each
other, which is inappropriate to represent a multi-modal distribution. Due to
this property, the whole model often fails to learn both clean and corrupted
data distribution, leaving the model vulnerable to noisy data. To this end, we
regularize epistemic uncertainty to be large, which naturally incentives to
utilize more mixtures where $\lambda_{1}$ is the weight parameter.

We illustrate effect of this regularizer in Figure 3. The synthetic dataset
consists of two-dimensional inputs (i.e., $x_{0}$ and $x_{1}$) for a binary
classification problem where we assign label $0$ to instances on the upper
moon and label $1$, otherwise. We corrupt the label information by flipping
the labels at the rate of 30%, and train the MLN with three mixtures. In the
first column, we illustrate the clean half-moon dataset, noisy half-moon
dataset, and decision boundary trained by the clean half-moon dataset. The
second and third columns present the effect of the regularizer. The first row
presents the estimated flipping rate, computed from (14), which will be
discussed in the later section. The result indicates that the proposed
regularizer helps the better prediction of the noise distribution. The second
row of Figure 3 shows that the output of each mixture disagrees with each
other, making the better representation. Furthermore, the third row
illustrates that the proposed regularizer smooths the decision boundary in the
presence of outliers.

###  4.4 Corruption Pattern Estimation

We further use the output of the MLN to gain information about the noise
corruption pattern. This can be done by estimating the noise transition
matrix. The noise transition matrix $T_{ij}(\mathbf{x})$ [20, 13] indicates
the probabilities of the clean labels flipping to noisy labels. The notation
represents the probability that the instance $\mathbf{x}$ with the clean label
$y=i$ will have a noisy label $\tilde{y}=j$. Formally, the noise transition
matrix is defined as follows.

$T_{ij}(\mathbf{x})=P(\tilde{y}=j|y=i,\mathbf{x})$ (5)

Capturing multi-modality is one of the strengths that the mixture model
possesses. This property leads the MLN to model a multi-modal distribution on
a noisy instance, representing both the clean and noisy label distribution. As
the label corruption patterns can be regarded as a multi-modal distribution,
we introduce an auxiliary random variable $z$ to estimate the noise transition
matrix.

$T_{ij}(\mathcal{D}_{l};z)=\frac{1}{|\mathcal{D}_{l}(i)|}\sum_{\mathbf{x}\in\mathcal{D}_{l}(i)}P(\tilde{y}=j|\mathbf{x})$
(6)

here, $\mathcal{D}_{l}$ denotes the set indexed $l$ and
$\mathcal{D}_{l}(i)=\\{\mathbf{x}|y=i,(\mathbf{x},y)\in\mathcal{D}_{l}\\}$.
For SDN setting set index can be 0 (clean) or 1 (ambiguous) and for CCN set
index will be 0 (total).

$T_{ij}(\mathcal{D}_{l};z)=\frac{1}{|\mathcal{D}_{l}(i)|}\sum_{\mathbf{x}\in\mathcal{D}_{l}(i)}\sum_{k=1}^{K}{P(z=k)P(\tilde{y}=j|z=k,\mathbf{x}})$
(7)

where K is total number of mixtures, z is latent variable, and $P(z=k)$ is
weight of component distribution denoted as $\pi_{k}$ above.

Starting from (7), we denote the $\widehat{P}(\mathbf{y}|\mathbf{x})$ as the
the soft-max output vector approximating $P(\mathbf{y}|\mathbf{x})$ by
parametrized model and estimator for the noise transition matrix as
$\widehat{T}$. In addition, since we cannot observe clean $y$, we assume $y$
as Bayes optimal labels, the class labels that maximize the clean class
posteriors
$\hat{f}(\mathbf{x}):=\underset{y}{\text{argmax}}\hat{P}(y|\mathbf{x})$.
Furthermore, we define a subset $\widehat{\mathcal{D}}_{l}(i)$ with the Bayes
optimal label index $i$ in the set $\widehat{\mathcal{D}}_{l}$, as we do not
have any prior knowledge about the clean $y$.

$\widehat{\mathcal{D}}_{l}(i)=\\{\mathbf{x}|\hat{f}(\mathbf{x})=i,\mathbf{x}\in\mathcal{D}_{l}\\}$
(8)

Then, the (7) can be modified as follows.

$\widehat{T}_{ij}(\mathcal{D}_{l};z)=\frac{1}{|\widehat{\mathcal{D}}_{l}(i)|}\sum_{\mathbf{x}\in\widehat{\mathcal{D}}_{l}(i)}\sum_{k=1}^{K}{\widehat{P}(z=k)\widehat{P}(\tilde{y}=j|z=k,\mathbf{x})}$
(9)

The Lemma 4.1 shows (9) is a valid transition matrix, since the row-wise sum
of predicted transition matrix becomes one.

######  Lemma 4.1.

The row wise sum of the proposed transition matrix estimation in (9) becomes
one.

$\sum_{j=1}^{C}{\widehat{T}_{ij}(D_{l};z)}=1$ (10)

###### Proof.

The row wise sum of the proposed estimated noise transition matrix becomes as
follows.

$\sum_{j=1}^{C}\widehat{T}_{ij}(\mathcal{D}_{l};z)=\sum_{j=1}^{C}\frac{1}{|\widehat{\mathcal{D}}_{l}(i)|}\sum_{\mathbf{x}\in\widehat{\mathcal{D}}_{l}(i)}\sum_{k=1}^{K}{\widehat{P}(z=k)\widehat{P}(\tilde{y}=j|z=k,\mathbf{x})}$
(11)

Since $j$ is not dependent to $|\widehat{\mathcal{D}}_{l}(i)|$ and
$\widehat{P}(z=k)$, the equation can be rewritten as follows.

$\sum_{j=1}^{C}\widehat{T}_{ij}(\mathcal{D}_{l};z)=\frac{1}{|\widehat{\mathcal{D}}_{l}(i)|}\sum_{\mathbf{x}\in\widehat{\mathcal{D}}_{l}(i)}\sum_{k=1}^{K}{\widehat{P}(z=k)\sum_{j=1}^{C}\widehat{P}(\tilde{y}=j|z=k,\mathbf{x})}$
(12)

By the definition of the categorial distribution and mixture weight,
$\sum_{j=1}^{C}\widehat{P}(\tilde{y}=j|z=k,\mathbf{x})=1$ and
$\sum_{k=1}^{K}\widehat{P}(z=k)=1$.

$\sum_{j=1}^{C}\widehat{T}_{ij}(\mathcal{D}_{l};z)=\frac{1}{|\widehat{\mathcal{D}}_{l}(i)|}\sum_{\mathbf{x}\in\widehat{\mathcal{D}}_{l}(i)}1$
(13)

As we define the set $\widehat{\mathcal{D}}_{l}(i)$ as (8), the proposed noise
transition matrix holds $\sum_{j=1}^{C}{\widehat{T}_{ij}(D_{l};z)}=1$. ∎

As the confidence of the softmax-output decreases on the noisy dataset, the
predicted transition matrix often suffers from being too smooth. To better
estimate the noise transition matrix, inspired by Liang et al. [27], we apply
temperature scaling for the softmax activation. We set the temperature to
zero, which makes the softmax function an indicator function, driving to the
predicted noise transition matrix inherent to the confidence score.
$\hat{P}_{\text{scaled}}$ is defined as follows.

$\hat{P}_{\text{scaled}}(\tilde{y}=j|z=k,\mathbf{x})=\mathbb{I}_{j}(\underset{c}{\text{argmax}}\hat{P}(\tilde{y}=c|z=k,\mathbf{x}))$
(14)

The scaled transition matrix is defined by replacing $\hat{P}$ to
$\hat{P}_{\text{scaled}}$ in (9).

##  5 Experiments

In this section, we present experimental results of validating the robustness
of the proposed method. We first describe the implementation details for
experiment settings, including datasets, corruption patterns, and
hyperparameters. Next, we present the results in the CCN setting and compare
them with benchmarks. Furthermore, we utilize the estimated uncertainty
measures to distinguish the collective outliers in the SDN settings where the
noise transition matrices of each partition are estimated and compared with
the ground truth.

###  5.1 Implementation Details

#### Class conditioned noise setting

We first construct a class-conditional noise dataset with clean instances and
noisy labels whose corruption rate is solely a function of class information.
We evaluate the proposed method on four different datasets, MNIST, CIFAR10,
CIFAR100, and TREC. The former three are popularly used for evaluating the
robustness of the image classification algorithms. We further validate our
method on sentence classification problems, TREC dataset. Following JoCoR [5],
we conduct experiments on four different label corruption patterns:
Symmetry-20%, Symmetry-50%, Symmetry-80%, and Asymmetry-40%. We set
Symmetry-70% instead of Symmetry-80% on the TREC dataset since it contains
imbalanced labels; too severe label noise makes the number of clean instances
not enough for learning. Furthermore, to show the generalization of noise
transition matrix estimation, we test on two more noise patterns, Dual-40% and
Tridigagonal-60%. These noise patterns are similar to asymmetric noise, but
they shift the label randomly to assigned two or three labels. For example,
Dual-40% patterns denote the label can be shifted to two labels with each
probability of 20%.

#### Set-dependent noise setting

A set-dependent noise setting is utilized where the dataset is partitioned
into two subsets: clean set and ambiguous set. We define an ambiguous set as a
set containing the pair of corrupted instances and noisy labels. We conduct an
experiment on two different datasets: Dirty-MNIST and Dirty-CIFAR10. Dirty-
MNIST, proposed by Mukhoti et.al. [28] is formed as the union of MNIST set and
Ambiguous-MNIST set. Ambiguous MNIST contains corrupted instances where it has
multiple plausible labels but containing only one GT label. To conduct set-
dependent noise on the Dirty-MNIST dataset, we additionally added label noise
on the Ambiguous-MNIST set. We validate on four different label noise
patterns: Symmetry-20%, Symmetry-50%, Symmetry-80%, and Asymmetry-40%. We
define the Dirty-CIFAR10 dataset, which contains half of the original CIFAR10
dataset and the other half ambiguated with the CutMix [25] method. We choose
the CutMix method to maintain the scheme that samples on ambiguous sets should
have multiple possible labels but has one GT label. Again, to form the SDN
setting, we added label noise on the ambiguous set.

#### Hyperparameters

We use a three-layer CNN for MNIST and seven-layer CNN for both CIFAR10 and
CIFAR100. For the TREC dataset, we use the CNN-multichannel model, where
utilizes both static and non-static word2vec initialized CNN channels, as
implemented in Kim et al. [29]. We set the batch size as 128 and use an Adam
optimizer with the learning rate $10^{-3}$ for the vision task and train the
model with 200 epochs for CIFAR10, CIFAR100, and 20 epochs for MNIST. For the
NLP task, we utilize an Adadelta optimizer with a learning rate of $0.1$ for
100 epochs following [29]. The learning rate is decayed 0.2 times for every
ten epochs for CIFAR10, CIFAR100, and TREC, and 0.2 decay rate for every five
epochs on the MNIST dataset. We set the minimum of
$\\{\sigma_{k}\\}_{k=1}^{K}$ as one and maximum as ten by using a sigmoid
function. In addition, we set the number of the mixtures to be $20$ for all
experiments, which should be large enough to cover all the noise distribution.
Furthermore, we set regularizer hyperparameters as $\lambda_{1}=1$,
$\lambda_{2}=1$ except for CIFAR100, where we scale the parameter to
$\lambda_{1}=0.1$, $\lambda_{2}=1$. Regularizer parameters are selected using
cross-validation results. On cross-validation, we assume there exists a small
clean set and use 10% of the clean test set as the validation set.

###  5.2 Robust Learning Performance

We conduct robust learning experiments with the class-conditioned noise (CCN)
setting to investigate the performance of the MLN. We evaluate the test
accuracy on four datasets with four different noise patterns and compare with
F-correction [30], Co-teaching [2], Co-teaching+ [4] and JoCoR [5]. The test
accuracy on MNIST is shown in Table 2. The proposed method outperforms on the
Symmetry-80% setting, and on other noise settings, it is compatible with the
compared methods. However, the results on CIFAR10 in Table 3 show that our
method outperforms the compared methods on Symmetry-80% and Asymmetry-40%,
with the second-best performance on other noise patterns. Furthermore, Table 4
presents the test accuracy on CIFAR100 dataset. The MLN outperforms on
Symmetry-80% and Asymmetry-40% noise patterns and performs second-best on the
rest of the noise patterns. We would like to note that the proposed method
shows its strengths in heavy corruptions, such as Symmetry-80% and
Asymmetry-40%. We also evaluate the MLN on the NLP domain using the TREC
dataset and compare it with the baseline method proposed in [29]. The result
is shown in Table 5 where the proposed method outperforms the baseline in all
cases. The proposed method shows a significant performance margin on large
corruption rates, such as Symmetry-80% and Asymmetry-40%.

Table 2: MNIST Test Accuracy

Noise Rate | F-correction[3] | Co-teaching[2] | Co-teaching+[4] | JoCoR[5] | MLN (ours)
---|---|---|---|---|---
Symmetry-20% | 98.37$\pm$0.28 | 99.08$\pm$0.04 | 99.00$\pm$0.10 |  99.20$\pm$0.07 | 98.97$\pm$0.01
Symmetry-50% | 95.70$\pm$0.6 | 98.19$\pm$0.09 |  98.83$\pm$0.11 | 98.76$\pm$0.06 | 98.32$\pm$0.01
Symmetry-80% | 85.33$\pm$2.28 | 85.26$\pm$0.11 | 86.24$\pm$0.16 | 86.00$\pm$0.15 |  87.47$\pm$0.11
Asymmetry-40% | 95.19$\pm$1.29 | 96.69$\pm$0.70 |  98.65$\pm$0.18 | 98.46$\pm$0.21 | 97.13$\pm$0.10

Table 3: CIFAR10 Test Accuracy

Noise Rate | F-correction[3] | Co-teaching[2] | Co-teaching+[4] | JoCoR[5] | MLN (ours)
---|---|---|---|---|---
Symmetry-20% | 68.74$\pm$0.20 | 78.23$\pm$0.27 | 78.71$\pm$0.34 |  85.73$\pm$0.19 | 84.20$\pm$0.05
Symmetry-50% | 42.71$\pm$0.42 | 71.30$\pm$0.13 | 57.05$\pm$0.54 |  79.41$\pm$0.25 | 77.88$\pm$0.07
Symmetry-80% | 15.88$\pm$0.42 | 26.58$\pm$2.22 | 24.19$\pm$2.74 | 27.78$\pm$3.06 |  41.83$\pm$0.10
Asymmetry-40% | 70.60$\pm$0.40 | 73.78$\pm$0.22 | 68.84$\pm$0.20 | 76.36$\pm$0.49 |  76.62$\pm$0.07

Table 4: CIFAR100 Test Accuracy

Noise Rate | F-correction[3] | Co-teaching[2] | Co-teaching+[4] | JoCoR[5] | MLN (ours)
---|---|---|---|---|---
Symmetry-20% | 37.95$\pm$0.10 | 43.73$\pm$0.16 | 49.27$\pm$0.03 |  53.01$\pm$0.04 | 51.60$\pm$0.08
Symmetry-50% | 24.98$\pm$1.82 | 34.96$\pm$0.50 | 40.04$\pm$0.70 |  43.49$\pm$0.46 | 42.22$\pm$0.07
Symmetry-80% | 2.10$\pm$2.23 | 15.15$\pm$0.46 | 13.44$\pm$0.37 | 15.49$\pm$0.98 |  19.88$\pm$0.14
Asymmetry-40% | 25.94$\pm$0.44 | 28.35$\pm$0.25 | 33.62$\pm$0.39 | 32.79$\pm$0.35 |  36.68$\pm$0.10

Table 5: TREC Test Accuracy

Flipping Rate | Baseline[29] | MLN (ours)
---|---|---
Symmetry-20% | 84.88$\pm$0.14 |  87.38$\pm$0.12
Symmetry-50% | 57.28$\pm$0.20 |  74.79$\pm$0.90
Symmetry-70% | 29.14$\pm$0.61 |  55.78$\pm$1.39
Asymmetry-40% | 72.63$\pm$0.96 |  74.17$\pm$0.28

###  5.3 Class-conditional Noise Transition Matrix Estimation

In this section, we evaluate the noise transition matrix estimation on the
class-conditional noise setting. We estimate the noise transition matrix using
(14), which is an anchor-free method that does not require a clean validation
set. We evaluate the noise transition matrix on MNIST and CIFAR10 datasets
with Symmetry-20%, Symmetry-50%, Symmetry-80%, Asymmetry-40%, Dual-40%, and
Tridiagonal-60% noise patterns 111We call Dual for two mislabeled classes and
Tridiagonal for three mislabeled classes.

Table 6: Evaluation of Noise Transition Matrix ATV (average total
variation)(x100), KTD(Kendall-Tau distance), Test Accuracy

|  | TVD[14] | MLN(Ours)
---|---|---|---
MNIST | Noise Rate | ATV | KDT | Accuracy | ATV | KDT | Accuracy
Symmetry-20% | 2.666 | 0.4472 | 98.65 | 14.46 | 0.4472 | 98.97
Symmetry-50% | 3.337 | 0.4472 | 97.80 | 7.822 | 0.4472 | 98.32
Symmetry-80% | 5.966 | 0.4472 | 86.37 | 2.842 | 0.4472 | 87.47
Asymmetry-40% | 2.002 | 0.5309 | 98.49 | 15.27 | 0.5087 | 97.13
Dual-40% | 2.257 | 0.5810 | 98.77 | 13.64 | 0.4940 | 98.97
| Tridiagonal-60% | 3.791 | 0.6109 | 98.74 | 10.96 | 0.5937 | 98.06
CIFAR10 | Symmetry-20% | 9.134 | 0.4472 | 81.47 | 20.71 | 0.4472 | 84.20
Symmetry-50% | 8.048 | 0.4472 | 73.94 | 9.69 | 0.4472 | 77.88
Symmetry-80% | 6.117 | 0.4772 | 43.50 | 5.85 | 0.4373 | 41.83
Asymmetry-40% | 2.002 | 0.5309 | 75.95 | 16.81 | 0.4948 | 76.63
Dual-40% | 9.320 | 0.5811 | 79.71 | 15.97 | 0.5811 | 81.31
Tridiagonal-60% | 12.46 | 0.6109 | 74.38 | 20.74 | 0.5822 | 71.29

Figure 4: Noise Transition Matrix on MNIST Figure 5: Noise Transition Matrix
on CIFAR10

We report the average total variation (ATV) and Kendall Tau rank distance
(KTD) [31] to evaluate the transition matrix estimation, which are defined as
follows:

$\text{Average total
variance}=\frac{1}{C}\sum_{i=1}^{C}\frac{1}{2}\sum_{j=1}^{C}{|T_{ij}-\hat{T}_{ij}|}$
(15) $\text{Average Kendall Tau
Distance}=\frac{1}{C}\sum_{i=1}^{C}{\sum_{j,k}\bar{K}_{j,k}(\mathbf{t}_{i},\hat{\mathbf{t}}_{i}})$
(16)

where $\mathbf{t}_{i}$ denotes for $i$-th row vector of transition matrix $T$
and $\hat{\mathbf{t}}_{i}$ for $i$-th row vector of $\widehat{T}$.

The function $\bar{K}$ for two arbitrary vectors $\mathbf{t}_{1}$ and
$\mathbf{t}_{2}$ is defined as follows:

$\bar{K}_{j,k}(\mathbf{t}_{1},\mathbf{t}_{2})=\begin{cases}0&\text{if j and k
are in same order of ranking}\\\ 1&\text{if not}\end{cases}$ (17)

The total variance is an average l1 norm between two matrices, which denotes
the absolute difference between the estimated and ground-truth matrices.
Kendall-tau rank distance is defined as a metric that counts the number of
pairwise disagreements between two ranking lists, which means comparing the
ranking of two matrices.

The Table 6 shows the ATV and KTD measure with comparison to TVD [14]. TVD
uses total variation regularization to estimate the noise transition matrix
with the prior confusion matrix and Dirichlet posterior update. Although the
proposed method underperforms for every noise rate except for Symmetry-80% on
ATV, the KDT measure outperforms TVD, especially in asymmetric noise patterns,
i.e., Asymmetry-80%, Dual-40%, Tridiagonal-60%. The results illustrate that
the MLN shows a better estimation performance compared to TVD in terms of the
row-wise ranking of the noise transition matrix. However, it shows slightly
inferior performances to TVD in terms of average total variances. We further
discuss this in Section 5.6. The transition matrix estimated by the MLN is
shown in Figure 4 for MNIST dataset and Figure 5 for CIFAR10 dataset.

###  5.4 Partitioning Sets on the SDN Setting

Figure 6: Example of Set-Dependent Noise Dataset. The arrows denote the label
corruption. Table 7: Measured AUROC over different uncertainty measures

| Noise Rate | Aleatoric | Epistemic |  $\pi$ Entropy | Max Softmax | Softmax Entropy
---|---|---|---|---|---|---
Dirty MNIST | Symmetry-20% | 0.9998 | 0.9991 | 0.9907 | 0.9978 | 0.9987
Symmetry-50% | 1.0000 | 0.9993 | 0.9999 | 0.9992 | 0.9997
Symmetry-80% | 0.9999 | 0.9992 | 0.9996 | 0.9997 | 0.9997
Asymmetry-40% | 0.9811 | 0.9293 | 0.9846 | 0.9927 | 0.9933
Dirty CIFAR10 | Symmetry-20% | 0.9181 | 0.7300 | 0.7952 | 0.8739 | 0.8971
Symmetry-50% | 0.9874 | 0.8909 | 0.9604 | 0.9610 | 0.9756
Symmetry-80% | 0.9976 | 0.9848 | 0.9886 | 0.9165 | 0.9559
Asymmetry-40% | 0.6523 | 0.5616 | 0.6563 | 0.7234 | 0.7060

In this section, we show the ability of the proposed method to partition the
dataset leveraging aleatoric uncertainty estimated from the MLN. In
particular, the collective outliers, partitions of training data with
corrupted labels, are well captured via aleatoric uncertainty. We use the
Dirty-MNIST dataset and Dirty-CIFAR10 dataset explained in Figure 6. The
datasets contain a clean set and an ambiguous set, where an ambiguous set is
composed of ambiguous instances with corrupted labels. There exist different
uncertainty measures than the ones explained in (1) and (2), for example, it
can be measured by max-softmax [32] or softmax entropy computed from the
entropy of mixture weights. The Pi-entropy denotes the entropy of the mixture
weights. As bigger entropy can be interpreted as a lower weight of target
distribution, this can be seen as the uncertainty of the inputs. In this
experiment, we assume the size of two sets is the same and set threshold as
the median of the uncertainty measures to partition these sets. The
uncertainty measures are as follows.

$\text{max-softmax}=1-\underset{c}{\text{max}}\
\mu_{k}^{(c)}(x)\quad\text{where}\quad k=\underset{i}{\text{argmax}}\
\pi_{i}(x)$ (18) $\text{softmax-
entropy}=-\sum_{c}^{C}\mu_{k}^{(c)}(x)\log\left(\mu_{k}^{(c)}(x)\right)\quad\text{where}\
k=\underset{i}{\text{argmax}}\ \pi_{i}(x)$ (19) $\text{pi-
entropy}=-\sum_{k}^{K}\pi_{k}(x)\log\left(\pi_{k}(x)\right)$ (20)

As the observation noise is an exemplar case of aleatoric uncertainty,
instance corruption patterns can be captured by aleatoric uncertainty. Table 7
reports the aleatoric uncertainty measure can partition clean and ambiguous
set on symmetric noise compared to other uncertainty measures except for
Asymmetry-40% noise rate. Figure 7 shows the average of aleatoric uncertainty
for both clean and ambiguous sets among each class. The result first shows
that the aleatoric uncertainty is higher in ambiguous instances. Furthermore,
shown in the symmetric noise cases, a heavy label corruption rate leads to
higher aleatoric uncertainty. Finally, Dirty-MNIST with Asymmetry-40% label
noise case shows that aleatoric uncertainty increases in the corrupted labels
compared to clean labels in ambiguous instances. This demonstrates that both
corruption in instances and corruption in the label are related to aleatoric
uncertainty.

Figure 7: Aleatoric uncertainty along each label on Dirty MNIST and Dirty
CIFAR10

###  5.5 Set-Dependent Noise Transition Matrix Estimation

Figure 8: Noise Transition Matrix on Dirty-MNIST Figure 9: Noise Transition
Matrix on Dirty-CIFAR10

In this section, we examine the ability of our method to estimate noise
transition matrices per each group where we partition the training data into
multiple sets using the predicted aleatoric uncertainty in Section 5.4. The
experimental results on the Dirty-MNIST and Dirty-CIFAR10 datasets are shown
in Figure 8 and 9, respectively. Each quarter denotes a single experiment for
each corruption pattern, with the upper and lower rows showing the predicted
noise transition matrix and the ground-truth, respectively. Here, clean labels
on the ambiguous set denote the ground-truth label of each instance after
ambiguating the instances. We can see that our proposed method is able to
correctly estimate the noise transition matrix for both clan and ambiguous
sets in terms of a row-wise ranking manner. Furthermore, Figure 9 suggests
that our proposed method can also capture the noise induced in inputs (i.e.,
CutMix [25]). In other words, the images of cats are cut-mixed with the images
with dogs (and vice versa), and these corruption patterns are well captured by
the noise transition matrix.

###  5.6 Limitations

In this section, we will discuss the limitations of the proposed method. While
the MLN correctly estimates the row-wise ranking of the noise transition
matrix, it under-performs TVD [14] in terms of the absolute error between the
ground-truth and the estimated transition matrices (average total variance).
This is mainly due to the fact that TVD uses an explicit confusion matrix
prior to update the estimated noise transition matrix with a Dirichlet
posterior. In contrast, the proposed method indirectly estimates the noise
transition matrix using the mixture weights. However, our proposed method be
used in both CCN and SDN settings, whereas TVD can only be applicable to the
CCN setting.

##  6 Conclusion

We have introduced an uncertainty-aware robust learning framework by
leveraging a mixture of the experts’ model named mixture logit networks (MLN).
The MLN can estimate two different types of uncertainty, epistemic and
aleatoric, where the predictive uncertainty is further utilized to define a
novel regularization method. We showed that the MLN can represent multi-modal
distributions, making the model not only robust to outliers but also able to
estimate noise patterns. In particular, we presented a set-dependent noise
(SDN) learning problem where multiple corruption patterns exist per each
partition and proposed a novel validation scheme for estimating the corruption
patterns. To tackle this problem, we leveraged aleatoric uncertainty to detect
the corrupted partition and estimated the SDN patterns using the multi-modal
target distribution computed from the MLN. We would like to note that
uncertainty estimation on the robust learning framework plays a significant
role in providing information about the corruption of each instance. The
current evaluation scheme for the SDN setting relies on two assumptions: the
collective outliers can be separated via the estimated aleatoric uncertainty,
and a particular label transition matrix exists per each partition. One
promising future research direction could be examining our proposed method to
real-world datasets without applying artificial noises to both inputs and
outputs. Clothing1M dataset can be used for this purpose as it is known to
contain a nontrivial amount of mislabeled data.

## References

  * [1] S. Choi, S. Hong, K. Lee, S. Lim, Task agnostic robust learning on corrupt outputs by correlation-guided mixture density networks, in: Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 3872–3881.
  * [2] B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, M. Sugiyama, Co-teaching: Robust training of deep neural networks with extremely noisy labels, in: Proc. of the Conference on Neural Information Processing Systems (NeurIPS), 2018.
  * [3] G. Patrini, A. Rozza, A. Krishna Menon, R. Nock, L. Qu, Making deep neural networks robust to label noise: A loss correction approach, in: Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 1944–1952.
  * [4] X. Yu, B. Han, J. Yao, G. Niu, I. Tsang, M. Sugiyama, How does disagreement help generalization against label corruption?, in: Proc. of the International Conference on Machine Learning (ICML), PMLR, 2019, pp. 7164–7173.
  * [5] H. Wei, L. Feng, X. Chen, B. An, Combating noisy labels by agreement: A joint training method with co-regularization, in: Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 13726–13735.
  * [6] A. Berthon, B. Han, G. Niu, T. Liu, M. Sugiyama, Confidence scores instance-dependent label-noise learning possible, in: Proc. of the International Conference on Machine Learning (ICML), PMLR, 2021, pp. 825–836.
  * [7] J. Cheng, T. Liu, K. Ramamohanarao, D. Tao, Learning with bounded instance and label-dependent label noise, in: Proc. of the International Conference on Machine Learning (ICML), PMLR, 2020, pp. 1789–1799.
  * [8] J. Bootkrajang, J. Chaijaruwanich, Towards instance-dependent label noise-tolerant classification: a probabilistic approach, Pattern Analysis and Applications 23 (1) (2020) 95–111.
  * [9] S. Karamcheti, R. Krishna, L. Fei-Fei, C. D. Manning, Mind your outliers! investigating the negative impact of outliers on active learning for visual question answering, in: Annual Meeting for the Association of Computational Linguistics (ACL-IJCNLP), 2021.
  * [10] X. Wang, Y. Hua, E. Kodirov, N. M. Robertson, Imae for noise-robust learning: Mean absolute error does not treat examples equally and gradient magnitude’s variance matters, arXiv preprint arXiv:1903.12141.
  * [11] X. Ma, H. Huang, Y. Wang, S. Romano, S. Erfani, J. Bailey, Normalized loss functions for deep learning with noisy labels, in: Proc. of the International Conference on Machine Learning (ICML), PMLR, 2020, pp. 6543–6553.
  * [12] Y. Lyu, I. W. Tsang, Curriculum loss: Robust learning and generalization against label corruption, in: Proc. of the International Conference on Learning Representations (ICLR), 2020.
  * [13] Y. Yao, T. Liu, B. Han, M. Gong, J. Deng, G. Niu, M. Sugiyama, Dual t: Reducing estimation error for transition matrix in label-noise learning, in: Proc. of the Conference on Neural Information Processing Systems (NeurIPS), 2020.
  * [14] Y. Zhang, G. Niu, M. Sugiyama, Learning noise transition matrix from only noisy labels via total variation regularization, in: Proc. of the International Conference on Machine Learning (ICML), 2021.
  * [15] H. Cheng, Z. Zhu, X. Li, Y. Gong, X. Sun, Y. Liu, Learning with instance-dependent label noise: A sample sieve approach, in: Proc. of the International Conference on Learning Representations (ICLR), 2020.
  * [16] H. Cheng, Z. Zhu, X. Li, Y. Gong, X. Sun, Y. Liu, Learning with instance-dependent label noise: A sample sieve approach, in: Proc. of the International Conference on Learning Representations (ICLR), 2021.
  * [17] X. Xia, T. Liu, B. Han, N. Wang, M. Gong, H. Liu, G. Niu, D. Tao, M. Sugiyama, Part-dependent label noise: Towards instance-dependent label noise, in: Proc. of the Conference on Neural Information Processing Systems (NeurIPS), Vol. 33, 2020.
  * [18] S. Yang, E. Yang, B. Han, Y. Liu, M. Xu, G. Niu, T. Liu, Estimating instance-dependent label-noise transition matrix using dnns, arXiv preprint arXiv:2105.13001.
  * [19] J. Li, R. Socher, S. C. Hoi, Dividemix: Learning with noisy labels as semi-supervised learning, in: Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [20] X. Xia, T. Liu, N. Wang, B. Han, C. Gong, G. Niu, M. Sugiyama, Are anchor points really indispensable in label-noise learning?, in: Proc. of the Conference on Neural Information Processing Systems (NeurIPS), 2019.
  * [21] Q. Wang, B. Han, T. Liu, G. Niu, J. Yang, C. Gong, Tackling instance-dependent label noise via a universal probabilistic model, arXiv preprint arXiv:2101.05467.
  * [22] W. Shao, Z. Ge, Z. Song, J. Wang, Semisupervised robust modeling of multimode industrial processes for quality variable prediction based on student’s t mixture model, IEEE Transactions on Industrial Informatics 16 (5) (2019) 2965–2976.
  * [23] K. Irie, S. Kumar, M. Nirschl, H. Liao, Radmm: recurrent adaptive mixture model with applications to domain robust language modeling, in: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2018, pp. 6079–6083.
  * [24] G. J. McLachlan, T. Krishnan, The EM algorithm and extensions, Vol. 382, John Wiley & Sons, 2007.
  * [25] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, Y. Yoo, Cutmix: Regularization strategy to train strong classifiers with localizable features, in: Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 6023–6032.
  * [26] A. Kendall, Y. Gal, What uncertainties do we need in bayesian deep learning for computer vision?, in: Proc. of the Conference on Neural Information Processing Systems (NIPS), 2017.
  * [27] S. Liang, Y. Li, R. Srikant, Enhancing the reliability of out-of-distribution image detection in neural networks, in: Proc. of the International Conference on Learning Representations (ICLR), 2018.
  * [28] J. Mukhoti, A. Kirsch, J. van Amersfoort, P. H. Torr, Y. Gal, Deterministic neural networks with appropriate inductive biases capture epistemic and aleatoric uncertainty, arXiv preprint arXiv:2102.11582.
  * [29] Y. Kim, Convolutional neural networks for sentence classification, in: Proc. of the Empirical Methods in Natural Language Processing (EMNLP), 2014.
  * [30] G. Patrini, A. Rozza, A. Krishna Menon, R. Nock, L. Qu, Making deep neural networks robust to label noise: A loss correction approach, in: Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 1944–1952.
  * [31] M. G. Kendall, A new measure of rank correlation, Biometrika 30 (1/2) (1938) 81–93.
  * [32] D. Hendrycks, K. Gimpel, A baseline for detecting misclassified and out-of-distribution examples in neural networks, in: Proc. of the International Conference on Learning Representations (ICLR), 2017.


# Live Blog Corpus for Summarization

###### Abstract

Live blogs are an increasingly popular news format to cover breaking news and
live events in online journalism. Online news websites around the world are
using this medium to give their readers a minute by minute update on an event.
Good summaries enhance the value of the live blogs for a reader but are often
not available. In this paper, we study a way of collecting corpora for
automatic live blog summarization. In an empirical evaluation using well-known
state-of-the-art summarization systems, we show that live blogs corpus poses
new challenges in the field of summarization. We make our tools publicly
available to reconstruct the corpus to encourage the research community and
replicate our results. https://github.com/UKPLab/lrec2018-live-blog-corpus

Keywords: Live blogs, Summarization Corpus, Corpus Construction, Focused
Crawling, Online Journalism

languageresourceLanguage Resources

Live Blog Corpus for Summarization

Avinesh P.V.S., Maxime Peyrard, Christian M. Meyer
---
Research Training Group AIPHES and UKP Lab
Computer Science Department, Technische Universität Darmstadt
www.aiphes.tu-darmstadt.de, www.ukp.tu-darmstadt.de

Abstract content

##  1\. Introduction

A live blog is a dynamic news article providing a rolling textual coverage of
an ongoing event. It is a single article continuously updated by one or many
journalists with timestamped micro-updates typically displayed in
chronological order. Live blogs can contain a wide variety of media, including
text, video, audio, images, social media snippets and links. At the end of the
broadcasting, a journalist usually summarizes the main information about the
event. For more extended events, journalists may also write intermediate
summaries. Figure 1 and 2 show an example live blog provided by the _BBC_ on
“Last day of Supreme Court Brexit Case” and _The Guardian_ on “US elections
2016 campaign”. The timestamped information snippets are on the right, the
human-written bullet-point summary is at the top left.

Figure 1: BBC.com live blog on “Last day of Supreme Court Brexit Case” Figure
2: TheGuardian.com live blog on “US elections 2016 campaign”

In the last decade, live-blogging has become very popular. It is commonly used
by major news organizations, such as the _BBC_ , _The Guardian_ or _The New
York Times_. Several different kinds of events are regularly covered by live
blogs, including sport games, elections, ceremonies, protests, conflicts and
natural disasters. ?, p.1) report a journalist’s view that “live blogs have
transformed the way we think about news, our sourcing, and everything”. Thanks
to this new journalistic trend, many live blogs – and their human-written
summaries – are available online and new ones are generated every day.

In this work, we propose to leverage this data and investigate the task of
automatic live blog summarization by crawling a new dataset. Live blog
summarization has more direct applications in Journalism than the traditional
but rather artificial tasks of a single document and multi-document
summarization. Systems capable of summarizing live streams of heterogeneous
content can be directly beneficial to users and even assist journalists during
their daily work.

However, this new task also comes with new challenges. Live blogs are a list
of short snippets of heterogeneous information and they do not form one
coherent piece of text. The non-cohesive snippets make the task different from
single document summarization. Furthermore, most single documents are easily
summarized by the baseline extracting the first few sentences. Such an
approach is not effective for live blogs due to their heterogeneity and
chronological order. The snippets are typically small, focused, numerous and
rarely redundant which contrasts with the well-studied task of multi-document
summarization. The topic is continually shifting and many sub-topics may arise
and become central at some point. This even differs from the single topic
shift found in classical update summarization tasks. For example, the live
blog in Figure 1 on “Last day of Supreme Court Brexit case” consists of topic
shift across “Supreme court judgment”, “Government appeal”, “Opinions of MP’s
on Brexit” and others. Moreover, when summarizing a live blog, one has to
account for the whole past and all sub-topics previously discussed, which
differs from real-time summarization setups like TREC.

We focus on two online news websites for acquiring live blogs, the _BBC_
111http://www.bbc.com and _The Guardian_ 222https://www.theguardian.com,
because they contain a lot of easily accessible live blogs that we
automatically crawl and process.

In summary, our contributions are:

  * •

We introduce a new task: live blog summarization.

  * •

We suggest a pipeline to collect and extract live blogs with human-written
summaries from two major online newspapers and release it for the
community333https://github.com/UKPLab/lrec2018-live-blog-corpus.

  * •

We benchmark the dataset with commonly used summarization methods to stimulate
further research into this challenging task.

The rest of the paper is structured as follows: Section 2. details existing
summarization corpora and related works. Section 3. discusses our approach to
collect live blogs from _BBC_ and _The Guardian_ , followed by a discussion on
the statistics and properties of our live blog corpus in section 4.. The
performance of well-established summarization baselines on this new dataset is
discussed in section 5., followed by the conclusion and future work.

##  2\. Related Work

In this section, we describe previous works related to summarization corpora.
They were focused on single and multi-document summarization, update
summarization, and real-time summarization. We are not aware of any previous
work on live blog summarization.

#### Single and multi-document summarization.

The most widely used summarization datasets have been published in the
Document Understanding Conference444http://duc.nist.gov/ (DUC) series. In
total, there are 139 document clusters with 376 human-written reference
summaries across DUC ’01, ’02, and ’04. Although the research community has
often used these corpora, creating the manual summaries is time-consuming and
labor-intensive.

Large datasets typically exist for single document summarization tasks, for
example, the ACL Anthology Reference Corpus [Bird et al., 2008] and the
CNN/Daily Mail dataset [Hermann et al., 2015]. The latter contains large pairs
of 312k online news articles and multi-sentence summaries used for neural
summarization approaches [Nallapati et al., 2016, See et al., 2017]. However,
their dataset contains only one source document, whereas live blogs have a
larger number of information snippets, typically more than 100.

Another recent work uses social media’s reactions on Twitter to create large-
scale multi-document summaries for news [Lloret and Palomar, 2013, Cao et al.,
2016]. ?) use hashtags to cluster the documents into the same topic and use
tweets with hyperlinks to generate optimal reference summaries. Their corpus
consists of 204 document clusters with 1,114 documents and 4,658 reference
tweets. Although this approach uses social media information to create a
summarization corpus, they produce synthetic summaries, which are not written
by a human. Moreover, they only use the corpus for training supervised
learning approaches and not for evaluating summarization systems.

Other multi-document summarization datasets focus on heterogeneous sources
[Zopf et al., 2016, Benikova et al., 2016, Nakano et al., 2010], multiple
languages [Giannakopoulos et al., 2015], and reader-aware multi-document
summaries [Li et al., 2017], which jointly aggregate news documents and reader
comments.

#### Update summarization.

After the DUC series, the Text Analysis Conference555http://www.nist.gov/tac/
(TAC) series (’08, ’09) introduced the update summarization task [Dang and
Owczarzak, 2008]. In this task, two summaries are provided for two sets of
documents and the summary of the second set of documents is an update of the
first set. Although the importance of text to be included in the summary
solely depends on the novelty of the information, the task usually observes
only a single topic shift. In live blogs, however, there are multiple sub-
topics and the importance of the sub-topics changes over time.

#### Real-time summarization.

Real-time summarization began at the Text REtrieval
Conference666http://trec.nist.gov/ (TREC) 2016 and represents an amalgam of
the microblog track and the temporal summarization track [Lin et al., 2016].
In real-time summarization, the goal is to automatically monitor the stream of
documents to keep a user up to date on topics of interest and create email
digests that summarize the events of that day for their interest profile. The
drawback of this task is that they have a predefined time frame for evaluation
due to the real-time constraint, which makes the development of systems and
replicating results arduous. Note that live blog summarization is very similar
to real-time summarization, as the real-time constraint also holds true for
live blogs if the summarization system is applied to the stream of snippets.
Moreover, the Guardian live blogs do consist of updated and real-time
summaries, but this requires different real-time crawling strategies which are
out of the scope of this work.

##  3\. Corpus Construction

In this section, we describe the three steps to construct our live blogs
summarization corpus: (1) live blog crawling yielding a list of URLs, (2)
content parsing and processing, where the documents and corresponding
summaries with the metadata are extracted from the URLs and stored in a JSON
format, and (3) live blog pruning as a final step for creating a high-quality
gold standard live blog summarization corpus.

#### Live blog Crawling.

On the Guardian, a frequently updated index
webpage777http://www.theguardian.com/tone/minutebyminute references all
archived live blogs. We took a snapshot of this page that provided us with
16,246 unique live blogs.

In contrast, the BBC website has no such live blog archive. Thus, we use an
iterative approach similar to BootCaT [Baroni and Bernardini, 2004] as
described in Algorithm 1 to bootstrap a corpus utilizing a set of seed terms
extracted from ten BBC live blog links from the web. The iterative procedure
starts with a small set of seed terms ($K_{0}$) and gathers new live blog
links using automated Bing queries888https://azure.microsoft.com/en-
us/services/cognitive-services/bing-web-search-api by exploiting patterns
($P$) in live blog URLs (i.e. “site:http://www.bbc.com/news/live/[key term]”
as in line 5). We collect all the valid links returned by the Bing queries
(line 6) and look for new key terms in the recently retrieved live blogs (line
10). In our implementation, key terms are terms with high TF*IDF scores. The
new key terms are then used in the Bing queries of the subsequent iterations.
The process is repeated until no new live blogs are discovered anymore (line
7). With this process, we ran 4,000 search queries returning each around 1,000
results on average and we collected 9,931 unique URLs.

Although our method collected a majority of the live blogs in the 4,000 search
queries, a more sophisticated key terms selection could minimize the search
queries and maximize the unique URLs. Additionally, this methodology can be
applied to other news websites featuring live blogs like _The New York Times_
, _Washington Post_ or _Der Spiegel_.

An important point to note is that we find the collected BBC live blog URLs
predominantly cover more recent years. This usage could be due to the Bing
Search API preferring recent articles for the first 100 results. To collect a
broad range of news articles the queries need to be precise.

Algorithm 1 Iterative Live Blog Retrieval

1:procedure LiveBlogRetrieval()

2: input: Seed terms $K_{0}$, Live blog Pattern $P$

3: $L_{0}\leftarrow\emptyset$

4: for $t=1...T$ do

5: $Q_{t}\leftarrow\textrm{makeQueries}(K_{t-1},P)$

6: $L_{t}\leftarrow\textrm{getLinks}(Q_{t})$

7: if $\cup_{i=0}^{t-1}L_{i}=\cup_{i=0}^{t}L_{i}$ then

8: return $\cup_{i=0}^{t}L_{i}$

9: else

10: $K_{t}\leftarrow\textrm{extractKeyTerms}(L_{t})-\cup_{i=0}^{t-1}K_{i}$

11: end if

12: end for

13:end procedure

#### Content Parsing and Processing.

Once the URLs are retrieved, we fetch the HTML content, remove the boiler-
plate and store the cleaned data in a JSON file.

During this step, unreachable URLs were filtered out. We discard live blogs
for which we could not retrieve the summary or correctly parse the information
snippets. Indeed, live blogs can have changing patterns over time rendering
the automatic extraction difficult.

Parsing of BBC live blogs can be automated easily because both bullet-point
summaries and information snippets follow a consistent pattern. For the
Guardian, we identify several recurring patterns which cover most of the live
blogs. The Guardian live blogs were in use since 2001 but were in experimental
phase till 2008. Due to the lack of a specific structure or a summary during
this experimental phase, we remove 10k of the crawled live blogs. However,
after 2008, live blogs have had a prominent place in the editorial with a
consistent structure.

We parse metadata like URL, author, date, genre, summaries and documents for
each live blog using site-specific regular expressions on the HTML source
files.

After this step, 7,307 live blogs remain for BBC and 6,450 for Guardian.

#### Live blog Pruning.

To further clean the data, we decided to remove live blogs exhibiting several
topics as they can be quite noisy. For example, BBC provides some live blogs
covering all events happening in a given region within a given time frame
(e.g., _Essex: Latest updates_). We also prune live blogs about sport games
and live chats, because the summaries are based on simple templates.

We further prune live blogs based on their summaries. We first remove a
sentence of a summary if it has less than three words. Then, we discarded live
blogs whose summaries have less than three sentences. This is to ensure the
quality of the corpus, as overly short summaries would yield a different
summarization goal similar to headline generation and they are typically an
indicator for a non-standard live blog layout.

After the whole pruning step, 974 live blogs remained for BBC and 1,681 for
the Guardian.

Overall, 10% of the initial set of live blogs, both for BBC and Guardian
remained after selective pruning. This is to ensure high-quality summaries for
the live blogs. Although the pruning rejects 90% of the live blogs, the size
of the live blog corpus is 20–30 times larger than the classical corpora
released during DUC, TREC and TAC tasks.

Dataset | Crawling | Processing | Pruning
---|---|---|---
BBC | 9,931 | 7,307 | 974
Guardian | 16,246 | 6,405 | 1,681
Table 1: Number of topics for BBC and the Guardian

#### Code Repository.

To replicate our results and advance research in live blog summarization we
publish our tools for reconstructing the live blog corpus open-source under
the Apache License 2.0. The repository consists of (a) raw and processed URLs,
(b) tools for crawling live blogs, (c) tools for parsing the content of the
URLs and transforming content into JSON, and (d) code for calculating
baselines and corpus statistics.

##  4\. Corpus Statistics

We compute several statistics about the corpora and report them in Table 2.
The number of documents (or snippets) per topic is around 95 for BBC and 56
for the Guardian. In comparison, standard multi-document summarization
datasets like DUC ’04999http://duc.nist.gov/duc2004 and TAC
’08A101010https://tac.nist.gov/2008 have only 10 documents per topic.

Furthermore, we observe that snippets are quite short as there is an average
of 62 words per snippet for BBC and 108 for the Guardian. Summaries are also
shorter than summaries in standard datasets. Indeed, in DUC2004 and TAC2008A
summaries are expected to contain 100 words.

Our corpora are larger because, together, they contain 2,655 topics and
186,999 documents. With many data points, machine learning approaches become
readily applicable.

Statistic | BBC | Guardian
---|---|---
# topics | 974 | 1,681
# documents | 92,537 | 94,462
# documents / topic | 95.01 | 56.19
# words / document | 61.75 | 107.53
# words / summary | 59.48 | 42.23
Table 2: Corpus statistics for BBC and the Guardian

#### Domain Distribution.

Live blogs cover a wide range of subjects from multiple domains. In Table 3,
we report the distribution of different domains in our combined datasets (BBC
and the Guardian). While we observe that politics, business and news are the
most prominent domains, there is also a number of well-represented domains
like local and international events or culture.

Domain | # topics | proportion (%)
---|---|---
Politics | 834 | 31.41
Business | 421 | 15.86
General News | 369 | 13.90
UK local events | 368 | 13.86
International events | 337 | 12.69
Culture | 186 | 7.01
Science | 60 | 2.26
Society | 27 | 1.02
Others | 53 | 2.00
Table 3: Corpus distribution across multiple domains for BBC and the Guardian

#### Heterogeneity.

The resulting corpus is expected of exhibiting various levels of
heterogeneity. Indeed, there contain various topics with mixed writing styles
(short-to-the-point snippets vs. longer descriptive snippets). Furthermore,
live blogs are subject to topic shifts which could be observed by the change
in words used.

To measure this textual heterogeneity, we use information theoretic metrics on
word probability distributions like it was done before in analyzing the
heterogeneity of summarization corpora [Zopf et al., 2016]. Based on Jensen-
Shannon (JS) divergence, they defined a measure of textual heterogeneity $TH$
for a topic $T$ composed of documents $d_{1},\cdots,d_{n}$ as

$TH_{JS}(T)=\frac{1}{n}\sum\limits_{d_{i}\in T}JS(P_{d_{i}},P_{T\setminus
d_{i}})$ (1)

Here, $P_{d_{i}}$ is the frequency distribution of words in document $d_{i}$
and $P_{T\setminus d_{i}}$ is the frequency distribution of words in all other
documents of the topic except $d_{i}$. The final quantity $TH_{JS}$ is the
average divergence of documents with all the others and provides, therefore, a
measure of diversity among documents of a given topic.

| BBC | Guardian | DUC ’04 | TAC ’08A
---|---|---|---|---
$TH_{JS}$ | 0.5917 | 0.5689 | 0.3019 | 0.3188
Table 4: Average textual heterogeneity of our corpora compared to standard
datasets

We report the results in Table 4. To put the numbers in perspective, we also
report the textual heterogeneity of the two standard summarization datasets
DUC ’04 and TAC ’08A. These corpora were created during shared tasks and
focused on multi-document news summarization. The heterogeneity in BBC and
Guardian are similar and both much higher than DUC ’04 and TAC ’08A, meaning
that our corpora contain more lexical variation inside topics.

##  5\. Results and Analysis

Systems | BBC ($L$) | Guardian ($L$) | BBC ($2*L$) | Guardian ($2*L$)
---|---|---|---|---
R1 | R2 | SU4 | R1 | R2 | SU4 | R1 | R2 | SU4 | R1 | R2 | SU4
TF*IDF | .227 | .067 | .064 | .153 | .021 | .027 | .367 | .115 | .147 | .248 | .037 | .065
LexRank | .276 | .080 | .079 | .188 | .029 | .038 | .421 | .138 | .176 | .297 | .051 | .089
LSA | .212 | .046 | .052 | .135 | .013 | .021 | .341 | .084 | .123 | .220 | .024 | .051
KL | .267 | .086 | .080 | .178 | .026 | .035 | .397 | .132 | .165 | .272 | .041 | .076
ICSI | .302 | .104 | .091 | .210 | .046 | .046 | .461 | .176 | .201 | .322 | .071 | .101
UB-1 | .514 | .273 | .218 | .422 | .177 | .145 | .754 | .388 | .435 | .640 | .256 | .304
UB-2 | .494 | .312 | .210 | .389 | .230 | .137 | .709 | .453 | .419 | .584 | .334 | .277
Table 5: ROUGE-1 (R1), ROUGE-2 (R2), and ROUGE-SU4 (SU4) scores of multiple
systems compared to the extractive upper bounds for ROUGE-1 (UB-1) and ROUGE-2
(UB-2) extractive for summary lengths of $L$ and $2*L$ Figure 3: BBC.com live
blog on “FIFA corruption inquiry”

In this section, we describe the automatic summarization methods and the upper
bounds we compute for our live blog summarization dataset.

###  5.1. Baselines

As benchmark results, we employ methods that have been successfully used for
both single and multi-document summarization. Some variants of them have also
been applied to update summarization tasks.

TF$\star$IDF [Luhn, 1958] scores sentences with the TF*IDF of their terms. The
best sentences are then greedily extracted.

LexRank [Erkan and Radev, 2004] is a well-known graph-based approach. A
similarity graph $G(V,E)$ is constructed where $V$ is the set of sentences and
an edge $e_{ij}$ is drawn between sentences $v_{i}$ and $v_{j}$ if and only if
the cosine similarity between them is above a given threshold. Sentences are
then scored according to their PageRank in $G$.

LSA [Steinberger and Jezek, 2004] is an approach involving a dimensionality
reduction of the term-document matrix via singular value decomposition (SVD).
The sentences extracted should cover the most important latent topics.

KL-Greedy [Haghighi and Vanderwende, 2009] minimizes the Kullback-Leibler (KL)
divergence between the word distributions in summary and the documents.

ICSI [Gillick and Favre, 2009] is a global linear optimization that extracts a
summary by solving a maximum coverage problem considering the most frequent
bigrams in the source documents. ICSI has been among the state-of-the-art MDS
systems when evaluated with ROUGE [Hong et al., 2014].

###  5.2. Upper bound

For comparison, we compute two upper bounds. The upper bound for extractive
summarization is retrieved by solving the maximum coverage of n-grams from the
reference summary [Takamura and Okumura, 2010, Peyrard and Eckle-Kohler, 2016,
P.V.S. and Meyer, 2017]. This is cast as an Integer Linear Programming (ILP)
and depends on two parameters: $N$, the size of n-grams considered and $L$,
the maximum length of the summaries. In our work, we set $N=1$ and $N=2$ and
compute the upper bound for ROUGE-1 (UB-1) and ROUGE-2 (UB-2) respectively.

###  5.3. Experimental Setup

We report scores for the ROUGE metrics identified by ?) as strongly
correlating with human evaluation methods: ROUGE-1 (R1) and ROUGE-2 (R2)
recall with stemming and stop words not removed. For completeness, we also
report the best skip-grams matching metric: ROUGE-SU4 (SU4).

###  5.4. Analysis

Table 5 shows the results of benchmark summarization methods widely used in
the summarization community on our live blog corpus. We explore two different
summary lengths: $L$, length of the human-written bullet-point summary, and
$2*L$, twice the length of the human-written summary to give leeway for
compensating the excessive compression ratio of the human live blog summaries.
The results show the state-of-the-art ICSI system is .2 ROUGE-1 and .3 ROUGE-2
lower than the upper bounds for both BBC and the Guardian with length
constraint $L$ and $2*L$ respectively. ICSI is only able to reach one-third of
the upper bound, which emphasizes that live blog summarization is a
challenging task and we need new techniques tackling live blog summarization.

Figure 3 shows the output of the ICSI system as compared to the extractive
upper bound on BBC live blog on “FIFA corruption
inquiry”.111111http://www.bbc.com/news/live/world-europe-32897157 It can be
seen that the ICSI system extracts sentences with most frequent concepts
(e.g., FIFA, president, world cup), but misses to identify topic shifts in
these information snippets. Although the information snippets collected by the
ICSI system are related to FIFA corruption, it misses capturing relative
importance of the information snippets.

Additionally, factors which determine the difficulty of the summarization task
are the length of the source documents and the summary [Nenkova and Louis,
2008]. The input document sizes of the BBC and the Guardian are on an average
5,890 and 6,048 words, whereas the summary sizes are around 59 and 42 words
respectively. Thus, the high compression ratio makes live blog summarization
even more challenging.

##  6\. Conclusion and Future Work

We introduce a new task: live blog summarization which has direct applications
for journalists and news readers. Our goal is constructing a reference corpus
for this new task. In this paper, we suggest a pipeline to collect live blogs
with human written bullet-point summaries from two major online newspapers,
which can be extended to live blogs from other news agencies like _The New
York Times_ , _Washington Post_ or _Der Spiegel_.

We further analyze the live blog corpus and provide benchmark results for this
dataset by applying commonly used summarization methods. Our results show that
off-the-shelf summarization systems cannot be used, as they are far from
reaching the upper bound. This calls for new solutions that take the task
characteristics into account. As future work, we plan to research novel
approaches to live blog summarization and investigate algorithms to identify
important information from multiple topic shifts and a large number of
information snippets.

Code for constructing and reproducing the live blog corpus and the automatic
summarization experiments are published under the permissive Apache License
2.0 and can be obtained from https://github.com/UKPLab/lrec2018-live-blog-
corpus.

## Acknowledgments

This work has been supported by the German Research Foundation as part of the
Research Training Group “Adaptive Preparation of Information from
Heterogeneous Sources” (AIPHES) under grant No. GRK 1994/1. We also
acknowledge the useful comments and suggestions of the anonymous reviewers.

##  7\. Bibliographical References

## References

  * Baroni and Bernardini, 2004 Baroni, M. and Bernardini, S.  (2004).  BootCaT: Bootstrapping Corpora and Terms from the Web.  In Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC), pages 1313–1316, Lisbon, Portugal.
  * Benikova et al., 2016 Benikova, D., Mieskes, M., Meyer, C. M., and Gurevych, I.  (2016).  Bridging the gap between extractive and abstractive summaries: Creation and evaluation of coherent extracts from heterogeneous sources.  In Proceedings of the 26th International Conference on Computational Linguistics (COLING), pages 1039–1050, Osaka, Japan.
  * Bird et al., 2008 Bird, S., Dale, R., J. Dorr, B., Gibson, B., Joseph, M., Kan, M.-Y., Lee, D., Powley, B., Radev, D., and Fan Tan, Y.  (2008).  The ACL Anthology Reference Corpus: A Reference Dataset for Bibliographic Research in Computational Linguistics.  In Proceedings of Language Resources and Evaluation Conference (LREC), Marrakech, Morocco.
  * Cao et al., 2016 Cao, Z., Chen, C., Li, W., Li, S., Wei, F., and Zhou, M.  (2016).  Tgsum: Build tweet guided multi-document summarization dataset.  In Proceedings of the Thirtieth Conference on Artificial Intelligence (AAAI), pages 2906–2912, Phoenix, AZ, USA.
  * Dang and Owczarzak, 2008 Dang, H. and Owczarzak, K.  (2008).  Overview of the TAC 2008 update summarization task.  In Proceedings of the First Text Analysis Conference (TAC), pages 1–16, Gaithersburg, MD, USA.
  * Erkan and Radev, 2004 Erkan, G. and Radev, D. R.  (2004).  LexRank: Graph-based Lexical Centrality As Salience in Text Summarization.  Journal of Artificial Intelligence Research, 22:457–479.
  * Giannakopoulos et al., 2015 Giannakopoulos, G., Kubina, J., Conroy, J., Steinberger, J., Favre, B., Kabadjov, M., Kruschwitz, U., and Poesio, M.  (2015).  MultiLing 2015: Multilingual Summarization of Single and Multi-Documents, On-line Fora, and Call-center Conversations.  In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 270–274, Prague, Czech Republic.
  * Gillick and Favre, 2009 Gillick, D. and Favre, B.  (2009).  A scalable global model for summarization.  In Proceedings of the Workshop on Integer Linear Programming for Natural Langauge Processing, pages 10–18, Boulder, CO, USA.
  * Haghighi and Vanderwende, 2009 Haghighi, A. and Vanderwende, L.  (2009).  Exploring Content Models for Multi-document Summarization.  In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 362–370, Boulder, CO, USA.
  * Hermann et al., 2015 Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., and Blunsom, P.  (2015).  Teaching Machines to Read and Comprehend.  In Proceedings of the 28th International Conference on Neural Information Processing Systems (NIPS), pages 1693–1701, Montreal, Canada.
  * Hong et al., 2014 Hong, K., Conroy, J., Favre, b., Kulesza, A., Lin, H., and Nenkova, A.  (2014).  A repository of state of the art and competitive baseline summaries for generic news summarization.  In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC), pages 1608–1616, Reykjavik, Iceland.
  * Li et al., 2017 Li, P., Bing, L., and Lam, W.  (2017).  Reader-Aware Multi-Document Summarization: An Enhanced Model and The First Dataset.  In Proceedings of the EMNLP Workshop on New Frontiers in Summarization, pages 91–99, Copenhagen, Denmark.
  * Lin et al., 2016 Lin, J., Roegiest, A., Tan, L., McCreadie, R., Voorhees, E., and Diaz, F.  (2016).  Overview of the trec 2016 real-time summarization track.  In Proceedings of The Twenty-Fifth Text REtrieval Conference (TREC), Gaithersburg, MD, USA.
  * Lloret and Palomar, 2013 Lloret, E. and Palomar, M.  (2013).  Towards automatic tweet generation: A comparative study from the text summarization perspective in the journalism genre.  Expert Systems with Applications, 40(16):6624–6630.
  * Luhn, 1958 Luhn, H. P.  (1958).  The automatic creation of literature abstracts.  IBM Journal of Research Development, 2:159–165.
  * Nakano et al., 2010 Nakano, M., Shibuki, H., Miyazaki, R., Ishioroshi, M., Kaneko, K., and Mori, T.  (2010).  Construction of Text Summarization Corpus for the Credibility of Information on the Web.  In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC), pages 3125–3131, Valletta, Malta.
  * Nallapati et al., 2016 Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., and Xiang, B.  (2016).  Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond.  In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290, Berlin, Germany.
  * Nenkova and Louis, 2008 Nenkova, A. and Louis, A.  (2008).  Can you summarize this? identifying correlates of input difficulty for multi-document summarization.  In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics (ACL), pages 825–833, Columbus, OH, USA.
  * Owczarzak et al., 2012 Owczarzak, K., Conroy, J. M., Dang, H. T., and Nenkova, A.  (2012).  An assessment of the accuracy of automatic evaluation in summarization.  In Proceedings of Workshop on Evaluation Metrics and System Comparison for Automatic Summarization, pages 1–9, Montreal, Canada.
  * Peyrard and Eckle-Kohler, 2016 Peyrard, M. and Eckle-Kohler, J.  (2016).  Optimizing an Approximation of ROUGE – a Problem-Reduction Approach to Extractive Multi-Document Summarization.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1825–1836, Berlin, Germany.
  * P.V.S. and Meyer, 2017 P.V.S., A. and Meyer, C. M.  (2017).  Joint Optimization of User-desired Content in Multi-document Summaries by Learning from User Feedback.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1353–1363, Vancouver, Canada.
  * See et al., 2017 See, A., Liu, P. J., and Manning, C. D.  (2017).  Get to the point: Summarization with pointer-generator networks.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1073–1083, Vancouver, Canada.
  * Steinberger and Jezek, 2004 Steinberger, J. and Jezek, K.  (2004).  Using latent semantic analysis in text summarization and summary evaluation.  In Proceedings of the 7th International Conference on Information Systems Implementation and Modelling (ISIM), pages 93–100, Rožnov pod Radhoštěm, Czech Republic.
  * Takamura and Okumura, 2010 Takamura, H. and Okumura, M.  (2010).  Learning to generate summary as structured output.  In Proceedings of the 19th ACM international Conference on Information and Knowledge Management, pages 1437–1440, Toronto, Canada.
  * Thurman and Schapals, 2017 Thurman, N. and Schapals, A. K.  (2017).  Live blogs, sources, and objectivity: The contradictions of real-time online reporting.  In The Routledge Companion to Digital Journalism Studies, pages 283–292. London/New York: Routledge.
  * Zopf et al., 2016 Zopf, M., Peyrard, M., and Eckle-Kohler, J.  (2016).  The Next Step for Multi-Document Summarization: A Heterogeneous Multi-Genre Corpus Built with a Novel Construction Approach.  In Proceedings of the 26th International Conference on Computational Linguistics (COLING), pages 1535–1545, Osaka, Japan.


11institutetext: Precision Medicine Centre of Excellence, The Patrick G
Johnston Centre for Cancer Research, Queen’s University Belfast, Belfast BT9
7AE, UK 22institutetext: Northern Ireland Biobank, The Patrick G Johnston
Centre for Cancer Research, Queen’s University Belfast, Belfast BT9 7AE, UK
33institutetext: Sonrai Analytics Ltd., Lisburn Road, Belfast BT9 7BL, UK

# ICOS Protein Expression Segmentation: Can Transformer Networks Give Better
Results?

Vivek Kumar Singh 11 Paul O’Reilly 33 Jacqueline James 11 2 2 Manuel Salto-
Tellez 11 Perry Maxwell 11

###### Abstract

Biomarkers identify a patient’s response to treatment. With the recent
advances in artificial intelligence based on the Transformer networks, there
is only limited research has been done to measure the performance on
challenging histopathology images. In this paper, we investigate the efficacy
of the numerous state-of-the-art Transformer networks for immune-checkpoint
biomarker, Inducible T-cell COStimulator (ICOS) protein cell segmentation in
colon cancer from immunohistochemistry (IHC) slides. Extensive and
comprehensive experimental results confirm that MiSSFormer achieved the
highest Dice score of $74.85\%$ than the rest evaluated Transformer and
Efficient U-Net methods.

###### Keywords:

Artificial intelligence Immunohistochemistry Transformer.

##  1 Introduction

Inducible Co-Stimulator (ICOS) may be a biomarker of interest in checkpoint
inhibitor therapy and as a means of assessing T-cell regulation as part of a
complex process of adaptive immunity. Over the last decade, deep learning-
based convolutional neural networks (CNNs) have achieved outstanding
performance in multiple applications [3]. Capturing long-term dependencies
requires increasing the size of the convolution kernel, which can slow down
the system while improving the representation of functionality. Recently,
Transformer-based networks developed that use self-attention mechanisms to
extract long-distance dependencies without slowing them down. This
demonstrates significant benefits for natural image classification,
segmentation, and recognition tasks. Due to the higher parameters and the need
of larger datasets in training, the work done in the medical field is limited
[6]. Therefore, the main motivation for this study is to measure the
performance of the multiple Transformer methods, which capture a wide range of
contextual information of different shapes and sizes of ICOS cells.

Figure 1: Illustration of the proposed pipeline for ICOS cells segmentation.

Figure 1 shows the general pipeline of proposed framework. To the best of our
knowledge, we believe this is the first adaptation of analyzing the multiple
Transformer network (i.e., SETR [8], TransUNet [2], Swin-UNet [1], MedT [7]
and MiSSFormer [4]) performance with self-attention mechanism in the
application for ICOS cell segmentation from IHC images.

##  2 Material and Methods

Dataset: In this study, we used ICOS IHC data from our previous research [5].
The entire dataset of 534 image patches is divided into training, validation,
and test sets at 60%, 10%, and 30%, respectively.

Transformer-based segmentation methods: we used the five methods describe as
follows.

SETR [8] substitutes the convolutions in the encoder with a spatial resolution
Transformer. The input is converted to a sequence of many patches via learned
patch embedding. It transformed through a global self-attention delivering an
identifiable feature representation. The decoder generates the binary mask.
The source code is available at https://github.com/920232796/SETR-pytorch.

TransUNet [2] incorporated the CNN-Transformer network. The CNN layers capture
spatial information, and Transformer responsible for global feature. It has
u-shape, where extracted self-attention features maps upsampled to be merged
with varying CNN features skipped from the encoder. The implementation is
available at https://github.com/Beckschen/TransUNet.

Swin-UNet [1] used the Swin Transformer in the encoder with shifted windows
approach. The image is split into several patches and fed to the encoder
extracting spatial and global features. The decoder uses a symmetric Swin
Transformer consisting patch expanding layer to create the mask. The source
code is available at https://github.com/HuCaoFighting/Swin-Unet.

MedT [7] combines the extracted encoding layers local and global features. It
applied a Local-Global training process for Transformers that utilize a
narrower global path and a deep local path to works on the image patches. It
leverages the axial attention technique that maintains the information passed
to key, query, and value via a positional embedding. The implementation is
available at https://github.com/jeya-maria-jose/Medical-Transformer.

MISSFormer [4] used the enhanced Transformer block to obtain better features
of ICOS cells. It includes the encoder, bridge, and decoder network with skip
connection to narrow down the semantic gaps. The encoder layer extracts the
relevant features of overlapped patches. The local and global features are
combined via the bridge mechanism. The decoder network provide the
segmentation map. The implementation is available at
https://github.com/ZhifangDeng/MISSFormer.

Implementation details: We used the input size of $256\times 256$ and applied
the data augmentation with rotation, and horizontal flipping. An Adam
optimizer was employed with a learning rate of 0.0002. We used the binary
cross-entropy and dice losses. All the models has trained with a batch size of
4 with 50 epochs.

##  3 Experimental Results

Table 1 shows the five state-of-the-art Transformer-based methods for ICOS
cell segmentation. It includes the SETR [8], TransUNet [2], Swin-UNet [1],
MedT [7] and MiSSFormer [4]. Experimental results confirm that MiSSFormer has
achieved the highest Dice and aggregated Jaccard index (AJI) scores of
$74.85\%$ and $59.26\%$ respectively, than existing methods. This method
allows capturing the long-range dependencies and local context of multi-scale
ICOS cells features, improving segmentation results. SETR has achieved the
lowest Dice and AJI scores, which were $17\%$ lower against the MiSSFormer. We
found that TransUNet, Swin-UNet, and MedT performed relatively well and were
very similar in all the metrics. We compared the Transformer methods to recent
work [5] that achieved a $73.44\%$ Dice score, which was $1.5\%$ lower than
MiSSFormer. Overall, Transformer-based method shows better segmentation
results than CNN.

Table 1: Comparison of the latest Transformer-based segmentation models with
ICOS.

Model | Dice coefficient | AJI | Sensitivity | Specificity
---|---|---|---|---
SETR | 57.90 | 41.95 | 64.47 | 99.03
TransUNet | 71.09 | 56.39 | 78.78 | 99.28
Swin-UNet | 71.52 | 55.61 | 77.92 | 99.42
MedT | 71.61 | 56.91 | 80.26 | 99.28
MiSSFormer | 74.85 | 59.26 | 81.13 | 99.49

Figure 2 provide the qualitative comparison of Transformer-based methods. We
provide the two examples has variability in cell structures. The reported AJI
score confirms that the MiSSFormer performed better than other methods and had
fewer false positives. We also show the heatmap generated by the intermediate
layer of the MiSSFormer. The higher red highlight network precisely captures
the cells and ignores the background shown in blue.

Figure 2: Illustration of five Transformer methods for ICOS cells
segmentation.

##  4 Conclusion

This paper presents novel research to investigate Transformer-based networks
for ICOS cell segmentation. We examined the effectiveness of five state-of-
the-art segmentation methods. Current studies comparing Transformer results to
the recent CNN-based method show MiSSFomer attained improved segmentation
results in all metrics. It can learn better feature representation than CNN.
Future work aims to generalize the Transformer network to other protein cell
datasets.

## References

  * [1] Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q., Wang, M.: Swin-unet: Unet-like pure transformer for medical image segmentation. arXiv preprint arXiv:2105.05537 (2021)
  * [2] Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A.L., Zhou, Y.: Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306 (2021)
  * [3] Dong, S., Wang, P., Abbas, K.: A survey on deep learning and its applications. Computer Science Review 40, 100379 (2021)
  * [4] Huang, X., Deng, Z., Li, D., Yuan, X.: Missformer: An effective medical image segmentation transformer. arXiv preprint arXiv:2109.07162 (2021)
  * [5] Sarker, M.M.K., Makhlouf, Y., Craig, S.G., Humphries, M.P., Loughrey, M., James, J.A., Salto-Tellez, M., O’Reilly, P., Maxwell, P.: A means of assessing deep learning-based detection of icos protein expression in colon cancer. Cancers 13(15), 3825 (2021)
  * [6] Shamshad, F., Khan, S., Zamir, S.W., Khan, M.H., Hayat, M., Khan, F.S., Fu, H.: Transformers in medical imaging: A survey. arXiv preprint arXiv:2201.09873 (2022)
  * [7] Valanarasu, J.M.J., Oza, P., Hacihaliloglu, I., Patel, V.M.: Medical transformer: Gated axial-attention for medical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 36–46. Springer (2021)
  * [8] Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P.H., et al.: Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 6881–6890 (2021)