#  IMAE for Noise-Robust Learning: Mean Absolute Error Does Not Treat Examples
Equally and Gradient Magnitude’s Variance Matters

Xinshao Wang∗ Yang Hua  Elyor Kodirov  Neil M. Robertson

#  Supplementary Material for IMAE

Xinshao Wang∗ Yang Hua  Elyor Kodirov  Neil M. Robertson

###### Abstract

In this work, we study robust deep learning against abnormal training data
from the perspective of example weighting built in empirical loss functions,
i.e., gradient magnitude with respect to logits, an angle that is not
thoroughly studied so far. Consequently, we have two key findings: (1) Mean
Absolute Error (MAE) Does Not Treat Examples Equally. We present new
observations and insightful analysis about MAE, which is theoretically proved
to be noise-robust. First, we reveal its underfitting problem in practice.
Second, we analyse that MAE’s noise-robustness is from emphasising on
uncertain examples instead of treating training samples equally, as claimed in
prior work. (2) The Variance of Gradient Magnitude Matters. We propose an
effective and simple solution to enhance MAE’s fitting ability while
preserving its noise-robustness. Without changing MAE’s overall weighting
scheme, i.e., what examples get higher weights, we simply change its weighting
variance non-linearly so that the impact ratio between two examples are
adjusted. Our solution is termed Improved MAE (IMAE). We prove IMAE’s
effectiveness using extensive experiments: image classification under clean
labels, synthetic label noise, and real-world unknown noise. We conclude IMAE
is superior to CCE, the most popular loss for training DNNs.

Cross Entropy, Mean Absolute Error, Underfitting, Overfitting, Label Noise,
Robustness

##  1 Introduction

In this work, we target at robust deep learning, which is indispensable when
it comes to large-scale industrial applications. It is non-affordable to
guarantee the quality of training data as its scale grows dramatically.
Consequently, abnormal examples111A training example is denoted as an
observation-label pair, where the observation can be an image or video while
the label defines its semantic information. We regard a training example as
abnormal unrestrictedly whenever its observation and label are semantically
unmatched, e.g., out-of-distribution examples (the observations contain only
background or objects that do not belong to any training class), or examples
with wrongly annotated labels. generally exist in large-scale real-world
scenarios (Berrada et al., 2018), which is caused by many factors, such as
incomplete annotation, wrong labelling, subjectiveness, bias and so forth.
Unfortunately, DNNs trained with categorical cross entropy (CCE) can fit
random patterns (Zhang et al., 2017).

Great advances have been made towards training DNNs robustly when abnormal
training examples exist (Arpit et al., 2017; Chang et al., 2017; Ren et al.,
2018; Jiang et al., 2018). The robust loss function is one of them. In this
paper, we study a so-claimed robust loss function, mean absolute error (MAE)
following (Ghosh et al., 2017; Zhang & Sabuncu, 2018). According to the
theoretical analysis of CCE and MAE in (Ghosh et al., 2017), CCE is sensitive
to label noise while MAE is noise-tolerant. Thereafter, generalised cross
entropy (GCE) (Zhang & Sabuncu, 2018) concludes MAE treats training samples
equally, thus being noise-robust.

However, our empirical observation and technical analysis lead us to a
contradictory and more reasonable conclusion.

Observation: In Table 1, when 40% noise exists, compared with CCE, MAE
underfits to clean training data points, thus fitting much fewer abnormal
examples.

Conclusion: In Figure 1, MAE emphasises more on uncertain examples, whose
probabilities of being classified to its labelled class are around 0.5, thus
being noise-robust.

Specifically, according to Table 1, MAE is much more noise-tolerant than CCE.
However, its ability of learning meaningful patterns is much weaker, fitting
only 74.3% of the clean subset. We provide an intuitive interpretation for
this according to Figure 1: The variance of MAE’s weight curve along with
probability is only 0.09. As a result, the impact ratio between two examples
is too small.222The terms, examples’ weight or impact, and examples’ gradient
magnitude w.r.t. logits, are used interchangeably because we define the weight
by gradient’s magnitude. The impact ratio between two examples is changed only
when gradients’ magnitude is scaled non-linearly. The impact ratio reflects
the relative impact of one example versus another for updating parameters. Due
to MAE’s small weight variance, informative samples cannot contribute enough
against non-informative ones. Therefore, MAE cannot learn meaningful patterns
well and is not widely used.

To adjust MAE’s weight variance, we design an effective and simple solution,
IMAE, which non-linearly transforms MAE’s weighting scheme by an exponential
function. On the one hand, by preserving MAE’s overall weighting scheme, IMAE
is noise-robust. On the other hand, by making the gradient magnitude’s
variance over training examples controllable, it learns meaningful patterns
much better.

We demonstrate the effectiveness of IMAE under different scenarios. Most
importantly, these empirical evidences justify that our interpretation of
MAE’s underfitting problem is reasonable and our proposed solution is
superior. Our key findings are summarised as follows:

  * •

CCE overfits to noise easily because it emphasises on low-probability examples
to which abnormal ones generally belong. Although CCE’s weight variance is not
large (0.33), its fitting ability benefits from emphasising on low-probability
examples.

  * •

MAE is noise-robust by focusing on uncertain (medium-probability) examples
instead of treating all equally. However, MAE generally underfits due to its
small weights variance (0.09), leading to small impact ratio between even far
different examples.

  * •

Our proposed IMAE achieves new state-of-the-art on robust training against
synthetic label noise and realistic unknown noise simply by adjusting MAE’s
weight variance, which is inspiring.

Table 1:  Classification accuracy (%) of CCE, MAE, and IMAE on CIFAR-10 (Krizhevsky, 2009). 40% of training examples, i.e., the noisy subset, have wrong labels. We test each model’s performance on test set, noisy subset and clean subset of training data. The backbone is ResNet56 owning enough capacity (He et al., 2016).  Loss |  Test set (Generalisation) |  Noisy subset (Noise-tolerance) |  Clean subset (Learning ability)
---|---|---|---
CCE | 63.3 | 75.0 | 96.2
MAE | 66.9 | 8.1 |  74.3 (worst)
IMAE |  81.5 (best) |  6.5 (best) | 93.1
Figure 1:  Sample’s weight along with sample’s probability being classified to
its labelled class in CCE, MAE, IMAE with $T=8$. If probabilities are
uniformly distributed, the variances of CCE’s, MAE’s and IMAE’s weighting
curves are 0.33, 0.09 and 4.55, respectively.

##  2 Preliminaries

We denote a training mini-batch as
$\mathbf{X}=\\{(\mathbf{x}_{i},y_{i})\\}_{i=1}^{N}$, where there are $N$
samples. $(\mathbf{x}_{i},y_{i})$ represents $i$-th training sample
$\mathbf{x}_{i}\in\mathbb{R}^{D}$ and its annotated class label
$y_{i}\in\\{1,2,...,C\\}$. $D$ is the dimensionality of input samples and $C$
is the number of all training classes. Let $f_{\theta}$ be a deep neural
network, which transforms $\mathbf{x}_{i}$ to a representation
$\mathbf{f}_{i}=f_{\theta}(\mathbf{x}_{i})\in\mathbb{R}^{E}$, $E$ is the
dimensionality of target space and $\theta$ indicates the parameters to be
learned.

To optimise $f_{\theta}$ during training, a linear classifier is generally
trained jointly (Liu et al., 2016). In general, the linear classifier follows
the output embeddings and is composed of one $C$-neuron fully connected (FC)
layer, one softmax normalisation layer and one loss layer. The FC layer can be
represented as
$\mathbf{z}_{i}=\mathbf{W}^{\top}\mathbf{f}_{i}\in\mathbb{R}^{C}$, where
$\mathbf{W}=[\mathbf{w}_{1},\mathbf{w}_{2},...,\mathbf{w}_{C}]\in\mathbb{R}^{E\times
C}$ consists of $C$ weight vectors (the bias term is omitted for brevity).
$\mathbf{z}_{ij}=\mathbf{w}_{j}^{\top}\mathbf{f}_{i}$ is a logit which
indicates the compatibility between sample $\mathbf{x}_{i}$ and class $j$. To
produce the probabilities of sample $\mathbf{x}_{i}$ belonging to different
classes, we normalise its logit vector $\mathbf{z}_{i}$ using a softmax
function:

$p(j|\mathbf{x}_{i})=\frac{\exp(\mathbf{z}_{ij})}{\sum_{m=1}^{C}\exp(\mathbf{z}_{im})},$
(1)

where $p(j|\mathbf{x}_{i})$ is the probability of sample $\mathbf{x}_{i}$
being predicted to class $j$.

Let $q(j|\mathbf{x}_{i})$ be the ground-truth probability of $\mathbf{x}_{i}$
belonging to class $j$, i.e., $q(j|\mathbf{x}_{i})=1$ if $j=y_{i}$,
$q(j|\mathbf{x}_{i})=0$ otherwise. In the loss layer, if we use CCE, the
minimisation objective per iteration is:

$\displaystyle L_{\mathrm{CCE}}(\mathbf{X};f_{\theta},\mathbf{W})$
$\displaystyle=-\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{C}q(j|\mathbf{x}_{i})\log
p(j|\mathbf{x}_{i})$ (2) $\displaystyle=-\frac{1}{N}\sum_{i=1}^{N}\log
p(y_{i}|\mathbf{x}_{i}).$

If MAE is applied, the minimisation objective becomes:

$\displaystyle L_{\mathrm{MAE}}(\mathbf{X};f_{\theta},\mathbf{W})$
$\displaystyle=\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{C}|p(j|\mathbf{x}_{i})-q(j|\mathbf{x}_{i})|$
(3) $\displaystyle=\frac{2}{N}\sum_{i=1}^{N}(1-p(y_{i}|\mathbf{x}_{i})),$

where $|\cdot|$ is the absolute function.

In summary, we learn a softmax deep network $g_{\theta,\mathbf{W}}$, which
outputs logits:
$\mathbf{z}_{i}=g_{\theta,\mathbf{W}}(\mathbf{x}_{i})=\mathbf{W}^{\top}f_{\theta}(\mathbf{x}_{i})\in\mathbb{R}^{C}.$
In classification tasks, we use $\mathbf{z}=g_{\theta,\mathbf{W}}(\mathbf{x})$
to produce logits for a test image $\mathbf{x}$. While in verification or
retrieval tasks (Wang et al., 2019a, b, d), we only use
$\mathbf{f}=f_{\theta}(\mathbf{x})$ as an embedding function. The overall
pipeline is described in Figure 2. The output of the softmax layer is
$\mathbf{p}$.

Figure 2:  Pipeline of a softmax deep network. There are two reasons for
analysing loss functions based on $\frac{\partial L}{\partial\mathbf{z}}$: (1)
In gradient back-propagation, the gradients of examples in a mini-batch are
fused when computing $\frac{\partial L}{\partial\mathbf{z}}$. (2) Intermediate
differences of $\frac{\partial L}{\partial\mathbf{p}}$ lead to ultimate
differences of $\frac{\partial L}{\partial\mathbf{z}}$. Therefore, our
analysis of $\frac{\partial L}{\partial\mathbf{z}}$ is more direct versus that
of $\frac{\partial L}{\partial\mathbf{p}}$ in (Zhang & Sabuncu, 2018).

Definition 1 (Uncertain Examples). We define uncertain examples to be those
data points whose $p(y_{i}|\mathbf{x}_{i})$ are around 0.5. Given an example
$\mathbf{x}_{i}$, if its $p(y_{i}|\mathbf{x}_{i})$ is closer to 0.5, its
uncertainty is higher.

Remark 1. This definition of uncertain examples is intuitive. If
$p(y_{i}|\mathbf{x}_{i})$ is closer to 1, the confidence of $\mathbf{x}_{i}$
being class $y_{i}$ is higher. If $p(y_{i}|\mathbf{x}_{i})$ is closer to 0,
the confidence of $\mathbf{x}_{i}$ belonging to one of other classes is
higher. However, if $p(y_{i}|\mathbf{x}_{i})$ is around 0.5, we are more
uncertain about whether $\mathbf{x}_{i}$ being class $y_{i}$. Therefore, we
can understand uncertainty from the perspective of binary classification
(Logistic Regression), i.e., whether $\mathbf{x}_{i}$ being class $y_{i}$ or
not.

Remark 2. We have the premise that abnormal (noisy) examples have smaller
probabilities in general. This premise is widely used and demonstrated by our
empirical observations. For example, in Figure 4 and Tables 1, 6, the accuracy
of noisy subset is less than that of clean subset consistently.

Remark 3. The uncertainty of an example is determined by its probability of
being classified to its annotated label. This example can belong to one of the
training classes (uncertain in-distribution example), or a class which does
not exist in the training set (uncertain out-of-distribution example).

Table 2:  Summary of CCE, MAE and IMAE. $(\mathbf{x},y)$ is a training example. For simplicity, $p_{y}=p(y|\mathbf{x})$, and $p_{j}=p(j|\mathbf{x}),j\neq y$, $\sum_{j\neq y}p_{j}+p_{y}=1$. Prior analysis on loss functions is based on the loss expression or $\frac{\partial L}{\partial\mathbf{p}}$. Instead, we are the first to study the differences of loss functions according to $||\frac{\partial L}{\partial\mathbf{z}}||_{1}$. Our empirical evidences justifies its rationality. Note that we have $L(p_{y})=\int\frac{\partial L}{\partial p_{y}}dp_{y},L(1)=0$, therefore $L(p_{y})=\int_{p_{y}}^{1}-\frac{\partial L}{\partial p_{y}}dp_{y}$. We remark IMAE is neither symmetric nor bounded, which challenges the robustness theories studied in (Ghosh et al., 2017; Zhang & Sabuncu, 2018; Wang et al., 2019e).  |  | $\frac{\partial L}{\partial\mathbf{p}}$ | $\frac{\partial p_{y}}{\partial\mathbf{z}}$ | $\frac{\partial L}{\partial\mathbf{z}}=\sum_{j=1}^{C}\frac{\partial L}{\partial p_{j}}\times\frac{\partial p_{j}}{\partial\mathbf{z}}$ |
---|---|---|---|---|---
|  Loss Expression $L=L(p_{y})=$ $\int_{p_{y}}^{1}-\frac{\partial L}{\partial p_{y}}dp_{y}$ | $\frac{\partial L}{\partial p_{y}}$ |  $\frac{\partial L}{\partial p_{j}},$ $j\neq y$ | $\frac{\partial p_{y}}{\partial\mathbf{z}_{y}}$ |  |  $\frac{\partial p_{y}}{\partial\mathbf{z}_{j}}$,
---
$j\neq y$
$\frac{\partial L}{\partial\mathbf{z}_{y}}$ |  |  $\frac{\partial L}{\partial\mathbf{z}_{j}}$,
---
$j\neq y$
$||\frac{\partial L}{\partial\mathbf{z}}||_{1}$
CCE | $-\log p_{y}$ | $-\frac{1}{p_{y}}$ | 0 | $p_{y}(1-p_{y})\text{~{}~{}}$ | $-p_{y}p_{j}$ | $p_{y}-1$ | $p_{j}$ | $2(1-p_{y})$
MAE | $2(1-p_{y})$ | -2 | 0 | $p_{y}(1-p_{y})\text{~{}~{}}$ | $-p_{y}p_{j}$ | $2p_{y}(p_{y}-1)$ | $2p_{y}p_{j}$ | $4p_{y}(1-p_{y})$
IMAE |  $\int_{p_{y}}^{1}\frac{\exp(Tp_{y}(1-p_{y}))}{2p_{y}(1-p_{y})}dp_{y}$ | $\frac{\exp(Tp_{y}(1-p_{y}))}{2p_{y}(p_{y}-1)}$ | 0 | $p_{y}(1-p_{y})\text{~{}~{}}$ | $-p_{y}p_{j}$ | $\frac{\exp(Tp_{y}(1-p_{y}))}{-2}$ | $\frac{\exp(Tp_{y}(1-p_{y}))p_{j}}{2(1-p_{y})}$ |  $\exp(Tp_{y}(1-p_{y}))$

##  3 Gradient Magnitude Serving as Weight

As shown in Figure 2, $g_{\theta,\mathbf{W}}$ can be viewed as a black box and
the update of $\theta$ and $\mathbf{W}$ is based on the back-propagation of
logits’ gradient. Therefore, an example’s contribution can be measured by the
magnitude of its partial derivative w.r.t. $\mathbf{z}$. It can be regarded as
example weighting that is naturally built-in in loss functions.

For brevity and clarity, we summarise the results here and put the detailed
derivation in our supplementary material.

###  3.1 Derivation of Softmax, CCE and MAE Layers

Softmax layer. According to Eq. (1), the derivation of softmax layer is:

$\displaystyle\frac{\partial
p(y_{i}|\mathbf{x}_{i})}{\partial\mathbf{z}_{ij}}$
$\displaystyle=\begin{cases}p(y_{i}|\mathbf{x}_{i})(1-p(y_{i}|\mathbf{x}_{i}))\text{,
}&j=y_{i}\\\ -p(y_{i}|\mathbf{x}_{i})p(j|\mathbf{x}_{i})\text{, }&j\neq
y_{i}\end{cases}$ (4)

CCE layer. According to Eq. (2), we have

$\displaystyle L_{\mathrm{CCE}}(\mathbf{x}_{i};f_{\theta},\mathbf{W})$
$\displaystyle=-\log p(y_{i}|\mathbf{x}_{i}).$ (5)

Therefore, we obtain (the parameters $\theta,\mathbf{W}$ are omitted),

$\displaystyle\frac{\partial L_{\mathrm{CCE}}(\mathbf{x}_{i})}{\partial
p(j|\mathbf{x}_{i})}$
$\displaystyle=\begin{cases}-p(y_{i}|\mathbf{x}_{i})^{-1}\text{, }&j=y_{i}\\\
0\text{, }&j\neq y_{i}\end{cases}.$ (6)

MAE layer. According to Eq. (3), we have

$\displaystyle L_{\mathrm{MAE}}(\mathbf{x}_{i};f_{\theta},\mathbf{W})$
$\displaystyle=2(1-(p(y_{i}|\mathbf{x}_{i})).$ (7)

Therefore, we obtain

$\displaystyle\frac{\partial L_{\mathrm{MAE}}(\mathbf{x}_{i})}{\partial
p(j|\mathbf{x}_{i})}$ $\displaystyle=\begin{cases}-2\text{, }&j=y_{i}\\\
0\text{, }&j\neq y_{i}\end{cases}.$ (8)

###  3.2 Perspective of Derivatives w.r.t. Logits Other Than Probabilities

Prior conclusion according to $\frac{\partial
L_{\mathrm{CCE}}(\mathbf{x}_{i})}{\partial p(j|\mathbf{x}_{i})}$,
$\frac{\partial L_{\mathrm{MAE}}(\mathbf{x}_{i})}{\partial
p(j|\mathbf{x}_{i})}$: (Zhang & Sabuncu, 2018) concludes that CCE is sensitive
to abnormal examples while MAE is robust by treating all data points equally
according to Eq. (6) and Eq. (8), respectively.

In this work, we propose to further analyse $\frac{\partial
L_{\mathrm{CCE}}(\mathbf{x}_{i})}{\partial\mathbf{z}_{ij}}$, $\frac{\partial
L_{\mathrm{MAE}}(\mathbf{x}_{i})}{\partial\mathbf{z}_{ij}}$ as discussed in
Figure 2.

According to Eq. (6) and (4), we calculate $\partial
L_{\mathrm{CCE}}(\mathbf{x}_{i})/\partial\mathbf{z}_{i}$:

$\displaystyle\frac{\partial
L_{\mathrm{CCE}}(\mathbf{x}_{i})}{\partial\mathbf{z}_{ij}}$
$\displaystyle=\begin{cases}p(y_{i}|\mathbf{x}_{i})-1\text{, }&j=y_{i}\\\
p(j|\mathbf{x}_{i})\text{, }&j\neq y_{i}\end{cases}.$ (9)

Analogously, according to Eq. (8) and (4), we have:

$\displaystyle\frac{\partial
L_{\mathrm{MAE}}(\mathbf{x}_{i})}{\partial\mathbf{z}_{ij}}$
$\displaystyle=\begin{cases}2p(y_{i}|\mathbf{x}_{i})(p(y_{i}|\mathbf{x}_{i})-1)\text{,}&j=y_{i}\\\
2p(y_{i}|\mathbf{x}_{i})p(j|\mathbf{x}_{i})\text{,}&j\neq y_{i}\end{cases}.$
(10)

Gradient magnitude treated as weight. In CCE and MAE, training samples are
weighted because different ones own different gradient magnitude w.r.t. logit
vector $\mathbf{z}$. We choose to measure one gradient’s magnitude by its
$L_{1}$ norm because of its simpler statistics than other norms. If one
sample’s gradient is larger, its impact is larger during gradient back-
propagation.

For CCE, based on Eq. (9), the weight of sample $\mathbf{x}_{i}$ is:

$w_{\mathrm{CCE}}(\mathbf{x}_{i})=||\frac{\partial
L_{\mathrm{CCE}}(\mathbf{x}_{i})}{\partial\mathbf{z}_{i}}||_{1}=2(1-p(y_{i}|\mathbf{x}_{i})),$
(11)

where $||\cdot||_{1}$ denotes $L_{1}$ norm. For MAE, based on Eq. (10), the
weight of sample $\mathbf{x}_{i}$ is:

$w_{\mathrm{MAE}}(\mathbf{x}_{i})=||\frac{\partial
L_{\mathrm{MAE}}(\mathbf{x}_{i})}{\partial\mathbf{z}_{i}}||_{1}=4p(y_{i}|\mathbf{x}_{i})(1-p(y_{i}|\mathbf{x}_{i})).$
(12)

According to Eq. (11) and Eq. (12), in both CCE and MAE, examples’ impact is
determined by their probabilities being predicted to annotated labels.

Figure 3:  Although the loss expression of IMAE is not an elementary function,
we visualise it by integral, i.e., the area under curve from $p_{y}$ to 1.

##  4 Improved MAE

IMAE transforms MAE’s weighting scheme non-linearly:

$w_{\mathrm{IMAE}}(\mathbf{x}_{i})=\exp(Tp(y_{i}|\mathbf{x}_{i})(1-p(y_{i}|\mathbf{x}_{i}))),$
(13)

where $T$ controls the exponential base. In back-propagation, we simply scale
the gradient w.r.t. logits as follows:

$\displaystyle\frac{\partial
L_{\mathrm{IMAE}}(\mathbf{x}_{i})}{\partial\mathbf{z}_{i}}=\frac{\partial
L_{\mathrm{MAE}}(\mathbf{x}_{i})}{\partial\mathbf{z}_{i}}\frac{w_{\mathrm{IMAE}}(\mathbf{x}_{i})}{w_{\mathrm{MAE}}(\mathbf{x}_{i})}$
(14) $\displaystyle=>||\frac{\partial
L_{\mathrm{IMAE}}(\mathbf{x}_{i})}{\partial\mathbf{z}_{i}}||_{1}=w_{\mathrm{IMAE}}(\mathbf{x}_{i}).$

IMAE is a family of robust losses when $T$ changes. Its details are summarised
in Table 2 and Figure 3.

###  4.1 Design Motivation: To Adjust Gradient Magnitude’s Variance and Impact
Ratio

Linear scaling also changes magnitude variance. However, it cannot adjust
impact ratio, i.e., the ratio between two gradients’ magnitude. That is why we
have tried linear scaling and find it does not work.

Instead, the exponential function is non-linear so that the impact ratio of
one sample versus another is re-adjusted compared with original MAE. The
hyper-parameter $T$ controls how significant gradient magnitude’s variance and
impact ratio are changed.

Furthermore, assuming that samples’ probabilities are uniformly distributed,
we compute the gradients’ variance of MAE and IMAE over training data points:

$\displaystyle\sigma_{\mathrm{MAE}}$
$\displaystyle=\int^{1}_{0}w_{\mathrm{MAE}}^{2}(p)\,\mathrm{d}p-(\int^{1}_{0}w_{\mathrm{MAE}}(p)\,\mathrm{d}p)^{2}$
(15) $\displaystyle\sigma_{\mathrm{IMAE}}$
$\displaystyle=\int^{1}_{0}w_{\mathrm{\mathrm{IMAE}}}^{2}(p)\,\mathrm{d}p-(\int^{1}_{0}w_{\mathrm{IMAE}}(p)\,\mathrm{d}p)^{2}.$
(16)

We have $\sigma_{\mathrm{MAE}}=0.09$. When $T=8$,
$\sigma_{\mathrm{IMAE}}=4.55$.

###  4.2 Discussion of MAE and CCE

The weighting curves of CCE, MAE and IMAE are compared in Figure 1. Our key
findings are summarised in the end of introduction. We further discuss them as
follows:

  * •

MAE’s weighting scheme is appealing and practical in that samples with medium
probabilities are emphasized. Generally, high-probability samples are clean
and already trained well. While low-probability ones are highly likely to be
noisy as a model improves during training. Although all samples are not
trained well and probabilities are not meaningful at the beginning, it also
does not hurt to focus on medium-probability ones.

  * •

MAE’s gradient magnitude’s variance over data points is only 0.09. As a
consequence, the impact ratio of one example versus another is too small.
Therefore, the majority contribute almost equally. Therefore, MAE generally
underfits to training data.

  * •

Does high loss value usually back-propagate high gradients to update
parameters? The answer is NO. Therefore, those theorems based on loss values,
e.g., symmetric or bounded conditions are insufficient for analysing
robustness of DNNs (Ghosh et al., 2017). Actually, IMAE is neither symmetric
nor bounded. However, it is proved to be noise-robust empirically.

These analytical discussions are demonstrated in our empirical studies in
Table 6 and Figures 4, 10.

##  5 Experiments

We demonstrate the effectiveness of IMAE as follows:

Outperforming the state-of-the-art. IMAE is compared with recent baselines in
Sections 5.1 and 5.2 in different scenarios: (1) Clean labels; (2) Synthetic
symmetric and asymmetric noisy labels; (3) Realistic agnostic noise.

Analysis of the training dynamics of IMAE against CCE and MAE. We thoroughly
visualise and compare the training dynamics of IMAE, CCE and MAE in Section
5.3 for empirical justification.

Supplementary studies. In our supplementary material, we further prove IMAE’s
effectiveness by: (1) The results on a video retrieval task (video person re-
identification); (2) The results of different stochastic optimisers; (3) The
ablation study of $T$.

Table 3:  The results on CIFAR-100 using ResNet44. Results from SL and D2L are different due to different optimisation details. In our experiments, we fix the random seed as 123 and do not use any random computational accelerator for the purpose of exact reproducibility. The best results on each block and our IMAE are bolded.  | Method |  Clean Labels | Symmetric Noisy Labels
---|---|---|---
|  $r$=0.2 |  $r$=0.4 |  $r$=0.6
Results From SL | CCE | 64.3 | 59.3 | 50.8 | 25.4
LS | 63.7 | 58.8 | 50.1 | 24.7
Boot-hard | 63.3 | 57.9 | 48.2 | 12.3
Forward | 64.0 | 59.8 | 53.1 | 24.7
D2L | 64.6 | 59.2 | 52.0 | 35.3
GCE | 64.4 | 59.1 | 53.3 | 36.2
SL | 66.8 | 60.0 | 53.7 | 41.5
Results From D2L | CCE | 68.2 | 52.9 | 42.9 | 30.1
Boot-hard | 68.3 | 58.5 | 44.4 | 36.7
Boot-soft | 67.9 | 57.3 | 41.9 | 32.3
Forward | 68.5 | 60.3 | 51.3 | 41.2
Backward | 68.5 | 58.7 | 45.4 | 34.5
D2L | 68.6 | 62.2 | 52.0 | 42.3
Our Trained Results | CCE | 70.0 | 60.4 | 53.2 | 42.1
MAE | 8.2 | 6.4 | 7.3 | 5.2
IMAE | 69.2 | 63.4 | 54.7 | 43.9
Table 4:  The results on CIFAR-100 using ResNet44. The best results on each block are bolded.  | Method | Asymmetric Noisy Labels
---|---|---
$r$=0.2 |  $r$=0.3 |  $r$=0.4
Results From SL (Wang et al., 2019e) | CCE | 63.0 | 63.1 | 61.9
LS | 63.0 | 62.3 | 61.6
Bootstrap | 63.4 | 63.2 | 62.1
Forward | 64.1 | 64.0 | 60.9
D2L | 62.4 | 63.2 | 61.4
GCE | 63.0 | 63.2 | 61.7
SL | 65.6 | 65.1 | 63.1
Our trained Results | CCE | 66.4 | 64.7 | 60.3
MAE | 7.3 | 6.3 | 7.3
IMAE | 67.5 | 65.8 | 63.3
Table 5:  Classification accuracy (%) on Clothing1M with ResNet50 (He et al., 2016). The leftmost block’s results are from SL (Wang et al., 2019e) while the middle block’s are from Masking (Han et al., 2018).  CCE |  Boot-hard | Forward | D2L | GCE | SL |  S-adaptation | Masking |  Joint Optim. | Our trained results
---|---|---|---|---|---|---|---|---|---
CCE | MAE | IMAE
68.8 | 68.9 | 69.8 | 69.5 | 69.8 | 71.0 | 70.3 | 71.1 | 72.2 | 71.7 | 39.7 | 73.2

###  5.1 Image Classification on CIFAR-100 with Synthetic Noise

Dataset. CIFAR-100 (Krizhevsky, 2009) contains 100 classes, 500 images per
class for training and 100 images per class for testing. The image size is
$32\times 32$.

Synthetic label noise generation. (1) Class-independent (uniform or symmetric)
noise: With a probability of $r$, the label of each image is replaced by one
of the other class labels uniformly. (2) Class-dependent (non-uniform or
asymmetric) noise: The 100 classes of CIFAR-100 are grouped into 20 coarse
ones. Every coarse one has 5 fine classes. Following (Wang et al., 2019e), we
first randomly select 2 out of 5 classes, and then their labels are flipped to
each other with a probability of $r$. $r$ denotes the noise rate. All
instances generated from the same original image by data augmentation share
the same label. All test labels are kept intact.

Implementation details. We follow the settings of recent SL (Wang et al.,
2019e) and train ResNet44 (He et al., 2016) for a fair comparison with their
reported results. We also use the same data augmentation techniques: random
horizontal flips and crops of $32\times 32$ on the images after being padded
with 4 pixels on each side. All networks are trained using SGD with a momentum
of 0.9, a weight decay of 0.0005 and an initial learning rate of 0.1.

Baselines. IMAE is compared against standard CCE, MAE, and six recent robust
training baselines: 1) Forward (or Backward) applies a noise-transition matrix
to multiply the network’s predictions (or losses) for label correction purpose
(Patrini et al., 2017); 2) Bootstrapping learns on new labels generated by a
convex combination (soft or hard combinations) of the original ones and their
predictions (Reed et al., 2015). 3) D2L achieves noise-robustness by
restricting the dimensionality expansion of learned subspaces during training
(Ma et al., 2018); 4) SL boosts CCE with a noise-robust counterpart, i.e.,
reverse cross entropy (Wang et al., 2019e); 5) GCE aims to achieve a balance
between MAE and CCE (Zhang & Sabuncu, 2018); 6) Label Smoothing (LS) trains
DNNs on softly smoothed labels instead of one-hot ones; We remark that (Lee et
al., 2019) is not benchmarked for two reasons: (1) The used network is not
ResNet-44 by checking with the authors; (2) The proposed algorithm is
orthogonal to ours because it targets at the inference stage and is a
generative classifier on top of pre-trained deep representations. Our IMAE
focuses on the training stage and is a softmax-based neural classifier.

Results. We display the results in Tables 3 and 4. We observe that IMAE is
superior to the state-of-the-art. We fix the random seed as 123 and do not use
any random computational accelerator for the purpose of exact reproducibility.

###  5.2 Image Classification on Clothing1M with Realistic Unknown Noise

Dataset. Clothing1M (Xiao et al., 2015) contains one million clothing images
of fourteen classes from online shopping websites. Its noise type is agnostic.
The noise rate is around 38.46%. Additionally, it includes 50k, 14k, and 10k
images with clean labels for training, validation, and testing, respectively.
To compare fairly with existing algorithms without exploiting auxiliary
information from trusted clean data, we also train only on the noisy training
data.

Implementation details. We follow (Patrini et al., 2017; Tanaka et al., 2018;
Wang et al., 2019e) and train ResNet50 initialised by pretrained ImageNet
model (Russakovsky et al., 2015). We apply an SGD optimiser with a momentum of
0.9 and a weight decay of 0.00002. We set the initial learning rate to 0.01
and divide it by 10 after 10k and 15k iterations. We stop training at 30k
iterations. Regarding data augmentation, a raw input image is warped to
256$\times$256, followed by a random crop of 227$\times$227 and a random
horizontal mirroring. The batch size is 84. Every program is run on a single
Tesla V100 GPU with 32 GB RAM.

Competitors. Some recent baselines are compared: 1) S-adaptation explicitly
estimates latent true labels by an additional softmax layer (Goldberger & Ben-
Reuven, 2017); 2) Masking speculates the structure of a noise-transition
matrix with human cognition (Han et al., 2018); 3) Joint Optim. iteratively
optimises model’s parameters and latent true labels (Tanaka et al., 2018).
Others are introduced in Section 5.1. Note that (Han et al., 2019) corrects
labels gradually and (Li et al., 2019) exploits meta-learning. They are not
technically related and not benchmarked consequently.

Results. We display the results in Table 5. IMAE outperforms the state-of-the-
art, which proves IMAE’s effectiveness under real-world scenarios with
agnostic noise. Beyond, we remark that IMAE is much simpler than those
competitors except CCE, MAE.

Table 6:  Results (%) of CCE, MAE and IMAE on CIFAR-10 with different noise rates. For classification accuracy on the testing set, we show the best result achieved during training and the final result when training stops, which are indicated by ‘Best’ and ‘Final’, respectively. For training accuracy, the results on noisy and clean subsets are displayed. The hybrid accuracy represents the result on the combination of testing set and clean training set. We report training and hybrid accuracies of the final model when training terminates. The ultimate objective is to achieve high hybrid accuracy, since both training and testing data points may occur in a deployed system.  The best result on each column block is bolded. ‘–’ indicates there is no noisy subset.  Backbone | Noise rate | Loss | Testing accuracy |  Training accuracy: Naive fitting |  Hybrid accuracy: Meaningful patterns
---|---|---|---|---|---
Best | Final | Noisy subset | Clean subset
ResNet20 | 0% | CCE | 91.5 | 91.3 | – | 100 | 98.5
MAE | 89.3 | 89.2 | – | 95.8 | 94.7
IMAE | 91.7 | 91.4 | – | 99.8 | 98.4
40% | CCE | 81.2 | 67.0 | 34.3 | 93.3 | 72.6
MAE | 76.2 | 75.9 | 6.8 | 84.6 | 79.7
IMAE | 84.3 | 84.0 | 5.5 | 94.0 | 88.2
80% | CCE | 43.0 | 20.3 | 38.3 | 57.0 | 22.0
MAE | 27.7 | 27.5 | 9.7 | 29.4 | 27.8
IMAE | 52.0 | 41.0 | 16.8 | 64.8 | 41.5
ResNet56 | 0% | CCE | 92.4 | 92.2 | – | 100 | 98.7
MAE | 89.0 | 89.0 | – | 96.1 | 94.9
IMAE | 92.2 | 92.2 | – | 99.8 | 98.5
40% | CCE | 81.6 | 63.3 | 75.0 | 96.2 | 63.6
MAE | 67.0 | 66.9 | 8.1 | 74.3 | 70.2
IMAE | 82.2 | 81.5 | 6.5 | 93.1 | 86.5
80% | CCE | 38.2 | 16.4 | 52.5 | 62.3 | 18.7
MAE | 15.2 | 15.1 | 9.6 | 15.6 | 15.1
IMAE | 37.1 | 34.0 | 13.0 | 44.7 | 34.8

(a) ResNet20: Testing set (higher is better).

(b) ResNet20: Noisy subset (lower is better).

(c) ResNet20: Clean subset (higher is better).

(d) ResNet56: Testing set (higher is better).

(e) ResNet56: Noisy subset (lower is better).

(f) ResNet56: Clean subset (higher is better).

Figure 4: CIFAR-10 with noise rate $r=40\%$. The accuracies on testing set,
noisy subset and clean subset of training set along with training iterations.
The legend on the top left is shared by all subfigures. Better viewed in
colour.

###  5.3 Empirical Analysis of IMAE Against Basic Baselines CCE and MAE on
CIFAR-10

Dataset. CIFAR-10 (Krizhevsky, 2009) contains 10 classes, 5k images per class
for training and 1k images per class for testing. The image size is $32\times
32$.

Implementation details 333Our purpose is to study the behaviours of CCE, MAE
and IMAE on CIFAR-10 instead of pushing its state-of-the-art results.. We
follow the study on CIFAR-10 in (He et al., 2016), which means we use exactly
the same architectures (ResNet20, ResNet56) and training settings: a weight
decay of 0.0001, a momentum of 0.9, a batch size of 128. The learning rate
starts at 0.1, then is divided by 10 at 32k and 48k iterations. Training stops
at 100k iterations. Data augmentation is the same as CIFAR-100. For IMAE,
without tuning $T$ case by case, we fix $T=0.5$ when training data is clean
and $T=8$ when noise exists although noisy rate is different.444More
discussion about the hyper-parameter $T$ is given in our supplementary
material.

A well-accepted way to improve data fitting ability is increasing a model’s
capacity. Therefore, we train a shallower net ResNet20 and a deeper net
ResNet56 for better analysis.

####  5.3.1 CIFAR-10 with intact labels

In Table 6, we first compare IMAE with CCE and MAE on clean CIFAR-10 using
different nets (ResNet20, ResNet56). We observe that IMAE is competitive with
CCE and outperforms MAE significantly.

####  5.3.2 CIFAR-10 with corrupted labels

Following (Zhang et al., 2017; Arpit et al., 2017), we test the robustness of
deep models against corrupted labels. We evaluate on uniform noise because it
is more challenging than asymmetric noise which is verified in (Vahdat, 2017).

Majority voting assumption. When generating uniform noise on CIFAR-10, even up
to 80% noise rate, clean examples are still the majority because 80% labels
are corrupted to other 9 classes evenly. We remark that the majority voting is
our reasonable assumption. We believe that if the noise becomes the majority,
it is hard to discover meaningful patterns. Being natural and intuitive, the
majority define the meaningful data patterns to learn.

Results. The results are summarised in Table 6. For more comprehensive and
clear comparison, we display the training dynamics in Figures 4 (40% noise)
and 10 (80% noise) of the supplementary material. Note that general learning
objectives are high final testing accuracy, low accuracy on the noisy training
subset, and high accuracy on the clean training subset. Therefore, we report
the hybrid accuracy on the combination of testing set and clean training set.
We have the following observations:

  * •

Regarding CCE’s test accuracies, the best is always much higher than the
final. In Figures 4 and 10, as training goes, CCE always tries to fit the
noisy training subset better. Therefore, CCE learns a lot of error information
when severe noise exists. When it comes to MAE and IMAE, the gap between the
best and final accuracies is significantly smaller than that of CCE regardless
of net’s capacity.

  * •

The training accuracies on both noisy and clean subsets are compared. Whatever
the noise rate and net’s capacity are, CCE fits the noisy subset much more.
Although MAE fits the noisy subset much less, it fits the clean subset worst.
Instead, our IMAE fits the noisy subset little and the clean subset
competitively with CCE.

  * •

IMAE obtains the best hybrid accuracy consistently.

##  6 Related Work

IMAE is a family of robust loss functions, inspired by the intrinsic example
weighting scheme of MAE. Therefore, our work is related to some prior work
about example weighting and robust loss functions.

###  6.1 Example Weighting

In (Ren et al., 2018), a meta-learning algorithm weights data points according
to their gradient directions. The meta-learning algorithm is optimised on a
clean validation set. In contrast, our IMAE assigns weights to samples based
on their gradient magnitude and does not require extra clean set. MentorNet
(Jiang et al., 2018) learns data-driven weighting scheme, which guides
StudentNet to focus on samples whose labels are more trustful. In Active Bias
(Chang et al., 2017) and Focal Loss (Lin et al., 2017), uncertain and hard
examples are emphasised, respectively. Other related work on weighting samples
includes curriculum learning (Bengio et al., 2009), self-paced learning (Kumar
et al., 2010), and hard examples mining (Shrivastava et al., 2016; Wang et
al., 2019b). In summary, what makes ours special is that the weighting scheme
inherits from MAE, which is naturally built-in in the loss function without
intuitive designing.

###  6.2 Noise-Robust Theorems on Loss Functions

Noise-robust theorems on loss functions from the angle of symmetric and
bounded conditions on loss values have been studied recently (Ghosh et al.,
2017; Zhang & Sabuncu, 2018; Wang et al., 2019e). Does a robust loss function
have to be symmetric or bounded? The answer is NO according to this work.
Although IMAE is neither symmetric nor bounded, we have extensive empirical
studies to support its effectiveness.

##  7 Conclusion

In this work, we firstly present a thorough study of CCE and MAE technically
and empirically. Compared with previous work, we introduce our observations
and new conclusions: 1) MAE underfits to meaningful patterns; 2) MAE is noise-
tolerant because of emphasising on medium-probability (uncertain) examples
instead of treating all samples equally. Secondly, we claim gradient
magnitude’s variance matters. As a consequence, we propose an effective and
simple solution for addressing MAE’s underfitting issue while preserving its
noise-robustness. IMAE is a family of robust loss functions whose gradient
magnitude’s variance is adjustable.

We remark that our empirically demonstrated claim–“Gradient Magnitude’s
Variance Matters”–can be applied for other algorithms as well, for example,
CCE. However, it is beyond the scope of this work since we focus on analysing
MAE and how to improve MAE here. We will investigate this claim in other loss
functions in our future work.

## References

  * Arpit et al. (2017) Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., and Lacoste-Julien, S.  A closer look at memorization in deep networks.  In _ICML_ , 2017.
  * Bengio et al. (2009) Bengio, Y., Louradour, J., Collobert, R., and Weston, J.  Curriculum learning.  In _ICML_ , 2009.
  * Berrada et al. (2018) Berrada, L., Zisserman, A., and Pawan Kumar, M.  Smooth loss functions for deep top-k classification.  _ICLR_ , 2018.
  * Chang et al. (2017) Chang, H.-S., Learned-Miller, E., and McCallum, A.  Active bias: Training more accurate neural networks by emphasizing high variance samples.  In _NeurIPS_ , 2017.
  * Ghosh et al. (2017) Ghosh, A., Kumar, H., and Sastry, P.  Robust loss functions under label noise for deep neural networks.  In _AAAI_ , 2017.
  * Goldberger & Ben-Reuven (2017) Goldberger, J. and Ben-Reuven, E.  Training deep neural-networks using a noise adaptation layer.  In _ICLR_ , 2017.
  * Han et al. (2018) Han, B., Yao, J., Niu, G., Zhou, M., Tsang, I., Zhang, Y., and Sugiyama, M.  Masking: A new perspective of noisy supervision.  In _NeurIPS_ , 2018.
  * Han et al. (2019) Han, J., Luo, P., and Wang, X.  Deep self-learning from noisy labels.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 2019.
  * He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _CVPR_ , 2016.
  * Ioffe & Szegedy (2015) Ioffe, S. and Szegedy, C.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _ICML_ , 2015.
  * Jiang et al. (2018) Jiang, L., Zhou, Z., Leung, T., Li, L.-J., and Fei-Fei, L.  Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels.  In _ICML_ , 2018.
  * Krizhevsky (2009) Krizhevsky, A.  Learning multiple layers of features from tiny images.  2009\.
  * Kumar et al. (2010) Kumar, M. P., Packer, B., and Koller, D.  Self-paced learning for latent variable models.  In _NeurIPS_ , 2010.
  * Law et al. (2017) Law, M. T., Urtasun, R., and Zemel, R. S.  Deep spectral clustering learning.  In _ICML_ , 2017.
  * Lee et al. (2019) Lee, K., Yun, S., Lee, K., Lee, H., Li, B., and Shin, J.  Robust inference via generative classifiers for handling noisy labels.  In _ICML_ , 2019.
  * Li et al. (2019) Li, J., Wong, Y., Zhao, Q., and Kankanhalli, M. S.  Learning to learn from noisy labeled data.  In _CVPR_ , 2019.
  * Lin et al. (2017) Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollar, P.  Focal loss for dense object detection.  In _ICCV_ , 2017.
  * Liu et al. (2016) Liu, W., Wen, Y., Yu, Z., and Yang, M.  Large-margin softmax loss for convolutional neural networks.  In _ICML_ , 2016.
  * Liu et al. (2017) Liu, Y., Yan, J., and Ouyang, W.  Quality aware network for set to set recognition.  In _CVPR_ , 2017.
  * Ma et al. (2018) Ma, X., Wang, Y., Houle, M. E., Zhou, S., Erfani, S. M., Xia, S.-T., Wijewickrema, S., and Bailey, J.  Dimensionality-driven learning with noisy labels.  In _ICML_ , 2018.
  * Movshovitz-Attias et al. (2017) Movshovitz-Attias, Y., Toshev, A., Leung, T. K., Ioffe, S., and Singh, S.  No fuss distance metric learning using proxies.  In _ICCV_ , 2017.
  * Patrini et al. (2017) Patrini, G., Rozza, A., Menon, A. K., Nock, R., and Qu, L.  Making deep neural networks robust to label noise: A loss correction approach.  In _CVPR_ , 2017.
  * Reed et al. (2015) Reed, S., Lee, H., Anguelov, D., Szegedy, C., Erhan, D., and Rabinovich, A.  Training deep neural networks on noisy labels with bootstrapping.  In _ICLR Workshop_ , 2015.
  * Ren et al. (2018) Ren, M., Zeng, W., Yang, B., and Urtasun, R.  Learning to reweight examples for robust deep learning.  In _ICML_ , 2018.
  * Russakovsky et al. (2015) Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.  Imagenet large scale visual recognition challenge.  _International Journal of Computer Vision_ , pp. 211–252, 2015\.
  * Shrivastava et al. (2016) Shrivastava, A., Gupta, A., and Girshick, R.  Training region-based object detectors with online hard example mining.  In _CVPR_ , 2016.
  * Tanaka et al. (2018) Tanaka, D., Ikami, D., Yamasaki, T., and Aizawa, K.  Joint optimization framework for learning with noisy labels.  In _CVPR_ , 2018.
  * Vahdat (2017) Vahdat, A.  Toward robustness against label noise in training deep discriminative neural networks.  In _NeurIPS_ , 2017.
  * Wang et al. (2019a) Wang, X., Hua, Y., Kodirov, E., Hu, G., Garnier, R., and Robertson, N. M.  Ranked list loss for deep metric learning.  _CVPR_ , 2019a.
  * Wang et al. (2019b) Wang, X., Hua, Y., Kodirov, E., Hu, G., and Robertson, N. M.  Deep metric learning by online soft mining and class-aware attention.  In _AAAI_ , 2019b.
  * Wang et al. (2019c) Wang, X., Kodirov, E., Hua, Y., and Robertson, N. M.  Derivative manipulation for general example weighting.  _arXiv preprint arXiv:1905.11233_ , 2019c.
  * Wang et al. (2019d) Wang, X., Kodirov, E., Hua, Y., and Robertson, N. M.  Id-aware quality for set-based person re-identification.  _arXiv preprint arXiv:1911.09143_ , 2019d.
  * Wang et al. (2019e) Wang, Y., Ma, X., Chen, Z., Luo, Y., Yi, J., and Bailey, J.  Symmetric cross entropy for robust learning with noisy labels.  In _ICCV_ , 2019e.
  * Xiao et al. (2015) Xiao, T., Xia, T., Yang, Y., Huang, C., and Wang, X.  Learning from massive noisy labeled data for image classification.  In _CVPR_ , 2015.
  * Zhang et al. (2017) Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O.  Understanding deep learning requires rethinking generalization.  In _ICLR_ , 2017.
  * Zhang & Sabuncu (2018) Zhang, Z. and Sabuncu, M. R.  Generalized cross entropy loss for training deep neural networks with noisy labels.  In _NeurIPS_ , 2018.
  * Zheng et al. (2016) Zheng, L., Bie, Z., Sun, Y., Wang, J., Su, C., Wang, S., and Tian, Q.  Mars: A video benchmark for large-scale person re-identification.  In _ECCV_ , 2016.

##  1 The Impact of $T$ on Gradient Magnitude’s Variance

Assuming samples’ probabilities are uniformly distributed, we calculate the
variances of IMAE’s weighting curves with different $T$. As illustrated in
Sec. 4 of the main paper, we rewrite the Eq. (13) (We use $e$ to replace
$\exp$ for brevity):

$w_{\mathrm{IMAE}}(p)=e^{T\cdot p(1-p)},$ (17)

where $p$ is the probability of one randomly sampled example being predicted
to its annotated label. According to Eq. (16) in the main paper, we have,

$\displaystyle\sigma_{\mathrm{IMAE}}$
$\displaystyle=\int^{1}_{0}w_{\mathrm{\mathrm{IMAE}}}^{2}(p)\,\mathrm{d}p-(\int^{1}_{0}w_{\mathrm{IMAE}}(p)\,\mathrm{d}p)^{2}$
(18)
$\displaystyle=\int^{1}_{0}\mathrm{e}^{2Tp{}\left(1-p\right)}\,\mathrm{d}p-(\int^{1}_{0}\mathrm{e}^{Tp{}\left(1-p\right)}\,\mathrm{d}p)^{2}$
$\displaystyle={{\sqrt{\pi}\,\mathrm{erf}\left({{\sqrt{2T}}\over{{2}}}\right)\,e^{{{T}\over{2}}}}\over{\sqrt{2T}}}-{{{\pi}\,\mathrm{erf^{2}}\left({{\sqrt{T}}\over{2}}\right)\,e^{{{T}\over{2}}}}\over{{T}}}.$

$\mathrm{erf}$ is the error function. Therefore we obtain the weighting
variances $\sigma_{\mathrm{IMAE}}$ of IMAE with different $T$, as displayed in
Table 4.

Table 4:  The weight variance (gradient magnitude’s variance) of IMAE when $T$ changes.  $T$ | 16 | 8 | 4 | 2 | 1 | 0.5 | 0
---|---|---|---|---|---|---|---
$\sigma_{\mathrm{IMAE}}$ | 354.113 | 4.546 | 0.299 | 0.040 | 0.007 | 0.002 | 0
Figure 5:  Sample’s weight along with probability in IMAE with different $T$
(IMAE-$T$). The hyper-parameter $T$ controls gradient magnitude’s variance,
and impact ratio between examples consequently. Better viewed in colour.

##  2 The Impact of $T$ on Validation Accuracy

We visualise and compare the effect of $T$ on CIFAR-10 test performance. These
experiments follow exactly the same settings of the main paper.

We try two cases: (1). Training labels are intact ($r=0$); (2). Training
labels are corrupted randomly with a probability of 0.4 ($r=40\%$). In both
cases, the test set is kept intact for evaluation. The backbone network is
ResNet20.

###  2.1 CIFAR-10 with intact training labels

The test results are shown and compared in Figure 6(a).

When training labels are clean, it is unhelpful to differentiate training
samples in a high degree, e.g., the performance is even lower when $T=16$. The
final test accuracies are similar when $T$ ranges from 0 to 8.

(a) Test accuracies of IMAE-$T$ trained on intact training labels.

(b) Test accuracies of IMAE-$T$ trained on corrupted training labels.

Figure 6: The accuracy on CIFAR-10 test set along with training iterations. We
display the results when training on intact training set and corrupted
training set. Better viewed in colour.

(a) The training accuracies of IMAE-$T$ on intact training set.

(b) The training accuracies of IMAE-$T$ on corrupted training set.

Figure 7: The accuracy on CIFAR-10 training sets along with training
iterations. We show the results when training on intact training set and
corrupted training set. Better viewed in colour.

###  2.2 CIFAR-10 with corrupted training labels

The results are presented and compared in Figure 6(b). Because there exists
40% label noise, as training goes, the test accuracy drops, which means the
model overfits noisy data gradually.

However, we observe that higher differentiation degree (larger $T$) works
better and is much less susceptible to overfitting to noisy data. In Figure
6(b), the final test accuracies of IMAE-16 and IMAE-8 are much higher than
those of other models.

##  3 The Impact of $T$ on Training Accuracy

Following the practice in the main paper, we also visualise and compare the
accuracies on the training sets, which indicate how different models fit to
training data as training goes, thus leading to different generalisation
performance in the test phase. We present how each model fits its
corresponding training set in Figure 7.

###  3.1 Fitting of intact training set

As compared in Figure 7(a), all models fit training data similarly when $T$
ranges from 0 to 8. However, when $T=16$, the differentiation degree becomes
too large as shown in Table 4. When differentiation degree is too large, only
a quite small proportion of training data can contribute. Consequently,
IMAE-16 underfits training data compared with other models. That is why
IMAE-16 has the worst test performance as shown in Figure 6(a).

###  3.2 Fitting of corrupted training set

The training accuracies of corrupted training set are displayed in Figure
7(b). We have two observations:

  * •

In cases where noise rate is high, as $T$ increases, the fitting of training
data first becomes better, and then becomes worse. Specifically, when $T$
increases from 0 to 8, the training accuracy grows gradually, which means the
fitting of training data becomes better. However, when $T=16$, the weighting
variance becomes very large (Table 4). As a result, IMAE-16’s fitting of
training data becomes much worse than IMAE-8’s.

  * •

Fitting corrupted training data better does not mean better generalisation
performance. On the one hand, although IMAE-16 fits the training data much
worse than IMAE-8 (Figure 7(b)), IMAE-16’s test accuracy is slightly better
than IMAE-8’s (Figure 6(b)). On the other hand, similar to IMAE-8, IMAE-4 fits
its training data well (Figure 7(b)), but IMAE-4’s test performance is much
worse than IMAE-8’s (Figure 6(b)).

##  4 Choosing $T$ in Practice

Figure 8:  IMAE-16’s, IMAE-8’s and IMAE-4’s accuracies on the clean training
set when they are trained on the corrupted training set. The overlap rate
between corrupted and intact training sets is only $(1-r)=60\%$. Therefore, we
can use the original training set as a validation set. Better viewed in
colour.

In summary, the training accuracy (fitting of training data) is uninformative
for estimating a model’s generalisation performance according to our findings
in Section 3.2. Therefore, it is better to optimise $T$ on a validation set in
practice.

For empirical demonstration, since the overlap rate between corrupted and
intact training sets is only $(1-r)=60\%$, we treat the original intact
training set as a validation set. The validation performance of IMAE-16,
IMAE-8 and IMAE-4 is compared in Figure 8. We observe that IMAE-16 and IMAE-8
own similar validation performance, while IMAE-4’s validation accuracy is
lower. Furthermore, their validation performance is consistent with their test
performance (Figure 6(b)). Therefore, we conclude that it is a good practice
to optimise $T$ on a validation set in different cases.

Figure 9:  Display of abnormal training examples highlighted by red boxes. The 1st row shows synthetic abnormal examples from corrupted CIFAR-10 (Krizhevsky, 2009). The 2nd and 3rd rows present realistic abnormal examples from video person re-identification benchmark MARS (Zheng et al., 2016). We remark three representatives: 1) The abnormal images with no person in 3rd row contain no semantic information at all. 2) The last abnormal image in 2nd or 3rd row may contain a person that does not belong to any person in the training set. 3) We cannot decide the object of interest without any prior when an image contains more than one object, e.g., the 2nd and 3rd last images in 2nd row contain two persons. Better viewed in colour. Table 5:  The results of algorithms using different stochastic optimisers on CIFAR-10 with 40% class-independent (symmetric) label noise. The trained network is ResNet56 (He et al., 2016). The key hyper-parameters of all optimisers are shown. Other settings are fixed to be the same as presented in the implementation details of Section 5.3, e.g., weight decay = 0.0001. Since Adam is an adaptive gradient method, we show several variants of it.  |  SGD (lr: 0.01) |  SGD + Momentum (lr: 0.01) |  Nesterov (lr: 0.01) |  Adam (lr: 0.01, delta: 0.1) |  Adam (lr: 0.005, delta: 0.1) |  Adam (lr: 0.005, delta: 1)
---|---|---|---|---|---|---
CCE | 64.3 | 60.6 | 56.4 | 42.5 | 44.5 | 50.3
MAE | 39.3 | 64.7 | 64.1 | 68.2 | 59.9 | 41.4
GCE | 68.8 | 80.5 | 79.7 | 73.2 | 70.6 | 69.3
IMAE | 82.0 | 83.5 | 83.7 | 75.5 | 76.3 | 78.6

##  5 Video person re-identification

Dataset and evaluation settings. MARS contains 20,715 videos of 1,261 persons
(Zheng et al., 2016). There are 1,067,516 frames in total. Because person
videos are collected by tracking and detection algorithms, abnormal examples
exist as shown in Figure 9. The exact noise rate is unknown. Following
standard settings, we use 8,298 videos of 625 persons for training and 12,180
videos of other 636 persons for testing. We report the cumulated matching
characteristics (CMC) and mean average precision (mAP) results.

Implementation details.555We explore the performance of different losses in
real-world applications instead of pushing the state-of-the-art results.
Following (Liu et al., 2017; Wang et al., 2019b), we train GoogleNet V2. We
also treat a video as an image set, which means we use only appearance
information without exploiting latent temporal information. A video’s
representation is simply the average fusion of its frames’ representations. We
apply the same training settings for each loss. The learning rate starts from
0.01 and is divided by 2 every 10k iterations. We stop training at 50k
iterations. We choose SGD optimiser with a weight decay of 0.0005 and momentum
of 0.9. The batch size is set to 180. We use standard data augmentation: a
$227\times 227$ crop is randomly sampled and flipped after resizing an
original image to $256\times 256$. At testing, following (Wang et al., 2019b;
Movshovitz-Attias et al., 2017; Law et al., 2017), we first $L_{2}$ normalise
videos’ features and then calculate the cosine similarity between every two
features.

Results. We compare our method with CCE, MAE and GCE. We implement GCE with
its best settings. The results are shown in Table 6. IMAE outperforms other
related methods by a significant margin.

Table 6:  The retrieval results of CCE, MAE, GCE and IMAE on MARS with GoogLeNet V2 (Ioffe & Szegedy, 2015).  Metric | CCE | MAE | GCE | IMAE
---|---|---|---|---
mAP (%) | 58.1 | 12.0 | 31.6 | 70.9
CMC-1 (%) | 73.8 | 26.0 | 51.5 | 83.5

(a) ResNet20: Testing set (higher is better).

(b) ResNet20: Noisy subset (lower is better).

(c) ResNet20: Clean subset (higher is better).

(d) ResNet56: Testing set (higher is better).

(e) ResNet56: Noisy subset (lower is better).

(f) ResNet56: Clean subset (higher is better).

Figure 10: CIFAR-10 with noise rate $r=80\%$. The accuracies on testing set,
noisy subset and clean subset of training set along with training iterations.
The legend on the top left is shared by all subfigures. Better viewed in
colour.

##  6 The Results of IMAE Using Different Stochastic Optimisers

In this section, we study the performance of IMAE when different stochastic
optimisers are used. The results are presented in Table 5. We observe that
IMAE’s results are the best consistently.

##  7 Derivation of Softmax, CCE and MAE Layers

###  7.1 Derivation of softmax layer

As the softmax layer is shared by CCE and MAE, we present the derivation of
softmax layer first. Based on Eq. (1), we have

$\displaystyle p(y_{i}|\mathbf{x}_{i})^{-1}$ $\displaystyle=1+\sum_{j\neq
y_{i}}\exp(\mathbf{z}_{ij}-\mathbf{z}_{iy_{i}}).$ (19)

If $j=y_{i}$, for left and right sides of Eq. (19), we calculate their
derivatives w.r.t. $\mathbf{z}_{iy_{i}}$ simultaneously:

$\displaystyle\frac{-1}{p(y_{i}|\mathbf{x}_{i})^{2}}\frac{\partial
p(y_{i}|\mathbf{x}_{i})}{\mathbf{z}_{iy_{i}}}=-\sum_{j\neq
y_{i}}\exp(\mathbf{z}_{ij}-\mathbf{z}_{iy_{i}})$ (20)
$\displaystyle=>\frac{\partial
p(y_{i}|\mathbf{x}_{i})}{\mathbf{z}_{iy_{i}}}=p(y_{i}|\mathbf{x}_{i})(1-p(y_{i}|\mathbf{x}_{i})).$

If $j\neq y_{i}$, analogously we have:

$\displaystyle\frac{-1}{p(y_{i}|\mathbf{x}_{i})^{2}}\frac{\partial
p(y_{i}|\mathbf{x}_{i})}{\mathbf{z}_{ij}}=\exp(\mathbf{z}_{ij}-\mathbf{z}_{iy_{i}})$
(21) $\displaystyle=>\frac{\partial
p(y_{i}|\mathbf{x}_{i})}{\mathbf{z}_{ij}}=-p(y_{i}|\mathbf{x}_{i})p(j|\mathbf{x}_{i}).$

In summary, the derivation of softmax layer is:

$\displaystyle\frac{\partial
p(y_{i}|\mathbf{x}_{i})}{\partial\mathbf{z}_{ij}}$
$\displaystyle=\begin{cases}p(y_{i}|\mathbf{x}_{i})(1-p(y_{i}|\mathbf{x}_{i}))\text{,
}&j=y_{i}\\\ -p(y_{i}|\mathbf{x}_{i})p(j|\mathbf{x}_{i})\text{, }&j\neq
y_{i}\end{cases}$ (22)

###  7.2 Derivation of loss layer: CCE

According to Eq. (2), we have

$\displaystyle L_{\mathrm{CCE}}(\mathbf{x}_{i};f_{\theta},\mathbf{W})$
$\displaystyle=-\log p(y_{i}|\mathbf{x}_{i}).$ (23)

Therefore, we obtain (the parameters are omitted for brevity),

$\displaystyle\frac{\partial L_{\mathrm{CCE}}(\mathbf{x}_{i})}{\partial
p(j|\mathbf{x}_{i})}$
$\displaystyle=\begin{cases}-p(y_{i}|\mathbf{x}_{i})^{-1}\text{, }&j=y_{i}\\\
0\text{, }&j\neq y_{i}\end{cases}.$ (24)

###  7.3 Derivation of loss layer: MAE

According to Eq. (3), we have

$\displaystyle L_{\mathrm{MAE}}(\mathbf{x}_{i};f_{\theta},\mathbf{W})$
$\displaystyle=2(1-(p(y_{i}|\mathbf{x}_{i})).$ (25)

Therefore, we obtain

$\displaystyle\frac{\partial L_{\mathrm{MAE}}(\mathbf{x}_{i})}{\partial
p(j|\mathbf{x}_{i})}$ $\displaystyle=\begin{cases}-2\text{, }&j=y_{i}\\\
0\text{, }&j\neq y_{i}\end{cases}.$ (26)

###  7.4 Derivatives w.r.t. $\mathbf{z}_{i}$

$\partial L_{\mathrm{CCE}}(\mathbf{x}_{i})/\partial\mathbf{z}_{i}$: The
calculation is based on Eq. (24) and Eq. (22).

If $j=y_{i}$, we have:

$\displaystyle\frac{\partial
L_{\mathrm{CCE}}(\mathbf{x}_{i})}{\partial\mathbf{z}_{iy_{i}}}$
$\displaystyle=\sum_{j=1}^{C}\frac{\partial
L_{\mathrm{CCE}}(\mathbf{x}_{i})}{\partial p(j|\mathbf{x}_{i})}\frac{\partial
p(y_{i}|\mathbf{x}_{i})}{\mathbf{z}_{iy_{i}}}$ (27)
$\displaystyle=p(y_{i}|\mathbf{x}_{i})-1.$

If $j\neq y_{i}$, it becomes:

$\displaystyle\frac{\partial
L_{\mathrm{CCE}}(\mathbf{x}_{i})}{\partial\mathbf{z}_{ij}}$
$\displaystyle=\sum_{j=1}^{C}\frac{\partial
L_{\mathrm{CCE}}(\mathbf{x}_{i})}{\partial p(j|\mathbf{x}_{i})}\frac{\partial
p(y_{i}|\mathbf{x}_{i})}{\mathbf{z}_{ij}}$ (28)
$\displaystyle=p(j|\mathbf{x}_{i}).$

In summary, $\partial L_{\mathrm{CCE}}(\mathbf{x}_{i})/\partial\mathbf{z}_{i}$
can be represented as:

$\displaystyle\frac{\partial
L_{\mathrm{CCE}}(\mathbf{x}_{i})}{\partial\mathbf{z}_{ij}}$
$\displaystyle=\begin{cases}p(y_{i}|\mathbf{x}_{i})-1\text{, }&j=y_{i}\\\
p(j|\mathbf{x}_{i})\text{, }&j\neq y_{i}\end{cases}.$ (29)

$\partial L_{\mathrm{MAE}}(\mathbf{x}_{i})/\partial\mathbf{z}_{i}$: The
calculation is analogous with that of $\partial
L_{\mathrm{CCE}}(\mathbf{x}_{i})/\partial\mathbf{z}_{i}$. According to Eq.
(26) and Eq. (22), if $j=y_{i}$:

$\displaystyle\frac{\partial
L_{\mathrm{MAE}}(\mathbf{x}_{i})}{\partial\mathbf{z}_{iy_{i}}}$
$\displaystyle=\sum_{j=1}^{C}\frac{\partial
L_{\mathrm{MAE}}(\mathbf{x}_{i})}{\partial p(j|\mathbf{x}_{i})}\frac{\partial
p(y_{i}|\mathbf{x}_{i})}{\mathbf{z}_{iy_{i}}}$ (30)
$\displaystyle=-2p(y_{i}|\mathbf{x}_{i})(1-p(y_{i}|\mathbf{x}_{i})).$

otherwise ($j\neq y_{i}$):

$\displaystyle\frac{\partial
L_{\mathrm{MAE}}(\mathbf{x}_{i})}{\partial\mathbf{z}_{ij}}$
$\displaystyle=\sum_{j=1}^{C}\frac{\partial
L_{\mathrm{MAE}}(\mathbf{x}_{i})}{\partial p(j|\mathbf{x}_{i})}\frac{\partial
p(y_{i}|\mathbf{x}_{i})}{\mathbf{z}_{ij}}$ (31)
$\displaystyle=2p(y_{i}|\mathbf{x}_{i})p(j|\mathbf{x}_{i}).$

In summary, $\partial L_{\mathrm{MAE}}(\mathbf{x}_{i})/\partial\mathbf{z}_{i}$
is:

$\displaystyle\frac{\partial
L_{\mathrm{MAE}}(\mathbf{x}_{i})}{\partial\mathbf{z}_{ij}}$
$\displaystyle=\begin{cases}2p(y_{i}|\mathbf{x}_{i})(p(y_{i}|\mathbf{x}_{i})-1)\text{,
}&j=y_{i}\\\ 2p(y_{i}|\mathbf{x}_{i})p(j|\mathbf{x}_{i})\text{, }&j\neq
y_{i}\end{cases}.$ (32)


uart- | u | g | 0.02 | 0.25 | 0.30 | 0.18 | 0.07 | 0.10 | 0.02 | 0.02 | 0.02 | 0.02
uart- | u | w | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
uasī | u | b | 0.81 | 0.05 | 0.05 | 0.02 | 0.01 | 0.04 | 0.00 | 0.00 | 0.00 | 0.00
uasī | u | w | 0.62 | 0.11 | 0.12 | 0.08 | 0.03 | 0.01 | 0.01 | 0.01 | 0.01 | 0.00
uat-caka | u | b | 0.63 | 0.08 | 0.14 | 0.06 | 0.03 | 0.04 | 0.01 | 0.01 | 0.01 | 0.01
uat-caka | u | w | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
uata | u | b | 0.03 | 0.22 | 0.32 | 0.16 | 0.08 | 0.12 | 0.02 | 0.02 | 0.02 | 0.02
uata | u | w | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
uićati | u | b | 0.01 | 0.23 | 0.34 | 0.17 | 0.08 | 0.09 | 0.02 | 0.02 | 0.02 | 0.02
uićati | u | g | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
uićati | u | w | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
uiiapāna | u | b | 0.05 | 0.24 | 0.26 | 0.14 | 0.07 | 0.17 | 0.02 | 0.02 | 0.02 | 0.01
uiiapāna | u | g | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
uiiapāna | u | w | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
uicāra | u | g | 0.69 | 0.08 | 0.09 | 0.05 | 0.03 | 0.05 | 0.01 | 0.01 | 0.01 | 0.00
uidāna | u | b | 0.57 | 0.10 | 0.10 | 0.08 | 0.03 | 0.08 | 0.01 | 0.01 | 0.01 | 0.01
uidāna | u | g | 0.98 | 0.00 | 0.02 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
uidāna | u | w | 0.90 | 0.00 | 0.10 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
uidauā | u | b | 0.02 | 0.24 | 0.33 | 0.17 | 0.09 | 0.08 | 0.02 | 0.02 | 0.02 | 0.01
uidauā | u | w | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
uināća | ć | h | 0.74 | 0.06 | 0.09 | 0.04 | 0.02 | 0.03 | 0.01 | 0.00 | 0.00 | 0.00
uināća | u | g | 0.02 | 0.22 | 0.34 | 0.15 | 0.08 | 0.12 | 0.02 | 0.02 | 0.02 | 0.01
uināća | u | w | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
urinji | u | b | 0.72 | 0.07 | 0.09 | 0.05 | 0.02 | 0.02 | 0.01 | 0.00 | 0.00 | 0.00
urinji | u | g | 0.81 | 0.05 | 0.05 | 0.02 | 0.01 | 0.04 | 0.00 | 0.00 | 0.00 | 0.00
aćua | ću | s | 0.54 | 0.11 | 0.14 | 0.09 | 0.02 | 0.07 | 0.01 | 0.01 | 0.01 | 0.01
aćua | ću | sp | 0.71 | 0.07 | 0.10 | 0.05 | 0.02 | 0.02 | 0.01 | 0.01 | 0.00 | 0.00
aćru | ćr | rs | 0.02 | 0.24 | 0.28 | 0.17 | 0.08 | 0.14 | 0.02 | 0.02 | 0.02 | 0.02
aćru | ćr | sr | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
aćru | pro | no pro | 0.26 | 0.18 | 0.21 | 0.12 | 0.06 | 0.12 | 0.02 | 0.01 | 0.01 | 0.01
aćru | pro | pro | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
arša | pro | no pro | 0.00 | 0.18 | 0.50 | 0.12 | 0.10 | 0.06 | 0.01 | 0.01 | 0.01 | 0.01
arša | pro | pro | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
arša | rs | š | 0.67 | 0.10 | 0.06 | 0.08 | 0.05 | 0.01 | 0.01 | 0.01 | 0.01 | 0.01
arša | rs | rch | 0.43 | 0.00 | 0.57 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
arša | rs | rs | 0.63 | 0.00 | 0.37 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
brzant | rj́ | l | 0.58 | 0.11 | 0.15 | 0.08 | 0.04 | 0.02 | 0.01 | 0.01 | 0.01 | 0.01
brzant | rj́ | rj́ | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
barda | rd | l | 0.02 | 0.23 | 0.35 | 0.16 | 0.08 | 0.10 | 0.02 | 0.02 | 0.01 | 0.01
barda | rd | rd | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
barzād | rj́ | l | 0.68 | 0.08 | 0.12 | 0.05 | 0.03 | 0.02 | 0.01 | 0.01 | 0.00 | 0.00
barzād | rj́ | rj́ | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
caurćat | rć | l | 0.00 | 0.25 | 0.34 | 0.18 | 0.08 | 0.07 | 0.03 | 0.02 | 0.02 | 0.02
caurćat | rć | rć | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
daća | ć | h | 0.71 | 0.07 | 0.10 | 0.05 | 0.02 | 0.02 | 0.01 | 0.01 | 0.00 | 0.00
daća | ć | s | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
darn | rn | (r)n | 0.03 | 0.20 | 0.41 | 0.11 | 0.09 | 0.12 | 0.01 | 0.01 | 0.01 | 0.01
darn | rn | (r)r | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
g(a)rna-ka | rn | (r)n | 0.02 | 0.15 | 0.53 | 0.10 | 0.11 | 0.06 | 0.01 | 0.01 | 0.01 | 0.01
g(a)rna-ka | rn | l | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
gau-ćuanta- | ću | s | 0.82 | 0.05 | 0.05 | 0.04 | 0.01 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
gau-ćuanta- | ću | sp | 0.76 | 0.06 | 0.08 | 0.04 | 0.02 | 0.02 | 0.01 | 0.00 | 0.00 | 0.00
hij́uāna | j́u | zb | 0.01 | 0.25 | 0.32 | 0.18 | 0.06 | 0.10 | 0.03 | 0.02 | 0.02 | 0.02
hij́uāna | j́u | zm | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
hij́uāna | j́u | zw | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
krmi | r | l | 0.58 | 0.11 | 0.11 | 0.08 | 0.02 | 0.06 | 0.01 | 0.01 | 0.01 | 0.01
krmi | r | r | 0.69 | 0.08 | 0.11 | 0.06 | 0.02 | 0.02 | 0.01 | 0.01 | 0.00 | 0.00
kaćiapa | ći | š | 0.03 | 0.24 | 0.28 | 0.15 | 0.06 | 0.18 | 0.02 | 0.02 | 0.01 | 0.01
kaćiapa | ći | s | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
laupāća | ć | h | 0.64 | 0.09 | 0.14 | 0.06 | 0.03 | 0.02 | 0.01 | 0.01 | 0.01 | 0.01
laupāća | ć | s | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
laupāća | l | l | 0.59 | 0.11 | 0.14 | 0.07 | 0.04 | 0.02 | 0.01 | 0.01 | 0.01 | 0.01
laupāća | l | r | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
mušti | št | št | 0.53 | 0.12 | 0.17 | 0.08 | 0.04 | 0.03 | 0.01 | 0.01 | 0.01 | 0.01
mušti | št | st | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
prtu | rt | l | 0.58 | 0.13 | 0.11 | 0.09 | 0.04 | 0.03 | 0.01 | 0.01 | 0.01 | 0.01
prtu | rt | rt | 0.91 | 0.00 | 0.09 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
parna | rn | (r)n | 0.01 | 0.22 | 0.29 | 0.16 | 0.12 | 0.13 | 0.02 | 0.02 | 0.02 | 0.02
parna | rn | (r)r | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
parna | rn | l | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
pura | tr | s | 0.70 | 0.08 | 0.09 | 0.05 | 0.02 | 0.04 | 0.01 | 0.01 | 0.01 | 0.00
pura | tr | tr | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
rāšta | št | št | 0.00 | 0.25 | 0.38 | 0.18 | 0.10 | 0.02 | 0.03 | 0.02 | 0.02 | 0.01
rāšta | št | st | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
skar | r | l | 0.09 | 0.20 | 0.24 | 0.16 | 0.12 | 0.11 | 0.02 | 0.02 | 0.02 | 0.02
skar | r | r | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
sprzan | rj́ | l | 0.37 | 0.14 | 0.22 | 0.12 | 0.05 | 0.05 | 0.01 | 0.01 | 0.01 | 0.01
sprzan | rj́ | rj́ | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
xšuifta- | t | r | 0.62 | 0.11 | 0.13 | 0.08 | 0.03 | 0.01 | 0.01 | 0.01 | 0.01 | 0.00
xšuifta- | t | t | 0.57 | 0.13 | 0.14 | 0.08 | 0.04 | 0.01 | 0.01 | 0.01 | 0.01 | 0.01

### Posterior distributions over dialect components for sound change
instances, raw values

https://github.com/chundrac/w_ir_layers/blob/master/p_z_all gives
$P(z_{i}|\boldsymbol{\theta},\boldsymbol{\phi})$ for every sound change
instance in our data set.