# Rényi Divergence Deep Mutual Learning

Weipeng Fuzzy Huang Security Big Data Lab, Tencent Inc., China. {fuzzyhuang,
changbodeng, johnnywan<EMAIL_ADDRESS>Junjie Tao School of Software
Engineering, Xi’an Jiaotong University, China<EMAIL_ADDRESS><EMAIL_ADDRESS>Changbo Deng11footnotemark: 1 Ming Fan22footnotemark:
2 Wenqiang Wan11footnotemark: 1 Qi Xiong11footnotemark: 1 Guangyuan Piao
Department of Computer Science, Maynooth University, Ireland.
<EMAIL_ADDRESS>
###### Abstract

This paper revisits an incredibly simple yet exceedingly effective computing
paradigm, Deep Mutual Learning (DML). We observe that the effectiveness
correlates highly to its excellent generalization quality. In the paper, we
interpret the performance improvement with DML from a novel perspective that
it is roughly an approximate Bayesian posterior sampling procedure. This also
establishes the foundation for applying the Rényi divergence to improve the
original DML, as it brings in the variance control of the prior (in the
context of DML). Therefore, we propose Rényi Divergence Deep Mutual Learning
(RDML). Our empirical results represent the advantage of the marriage of DML
and the Rényi divergence. The flexible control imposed by the Rényi divergence
is able to further improve DML to learn better generalized models.

##  1 Introduction

A compelling virtue of certain machine learning techniques is that they show
strong connections to realworld phenomena. One member among these techniques
is Deep Mutual Learning (DML). It is an empirically very powerful computing
paradigm even though it is conceptually simple [34]. The metaphor for DML is
simply that, the learning procedure organizes a class of students to learn
knowledge not only from the ground truth, but also from their peers.
Intuitively, this learning paradigm encourages the students to learn others’
strengths. The students therefore perform better than learning knowledge only
from the teacher. A possible reason is that this paradigm protects the
students from pushing their understanding of certain knowledge to an extreme,
as the peers’ knowledge can regulate the extreme understanding. After all, DML
is a fine paradigm, as that simplicity is better than complexity—as suggested
by Occam’s razor.

Interestingly, even when each student uses the same network architecture (but
with distinct weight initializations), each single model still benefits from
the paradigm and outperforms itself [34]. This is reasonable as initialization
plays a crucial role in Deep Learning (DL) optimizations because some initial
points can lead to wider optima than others, in the highly non-convex
optimization tasks [3]. As regards of pretrained models, the dropout mechanism
randomly dropping a proportion of parameter values [25] maintains the
diversity. In other words, we are able to generate sufficient diversity among
a number of instances of the identical architectures, in most cases.

Despite the empirical efficiency, the question for why DML really works
remains open to discussion. One way to look at this paradigm is from the
perspective of Knowledge Distillation (KD). In fact, KD is a methodology
designed for distilling knowledge from an ensemble of large fine trained
networks into a smaller network so that the small network can obtain a close
performance but be more computationally efficient [9]. DML is an online KD
method where every model starts from the same stage (with diverse weight
initializations) and shares knowledge from the very beginning [6]. Neither
offline nor online KD has been analyzed with a thorough theoretical study. The
work [20] provided a preliminary theoretical justification on KD; however, it
is limited to basic linear models. Nonetheless, this is expected as DL is
inherently challenging for theoretical analysis.

DML is remarkable in constraining the generalization errors and hence nicely
protects the learned models from overfitting [18]. Heuristically, it helps the
students find wider local optima [34], since the students share diversity with
others and avoid being optimized in only very few directions. In this paper,
we interpret that DML is approximately equivalent to the process of a Bayesian
posterior sampling. Bayesian posterior sampling supports good generalization,
since it focuses on the posterior distribution (i.e., the distribution of the
model parameters) rather than estimating the parameters via _Maximum
Likelihood Estimation_ approach. This implies that a good amount of
uncertainty is well maintained in the entire model. On top of this
observation, we propose to replace the Kullback–Leibler (KL) divergence for
learning the differences from others by the Rényi divergence [4], which can be
thought of as a superclass of KL divergence. It allows us to further tune the
variance of the priors approximated by DML. Finally, we coin this paradigm
Rényi Divergence Deep Mutual Learning (RDML).

Our contributions are summarized here. We connect the DML and the Rényi
divergence to the Bayesian learning for interpreting their working machinery
in Section 4. In the same section, we propose to replace the KL divergence by
the Rényi divergence which is more controllable and tunable to further improve
the model performance. Moreover, we empirically show that the Rényi divergence
can further improve the model performance and achieve better generalizations
in Section 5. Finally, the codebase is available at
http://github.com/parklize/rdml.

##  2 Related Work

In this section, we review related work, with emphasis on DML and the Rényi
divergence.

DML is one of the main knowledge distillation schemes which distill knowledge
from a deep neural network into another network. In contrast to offline
distillation in which knowledge is transferred from a pre-trained model into
another model, DML allows multiple neural networks to transfer knowledge in a
collaborative manner during the training process and provides flexibility to
train different networks as well as the same networks [6]. Due to its
effectiveness, DML has been used in a wide variety of contexts and
applications such as tracking visual objects [35], machine translation [36],
speech recognition [15], and COVID19 recognition [31]. For example, MutualNet
[30] has been proposed to train a cohort of sub-(convolutional) networks with
different configurations of network widths and input resolutions via DML to
achieve adaptive accuracy-efficiency tradeoffs. Park et al. [18] have extended
and applied DML in the context of deep metric learning beyond classification
tasks and showed the effectiveness of DML over individual models.

The Rényi divergence [4] has drawn growing attention for the community and has
been employed to more applications. For instance, the Rényi divergence has
been employed to replace KL divergence in variational inference [13] wherein
examples show that the Rényi divergence can control the regularization power
of the inferred variational distribution. There are also a number of related
applications in cryptography [1, 21] as the Rényi divergence is able to
provide tighter security bounds. Recently, the Rényi divergence is also
applicable in analyzing human beings’ brain behaviors [23]. These have
illustrated the potential of this divergence in a broad range of tasks. Our
work proposes to employ the Rényi divergence instead of KL divergence in the
DML framework to achieve improvements through greater flexibility.

##  3 Deep Mutual Learning

Before proceeding to explain the frameworks, we first introduce the common
notation which will be consistent throughout the paper. Considering a data
size $N$, let us define the collection of data
$\mathcal{D}=\\{(x_{n},y_{n})\\}_{1\leq n\leq N}$ where $x_{n}$ is the $n$th
datum and $y_{n}$ is the corresponding ground truth. In addition, we write
$\mathbf{x}=\\{x_{n}\\}_{1\leq n\leq N}$ and $\mathbf{y}=\\{y_{n}\\}_{1\leq
n\leq N}$. Let $\operatorname{\mathbb{KL}}(\cdot)$ denote the KL divergence
and $D_{\alpha}(\cdot)$ denote the Rényi divergence. If an arbitrary random
variable $X$ is distributed by distribution $p(X)$, it is written as $X\sim
p(X)$. Furthermore, $\simeq$ reads “be approximately equivalent to”.
Meanwhile, we denote by $\bm{\theta}$ the model parameters in any generic
model. Let $K$ be the number of models in the DML paradigm. Finally, let
$\beta$ denote the learning rate in the optimization techniques.

In the remainder of the section, we first introduce the vanilla DML framework
and later demonstrate the DML with the Rényi divergence.

###  3.1 Vanilla DML

Imagine that there are a cohort of students learning a task together. It
consists of a base loss and a divergence loss for every single student.
Specifically, the DML loss $\operatorname{\mathcal{L}}^{k}_{dml}$ for student
$k$ is defined by

(3.1)
$\displaystyle\operatorname{\mathcal{L}}_{dml}^{k}\coloneqq\operatorname{\mathcal{L}}^{k}_{base}+\operatorname{\mathcal{L}}^{k}_{div}\,,$

where $\operatorname{\mathcal{L}}_{base}^{k}$ is the loss objective of the
input data and the ground truth, and $\operatorname{\mathcal{L}}_{div}^{k}$ is
the divergence loss from this student to others. In general, the base loss is
selected depending on the task and the data, e.g., a cross entropy for
classification. For example, in a multiclass classification task containing
$M$ classes, the base loss is

(3.2)
$\displaystyle\operatorname{\mathcal{L}}_{base}^{k}(\mathbf{y};\mathbf{x})=-\frac{1}{N}\sum_{n=1}^{N}\sum_{m=1}^{M}\mathbbm{1}(y_{n}=m)\log
p^{(k)}_{nm}$

where $p_{nm}^{(k)}$ denotes the probability of the $n$th datum belonging to
class $m$ with regard to model $k$111We use “$(k)$” instead of “$k$” in the
superscript for the index of model when there is confusion between exponent
and itself.

The divergence loss $\operatorname{\mathcal{L}}_{div}^{k}$ is based on the
predictive probability distribution $P^{k}$ for each student $k$. In the
vanilla DML, the KL divergence is applied to compute the distance from $P^{k}$
to $P^{k^{\prime}}$ such that

(3.3)
$\displaystyle\operatorname{\mathcal{L}}_{div}^{k}=\frac{1}{K-1}\sum_{k^{\prime}\neq
k}\operatorname{\mathbb{KL}}(P^{k^{\prime}}~{}||~{}P^{k})\,,$

which indicates that the model $k$ will be calibrated by other models.
Concerning the formalism defined above, the learning process is sketched in
Algorithm 1. It iterates through each student and optimizes the parameters
upon the corresponding DML loss. Although we exemplify it using
classification, we emphasize that this framework can be trivially extended to
a diversity of machine learning tasks.

Algorithm 1 Learning Procedure of DML

1:$\bm{\theta}_{k}$: the parameters for student $k$

2:while not converged do

3: for $k=1,\dots,K$ do

4:
$\bm{\theta}_{k}\leftarrow\bm{\theta}_{k}-\beta\nabla_{\bm{\theta}_{k}}\operatorname{\mathcal{L}}^{k}_{dml}$

###  3.2 DML with the Rényi Divergence

The Rényi divergence, as a superclass of the KL divergence, introduces more
flexibility of managing the divergence value. In DML, it facilitates the
control of the degree of effort that one student should spend on learning from
others. We propose to replace the KL divergence in
$\operatorname{\mathcal{L}}_{div}^{k}$ in DML to acquire a more controllable
computation paradigm, which we name RDML. Next, we discuss on the Rényi
divergence.

####  3.2.1 Rényi divergence

The divergence is a statistical distance quantity with a controlling parameter
$\alpha\in[0,1)\cup(1,\infty)$ and thus is also called $\alpha$-divergence. It
measures the distribution distance from $Q$ to $P$, which is defined by

(3.4) $\displaystyle D_{\alpha}(P||Q)$
$\displaystyle=\frac{1}{\alpha-1}\log\int
p(\mu)^{\alpha}q(\mu)^{1-\alpha}d\mu\,.$

Setting $\alpha=1$ is invalid according to the above definition. Same as the
KL divergence, the Rényi divergence is asymmetric and hence is in general not
a metric [4]. The Rényi divergence covers a family of statistical distances.
For instance, $\alpha=0.5$ leads the Rényi divergence to the squared Hellinger
divergence and $\alpha\to 1$ leads to the KL divergence, etc. [13].

Furthermore, the following essential remarks have been proved in [4] (however,
we still provide two simpler proofs in the supplemental material for a quick
verification of these two facts).

###### Remark 1

For any distribution $P$, $Q$, and $\alpha\in[0,1)\cup(1,\infty)$, the Rényi
divergence $D_{\alpha}(P||Q)\geq 0$. The equality holds if and only if
$P\equiv Q$.

###### Remark 2

Given $\alpha\in[0,1)\cup(1,\infty)$, the Rényi divergence $D_{\alpha}(P||Q)$
is non-decreasing in $\alpha$, for any distribution $P$ and $Q$.

The first remark fixes the lower bound of the divergence. This property
ensures $\operatorname{\mathcal{L}}_{dml}$ to be non-negative, which is always
a desired property for the loss function. Remark 2 indicates that $\alpha$
controls the distance values when $P$ and $Q$ are fixed. In the context of
DML, a larger $\alpha$ hence pushes the students to learn more from the peers
considering that the gradients for updating the parameters become larger.

Figure 1 illustrates an example of the $D_{\alpha}(P||Q)$, for various
$\alpha$ with fixed $P$ and $Q$ respectively. In the example, the Rényi
divergence with $\alpha\to 1$ is the KL divergence. We observe that the
correlation between $D_{\alpha}(P||Q)$ and $\alpha$ perfectly embodies Remark
2. On the other hand, the case of focusing on $D_{\alpha}(P||Q)$ with fixed
$P$, spans a much greater range. The divergence is more sensitive for $P\neq
Q$ in this scenario with sufficiently large $\alpha$. That said, the gap
between $\alpha=10$ and $\alpha=100$ is smaller, which implies that the
divergence growth will become much slower as $\alpha$ increases.

(a)  (b)

Figure 1:  Example plots of the Rényi divergence for distributions containing
two events. The orange dashed line is for the KL divergence in both plots. The
first plot fixes distribution $Q$ to $(0.4,0.6)$ and shows the divergence
change over $p$ and $1-p$. The second plot fixes $P=(0.4,0.6)$ and shows the
divergence change over $q$ and $1-q$. Note that, either $q=0$ or $q=1$ obtains
the divergence value $\infty$ for all possible $\alpha$. Since infinity is
technically not drawable, the x-axis in the second plot ranges from $0.001$ to
$0.999$.  Figure 2: Heatmap of the best $\alpha$ values of 18 experiments in
Section 5 where each experiment is a unique combination of a method $M$ and a
dataset $D$. A method with different datasets (and vice versa) will require
different $\alpha$ to perform better.

As shown, $\alpha$ introduces the flexibility of controlling the degree any
student learns from others in the context of DML. The degree that a student
should learn from her peers depends on the situation. This can also be noticed
from Figure 2 in which the best $\alpha$ values vary across 18 experiments
depending on the method and dataset in each experiment in Section 5. Analogous
to those regularized approaches, the model performance always benefits from a
better tuned regularization power, i.e., the coefficient for the
regularization part.

##  4 Why DML and the Rényi Divergence Work

In our findings, the major reason for why DML works is that DML better
mitigates the overfitting for each single model and thus obtains better
generalization for training the models. A general reason for causing
overfitting is the ignorance to uncertainty [14]. The advantage of the DML
framework lies on its ability to generate good perturbations during the
parameter updating phase. Hence, a reasonable amount of uncertainty is
incorporated in the training procedure.

In the sequel, we provide a Bayesian interpretation of DML. To keep our
interpretation sufficiently precise, we restrict that all students use the
same architecture.

###  4.1 A Bayesian Interpretation of DML and the Rényi Divergence

To summarize, a gradient-based parameter updating step in DML is approximately
equivalent to Bayesian posterior sampling, even though only a small batch of
data is drawn to update the parameters at each timestamp in this sort of
approach. The parameter $\alpha$ in the Rényi divergence acts as a control to
indirectly manage the prior variance.

Stochastic Gradient Descent (SGD) and its variants are usually not preferred
in inferring Bayesian models, as they will converge to the _Maximum a
Posteriori_ rather than the true posterior. This is partly due to the fact
that the parameter set of a Bayesian model is in non-Euclidean geometric space
[5]. That being said, SGD still strongly correlates to the Markov Chain Monte
Carlo (MCMC) algorithms, the standard approach for inferring the Bayesian
models. For example, Stochastic Gradient Langevin Dynamics [28] is a gradient-
based method applying the Langevin Dynamics mechanism to inject proper noise
to ensure the parameters converge to that of the true posterior. Most
recently, Mingard et al. [16] show that with some neural network
architectures, the SGD-obtained posterior probability
$p_{S}(\bm{\theta}|\mathcal{D})$ correlates extraordinarily to the Bayesian-
obtained posterior probability $p_{B}(\bm{\theta}|\mathcal{D})$, so that SGD
can be viewed as a Bayesian sampler in certain circumstances. Interestingly
but unsurprisingly, this result also holds with other gradient-based
algorithms including Adam, AdaGrad, and RMSprop, etc. [16]. Therefore, noting
that posterior $p(\bm{\theta}|\mathcal{D})$ follows
$p(\bm{\theta}|\mathcal{D})\propto
p(\bm{\theta},\mathcal{D})=p(\mathcal{D}|\bm{\theta})p(\bm{\theta})$, we set
up the following assumption in order to verify our interpretation later.

###### Assumption 1

When SGD is applied to minimize the negative joint probability of data and
parameters, such that $\bm{\theta}=\bm{\theta}-\beta\nabla_{\bm{\theta}}(-\log
p(\bm{\theta},\mathcal{D}))=\bm{\theta}+\beta\nabla_{\bm{\theta}}\log
p(\bm{\theta},\mathcal{D})$, it is approximately equivalent to ($\simeq$) that
$\bm{\theta}\sim p(\bm{\theta}|\mathcal{D})$.

###  4.2 Interpreting DML

A neural network can considerably be regarded as a blackbox function
approximating a hard-to-define distribution. Let us follow [11] to formulate
$f(\mathbf{x},\bm{\theta})$ a neural network where $\bm{\theta}$ is the
parameter set in the network, where $P=f(\mathbf{x},\bm{\theta})$ are the
approximated probabilities for certain tasks. Taking multiclass classification
as an example again, we write
$p(\mathbf{y}|\mathbf{x},\bm{\theta})=\operatorname{\mathrm{Categorical}}(\mathbf{y};P)$
such that the label assignment $\mathbf{y}$ is thought to be sampled from a
categorical distribution upon distribution $P$. The negative logarithm of this
likelihood function is identical to the cross entropy loss, i.e.,

(4.5) $\displaystyle\operatorname{\mathcal{L}}_{base}(\mathbf{y};P)=-\log
p(\mathbf{y}|\mathbf{x},\bm{\theta})\,.$

Minimizing the base loss is in effect to maximize the log likelihood.

Similar to Bayesian linear regression, $\mathbf{x}$ is the known covariate and
thus a constant. If $\bm{\theta}$ is a random variable then $P\sim
f(\mathbf{x},\bm{\theta})$, where $P^{k}=f(\mathbf{x},\bm{\theta}_{k})$ and
$\bm{\theta}_{k}$ is sampled from a hard-to-define prior $p(\bm{\theta})$. In
(R)DML, we have

$\displaystyle\nabla_{\bm{\theta}_{k}}\operatorname{\mathcal{L}}_{div}^{k}$
$\displaystyle=\frac{1}{K-1}\sum_{k^{\prime}\neq
k}\nabla_{\bm{\theta}_{k}}D_{\alpha}[f(\mathbf{x},\bm{\theta}_{k^{\prime}})||f(\mathbf{x},\bm{\theta}_{k})]\,.$

Considering that $\mathbf{x}$ is constant, the randomness of
$f(\mathbf{x},\bm{\theta}_{k})$ is completely decided by $\bm{\theta}$. Using
a proper learning rate, the minimization step moves $\bm{\theta}_{k}$ towards
every $\bm{\theta}_{k^{\prime}}$ with a reasonably small pace, in the
probability space. While temporarily ignoring
$\operatorname{\mathcal{L}}_{base}^{k}$, the SGD step related to
$\operatorname{\mathcal{L}}_{div}^{k}$ simulates a stochastic process for
moving each $\bm{\theta}_{k}$ reasonably closer to the true mean
${\bm{\theta}}^{*}$ (in a non-Euclidean space). The group of $\bm{\theta}_{k}$
are then random scatters around the true mean $\bm{\theta}^{*}$. Hence, we
regard $\bm{\theta}_{k}\sim p(\bm{\theta}_{k}|\bm{\theta}^{*},\eta)$ where
$\eta$ is the parameter for the unknown prior. As the prior is unknown, it is
challenging to decide $\eta$ and therefore an empirical approach for
simulating this prior is well suited in the scenario. Hence, omitting the
hyperparameter set $(\bm{\theta}^{*},\eta)$, we obtain

(4.6)
$\displaystyle\nabla_{\bm{\theta}_{k}}\operatorname{\mathcal{L}}_{div}^{k}\quad\simeq\quad\nabla_{\bm{\theta}_{k}}[-\log
p(\bm{\theta}_{k})]\,$

as moving towards the prior mean should increase the prior probability
$p(\bm{\theta}_{k}|\bm{\theta}^{*},\eta)$ in most cases. This approach relies
on the empirical information for updating the information in the prior.

Figure 3: The illustration of $\bm{\theta}_{k}$’s movements in DML. Curly
arrows are used to emphasize that the movements are nonlinear. In this case,
$\bm{\theta}_{4}$ approaches the true $\bm{\theta}^{*}$ by moving towards
other peers. Besides its movement towards the true $\bm{\theta}^{*}$, its
update will still be influenced by the likelihood.

DML considers both the likelihood (Equation 4.5) and prior part (Equation
4.6), such that

$\displaystyle\bm{\theta}_{k}$
$\displaystyle=\bm{\theta}_{k}-\beta\left(\nabla_{\bm{\theta}_{k}}\operatorname{\mathcal{L}}^{k}_{base}+\nabla_{\bm{\theta}_{k}}\operatorname{\mathcal{L}}_{div}^{k}\right)$
$\displaystyle\simeq\bm{\theta}_{k}-\beta\nabla_{\bm{\theta}_{k}}\left(-\log
p(\mathbf{y}|\mathbf{x},\bm{\theta}_{k})-\log p(\bm{\theta}_{k})\right)$
$\displaystyle=\bm{\theta}_{k}+\beta\nabla_{\bm{\theta}_{k}}\log
p(\mathbf{y},\mathbf{x},\bm{\theta}_{k})=\bm{\theta}_{k}+\beta\nabla_{\bm{\theta}_{k}}\log
p(\mathcal{D},\bm{\theta}_{k})$

where the last line comes from that $p(\mathbf{x})$ is constant. According to
1, we arrive at

$\displaystyle\bm{\theta}_{k}=\bm{\theta}_{k}-\beta\nabla_{\bm{\theta}_{k}}\operatorname{\mathcal{L}}^{k}_{dml}\quad\simeq\quad\bm{\theta}_{k}$
$\displaystyle\sim p(\bm{\theta}_{k}|\mathcal{D})\,.$

Hence, we reach the conclusion that DML is approximately equivalent to an MCMC
posterior sampling for the parameter $\bm{\theta}$ while its prior is defined
by an empirical approximation of the unknown prior. Figure 3 depicts an
illustrative example of this process.

###  4.3 Advantage of the Rényi Divergence in DML

Adopting the Rényi divergence in DML incorporates another flexible control to
this stochastic process. In other words, $\alpha$ controls the variance of
this empirical prior, and a larger $\alpha$ results in smaller variance.
Additionally, $\alpha$ imposes nonlinear change to the control of the movement
of each $\bm{\theta}$ in the probability space. Given Remark 2, at each
timestamp, $\bm{\theta}$ moves in a larger pace to the mean with a greater
$\alpha$, while it can move slower with a smaller $\alpha$. As pointed out in
[14], a prior better explaining the data does not necessarily lead to a better
generalized posterior. Thus, we should embrace the flexibility to allow for
broader prior choices wherein there is a greater chance to seek a better
solution.

###  4.4 Comparing with DML & Linear Scaling

Alternatively, one may consider multiplying the KL divergence part in the loss
with a linear scale factor $\psi>0$. Following that, the objective can be
written as

$\max_{\bm{\theta}}\log p(\mathbf{y}|\bm{\theta},\mathbf{x})+\psi\log
p(\bm{\theta})\\\ \equiv\max_{\bm{\theta}}1/\psi\log
p(\mathbf{y}|\bm{\theta},\mathbf{x})+\log p(\bm{\theta})\,.$

This implies that the corresponding posterior roughly assumes
$p(\bm{\theta}|\mathcal{D},\psi)\propto
p(\mathbf{y}|\bm{\theta},\mathbf{x})^{1/\psi}p(\bm{\theta})$. If $0<1/\psi\leq
1$ is satisfied, this is namely the _power posterior_ , which is proved as a
more robust Bayesian posterior [32, 7]. The power posterior provides better
performance when the prior is misspecified (in some sense, overfitting). While
the prior is sufficiently good, general posteriors will work as well as power
posteriors. That is, seeking a nicer prior is still in a higher order.
Referring to our situation, it means that this approach and RDML are not
competitors and can collaborate with no conflict.

##  5 Empirical Study

The empirical study aims to explore if RDML is capable of uplifting the model
performance more than the vanilla DML and the single models. Hence, we reduce
the effort in tuning the models but instead will follow some general settings.
We focus on the following research questions.

  1. 1.

How does RDML perform compared with the vanilla DML and the single model on
its own?

  2. 2.

Does RDML generalize better?

Table 1:  Top-1 accuracy ($\%$) results for CIFAR10 and CIFAR100. For a stabler outcome, we average the accuracy of the test set in the last 10 epochs. Also, the same models are reindexed by their values in an ascending order. In the table, GN is abbreviated for GoogLeNet, RN is short for ResNet34, and VG is short for VGG16. The subscript of a model is the model index, starting from 0. $\mathrm{RDML{}}_{a}$ means that it sets the $\alpha$ value as $a$. The independent case (Ind.) has only one result.  |  Dataset: CIFAR10 |  Dataset: CIFAR100
---|---|---
$\mathrm{GN}_{0}$ | $\mathrm{GN}_{1}$ | $\mathrm{RN}_{0}$ | $\mathrm{RN}_{1}$ | $\mathrm{VG}_{0}$ | $\mathrm{VG}_{1}$ | $\mathrm{GN}_{0}$ | $\mathrm{GN}_{1}$ | $\mathrm{RN}_{0}$ | $\mathrm{RN}_{1}$ | $\mathrm{VG}_{0}$ | $\mathrm{VG}_{1}$
Ind. | 91.87 |  | 92.56 |  | 90.71 |  | 69.61 |  | 76.26 |  | 72.21 |
$\mathrm{RDML}_{0.5}$ | 91.91 | 92.06 | 92.86 | 93.03 | 91.03 | 91.26 | 70.22 | 70.24 | 76.71 | 77.14 | 72.09 | 72.53
$\mathrm{DML}$ | 92.28 | 92.37 | 93.27 | 93.34 | 91.22 | 91.33 | 72.34 | 72.49 | 77.14 | 77.24 | 73.01 | 73.17
$\mathrm{RDML}_{1.5}$ | 92.61 | 92.62 | 93.28 | 93.35 | 91.17 | 91.29 | 71.65 | 72.1 | 77.61 | 77.94 | 73.32 | 73.6
$\mathrm{RDML}_{2.0}$ | 92.26 | 92.28 | 93.18 | 93.19 | 91.17 | 91.23 | 72.13 | 72.26 | 78.31 | 78.5 | 73.30 | 73.74
Table 2:  Top-1 accuracy ($\%$) results for DTD and Flowers102. We average the accuracy results of the last 5 epochs in the test set. In the table, IV4 is abbreviated for InceptionV4 and YV3 is short for YoloV3. Ind. is again the independent case and the subscript of RDML is the $\alpha$ value. The same models are reindexed by their values in an ascending order.  |  Dataset: DTD |  Dataset: Flowers102
---|---|---
$\mathrm{IV4}_{0}$ | $\mathrm{IV4}_{1}$ | $\mathrm{ViT}_{0}$ | $\mathrm{ViT}_{1}$ | $\mathrm{YV3}_{0}$ | $\mathrm{YV3}_{1}$ | $\mathrm{IV4}_{0}$ | $\mathrm{IV4}_{1}$ | $\mathrm{ViT}_{0}$ | $\mathrm{ViT}_{1}$ | $\mathrm{YV3}_{0}$ | $\mathrm{YV3}_{1}$
Ind. | 64.66 |  | 72.55 |  | 68.95 |  | 85.70 |  | 98.23 |  | 82.94 |
$\mathrm{RDML}_{0.5}$ | 64.40 | 65.89 | 72.77 | 73.94 | 69.50 | 69.89 | 84.14 | 84.21 | 98.28 | 98.52 | 78.01 | 78.47
$\mathrm{DML}$ | 65.87 | 66.20 | 74.41 | 75.11 | 69.93 | 70.23 | 84.94 | 85.01 | 98.23 | 98.54 | 78.89 | 79.25
$\mathrm{RDML}_{1.5}$ | 67.57 | 67.67 | 75.96 | 76.01 | 70.90 | 71.72 | 83.82 | 84.11 | 98.68 | 99.09 | 80.03 | 80.21
$\mathrm{RDML}_{2.0}$ | 67.86 | 68.97 | 75.37 | 76.12 | 71.17 | 71.26 | 83.53 | 84.26 | 98.99 | 99.01 | 80.74 | 81.1

###  5.1 Setup

The tested data covers the two popular fields Computer Vision (CV) and Natural
Language Processing (NLP). The CV data contains CIFAR10 and CIFAR100 [12] for
resolution $32\times 32$. We examine on DTD [2] and Flowers102 [17] for
resolution $224\times 224$. These datasets are used to conduct experiments for
the image classification task. The NLP data for text classification task
contains AGNews, Yahoo!Answers, and YelpReviewFull. All data is handled
through either torchvision222https://pytorch.org/vision/stable/datasets.html
or torchtext333https://pytorch.org/text/stable/datasets.html.

Table 3:  Top-1 accuracy ($\%$) results for the test sets of AGNews, Yahoo!Answers, and YelpReviewFull. The best performing $\alpha$ of RDML is chosen based on the mean accuracy of $K$ models on each test set.  Dataset | fastText | CharCNN
---|---|---
Independent | fastText-0 | fastText-1 | $\alpha$ | Independent | CharCNN-0 | CharCNN-1 | $\alpha$
AGNews | 89.724 | 89.763 | 89.763 | 2.0 | 87.132 | 89.632 | 89.171 | 1.5
Yahoo!Answers | 65.645 | 65.650 | 65.650 | 0.5 | 63.538 | 64.542 | 64.507 | 2.0
YelpReviewFull | 54.262 | 54.308 | 54.312 | 1.5 | 55.014 | 55.358 | 55.582 | 1.5

Regarding CIFAR10 and CIFAR100, we utilize the SGD with Nesterov momentum
$0.9$. The initial learning rate is set to $0.1$ and the weight decay is set
to $0.0005$. Apart from that, we run $200$ epochs in total for a batch size
$128$. The learning rate drops by $0.2$ at every $60$th epoch. In light of the
architectures, we adopt GoogLeNet [27], ResNet34 [8], and VGG16 [24]444The
implementations of these three architectures are from
https://github.com/weiaicunzai/pytorch-cifar100. As regards of DTD and
Flowers192, we set the learning rate to $0.005$ and set weight decay as
$0.0001$ using the SGD with momentum $0.9$. We run 30 epochs with batch size
$32$ and decay the learning rate by $0.1$ at epochs $(16,22)$. Also, the
gradients are clipped by the max norm $5$. For these two datasets,
timm555https://rwightman.github.io/pytorch-image-models is applied to acquire
the pretrained weights for the models: InceptionV4 [26], ViT [29], and YoloV3
[22]. In text classification, we examine fastText [10] with GloVe [19]
(pretrained word embeddings) and CharCNN [33] using a public
implementation666https://github.com/AnubhavGupta3377/Text-Classification-
Models-Pytorch. We empirically set an initial learning rate to $0.0001$ and
set the weight decay to $0.9$ for the experiments with CharCNN for
Yahoo!Answers and YelpReviewFull. Apart from that, we use the default settings
of the implementation with an initial learning rate to $0.001$ and set the
weight decay to $0.5$ for all other experiments. Finally, 20 epochs are run
and the learning rate is decayed every three epochs with a batch size of 128,
for each dataset and method.

Although a larger $K$ must be in favor for better approximating the prior, a
pragmatic solution has to cope with the runtime efficiency. We follow [34] to
set $K=2$ which suffices to raise the model performance.

(a)

(b)

(c)

Figure 4: Test loss of RDML for various $\alpha$ on the CIFAR100 dataset

(a)

(b)

(c)

Figure 5: Test accuracy of RDML for various $\alpha$ on the CIFAR100 dataset

###  5.2 Evaluation

This subsection is dedicated to answer the first question.

##### Image Classification.

The training set and test set are split automatically through torchvision. As
for classification, we employ the top-1 accuracy to examine the model
performance. The experiments are run upon a range of configurations, including
the independent case and $\alpha\in\\{0.5,1,1.5,2\\}$. The results for CIFAR10
and CIFAR100 are shown in Table 1, and those for DTD and Flowers102 are
displayed in Table 2. The indices of the models are sorted by their values. We
label the column-wise best performer using the bold face.

With respect to CIFAR10, the results show that for GoogLeNet and ResNet34,
RDML with $\alpha=1.5$ are the winner, while vanilla DML obtains the best
accuracy with VGG16. In the experiments for CIFAR100, the three selected
architectures achieve the best outcomes with $\alpha=2$ in most cases.
Observed from the results, RDML under certain configuration always outperforms
the independent case. It demonstrates that tuning $\alpha$ is helpful in
learning better model parameters. Regarding DTD, $\alpha=2$ and $\alpha=1.5$
are still best options for the selected architectures. Same as DML, RDML also
perfectly collaborates with the pretrained models which are agreed to be more
powerful in modern DL tasks [3]. Additionally, we notice that the improvements
over the independent model are greater with these models.

Flowers102 is an interesting case that the performance of InceptionV4 and
YoloV3 both downgrade when employing DML or RDML. After investigating more, we
notice that this strongly points to generalization and leave more discussion
in Figure 6.

##### Text Classification.

Similar to the image classification, we adopt the top-1 accuracy for
evaluation. Here, we demonstrate $\alpha$ can also be tuned with a grid search
on validation set as a hyperparameter. To this end, we choose $\alpha$ based
on a grid search $\in\\{0.5,1,1.5,2\\}$ using a validation set (which is 20%
of the training set), and investigate what the best performing $\alpha$ values
are and the corresponding performance on the test set. More specifically, the
$\alpha$ leading to the highest average accuracy on the test set is presented.
The results are summarized in Table 3.

Clearly, the best performing $\alpha$ values vary depending on models and
datasets, and improves the independent model consistently, which is in line
with the observations of our results on image classification. We also note
that, regularization via RDML is more beneficial for the larger models with
respect to improving the performance over the corresponding independent model,
e.g., CharCNN (with $\sim$2M parameters) versus fastText (with $\sim$3K
parameters). This coincides with the fact that larger models are generally
more prone to overfitting and hence RDML is a valuable addition to them.

###  5.3 Generalization Results

Figure 6: Test loss plots for Flowers102

In this section, we examine the generalization ability of the RDML over the
choice of $\alpha$, for answering the second research question. We focus on
the test performance of CIFAR100 and Flowers102. The other results are placed
in the supplemental material.

First of all, we check the base loss for the test data with regard to the
CIFAR100 dataset in Figure 4. It is commonly agreed in the machine learning
community that the training loss can be constantly decreased while the test
loss will be climbing after a certain point. In Figure 4, the test loss of a
single model (either GoogLeNet, ResNet34, or VGG16) perfectly replicates this
agreement. However, RDML is able to contain this growth tendency of test loss
for all students; and, clearly, the capability of preventing this increase
gets enhanced as $\alpha$ becomes larger, i.e. setting a prior with smaller
variance. Moreover, we observe that RDML with larger $\alpha$ significantly
reduces the fluctuations in the test loss, i.e., the variance is being reduced
for the test loss (though it still maintains a reasonable amount of
uncertainty). It shows that $\alpha=2$ for GoogLeNet over-regularizes the
model and thus performs slightly worse than that with $\alpha=1.5$. In the
classification problem, the test base loss is the negative likelihood of the
unseen data and it can be optimized via RDML. The results indicate that RDML
is able to achieve better generalizations through tuning the parameter
$\alpha$.

Similar patterns are observed while checking the test accuracy in Figure 5. In
regard to ResNet34, starting from epoch $60$, the single model boosts its
accuracy since the learning rate is decayed, but rapidly receives a drop in
the accuracy. However, it shows that this drop can be significantly mitigated
by RDML and achieve the best outcome when $\alpha=2$ within the range of our
choices. Correspondingly, the results for ResNet34 in Figure 4 show a loss
increase during the same period. As shown, RDML can limit this loss increasing
speed and can limit harder with greater $\alpha$.

#####  Flowers102.

We argue that Flowers102 is a considerably simpler dataset since all models
can easily achieve high test accuracy. Figure 6 depicts that the independent
cases of both InceptionV4 and YoloV3 in this task (along with the
configuration) have been generalizing very well. The red solid and circled
lines, for the independent runs, are clearly distinguishable from others.
Though, this generalization does not necessarily help the model outperform
others due to the model capacity but indicates that it may highly exploit its
own potential. Since the model on its own has been outstanding in
generalization, the uncertainty that RDML brings to the model will be
excessive and redundant. For ViT, the solid red line performs nicely in the
first 15-16 epochs. However, RDML with $\alpha=1.5$ and $\alpha=2$ catch up in
the latter epochs. Finally, it excels the independent case but the improvement
is not very significant.

###  5.4 Discussion

Through this empirical study, we demonstrate that RDML is able to improve on
the vanilla DML and the independent model in most cases. This covers a broad
collection of network architectures and datasets. In the selection of our
experiments, RDML is shown more valuable when dealing with harder tasks and
underperforming configurations, which is also reasonable since a perfect model
for a simple task already suffices to reach the maximum performance. Imposing
extra components to such a process can ruin its performance.

##  6 Conclusion

In this paper, we have revisited DML and interpreted it from a Bayesian
perspective. Inspired by the interpretation that DML and Rényi divergence are
a good combination for learning more generalized models, we propose RDML which
is more flexible for further tuning the generalization. The experimental
results agree with our findings and demonstrate that RDML has greater capacity
than DML for improving the model performance. Last but not least, we expect
that our Bayesian interpretation of DML may inspire theorists to conduct more
versatile and interesting analysis.

## References

  * [1] S. Bai, T. Lepoint, A. Roux-Langlois, A. Sakzad, D. Stehlé, and R. Steinfeld, Improved Security Proofs in Lattice-based Cryptography: Using the Rényi Divergence Rather Than the Statistical Distance, Journal of Cryptology, 31 (2018), pp. 610–640.
  * [2] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi, Describing Textures in the Wild, in CVPR, 2014.
  * [3] D. Erhan, A. Courville, Y. Bengio, and P. Vincent, Why Does Unsupervised Pre-training Help Deep Learning?, JMLR, (2010), pp. 201–208.
  * [4] T. V. Erven and P. Harrëmos, Rényi Divergence and Kullback-Leibler Divergence, IEEE Transactions on Information Theory, 60 (2014), pp. 3797–3820.
  * [5] M. Girolami and B. Calderhead, Riemann Manifold Langevin and Hamiltonian Monte Carlo methods, Journal of the Royal Statistical Society: Series B (Statistical Methodology), 73 (2011), pp. 123–214.
  * [6] J. Gou, B. Yu, S. J. Maybank, and D. Tao, Knowledge Distillation: A Survey, ICCV, 129 (2021), pp. 1789–1819.
  * [7] P. Grünwald, The Safe Bayesian: Learning the Learning Rate via the Mixability Gap, Lecture Notes in Computer Science, 7568 LNAI (2012), pp. 169–183.
  * [8] K. He, X. Zhang, S. Ren, and J. Sun, Deep Residual Learning for Image Recognition, in CVPR, 2016, pp. 770–778.
  * [9] G. Hinton, O. Vinyals, and J. Dean, Distilling the Knowledge in A Neural Network, arXiv preprint arXiv:1503.02531, 2 (2015).
  * [10] A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov, Bag of tricks for efficient text classification, arXiv preprint arXiv:1607.01759, (2016).
  * [11] D. P. Kingma and M. Welling, An Introduction to Variational Autoencoders, Foundations and Trends® in Machine Learning, 12 (2019), pp. 307–392.
  * [12] A. Krizhevsky and G. Hinton, Learning multiple layers of features from tiny images, (2009).
  * [13] Y. Li and R. E. Turner, Rényi Divergence Variational Inference, NeurIPS, 29 (2016).
  * [14] S. Lotfi, P. Izmailov, G. Benton, M. Goldblum, and A. G. Wilson, Bayesian Model Selection, the Marginal Likelihood, and Generalization, in ICML, vol. 162, PMLR, 17–23 Jul 2022, pp. 14223–14247.
  * [15] R. Masumura, M. Ihori, A. Takashima, T. Tanaka, and T. Ashihara, End-to-end Automatic Speech Recognition with Deep Mutual Learning, in APSIPA ASC, IEEE, 2020, pp. 632–637.
  * [16] C. Mingard, G. Valle-Pérez, J. Skalse, and A. A. Louis, Is SGD a Bayesian sampler? Well, almost., JMLR, 22 (2021), pp. 1–64.
  * [17] M.-E. Nilsback and A. Zisserman, Automated Flower Classification over a Large Number of Classes, in Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.
  * [18] W. Park, W. Kim, K. You, and M. Cho, Diversified mutual learning for deep metric learning, in ECCV, Springer, 2020, pp. 709–725.
  * [19] J. Pennington, R. Socher, and C. D. Manning, Glove: Global vectors for word representation, in EMNLP, 2014, pp. 1532–1543.
  * [20] M. Phuong and C. Lampert, Towards understanding knowledge distillation, in ICML, 2019, pp. 5142–5151.
  * [21] T. Prest, Sharper Bounds in Lattice-Based Cryptography using the Rényi Divergence, in International Conference on the Theory and Application of Cryptology and Information Security, 2017, pp. 347–374.
  * [22] J. Redmon and A. Farhadi, Yolov3: An incremental improvement, arXiv preprint arXiv:1804.02767, (2018).
  * [23] N. Sajid, F. Faccio, L. Da Costa, T. Parr, J. Schmidhuber, and K. Friston, Bayesian Brains and the Rényi Divergence, Neural Computation, 34 (2022), pp. 829–855.
  * [24] K. Simonyan and A. Zisserman, Very Deep Convolutional Networks for Large-Scale Image Recognition, in ICLR, 2015.
  * [25] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, Dropout: A Simple Way to Prevent Neural Networks from Overfitting, JMLR, 15 (2014), pp. 1929–1958.
  * [26] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi, Inception-v4, inception-resnet and the impact of residual connections on learning, in AAAI, 2017.
  * [27] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, Going deeper with convolutions. 2014, arXiv preprint arXiv:1409.4842, 10 (2014).
  * [28] M. Welling and Y. W. Teh, Bayesian Learning via Stochastic Gradient Langevin Dynamics, in ICML, 2011, pp. 681–688.
  * [29] B. Wu, C. Xu, X. Dai, A. Wan, P. Zhang, Z. Yan, M. Tomizuka, J. Gonzalez, K. Keutzer, and P. Vajda, Visual transformers: Token-based image representation and processing for computer vision, arXiv preprint arXiv:2006.03677, (2020).
  * [30] T. Yang, S. Zhu, C. Chen, S. Yan, M. Zhang, and A. Willis, Mutualnet: Adaptive convnet via mutual learning from network width and resolution, in ECCV, Springer, 2020, pp. 299–315.
  * [31] H. Zhang, W. Liang, C. Li, Q. Xiong, H. Shi, L. Hu, and G. Li, DCML: Deep Contrastive Mutual Learning for COVID-19 Recognition, Biomedical Signal Processing and Control, 77 (2022), p. 103770.
  * [32] T. Zhang, From $\epsilon$-entropy to KL-entropy: Analysis of minimum information complexity density estimation, The Annals of Statistics, 34 (2006), pp. 2180–2210.
  * [33] X. Zhang, J. Zhao, and Y. LeCun, Character-level convolutional networks for text classification, NeurIPS, 28 (2015).
  * [34] Y. Zhang, T. Xiang, T. M. Hospedales, and H. Lu, Deep Mutual Learning, in CVPR, 2018, pp. 4320–4328.
  * [35] H. Zhao, G. Yang, D. Wang, and H. Lu, Deep Mutual Learning for Visual Object Tracking, Pattern Recognition, 112 (2021), p. 107796.
  * [36] J. Zhao, W. Luo, B. Chen, and A. Gilman, Mutual-learning improves end-to-end speech translation, in EMNLP, 2021, pp. 3989–3994.