# Conditional Mutual Information Constrained Deep Learning for Classification

En-Hui Yang, Shayan Mohajer Hamidi∗, Linfeng Ye∗, Renhao Tan, and Beverly Yang
∗ Authors contributed equally.
En-Hui Yang, Shayan Mohajer Hamidi, Linfeng Ye and Renhao Tan are with the
Department of Electrical and Computer Engineering, University of Waterloo,
Waterloo, ON N2L 3G1, Canada (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>cameron.tan@uwaterloo.ca).
Beverly Yang is with the NBK Institute of Mining Engineering, University of
British Columbia, Vancouver, BC V6T 1Z4, Canada (e-mail: beverly.yang@ubc.ca).

###### Abstract

The concepts of conditional mutual information (CMI) and normalized
conditional mutual information (NCMI) are introduced to measure the
concentration and separation performance of a classification deep neural
network (DNN) in the output probability distribution space of the DNN, where
CMI and the ratio between CMI and NCMI represent the intra-class concentration
and inter-class separation of the DNN, respectively. By using NCMI to evaluate
popular DNNs pretrained over ImageNet in the literature, it is shown that
their validation accuracies over ImageNet validation data set are more or less
inversely proportional to their NCMI values. Based on this observation, the
standard deep learning (DL) framework is further modified to minimize the
standard cross entropy function subject to an NCMI constraint, yielding CMI
constrained deep learning (CMIC-DL). A novel alternating learning algorithm is
proposed to solve such a constrained optimization problem. Extensive
experiment results show that DNNs trained within CMIC-DL outperform the state-
of-the-art models trained within the standard DL and other loss functions in
the literature in terms of both accuracy and robustness against adversarial
attacks. In addition, visualizing the evolution of learning process through
the lens of CMI and NCMI is also advocated.

###### Index Terms:

Alternating minimization, concentration and separation, conditional mutual
information, cross entropy, deep learning.

##  I Introduction

In recent years, deep neural networks (DNNs) have been applied in a wide range
of applications, revolutionizing fields like computer vision, natural language
processing, and speech recognition [1, 2]. Typically, a DNN consists of
cascaded non-linear layers that progressively produce multi-layers of
representations with increasing levels of abstraction, starting from raw input
data and ending with a predicted output label. The success of DNNs is largely
attributable to their ability to learn these multi-layers of representations
as features from the raw data through a deep learning (DL) process.

Putting its neural architecture aside, a classification DNN is,
mathematically, a mapping from raw data $x\in\mathbb{R}^{d}$ to a probability
distribution $P_{x}$ over the set of class labels, predicting an output label
$\hat{y}$ with probability $P_{x}(\hat{y})$. Given a pair of random variables
$(X,Y)$, the distribution of which governs either a training set or testing
set, where $X\in\mathbb{R}^{d}$ represents the raw data and $Y$ is the ground
truth label of $X$, the prediction performance of the DNN is often measured by
its error rate

$\epsilon=\Pr\\{\hat{Y}\not=Y\\},$

where $\hat{Y}$ is the label predicted by the DNN with probability
$P_{X}(\hat{Y})$ in response to the input $X$. The accuracy of the DNN is
equal to $1-\epsilon$. The error rate is further upper bounded by the average
of the cross entropy between the conditional distribution of $Y$ given $X$ and
$P_{X}$ (see Section II). To have better prediction performance, a DL process
is then applied to minimize the error rate $\epsilon$ or its cross entropy
upper bound [1, 2].

Although the error rate of a DNN is its most important performance as far as
its prediction is concerned, focusing entirely on the error rate is not
enough, and can actually lead to several problems. First, the error rate of a
DNN depends not only on the DNN itself, but also on the governing joint
distribution of $(X,Y)$. When a DNN has a small error rate for one governing
joint distribution of $(X,Y)$, it does not necessarily imply that it would
have a small error rate for another governing joint distribution of $(X,Y)$,
especially when two distributions are quite different. This is essentially
related to the well-known overfitting and robustness problems [2, 3, 4, 5].
Second, even when a DNN works well across different governing distributions of
$(X,Y)$, it remains a block box to us, especially when its architecture is
huge. We don’t know why it works and how it works. Its error rate does not
reveal any useful information about the intrinsic mapping structure such as
the intra-class concentration and inter-class separation of the DNN in its
output probability distribution space.

To gain deep insights into the intrinsic mapping structure of a DNN as a
mapping from $x\in\mathbb{R}^{d}$ to $P_{x}$, in this paper we introduce
information quantities from information theory [6] to measure intra-class
concentration and inter-class separation of the DNN. Specifically, we propose
to use the conditional mutual information (CMI) $I(X;\hat{Y}|Y)$ between $X$
and $\hat{Y}$ given $Y$ as the measure for the intra-class concentration of
the DNN as a mapping $x\in\mathbb{R}^{d}\to P_{x}$. For each class label $y$,
the conditional mutual information $I(X;\hat{Y}|Y=y)$ between $X$ and
$\hat{Y}$ given $Y=y$ tells how all output probability distributions $P_{X}$
given $Y=y$ are concentrated around its “centroid”, the conditional
probability distribution $P_{\hat{Y}|Y=y}$. The smaller $I(X;\hat{Y}|Y=y)$ is,
the more concentrated all output probability distributions $P_{X}$ given $Y=y$
are around its centroid. We further introduce another information quantity
(see Section II) to measure the inter-class separation of the DNN as a mapping
$x\in\mathbb{R}^{d}\to P_{x}$. Define the ratio between $I(X;\hat{Y}|Y)$ and
the inter-class separation as the normalized conditional mutual information
(NCMI) between $X$ and $\hat{Y}$ given $Y$. One may interpret CMI and NCMI as
certain mapping structure traits of the DNN. Then in addition to its error
rate, the DNN can also be evaluated in terms of its CMI and NCMI.

Equipped with our new concepts of CMI and NCMI, we further evaluate popular
DNNs pretrained in the literature over ImageNet in terms of their respective
CMI and NCMI. It turns out that their validation accuracies over the ImageNet
validation data set are more or less inversely proportional to their NCMI
values. In other words, even though these DNNs have different architectures
and different sizes, their error rates and NCMI values have more or less a
positive linear relationship. Indeed, the correlation between the error rate
and NCMI is above $0.99$. This implies that given a DNN architecture, one may
be able to further improve the effectiveness of DL by simultaneously
minimizing the error rate (or cross entropy upper bound) and NCMI of the DNN
during the learning process, where the error rate and NCMI represent the
prediction performance and the concentration/separation mapping structure
performance of the DNN, respectively. This in turn motivates us to modify the
standard DL framework to minimize the standard cross entropy function subject
to an NCMI constraint, yielding CMI constrained deep learning (CMIC-DL). To
this end, a novel alternating learning algorithm is further proposed to solve
such a constrained optimization problem. Extensive experiment results show
that DNNs trained within CMIC-DL outperform the state-of-the-art models
trained within the standard DL and other loss functions in the literature in
terms of both accuracy and robustness against adversarial attacks.

The remainder of this paper is organized as follows. In Section II, we
formally introduce the concepts of CMI and NCMI to measure intra-class
concentration and inter-class separation structure performance of a DNN when
it is viewed as a mapping from $x\in\mathbb{R}^{d}$ to $P_{x}$. In Section
III, we use NCMI to evaluate and compare popular DNNs pretrained in the
literature over ImageNet. These DNNs have different architectures and
different sizes. Section IV is devoted to the full development of CMIC-DL. In
Section V, extensive experiment results are presented and compared with the
prior art in the literature; visualizing the evolution of learning process
through the lens of CMI and NCMI is also advocated. Finally, conclusions are
drawn along with some open problems in Section VI.

##  II Performance of DNNs: Concentration and Separation

A DNN can be described either by its neural architecture along with its
connection weights, the number of which can be in billions, or by its
mathematical mapping from $x\in\mathbb{R}^{d}$ to $P_{x}$. Both perspectives
are useful. In this and next sections, we will take the second perspective and
regard a DNN simply as a mapping $x\in\mathbb{R}^{d}\to P_{x}$. Before
formally introducing CMI and NCMI, we set up notation to be used throughout
the paper.

###  II-A Notation

For a positive integer $K$, let $[K]\triangleq\\{1,\dots,K\\}$. Assume that
there are $C$ class labels with $[C]$ as the set of class labels. Let ${\cal
P}([C])$ denote the set of all probability distributions over $[C]$. For any
two probability distributions $P_{1},P_{2}\in{\cal P}([C])$, the cross entropy
of $P_{1}$ and $P_{2}$ is defined as

$H(P_{1},P_{2})=\sum_{i=1}^{C}-P_{1}(i)\ln P_{2}(i),$ (1)

where $\ln$ denotes the logarithm with base $e$; the Kullback–Leibler (KL)
divergence (or relative entropy) between $P_{1}$ and $P_{2}$ is defined as

$D(P_{1}||P_{2})=\sum_{i=1}^{C}P_{1}(i)\ln{P_{1}(i)\over P_{2}(i)}.$ (2)

For any $y\in[C]$ and $P\in{\cal P}([C])$, write the cross entropy of the one-
hot probability distribution corresponding to $y$ and $P$ as

$H(y,P)=-\ln P(y).$ (3)

Given a DNN: $x\in\mathbb{R}^{d}\to P_{x}$, let $\mathbf{\theta}$ denote its
weight vector consisting of all its connection weights; whenever there is no
ambiguity, we also write $P_{x}$ as $P_{x,\mathbf{\theta}}$, and $P_{x}(y)$ as
$P(y|x,\mathbf{\theta})$ for any $y\in[C]$.

###  II-B Error Rate

Fix a DNN: $x\in\mathbb{R}^{d}\to P_{x}$. As before, let $(X,Y)$ be a pair of
random variables representing the raw input data and the corresponding ground
truth label; let $\hat{Y}$ be the label predicted by the DNN with probability
$P_{X}(\hat{Y})$ in response to the input $X$, that is, for any input
$x\in\mathbb{R}^{d}$ and any $\hat{y}\in[C]$

$P(\hat{Y}=\hat{y}|X=x)=P_{x}(\hat{y})=P(\hat{y}|x,\mathbf{\theta}).$ (4)

Note that $Y\to X\to\hat{Y}$ forms a Markov chain in the indicated order.
Therefore, given $X=x$, $Y$ and $\hat{Y}$ are conditionally independent.

The error rate of the DNN for $(X,Y)$ is equal to

$\epsilon=\Pr\\{\hat{Y}\not=Y\\}$

which can be upper bounded by the average of the cross entropy of the
conditional probability distribution of $Y$ given $X$,
$P_{Y|X}=P_{Y|X}(\cdot|X)$, and $P_{X}$, as shown in the following theorem.

######  Theorem 1.

For any DNN: $x\in\mathbb{R}^{d}\to P_{x}$ and any $(X,Y)$,

$\epsilon\leq\mbox{$\bf E$}_{X}\left[H(P_{Y|X},P_{X})\right]$ (5)

where $\mbox{$\bf E$}_{X}$ denotes the expectation with respect to $X$.

###### Proof:

Let $I_{\\{\hat{Y}\not=Y\\}}$ denote the indicator function of the event
$\\{\hat{Y}\not=Y\\}$. Then

$\displaystyle\epsilon$ $\displaystyle=$ $\displaystyle\Pr\\{\hat{Y}\not=Y\\}$
$\displaystyle=$ $\displaystyle\mbox{$\bf E$}[I_{\\{\hat{Y}\not=Y\\}}]$
$\displaystyle=$ $\displaystyle\mbox{$\bf E$}_{X}\left[\mbox{$\bf
E$}[I_{\\{\hat{Y}\not=Y\\}}|X]\right]$ $\displaystyle=$
$\displaystyle\mbox{$\bf
E$}_{X}\left[1-\sum_{i=1}^{C}P_{Y|X}(i|X)P_{X}(i)\right]$ $\displaystyle=$
$\displaystyle\mbox{$\bf
E$}_{X}\left[\sum_{i=1}^{C}P_{Y|X}(i|X)(1-P_{X}(i))\right]$
$\displaystyle\leq$ $\displaystyle\mbox{$\bf
E$}_{X}\left[\sum_{i=1}^{C}-P_{Y|X}(i|X)\ln P_{X}(i)\right]$ (7)
$\displaystyle=$ $\displaystyle\mbox{$\bf
E$}_{X}\left[H(P_{Y|X},P_{X})\right]$ (8)

where (II-B) follows from the fact that $Y$ and $\hat{Y}$ are conditionally
independent given $X$, and (7) is due to the inequality $\ln z\leq z-1$ for
any $z>0$. This completes the proof of Theorem 1. ∎

Given $X=x$, what happens if the DNN outputs instead the top one label
$\hat{Y}^{*}$

$\hat{Y}^{*}=\operatornamewithlimits{arg\,max}_{i\in[C]}P_{x}(i)?$

In this case, the error rate of the DNN for $(X,Y)$ is equal to

$\epsilon^{*}=\Pr\\{\hat{Y}^{*}\not=Y\\}$

which can also be upper bounded in terms of $\mbox{$\bf
E$}_{X}\left[H(P_{Y|X},P_{X})\right]$.

######  Corollary 1.

For any DNN: $x\in\mathbb{R}^{d}\to P_{x}$ and any $(X,Y)$,

$\epsilon^{*}\leq C\epsilon\leq C\mbox{$\bf
E$}_{X}\left[H(P_{Y|X},P_{X})\right].$ (9)

###### Proof:

$\displaystyle\epsilon^{*}$ $\displaystyle=$
$\displaystyle\Pr\\{\hat{Y}^{*}\not=Y\\}$ $\displaystyle=$
$\displaystyle\mbox{$\bf E$}_{X}\left[1-P_{Y|X}(\hat{Y}^{*}|X)\right]$
$\displaystyle\leq$ $\displaystyle C\mbox{$\bf
E$}_{X}\left[P_{X}(\hat{Y}^{*})(1-P_{Y|X}(\hat{Y}^{*}|X))\right]$
$\displaystyle\leq$ $\displaystyle C\mbox{$\bf
E$}_{X}\left[\sum_{i=1}^{C}P_{X}(i)(1-P_{Y|X}(i|X))\right]$ $\displaystyle=$
$\displaystyle C\mbox{$\bf
E$}_{X}\left[1-\sum_{i=1}^{C}P_{Y|X}(i|X)P_{X}(i)\right]$ $\displaystyle=$
$\displaystyle C\epsilon\leq C\mbox{$\bf
E$}_{X}\left[H(P_{Y|X},P_{X})\right],$ (11)

where (II-B) follows from the fact that $P_{X}(\hat{Y}^{*})\geq 1/C$, and (11)
is due to (II-B) and (8). ∎

In view of Theorem 1 and Corollary 1, no matter which form of error rate
$\epsilon$ or $\epsilon^{*}$ is used, minimizing the average of the cross
entropy $\mbox{$\bf E$}_{X}[H(P_{Y|X},P_{X})]$ would have an effect to reduce
$\epsilon$ and $\epsilon^{*}$. This provides mathematical justifications for
the use of the average of the cross entropy $\mbox{$\bf
E$}_{X}[H(P_{Y|X},P_{X})]$ as an objective function or a major component
thereof in DL and knowledge distillation, where $P_{Y|X}$ is approximated by
the one-hot probability vector corresponding to $Y$ in DL[1, 2], and by the
output probability distribution of the teacher in knowledge distillation [7,
8, 9].

Figure 1: The mappings from the label space to the input space, and from the
input space to the output space of a DNN. Here caricatures are used to depict
label and input spaces, where each of the three instances in the label space
are mapped to two instances in input space according to
$P_{X|Y}(\cdot|Y=y_{i})$, for $i\in\\{1,2,3\\}$. On the other hand, the figure
for the output space is obtained from a real example, where for the ResNet56
model trained on CIFAR-100 dataset, the output probability vectors
corresponding to all validation sample instances from three randomly-picked
classes are projected over the two-dimensional probability simplex.

###  II-C Concentration

The error rates $\epsilon$ and $\epsilon^{*}$ of the DNN:
$x\in\mathbb{R}^{d}\to P_{x}$ for $(X,Y)$ do not provide any useful
information on the intrinsic mapping structure of the DNN in the probability
distribution space ${\cal P}([C])$. Two important mapping structure properties
the DNN: $x\in\mathbb{R}^{d}\to P_{x}$ possesses, are its intra-class
concentration and inter-class separation in the space ${\cal P}([C])$. In this
and next subsections, we formally introduce information quantities to quantify
these two mapping structure properties, respectively.

Visualize the DNN: $x\in\mathbb{R}^{d}\to P_{x}$ according to Fig. 1. Given
$Y=y$, $y\in[C]$, the input data $X$ is conditionally distributed according to
the conditional distribution $P_{X|Y}(\cdot|y)$ and then mapped into $P_{X}$,
a random point in the space ${\cal P}([C])$. The instances (or realizations)
of this random point $P_{X}$ form a cluster in the space ${\cal P}([C])$. The
centroid of this cluster is the average of $P_{X}$ with respect to the
conditional distribution $P_{X|Y}(\cdot|y)$, which is exactly the conditional
distribution of $\hat{Y}$ given $Y=y$

$P_{\hat{Y}|y}=\mbox{$\bf E$}[P_{X}|Y=y].$ (12)

Measure the “distance” between each $P_{X}$ and the centroid $P_{\hat{Y}|y}$
by their KL divergence $D(P_{X}||P_{\hat{Y}|y})$. Then the average of KL
divergence $D(P_{X}||P_{\hat{Y}|y})$ with respect to the conditional
distribution $P_{X|Y}(\cdot|y)$ is equal to

$\displaystyle\mbox{$\bf E$}\left[D(P_{X}||P_{\hat{Y}|y})|Y=y\right]$ (13)
$\displaystyle=$ $\displaystyle\mbox{$\bf
E$}\left[\left(\sum_{i=1}^{C}P_{X}(i)\ln{P_{X}(i)\over
P_{\hat{Y}|y}(\hat{Y}=i|Y=y)}\right)\left|Y=y\right.\right]$ $\displaystyle=$
$\displaystyle\sum_{x}P_{X|Y}(x|y)\left[\sum_{i=1}^{C}P(\hat{Y}=i|x)\times\right.$
$\displaystyle\left.\ln{P(\hat{Y}=i|x)\over
P_{\hat{Y}|y}(\hat{Y}=i|Y=y)}\right]$ $\displaystyle=$ $\displaystyle
I(X;\hat{Y}|y),$ (14)

where $I(X;\hat{Y}|y)$ is the conditional mutual information between $X$ and
$\hat{Y}$ given $Y=y$. (Please refer to [6] for the notions of mutual
information and conditional mutual information.) In (13), $X$ is assumed to be
discrete; if $X$ is continuous, then the average $\sum_{x}P_{X|Y}(x|y)$ should
be replaced by the integral

$\int_{x}dP_{X|Y}(x|y).$

Note that (14) is due to the fact that $Y\to X\to\hat{Y}$ forms a Markov
chain.

The information quantity $I(X;\hat{Y}|y)$ quantifies the concentration of the
cluster formed by the instances of the random point $P_{X}$ given $Y=y$ around
its centroid $P_{\hat{Y}|y}$. Averaging $I(X;\hat{Y}|y)$ with respect to the
distribution $P_{Y}(y)$ of $Y$, we get the conditional mutual information
$I(X;\hat{Y}|Y)$ between $X$ and $\hat{Y}$ given $Y$:

$\displaystyle I(X;\hat{Y}|Y)$ $\displaystyle=$
$\displaystyle\sum_{y\in[C]}P_{Y}(y)I(X;\hat{Y}|y)$ (15) $\displaystyle=$
$\displaystyle\mbox{$\bf E$}\left[D(P_{X}||P_{\hat{Y}|Y})\right]$
$\displaystyle=$
$\displaystyle\sum_{y}\sum_{x}P(x,y)\left[\sum_{i=1}^{C}P(\hat{Y}=i|x)\times\right.$
$\displaystyle\left.\ln{P(\hat{Y}=i|x)\over
P_{\hat{Y}|y}(\hat{Y}=i|Y=y)}\right].$

The CMI $I(X;\hat{Y}|Y)$ can then be regarded as a measure for the intra-class
concentration of the DNN: $x\in\mathbb{R}^{d}\to P_{x}$ for $(X,Y)$.

In practice, the joint distribution $P(x,y)$ of $(X,Y)$ may be unknown. To
compute the CMI $I(X;\hat{Y}|Y)$ in this case, one may approximate $P(x,y)$ by
the empirical distribution of a data sample
$\\{(x_{1},y_{1}),(x_{2},y_{2}),\cdots,(x_{n},y_{n})\\}$. For any $y\in[C]$,
let

$n_{y}=|\\{(x_{j},y_{j}):y_{j}=y,1\leq j\leq n\\}|,$ (16)

where $|S|$ denotes the cardinality of a set $S$, and

$P_{y}={1\over n_{y}}\sum_{(x_{j},y_{j}):y_{j}=y}P_{x_{j}}.$ (17)

Then $I(X;\hat{Y}|Y)$ can be computed as follows

$\displaystyle I(X;\hat{Y}|Y)$
$\displaystyle=\sum_{y\in[C]}\sum_{(x_{j},y_{j}):y_{j}=y}{1\over
n}D(P_{x_{j}}||P_{y})$ $\displaystyle={1\over
n}\sum_{j=1}^{n}D(P_{x_{j}}||P_{y_{j}}).$ (18)

###  II-D Separation and NCMI

Let $(U,V)$ be a pair of random variables independent of $(X,Y)$, and having
the same joint distribution as that of $(X,Y)$. With reference to Fig. 1, we
define the following information quantity111Other information quantities can
also be defined and used as a measure for the inter-class separation of the
DNN: $x\in\mathbb{R}^{d}\to P_{x}$, which will be explored in Appendix B.
Although they are more or less equivalent, the information quantity $\Gamma$
defined here is more convenient for the selection of hyper parameters in our
proposed CMIC deep learning.

$\Gamma=\mbox{$\bf E$}\left[I_{\\{Y\not=V\\}}H(P_{X},P_{U})\right],$ (19)

and use $\Gamma$ as a measure for the inter-class separation of the DNN:
$x\in\mathbb{R}^{d}\to P_{x}$. It is clear that the larger $\Gamma$ is, the
further apart different clusters are from each other on average.

Ideally, we want $I(X;\hat{Y}|Y)$ to be small while keeping $\Gamma$ large.
This leads us to consider the ratio between $I(X;\hat{Y}|Y)$ and $\Gamma$:

$\hat{I}(X;\hat{Y}|Y)\mbox{$\
\stackrel{{\scriptstyle\Delta}}{{=}}$}{I(X;\hat{Y}|Y)\over\Gamma}.$ (20)

We call $\hat{I}(X;\hat{Y}|Y)$ the normalized conditional mutual information
between $X$ and $\hat{Y}$ given $Y$.

In case where the joint distribution $p(x,y)$ of $(X,Y)$ is unknown, it can be
approximated by the empirical distribution of a data sample
$\\{(x_{1},y_{1}),(x_{2},y_{2}),\cdots,(x_{n},y_{n})\\}$. In parallel with
(18), $\Gamma$ can be computed in this case as follows:

$\Gamma={1\over
n^{2}}\sum_{j=1}^{n}\sum_{k=1}^{n}I_{\\{y_{j}\not=y_{k}\\}}H(P_{x_{j}},P_{x_{k}}),$
(21)

from which and (18), $\hat{I}(X;\hat{Y}|Y)$ can be computed accordingly.

###  II-E Related Works

In the literature, intra-class concentration and inter-class separation of a
DNN have been mainly investigated in the feature space corresponding to the
penultimate layer of the DNN, and largely treated in an ad-hoc manner in a
deep learning process or algorithm. Specifically, it was observed numerically
in [10, 11, 12] that DNNs concentrate features of each class around their
separated mean. This observation was further analyzed in [13] under the
Gaussian mixture model assumption about features. In [14, 15, 16, 17, 18] and
references therein, different loss functions including the so-called center
loss, contrastive center loss, orthogonal project loss, constrained center
loss, and their variants, all of which are defined in the feature space, were
proposed and used in the respective learning processes to improve the intra-
class concentration and inter-class separation of such trained DNNs.

In contrast, in this paper we investigate the intra-class concentration and
inter-class separation of a DNN in its output probability distribution space
${\cal P}([C])$, where the DNN is viewed as a mapping from
$x\in\mathbb{R}^{d}$ to $P_{x}$. This perspective allows us to introduce
information quantities, CMI, $\Gamma$, and NCMI, to quantify the intra-class
concentration and inter-class separation of each DNN. In addition, our
introduced CMI and NCMI can also be regarded as additional performance metrics
for any DNN, which are in parallel with the error rate performance metric, are
independent of any learning process, and represent mapping structure
properties of a DNN. As additional performance metrics, they can be used to
evaluate and compare different DNNs regardless of the architectures and sizes
of DNNs.

Another related work in the sense of introducing information theoretic ideas
into DL is the so-called coded deep learning (CDL) [19], where information
theoretic coding ideas are embedded into the inner workings of DL. The
purposes of CDL are to eliminate essentially floating operations of a coded
DNN during its inference time and efficiently compress the coded DNN while
maintaining or even improving the error rate of the coded DNN.

In the next section, CMI and NCMI $\hat{I}(X;\hat{Y}|Y)$ will be used to
evaluate and compare popular DNNs pre-trained over ImageNet in the literature.

##  III NCMI Vs. Accuracy

The popular DNNs we selected for evaluation according to their respective CMI
and NCMI are ResNet-$\\{18,34,50,101,152\\}$ [20], VGG-$\\{11,13,16,19\\}$
[21], EfficientNet-$\\{\text{B0},\text{B1},\text{B2},\text{B3}\\}$ [22], Wide-
ResNet-$\\{50,101\\}$ [23], MobileNet-V3-$\\{\text{small},\text{large}\\}$
[24], and AlexNet [25]. They are all pre-trained on ImageNet dataset and
obtained from the Pytorch official
website222https://pytorch.org/vision/stable/models.html..

Table I lists the values of CMI, $\Gamma$, and NCMI of the selected DNNs,
which are calculated, according to (18), (21), and (20), over the ImageNet
validation set, along with their respective error rate $\epsilon^{*}$. From
Table I, it is clear that within the same family, as the model size increases,
the CMI value decreases. This shows that larger models have more compact
clusters in the output probability space ${\cal P}([C])$. For the $\Gamma$
value, although the general trend is that within the same family, the $\Gamma$
value increases as the model size gets larger, there does exist an exception.
Note that for the EfficientNet family, the smallest model EfficientNet-B0 has
the largest $\Gamma$ value.

Now turn our attention to the NCMI value. From Table I, it follows that as the
model size within the same family increases, the NCMI value decreases as well.
Even more interesting is the relationship between the NCMI and error rate
$\epsilon^{*}$. Across all models evaluated, as the NCMI value decreases, so
does the error rate $\epsilon^{*}$. To make the relationship between the NCMI
and error rate $\epsilon^{*}$ more transparent, Figure 2 illustrates the
relationship graphically. From Figure 2, it seems that the NCMI and error rate
$\epsilon^{*}$ have a positive linear relationship; indeed, the Pearson
correlation coefficient $\rho$ [26] between them is $\rho=0.9929$, strongly
supporting the former statement. As such, the NCMI value of a DNN can be used
to gauge the prediction performance of the DNN.

To conclude this section, let us draw some analogies. If a DNN is analogized
with a student, then the error rate and NCMI of the DNN can be analogized with
the testing score of the student in an exam and certain trait of the student,
respectively. In a way similar to using the trait of the student to predict
the student’s testing performance, one can also use the NCMI value of the DNN
to predict the DNN’s testing performance.

TABLE I: CMI, $\Gamma$, and NCMI values over the validation set of some pre-
trained models on ImageNet dataset along with their error rate $\epsilon^{*}$,
where the DNNs from the same family are highlighted by the same color.

Models | CMI | $\Gamma$ | NCMI | Error rate $\epsilon^{*}$ | Models | CMI | $\Gamma$ | NCMI | Error rate $\epsilon^{*}$
---|---|---|---|---|---|---|---|---|---
ResNet18 | 0.999 | 9.891 | 0.101 | 0.302 | AlexNet | 1.331 | 9.830 | 0.135 | 0.434
ResNet34 | 0.902 | 9.919 | 0.090 | 0.266 | EfficientNet-B0 | 0.692 | 9.433 | 0.073 | 0.220
ResNet50 | 0.815 | 9.929 | 0.082 | 0.238 | EfficientNet-B1 | 0.661 | 9.114 | 0.072 | 0.213
ResNet101 | 0.779 | 9.948 | 0.078 | 0.226 | EfficientNet-B2 | 0.639 | 9.224 | 0.069 | 0.193
ResNet152 | 0.749 | 9.953 | 0.075 | 0.216 | EfficientNet-B3 | 0.627 | 9.365 | 0.067 | 0.180
VGG11 | 0.959 | 9.899 | 0.096 | 0.296 | Wide-ResNet50 | 0.749 | 9.935 | 0.075 | 0.215
VGG13 | 0.930 | 9.909 | 0.094 | 0.284 | Wide-ResNet101 | 0.734 | 9.937 | 0.073 | 0.211
VGG16 | 0.878 | 9.925 | 0.088 | 0.266 | MobileNet-V3-Small | 1.088 | 9.898 | 0.110 | 0.323
VGG19 | 0.860 | 9.930 | 0.086 | 0.257 | MobileNet-V3-Large | 0.922 | 9.956 | 0.092 | 0.259

$\displaystyle
J_{\mathcal{B}}\left(\lambda,\beta,\theta,\\{Q_{c}\\}_{c\in[C]}\right)$
$\displaystyle=\frac{1}{|\mathcal{B}|}\sum_{(x,y)\in\mathcal{B}}H(y,P_{x,\mathbf{\theta}})+\lambda{1\over|\mathcal{B}|}\sum_{(x,y)\in\mathcal{B}}D(P_{x,\mathbf{\theta}}||Q_{y})-\beta\frac{1}{|\mathcal{B}|^{2}}\sum_{(x,y),(u,v)\in\mathcal{B}}I_{\\{y\not=v\\}}H(P_{x,\mathbf{\theta}},P_{u,\mathbf{\theta}}).$
(22) Figure 2: The error rate vs NCMI value over the validation set of popular
pre-trained models on ImageNet dataset. The sizes of the circles represent the
sizes of respective models in terms of the number of model parameters; the
larger the circle, the larger the model.

##  IV CMIC Deep Learning

The discussions in the above section suggest a new way of learning. In the
learning process, instead of minimizing the average of cross entropy
$\mbox{$\bf E$}_{X}\left[H(P_{Y|X},P_{X})\right]$ alone, one also needs to
look after the NCMI $\hat{I}(X;\hat{Y}|Y)$. This leads to a new form of
learning framework dubbed CMI constrained deep learning (CMIC-DL), which is
described next.

###  IV-A Optimization Problem Formulation

In CMIC-DL, the optimization problem to be solved is as follows:

$\displaystyle\min_{\mathbf{\theta}}~{}$ $\displaystyle\mbox{$\bf
E$}_{X}\left[H(P_{Y|X},P_{X,\mathbf{\theta}})\right]$ s.t.
$\displaystyle\hat{I}(X;\hat{Y}|Y)=r,$ (23)

where $r$ is a positive constant. By interpreting $\hat{I}(X;\hat{Y}|Y)$ as a
rate, and $\mbox{$\bf E$}_{X}\left[H(P_{Y|X},P_{X,\mathbf{\theta}})\right]$ as
a distortion, the above optimization problem resembles the rate distortion
problem in information theory [6, 27, 28]. By rewriting the constraint in
(IV-A), and using the Lagrange multiplier method, the constrained optimization
problem in (IV-A) could be formulated as the following unconstrained one

$\displaystyle\min_{\mathbf{\theta}}~{}$ $\displaystyle\mbox{$\bf
E$}_{X}\left[H(P_{Y|X},P_{X,\mathbf{\theta}})\right]$ $\displaystyle+\lambda
I(X;\hat{Y}|Y)-\beta\mbox{$\bf
E$}\left[I_{\\{Y\not=V\\}}H(P_{X,\mathbf{\theta}},P_{U,\mathbf{\theta}})\right],$
(24)

where $\lambda>0$ is a scalar, and $\beta=\lambda r$.

Note that in view of (15), the CMI $I(X;\hat{Y}|Y)$ in (IV-A) depends on
$P_{\hat{Y}|Y}$, which, for $Y=y$, is the average of $P_{X,\mathbf{\theta}}$
with respect to the conditional distribution $P_{X|Y}(\cdot|y)$ (see (12)). As
such, the unconstrained optimization problem in its form (IV-A) is not
amenable to numerical solutions. To overcome this, we first convert it into a
double unconstrained minimization problem by introducing a dummy distribution
$Q_{y}\in{\cal P}([C])$ for each $y\in[C]$, as shown in the following theorem,
which will be proved in Appendix A.

######  Theorem 2.

For any $\lambda>0$ and $\beta>0$,

$\displaystyle\min_{\mathbf{\theta}}~{}$ $\displaystyle\left\\{\mbox{$\bf
E$}_{X}\left[H(P_{Y|X},P_{X,\mathbf{\theta}})\right]\right.$
$\displaystyle\left.+\lambda I(X;\hat{Y}|Y)-\beta\mbox{$\bf
E$}\left[I_{\\{Y\not=V\\}}H(P_{X,\mathbf{\theta}},P_{U,\mathbf{\theta}})\right]\right\\}$
$\displaystyle=$
$\displaystyle\min_{\mathbf{\theta}}~{}\min_{\\{Q_{c}\\}_{c\in[C]}}~{}\left\\{\mbox{$\bf
E$}[H(P_{Y|X},P_{X,\mathbf{\theta}})+\lambda
D(P_{X,\mathbf{\theta}}||Q_{Y})]\right.$ $\displaystyle\left.-\beta\mbox{$\bf
E$}[I_{\\{Y\not=V\\}}H(P_{X,\mathbf{\theta}},P_{U,\mathbf{\theta}})]\right\\}.$
(25)

In practice, the joint distribution $P(x,y)$ of $(X,Y)$ may be unknown. In
this case, to solve (2) numerically, one may approximate $P(x,y)$ by the
empirical distribution of a data sample (such as a mini-batch in the DL
process)
$\mathcal{B}=\\{(x_{i_{1}},y_{i_{1}}),(x_{i_{2}},y_{i_{2}}),\cdots,(x_{i_{m}},y_{i_{m}})\\}$,
and $P_{Y|X}$ by the one-hot probability distribution corresponding to $Y$.
Accordingly, the objective function in the double minimization (2) can be
approximated by
$J_{\mathcal{B}}\left(\lambda,\beta,\theta,\\{Q_{c}\\}_{c\in[C]}\right)$ shown
in (22) (on the top of the page).

###  IV-B Algorithm for Solving the Optimization in (2)

Having addressed how to approximate the objection function in the double
minimization (2), we are now ready to present an algorithm for solving (2). In
fact, by reformulating the single minimization problem as a double
minimization problem, Theorem 2 lends us an alternating algorithm that
optimizes $\mathbf{\theta}$ and $\\{Q_{c}\\}_{c\in[C]}$ alternatively to
minimize the objective function in (2), given that the other is fixed.

Given $\\{Q_{c}\\}_{c\in[C]}$, $\mathbf{\theta}$ can be updated using the same
strategy as in the conventional DL through stochastic gradient descent
iterations over mini-batches, where the training set is divided into $B$ mini-
batches $\\{\mathcal{B}_{b}\\}_{b\in[B]}$ with each batch of size
$|\mathcal{B}|$. Given $\mathbf{\theta}$, how is $\\{Q_{c}\\}_{c\in[C]}$
updated? This is where differences arise. In view of (12) and (32), the
optimal $\\{Q_{c}\\}_{c\in[C]}$ given $\mathbf{\theta}$ is equal to

$Q_{c}=P_{\hat{Y}|y=c}=\sum_{x}P(x|y=c)P_{x,\mathbf{\theta}},$ (26)

for any $c\in[C]$. Therefore, to update $\\{Q_{c}\\}_{c\in[C]}$ given
$\mathbf{\theta}$, we construct, at each iteration, $C$ mini-batches
$\\{\mathfrak{B}_{c}\\}_{c\in[C]}$ in the following manner: to make
$\mathfrak{B}_{c}$, $\forall c\in[C]$, we randomly sample $|\mathfrak{B}_{c}|$
instances from the training samples whose ground truth labels are $c$. It then
follows from (26) that for any $c\in[C]$, $Q_{c}$ is updated as333To update
$\\{Q_{c}\\}_{c\in[C]}$, we may use momentum to make the updation more stable
and less noisy.

$Q_{c}=\frac{\sum_{x\in\mathfrak{B}_{c}}P_{x,\mathbf{\theta}}}{|\mathfrak{B}_{c}|}.$
(27)

The procedure for solving the optimization problem (2) is now summarized in
Algorithm 1, where we use $(\cdot)^{t}_{c,b}$ to indicate class $c$ at the
$b$-th batch updation during the $t$-th epoch. We also use $(\cdot)^{t}_{c,B}$
as $(\cdot)^{t}_{c}$ whenever necessary, and set
$(\cdot)^{t}_{c,0}=(\cdot)^{t-1}_{c}$.

Algorithm 1 The proposed alternating algorithm for solving the optimization
problem in (2)

0: The training set $\mathcal{T}$, all mini-batches
$\\{\mathcal{B}_{b}\\}_{b\in[B]}$, number of epochs $\mathit{T}$, $\lambda$,
and $\beta$.

1: Initialization:Initialize $\theta^{0}$ and $\\{Q_{c}^{0}\\}_{c\in[C]}$.

2: for $t=1$ to $T_{max}$ do

3: for $b=1$ to $B$ do

4:  _$[$ Updating $\theta$$]$_: Fix $\\{Q_{c,b-1}^{t}\\}_{c\in[C]}$. Update
$\theta^{t}_{b-1}$ to $\theta^{t}_{b}$ by using (stochastic) batch gradient
descent over the loss function
$J_{\mathcal{B}_{b}}\left(\lambda,\beta,\theta^{t}_{b-1},\\{Q_{c,b-1}^{t}\\}_{c\in[C]}\right)$.

5:  _$[$ Updating $\\{Q_{c}\\}_{c\in[C]}$$]$_: Fix $\theta^{t}_{b}$. Construct
mini-batches $\\{\mathfrak{B}_{c}\\}_{c\in[C]}$ from $\mathcal{T}$. Update
$Q_{c,b-1}^{t}$ to $Q_{c,b}^{t}$, $\forall c\in[C]$, according to (27), i.e.,
$\displaystyle
Q_{c,b}^{t}=\frac{\sum_{x\in\mathfrak{B}_{c}}P_{x,\mathbf{\theta}^{t}_{b}}}{|\mathfrak{B}_{c}|}.$
(28)

6: end for

7: end for

8: return model parameters $\theta^{T}$.

##  V Experiment Results

To demonstrate the effectiveness of CMIC-DL and compare it with some state-of-
the-art alternatives, we have conducted a series of experiments. Specifically,
we have performed experiments on two popular image classification datasets,
namely CIFAR-100 [29] and ImageNet [25]. In Subsections V-A and V-B, we
present their respective accuracy results. In Subsection V-C, we explore how
to visualize the concentration and separation of a DNN, which is made possible
by viewing the DNN as a mapping from $x\in\mathbb{R}^{d}$ to $P_{x}$; using
such a visualization method, the concentration and separation of ResNet-56
trained within our CMIC-DL framework are then compared with those of ResNet-56
trained within the standard DL framework.

In the literature, a deep learning process is typically analyzed
experimentally through the evolution curve of its error rate. With our newly
introduced performance metrics, CMI, $\Gamma$ (separation), and NCMI, the
learning process can also be analyzed through the evolution curves of CMI,
$\Gamma$, and NCMI, which show interestingly how the mapping structure in
terms of CMI, $\Gamma$, and NCMI evolves over the course of learning process.
In Subsection V-D, we use ResNet-56 as an example, and illustrate and compare
the evolution curves of CMI, $\Gamma$, NCMI, and error rate within our CMIC-DL
framework vs within the standard DL framework. Lastly, in Subsection V-E, we
evaluate the robustness of models trained within our CMIC-DL framework against
two different adversarial attacks, and show that in comparison with the
standard DL, CMIC-DL improves the robustness of DNNs as well.

###  V-A Experiments on CIFAR-100

CIFAR-100 dataset contains 50K training and 10K test colour images of size
$32\times 32$, which are labeled for 100 classes.

$\bullet$ Models: To show the effectiveness of CMIC-DL, we have conducted
experiments on three different model architectural families. Specifically, we
have selected (i) three models from ResNet family [20], namely
ResNet-$\\{32,56,110\\}$; (ii) VGG-13 from VGG family [21]; and (iii) Wide-
ResNet-28-10 from Wide-ResNet family [23].

$\bullet$ Benchmarks: We evaluate the performance of the DNNs trained via
CMIC-DL against those trained by conventional cross entropy loss (CE), center
loss (CL) [16] which promotes clustering the features, focal loss (FL) [30]
which uses regularization, large-margin Gaussian Mixture (L-GM) loss [31]
which imposes margin constraints, and orthogonal projection loss (OPL) [18]
which imposes orthogonality in the feature space.

$\bullet$ Training settings: We have deployed an SGD optimizer with a momentum
of 0.9, a weight decay of 0.0005, and a batch size of 64. We have trained the
models for 200 epochs, and adopted an initial learning rate of 0.1, which is
further divided by 10 at the 60-th, 120-th and 160-th epochs. To have a fair
comparison, we have reproduced the results of all the benchmark methods using
their respective best hyper-parameters reported in their original papers. In
addition, in Algorithm 1, we set $\\{Q_{c}^{0}(i)\\}_{c\in[C]}=\frac{1}{C}$,
for $i\in[C]$, use $|\mathfrak{B}_{c}|=8$, $\forall c\in[C]$, and also update
$Q_{c,b}^{t}$ using the momentum of 0.9999.

The results are summarized in Table II. As seen, the models trained within our
CMIC-DL framework outperform those trained by the benchmark methods.
Importantly, the improvement is consistent across the models from different
architectural families, showing that CMIC-DL can effectively train DNNs from
different families. As a rule of thumb, compared to the CE method, CMIC-DL
yields DNNs with almost 1.3% higher validation accuracy for the ResNet models.

Furthermore, in Table III we report the NCMI values $\hat{I}(X;\hat{Y}|Y)$,
over the validation set, for the models we trained in Table II, where we use
the notation $\hat{I}_{\text{Loss}}$ to denote the NCMI value when the
underlying DNN is trained using “Loss” method. As observed,
$\hat{I}_{\text{CMIC}}$ has the smallest value compared to the other
counterparts.

In addition, in Table IV, we report the $\lambda^{*}$ and $\beta^{*}$ values
for which we obtained the best validation accuracies. As observed, the
$\lambda^{*}$ and $\beta^{*}$ values are almost the same for all the models.

TABLE II:  The validation accuracies (%) of different models trained by CMIC-DL and different benchmark methods over CIFAR-100 dataset, which are averaged over three different random seeds, and where Bold and underlined values denote the best and second best results, respectively. Loss | Res32 | Res56 | Res110 | VGG13 | WRN-28-10
---|---|---|---|---|---
CL | 70.23 | 72.70 | 74.20 | 74.50 | 80.97
FL | 71.62 | 73.20 | 74.35 | 74.53 | 81.24
LGM | 71.50 | 73.06 | 74.39 | 74.57 | 81.29
OPL | 71.03 | 72.60 | 73.98 | 74.11 | 81.12
CE | 70.90 | 72.40 | 73.79 | 73.77 | 80.93
CMIC | 72.24 | 73.66 | 75.08 | 74.62 | 81.63
TABLE III: The respective NCMI values, measured over the validation set, of the models trained in Table II via different benchmark methods. The values are averaged over thee different runs. Loss | Res32 | Res56 | Res110 | VGG13 | WRN-28-10
---|---|---|---|---|---
$\hat{I}_{\text{CL}}$ | 0.057 | 0.045 | 0.0395 | 0.0395 | 0.0309
$\hat{I}_{\text{FL}}$ | 0.053 | 0.046 | 0.0393 | 0.0399 | 0.0312
$\hat{I}_{\text{LGM}}$ | 0.054 | 0.047 | 0.0390 | 0.0398 | 0.0310
$\hat{I}_{\text{OPL}}$ | 0.056 | 0.050 | 0.0397 | 0.0402 | 0.0314
$\hat{I}_{\text{CE}}$ | 0.057 | 0.053 | 0.0402 | 0.0408 | 0.0317
$\hat{I}_{\text{CMIC}}$ | 0.051 | 0.042 | 0.0382 | 0.0392 | 0.0303
TABLE IV: Hyper-parameters, $\lambda^{*}$ and $\beta^{*}$, that were used in
CMIC-DL in Table II.

Params. | Res32 | Res56 | Res110 | VGG13 | WRN-28-10
---|---|---|---|---|---
($\lambda^{*}$,$\beta^{*}$) | (0.7,0.4) | (0.7,0.4) | (0.7,0.2) | (0.8,0.3) | (0.7,0.4)

###  V-B Experiments on ImageNet

ImageNet is a large-scale dataset used in visual recognition tasks, containing
around 1.2 million training samples and 50,000 validation images.

$\bullet$ Models: We have conducted experiments on two models from ResNet
family, namely ResNet-18 and ResNet-50.

$\bullet$ Benchmarks: We evaluate the performance of CMIC-DL against CE and
OPL.

$\bullet$ Training settings: We have deployed an SGD optimizer with a momentum
of 0.9, a weight decay of 0.0001, and a batch size of 256. We have trained the
models for 90 epochs, and adopted an initial learning rate of 0.1, which is
further divided by 10 at the 30-th and 60-th epochs. In Algorithm 1, we set
$\\{Q_{c}^{0}(i)\\}_{c\in[C]}=\frac{1}{C}$, for $i\in[C]$, use
$|\mathfrak{B}_{c}|=8$, $\forall c\in[C]$, and also update $Q_{c,b}^{t}$ using
the momentum of 0.9999.

The top-$\\{1,5\\}$ accuracies are reported in Table V. As seen, in comparison
with the CE method, CMIC-DL increases the top-1 validation accuracy for
ResNet-18 and ResNet-50 by 0.56% and 0.37%, respectively. The improvement is
also consistent for the top-5 validation accuracy.

The hyper parameters $(\lambda^{*},\beta^{*})$ used in CMIC-DL for ResNet-18
and ResNet-50 are $(0.6,0.1)$ and $(0.6,0.2)$, respectively. The corresponding
NCMI values are $\hat{I}_{\text{CE}}=0.110$ and $\hat{I}_{\text{CMIC}}=0.102$
for ResNet-18, and $\hat{I}_{\text{CE}}=0.091$ and
$\hat{I}_{\text{CMIC}}=0.088$ for ResNet-50.

TABLE V: The validation accuracies (%) of different models trained by CMIC-DL and different benchmark methods on ImageNet dataset. | ResNet-18 | ResNet-50
---|---|---
Method | top-1 | top-5 | top-1 | top-5
CE (Baseline) | 69.91 | 89.08 | 76.15 | 92.87
OPL | 70.27 | 89.60 | 76.32 | 93.09
CMIC | 70.47 | 89.96 | 76.52 | 93.44

###  V-C Concentration and Separation Visualization

In this subsection, we explore how to visualize concentration and separation
of a DNN. Consider the data set CIFAR-100. To visualize concentration and
separation of a DNN in a dimension reduced probability space, we randomly
select three class labels. Restrict ourselves to a subset consisting of all
validation sample instances with labels from the three selected labels. Given
a DNN, feed each validation sample instance from the subset into the DNN, keep
only three logits corresponding to the three selected labels, and then convert
these three logits into a 3 dimension probability vector through the softmax
operation. Following these steps in the indicated order, the DNN then maps
each validation sample instance from the subset into a 3 dimension probability
vector. Further project the 3 dimension probability vector into the 2
dimension simplex. Then the concentration and separation properties of the DNN
for the three selected classes can be more or less visualized through the
projected 2 dimension simplex.

Using the above visualization method, Fig. 3 compares the concentration and
separation properties of ResNet-56 trained within our CMIC-DL framework with
those of ResNet-56 trained within the standard CE framework. From Fig. 3, it
is clear that the three clusters in the case CMIC-DL are more concentrated
than their counterparts in the case of CE, and also further apart from each
other than their counterparts in the case of CE. Again, this is consistent
with the NCMI values reported in Table III.

Figure 3: Visualization and comparison of concentration and separation:
ResNet56 trained via CE vs ResNet56 trained via CMIC, where different shapes
indicate different classes.

###  V-D Evolution of CMI, $\Gamma$, NCMI, and error rate

In this subsection, we analyze and visualize a learning process within either
our CMIC-DL framework or the conventional CE-based DL framework through the
lens of CMI, $\Gamma$, NCMI, and error rate. Fig. 4 shows the evolution curves
of CMI, $\Gamma$, NCMI, and error rate over the validation set during the
course of training ResNet-56 on CIFAR-100 dataset in each case, where the
training setup is the same as that used in Subsection V-A, and we use
$\lambda=0.7$ and $\beta=0.4$ in the case of CMIC-DL.

As seen in Fig. 4a, the CMI value in both CE and CMIC-DL cases is small at the
beginning of the training (epoch zero). This is because at the beginning, all
clusters in the output probability distribution space $\cal{P}([C])$ stick
around together, as shown from the separation distance curve (see Fig. 4b),
and probability distributions within each cluster are not separated at all.
After the training starts and for the first a few epochs, the clusters move
away from each other; during the course of movement, probability distributions
within each cluster move in different speed, and become separated. As such,
both the values of CMI and $\Gamma$ increase. Indeed, this is shown in Fig. 4a
and Fig. 4b. Hereafter, the clusters continue to move away from each other,
while at the same time, probability distributions within each cluster tend to
move together. Thus the $\Gamma$ value continues to increase, while the CMI
value decreases, as shown again in Fig. 4a and Fig. 4b.

The above summarizes the general behaviour of the CMI and $\Gamma$ evolution
curves in both CE and CMIC-DL cases. Let us now examine the differences
between them. From Fig. 4a, it is clear that the CMI evolution curve in the
case of CMIC-DL always remains below its counterpart in the CE case. On the
other hand, as shown in Fig. 4b, although initially the $\Gamma$ value
increases faster in the CE case than in the CMIC-DL case, after the first a
few epochs, the rate of increase in $\Gamma$ value is consistently higher in
the CMIC-DL case than in the CE case to the extent that the $\Gamma$ value in
the CMIC-DL case surpasses its counterpart in the CE case in the late stage of
the learning process.

From Fig. 4c and Fig. 4d, we can see that once the learning process is more or
less stabilized, both the NCMI value and error rate in the CMIC-DL case are
consistently smaller than their counterparts in the CE case. Once again, this
is consistent with our observation in Fig. 2: the smaller the NCMI value, the
lower the error rate. In conjunction with the visualization method discussed
in Subsection V-C, we have created a video available at
https://youtu.be/G0fDwv6o9Ek to illustrate the learning process during the
course of training ResNet-56 on CIFAR-100 dataset in each of the CE and NMIC-
DL cases through the lens of CMI and $\Gamma$, where concentration and
separation are shown for three randomly selected classes, and the evolution
curves of CMI and $\Gamma$ are shown for all classes.

(a) CMI ($I$).

(b) Separation ($\Gamma$).

(c) NCMI ($\hat{I}$).

(d) Error rate ($\epsilon^{*}$).

Figure 4: The evolution curves of (a) CMI, (b) $\Gamma$, (c) NCMI, and (d)
error rate over the course of training ResNet-56 over CIFAR-100 dataset using
CE and CMIC frameworks.

###  V-E Robustness against adversarial attacks

As a by-product, we would expect that DNNs trained within the CMIC-DL
framework are more robust against adversarial attacks, in comparison with
their counterparts trained within the standard CE-based DL framework. This is
because when a DNN is trained within our CMIC-DL framework, its clusters in
its output probability distribution space are more compact, and also further
separated from each other, in comparison with its counterpart trained within
the standard CE-based DL framework. As such, it is harder for an adversary to
craft a perturbation which, when added to a clean sample, would result in an
attacked sample falling into a cluster with a different label. Our purpose in
this subsection is to confirm this by-product. To this end, we have performed
the following experiments.

$\bullet$ Dataset: We have used MNIST dataset [32] comprising of 10-class
handwritten digits.

$\bullet$ Model: We have selected a simple DNN with three convolutional and
one fully connected layers.

$\bullet$ Attacks: Two white-box attacks have been selected, where the
adversary has an access to the gradients of the underlying model.
Specifically, FGSM [3] and PGD attack [5] with 5 iterations were employed with
attack perturbation budgets
$\|\epsilon\|_{\infty}=\\{0.05,0.10,0.15,0.20,0.25,0.30,0.35\\}$.

$\bullet$ Training settings: We have deployed an SGD optimizer with a batch
size of 64. We have trained the models for 15 epochs and adopted an step
learning rate annealing with decay factor of 0.7. The hyper parameters were
selected to be $\lambda^{*}=2$ and $\beta^{*}=9$ in our CMIC-DL framework due
to the fact that the classification task over MNIST dataset is far simpler
than that over CIFAR-100 and ImageNet dataset.

Fig. 5 illustrates the resulting trade-offs between robust accuracy and
perturbation budget. From Fig. 5, it is clear that the DNN trained within the
CMIC-DL framework is more robust against both FGSM and PGD attacks, in
comparison with its counterpart trained within the standard CE-based DL
framework, thus confirming the by-product. In addition, the clean accuracy for
the models trained within the CE-based DL and CMIC-DL frameworks are 99.14%
and 99.21%, respectively, showcasing that the accuracy over the benign samples
is not sacrificed for a higher robust accuracy.

(a)

(b)

Figure 5: The robustness of a simple DNN over MNIST dataset trained within the
conventional CE-based DL and CMIC-DL frameworks against (a) FGSM attack and
(b) PGD attack with 5 iteraions, respectively.

We conclude this subsection by pointing out that although CMIC-DL can improve
the robustness of DNNs trained therein against adversarial attacks, CMIC-DL
itself is not a framework for adversarial training. In our future work, we
will fully address CMIC adversarial training by extending the performance
metrics of CMI, $\Gamma$ (separation), and NCMI to the new concepts of robust
CMI, robust separation, and robust NCMI.

##  VI Conclusion

Viewing a DNN as a mapping from $x\in\mathbb{R}^{d}$ to $P_{x}$, in this paper
we have introduced conditional mutual information (CMI) and normalized
conditional mutual information (NCMI) as new performance metrics of the DNN to
measure the intra-class concentration and inter-class separation of the DNN.
As new performance metrics, CMI and NCMI are in parallel with error rate. We
then have used CMI and NCMI to evaluate and compare DNNs of different
architectures and sizes. It turns out that NCMI and error rate have
essentially a positive linear relationship with their correlation $\geq 0.99$.
As such, the NCMI value of a DNN can be used to gauge the prediction
performance of the DNN.

Based on NCMI, we have then developed a learning framework called CMI
constrained deep learning (CMI-DL) within which the conventional cross entropy
function is minimized subject to a NCMI constraint. An novel alternating
learning algorithm has been further proposed to solve such a constrained
optimization problem. Extensive experiment results consistently show that DNNs
trained within the CMIC-DL framework outperform those trained using the other
DL benchamrk methods discussed in the paper. In addition, with CMI and NCMI as
performance metrics for measuring the concentration and separation of a DNN,
the learning process of the DNN can also be analyzed and visualized through
the evolution of CMI and NCMI.

Open problems include (1) how to extend CMI and NCMI to define concepts of
robust CMI, robust separation, and robust NCMI; (2) how to extend CMIC-DL to
robust CMIC-DL to fully address adversarial training; (3) how to use CMI to
help estimate the conditional probability distribution of $Y$ given $X$; and
(4) the investigation of minimizing NCMI alone without using the standard
cross entropy objective function by modifying a predictor. These problems will
be addressed in the future.

##  Appendix A Proof of Theorem 2

Since $\lambda>0$ and $\beta>0$, it suffices to show that

$I(X;\hat{Y}|Y)=\min_{\\{Q_{c}\\}_{c\in[C]}}\mbox{$\bf
E$}[D(P_{X,\mathbf{\theta}}||Q_{Y})].$ (29)

To this end, we apply (15) to get the following:

$\displaystyle
I(X;\hat{Y}|Y)=\sum_{y}\sum_{x}P(x,y)\left[\sum_{i=1}^{C}P(\hat{Y}=i|x,\mathbf{\theta})\times\right.$
$\displaystyle\left.\ln{P(\hat{Y}=i|x,\mathbf{\theta})\over
P_{\hat{Y}|y}(\hat{Y}=i|Y=y)}\right]$
$\displaystyle=\sum_{y}\sum_{x}P(x,y)\left[\sum_{i=1}^{C}P(\hat{Y}=i|x,\mathbf{\theta})\times\left[\ln{P(\hat{Y}=i|x,\mathbf{\theta})\over
Q_{y}(i)}\right.\right.$ $\displaystyle\left.\left.+\ln{Q_{y}(i)\over
P_{\hat{Y}|y}(\hat{Y}=i|Y=y)}\right]\right]$
$\displaystyle=\sum_{y}\sum_{x}P(x,y)D(P_{x,\mathbf{\theta}}||Q_{y})+\sum_{y}\sum_{x}P(x,y)\times$
$\displaystyle\left[\sum_{i=1}^{C}P(\hat{Y}=i|x,\mathbf{\theta})\ln{Q_{y}(i)\over
P_{\hat{Y}|y}(\hat{Y}=i|Y=y)}\right]$ $\displaystyle=\mbox{$\bf
E$}[D(P_{X,\mathbf{\theta}}||Q_{Y})]$
$\displaystyle+\sum_{y}P(y)\left[\sum_{i=1}^{C}P_{\hat{Y}|y}(\hat{Y}=i|y)\ln{Q_{y}(i)\over
P_{\hat{Y}|y}(\hat{Y}=i|Y=y)}\right]$ $\displaystyle=\mbox{$\bf
E$}[D(P_{X,\mathbf{\theta}}||Q_{Y})]-\mbox{$\bf E$}[D(P_{\hat{Y}|Y}||Q_{Y})]$
$\displaystyle\leq\mbox{$\bf E$}[D(P_{X,\mathbf{\theta}}||Q_{Y})],$ (30)

for any $Q_{y}\in{\cal P}([C]),y\in[C]$, where the inequality above is due to
the nonnegativity of KL divergence. Thus

$I(X;\hat{Y}|Y)\leq\min_{\\{Q_{c}\\}_{c\in[C]}}\mbox{$\bf
E$}[D(P_{X,\mathbf{\theta}}||Q_{Y})].$ (31)

On the other hand, (30) becomes an equality whenever

$Q_{c}=P_{\hat{Y}|y=c},\forall c\in[C].$ (32)

This, together with (30), implies (29), and hence completes the proof of
Theorem 2.

##  Appendix B Other Information Quantities for Separation

In this Appendix, we explore other information quantities which can also be
defined and used as a measure for the inter-class separation of the DNN:
$x\in\mathbb{R}^{d}\to P_{x}$. Specifically, two more information quantities
$\Gamma^{\prime}$ and $\Gamma^{\prime\prime}$ are introduced and compared with
$\Gamma$ defined in (19). Although they are more or less equivalent, $\Gamma$
is more convenient for selecting hyper parameters in our CMIC-DL framework.

###  B-A Information Quantity $\Gamma^{\prime}$

A possible information quantity for measuring inter-class separation can be
defined as follows

$\displaystyle\Gamma^{\prime}=\mbox{$\bf
E$}\left[I_{\\{Y\not=V\\}}D(P_{X}||P_{U})\right],$ (33)

where the cross entropy function $H(P_{X},P_{U})$ in (19) is replaced by the
KL divergence $D(P_{X}||P_{U})$. To connect $\Gamma^{\prime}$ with CMI and
$\Gamma$, we simplify $\Gamma^{\prime}$ as follows:

$\displaystyle\Gamma^{\prime}$ $\displaystyle=\mbox{$\bf
E$}\left[I_{\\{Y\not=V\\}}\sum_{i=1}^{C}P(\hat{Y}=i|X)\ln{\frac{P(\hat{Y}=i|X)}{P(\hat{Y}=i|U)}}\right]$
$\displaystyle=\mbox{$\bf
E$}\left[I_{\\{Y\not=V\\}}\sum_{i=1}^{C}P(\hat{Y}=i|X)\left(\ln{\frac{P(\hat{Y}=i|X)}{P_{\hat{Y}|Y}(\hat{Y}=i|Y)}}\right.\right.$
$\displaystyle\left.\left.+\ln{\frac{P_{\hat{Y}|Y}(\hat{Y}=i|Y)}{P(\hat{Y}=i|U)}}\right)\right]$
$\displaystyle=\mbox{$\bf
E$}\left[I_{\\{Y\not=V\\}}D(P_{X}||P_{\hat{Y}|Y})\right]$
$\displaystyle+\mbox{$\bf
E$}\left[I_{\\{Y\not=V\\}}\sum_{i=1}^{C}P(\hat{Y}=i|X)\ln{\frac{P_{\hat{Y}|Y}(\hat{Y}=i|Y)}{P(\hat{Y}=i|U)}}\right]$
(34) $\displaystyle=\mbox{$\bf
E$}\left[(1-P(Y))D(P_{X}||P_{\hat{Y}|Y})\right]$ (35)
$\displaystyle+\mbox{$\bf
E$}\left[I_{\\{Y\not=V\\}}\sum_{i=1}^{C}P_{\hat{Y}|Y}(\hat{Y}=i|Y)\ln{\frac{P_{\hat{Y}|Y}(\hat{Y}=i|Y)}{P(\hat{Y}=i|U)}}\right]$
(36) $\displaystyle=\mbox{$\bf
E$}\left[(1-P(Y))D(P_{X}||P_{\hat{Y}|Y})\right]$ $\displaystyle+\mbox{$\bf
E$}\left[I_{\\{Y\not=V\\}}D(P_{\hat{Y}|Y}||P_{U})\right],$ (37)

where (35) is due to the fact that $V$ is independent of $(X,Y)$, and (36)
follows from the independence of $(X,Y)$ and $(U,V)$ and the Markov chain
$Y\to X\to\hat{Y}$.

Note that the first expectation in (37) is related to the CMI
$I(X;\hat{Y}|Y)$. Indeed, when $P(Y)$ is equal to a constant, i.e., $1/C$,
which is true in most empirical cases, it follows from (15) that

$\mbox{$\bf E$}\left[(1-P(Y))D(P_{X}||P_{\hat{Y}|Y})\right]=(1-{1\over
C})I(X,\hat{Y}|Y),$

which, together with (37), implies that

$\Gamma^{\prime}=(1-{1\over C})I(X,\hat{Y}|Y)+\mbox{$\bf
E$}\left[I_{\\{Y\not=V\\}}D(P_{\hat{Y}|Y}||P_{U})\right].$ (38)

Plugging (38) into the optimization problem in (IV-A), we get the following
optimization problem

$\displaystyle\min_{\mathbf{\theta}}~{}\mbox{$\bf E$}_{X}$
$\displaystyle\left[H(P_{Y|X},P_{X,\mathbf{\theta}})\right]+\left(\lambda-\left(\beta-\frac{\beta}{C}\right)\right)I(X;\hat{Y}|Y)$
$\displaystyle-\beta\mbox{$\bf
E$}\left[I_{\\{Y\not=V\\}}D(P_{\hat{Y}|Y}||P_{U,\mathbf{\theta}})\right].$
(39)

Thus, if $\Gamma^{\prime}$ was used as a measure for inter-class separation,
then it would cancel out part of the CMI, making the selection of hyper
parameters $\lambda$ and $\beta$ become harder.

###  B-B Information Quantity $\Gamma^{\prime\prime}$

Equations (38) and (B-A) suggest that one might use the following information
quantity as a measure for inter-class separation instead

$\displaystyle\Gamma^{\prime\prime}=\mbox{$\bf
E$}\left[I_{\\{Y\not=V\\}}D(P_{\hat{Y}|Y}||P_{U})\right].$ (40)

In fact, $\Gamma^{\prime\prime}$ has a descent physical meaning in the sense
that it measures the average of distances between the output distributions of
the DNN in response to input sample instances and the centroids of the
clusters with different ground truth labels.

To connect $\Gamma^{\prime\prime}$ with CMI and $\Gamma$, we further simplify
$\Gamma^{\prime\prime}$ as follows

$\displaystyle\Gamma^{\prime\prime}$ $\displaystyle=\mbox{$\bf
E$}\left[I_{\\{Y\not=V\\}}\sum_{i=1}^{C}P(\hat{Y}=i|X)\ln{\frac{P_{\hat{Y}|Y}(\hat{Y}=i|Y)}{P(\hat{Y}=i|U)}}\right]$
(41) $\displaystyle=\mbox{$\bf
E$}\left[I_{\\{Y\not=V\\}}H(P_{X},P_{U})\right]$ $\displaystyle+\mbox{$\bf
E$}\left[I_{\\{Y\not=V\\}}\sum_{i=1}^{C}P(\hat{Y}=i|X)\ln
P_{\hat{Y}|Y}(\hat{Y}=i|Y)\right]$ $\displaystyle=\Gamma+\mbox{$\bf
E$}\left[I_{\\{Y\not=V\\}}\sum_{i=1}^{C}P_{\hat{Y}|Y}(\hat{Y}=i|Y)\ln
P_{\hat{Y}|Y}(\hat{Y}=i|Y)\right]$ (42) $\displaystyle=\Gamma-\mbox{$\bf
E$}\left[(1-P(Y))H(P_{\hat{Y}|Y},P_{\hat{Y}|Y})\right].$ (43)

In the above, (41) follows from (34) and (37), (42) is due to the fact that
$X$ is independent of $V$, and $Y\to X\to\hat{Y}$ forms a Markov chain, and
(43) is attributable to the independence of $V$ and $Y$.

Note again that the second term in (43) is related to the CMI
$I(X;\hat{Y}|Y)$. Indeed, when $P(Y)$ is equal to a constant, i.e., $1/C$,
which is true in most empirical cases, it follows that

$\displaystyle\mbox{$\bf
E$}\left[(1-P(Y))H(P_{\hat{Y}|Y},P_{\hat{Y}|Y})\right]$ (44) $\displaystyle=$
$\displaystyle(1-{1\over C})H(\hat{Y}|Y)$ $\displaystyle=$
$\displaystyle(1-{1\over C})\left[I(X;\hat{Y}|Y)+H(\hat{Y}|X,Y)\right]$
$\displaystyle=$ $\displaystyle(1-{1\over
C})\left[I(X;\hat{Y}|Y)+H(\hat{Y}|X)\right],$

where $H(W|Z)$ denotes the Shannon conditional entropy of the random variable
$W$ given the random variable $Z$, and (44) is due to the Markov chain $Y\to
X\to\hat{Y}$. Combining (44) with (43) yields

$\Gamma^{\prime\prime}=\Gamma-(1-{1\over
C})\left[I(X;\hat{Y}|Y)+H(\hat{Y}|X)\right].$ (45)

Plugging (45) into the optimization problem in (IV-A), we get the following
optimization problem

$\displaystyle\min_{\mathbf{\theta}}~{}\mbox{$\bf E$}_{X}$
$\displaystyle\left[H(P_{Y|X},P_{X,\mathbf{\theta}})\right]+\left(\lambda+\left(\beta-\frac{\beta}{C}\right)\right)I(X;\hat{Y}|Y)$
$\displaystyle+\beta(1-{1\over C})H(\hat{Y}|X)-\beta\Gamma.$ (46)

Thus, if $\Gamma^{\prime\prime}$ was used as a measure for inter-class
separation, then it would further enhance the effect of the CMI, making the
selection of hyper parameters $\lambda$ and $\beta$ become harder as well.

## Acknowledgments

This work was supported in part by the Natural Sciences and Engineering
Research Council of Canada under Grant RGPIN203035-22, and in part by the
Canada Research Chairs Program.

## References

  * [1] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” _nature_ , vol. 521, no. 7553, pp. 436–444, 2015.
  * [2] I. Goodfellow, Y. Bengio, and A. Courville, _Deep learning_. MIT press, 2016.
  * [3] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” _arXiv preprint arXiv:1412.6572_ , 2014.
  * [4] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in _2017 ieee symposium on security and privacy (sp)_. Ieee, 2017, pp. 39–57.
  * [5] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” in _International Conference on Learning Representations_ , 2018.
  * [6] T. M. Cover, _Elements of information theory_. John Wiley & Sons, 1999.
  * [7] G. Hinton, O. Vinyals, J. Dean _et al._ , “Distilling the knowledge in a neural network.”
  * [8] K. Zheng and E.-H. Yang, “Knowledge distillation based on transformed teacher matching,” in _To be submitted to International Conference on Learning Representations_. International Conference on Learning Representations, ICLR, 2024.
  * [9] A. K. Menon, A. S. Rawat, S. Reddi, S. Kim, and S. Kumar, “A statistical perspective on distillation,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 7632–7642.
  * [10] E. Oyallon, “Building a regular decision boundary with deep networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 5106–5114.
  * [11] V. Papyan, “Traces of class/cross-class structure pervade deep learning spectra,” _The Journal of Machine Learning Research_ , vol. 21, no. 1, pp. 10 197–10 260, 2020.
  * [12] V. Papyan, X. Han, and D. L. Donoho, “Prevalence of neural collapse during the terminal phase of deep learning training,” _Proceedings of the National Academy of Sciences_ , vol. 117, no. 40, pp. 24 652–24 663, 2020.
  * [13] J. Zarka, F. Guth, and S. Mallat, “Separation and concentration in deep networks,” in _International Conference on Learning Representations_ , 2020\.
  * [14] Y. Wen, K. Zhang, Z. Li, and Y. Qiao, “A comprehensive study on center loss for deep face recognition,” _International Journal of Computer Vision_ , vol. 127, pp. 668–683, 2019.
  * [15] Z. Shi, H. Wang, and C.-S. Leung, “Constrained center loss for convolutional neural networks,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2021.
  * [16] Y. Wen, K. Zhang, Z. Li, and Y. Qiao, “A discriminative feature learning approach for deep face recognition,” in _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part VII 14_. Springer, 2016, pp. 499–515.
  * [17] C. Qi and F. Su, “Contrastive-center loss for deep neural networks,” in _2017 IEEE international conference on image processing (ICIP)_. IEEE, 2017, pp. 2851–2855.
  * [18] K. Ranasinghe, M. Naseer, M. Hayat, S. Khan, and F. S. Khan, “Orthogonal projection loss,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 12 333–12 343.
  * [19] S. M. Hamidi and E.-H. Yang, “Coded deep learning: framework and algorithms,” _Submitted to IEEE transactions on pattern analysis and machine intelligence_.
  * [20] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [21] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [22] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in _International conference on machine learning_. PMLR, 2019, pp. 6105–6114.
  * [23] S. Zagoruyko and N. Komodakis, “Wide residual networks,” _arXiv preprint arXiv:1605.07146_ , 2016.
  * [24] L. Zhao and L. Wang, “A new lightweight network based on mobilenetv3.” _KSII Transactions on Internet & Information Systems_, vol. 16, no. 1, 2022\.
  * [25] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Advances in neural information processing systems_ , vol. 25, 2012.
  * [26] I. Cohen, Y. Huang, J. Chen, J. Benesty, J. Benesty, J. Chen, Y. Huang, and I. Cohen, “Pearson correlation coefficient,” _Noise reduction in speech processing_ , pp. 1–4, 2009.
  * [27] T. Berger, “Rate distortion theory. englewood clis,” 1971.
  * [28] E.-H. Yang, Z. Zhang, and T. Berger, “Fixed-slope universal lossy data compression,” _IEEE Trans. Inf. Theory_ , vol. 43, no. 5, pp. 1465–1476, 1997.
  * [29] A. Krizhevsky, G. Hinton _et al._ , “Learning multiple layers of features from tiny images,” 2009.
  * [30] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2980–2988.
  * [31] W. Wan, Y. Zhong, T. Li, and J. Chen, “Rethinking feature distribution for loss functions in image classification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 9117–9126.
  * [32] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” _Proceedings of the IEEE_ , vol. 86, no. 11, pp. 2278–2324, 1998.

|  En-Hui Yang (M’97-SM’00-F’08) received the B.S. degree in applied
mathematics from Huaqiao University, China, Ph.D. degree in mathematics from
Nankai University, China, and Ph.D. degree in electrical engineering from the
University of Southern California, USA, in 1986, 1991, and 1996, respectively.
Since June 1997, he has been with the Department of Electrical and Computer
Engineering, University of Waterloo, ON, Canada, where he is currently a
Professor and Canada Research Chair, and the founding Director of the Leitch-
University of Waterloo multimedia communications lab. A co-founder of
SlipStream Data Inc. (now a subsidiary of BlackBerry) and the founder of
BicDroid Inc., he currently also serves as an Executive Council Member of
China Overseas Friendship Association, an Expert Advisor for the Overseas
Chinese Affairs Office of the State Council of China, a Board Governor of the
University of Waterloo, a Board Trustee of Huaqiao University, a member of
IEEE Founders Medal Committee, and advisors for other national and provincial
bodies. His current research interests are: multimedia compression, digital
communications, information theory, source and channel coding, image and video
coding, deep learning, big data analytics, and information security. Dr. Yang
is a recipient of several awards and honors, a partial list of which includes
the 2021 IEEE Eric E. Sumner Award, the prestigious Inaugural Premier’s
Catalyst Award in 2007 for the Innovator of the Year; the 2007 Ernest C.
Manning Award of Distinction, one of the Canada’s most prestigious innovation
prizes; the 2013 CPAC Professional Achievement Award; the 2014 IEEE
Information Theory Society Padovani Lecture; and the 2014 FCCP Education
Foundation Award of Merit. With over 230 papers and more than 230
patents/patent applications worldwide, his research work has benefited people
over 170 countries through commercialized products, video coding open sources,
and video coding standards. He is a Fellow of the Canadian Academy of
Engineering and a Fellow of the Royal Society of Canada: the Academies of
Arts, Humanities and Sciences of Canada. He served, inter alia, as a review
panel member for the International Council for Science; a general co-chair of
the 2008 IEEE International Symposium on Information Theory; an Associate
Editor for IEEE Transactions on Information Theory; a Technical Program Vice-
Chair of the 2006 IEEE International Conference on Multimedia and Expo (ICME);
the Chair of the award committee for the 2004 Canadian Award in
Telecommunications; a Co-Editor of the 2004 Special Issue of the IEEE
Transactions on Information Theory; and a Co-Chair of the 2003 Canadian
Workshop on Information Theory.
---|---
|  Shayan Mohajer Hamidi received the B.Sc. degree from Sharif University,
Tehran, Iran, in 2016, and the MASc. degree from the University of Waterloo,
Waterloo, ON, Canada, in 2018, both in electrical engineering. He was a
research assistant with the CST lab at the University of Waterloo from 2018 to
2020. He is currently working toward his Ph.D. degree in Electrical
Engineering at the University of Waterloo. His current research interests
include machine learning, optimization, information theory.
---|---
|  Linfeng Ye received the B.Sc. degree from Xi’an University of Science and
Technology, Xi’an, China in 2020, in microelectronics, and the MEng. degree
from the University of Waterloo, Waterloo, ON, Canada, in 2021, in electrical
engineering. He is currently working towards his MASc. degree in electrical
engineering at University of Waterloo. His current research interests include
machine learning, and information theory.
---|---
|  Renhao Tan received the Bachelor of Advanced Computing (Honours) degree
from the Australian National University, Canberra, ACT, Australia, in 2021. He
is currently pursuing a MASc. degree in Electrical & Computer Engineering at
University of Waterloo, Waterloo, ON, Canada. His research interests include
machine learning, computer vision and data mining.
---|---
|  Beverly Yang received the BASc degree in Geological Engineering from the
University of Waterloo in 2020. She is currently pursuing a PhD in mining
engineering at the University of British Columbia. Her research interests
include rock mechanics and machine learning.
---|---