# Understanding Neural Networks with Logarithm Determinant Entropy Estimator

Zhanghao Zhouyin1, and Ding Liu1 Manuscript received ; revised ** **, .
Corresponding author: Ding Liu (email: liuding@tiangong.edu.cn). 1School of
Computer Science and Technology, Tiangong University, Tianjin 300387, China

###### Abstract

Understanding the informative behaviour of deep neural networks is challenged
by misused estimators and the complexity of network structure, which leads to
inconsistent observations and diversified interpretation. Here we propose the
LogDet estimator – a reliable matrix-based entropy estimator that approximates
Shannon differential entropy. We construct informative measurements based on
LogDet estimator, verify our method with comparable experiments and utilize it
to analyse neural network behaviour. Our results demonstrate the LogDet
estimator overcomes the drawbacks that emerge from highly diverse and
degenerated distribution thus is reliable to estimate entropy in neural
networks. The Network analysis results also find a functional distinction
between shallow and deeper layers, which can help understand the compression
phenomenon in the Information bottleneck theory of neural networks.

###### Index Terms:

Entropy estimator, Logarithm Determinant, Information Bottleneck.

##  I Introduction

Using Information Bottleneck theory (IB) [1][2] to analyze neural network’s
behaviour has been found applicable in various domain. According to IB Theory,
the learning process can be characterized as finding an optimal representation
that captures most target information, while having the least dependency on
the original signal. However, recent analytical research of neural network’s
informative behaviour achieves highly diversified results, where both works
that approve and oppose are reported, hardly reaching a certain consensus.
Criticisms state that current information estimators are vulnerable to
different model and saturate in high dimension feature space[3][4][5], some
are accused fallacious measure the sample geometry, rather than informative
behaviour[6][7]. These debates and accuses discredit analytic results and
attach importance to revisit informative estimators. Facing the challenge,
this work provides a reliable matrix-based Logarithm Determinant (LogDet)
entropy estimator. Rather than counting numbers of resembling samples, it
recognizes feature-wise dependency and is derived directly from Shannon
differential entropy. Our experiments reveal LogDet estimator is robust to
noise and random perturbation in any dimension of feature space.

Information Bottleneck Theory (IB), as an extension of minimal sufficient
statistics and rate distortion theory [1], describe the learning process as
extracting minimal representation $T$ from input $X$, where it contains the
most relevant information of target $Y$. Thus, the objective of IB is
described with mutual information as:

$\min I(X;T)-\beta I(T;Y).$

Further, when applied in neural networks, it propose that the learning process
can be characterized into two distinct phases by tracking $I(X;T)$ and
$I(T;Y)$: (1) Fitting where Networks extract information from input with
rising $I(X;T)$ and $I(T;Y)$; (2) Compression that models reduce redundancies
and their dependency on input $X$, as a decrease of $I(X;T)$[2]. Therefore, by
tracking mutual information of NNs, we clearly understand the training
behaviour from an informative aspect and therefore “Opening the Black Box of
Neural Networks”[3]. However, some theoretical problems still exist in entropy
estimation. To any neural network, each layer output $T_{i}$ is determined
with the sure input $X$. When considering $T$ and $S$ are both discrete, then
the theoretical mutual information $I(X;T)=H(T)-H(T|X)$ is a constant since
neural networks mapping samples bijectively. While if we assume the output
feature space is continuous, then $I(X;T)$ would become positive infinity.
Moreover, high dimensional data itself is unavoidably ill-posed and
degenerated distributed, which complicates the definition of informative
functionals and their estimation. These problems challenge many estimators,
while some other approximation methods (i.e. Kernel-Density Estimation,
Stochastic Binning) using additional noise to produce Stochastic Neural
Networks, or dividing samples into discrete groups to avoid bijection (i.e.
Binning, K-nearest). Unfortunately, Most methods revealed limitations such as
sensitive to dimensions, samples variance and hyper-parameters[3][7][4].

Entropy estimation is a classical problem of information theory. Prior works
have raised many estimators with LogDet function, such as uniformly minimum
variance unbiased (UMVU)[8], bayesian method[9][10] and more recently
nonparanormal estimator [11]. However, these methods often involve complex
optimization, which is unsuitable for synchronous estimation in machine
learning. Recent proposed matrix-based estimator $\alpha$-$R\acute{e}nyi$
entropy[12][13] achieve good performance in deep neural
networks[14][15][16][17]. However, its still lack of verification when applied
in different models. In this work, we fully verified our proposed method. The
proposed LogDet estimator measuring entropy with continuous samples with added
noise. Such a continuity hypothesis enables us to directly approximate
differential entropy. Also, adding noise avoids the singular covariance matrix
in LogDet estimator, which commonly appear when with finite samples or
strongly correlated features. Further, we show LogDet estimator’s performance
by estimating entropy and mutual information. Then compare it with other
commonly used methods. Results demonstrate that it is accurate, noise-robust
and applicable to any dimension of features space, without worrying about
saturation.

Apart from the entropy estimation problem, current analytical works report
inconsistent conclusion about IB compression. Where its existence is observed
in [4][18] and absence is also argued in [19][6][20]. Moreover, the promised
connection of compression and generalization is also unclear. These results
are recently accused corrupted by misused estimators[7][3], and other works
argue that some meaningful results are connected with geometrical
clustering[17][6][7], rather than mutual information behaviour. Most of all,
these debates stuck in arguing the existence of compression, but lack of
sufficient verification of estimators. This prohibits the soundly
investigation of compression. Some works try to provide theoretical
analysis[21], however, the area still lacks of empirical understanding. Also
in need is the interpretation of compression, which is still unclear within
current literatures.

After verifying the LogDet estimator, we believe one difficulty that prevents
the understanding of IB behaviour comes from the complexity of $I(X;T)$
itself. Because even with a correct estimation of multi-layer neural networks,
directly interpreting $I(X;T_{i})$ of layer $i$ involve considering all former
layers. Then the deeper $T_{i}$ we like to discuss, the more complex its
behaviour will be. Taking the second layer as an example, in this case,
whatever we observe in $I(X;T_{2})$ is, it is difficult to confirm whether it
is caused by the second or the first layer. Thus, we propose the concept of
Layer Transmission Capacity (LTC) which estimates neural networks’ capacity to
transmit information in a layer-wise fashion. Based on this, we can discuss
mutual information within a single layer. Our results give a direct
interpretation of observed compression and provide a solid basis to its
understanding.

Our main contributions can be concluded as bellow:

  * -

We propose the LogDet entropy estimator, which resolve the inaccurate analysis
of misused estimators in previous works, and proved can be applied in any high
dimensional feature spaces of deep neural networks.

  * -

We compare different entropy estimators, which shows LogDet is robust to
noise, large variance of samples and saturation problems.

  * -

We proposed Layer Transmission Capacity (LTC) to provide a more precise mutual
information of each layer in IB theory. It reveals IB compression is mostly an
effect of the first layer, which has distinct behaviour against other layers
in neural networks.

##  II Method

Differential Entropy is the natural extension of discrete Shannon Entropy
$H(X)=-\sum_{X}p(x)\log{p(x)}$. By replacing probability function with its
density and integrating over x, the definition of differential entropy is:
$h(X)=-\int_{X}f(x)\log{f(x)}\rm{dx}$. Estimating differential entropy has
long been an active area in Information Theory [22][23][10][8]. In principle,
the differential entropy of multivariate Gaussian distribution can be
represented by the logarithm determinant of its covariance matrix. Here we
first introduce a approximation method LogDet for estimating differential
entropy and define basic information measurements in multivariate Gaussian
distribution. Next, we extend this method in multi -signal situation and
proposed the layer-wise estimation Layer Transmission Capacity of neural
networks. Finally, we interpret LogDet estimator from the perspective of
Coding Length Function[24], and discuss the parameter setting.

###  II-A Approximating Differential Entropy with LogDet estimator

The Logarithm Determinant (LogDet for short) is considered strongly connected
with information entropy[25]. To a multi-variable Gaussian Distribution
$X\sim{N}(\mu,\Sigma_{X})$, the Shannon differential entropy can be
represented by the log determinant of its covariance matrix as:

$\displaystyle\mathbf{h}(X)$
$\displaystyle=\frac{1}{2}\log\det\Sigma_{X}+\frac{d}{2}(\log 2\pi+1)$ (1)
$\displaystyle=\frac{1}{2}\sum_{i}^{d}\log\lambda_{i}+\frac{d}{2}(\log
2\pi+1)$

where $d$ is the dimension of variables and $\lambda_{i}$ denote the $ith$
eigenvalue of its covariance matrix. Obviously the second term is constant and
the measurement is determined by $\log\det{\Sigma_{x}}$. Later on, by
extending the generalization of Strong Sub-additive (SSA) condition[26], it
reveals that all matrix monotone functions $f$ would satisfy the generalized
SSA inequality: $Trf(V_{AC})+Trf(V_{BC})-Trf(V_{ABC})-Trf(V_{C})\geq{0}$. When
having $\log{x}$ as $f(x)$, differential $R\acute{e}nyi$’s $\alpha$-order
entropy [27] of multivariate Gaussian is also equivalent to the $\log\det$
representation[28], which is:

$\displaystyle\mathbf{h}_{\alpha}(X)$
$\displaystyle=\frac{1}{1-\alpha}\log{\int d^{n}xp_{A}(x)^{\alpha}}$ (2)
$\displaystyle=\frac{1}{2}\log\det\Sigma_{X}+\frac{n}{2}(\log
2\pi+\frac{1}{\alpha-1}\log{\alpha}).$

Since $\alpha$-$R\acute{e}nyi$ entropy is the generalization of a family of
entropy measurements, the above expression reveals the potential of LogDet
function as a universal entropy measurement to multivariate random Gaussian
variables. Indeed, $\log\det$ function has been acting as an ideal matrix-
based estimator for decades, applications such as Jensen-Bregman LogDet
Divergence [29], Alpha–Beta and Gamma Divergences[30] show the good properties
to semi-definite matrices as robust to noises, positional bias and outlines,
while maintaining the computational efficiency.

Despite the advantages, LogDet differential entropy is faced with two
challenges: (1) First, when having finite but strongly correlated variables,
the covariance matrix is often singular which leads to zero determinant.
Similar cases often appear in current machine learning, where samples tend to
be sparse and strongly correlated. (2) Denoting the continuous model’s input
signal as $X$ and intermediate variable as $T$, then $T$ is continuously
determined by $X$. When $X$ is settled, the conditional probability density of
each determined variable $T$ becomes a delta function. Therefore, the
conditional entropy $H(T|X)$ would be negative infinite and lead to a positive
infinite $I(X;T)=H(T)-H(T|X)$[31][6][5]. This proves that estimating
differential entropy directly in neural networks only returns the bias of
estimators, rather than meaningful information entropy.

Prior works of entropy estimators provide a solution. By adding independent
Gaussian noise $z\sim{N(0,I)}$ to each deterministic intermediate state $T$ as
$\hat{T}=T+z$, we have a stochastic $\hat{T}$ to any given $X$. In such cases,
deterministic problems is avoided and we can apply $\log\det$ operation to
covariance matrix $Cov(\hat{T})=\Sigma+I$ to estimate its entropy. To
eliminate artificial bias brings by added noise, rather than scale down
noise’s variance (which will cause a negative estimated entropy), we enlarge
the original covariance matrix by an expanding factor $\beta$ to decrease the
comparable weight of added noise. This leads to the following definition:

$Definition$ Given $n$ samples $X=[x_{1},x_{2},...,x_{n}]$ where each sample
have $d$ dimensional features that $X\in{\mathbb{R}^{n\times{d}}}$, its
covariance matrix is calculated by $\Sigma=\frac{X^{T}X}{n}$, the LogDet
entropy can be represented as:

$\mathbf{H}_{D}(X)=\frac{1}{2}\log\det(I+\beta\Sigma)=\frac{1}{2}\log\det(I+\beta\frac{X^{T}X}{n})$
(3)

where $\beta$ is the scaling parameter. Here, we show such definition describe
the differential entropy. By performing SVD to $X$, Eq.3 is equivalent to:

$\displaystyle\mathbf{H}_{D}(X)$
$\displaystyle=\sum_{i=1}^{k}\frac{1}{2}\log(1+\beta{\lambda_{i}})$ (4)
$\displaystyle=\sum_{i=1}^{k}\frac{1}{2}\log(\frac{1}{\beta}+\lambda_{i})+\frac{k}{2}\log\beta$

where $k=rank(X)$ and $\lambda_{i}$ denotes the $ith$ eigenvalue of
$\frac{X^{T}X}{n}$. We see the first term of Eq.4 approximate the dominant
term in Eq.1, which is the differential entropy of multivariate Gaussian
random variables (RVs). The second term is a constant $\frac{1}{2}\log\beta$
weighted by $k$, which also change simultaneously with differential entropy.
Therefore, LogDet is capable to estimate entropy of multivariate Gaussian RVs
by approximating the differential entropy. (Details can be found in Appendix.
A).

Showing the approaching property of LogDet estimator, we can further define
joint LogDet entropy as:

$Definition$ Given samples sets
$X_{1}\in\mathbb{R}^{n\times{d_{1}}}$,$X_{2}\in\mathbb{R}^{n\times{d_{2}}}$,
if denote $Z=[X_{1},X_{2}]\in\mathbb{R}^{n\times(d_{1}+d_{2})}$ as a
concatenation of $X_{1}$ and $X_{2}$, and its covariance matrix is
$\Sigma_{Z}=\begin{pmatrix}\Sigma_{1}&F\\\ F^{T}&\Sigma_{2}\end{pmatrix}$,
where $\Sigma_{1}$ and $\Sigma_{2}$ is covariance matrices of $X_{1}$ and
$X_{2}$ respectively, and $F$ equals to $\frac{X_{1}^{T}X_{2}}{n}$, the joint
LogDet entropy of $X_{1}$ and $X_{2}$ is:

$\displaystyle\mathbf{H}_{D}(X_{1},X_{2})$
$\displaystyle=\frac{1}{2}\log\det(I+\beta\Sigma_{Z})$ (5)
$\displaystyle=\frac{1}{2}\log\det(I+\beta\frac{Z^{T}Z}{n})$

.

Some prior works summarize determinant inequalities from the aspect of
information theory[32][33][34]. These inequalities gives a theoretical
background to define other LogDet information measurements.

$Proposition$ Given random variables $X_{1}$ and $X_{2}$, their joint LogDet
entropy satisfies the following inequalities (proof see Appendix. A):

$\displaystyle\mathbf{H}_{D}(X_{1},X_{2})$
$\displaystyle\leq\mathbf{H}_{D}(X_{1})+\mathbf{H}_{D}(X_{2})$ (6)
$\displaystyle\mathbf{H}_{D}(X_{1},X_{2})$
$\displaystyle\geq\max(\mathbf{H}_{D}(X_{1})+\mathbf{H}_{D}(X_{2}))$ (7)

Therefore, we can further compute conditional entropy and mutual information
as:

$\displaystyle\mathbf{H}_{D}(X_{1}|X_{2})=\mathbf{H}_{D}(X_{1},X_{2})-\mathbf{H}_{D}(X_{2})$
(8)
$\displaystyle\mathbf{I}_{D}(X_{1};X_{2})=\mathbf{H}_{D}(X_{1})+\mathbf{H}_{D}(X_{2})-\mathbf{H}_{D}(X_{1},X_{2})$
(9)

.

(a) $I_{D}(X;Y)$ via Dimensions with n=128 (b) $I_{D}(X;Y)$ via Sample Number
with d=1000

Figure 1: Experiment of $I_{D}(X;Y)$ with multivariate Gaussian samples. $d$
is feature dimensions, and correlation factor $\rho$ grows from -1 to 1. We
set $\beta=1$ in this case. All curves track the theoretical tendency of
mutual information.

We test Mutual information $I_{D}(X_{1};X_{2})$ using a experiment in Fig. 1.
$X$ and $Y$ are $d$ dimensional multivariate RVs that obey $N(0,I)$, while
variables between $X$ and $Y$ is controlled correlated with factor $\rho$ that
$Cov(X,Y)=\rho{I}$. In other words, the covariance between RVs of $X$ and
those of $Y$ is $\rho$. We alter $\rho$ from -1 to 1, and estimate the mutual
information $I_{D}(X;Y)$ with LogDet estimator. Similar experiment is adopted
in [17][35]. Besides, we repeat this experiment on samples with different
dimension $d\in${3,100,1000}. Experimental results show that the estimated
$I_{D}(X;Y)$ is not exactly theoretical mutual information, and the minimum
value is also not zero when $\rho=0$. But clearly, all curves follow the
tendency of theoretical mutual information, leading to a concave shape. The
result also suggest LogDet estimator can be applied in high-dimensional space
without saturation problems. In Fig. 1(b), we see adding more samples improves
the result, whose curve shows deeper reduction than those with fewer samples.
In all, we can use LogDet estimator to approximate mutual information.

###  II-B Multi-signal Extension

Beside the basic information measurements, LogDet estimator can be extended to
other complex data structures. For example, for multiple signals
$X=[X_{1},X_{2},...,X_{k}]$, where each $X_{i}$ is a multivariate RV. This
condition is common neural networks, such as multi-channel convolutional
neural networks, since features in different channel are often calculated by
different parameters, therefore, instead of regarding the whole feature map as
a multivariate RV, it is better to split apart features from different
channels as multiple signals, and calculate their joint entropy. Here we will
show LogDet estimator can be naturally extended to such multiple-signal
conditions by multivariate joint entropy.

$Definition$ Given the variable set $Z=[X_{1},X_{2},X_{3},...,X_{k}]$ where
$X_{i}$ is a multivariate RV, denote the covariance matrix between $X_{i}$ and
$X_{j}$ as $\Sigma_{i,j}$, we then have the covariance matrix of $Z$ equals to
$\Sigma_{1,2,...,k}=\begin{pmatrix}\Sigma_{1,1}&\Sigma_{1,2}&\cdots&\Sigma_{1,k}\\\
\Sigma_{2,1}&\Sigma_{2,2}&&\vdots\\\ \vdots&&\ddots\\\
\Sigma_{k,1}&\cdots&&\Sigma_{k,k}\end{pmatrix}$, the joint entropy of
$[X_{1},X_{2},...,X_{k}]$ is given by:

$\displaystyle\mathbf{H}_{D}(X_{1},X_{2},...,X_{k})=\frac{1}{2}\log\det(I+\beta\Sigma_{1,2,...,k})$
(10)

The definition above directly extend joint entropy with LogDet estiamtor.
However, the size of covariance matrix of multiple signal
$Cov(Z)=\Sigma_{1,2,...,k}\in\mathbb{R}^{kd\times{kd}}$ can rapidly increase
by signal number $k$ and feature dimension $d$, which is expensive to perform
$\log\det$ operation on and to save. Luckily, by applying $\log\det$’s
commutative property[24], it suggests an equivalent approach to calculate the
multi-signal entropy. Having $n$ estimating samples for each variable $X_{i}$,
the above expression equals to:

$\displaystyle\mathbf{H}_{D}(X_{1},X_{2},...,X_{k})$
$\displaystyle=\frac{1}{2}\log\det(I+\beta\frac{Z^{T}Z}{n})$
$\displaystyle=\frac{1}{2}\log\det(I+\beta\frac{ZZ^{T}}{n})$
$\displaystyle=\frac{1}{2}\log\det(I+\frac{\beta}{n}\sum_{i=1}^{k}X_{i}X_{i}^{T})$

Under this expression, the estimated matrix size become independent to
$X_{i}$’s feature dimension and signal number, limiting its size within
$\mathbb{R}^{n\times{n}}$. Therefore, the complexity only grows linearly with
signal numbers $k$. Moreover, this interpretation suggests covariance matrix
in LogDet entropy is equivilent to the sum of $X_{i}X_{i}^{T}$. Which is the
pair-wise similarity matrix that contains samples’ inner-production as
components. This means we can naturally introduce kernel methods to adjust
LogDet estimator to more complex input distribution. After all, we extend the
LogDet estimator to multi-signal situation and discover a efficient approach
in application.

Under the above expression, the following properties can be derived with
Minkowski determinant theorem and the quality of $\log\det$ function to
positive semi-definite matrix(proof See Appendix. A).

$Corollary$ Segmenting the variable set $X=[X_{1},X_{2},...,X_{k}]$ into
arbitrary number of subsets, denoted as [$X^{(1)},X^{(2)},...,X^{(t)}$], where
each allocated $X^{(i)}$ is the complement to the concatenation of the rest as
[$X^{(1)},X^{(2)},...,X^{(i-1)},X^{(i+1)},...,X^{(t)}$] with respect to
original variable set $X=[X_{1},X_{2},...,X_{k}]$, the following inequalities
hold:

$\displaystyle\mathbf{H}_{D}(X)$
$\displaystyle\leq\mathbf{H}_{D}(X^{(1)})+\mathbf{H}_{D}(X^{(2)})+...+\mathbf{H}_{D}(X^{(t)})$
(11) $\displaystyle\mathbf{H}_{D}(X)$
$\displaystyle\geq\max(\mathbf{H}_{D}(X^{(1)}),\mathbf{H}_{D}(X^{(2)}),...,\mathbf{H}_{D}(X^{(t)}))$
(12)

Where the first equality holds i.f.f. variables between each subset is fully
independent that has zero covariance, which means $\Sigma_{i,j}=0$ for any
$X_{i}$ and $X_{j}$ ($i\neq{j}$) that belong to different subsets. The second
inequality holds i.f.f. all the rest variables are fully determined, providing
no information to the largest one. We also notice that, when with equal
$X_{i}$ and $X_{j}$, the joint estimation only increase with a constant
$k\log{2}$ where $k$ is the matrix rank. It can be interpreted as representing
the same variables only requires additional $k$ bits for marking the
equivalence of variables between $X_{i}$ and $X_{j}$. These extended
inequalities and equality conditions are fully aligned with the definition of
information entropy, therefore, providing a theoretical guarantee for our
estimation.

Figure 2: Estimating activation function’s saturating effect using LogDet
mutual information estimator. Input $X$ is weighted by $w$ and operated by
$tanh$ and $ReLU$ respectively. More value would saturated in $tanh$ when $w$
grows, revealed as decrease in $I(X;T)$. The experiment is conducted in
different dimension $d$, and using 3000 samples.

###  II-C Layer Transmission Capacity

The computing process of neural networks from input to intermediate
representation $X\xrightarrow[]{}T$ often involves operations of numerous
layers, it is hard to locate which layer is responsible for changes in
$I(X;T)$. Therefore, we propose an estimation of Layer Transmission Capacity
to simplify the discussion of mutual information behaviour. By measuring
mutual information of each layer’s input and output, we can quantify how much
information is transmitted through. This enables the discussion of each
layer’s behaviour without most of interference, here we give its intuitive
understanding from the perspective of Gaussian Channels.

To any neural network, each layer has two main components: (1) Parameters
$\theta$ (2) Activation function $g(.)$. During computations, Input $T$ is
weighted by the parameters $\theta$, then activated by $g(.)$ as
$L(T)=g(\theta\cdot{T})$. Thereafter, $\theta$ is updated according to the
gradient feedback. Assuming we have an infinite number of input $T$ that
depend on a continuous distribution of task data, then each layer can be
modeled with Gaussian Feedback Channel[25], where model parameters $\theta$
function as an explicit encoder, and the activation function as special
’narrow’ channel that has structured interruption to encoded words
$\theta\cdot{T}$. Such interruption solely depends on the non-linearity of
each activation function and can therefore considered as coloured noise, which
is not independent Gaussian but with a particular Phase Structure. Naturally,
the back-propagated gradients are feedback to the encoder $\theta$, guiding
the codebook design.

Under the above description, assuming we have a continuous input signal $T$,
layer $L$’s simulated Channel Capacity is defined as the largest amount of
information transmitted through. which is:

$\displaystyle C$ $\displaystyle=\max_{p(T):E(T^{2})\leq{P}}I(T;L(T))$ (13)

where $L(T)$ is the output of layer $L$, P is variance constrain of channel
input $T$. This tells that, with proper distribution $p(T)$, we can precisely
calculate how much information layer $L$ can transmit as most. Since we only
need to consider $T$ from the task distribution, and the explicit encoder
$\theta$ can be optimized through training. Therefore, the Capacity of $L$
should be maximized concerning $\theta$ and $p(T)$. When the task distribution
$p(T)$ is settled (by choosing the training data in DNNs), we can track the
dynamic of such transmitted capacity as a function to parameter $\theta$. We
denote this function as Layer Transmission Capacity (LTC):

$Definition$ Given sufficient observing samples $X$, to a neural network layer
$L$, whose input denoted as $T$ and output as $L(T)$, the Layer Transmission
Capacity of $L$ is given by:

$LTC(L)[\theta]=I_{D}(T;L(T;\theta))$ (14)

It is worth notice that, LTC cannot fully remove the influence of the former
layer, because $I_{D}(T;L(T))$ can still be affected by $H_{D}(T)$, which is
determined by the former layer. However, it remove most elements that
irrelevant to layers’ operation and concentrate the discussion in the current
layer, which achieve us a better understanding. Also when the task is settled,
each layer’s input $T$ will have a converged distribution, so function
$LTC(L)[\theta]$ have an upper bond when maximized with $\theta$. This ensures
the value is varied within a constraint range, rather than change infinitely.
After all, we can use Layer Transmission Capacity function to evaluate the
transmitting behaviour of each layer.

###  II-D Relation to Coding Length Function

Differential entropy gives the amount of binary bits to encode continuous
variables, however in practice, it is difficult to represent arbitrary
continuous RVs with finite codes. Coding Length Function provides a
substitution[24], which measures the distortion rate from Rate Distortion
Theory. Our approach can also be considered as an extension of the Coding
Length Function. Here we briefly review the Coding Length Function, and
interpret our LogDet estimator to show how $\beta$ is related to reducing
distortion ratio. This discussion also helps to select a proper $\beta$.

#### Coding Length Function

Coding length function is raised to estimate Distortion Rate from Rate
Distortion Theory[25]. Despite the difficulty to encode continuous RVs into
finite representation, it is still proved approachable by allowing a small
error as encoding distortion. Then, the distortion rate function gives the
minimum number of bits (or distortion rate) of such codes. Formally, the
distortion rate is defined as:

$R(D)=R^{(I)}(D)=\min_{f:d(\mathbf{x},\mathbf{\hat{x}})=D}\mathbf{I}(\mathbf{x};\mathbf{\hat{x}})$

where $\mathbf{x}$ is the original signal, and $\mathbf{\hat{x}}$ is its
distorted representation, with distortion $D$. Then, the Coding Length
Function of $n$ samples from arbitrary distribution
$\mathbf{W}=[\mathbf{w}_{1},\mathbf{w}_{2},...,\mathbf{w}_{n}]$ where
$\mathbf{W}\in\mathbb{R}^{n\times{d}}$ is given by:

$\displaystyle
R(W)=\log_{2}(\frac{vol(\hat{W})}{vol(z)})=\frac{1}{2}\log\det(I+\frac{d}{n\epsilon^{2}}W^{T}W)$
(15)

$\epsilon$ is the allowable distortion. This formula is achieved by dividing
the volume of distorted samples
$vol(\hat{W})=\sqrt{\det(\frac{\epsilon^{2}}{d}I+\frac{W^{T}W}{n})}$ with the
noise sphere $vol(z)=\sqrt{\det(\frac{\epsilon^{2}}{d})}$, and labeling the
divided spheres with binary codes. Therefore, each sphere can be localized
with their position code, and represent samples within the distortion radius.
The length of such code gives the smallest number of bits to represent the
distorted sample.

(a) $H_{D}(X)$ Saturation Test (b) $H_{D}(X)$ Precision Test

Figure 3: Estimator’s saturation test (a) and precision test (b) comparing
with different estimation methods (REYI[13, 17], KNN[36, 37], KDE[5], BIN[38]
and ours LD(LogDet)). In Saturation Test, samples are varied with increasing
variance (from 0 to 1). In Precision test, samples $X$’s variance is fixed in
1, the curve is plotted along the inner correlation factors $\rho$, where
$Cov(x_{i},x_{j})=\rho$ for $i\neq{j}.$ For comparison, all estimated result
is unified in range of (0,10). In (a), BIN, KDE saturating when increased the
dimension. And in (b), only REYI, KNN and our method can precisely distinguish
with correlated variables in all dimensional cases, but comparably, REYI and
our method is observed better preciser increase than KNN. In all, LogDet can
theoretically estimate variate samples with high precision in all dimensional
cases.

It is easy to find the consensus between Eq.15 and Eq.3. Let
$\beta=\frac{n}{\epsilon^{2}}$, then increasing $\beta$ to eliminate bias of
LogDet entropy is equivalent to reducing distortion $\epsilon$ in $R(D)$.
Since $R(D)$ as a measure of distortion rate, should approach the Shannon
Entropy $\mathbf{H}$ when taking zero as limit of $\epsilon$. This proves the
plausibility of LogDet’s approximation method, explaining why an expand factor
$\beta$ is essential for accurate measuring. Moreover, prior analysis of
coding length function reveals it that, as a tight upper bond of coding length
of arbitrary distributed variables, it can be applied in a wide range of
machine learning tasks and datasets[24][39][40][41]. This shows LogDet’s
potential as a entropy measurement in realworld benchmarks.

The Distortion Rate for multivariate Gaussian $R(D)=\log\det\frac{\Sigma}{D}$
requires that D smaller than any eigenvalue of covariance matrix
$\Sigma$[25][24]. This indicates a small $D$ as well as a large $\beta$ in
LogDet estimator. In this work, we follow the prior applications in Coding
Length Function[40][42] where $D=\frac{\epsilon^{2}}{d}$, which successfully
measures the compactness of samples. Therefore, we let $\epsilon=0.1$ so
$\beta=100d$ in LogDet entropy $H_{D}(X)$. In $I_{D}(X_{1};X_{2})$, $\beta$ is
set as $100(d_{1}+d_{2})$, where $d_{1}$, $d_{2}$ are feature dimensions of
$X_{1}$, $X_{2}$ respectively.

We adopt the test of activation function in [5] to verify whether LogDet under
this setting can be used to estimate mutual information in neural networks.
Since large variables value is saturated after activated by doule-saturating
function $tanh$, the model output’s connection with it of input is reduced.
Such reduction of connection can be revealed as a decrease of $I(X;T)$ in
neural networks. In this test, we enlarge the weight of a single fully
connected layer, and expect LogDet estimator can track such decreases, which
validates its capacity to estimate informative behaviour in neural networks.
In Fig.2, such decreases occur in all tested feature dimension with $tanh$,
and absents when with $ReLU$ activation. This is consistent with our
expectation. It also shows that $LogDet$ mutual information estimator would
not be affected by the sample variance, but solely reveal the change in
variable dependency. In which case, we don’t need to concern the disruption of
variance changes during training. However, the result suggests a problem in
estimating high dimensional samples. As a method relies on matrix estimation,
the accuracy of $LogDet$ estimator and other matrix-based methods heavily
influenced by number of estimated sample. In this paper’s experiments, this
test shows 3000 samples is sufficient. As for the limitation of matrix-based
methods, we leave the discussion in Appendix. B.

##  III Experiments

Experiments are divided into three parts. First, we use generated Gaussian
samples to investigate LogDet’s property when estimating high dimensional
samples. Second, to show our method is applicable in real-world datasets, we
design an experiment with Machine Learning benchmarks MNIST, CIFAR10 and
STL10. Then, we utilize LogDet estimator for Information Bottleneck analysis
in neural networks, and compare the result with Layer Transmission Capacity to
give an empirical interpretation of compression phenomenon and Networks’
information behaviour. In all estimation, each variable is shifted with mean
zero.

The first two experiments also involve other commonly adopted estimators.
Methods we compare include: the improved version[17] of recent proposed
multivariate matrix based $R\acute{e}nyi$’s $\alpha$-order entropy(REYI) [13];
the high dimensional adaptation[37] of classical K-nearest neighbour(KNN)
method[36]; Kernel Density Estimation(KDE) method[5] and the Binning(BIN)
method[38].

Figure 4: Entroph Estimation using Machine Learning Image Benchmarks. Each row
corresponds to one of three test datasets: MNIST, STL and CIFAR10, the columns
is divided by the variance ratio $\sigma$ of added noise, from 0 to 1. All the
value is unified between 0 and 1, and shifted in different vertical positions.
The estimated value should decline as more single kind of sample appeared in
the sample set.

###  III-A LogDet as Entropy Estimator

Recall the fact that the definition of other informative functionals are based
on LogDet entropy $H_{D}(X)$. Therefore, it is essential to validate whether
LogDet can measure information entropy in different situation. Consider a
multivariate random variable $X\sim{N(0,\Sigma)}$, its differential entropy
$H(X)=\frac{1}{2}\log\det\Sigma+\frac{d}{2}(\log 2\pi+1)$ is controled by two
main components (determine diagonal and non-diagonal value of $\Sigma$
respectively): Variance and variables’ Correlation. Therefore, we verify
LogDet estimator’s performance with different variances of independent
Gaussian and changing Correlations of fix-variance samples. Moreover, this
experiment gives a comparable test with other methods. Although each estimator
often defines Mutual Information with different approximation methods (Such as
the hypothesis of $I(X;T)=H(T)$, and KL-divergence definition of MI), the
accurate entropy measure is still a common requirement of most estimators, in
which case, by comparing their performance in this test, different estimators
can be equally verified.

(a) IP of the FCN trained on CIFAR10 (b) IP of the FCN trained on MNIST

Figure 5: Information Plane of a fully connected neural network with
(784/3072-1024-20-20-10) neurons in each layer, trained on MNIST and CIFAR10
dataset. We sampled 100 from 500 epoch and displayed each layer separately.

We design two experiments with generated Gaussian samples. The first one
maintains the independence of variables as $X\sim{N(0,\sigma{I})}$. We
gradually increase $\sigma$ from 0 to 1, and show the curve of all compared
method. By conducting this experiment in different feature dimension, we can
verify whether these estimators suffer saturation problem with high variance
samples. We denote this as Saturation Test in Fig.3(a).

In the other test, samples are collected from a correlated multivariate Normal
Distribution(i.e. the variance of each variable is controlled as one). Let
$X\sim{N(0,\Sigma)}$, with $\Sigma=[s_{ij}]_{i,j=0}^{d}$ where $s_{ij}=1$ if
$i=j$ and otherwise $s_{ij}=\sigma$. We decrease the correlation factor
$\sigma$ from initial 1 to 0, as a change of correlation from fully
correlated, to completely independent. The reduction of correlation should
increase the estimated entropy. This experiment can find out whether
estimators can reveal precise dependence change, can be regarded as a test of
estimators’ accuracy in Fig.3 (b). We also repeat the experiment in feature
dimensions of [3,15,50,200].

In Fig.3 (a), it is clear that all methods can measure entropy with 3
variables. While in high dimensional cases, KDE(red) and BIN(purple)
estimators experience saturation in different degree. KDE(red) method grows
sharply with a slight increase of variance, indicating it is unable to
distinguish samples with more diverse distributions. BIN(purple) method
saturate in another way that can only distinguish those samples with high
variances, revealing it would be inaccurate in most high dimensional
estimation (because other samples are less diverse than Gaussian). In Fig.(b),
all methods perform well in 3-dimensional case, but REYI(blue) and our
LogDet(green) method have good precision when dimension increases. KNN(orange)
method can also show the correlation change, but it grows rapidly in begins,
which is less credible than REYI(blue) and LogDet(green).

Experiments reveal LogDet estimator can measure entropy in all dimensional
feature space, without concern of saturation, and is accurate to measure the
correlation between variables.

###  III-B LogDet Estimator on Realworld Benchmarks

When measuring samples with high feature dimension, their distribution is
unavoidably ill-posed and degenerated. Such as image data, for the strong
correlation between pixels, images often contain vast redundancies which make
them low-rank in their feature space. Meanwhile, the difference between images
is hard to evaluate. For pictures in the same class, their quantified
similarity often seems neglectable comparing to their differences. This makes
conventional measurement hard to define on such data. However, estimating in
such highly diversified and degenerated situation is a common requirement of
modern machine learning applications. In Fig.3, Our experiments show LogDet
estimator can precisely measure entropy with different dimensional features,
it is natural to wonder whether it can be applied in real world to reveal
correlation in such degenerated distribution.

To find out, we choose 1000 samples from random classes in Machine Learning
benchmarks MNIST (d=784), CIFAR10 (d=3072) and 500 samples from STL10
(d=3072). By gradually replacing the mix-class samples with images from a
single category, we would expect the estimated entropy to decrease
continuously. Because samples from the same class are more alike, their
features would be more inner correlated, which indicates a lower entropy than
their mix. We repeatedly perform the entropy estimation on each substitute
class in each dataset. So we will have 10 results per dataset per method. By
averaging the results, the bias brings by the entropy difference of each class
can be eliminated. To compare each method’s performance, we unify all values
from 0 to 1 without changing their distribution. We also compare the result on
the original image, and those with Gaussian noise by variance
$\epsilon\in$[0.2, 0.6, 0.8, 1.0] to further analyse these estimator’s
robustness to random noise.

The experiment result is displayed in Fig.4. When tested with MNIST and STL10,
LogDet estimator, $\alpha$-$R\acute{e}nyi$ estimator and K-nearest approach
show expected reduction. However, when with added noises,
$\alpha$-$R\acute{e}nyi$ is comparably more unstable. Its estimation fails
when noise variance reached 0.6. While in CIFAR10, the decline of
$\alpha$-$R\acute{e}nyi$ estimator and K-nearest approach become less obvious
than LogDet. Despite the fluctuation, the general curve of LogDet estimator’s
is still downward. Besides these three methods, the Binning approach saturates
in most cases. KDE fails in another way, showing a concave curve in all three
unbiased tests. Then it is randomly fluctuated with added noises, suggesting
it unsuitable to measure in image datasets.

Comparing the result, LogDet estimator shows statistically declines within all
noise situation and all tested datasets. This shows its capability in
estimating machine learning benchmarks with high-dimensional degenerated
distribution, and is robust to random noise. We think the stability is brought
by the logarithm determinant operation, which aligns with the findings of
prior works about LogDet based matrix divergence [29][30].

###  III-C LogDet Estimator for Information Behaviour Analysis

(a) IP of the 3-layer FCN (b) IP of the 5-layer FCN

(c) IP of the 7-layer FCN

Figure 6: Information Plane of fully connected neural networks with 3072-1024
in the first two layers and 10 in the last layer. Five and seven layers FCNs
have multiple 20-neurons Middle layers, with 2 and 4 layers respectively. All
models are trained on CIFAR10 dataset. Mutual information is estimated with
3000 test samples.

In this part, we employ LogDet estimator in deep neural network’s behaviour
analysis. Firstly, we display the information plane (IP) of a fully connected
neural network activated by $tanh$ and $ReLU$. Our results find the
compression in $I(X;T)$ occur in both $tanh$ and $ReLU$ activated networks.
Then we compare the mutual information $I(X;T)$ to the introduced Layer
Transmission Capacity $LTC(L)$ that measures the amount of information
transmitted through each layer. Based on this, we try to give a plausible
understanding of compression. Results reveal IB compression in $I(X;T)$ are
caused by the reduction of Transmission Capacity that only exist in the first
few layers. The Networks express two distinct LTC behaviour between shallow
layers and deeper ones.

As we mentioned in the introduction section, to avoid unreliable results
caused by misusing the estimator, we control the network conditions (models,
activation functions, feature dimensions etc.) that LogDet estimator is proved
effective in the above experiments. So we choose a fully connected neural
network, with (784/3072-1024-20-20-10) neurons for each layer. Except for the
input, the dimension of hidden layers is about up to 1000, which guarantees
LogDet works well with 3000 samples. Networks are trained on benchmarks MNIST
and CIFAR10, activated by $tanh$ and $ReLU$.

Firstly, we display the IP on the 5-layers FCN in Fig. 5. The results show the
phase transition in most of the cases, where $I(X;T)$ increasing at the
beginning then decreasing. In Fig. 5 (a) that trained with CIFAR10, it shows a
more sophisticated behaviour than (b) on MNIST, but their tendencies is
roughly similar. Comparing with different activation function, we see the
model with $tanh$ activation has rather a severe compression than operated by
$ReLU$. This observation objects the findings in [5], which account
compression for the effect of $tanh$’s saturation and proposed compression
doesn’t exist in with $ReLU$. Our results show compression still occur in
$ReLU$ where saturation is absent. Considering it is less severe than $tanh$,
we also inference that the double-saturating $tanh$ can cause compression, but
it is not the only reason. Then, to investigate whether such behaviour appears
generally, we alter numbers of 20-neurons layer change the model to 3 and 7
layers. All 3, 5 and 7 layers’ models are trained on CIFAR10. Our results
displayed in Fig. 6 show deeper models tend to have stronger compression than
shallow ones. Five and seven layers models show typical IB phase transition
from fitting to compression. In each model, the former layer compresses more
information about $X$ than the latter ones.

(a) LTC of the 3-layer FCN (b) LTC of the 5-layer FCN

(c) LTC of the 7-layer FCN

Figure 7: Layer Transmission Capacity of different layers’ fully connected
neural networks that possess same structure as the ones in information plane
demonstration. All models are trained on CIFAR10 dataset and estimated with
3000 test samples.

However, because each layer’s $I(X;T)$ behaviour is affected by all its former
layers, only observing such information plane cannot reveal precisely the
detailed behaviour of the neural network. To further understand the
compression phenomenon, we estimate Layer Transmission Capacity and construct
a layer-wise information plane, where the horizontal axis $I(X;T_{i})$ is
replaced by Layer Transmission Capacity $LTC(L_{i})$, which indicates how much
information is transmitted in $ith$ layer. The vertical axis of $I(T_{i};Y)$
is kept and interpreted as the amount of target-related information that
passes the $L_{i}$ layer. In Fig.7, we display the layer-wise Information
Planes of the three corresponding FCNs in Fig.6.

In all six results (with different activation and model depth), we see a
decrease of $LTC$ in the first few layers. This decrease directly tells the
first layer of FCN reduces the amount of information transmitted through it.
Except for the decrease in the first layer(and to some $tanh$ activated
situation in the second layer), there is hardly any reduction of $LTC$
observed. Most later layers’ $LTC$ keeps increasing, indicating the layers are
expanding their channel and allowing more information to pass by. Thus
compression never happens in their layer-wise demonstration. We argue that
this result doesn’t contradict with the compression observed in IB plane, but
rather gives an interpretation: The input layer reduces the information about
$X$ goes into deeper layers. Despite other layers keep increasing their
transmission capacity, the $I(X;T_{i})$ will still decrease for it is limited
in the beginning. In which case, the decrease of $I(X;T)$ displayed in Fig.6
is mainly an influence of compression in the first layer.

Also worth noticing is, the latter layers tend to have similar behaviour that
is distinct to the former ones. This reveals a functional distinction along
with the depth of neural network, where the former layer tends to compress the
data, acting like feature selector to get a useful representation. While the
latter ones are optimized to transmit information more effectively or change
its representation to fit the target. Such phenomenon that appears in three
models with different depth shows the generality of LTC behaviour.
Furthermore, we want to investigate whether such phenomenon is essential, we
fix the first layer’s parameters, then train the network as usual. If such
two-step compression-transmission phenomenon is essential to training neutral
network, we expect compression of LTC would appear in the second layer where
it doesn’t promised appear in former test. The result is displayed in Fig.8.
We see the first layer is fixed as a stable point. Surprisingly, the second
layer, that is not compressed in the former experiment, showing a similar
$LTC$ compression as the first layer. The Compression-Transmission distinction
happens again! This further support the essences of such compression behaviour
of the first layer. Also interesting is, when we fix more parameters that the
model can be hardly trained on the task, such compression in LTC disappeared.
Along with its absence, the prediction accuracy can be merely improved through
training. This observation connected the compression of the first layer to the
model’s performance, and again shows first layer’s compression is essential in
training neural networks.

(a) IP with fixed first layer (b) IP with fixed second layer (c) LTC with
fixed first layer (d) LTC with fixed second layer

(e) ACC with fixed first layer (f) ACC with fixed second layer

Figure 8: Information Plane, Layer Transmission Capacity and Accuracy of the
5-layer FCN with fixed parameters of first layer (a,c,e) and fixed first and
second layers (b,d,f). Trained on CIFAR10 dataset, the model compresses in
second layer of LTC when fixed the first layer. Where such compression is
absent without parameter fixing.

In all, these results explain the cause of IB compression as the effect of the
first layer. It also reveals the distinct behaviour between the first few
layers and other layers. We show such distinction is a general phenomenon when
training neural networks by changing the Network structures, activation
functions and fixing parameters, and such phenomenon is connected with
prediction performance.

##  IV Discussion and Future Works

By approximating Shannon Differential Entropy, this work shows LogDet
estimator is capable of estimating information entropy on artificial samples
and real-world benchmarks. Experiments show LogDet estimator can be applied to
any high dimensional features and is robust to random noise. Comparing to
other analytical works, these tests ensure the soundness of informative
behaviour analysis of neural networks with LogDet estimator.

Furthermore, in neural networks’ IB analysis, our work gives a layer-wise
interpretation for compression behaviour. First we find networks experience
compression in both double and single saturating activation. Experiments
reveal saturation of $tanh$ can cause compression. This explains why we often
witness more severe compression in $tanh$ activated network. Also, compression
occurs in $ReLU$ activated FCNs proves the double-saturating effect is not the
only reason. To better understand such phenomenon, We simplify the discussion
by a layer-wise Transmission Capacity analysis. The result shows only the
first or second layer reduces its transmission capacity LTC, causing the
decrease of $I(X;T)$ as compression phenomenon observed in IB plane. Besides,
the model learns to automatically divide into two areas: (1) shallow layers
transmit the input related information selectively; (2) deeper layers on the
opposite, keep increasing their capacity to avoid transmitting loss when
processing the features. These results reveal a functional difference among
neural network layers which gives an empirical guideline for future analysis
of neural networks informative behaviour.

We believe LogDet estimator provides a extendable framework for estimating
informative relations and constructing informational objectives. Some
potential directions are displayed below:

  1. 1.

Estimating covariance matrix: In this work, LogDet estimator measures entropy
with covariance matrix. In which case, we can adopt many covariance estimation
methods to improve. One possible approach is to introduce prior knowledge
about estimated samples. For example, when estimating with image data, the
locally correlated nature of pixels can be introduced. Therefore, we can
reduce covariance between distant variables, while keep only the correlation
of their $ith$ neighbours. This can be achieved by eliminating other but the
diagonal and off-diagonal values in covariance matrix $\Sigma$. Covariance
estimation has been well studied in modern statistics, we refer to [43][44]
for potential improvements.

  2. 2.

Kernel extension: Since the multi-signal extension reveals the potential of
kernelizing the similarity matrix $\Sigma$ in $\log\det(I+\beta\Sigma)$, we
can replace it with Gram matrices where each component is an arbitrary kernel
function $k(i,j)$. Similar kernel extension using $\log\det$ function has been
applied in [45][42] and is shown flexible in visual recognition problems. We
believe this can also be applied to enhance the LogDet information measuring
on more sophisticated sample distribution.

  3. 3.

Learning Objectives: LogDet expression has revealed the potential to act as an
objective in supervised learning tasks, and express robust performance to
corrupted labels[40]. By further analysing the network behaviour, it is
possible to use LogDet as objective function and optimize it through
backpropagation. For example, we tried to use a kernel function
$exp(x^{T}y-1)$ to force the similarity matrix to only captures positive
correlations. Then, we use $H_{D}(Y|T)=H_{D}(T,Y)-H_{D}(T)$ as a learning
objective. This expression can be used to training neural network classifiers,
and can increase accuracy through training. However, since $exp$ cannot
eliminate all negative correlations while retaining the positive ones, our
objective is not an ideal adaptation. We believe this is a quite promising
direction.

We also witness the limitations of LogDet estimator. As a matrix-based method,
it heavily relies on the amount of estimating samples. Although the
approximating method avoids the singular problem, the estimating result is
still discredited when the sample number is less than the feature dimension.
This currently is an unavoidable problem of matrix-based methods. Similar
limitations also occur in the recent proposed $\alpha$-$R\acute{e}nyi$
estimator. Despite it proposed independent with feature dimension, its
estimating result is less accurate when sample amount is fewer than feature
dimension(see Appendix. B for details test). We look forward to further
improvements to overcome such limitation.

## Acknowledgment

This work is supported by Natural Science Foundation of Tianjin
(20JCYBJC00500), the Science & Technology Development Fund of Tianjin
Education Commission for Higher Education (2018KJ217), the Tianjin Science and
Technology Program (19PTZWHZ00020) and Natural Science Foundation of China
(61772363).

## References

  * [1] Naftali Tishby, Fernando C Pereira, and William Bialek.  The information bottleneck method.  arXiv preprint physics/0004057, 2000.
  * [2] Ravid Shwartz-Ziv and Naftali Tishby.  Opening the black box of deep neural networks via information.  arXiv preprint arXiv:1703.00810, 2017.
  * [3] Bernhard C Geiger.  On information plane analyses of neural network classifiers–a review.  arXiv preprint arXiv:2003.09671, 2020.
  * [4] Hao Cheng, Dongze Lian, Shenghua Gao, and Yanlin Geng.  Utilizing information bottleneck to evaluate the capability of deep neural networks for image classification.  Entropy, 21(5):456, 2019.
  * [5] Andrew M Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox.  On the information bottleneck theory of deep learning.  Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124020, 2019.
  * [6] Ziv Goldfeld, Ewout van den Berg, Kristjan Greenewald, Igor Melnyk, Nam Nguyen, Brian Kingsbury, and Yury Polyanskiy.  Estimating information flow in deep neural networks.  arXiv preprint arXiv:1810.05728, 2018.
  * [7] Martin Schiemer and Juan Ye.  Revisiting the information plane.  2019\.
  * [8] Nabil Ali Ahmed and DV Gokhale.  Entropy expressions and their estimators for multivariate distributions.  IEEE Transactions on Information Theory, 35(3):688–692, 1989.
  * [9] Neeraj Misra, Harshinder Singh, and Eugene Demchuk.  Estimation of the entropy of a multivariate normal distribution.  Journal of multivariate analysis, 92(2):324–342, 2005.
  * [10] Santosh Srivastava and Maya R Gupta.  Bayesian estimation of the entropy of the multivariate gaussian.  In 2008 IEEE International Symposium on Information Theory, pages 1103–1107. IEEE, 2008.
  * [11] Shashank Singh and Barnabás Póczos.  Nonparanormal information estimation.  In International Conference on Machine Learning, pages 3210–3219. PMLR, 2017.
  * [12] Luis Gonzalo Sanchez Giraldo, Murali Rao, and Jose C Principe.  Measures of entropy from data using infinitely divisible kernels.  IEEE Transactions on Information Theory, 61(1):535–548, 2014.
  * [13] Shujian Yu, Luis Gonzalo Sanchez Giraldo, Robert Jenssen, and Jose C Principe.  Multivariate extension of matrix-based rényi’s $\alpha$-order entropy functional.  IEEE transactions on pattern analysis and machine intelligence, 42(11):2960–2966, 2019.
  * [14] Shujian Yu and Jose C Principe.  Understanding autoencoders with information theoretic concepts.  Neural Networks, 117:104–123, 2019.
  * [15] Shujian Yu, Kristoffer Wickstrøm, Robert Jenssen, and José C Príncipe.  Understanding convolutional neural networks with information theory: An initial exploration.  IEEE transactions on neural networks and learning systems, 2020\.
  * [16] Shujian Yu, Francesco Alesiani, Xi Yu, Robert Jenssen, and Jose C Principe.  Measuring dependence with matrix-based entropy functional.  arXiv preprint arXiv:2101.10160, 2021.
  * [17] Nicolás I Tapia and Pablo A Estévez.  On the information plane of autoencoders.  In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2020.
  * [18] Ivan Chelombiev, Conor Houghton, and Cian O’Donnell.  Adaptive estimators show information compression in deep neural networks.  arXiv preprint arXiv:1902.09037, 2019.
  * [19] Marylou Gabrié, Andre Manoel, Clément Luneau, Jean Barbier, Nicolas Macris, Florent Krzakala, and Lenka Zdeborová.  Entropy and mutual information in models of deep neural networks.  Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124014, 2019.
  * [20] Junjie Li and Ding Liu.  Information bottleneck theory on convolutional neural networks.  Neural Processing Letters, pages 1–16, 2021.
  * [21] Hassan Hafez-Kolahi, Shohreh Kasaei, and Mahdiyeh Soleymani-Baghshah.  Do compressed representations generalize better?  arXiv preprint arXiv:1909.09706, 2019.
  * [22] Maya Gupta and Santosh Srivastava.  Parametric bayesian estimation of differential entropy and relative entropy.  Entropy, 12(4):818–843, 2010.
  * [23] Barnabás Póczos and Jeff Schneider.  Nonparametric estimation of conditional information and divergences.  In Artificial Intelligence and Statistics, pages 914–923. PMLR, 2012.
  * [24] Yi Ma, Harm Derksen, Wei Hong, and John Wright.  Segmentation of multivariate mixed data via lossy data coding and compression.  IEEE transactions on pattern analysis and machine intelligence, 29(9):1546–1562, 2007.
  * [25] Thomas M Cover.  Elements of information theory.  John Wiley & Sons, 1999.
  * [26] Gerardo Adesso and R Simon.  Strong subadditivity for log-determinant of covariance matrices and its applications.  Journal of Physics A: Mathematical and Theoretical, 49(34):34LT02, 2016.
  * [27] Alfréd Rényi et al.  On measures of entropy and information.  In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics. The Regents of the University of California, 1961.
  * [28] Ludovico Lami, Christoph Hirche, Gerardo Adesso, and Andreas Winter.  From log-determinant inequalities to gaussian entanglement via recoverability theory.  IEEE Transactions on Information Theory, 63(11):7553–7568, 2017\.
  * [29] Anoop Cherian, Suvrit Sra, Arindam Banerjee, and Nikolaos Papanikolopoulos.  Jensen-bregman logdet divergence with application to efficient similarity search for covariance matrices.  IEEE transactions on pattern analysis and machine intelligence, 35(9):2161–2174, 2012.
  * [30] Andrzej Cichocki, Sergio Cruces, and Shun-ichi Amari.  Log-determinant divergences revisited: Alpha-beta and gamma log-det divergences.  Entropy, 17(5):2988–3034, 2015.
  * [31] Rana Ali Amjad and Bernhard C Geiger.  Learning representations for neural network-based classification using the information bottleneck principle.  IEEE transactions on pattern analysis and machine intelligence, 42(9):2225–2239, 2019.
  * [32] Thomas M Cover and A Thomas.  Determinant inequalities via information theory.  SIAM journal on Matrix Analysis and Applications, 9(3):384–392, 1988.
  * [33] Ivan Matic.  Inequalities with determinants of perturbed positive matrices.  Linear Algebra and its Applications, 449:166–174, 2014.
  * [34] T Tony Cai, Tengyuan Liang, and Harrison H Zhou.  Law of log determinant of sample covariance matrix and optimal estimation of differential entropy for high-dimensional gaussian distributions.  Journal of Multivariate Analysis, 137:161–172, 2015.
  * [35] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm.  Mine: mutual information neural estimation.  arXiv preprint arXiv:1801.04062, 2018.
  * [36] Alexander Kraskov, Harald Stögbauer, and Peter Grassberger.  Estimating mutual information.  Physical review E, 69(6):066138, 2004.
  * [37] Greg Ver Steeg and Aram Galstyan.  Information-theoretic measures of influence based on content dynamics.  In Proceedings of the sixth ACM international conference on Web search and data mining, pages 3–12, 2013.
  * [38] Hao Cheng, Dongze Lian, Shenghua Gao, and Yanlin Geng.  Evaluating capability of deep neural networks for image classification via information plane.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 168–182, 2018.
  * [39] Kwan Ho Ryan Chan, Yaodong Yu, Chong You, Haozhi Qi, John Wright, and Yi Ma.  Deep networks from the principle of rate reduction.  arXiv preprint arXiv:2010.14765, 2020.
  * [40] Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma.  Learning diverse and discriminative representations via the principle of maximal coding rate reduction.  Advances in Neural Information Processing Systems, 33, 2020.
  * [41] Ziyang Wu, Christina Baek, Chong You, and Yi Ma.  Incremental learning via rate reduction.  arXiv preprint arXiv:2011.14593, 2020.
  * [42] Xi Yu, Shujian Yu, and Jose C Principe.  Deep deterministic information bottleneck with matrix-based entropy functional.  arXiv preprint arXiv:2102.00533, 2021.
  * [43] Zongliang Hu, Kai Dong, Wenlin Dai, and Tiejun Tong.  A comparison of methods for estimating the determinant of high-dimensional covariance matrix.  The international journal of biostatistics, 13(2), 2017.
  * [44] Clifford Lam.  High-dimensional covariance matrix estimation.  Wiley Interdisciplinary Reviews: Computational Statistics, 12(2):e1485, 2020.
  * [45] John Wright, Yi Ma, Yangyu Tao, Zhuochen Lin, and Heung-Yeung Shum.  Classification via minimum incremental coding length (micl).  Technical report, Coordinated Science Laboratory, University of Illinois at Urbana-Champaign, 2007.

##  Appendix A Proof of main results

$Lemma$ 1 To any positive definite matrix $A$ and positive semi-definite
matrix $B$, we have the additive inequality:

$\det(A+B)\geq{\det(A)}$

$Proof$ since A is an arbitrary positive definite matrix, there exist a
invertable positive semi-definite matrix T where $T^{2}=A$. Then, we have:

$\displaystyle\det(A+B)$ $\displaystyle=\det(TIT+B)$
$\displaystyle=\det(T(I+T^{-1}BT^{-1})T)$
$\displaystyle=\det(T)\det(T)\det(I+T^{-1}BT^{-1})$
$\displaystyle=\det(A)\det(I+T^{-1}BT^{-1})$

Let $Q=T^{-1}BT^{-1}$, and eigenvalues of $Q$ as $[\lambda_{i}]_{i=1}^{n}$,
then Q is still positive semi-definite, now we have:

$\displaystyle\det(A)\det(I+T^{-1}BT^{-1})$ $\displaystyle=\det(A)\det(I+Q)$
$\displaystyle=\det(A)\prod_{i=1}^{n}(1+\lambda_{i})$
$\displaystyle\geq{\det(A)(1+\prod_{i=1}^{n}\lambda_{i}})$
$\displaystyle=\det(A)(1+\det(Q))$ $\displaystyle\geq{\det(A)}$

The last inequality is for $\det(Q)\geq{0}$ as it is positive semi-definite.

$Commutative$ $property$

$\displaystyle\frac{1}{2}\log\det(I+\beta\frac{X^{T}X}{n})=\frac{1}{2}\log\det(I+\beta\frac{XX^{T}}{n})$

$Proof$ of Eq.4: performing $SVD$ to $X$, we have $X=UDV$ where
$D=diag([n\lambda_{1},n\lambda_{2},...,n\lambda_{k}])$ that $\lambda_{i}$ is
$\frac{X^{T}X}{n}$’s ith eigenvalue, then:

$\displaystyle\mathbf{H}_{D}(X)$
$\displaystyle=\frac{1}{2}\log\det(I+\beta\frac{X^{T}X}{n})$
$\displaystyle=\frac{1}{2}\log\det(I+\frac{\beta}{n}V^{T}D^{2}V)$
$\displaystyle=\frac{1}{2}\log\det(V^{T}(I+\frac{\beta}{n}D^{2})V)$
$\displaystyle=\frac{1}{2}\log\det(I+\frac{\beta}{n}D^{2})$
$\displaystyle=\frac{1}{2}\log\prod_{i=1}^{k}(1+\beta\lambda_{i})$
$\displaystyle=\frac{1}{2}\sum_{i=1}^{k}\log(1+\beta\lambda_{i})$
$\displaystyle=\frac{1}{2}\sum_{i=1}^{k}\log(\frac{1}{\beta}+\lambda_{i})+\frac{k}{2}\log\beta$

$Proof$ of $Proposition$ 2.1: The first inequality
$\mathbf{H}_{D}(X_{1},X_{2})\leq\mathbf{H}_{D}(X_{1})+\mathbf{H}_{D}(X_{2})$
Let estimated covariance matrix
$Cov(X_{1})=\Sigma_{1}=\frac{X_{1}^{T}X_{1}}{n}$,
$Cov(X_{2})=\Sigma_{2}=\frac{X_{2}^{T}X_{2}}{n}$, and $Z=[X_{1},X_{2}]$,
$Cov(Z)=\begin{pmatrix}\Sigma_{1}&F\\\ F^{T}&\Sigma_{2}\end{pmatrix}$, we have

$\displaystyle\mathbf{H}_{D}(X_{1})+\mathbf{H}_{D}(X_{2})$
$\displaystyle=\frac{1}{2}\log\det(I+\beta\Sigma_{X_{1}})$
$\displaystyle+\frac{1}{2}\log\det(I+\beta\Sigma_{X_{2}})$
$\displaystyle=\frac{1}{2}\log\det(I+\beta\frac{X_{1}^{T}X_{1}}{n})(I+\beta\frac{X_{2}^{T}X_{2}}{n})$
$\displaystyle=\frac{1}{2}\log\det(I+\beta\frac{X_{1}X_{1}^{T}}{n})(I+\beta\frac{X_{2}X_{2}^{T}}{n})$
$\displaystyle=\frac{1}{2}\log\det(I+\beta\frac{X_{1}X_{1}^{T}}{n}+\beta\frac{X_{2}X_{2}^{T}}{n}$
$\displaystyle+\beta^{2}\frac{X_{1}X_{1}^{T}+X_{2}X_{2}^{T}}{n^{2}})$
$\displaystyle\geq{\frac{1}{2}\log\det(I+\beta\frac{X_{1}X_{1}^{T}}{n}+\beta\frac{X_{2}X_{2}^{T}}{n})}$
$\displaystyle=\frac{1}{2}\log\det(I+\beta\frac{ZZ^{T}}{n})$
$\displaystyle=\frac{1}{2}\log\det(I+\beta\frac{Z^{T}Z}{n})$
$\displaystyle=\mathbf{H}_{D}(X_{1},X_{2})$

The last inequality satisfied for $Lemma$ 1\. The second inequality
$\mathbf{H}_{D}(X_{1},X_{2})\geq\max(\mathbf{H}_{D}(X_{1}),\mathbf{H}_{D}(X_{2}))$
can be derived using above expression:

$\displaystyle\mathbf{H}_{D}(X_{1},X_{2})$
$\displaystyle=\frac{1}{2}\log\det(I+\beta\frac{Z^{T}Z}{n})$
$\displaystyle=\frac{1}{2}\log\det(I+\beta\frac{X_{1}X_{1}^{T}}{n}+\beta\frac{X_{2}X_{2}^{T}}{n})$

Applying $Lemma$ 1, we have:

$\displaystyle\mathbf{H}_{D}(X_{1},X_{2})$
$\displaystyle\geq{\frac{1}{2}\log\det(I+\beta\frac{X_{1}X_{1}^{T}}{n})}=\mathbf{H}_{D}(X_{1})$
$\displaystyle\mathbf{H}_{D}(X_{1},X_{2})$
$\displaystyle\geq{\frac{1}{2}\log\det(I+\beta\frac{X_{2}X_{2}^{T}}{n})}=\mathbf{H}_{D}(X_{2})$

So,

$\mathbf{H}_{D}(X_{1},X_{2})\geq{\max(\mathbf{H}_{D}(X_{1}),\mathbf{H}_{D}(X_{2}))}$

$Proof$ to $Corollary$ 4.1: The procedure is similar with the proof to
Proposition 2.1. By applying commutative property of $\log\det$ function and
$Lemma$ 1, the inequality is easy to prove.

##  Appendix B Limitations of Matrix-based Estimators

The matrix-based method heavily relies on the amount of estimating samples.
Although, with samples far less than the dimension, LogDet is express the
capacity to show the relations in $I_{D}(X;Y)$ Fig. 1. However, Gaussian
samples are too simple, the subtle dependence change in neural networks
requires more samples to reveal. We conduct the activation function’s
experiment to demonstrate:

(a) fixed amount of samples while increase $d$ (b) fixed $d$ while increase
the amount of samples

Figure 9: Limitation of matrix-based methods using $tanh$’s saturating test.

In Fig. 9(a), we compare the LogDet estimator and matrix-based
$\alpha$-$R\acute{e}nyi$ estimator on $tanh$’s saturation effect, and use 500
samples to estimate. It is clear, when the feature dimension increases, the
saturation effect is shown by both methods become less obvious. At last it
disappear when $d=1000$. But when we fix the dimension as 1000, and increase
the sample numbers from 500 to 2000 in (b), such decrease a.k.a. compression
shows again. We know such saturation exists, but when the dimension is larger
than samples, cannot be revealed by both matrix-based estimators. Also
interesting is that, in all results, LogDet estimator express more severe
change than $\alpha$-$R\acute{e}nyi$ estimator.