# An information-Theoretic Approach to Semi-supervised Transfer Learning

Daniel Jakubovitz, David Uliel, Miguel Rodrigues, Raja Giryes  Daniel
Jakubovitz, David Uliel and Raja Giryes are with the School of Electrical
Engineering, Tel Aviv University, Tel Aviv 69978, Israel. (email:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Miguel R. D. Rodrigues is with the Department of Electronic and Electrical
Engineering, University College London, London, United Kingdom. (email:
<EMAIL_ADDRESS>
###### Abstract

Transfer learning is a valuable tool in deep learning as it allows propagating
information from one ”source dataset” to another ”target dataset”, especially
in the case of a small number of training examples in the latter. Yet,
discrepancies between the underlying distributions of the source and target
data are commonplace and are known to have a substantial impact on algorithm
performance. In this work we suggest novel information-theoretic approaches
for the analysis of the performance of deep neural networks in the context of
transfer learning. We focus on the task of semi-supervised transfer learning,
in which unlabeled samples from the target dataset are available during
network training on the source dataset. Our theory suggests that one may
improve the transferability of a deep neural network by incorporating
regularization terms on the target data based on information-theoretic
quantities, namely the Mutual Information and the Lautum Information. We
demonstrate the effectiveness of the proposed approaches in various semi-
supervised transfer learning experiments.

###### Index Terms:

Information Theory, Lautum Information, Mutual Information, Transfer Learning,
Semi-supervised learning.

##  1 Introduction

Machine learning algorithms have lately come to the forefront of technological
advancements, providing state-of-the-art results in a variety of fields [1].
However, alongside their outstanding performance, these methods suffer from
sensitivity to data discrepancies - any inherent difference between the
training data and the test data may result in a substantial decrease in
performance. Moreover, to obtain good performance, a large amount of labeled
data is necessary for their training. Such a substantial amount of labeled
data is often either very expensive or simply unobtainable.

(a) Pre-transfer training stage.

(b) Post-transfer training stage.

Figure 1: Our semi-supervised transfer learning technique applies information-
based regularization. Omitting the blue part in the Pre-transfer training
stage (lower part of (a)), results in standard transfer learning.

One popular approach to mitigate this issue is using ”transfer learning”,
where training on a small labeled ”target” dataset is improved by using
information from another large labeled ”source” dataset. A common method for
transfer learning uses the result of training on the source as initialization
for training on the target, thereby improving performance on the latter [2].

Transfer learning has been the focus of substantial research attention over
the years. Plenty of different approaches have been proposed to encourage a
more effective transfer from a source dataset to a target dataset, many of
which aim at obtaining better system robustness to environment changes, so as
to allow an algorithm to perform well even under some variations in the
settings (e.g. changes in lighting conditions in computer vision tasks).
Several works take a more targeted approach and directly try to reduce the
generalization error by reducing the difference in performance on specific
source and target datasets [3]. It is often the case that the target dataset
has a large number of samples, though only a few of those samples are labeled.
In this scenario a semi-supervised learning approach could prove to be
beneficial by making effective use of the available unlabeled samples for
training.

In this work we focus on the task of semi-supervised transfer learning. We
make a distinction between transfer learning and the related task of domain
adaptation, where the former refers to the case of two sources of data which
are different in both content and ”styling” (e.g. the MNIST $\rightarrow$ SVHN
case), whereas the latter refers to the case of two sources of data with the
same content. Another relevant difference is that labeled data from the target
distribution is typically available in the transfer learning case, yet no such
labels are available in the domain adaptation case.

Plenty of works exist in the literature on transfer learning, semi-supervised
learning and the use of information theory for the analysis of machine
learning algorithms. The closest work to ours is [4] in which an information-
theoretic approach is used in order to decompose the cross-entropy _train_
loss of a machine learning algorithm into several separate terms. However,
unlike this work, we propose two different decompositions of the cross-entropy
_test_ loss and make the relation to semi-supervised transfer learning.

Contribution. We consider the case of semi-supervised transfer learning in
which plenty of labeled examples from a source distribution are available
alongside just a few labeled examples from a target distribution; yet, we are
also provided with a large number of unlabeled samples from the latter. This
setup combines transfer learning and semi-supervised learning, where both aim
at obtaining an improved performance on a target dataset with a small number
of labeled examples. In this work, we propose combining both methodologies to
compound their advantages. This setting represents the case where the learned
information from a large labeled source dataset is used to obtain good
performance when transferring to a mostly unlabeled target set, where the
unlabeled examples of the target are available at training time on the source.
This scenario is commonplace in situations where obtaining labels for target
data samples is an expensive or lengthy process, however labels for source
data samples are of abundance and readily available, most often due to their
open-source availability or an enhanced ease of their labeling. An example
scenario is when transferring from an open-source dataset (e.g. X-Ray images
of a variety of human organs from a variety of machines) with plenty of
available labeled samples, to a more specific target dataset (X-Ray images
from a specific type of machine, a specific point of view, a specific human
organ in view etc.) for which few labels are available.

We present two theoretical derivations that lead to novel semi-supervised
transfer learning techniques. We take an information-theoretic approach to
examine the cross-entropy test loss of machine learning methods; we present
two decompositions of the loss into several different terms that account for
different aspects of its behavior. These derivations lead to new
regularization terms: (i) ”Lautum regularization” which relies on the
maximization of the Lautum information [5] between unlabeled data samples
drawn from the target distribution and the learned model weights, and (ii)
”Mutual Information regularization”, derived from the Information Bottleneck
Theorem [6], which relies on the minimization of the mutual information
between the learned model parameters and the unlabeled target samples.

Figure 1 provides a general illustration of our two approaches. We corroborate
their effectiveness with experiments of semi-supervised transfer learning for
neural networks on image classification tasks. We examined the transfer in two
cases: from the MNIST dataset to the notMNIST dataset (which consists of the
letters A-J in grayscale images) and from the CIFAR-10 dataset to 10 specific
classes of the CIFAR-100 dataset. We compare our results to three other
methods: (1) Temporal Ensembling (TE) [7], a prominent method for semi-
supervised training which we apply in a transfer learning setup. We examine TE
both by itself and combined with Lautum regularization; (2) the Multi-kernel
Maximum Mean Discrepancy (Mk-MMD) method [8], which is popular in semi-
supervised transfer learning; (3) standard transfer learning which does not
use any of the unlabeled samples. We examine our two approaches both when
separately applied (i.e. each approach is applied and analysed without the
other), and when combined (i.e. when both approaches are applied at the same
time within the same training context). The advantages of our proposed methods
are demonstrated in our experimental results which outperform the other
compared methods.

##  2 Related Work

Plenty of works exist in the literature on transfer learning, semi-supervised
learning and using information theory for the analysis of machine learning
algorithms. We hereby overview the ones most relevant to our work.

Transfer learning. Transfer learning [9, 10] is a useful training technique
when the goal is to adapt a learning algorithm, which was trained on a source
dataset, to perform well on a target dataset that is potentially very
different in content compared to the source. This technique can provide a
significant advantage when the number of training samples in the source
dataset is large compared to a small number in the target, where the knowledge
extracted from the source dataset may be relevant also to the target.

The work in [11] relates to a core question in transfer learning: which layers
in the network are general and which are more task specific, and precisely how
transferability is affected by the distance between two tasks. In a recent
work [12] an analytical theory of how knowledge is transferred from one task
to another in deep linear networks is presented. A metric is defined to
quantify the amount of knowledge transferred between a pair of tasks.

Practical approaches to improving performance in transfer learning settings
have been proposed in many works. In [13], transfer learning in the context of
regression problems is examined. A transfer learning algorithm that does not
assume that the support of the target distribution is contained in the support
of the source distribution is proposed. This notion leads to a more flexible
transfer. In [14] the framework of ”learning to transfer” is proposed in order
to leverage previous transfer learning experiences for better transfer between
a new pair of source and target datasets. In [15] the structural relation
between different visual tasks is examined in the feature space. The result is
a taxonomic map that enables a more efficient transfer learning with a reduced
amount of labeled data. In [16] a regularized transfer learning framework that
preserves the outer layer outputs of the target network (post-transfer) is
proposed. In [17] the authors provide an empirical deep-dive into the inner
workings of catastrophic forgetting and negative transfer, and propose the
Batch Spectral Shrinkage approach to discourage the transfer of irrelevant
spectral components of the learned representations. In [18] the authors
propose the method of oracle Auto-focusing, which allows for a dynamic
correction of data distribution shifts.

Semi-supervised learning. Semi-supervised learning [19] is typically used when
there is little labeled data for training, yet more unlabeled data is
available. The literature on semi-supervised training is vast and describes a
variety of techniques for performing effective semi-supervised learning that
would make good use of the available unlabeled data in order to improve model
performance. Most of these techniques rely on projecting the relation between
the available labeled samples and their labels onto the unlabeled samples and
the model’s predictions for them.

In [20] minimum entropy regularization is proposed. This technique modifies
the cross-entropy loss used for training in order to encourage a deep neural
network to make confident predictions on unlabeled data. In [21], a new
framework for semi-supervised training of neural networks called ”associative
learning” is proposed. In this framework ”associations” are made between the
embeddings of the available labeled data and the unlabeled data. An
optimization process is then used to encourage correct ”associations”, which
make better use of the unlabeled data. In [22], a method is proposed for
combining several different semi-supervised learning techniques using Bayesian
optimization. In [23], a semi-supervised framework that allows labeled
training data privacy is proposed. In this framework, knowledge is transferred
from teacher models to a student model in a semi-supervised manner, thereby
precluding the student from gaining access to the labeled training data which
is available to the teachers. In [24], various semi-supervised learning
algorithms are evaluated on real-world applications, yet no specific attention
is paid to the transfer learning case and the effects of fine-tuning a pre-
trained network.

Two notable works that employ semi-supervised training techniques are [25] and
[26]. In [25], a semi-supervised deep kernel learning model is presented for
regression tasks. In [26], a GAN-based method is presented. It is proposed to
estimate the tangent space to the learned data manifold using GANs, infer the
relevant invariances and then inject these into the learned classifier during
training.

Other prominent works that focus on semi-supervised transfer learning are [27]
which examines semi-supervised transfer learning for sentiment classification,
[28] where semi-supervised transfer learning is examined for different
training strategies and model choices, and [29] where an adaptive consistency
regularization paradigm is suggested to optimize performance.

All of the above mentioned works differ from ours in their fundamental
approach - they address the discrepancy between the source and target data
either in the input space or in the feature space, yet disregard the effect of
the chosen loss function and its impact on the mitigation of this discrepancy.
In contrast, our work is focused on the mathematical analysis of cross-entropy
loss which is commonly used in classification tasks. This provides our
approach with a competitive advantage compared to other prominent methods.

Information theory and machine learning. Information theory has largely been
used to provide theoretical insight into the intricacies of machine learning
algorithms. In [30], the Information Bottleneck framework has been presented.
This framework formalizes the tradeoff between algorithm sufficiency
(fidelity) and complexity. It has been analyzed in various works such as [31]
and [32]. Following works [33, 34] explicitly apply information bottleneck
principles to deep learning. In [35] several claims from [34] are examined and
challenged. In [36] useful methods for computing information-theoretic
quantities are proposed for several deep neural network models. In [37] the
authors provide an information-theoretic analysis of the generalization
ability of Gibbs-based transfer learning algorithms. They show how the
benefits of transfer learning can be explained by the framework of the bias-
variance tradeoff.

In order to utilize information-theoretic quantities such as the Mutual
Information (MI) in neural network training, efficient estimation methods are
required. Recent works harness the strength of DNNs in order to estimate the
MI. The MINE approach [38] suggests using a DNN for estimating a lower bound
for the MI. The work in [39] compared several variational methods of MI
estimation and presented their bias-variance tradeoff. The work in [40]
analyzed the limitations of MI estimators and suggested a way to modify MINE
for reducing its variance.

The closest work to ours is [4], where an information-theoretic approach is
used to decompose the cross-entropy loss of a machine learning algorithm into
several separate terms. It is suggested that overfitting the training data is
mathematically encapsulated in the mutual information between the training
data labels and the learned model weights; i.e. the mutual information
essentially represents the ability of a neural network to _memorize_ the
training data [41]. Consequently, a regularizer that alleviates the phenomenon
of overfitting is proposed, and initial results of its efficiency are
presented. However, we propose two different decompositions of the cross-
entropy _test_ loss and make the connection to semi-supervised transfer
learning. We propose two regularizers: (i) based on the Lautum information
between the model weights and the unlabeled samples of the target dataset;
(ii) based on the mutual information between the model weights and the
unlabeled samples of the target dataset. Both of these lead to an improved
semi-supervised transfer learning technique, and yield experimental results
that corroborate our theoretical analysis.

##  3 The cross-entropy loss - an information theory perspective

Let $\mathcal{D}=\\{(x_{i},y_{i})\\}_{i=1}^{N}$ be a fixed training set with
$N$ training samples that is used to train a learning algorithm with a set of
weights $w$. Given $\mathcal{D}$, the learning algorithm selects a specific
hypothesis from the hypothesis class according to the distribution
$p(w_{\mathcal{D}})$. In the case of neural networks, selecting a hypothesis
is equivalent to training the network on the data.

We denote by $w_{\mathcal{D}}$ the model weights that were learned using the
fixed training set $\mathcal{D}$, and by $f(y|x,w_{\mathcal{D}})$ the learned
classification function which given the weights $w_{\mathcal{D}}$ and a
$D$-dimensional input $x\in\mathbb{R}^{D}$ computes the probability of the
$K$-dimensional label $y\in\mathbb{R}^{K}$. Note that we treat
$f(y|x,w_{\mathcal{D}})$ as a probability distribution over the random
variables $y,x,w_{\mathcal{D}}$, where $\mathcal{D}$ is a fixed parameter.

The learned classification function is then tested on data drawn from the true
underlying distribution $p(x,y)$. Ideally, as a result of the training process
the learned classification function $f(y|x,w_{\mathcal{D}})$ would highly
resemble the ground-truth label distribution $p(y|x)$, and similarly
$f(x,y|w_{\mathcal{D}})$ would highly resemble $p(x,y)$. In practice, the
learned function does not perfectly match the real distribution although this
is the goal of training.

Using these notations, we analyze the cross-entropy loss between the
predictions of a learned classification function and the true label
distribution, which is commonly employed for training in classification tasks.
In our derivations we used several information-theoretic measures which we
present hereafter. Let $X,Y$ be two random variables with respective
probability density functions $p(x),p(y)$ and the joint probability density
function $p(x,y)$. The following are the definitions of three information-
theoretic measures which are relevant to our derivations.

######  Definition 1 (Mutual information).

The mutual information (MI) between $X$ and $Y$ is defined by

$I(X;Y)=\iint p(x,y)\log\left\\{\frac{p(x,y)}{p(x)p(y)}\right\\}dxdy.$ (1)

The MI captures the dependence between two random variables. It is the
Kullback-Leibler divergence between the joint distribution and the product of
the marginal distributions.

######  Definition 2 (Lautum information).

The Lautum information between $X$ and $Y$ is

$L(X;Y)=\iint p(x)p(y)\log\left\\{\frac{p(x)p(y)}{p(x,y)}\right\\}dxdy.$ (2)

This measure is the Kullback-Leibler divergence between the product of the
marginal distributions and the joint distribution. Similar to the MI, the
Lautum information is related to the dependence between two random variables.
However, it has different properties than the MI, as outlined in [5].

######  Definition 3 (Differential entropy).

The differential entropy of a random variable $X$ is defined by

$H(X)=-\int p(x)\log p(x)dx.$ (3)

Main theoretical results. Having these definitions, we present our main
theoretical results in the two following theorems:

######  Theorem 1.

For a classification task with a ground-truth distribution $p(y|x)$, a fixed
training set $\mathcal{D}$, learned weights $w_{\mathcal{D}}$ and learned
classification function $f(y|x,w_{\mathcal{D}})$, the expected cross-entropy
loss of a machine learning algorithm on the test distribution is approximately

$\mathbb{E}_{w_{\mathcal{D}}}\left\\{KL(p(x,y)||f(x,y|w_{\mathcal{D}}))\right\\}+H(y|x)-L(w_{\mathcal{D}};x),$
(4)

where the expected cross-entropy test loss is defined by

$\mathbb{E}_{(x,y)\sim p(x,y)}\mathbb{E}_{w\sim p(w_{\mathcal{D}})}\\{-\log
f(y|x,w_{\mathcal{D}})\\}.$ (5)

Note that $KL$ signifies the Kullback-Leibler divergence and that we treat the
training set $\mathcal{D}$ as a fixed parameter, whereas $w_{\mathcal{D}}$ and
the examined test data $(x,y)$ are treated as random variables. We refer the
reader to Appendix A for the proof of Theorem 1.

In accordance with Theorem 1, the three terms that compose the expected cross-
entropy test loss represent three different aspects of the loss of a learning
algorithm performing a classification task:

  * •

Classifier mismatch:
$\mathbb{E}_{w_{\mathcal{D}}}\left\\{KL\left(p(x,y)||f(x,y|w_{\mathcal{D}})\right)\right\\}$,
measures the deviation of the learned classification function’s data
distribution $f(x,y|w_{\mathcal{D}})$ from the true distribution of the data
$p(x,y)$. It is measured by the KL-divergence, which is averaged over all
possible instances of $w$ parameterized by the fixed training set
$\mathcal{D}$. This term essentially measures the ability of the weights
learned from $\mathcal{D}$ to represent the true distribution of the data.

  * •

Intrinsic Bayes error: $H(y|x)$, represents the inherent uncertainty of the
labels given the data samples.

  * •

Lautum information: $L(w_{\mathcal{D}};x)$, represents the dependence between
$w_{\mathcal{D}}$ and $x$. Note that
$L(w_{\mathcal{D}};x)=\mathbb{E}_{w_{\mathcal{D}}}\\{KL(p(x)||p(x|w_{\mathcal{D}}))\\}$,
it essentially measures how much $p(x|w_{\mathcal{D}})$ deviates from $p(x)$
on average over the possible values of $w_{\mathcal{D}}$.

Our formulation suggests that a machine learning algorithm, which is trained
relying on empirical risk minimization, implicitly aims to maximize the Lautum
information $L(w_{\mathcal{D}};x)$ in order to minimize the cross-entropy
loss. At the same time, the algorithm aspires to minimize the KL-divergence
between the ground-truth distribution of the data and the learned
classification function. The intrinsic Bayes error cannot be minimized and
remains the inherent uncertainty in the task. Namely, the formulation in (4)
suggests that promoting a larger Lautum information between the data samples
and the learned model weights would be beneficial for reducing the model’s
test error on unseen data drawn from $p(x,y)$.

We next present a theorem that relates the classification error to the MI
rather than the Lautum information.

######  Theorem 2.

For a classification task with a ground-truth distribution $p(y|x)$, fixed
training set $\mathcal{D}$, learned weights $w_{\mathcal{D}}$ and learned
classification function $f(y|x,w_{\mathcal{D}})$, the expected cross-entropy
loss of a machine learning algorithm on the test distribution is equal to

$\displaystyle\begin{split}&\mathbb{E}_{x,w_{\mathcal{D}}}\left\\{KL(p(y|x,w_{\mathcal{D}})||f(y|x,w_{\mathcal{D}}))\right\\}+\\\
&H(x,y|w_{\mathcal{D}})-H(y)+I(x;w_{\mathcal{D}}),\end{split}$ (6)

where the expected cross-entropy test loss is defined by

$\mathbb{E}_{(x,y)\sim p(x,y)}\mathbb{E}_{w\sim p(w_{\mathcal{D}})}\\{-\log
f(y|x,w_{\mathcal{D}})\\}.$ (7)

The interplay between the different terms in Theorem 2 sheds light on several
aspects of the task’s expected cross-entropy test loss:

  * •

Conditional entropy: $H(x,y|w_{\mathcal{D}})$, a measurement of the model
weights’ capability to capture the data distribution.

  * •

Label entropy: $H(y)$, the intrinsic uncertainty in the labels.

  * •

Mutual information (MI): $I(x;w_{\mathcal{D}})$, an overfitting term, which
represents the memorization of the inputs by the model weights.

  * •

Model inefficiency: represented by the KL divergence term and relates to the
discrepancy between the learned model classification function
$f(y|x,w_{\mathcal{D}})$ and the true underlying distribution
$p(y|x,w_{\mathcal{D}})$.

We refer the reader to Appendix B for the proof of Theorem 2. Note that our
theorem, unlike in [4], uses the MI term to make the relation to the transfer
learning task.

##  4 Regularized information in Semi - Supervised transfer learning

We turn to show how our theory may be applied to the task of semi-supervised
transfer learning. In standard transfer learning, which consists of a pre-
transfer stage and a post-transfer stage, a neural network is trained on a
labeled source dataset and then fine-tuned on a smaller labeled target
dataset. In semi-supervised transfer learning, which we study here, we assume
that an additional large set of unlabeled examples from the target
distribution is available during training on the source data.

Semi-supervised transfer learning is highly beneficial in scenarios where the
available target dataset is only partially annotated. Making use of the
unlabeled part of this dataset, which is usually substantially bigger than the
labeled part, holds the potential to considerably improve the obtained
performance. Thus, if this unlabeled part is a-priori available, then using it
from the beginning of training can potentially improve the obtained
performance results.

In order to use the unlabeled samples of the target dataset during pre-
transfer training on the source dataset, we leverage the formulations in
Theorem 1, and Theorem 2. Both decompositions contain a term which relates to
the information that the neural network weights hold with respect to the input
samples. By introducing Lautum or MI regularizations in the pre-transfer
training stage, we promote a performance improvement on the target dataset.

###  4.1 Lautum Information regularization in semi-supervised transfer
learning

When inspecting the decomposition we obtained in Theorem 1, it is clear that
by using unlabeled samples the classifier mismatch term cannot be minimized
due to the lack of labels; the intrinsic Bayes error is a characteristic of
the task and cannot be minimized either; yet, the Lautum information does not
depend on the labels and can therefore be calculated and maximized.

When the Lautum information is calculated between the model weights and data
samples drawn from the target distribution, its maximization would encourage
the learned weights to better relate to these samples, and by extension to
better relate to the underlying probability distribution from which they were
drawn. Therefore, it is expected that an enlarged Lautum information will
yield an improved performance on the target test set. We accordingly aim at
maximizing $L(w_{\mathcal{D}};x)$ during training. The pre-transfer
maximization of the term $L(w_{\mathcal{D}};x)$, which is computed with
samples drawn from the target distribution, would make the learned weights
more inclined towards good performance on the target set right from the
beginning. At the same time, the cross-entropy loss at this stage is
calculated using labeled samples from the source dataset. In the post-transfer
stage, the cross-entropy loss is calculated using labeled samples from the
target dataset, and therefore $L(w_{\mathcal{D}};x)$ is implicitly maximized
during this stage as well. We have empirically observed that explicitly
maximizing the Lautum information between the unlabeled target samples and the
model weights during post-transfer training by imposing Lautum regularization
in addition to (or instead of) during pre-transfer training does not lead to
improved results.

To summarize, our semi-supervised transfer learning approach optimizes two
goals at the same time: (i) minimizing the classifier mismatch
$\mathbb{E}_{w_{\mathcal{D}}}\left\\{KL\left(p(x,y)||f(x,y|w_{\mathcal{D}})\right)\right\\}$,
which is achieved using the labeled data both for the source and the target
datasets during pre-transfer and post-transfer training respectively; and (ii)
maximizing the Lautum information $L(w_{\mathcal{D}};x)$, which is achieved
explicitly using the unlabeled target data during pre-transfer training by
imposing Lautum regularization, and in the post-transfer stage implicitly
through the minimization of the cross-entropy loss which is evaluated on the
labeled target data. Figure 1 summarizes our training scheme.

Estimating the Lautum information. We are interested in using the Lautum
information as a regularization term, which we henceforth refer to as ”Lautum
regularization”. Since computing the Lautum information between two random
variables requires knowledge of their probability distribution functions
(which are high-dimensional and hard to estimate), we assume that
$w_{\mathcal{D}}$ and $x$ are jointly Gaussian.111Note that this assumption is
not very plausible for the distribution of the input samples, however could be
reasonably plausible for the weights of the learned model This assumption
allows us to explicitly compute the terms that make up the Lautum information,
in addition to providing relatively simple computations and good experimental
results as shown in Section 5.

Since we only have one instance of the network weights at any specific point
during training, and due to the computational complexity of using the network
weights to estimate the Lautum information, we make use of the network’s
output logits to obtain an approximation for the model weights. Essentially,
the estimation is of the Jacobian matrix, i.e. the gradients of the learned
classifier output with respect to the input. An intuition to this
approximation may be obtained from the linear case of $f=Wx$, where the
derivative of the output with the respect to the input is
$W=\frac{\partial{f}}{\partial{x}}$. Estimating the Jacobian matrix comes with
a high computational cost as the matrix is large in any practical scenario.
Therefore, in order to perform more effective training and reduce the required
resources, we draw inspiration from the approaches described in [42, 43], and
use random projections of the classifier output logits to a scalar value, of
which we then take the derivative with respect to the inputs in order to
estimate the Jacobian. In this way we are able to efficiently use the network
outputs (logits) as a proxy for its weights.

In practice, to randomly project the logits to a scalar, we draw a random
vector from a normal distribution with zero mean and variance
$\frac{1}{\sqrt{K}}$ where $K$ represents the dimension of the logits vector.
Then, we calculate the inner product between the classifier output logits and
the random projection vector. Empirically, we noticed that by drawing the
random projection vector once every epoch and using it throughout the
iterations in the epoch, we obtain a more stable training and eventually
achieve better results.

As shown in [5], the Lautum information between two jointly Gaussian random
variables $(w,x)$ with covariance

$\displaystyle\left[\begin{array}[]{cc}\Sigma_{w}&\Sigma_{wx}\\\
\Sigma_{xw}&\Sigma_{x}\end{array}\right],$ (10)

where $\Sigma_{x}\succ 0$ and $\Sigma_{w}\succ 0$, is given by

$\displaystyle\begin{split}L(w;x)=&\log\left\\{\det(I-\Sigma_{x}^{-1}\Sigma_{xw}\Sigma_{w}^{-1}\Sigma_{wx})\right\\}\\\
&+2tr((I-\Sigma_{x}^{-1}\Sigma_{xw}\Sigma_{w}^{-1}\Sigma_{wx})^{-1}-I).\end{split}$
(11)

where $det$ represents the matrix Determinant, and $tr$ is the matrix Trace.

The covariance matrix of our target dataset $\Sigma_{x}$ is evaluated once
before training using the entire target training set, whereas
$\Sigma_{w},\Sigma_{wx},\Sigma_{xw}$ are evaluated during training using the
current mini-batch in every iteration. All these matrices are estimated using
standard sample covariance estimation based on the current mini-batch in every
iteration, e.g.

$\displaystyle\Sigma_{x}=\frac{1}{N_{batch}}\sum_{i=1}^{N_{batch}}(x_{i}-\mu_{x})(x_{i}-\mu_{x})^{T},$
(12)

and

$\displaystyle\Sigma_{xw}=\frac{1}{N_{batch}}\sum_{i=1}^{N_{batch}}(x_{i}-\mu_{x})(w_{i}-\mu_{w})^{T},$
(13)

where

$\mu_{x}=\frac{1}{N_{batch}}\sum_{i=1}^{N_{batch}}x_{i},~{}~{}~{}~{}~{}~{}\mu_{w}=\frac{1}{N_{batch}}\sum_{i=1}^{N_{batch}}w_{i}$

represent the sample mean values of $x$ and $w$ respectively (i.e. their
average values in the current mini-batch) and $N_{batch}$ denotes the mini-
batch size. The dimensions of these matrices are
$\Sigma_{x}\in\mathbb{R}^{D\times D},\Sigma_{w}\in\mathbb{R}^{K\times
K},\Sigma_{xw}\in\mathbb{R}^{D\times K},\Sigma_{wx}\in\mathbb{R}^{K\times D}$.
Note that $\Sigma_{wx}=\Sigma_{xw}^{T}$ hence only one of these matrices has
to be calculated from the samples in every training iteration.

Since these are high-dimensional matrices, obtaining a numerically stable
sample estimate would require a large amount of data, which would require the
use of a very large mini-batch. This constraint poses both a hardware problem
(since standard GPUs cannot fit mini-batches of thousands of examples) and a
potential generalization degradation (as smaller mini-batches have been linked
to improved generalization [44]). To overcome this issue, similarly to batch
normalization, we normalize both the logits derivative vector (which is our
proxy for the weights), and the input images by their respective standard
deviations along the batch dimension. By doing so, we improve the numerical
stability of the estimated matrices towards the inversion in the following
Lautum estimation.

We additionally use a standard exponentially decaying moving-average
estimation of the three matrices $\Sigma_{w},\Sigma_{wx},\Sigma_{xw}$, to
obtain numerical stability. We denote by $\alpha$ the decay rate, and get the
following update rule for the three covariance matrices in every training
iteration:

$\displaystyle\Sigma_{(n)}=\alpha\Sigma_{(n-1)}+(1-\alpha)\Sigma_{batch},$
(14)

where $n$ denotes the training iteration and $\Sigma_{batch}$ denotes the
sample covariance matrix calculated using the current mini-batch. Using an
exponentially decaying moving-average is of particularly high importance for
$\Sigma_{w}$, which is inverted to compute the Lautum regularization term.

Training with Lautum regularization. Once the Lautum information has been
estimated, our loss function for pre-transfer training is:

$\displaystyle\begin{split}&Loss=\sum_{i=1}^{N}\sum_{k=1}^{K}-y_{ik}^{s}\log
f_{k}(x_{i}^{s}|w_{\mathcal{D}})-\lambda L(w_{\mathcal{D}};x^{t}),\end{split}$
(15)

where the superscript $s$ represents the source dataset, and $t$ represents
the target dataset. Note that the cross-entropy loss is calculated using
labeled samples from the source training set whereas the Lautum regularization
uses only unlabeled samples from the target training set. Also note that
$y_{i}$ represents the ground truth label of the sample $x_{i}$;
$f(x_{i}|w_{\mathcal{D}})$ represents the network’s estimated post softmax
label for that sample; and $L(w_{\mathcal{D}};x)$ is calculated as detailed
above. We emphasize that the Lautum regularization term is subtracted and not
added to the cross-entropy loss as we aim at _maximizing_ the Lautum
information during training.

The loss function for post-transfer training consists of a standard cross-
entropy loss:

$\displaystyle\begin{split}&Loss=\sum_{i=1}^{N}\sum_{k=1}^{K}-y_{ik}^{t}\log
f_{k}(x_{i}^{t}|w_{\mathcal{D}}).\end{split}$ (16)

Note that at this stage the cross-entropy loss, which is calculated using
labeled target samples, implicitly includes the Lautum term of the target data
(see Theorem 1).

###  4.2 Mutual Information (MI) regularization in semi-supervised transfer
learning

We now turn to inspect the decomposition we obtained in Theorem 2. Firstly,
the KL divergence term, $KL(p(y|x,w_{\mathcal{D}})||f(y|x,w_{\mathcal{D}}))$
and the conditional entropy term $H(x,y|w_{\mathcal{D}})$ can both be
minimized where labels are available only. Second, the term $H(y)$, which
represents the intrinsic uncertainty of the labels, is inherent to the data
and therefore cannot be minimized. The last term, $I(x;w_{\mathcal{D}})$, does
not depend on the task labels. Therefore, by leveraging the unlabeled samples
from the target dataset, it is possible to explicitly regularize the
information that the model parameters hold with respect to the inputs, and
thereby improve model transferability.

Another motivation for minimizing the information the model parameters hold
with respect to the inputs stems from the information bottleneck (IB) theory
[6], which defines the IB Lagrangian by

$\mathcal{L}(p(z|x))=H(y|z)+\beta I(x;z).$ (17)

The IB Lagrangian addresses the relevance of the information in a
representation via a tradeoff between its sufficiency to the task $H(y|z)$
)which is simply the cross-entropy loss), and its minimality $I(x;z)$, where
the parameter $\beta$ allows tradeoff tuning. By regularizing the learned
representation $z$ to hold as minimal information as possible with respect to
the inputs, we promote the representation to only hold information relevant to
the task without overfitting the training data.

The regularization term in the IB is inherently different from our analysis in
Theorem 2. The former relates the representation $z$ to the inputs, whereas
our analysis relates the model weights to the inputs. To bridge this
difference we rely on the work in [4], which showed that
$I(x;w_{\mathcal{D}})$ tightly bounds $I(x;z)+TC(z)$, which is the sum of the
mutual information between the inputs and the representation and the total
correlation of the representation $z$. Combining this argument with Theorem 2,
it follows that we may minimize the MI between the target inputs and their
respective feature representations, thereby inducing the minimization of the
MI between the target inputs and the network weights. Relating this insight to
the context of semi-supervised transfer learning, by using the unlabeled data
of the target dataset, and regularizing the information the representations
hold with respect to their inputs, we are able to optimize the cross-entropy
loss of the target dataset.

Estimating the Mutual Information. Estimating the MI can be challenging,
especially for high-dimensional distributions. To this end, we use the
variational approach suggested in [40], which is derived from the MINE method
[38].

The MINE [38] approach uses a neural network to estimate a lower bound for the
Donsker-Varadhan representation [45] of the KL divergence. Plugging in the MI
as defined in Definition 1, we get a lower bound for the MI value. The
expressive power of a neural network enables this estimator to both be
consistent and tightly estimate the MI. The MINE bound reads as follows:

$KL(P,Q)\geq\sup_{M\in\mathcal{F}}(\mathbb{E}_{P}[M]-log(\mathbb{E}_{Q}[e^{M}]),$
(18)

where $\mathcal{F}$ is a class of functions (neural networks in our case),
$M:\Omega\xrightarrow{}\mathbb{R}$ is a specific neural network defined by the
selected architecture and its parameters, $P$ is the empirical $p(x,z)$ which
is the joint distribution of the target dataset inputs $x$ and their
respective representations $z$, $Q$ is the empirical product of the marginals
$p(x)$, $p(z)$ which are derived from the empirical samples of the target
dataset, and lastly the neural network features are
$z_{i}=f_{w_{\mathcal{D}}}(x_{i})$.

Note that even though the MINE [38] estimator is unbiased, its variance is
proportional to the MI value. Therefore, training the MI neural estimator in
cases where the true MI value is large may lead to an unstable training and
noisy results. To mitigate this, we use the suggested estimator from [40],
which is biased and controls the bias-variance tradeoff using a clipping
parameter. Namely, the $\log(\mathbb{E}_{Q}[e^{M}])$ term from the lower bound
in (18) is modified to $log(\mathbb{E}_{Q}[Clip(e^{M},e^{-\tau},e^{\tau})])$.
Tuning the clipping value $\tau$ allows one to tradeoff the bias and variance.

Training with Mutual Information regularization. The estimated MI is used in
the pre-transfer learning stage via a regularization term added to the cross-
entropy loss, yielding the following optimization function for training:

$\displaystyle\begin{split}&Loss=\sum_{i=1}^{N}\sum_{k=1}^{K}-y_{ik}^{s}\log
f_{k}(x_{i}^{s}|w_{\mathcal{D}})+\lambda I(x^{t};z^{t}).\end{split}$ (19)

where in this case as well, the superscript $s$ represents the source dataset,
and $t$ represents the target dataset. The hyper-parameter $\lambda$ is
optimized as part of our experiments. Similarly to the Lautum regularization,
we aim at minimizing the estimated value of the MI only for the part of the
target dataset for which labels are not available.

Note that there is an interplay between the MI estimation and the learned
classification function in the training process; the MI value stems from the
MI neural estimator which makes use of the output features of the
classification network. At the same time, training the classification network
requires the MI estimation to calculate the regularization term. To address
this training inter-dependency, both networks are trained conjointly. In order
to facilitate a stable training process, we alternately train the two
networks, i.e. we train the MI estimator for a few epochs, and then train the
classification network for a few epochs. This process is repeated until
convergence.

The post-transfer-learning stage follows the same process as in the Lautum
case, i.e., the loss in (16) is employed. In this case as well, MI
regularization is not explicitly used but implicitly encapsulated in the
cross-entropy loss.

###  4.3 Comparing the Mutual Information and the Lautum Information
regularizations

Both MI regularization and Lautum Information regularization aim at
controlling the information that the model parameters hold with respect to the
inputs. However, each of them has an advantage over the other.

One fundamental difference is related to the assumptions that each method
makes about the data: Lautum regularization assumes that the inputs and model
weights are jointly Gaussian, whereas MI makes no such assumptions about
either the data or model parameter distributions.

Another difference is that the Lautum regularization aims at maximizing an
unbounded regularization term, a fact that may introduce a stability challenge
in the training process and hyperparameter tuning. In contrast, MI
regularization minimizes a bounded quantity, which makes the training process
significantly more stable.

Computational complexity is a relevant differentiating factor as well. In
cases of large inputs, Lautum regularization involves the computation of very
large covariance matrices and their inversion, which requires heavy computing
resources and therefore slows down training.

Remarkably, one key disadvantage MI regularization has compared to Lautum
regularization lies in the estimated regularization term itself: the Lautum
Information approach estimates the value of the Lautum quantity itself,
whereas the MI approach estimates a lower bound of the regularization
quantity. Therefore, MI regularization has an implicit impact on the value we
actually aim at controlling, contrary to Lautum regularization’s explicit
impact.

Note that although the MI regularization holds several key advantages compared
to the Lautum information regularization, including superior experimental
results as outlined in Section 5, we still find the latter a relevant case
study. It paves a path for an information-theoretic-led analysis while also
serving as an experimental stepping-stone in our research.

##  5 Experiments

In order to demonstrate the advantages of semi-supervised transfer learning
both with the Lautum regularization and with the MI regularization, we
performed several experiments on image classification tasks using deep neural
networks. Note, though, that our theoretical derivations also apply to other
machine learning models.

###  5.1 Experimental setup

We train deep neural networks and perform transfer learning from the original
source dataset to the target dataset. In our experiments we use the original
labeled source training set as is, and split the target training set into two
parts. The first part is very small and contains labeled samples, whereas the
second part consists of the remainder of the target training set and contains
unlabeled samples only (the labels are discarded). The performance is
evaluated by the final accuracy on the target test set.

We examined different methods of transfer learning: (1) standard supervised
transfer which uses the labeled samples only. (2) Temporal Ensembling semi-
supervised learning as outlined in [7], applied in a transfer learning
setting; Temporal Ensembling is applied in the post-transfer training stage.
(3) Mk-MMD [8], which is based on 19 different Gaussian kernels with different
standard deviations. Mk-MMD is applied in the pre-transfer training stage. (4)
Lautum regularization - our technique as described in Section 4.1. (5) Both
Temporal Ensembling and Lautum regularization. Note that Temporal Ensembling
is applied in the post-transfer training stage, whereas Lautum regularization
is applied in the pre-transfer training stage. (6) MI regularization, our
second technique which is described in Section 4.2. (7) Both Lautum
regularization and MI regularization applied in conjunction.

As presented in Sections 4.1 and 4.2 our training paradigm consists of two
stages. When employing our proposed methods, in the first stage we train the
network using our fully labeled source training set while imposing either
Lautum or MI or both regularizations using the unlabeled samples from the
target training set. We use the same mini-batch size both for the calculation
of the cross-entropy loss (using labeled source samples) and for the
computation of the regularization terms (using unlabeled target samples).

In the Lautum experiments, we also use an exponentially decaying moving
average to ascertain numerical stability in the estimation of the covariance
matrices $\Sigma_{w},\Sigma_{wx},\Sigma_{xw}$. Note that the matrix
$\Sigma_{x}$ is calculated once before training and remains constant
throughout it. In the MI experiments we train the MI neural estimator
alternately with the classification network training. We empirically observed
that this methodology provides a more stable training process, as particularly
in our case we have a mutual dependency between the MI estimator training,
which relies on the extracted features from the classifier, and the classifier
training, which is regularized using the MI estimation.

In the second stage, we perform a transfer to the target set by training
(fine-tuning) the entire network using the labeled samples from the target
training set, where the mini-batch size remains the same as before. As in
[28], we fine-tune the entire network as this approach best fits the settings
of semi-supervised transfer learning. As previously mentioned, we do not apply
any information regularization at this stage as we empirically found it does
not improve the obtained performance results.

We performed our experiments on the MNIST and CIFAR-10 datasets. For MNIST we
examined the transfer to the notMNIST dataset, which consists of 10 classes
representing the letters A-J. The notMNIST dataset is similar to the MNIST
dataset in its grayscale styling and image size, yet it differs in content.
For CIFAR-10 we examined the transfer to 10 specific classes of the CIFAR-100
dataset (specifically, classes 0, 10, 20,…,90 of CIFAR-100, which we
reclassified as classes 0, 1, 2,…,9 respectively). These CIFAR-100 classes are
different from the corresponding CIFAR-10 ones in content. For example, class
0 in CIFAR-10 represents airplanes whereas class 0 in CIFAR-100 represents
beavers. Both in the MNIST $\rightarrow$ notMNIST case and the CIFAR-10
$\rightarrow$ CIFAR-100 (10 classes) case we used the same CNN as in [7].
Appendix C details the network architecture.

###  5.2 MNIST to notMNIST results

In order for the input images to fit the network’s input we resized the MNIST
and notMNIST images to $32$x$32$ pixels and transformed each of them into RGB
format. Training was done using the Adam optimizer [46] and a mini-batch size
of 50 inputs. Standard supervised training with this network on the entire
MNIST dataset lead to a test accuracy of $99.01\%$ on MNIST.

For each of the transfer learning methods we examined three different splits
of the notMNIST training dataset which consists of 200,000 samples to an
unlabeled part and a labeled part: (1) unlabeled part of 199,950 samples and a
labeled part of 50 samples; (2) unlabeled part of 199,900 samples and a
labeled part of 100 samples; (3) unlabeled part of 199,800 samples and a
labeled part of 200 samples. All three options use very few labeled samples in
order to fairly represent realistic semi-supervised learning scenarios; in all
three options 99.9% or more of the training data is unlabeled.

In the Lautum regularization case, we used a decay rate of $\alpha=0.999$ for
the exponentially decaying moving average estimation of the three matrices
$\Sigma_{w},\Sigma_{wx},\Sigma_{xw}$. Further comprehensive hyper-parameter
tuning (e.g. for the value of $\lambda$ which controls the regularization term
weight) was done for all examined methods.

Using the settings outlined above we obtained the results shown in Table I for
the MNIST $\rightarrow$ notMNIST case. The advantage of using information-
based regularization is evident from the results, as it outperforms the other
compared methods in all the examined target training set splits. In general,
the Temporal Ensembling method by itself does not yield very competitive
results compared to standard transfer learning. However, the MI regularization
provides the best target test set accuracy over all the examined target
training set splits. Though our analysis in Section 3 does not establish
theoretical foundations to support using both MI and Lautum regularizations in
conjunction, we performed this experiment to present a comprehensive empirical
setup. Our experiments show that imposing both regularizations degrades the
target test accuracy compared to their individual applications.

In addition, we examined the contributions of Lautum and MI regularizations
with respect to the number of labeled target training samples even further. We
tested several additional ratios of labeled target training samples up to
2.5%, and compared our suggested methods to standard transfer learning. The
results are shown in Table II. As the proportion of labeled samples in the
target training set increases, the contribution of the regularizations is
diminished as expected. This behavior is explained by Theorem 1 and Theorem 2,
as in both decompositions the role of the KL divergence term is enhanced by
the increased number of labeled samples during cross-entropy loss
minimization.

Method | # labeled | Accuracy
---|---|---
Standard | 50 | 34.02%
TE | 50 | 37.28%
Mk-MMD | 50 | 46.72%
Lautum + MI | 50 | 46.51%
Lautum | 50 | 48.42%
Lautum + TE | 50 | 67.3%
MI | 50 | 70.50%
Standard | 100 | 57.58%
TE | 100 | 61.45%
Mk-MMD | 100 | 63.32%
Lautum + MI | 100 | 62.01%
Lautum | 100 | 65.94%
Lautum + TE | 100 | 76.8%
MI | 100 | 78.38%
Standard | 200 | 67.78%
TE | 200 | 74.87%
Mk-MMD | 200 | 80.35%
Lautum + MI | 200 | 76.31%
Lautum | 200 | 82.8%
Lautum + TE | 200 | 84.5%
MI | 200 | 85.85%
TABLE I: MNIST $\rightarrow$ notMNIST, target test set accuracy comparison between various methods. Results are presented for different amounts of labeled target training samples.  # labeled | Standard TL | Lautum | MI
---|---|---|---
50 | 34.02% | 48.42% | 70.50%
100 | 57.58% | 65.94% | 78.38%
200 | 67.78% | 82.8% | 85.85%
500 | 75.01% | 87.91% | 89.40%
1000 | 81.26% | 90.04% | 90.53%
2500 | 86.10% | 91.85% | 92.64%
5000 | 89.40% | 93.78% | 94.45%
TABLE II: MNIST $\rightarrow$ notMNIST, the contirbution of (i) Lautum
regularization, and (ii) MI regularization, to the target test set accuracy as
labeled sample size increases.

###  5.3 CIFAR-10 to CIFAR-100 (10 classes) results

In the CIFAR-10 $\rightarrow$ CIFAR-100 (10 classes) case training was done
using the Adam optimizer [46] and a mini-batch size of 100 inputs. Standard
supervised training with this network on the entire CIFAR-10 dataset lead to a
test accuracy of $85.09\%$ on CIFAR-10.

Our target set consists of 10 classes from the CIFAR-100 dataset. Accordingly,
our training target set consists of 5,000 samples and our test target set
consists of 1,000 samples. We examined the same seven transfer learning
techniques as in the MNIST $\rightarrow$ notMNIST case, where for each we
examined three different splits of the CIFAR-100 (10 classes) training set
into an unlabeled part and a labeled part: (1) unlabeled part of 4,900 samples
and a labeled part of 100 samples; (2) unlabeled part of 4,800 samples and a
labeled part of 200 samples; (3) unlabeled part of 4,500 samples and a labeled
part of 500 samples. All three options use a small number of labeled samples
in order to fairly represent realistic semi-supervised learning scenarios; in
all three options 90% or more of the data is unlabeled. We used a decay rate
of $\alpha=0.999$ for the exponentially decaying moving average estimation of
the three matrices $\Sigma_{w},\Sigma_{wx},\Sigma_{xw}$. In this case as well,
further comprehensive hyper-parameter tuning was done for all examined
methods.

Using the settings outlined above we obtained the results shown in Table III
for the CIFAR-10 $\rightarrow$ CIFAR-100 (10 classes) case. It is evident from
the results that in the CIFAR-10 $\rightarrow$ CIFAR-100 (10 classes) case as
well, using either Lautum regularization or MI regularization improves the
post-transfer performance on the target test set and outperforms Temporal
Ensembling and Mk-MMD. These results are consistent with the MNIST
$\rightarrow$ NotMNIST results, as the MI regularization outperforms all other
methods in this case too. Particularly in the case of 100 labeled target
training samples, the obtained performance boost is significantly larger than
the other methods.

Method | # labeled | Accuracy
---|---|---
Standard | 100 | 39.90%
TE | 100 | 42.20%
Mk-MMD | 100 | 45.30%
Lautum + MI | 100 | 43.14%
Lautum | 100 | 46.42%
Lautum + TE | 100 | 46.80%
MI | 100 | 53.10%
Standard | 200 | 52.80%
TE | 200 | 54.60%
Mk-MMD | 200 | 59.30%
Lautum + MI | 200 | 55.44%
Lautum | 200 | 60.70%
Lautum + TE | 200 | 61.40%
MI | 200 | 63.20%
Standard | 500 | 64.50%
TE | 500 | 66.50%
Mk-MMD | 500 | 68.00%
Lautum + MI | 500 | 66.93%
Lautum | 500 | 70.60%
Lautum + TE | 500 | 71.05%
MI | 500 | 72.82%
TABLE III: CIFAR-10 $\rightarrow$ CIFAR-100 (10 classes), target test set
accuracy comparison between various methods. Experimental results are
presented for different amounts of labeled training target samples.

##  6 Conclusions

We have proposed a new semi-supervised transfer learning approach for machine
learning algorithms that are trained using the cross-entropy loss. Our
approach is backed by information-theoretic derivations and exemplifies how
one may make good use of unlabeled target set samples along with just a few
labeled samples to improve model performance on the target dataset. Our
approach relies on two techniques: (i) the maximization of the Lautum
information between unlabeled samples from the target set and an algorithm’s
learned features by using the Lautum information as a regularization term;
(ii) the minimization of the Mutual information between unlabeled samples from
the target set and a model’s learned parameters by using the Mutual
information as a regularization term. As shown, regularizing either of these
terms minimizes the cross-entropy test loss on the target dataset and thereby
improves performance as indicated by our experimental results.

The advantages of the MI regularization compared to the Lautum regularization
are also supported by the experimental results where the performance of the
former surpasses that of the latter in all tested settings.

We have also shown that our approaches surpass the performance of prominent
semi-supervised techniques in a transfer learning setting.

Future work will focus on alternative approximations of the used information-
theoretic quantities (Lautum information and MI) which could potentially yield
better performance or reduce the introduced computational overhead. In
addition, our approach has the potential to be applied to other tasks as well,
such as multi-task learning or domain adaptation. Incorporating new techniques
to mitigate the effects of training using an imbalanced dataset could also be
of interest. We defer these directions to future research.

## Acknowledgments

This work was supported by the ERC-StG SPADE grant.

## References

  * [1] I. Goodfellow, Y. Bengio, and A. Courville, _Deep Learning_. MIT Press, 2016.
  * [2] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell, “Decaf: A deep convolutional activation feature for generic visual recognition,” in _Proceedings of the 31st International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, E. P. Xing and T. Jebara, Eds., vol. 32, no. 1. Bejing, China: PMLR, 22–24 Jun 2014, pp. 647–655. [Online]. Available: http://proceedings.mlr.press/v32/donahue14.html
  * [3] X. LI, Y. Grandvalet, and F. Davoine, “Explicit inductive bias for transfer learning with convolutional networks,” in _Proceedings of the 35th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, J. Dy and A. Krause, Eds., vol. 80. Stockholm Sweden: PMLR, 10–15 Jul 2018, pp. 2825–2834. [Online]. Available: http://proceedings.mlr.press/v80/li18a.html
  * [4] A. Achille and S. Soatto, “Emergence of invariance and disentanglement in deep representations,” _The Journal of Machine Learning Research_ , vol. 19, no. 1, pp. 1947–1980, 2018.
  * [5] D. P. Palomar and S. Verdu, “Lautum information,” _IEEE Trans. Inform. Theory_ , vol. 54, no. 3, pp. 964–975, March 2008.
  * [6] N. Tishby, F. C. Pereira, and W. Bialek, “The information bottleneck method,” _arXiv preprint physics/0004057_ , 2000.
  * [7] S. Laine and T. Aila, “Temporal ensembling for semi-supervised learning,” in _ICLR_ , 2017.
  * [8] A. Gretton, D. Sejdinovic, H. Strathmann, S. Balakrishnan, M. Pontil, K. Fukumizu, and B. K. Sriperumbudur, “Optimal kernel choice for large-scale two-sample tests,” in _Advances in Neural Information Processing Systems 25_ , F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2012, pp. 1205–1213.
  * [9] S. J. Pan and Q. Yang, “A survey on transfer learning,” _IEEE Trans. Knowl. Data Eng._ , vol. 22, no. 10, pp. 1345–1359, 2010. [Online]. Available: https://doi.org/10.1109/TKDE.2009.191
  * [10] C. Tan, F. Sun, T. Kong, W. Zhang, C. Yang, and C. Liu, “A survey on deep transfer learning,” in _ICANN_ , 2018.
  * [11] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable are features in deep neural networks?” in _Advances in Neural Information Processing Systems 27_ , Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2014, pp. 3320–3328. [Online]. Available: http://papers.nips.cc/paper/5347-how-transferable-are-features-in-deep-neural-networks.pdf
  * [12] A. K. Lampinen and S. Ganguli, “An analytic theory of generalization dynamics and transfer learning in deep linear networks,” in _ICLR_ , 2019.
  * [13] X. Wang and J. Schneider, “Flexible transfer learning under support and model shift,” in _Advances in Neural Information Processing Systems 27_ , Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2014, pp. 1898–1906. [Online]. Available: http://papers.nips.cc/paper/5632-flexible-transfer-learning-under-support-and-model-shift.pdf
  * [14] Y. Wei, Y. Zhang, J. Huang, and Q. Yang, “Transfer learning via learning to transfer,” in _Proceedings of the 35th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, J. Dy and A. Krause, Eds., vol. 80. Stockholm Sweden: PMLR, 10–15 Jul 2018, pp. 5085–5094. [Online]. Available: http://proceedings.mlr.press/v80/wei18a.html
  * [15] A. R. Zamir, A. Sax, W. Shen, L. J. Guibas, J. Malik, and S. Savarese, “Taskonomy: Disentangling task transfer learning,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2018.
  * [16] X. Li, H. Xiong, H. Wang, Y. Rao, L. Liu, and J. Huan, “Delta: Deep learning transfer using feature map with attention for convolutional networks,” in _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_ , 2019.
  * [17] X. Chen, S. Wang, B. Fu, M. Long, and J. Wang, “Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning,” in _Advances in Neural Information Processing Systems_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds. Curran Associates, Inc., 2019.
  * [18] C. Fannjiang and J. Listgarten, “Autofocused oracles for model-based design,” in _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 12 945–12 956. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2020/file/972cda1e62b72640cb7ac702714a115f-Paper.pdf
  * [19] X. Zhu, “Semi-supervised learning literature survey,” in _University of Wisconsin-–Madison, Tech. Rep. 1530_ , 2006.
  * [20] Y. Grandvalet and Y. Bengio, “Semi-supervised learning by entropy minimization,” in _Advances in Neural Information Processing Systems 17_ , L. K. Saul, Y. Weiss, and L. Bottou, Eds. MIT Press, 2005, pp. 529–536.
  * [21] P. Häusser, A. Mordvintsev, and D. Cremers, “Learning by association - A versatile semi-supervised training method for neural networks,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_ , 2017, pp. 626–635. [Online]. Available: https://doi.org/10.1109/CVPR.2017.74
  * [22] H. Sun, W. W. Cohen, and L. Bing, “Semi-supervised learning with declaratively specified entropy constraints,” in _Advances in Neural Information Processing Systems 31_ , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds. Curran Associates, Inc., 2018, pp. 4430–4440.
  * [23] N. Papernot, M. Abadi, Úlfar Erlingsson, I. Goodfellow, and K. Talwar, “Semi-supervised knowledge transfer for deep learning from private training data,” in _ICLR_ , 2017.
  * [24] A. Oliver, A. Odena, C. A. Raffel, E. D. Cubuk, and I. J. Goodfellow, “Realistic evaluation of deep semi-supervised learning algorithms,” in _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada._ , 2018, pp. 3239–3250.
  * [25] N. Jean, S. M. Xie, and S. Ermon, “Semi-supervised deep kernel learning: Regression with unlabeled data by minimizing predictive variance,” in _Advances in Neural Information Processing Systems 31_ , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds. Curran Associates, Inc., 2018, pp. 5322–5333.
  * [26] A. Kumar, P. Sattigeri, and T. Fletcher, “Semi-supervised learning with gans: Manifold invariance with improved inference,” in _Advances in Neural Information Processing Systems 30_ , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds. Curran Associates, Inc., 2017, pp. 5534–5544.
  * [27] R. Gupta, S. Sahu, C. Espy Wilson, and S. Narayanan, “Semi-supervised and transfer learning approaches for low resource sentiment classification,” in _Proceedings of ICASSP_ , April 2018.
  * [28] H.-Y. Zhou, A. Oliver, J. Wu, and Y. Zheng, “When semi-supervised learning meets transfer learning: Training strategies, models and datasets,” _arXiv_ , vol. abs/1812.05313, 2018.
  * [29] A. Abuduweili, X. Li, H. Shi, C. Xu, and D. Dou, “Adaptive consistency regularization for semi-supervised transfer learning,” in _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021_. Computer Vision Foundation / IEEE, 2021, pp. 6923–6932.
  * [30] N. Tishby, F. C. Pereira, and W. Bialek, “The information bottleneck method,” _The 37’th Allerton Conference on Communication, Control, and Computing_ , 1999.
  * [31] O. Shamir, S. Sabato, and N. Tishby, “Learning and generalization with the information bottleneck,” _Theor. Comput. Sci._ , vol. 411, no. 29-30, pp. 2696–2711, 2010.
  * [32] N. Friedman, O. Mosenzon, N. Slonim, and N. Tishby, “Multivariate information bottleneck,” in _UAI ’01: Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence, University of Washington, Seattle, Washington, USA, August 2-5, 2001_ , 2001, pp. 152–161.
  * [33] N. Tishby and N. Zaslavsky, “Deep learning and the information bottleneck principle,” in _2015 IEEE Information Theory Workshop, ITW 2015, Jerusalem, Israel, April 26 - May 1, 2015_ , 2015, pp. 1–5. [Online]. Available: https://doi.org/10.1109/ITW.2015.7133169
  * [34] R. Shwartz-Ziv and N. Tishby, “Opening the black box of deep neural networks via information,” _CoRR_ , vol. abs/1703.00810, 2017. [Online]. Available: http://arxiv.org/abs/1703.00810
  * [35] A. M. Saxe, Y. Bansal, J. Dapello, M. Advani, A. Kolchinsky, B. D., and T. D. D. Cox, “On the information bottleneck theory of deep learning,” in _ICLR_ , 2018.
  * [36] M. Gabrié, A. Manoel, C. Luneau, j. barbier, N. Macris, F. Krzakala, and L. Zdeborová, “Entropy and mutual information in models of deep neural networks,” in _Advances in Neural Information Processing Systems 31_ , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds. Curran Associates, Inc., 2018, pp. 1821–1831.
  * [37] Y. Bu, G. Aminian, L. Toni, G. W. Wornell, and M. Rodrigues, “Characterizing and understanding the generalization error of transfer learning with gibbs algorithm,” in _Proceedings of The 25th International Conference on Artificial Intelligence and Statistics_ , ser. Proceedings of Machine Learning Research, G. Camps-Valls, F. J. R. Ruiz, and I. Valera, Eds., vol. 151. PMLR, 28–30 Mar 2022, pp. 8673–8699.
  * [38] M. I. Belghazi, A. Baratin, S. Rajeshwar, S. Ozair, Y. Bengio, A. Courville, and D. Hjelm, “Mutual information neural estimation,” in _Proceedings of the 35th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, J. Dy and A. Krause, Eds., vol. 80. PMLR, 10–15 Jul 2018, pp. 531–540.
  * [39] B. Poole, S. Ozair, A. Van Den Oord, A. Alemi, and G. Tucker, “On variational bounds of mutual information,” in _Proceedings of the 36th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09–15 Jun 2019, pp. 5171–5180.
  * [40] J. Song and S. Ermon, “Understanding the limitations of variational mutual information estimators,” in _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_ , 2020\.
  * [41] D. Jakubovitz, R. Giryes, and M. R. D. Rodrigues, _Generalization Error in Deep Learning_. Cham: Springer International Publishing, 2019, pp. 153–193.
  * [42] J. Hoffman, D. A. Roberts, and S. Yaida, “Robust learning with jacobian regularization,” 2019.
  * [43] Z. Goldfeld and K. Greenewald, “Sliced mutual information: A scalable measure of statistical dependence,” in _Advances in Neural Information Processing Systems_ , M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 17 567–17 578.
  * [44] N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang, “On large-batch training for deep learning: Generalization gap and sharp minima,” in _ICLR_ , 2017. [Online]. Available: https://arxiv.org/pdf/1609.04836.pdf
  * [45] M. D. Donsker and S. S. Varadhan, “Asymptotic evaluation of certain markov process expectations for large time. iv,” _Communications on pure and applied mathematics_ , vol. 36, no. 2, pp. 183–212, 1983.
  * [46] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _ICLR_ , 2015. [Online]. Available: http://arxiv.org/abs/1412.6980

##  Appendix A Proof of Theorem 1

Let us reiterate Theorem 1 before formally proving it.

Theorem 1 _For a classification task with a ground-truth distribution
$p(y|x)$, fixed training set $\mathcal{D}$, learned weights $w_{\mathcal{D}}$
and learned classification function $f(y|x,w_{\mathcal{D}})$, the expected
cross-entropy loss of a machine learning algorithm on the test distribution is
approximately_

$\mathbb{E}_{w_{\mathcal{D}}}\left\\{KL(p(x,y)||f(x,y|w_{\mathcal{D}}))\right\\}+H(y|x)-L(w_{\mathcal{D}};x).$
(20)

Proof. The expected cross-entropy loss of the learned classification function
$f(y|x,w_{\mathcal{D}})$ on the test distribution $p(x,y)$ can be approximated
by

$\mathbb{E}_{(x,y)\sim p(x,y)}\mathbb{E}_{w\sim p(w_{\mathcal{D}})}\\{-\log
f(y|x,w_{\mathcal{D}})\\},$ (21)

where instead of taking the expectation over $p(x,y,w_{\mathcal{D}})$, we take
the expectation over $p(x,y)$ and $p(w_{\mathcal{D}})$ separately. Our
approximation is motivated by the fact that the test distribution $p(x,y)$ is
independent from $p(w_{\mathcal{D}})$ as post-training $w_{\mathcal{D}}$ is no
longer a random variable but rather a fixed parameter of the problem.
Explicitly, (21) can be written as 222Since the values of $y$ are discrete it
is more accurate to sum instead of integrate over them. Yet, for the
simplicity of the proof we present the derivations using integration.

$-\iiint p(x,y)p(w_{\mathcal{D}})\log f(y|x,w_{\mathcal{D}})\hskip
1.42271ptdx\hskip 1.42271ptdy\hskip 1.42271ptdw_{\mathcal{D}}.$ (22)

To compare the learned classifier with the true classification of the data we
develop (22) further as follows:

$=-\iiint
p(x,y)p(w_{\mathcal{D}})\log\left\\{\frac{f(y|x,w_{\mathcal{D}})}{p(y|x,w_{\mathcal{D}})}p(y|x,w_{\mathcal{D}})\right\\}\hskip
1.42271ptdx\hskip 1.42271ptdy\hskip 1.42271ptdw_{\mathcal{D}}.$ (23)

Using standard logarithm arithmetic we get the following expression:

$\begin{split}=&\underbrace{-\iiint
p(x,y)p(w_{\mathcal{D}})\log\left\\{\frac{f(y|x,w_{\mathcal{D}})}{p(y|x,w_{\mathcal{D}})}\right\\}\hskip
1.42271ptdx\hskip 1.42271ptdy\hskip 1.42271ptdw_{\mathcal{D}}}_{(\star)}\\\
&\underbrace{-\iiint p(x,y)p(w_{\mathcal{D}})\log p(y|x,w_{\mathcal{D}})\hskip
1.42271ptdx\hskip 1.42271ptdy\hskip
1.42271ptdw_{\mathcal{D}}}_{(\star\star)}.\end{split}$ (24)

We separate the derivations of the two terms in (24). First, we develop the
term $(\star\star)$ further:

$\begin{split}(\star\star)=&-\iiint p(x,y)p(w_{\mathcal{D}})\log
p(y|x,w_{\mathcal{D}})\hskip 1.42271ptdx\hskip 1.42271ptdy\hskip
1.42271ptdw_{\mathcal{D}}\\\ =&-\iiint
p(x,y)p(w_{\mathcal{D}})\log\left\\{\frac{p(x,y,w_{\mathcal{D}})}{p(x,w_{\mathcal{D}})}\hskip
1.42271pt\right\\}dx\hskip 1.42271ptdy\hskip
1.42271ptdw_{\mathcal{D}}.\end{split}$ (25)

Using logarithm arithmetic and adding and subtracting terms we get:

$\begin{split}(\star\star)=&-\iiint
p(x,y)p(w_{\mathcal{D}})\log\left\\{\frac{p(x,y,w_{\mathcal{D}})}{p(x,y)p(w_{\mathcal{D}})}\right\\}dx\hskip
1.42271ptdy\hskip 1.42271ptdw_{\mathcal{D}}\\\ &-\iiint
p(x,y)p(w_{\mathcal{D}})\log\left\\{p(x,y)p(w_{\mathcal{D}})\right\\}dx\hskip
1.42271ptdy\hskip 1.42271ptdw_{\mathcal{D}}\\\ &-\iiint
p(x,y)p(w_{\mathcal{D}})\log\left\\{\frac{p(x)p(w_{\mathcal{D}})}{p(x,w_{\mathcal{D}})}\right\\}dx\hskip
1.42271ptdy\hskip 1.42271ptdw_{\mathcal{D}}\\\ &+\iiint
p(x,y)p(w_{\mathcal{D}})\log\left\\{p(x)p(w_{\mathcal{D}})\right\\}dx\hskip
1.42271ptdy\hskip 1.42271ptdw_{\mathcal{D}}.\end{split}$ (26)

Using the law of total probability along with the definitions of the
differential entropy and the Lautum information we can reformulate (26) as
follows:

$\begin{split}(\star\star)&=L(w_{\mathcal{D}};(x,y))\\\
&+H(w_{\mathcal{D}})+H(x,y)\\\ &-L(w_{\mathcal{D}};x)\\\
&-H(x)-H(w_{\mathcal{D}}).\end{split}$ (27)

Since $H(y|x)=H(x,y)-H(x)$ we get that:

$(\star\star)=L(w_{\mathcal{D}};(x,y))+H(y|x)-L(w_{\mathcal{D}};x).$ (28)

We next analyze the expression of $(\star)$:

$\begin{split}(\star)=&-\iiint
p(x,y)p(w_{\mathcal{D}})\log\left\\{\frac{f(y|x,w_{\mathcal{D}})}{p(y|x,w_{\mathcal{D}})}\right\\}\hskip
1.42271ptdx\hskip 1.42271ptdy\hskip 1.42271ptdw_{\mathcal{D}}.\end{split}$
(29)

Within the $\log$ operation we multiply and divide by the term
$\frac{p(x,y)p(w_{\mathcal{D}})}{p(x,w_{\mathcal{D}})}$ and get:

$\begin{split}&(\star)=\iiint p(x,y)p(w_{\mathcal{D}})\cdot\\\
&\log\left\\{\frac{p(x,y)p(w_{\mathcal{D}})}{f(y|x,w_{\mathcal{D}})p(x,w_{\mathcal{D}})}\cdot\frac{p(y|x,w_{\mathcal{D}})p(x,w_{\mathcal{D}})}{p(x,y)p(w_{\mathcal{D}})}\right\\}dx\hskip
1.42271ptdy\hskip 1.42271ptdw_{\mathcal{D}}.\end{split}$ (30)

Since $f(y|x,w_{\mathcal{D}})$ is the learned classifier which outputs the
probability of the label $y$ for an input $x$ given the model weights
$w_{\mathcal{D}}$, without the labels it has no affect on the joint
distribution of the weights and inputs, i.e.
$f(x,w_{\mathcal{D}})=p(x,w_{\mathcal{D}})$,
$f(w_{\mathcal{D}})=p(w_{\mathcal{D}})$. Accordingly,

$\begin{split}(\star)=&\iiint
p(x,y)p(w_{\mathcal{D}})\log\left\\{\frac{p(x,y)p(w_{\mathcal{D}})}{f(x,y,w_{\mathcal{D}})}\right\\}dx\hskip
1.42271ptdy\hskip 1.42271ptdw_{\mathcal{D}}\\\ -&\iiint
p(x,y)p(w_{\mathcal{D}})\log\left\\{\frac{p(x,y)p(w_{\mathcal{D}})}{p(x,y,w_{\mathcal{D}})}\right\\}dx\hskip
1.42271ptdy\hskip 1.42271ptdw_{\mathcal{D}}.\end{split}$ (31)

Since
$f(x,y|w_{\mathcal{D}})=\frac{f(x,y,w_{\mathcal{D}})}{p(w_{\mathcal{D}})}$ we
get that:

$\begin{split}(\star)=&\iiint
p(x,y)p(w_{\mathcal{D}})\log\left\\{\frac{p(x,y)}{f(x,y|w_{\mathcal{D}})}\right\\}dx\hskip
1.42271ptdy\hskip 1.42271ptdw_{\mathcal{D}}\\\ -&\iiint
p(x,y)p(w_{\mathcal{D}})\log\left\\{\frac{p(x,y)p(w_{\mathcal{D}})}{p(x,y,w_{\mathcal{D}})}\right\\}dx\hskip
1.42271ptdy\hskip 1.42271ptdw_{\mathcal{D}}.\end{split}$ (32)

In the first term we have the expectation over $w_{\mathcal{D}}$ and so:

$\begin{split}(\star)=&\mathbb{E}_{w_{\mathcal{D}}}\left\\{\iint
p(x,y)\log\left\\{\frac{p(x,y)}{f(x,y|w_{\mathcal{D}})}\right\\}dx\hskip
1.42271ptdy\right\\}\\\ -&\iiint
p(x,y)p(w_{\mathcal{D}})\log\left\\{\frac{p(x,y)p(w_{\mathcal{D}})}{p(x,y,w_{\mathcal{D}})}\right\\}dx\hskip
1.42271ptdy\hskip 1.42271ptdw_{\mathcal{D}}.\end{split}$ (33)

We get that the first term is the expectation over $w_{\mathcal{D}}$ of the
KL-divergence between $p(x,y)$ and $f(x,y|w_{\mathcal{D}})$, whereas the
second term is the negative Lautum information between $(x,y)$ and
$w_{\mathcal{D}}$:

$\begin{split}(\star)=\mathbb{E}_{w_{\mathcal{D}}}\left\\{KL(p(x,y)||f(x,y|w_{\mathcal{D}}))\right\\}-L(w_{\mathcal{D}};(x,y)).\end{split}$
(34)

Plugging the expressions we got for $(\star\star)$ from (28) and for $(\star)$
from (34) into (24) we obtain the expression in (20):

$\begin{split}&(\star)\hskip 1.42271pt+\hskip
1.42271pt(\star\star)=\underbrace{\mathbb{E}_{w_{\mathcal{D}}}\left\\{KL(p(x,y)||f(x,y|w_{\mathcal{D}}))\right\\}-L(w_{\mathcal{D}};(x,y))}_{(\star)}\\\
&+\underbrace{L(w_{\mathcal{D}};(x,y))+H(y|x)-L(w_{\mathcal{D}};x)}_{(\star\star)}\\\
&=\mathbb{E}_{w_{\mathcal{D}}}\left\\{KL(p(x,y)||f(x,y|w_{\mathcal{D}}))\right\\}+H(y|x)-L(w_{\mathcal{D}};x).\end{split}$
(35)

##  Appendix B Proof of Theorem 2

Let us reiterate Theorem 2 before formally proving it.

Theorem 2 _For a classification task with a ground-truth distribution
$p(y|x)$, fixed training set $\mathcal{D}$, learned weights $w_{\mathcal{D}}$
and learned classification function $f(y|x,w_{\mathcal{D}})$, the expected
cross-entropy loss of a machine learning algorithm on the test distribution is
equal to_

$\displaystyle\begin{split}&\mathbb{E}_{x,w_{\mathcal{D}}}\left\\{KL(p(y|x,w_{\mathcal{D}})||f(y|x,w_{\mathcal{D}}))\right\\}+\\\
&H(x,y|w_{\mathcal{D}})-H(y)+I(x;w_{\mathcal{D}}).\end{split}$ (36)

Proof. Recall that the cross-entropy can be written as:

$H(p,q)=H(p)+KL(p,q)$ (37)

In our notation:

$H(y\lvert
x,w_{\mathcal{D}})+\mathbb{E}_{x,w_{\mathcal{D}}}\left\\{KL(p(y\lvert
x,w_{\mathcal{D}})||f(y\lvert x,w_{\mathcal{D}}))\right\\}.$ (38)

Using the entropy chain rule we can write the left term as:

$H(y\lvert
x,w_{\mathcal{D}})=H(y,x,w_{\mathcal{D}})-H(y)-H(w_{\mathcal{D}}\lvert x).$
(39)

From the following identity of MI, we can rewrite the relative entropy term:

$\displaystyle I(x;w_{\mathcal{D}})$
$\displaystyle=H(w_{\mathcal{D}})-H(w_{\mathcal{D}}\lvert x)\Rightarrow$ (40)
$\displaystyle H(w_{\mathcal{D}}\lvert x)$
$\displaystyle=H(w_{\mathcal{D}})-I(x;w_{\mathcal{D}})$

In addition, the joint Entropy satisfies

$H(y,x,w_{\mathcal{D}})=H(w_{\mathcal{D}})+H(y,x\lvert w_{\mathcal{D}}).$ (41)

Therefore, we have that

$H(y\lvert x,w_{\mathcal{D}})=H(w_{\mathcal{D}})+H(y,x\lvert
w_{\mathcal{D}})-H(y)+I(x;w_{\mathcal{D}})-H(w_{\mathcal{D}}),$ (42)

which leads to our final cross-entropy decomposition:

$\displaystyle\begin{split}&\mathbb{E}_{x,w_{\mathcal{D}}}\left\\{KL(p(y|x,w_{\mathcal{D}})||f(y|x,w_{\mathcal{D}}))\right\\}+\\\
&H(x,y|w_{\mathcal{D}})-H(y)+I(x;w_{\mathcal{D}}).\end{split}$ (43)

Figure 2: The network architecture used in our experiments.

##  Appendix C The CNN architecture used in the experiments

Both in the MNIST $\rightarrow$ notMNIST case and the CIFAR-10 $\rightarrow$
CIFAR-100 (10 classes) case we used the same CNN as in [7]. The architecture
is illustrated in Figure 2.