# On Data-Augmentation and Consistency-Based Semi-Supervised Learning

Atin Ghosh & Alexandre H. Thiery
Department of Statistics and Applied Probability
National University of Singapore
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

Recently proposed consistency-based Semi-Supervised Learning (SSL) methods
such as the $\Pi$-model, temporal ensembling, the mean teacher, or the virtual
adversarial training, have advanced the state of the art in several SSL tasks.
These methods can typically reach performances that are comparable to their
fully supervised counterparts while using only a fraction of labelled
examples. Despite these methodological advances, the understanding of these
methods is still relatively limited. In this text, we analyse (variations of)
the $\Pi$-model in settings where analytically tractable results can be
obtained. We establish links with Manifold Tangent Classifiers and demonstrate
that the quality of the perturbations is key to obtaining reasonable SSL
performances. Importantly, we propose a simple extension of the Hidden
Manifold Model that naturally incorporates data-augmentation schemes and
offers a framework for understanding and experimenting with SSL methods.

##  1 Introduction

Consider a dataset $\mathcal{D}=\mathcal{D}_{L}\cup\mathcal{D}_{U}$ that is
comprised of labelled samples
$\mathcal{D}_{L}=\\{x_{i},y_{i}\\}_{i\in\mathbf{I}_{L}}$ as well as unlabelled
samples $\mathcal{D}_{U}=\\{x_{i}\\}_{i\in\mathbf{I}_{U}}$. Semi-Supervised
Learning (SSL) is concerned with the use of both the labelled and unlabeled
data for training. In many scenarios, collecting labelled data is difficult or
time consuming or expensive so that the amount of labelled data can be
relatively small when compared to the amount of unlabelled data. The main
challenge of SSL is in the design of methods that can exploit the information
contained in the distribution of the unlabelled data (Zhu, 2005; Chapelle et
al., 2009).

In modern high-dimensional settings that are common to computer vision, signal
processing, Natural Language Processing (NLP) or genomics, standard
graph/distance based methods (Blum & Chawla, 2001; Zhu & Ghahramani, 2002; Zhu
et al., 2003; Belkin et al., 2006; Dunlop et al., 2019) that are successful in
low-dimensional scenarios are difficult to implement. Indeed, in high-
dimensional spaces, it is often difficult to design sensible notions of
distances that can be exploited within these methods. We refer the interested
reader to the book-length treatments (Zhu, 2005; Chapelle et al., 2009) for
discussion of other approaches.

The manifold assumption is the fundamental structural property that is
exploited in most modern approaches to SSL: high-dimensional data samples lie
in a small neighbourhood of a low-dimensional manifold (Turk & Pentland, 1991;
Basri & Jacobs, 2003; Peyré, 2009; Cayton, 2005; Rifai et al., 2011a). In
computer vision, the presence of this low-dimensional structure is
instrumental to the success of (variational) autoencoder and generative
adversarial networks: large datasets of images can often be parametrized by a
relatively small number of degrees of freedom. Exploiting the unlabelled data
to uncover this low-dimensional structure is crucial to the design of
efficient SSL methods. A recent and independent evaluation of several modern
methods for SSL can be found in (Oliver et al., 2018). It is found there that
consistency-based methods (Bachman et al., 2014; Sajjadi et al., 2016; Laine &
Aila, 2016; Tarvainen & Valpola, 2017; Miyato et al., 2018; Luo et al., 2018;
Grill et al., 2020), the topic of this paper, achieve state-of-the art
performances in many realistic scenarios.

#### Contributions:

consistency-based semi-supervised learning methods have recently been shown to
achieve state-of-the-art results. Despite these methodological advances, the
understanding of these methods is still relatively limited when compared to
the fully-supervised setting (Saxe et al., 2013; Advani & Saxe, 2017; Saxe et
al., 2018; Tishby & Zaslavsky, 2015; Shwartz-Ziv & Tishby, 2017). In this
article, we do not propose a new SSL method. Instead, we analyse consistency-
based methods in settings where analytically tractable results can be
obtained, when the data-samples lie in the neighbourhood of well-defined and
tractable low-dimensional manifolds, and simple and controlled experiments can
be carried out. We establish links with Manifold Tangent Classifiers and
demonstrate that consistency-based SSL methods are in general more powerful
since they can better exploit the local geometry of the data-manifold if
efficient data-augmentation/perturbation schemes are used. Furthermore, in
section 4.1 we show that the popular Mean Teacher method and the conceptually
more simple $\Pi$-model approach share the same solutions in the regime when
the data-augmentations are small; this confirms often reported claim that the
data-augmentation schemes leveraged by the recent SSL, as well as fully
unsupervised algorithms, are instrumental to their success. Finally, in
section 4.3 we propose an extension of the Hidden Manifold Model (Goldt et
al., 2019; Gerace et al., 2020). This generative model allows us to
investigate the properties of consistency-based SSL methods, taking into
account the data-augmentation process and the underlying low-dimensionality of
the data, in a simple and principled manner, and without relying on a specific
dataset. For gaining understanding of SSL, as well as self-supervised learning
methods, we believe it to be important to develop a framework that (i) can
take into account the geometry of the data (ii) allows the study of the
influence of the quality of the data-augmentation schemes (iii) does not rely
on any particular dataset. While the understanding of fully-supervised methods
have largely been driven by the analysis of simplified model architectures
(eg. linear and two-layered models, large dimension asymptotic such as the
Neural Tangent Kernel), these analytical tools alone are unlikely to be enough
to explain the mechanisms responsible for the success of SSL and self-
supervised learning methods Chen et al. (2020); Grill et al. (2020), since
they do not, and cannot easily be extended to, account for the geometry of the
data and data-augmentation schemes. Our proposed framework offers a small step
in that direction.

##  2 Consistency-Based Semi-Supervised Learning

For concreteness and clarity of exposition, we focus the discussion on
classification problems. The arguments described in the remaining of this
article can be adapted without any difficulty to other situations such as
regression or image segmentation. Assume that the samples
$x_{i}\in\mathcal{X}\subset\mathbb{R}^{D}$ can be represented as
$D$-dimensional vectors and that the labels belong to $C\geq 2$ possible
classes, $y_{i}\in\mathcal{Y}\equiv\\{1,\ldots,C\\}$. Consider a mapping
$\mathcal{F}_{\theta}:\mathbb{R}^{D}\to\mathbb{R}^{C}$ parametrized by
$\theta\in\Theta\subset\mathbb{R}^{|\Theta|}$. This can be a neural network,
although that is not necessary. For $x\in\mathcal{X}$, the quantity
$\mathcal{F}_{\theta}(x)$ can represent probabilistic output of the
classifier, or , for example, the pre-softmax activations. Empirical risk
minimization consists in minimizing the function

$\displaystyle\mathcal{L}_{L}(\theta)=\frac{1}{|\mathcal{D}_{L}|}\sum_{i\in\mathbf{I}_{L}}\,\ell{\left(\mathcal{F}_{\theta}(x_{i}),y_{i}\right)}$

for a loss function $\ell:\mathbb{R}^{C}\times\mathcal{Y}\mapsto\mathbb{R}$.
Maximum likelihood estimation corresponds to choosing the loss function as the
cross entropy. The optimal parameter $\theta\in\Theta$ is found by a variant
of stochastic gradient descent (Robbins & Monro, 1951) with estimated gradient

$\displaystyle\nabla_{\theta}\;{\left\\{\frac{1}{|\mathcal{B}_{L}|}\sum_{i\in\mathcal{B}_{L}}\,\ell{\left(\mathcal{F}_{\theta}(x_{i}),y_{i}\right)}\right\\}}$

for a mini-batch $\mathcal{B}_{L}$ of labelled samples. Consistency-based SSL
algorithms regularize the learning by enforcing that the learned function
$x\mapsto\mathcal{F}_{\theta}(x)$ respects local derivative and invariance
constraints. For simplicity, assume that the mapping
$x\mapsto\mathcal{F}_{\theta}(x)$ is deterministic, although the use of drop-
out (Srivastava et al., 2014) and other sources of stochasticity are popular
in practice. The $\Pi$-model (Laine & Aila, 2016; Sajjadi et al., 2016) makes
use of a stochastic mapping
$\mathscr{S}:\mathcal{X}\times\Omega\to\mathcal{X}$ that maps a sample
$x\in\mathcal{X}$ and a source of randomness
$\omega\in\Omega\subset\mathbb{R}^{d_{\Omega}}$ to another sample
$\mathscr{S}_{\omega}(x)\in\mathcal{X}$. The mapping $\mathscr{S}$ describes a
stochastic data augmentation process. In computer vision, popular data-
augmentation schemes include random translations, rotations, dilatations,
croppings, flippings, elastic deformations, color jittering, addition of
speckle noise, and many more domain-specific variants. In NLP, synonym
replacements, insertions and deletions, back-translations are often used
although it is often more difficult to implement these data-augmentation
strategies. In a purely supervised setting, data-augmentation can be used as a
regularizer. Instead of directly minimizing $\mathcal{L}_{L}$, one can
minimize instead

$\displaystyle\theta\mapsto\frac{1}{|\mathcal{D}_{L}|}\sum_{i\in\mathbf{I}_{L}}\,\mathbb{E}_{\omega}{\left[\ell{\left(\mathcal{F}_{\theta}[\mathscr{S}_{\omega}(x_{i})],y_{i}\right)}\right]}.$

In practice, data-augmentation regularization, although a simple strategy, is
often crucial to obtaining good generalization properties (Perez & Wang, 2017;
Cubuk et al., 2018; Lemley et al., 2017; Park et al., 2019). The idea of
regularizing by enforcing robustness to the injection of noise can be traced
back at least to (Bishop, 1995). In the $\Pi$-model, the data-augmentation
mapping $\mathscr{S}$ is used to define a consistency regularization term,

$\displaystyle\mathcal{R}(\theta)=\frac{1}{|\mathcal{D}|}\sum_{i\in\mathbf{I}_{L}\cup\mathbf{I}_{U}}\mathbb{E}_{\omega}{\left\\{\big{\|}\mathcal{F}_{\theta}[\mathscr{S}_{\omega}(x_{i})]-\mathcal{F}_{\theta_{\star}}(x_{i})\big{\|}^{2}\right\\}}.$
(1)

The notation $\theta_{\star}$ designates a copy of the parameter $\theta$,
i.e. $\theta_{\star}=\theta$, and emphasizes that when differentiating the
consistency regularization term $\theta\mapsto\mathcal{R}(\theta)$, one does
not differentiate through $\theta_{\star}$. In practice, a stochastic estimate
of $\nabla\mathcal{R}(\theta)$ is obtained as follows. For a mini-batch
$\mathcal{B}$ of samples $\\{x_{i}\\}_{i\in\mathcal{B}}$, the current value
$\theta_{\star}\in\Theta$ of the parameter and the current predictions
$f_{i}\equiv\mathcal{F}_{\theta_{\star}}(x_{i})$, the quantity

$\displaystyle\nabla{\left\\{\frac{1}{|\mathcal{B}|}\sum_{i\in\mathcal{B}}\big{\|}\mathcal{F}_{\theta}[\mathscr{S}_{\omega}(x_{i})]-f_{i}\big{\|}^{2}\right\\}}$

is an approximation of $\nabla\mathcal{R}(\theta)$. There are indeed many
variants (eg. use of different norms, different manners to inject noise), but
the general idea is to force the learned function
$x\mapsto\mathcal{F}_{\theta}(x)$ to be locally invariant to the data-
augmentation scheme $\mathscr{S}$. Several extensions such as the Mean Teacher
(Tarvainen & Valpola, 2017) and the VAT (Miyato et al., 2018) schemes have
been recently proposed and have been shown to lead to good results in many SSL
tasks. The recently proposed and state-of-the-art BYOL approach Grill et al.
(2020) is relying on mechanisms that are very close to the consistency
regularization methods discussed on this text.

If one recalls the manifold assumption, this approach is natural: since the
samples corresponding to different classes lie on separate manifolds, the
function $\mathcal{F}_{\theta}:\mathcal{X}\to\mathbb{R}^{C}$ should be
constant on each one of these manifolds. Since the correct value of
$\mathcal{F}_{\theta}$ is typically well approximated or known for labelled
samples $(x_{i},y_{i})\in\mathcal{D}_{L}$, the consistency regularization term
equation 1 helps propagating these known values across these manifolds. This
mechanism is indeed similar to standard SSL graph-based approaches such as
label propagation (Zhu & Ghahramani, 2002). Graph-based methods are difficult
to directly implement in computer vision, or NLP, when a meaningful notion of
distance is not available. This interpretation reveals that it is crucial to
include the labelled samples in the regularization term equation 1 in order to
help propagating the information contained in the labelled samples to the
unlabelled samples. Our numerical experiments suggest that, in the standard
setting when the number of labelled samples is much lower than the number of
unlabeled samples, i.e. $|\mathcal{D}_{L}|\ll|\mathcal{D}_{U}|$, the
formulation equation 1 of the consistency regularization leads to sub-optimal
results and convergence issues: the information contained in the labelled data
is swamped by the number of unlabelled samples. In all our experiments, we
have adopted instead the following regularization term

$\displaystyle\mathcal{R}(\theta)$
$\displaystyle=\frac{1}{|\mathcal{D}_{L}|}\sum_{i\in\mathbf{I}_{L}}\mathbb{E}_{\omega}{\left\\{\big{\|}\mathcal{F}_{\theta}[\mathscr{S}_{\omega}(x_{i})]-\mathcal{F}_{\theta_{\star}}(x_{i})\big{\|}^{2}\right\\}}$
(2)
$\displaystyle+\frac{1}{|\mathcal{D}_{U}|}\sum_{j\in\mathbf{I}_{U}}\mathbb{E}_{\omega}{\left\\{\big{\|}\mathcal{F}_{\theta}[\mathscr{S}_{\omega}(x_{j})]-\mathcal{F}_{\theta_{\star}}(x_{j})\big{\|}^{2}\right\\}}$

that balances the labelled and unlabelled data samples more efficiently.
Furthermore, it is clear that the quality and variety of the data-augmentation
scheme $\mathscr{S}:\mathcal{X}\times\Omega\to\mathcal{X}$ is pivotal to the
success of consistency-based SSL methods. We argue in this article that it is
the dominant factor contributing to the success of this class of methods.
Effort spent on building efficient local data-augmentation schemes will be
rewarded in terms of generalization performances. Designing good data-
augmentation schemes is an efficient manner of injecting expert/prior
knowledge into the learning process. It is done by leveraging the
understanding of the local geometry of the data manifold. As usual and not
surprisingly (Niyogi et al., 1998; Montavon et al., 2012), in data-scarce
settings, any type of domain-knowledge needs to be exploited and we argue that
consistency regularization approaches to SSL are instances of this general
principle.

##  3 Approximate Manifold Tangent Classifier

It has long been known (Simard et al., 1998) that exploiting the knowledge of
derivatives, or more generally enforcing local invariance properties, can
greatly enhance the performance of standard classifiers/regressors (Haasdonk &
Keysers, 2002; Chapelle & Schölkopf, 2002). In the context of deep-learning,
the Manifold Tangent Classifier (Rifai et al., 2011a) is yet another
illustration of this idea. Consider the data manifold
$\mathcal{M}\subset\mathcal{X}\subset\mathbb{R}^{D}$ and assume that the data
samples lie on a neighbourhood of it. For $x\in\mathcal{M}$, consider as well
the tangent plane $T_{x}$ to $\mathcal{M}$ at $x$. Assuming that the manifold
$\mathcal{M}$ is of dimension $1\leq d\leq D$, the tangent plane $T_{x}$ is
also of dimension $d$ with an orthonormal basis
$\mathbf{e}_{1}^{x},\ldots,\mathbf{e}_{d}^{x}\in\mathbb{R}^{D}$. This
informally means that, for suitably small coefficients
$\omega_{1},\ldots,\omega_{d}\in\mathbb{R}$, the transformed sample
$\overline{x}\in\mathcal{X}$ defined as

$\displaystyle\overline{x}\;=\;x+\sum_{j=1}^{d}\omega_{j}\,\mathbf{e}^{x}_{j}$

also lies, or is very close to, the data manifold $\mathcal{M}$. A possible
stochastic data-augmentation scheme can therefore be defined as
$\mathscr{S}_{\omega}(x)=x+V_{\omega}$ where
$V_{\omega}=\sum_{j=1}^{d}\omega_{j}\,\mathbf{e}^{x}_{j}$. If $\omega$ is a
multivariate $d$-dimensional centred Gaussian random vector with suitably
small covariance matrix, the perturbation vector $V_{\omega}$ is also centred
and normally distributed. To enforce that the function
$x\to\mathcal{F}_{\theta}(x)$ is locally approximately constant along the
manifold $\mathcal{M}$, one can thus penalize the derivatives of
$\mathcal{F}_{\theta}$ at $x$ in the directions $V_{\omega}$. Denoting by
$\mathbf{J}_{x}\in\mathbb{R}^{C,D}$ the Jacobian with respect to
$x\in\mathbb{R}^{D}$ of $\mathcal{F}_{\theta}$ at $x\in\mathcal{M}$, this can
be implemented by adding a penalization term of the type
$\mathbb{E}_{\omega}[\|\mathbf{J}_{x}\,V_{\omega}\|^{2}]=\mathbf{Tr}{\left(\Gamma\otimes\mathbf{J}^{T}_{x}\mathbf{J}_{x}\right)}$,
where $\Gamma\in\mathbb{R}^{D,D}$ is the covariance matrix of the random
vector $\omega\to V_{\omega}$. This type of regularization of the Jacobian
along the data-manifold is for example used in (Belkin et al., 2006). More
generally, if one assumes that for any $x,\omega\in\mathcal{X}\times\Omega$ we
have
$\mathscr{S}_{\varepsilon\,\omega}(x)=x+\varepsilon\,\mathbf{D}(x,\omega)+\mathcal{O}(\varepsilon^{2})$,
for some derivative mapping
$\mathbf{D}:\mathcal{X}\times\Omega\to\mathcal{X}$, it follows that

$\displaystyle\lim_{\varepsilon\to
0}\;\frac{1}{\varepsilon^{2}}\,\mathbb{E}_{\omega}{\left[\|\mathcal{F}_{\theta}[\mathscr{S}_{\varepsilon\,\omega}(x)]-\mathcal{F}_{\theta}(x)\|^{2}\right]}\;=\;\mathbb{E}_{\omega}{\left[\|\mathbf{J}_{x}\,\mathbf{D}(x,\omega)\|^{2}\right]}\;=\;\mathbf{Tr}{\left(\Gamma_{x,\mathscr{S}}\otimes\mathbf{J}_{x}^{T}\,\mathbf{J}_{x}\right)}$

where $\Gamma_{x,\mathscr{S}}$ is the covariance matrix of the
$\mathcal{X}$-valued random vector
$\omega\mapsto\mathbf{D}(x,\omega)\in\mathcal{X}$. This shows that
consistency-based methods can be understood as approximated Jacobian
regularization methods, as proposed in (Simard et al., 1998; Rifai et al.,
2011a).

###  3.1 Limitations

In practice, even if many local dimension reduction techniques have been
proposed, it is still relatively difficult to obtain a good parametrization of
the data manifold. The Manifold Tangent Classifier (MTC) (Rifai et al., 2011a)
implements this idea by first extracting in an unsupervised manner a good
representation of the dataset $\mathcal{D}$ by using a Contractive-Auto-
Encoder (CAE) (Rifai et al., 2011b). This CAE can subsequently be leveraged to
obtain an approximate basis of each tangent plane $T_{x_{i}}$ for
$x_{i}\in\mathcal{D}$, which can then be used for penalizing the Jacobian of
the mapping $x\mapsto\mathcal{F}_{\theta}(x)$ in the direction of the tangent
plane to $\mathcal{M}$ at $x$.

Figure 1: Left: Jacobian (i.e. first order) Penalization method are short-
sighted and do not exploit fully the data-manifold Right: Data-Augmentation
respecting the geometry of the data-manifold.

The above discussion shows that the somewhat simplistic approach consisting in
adding an isotropic Gaussian noise to the data samples is unlikely to deliver
satisfying results. It is equivalent to penalizing the Frobenius norm
$\|\mathbf{J}_{x}\|_{\text{F}}^{2}$ of the Jacobian of the mapping
$x\mapsto\mathcal{F}_{\theta}(x)$; in a linear model, that is equivalent to
the standard ridge regularization. This mechanism does not take at all into
account the local-geometry of the data-manifold. Nevertheless, in medical
imaging applications where scans are often contaminated by speckle noise, this
class of approaches which can be thought off as adding artificial speckle
noise, can help mitigate over-fitting (Devalla et al., 2018).

There are many situations where, because of data scarcity or the sheer
difficulty of unsupervised representation learning in general, domain-specific
data-augmentation schemes lead to much better regularization than Jacobian
penalization. Furthermore, as schematically illustrated in Figure 1, Jacobian
penalization techniques are not efficient at learning highly non-linear
manifolds that are common, for example, in computer vision. For example, in
“pixel space", a simple image translation is a highly non-linear
transformation only well approximated by a first order approximation for very
small translations. In other words, if $x\in\mathcal{X}$ represents an image
and $g(x,v)$ is its translated version by a vector $v$, the approximation
$g(x,v)\approx x+\nabla_{v}g(x)$, with
$\nabla_{v}g(x)\equiv\lim_{\varepsilon\to
0}\,(g(x,\varepsilon\,v)-g(x)/\varepsilon$, becomes poor as soon as the
translation vector $v$ is not extremely small.

In computer vision, translations, rotations and dilatations are often used as
sole data-augmentation schemes: this leads to a poor local exploration of the
data-manifold since this type transformations only generate a very low
dimensional exploration manifold. More precisely, the exploration manifold
emanating from a sample $x_{0}\in\mathcal{X}$, i.e.
$\\{\mathscr{S}(x_{0},\omega)\;:\;\omega\in\Omega\\}$, is very low
dimensional: its dimension is much lower than the dimension $d$ of the data-
manifold $\mathcal{M}$. Enriching the set of data-augmentation degrees of
freedom with transformations such as elastic deformation or non-linear pixel
intensity shifts is crucial to obtaining a high-dimensional local exploration
manifold that can help propagating the information on the data-manifold
efficiently (Cubuk et al., 2019a; Park et al., 2019).

##  4 Asymptotic Properties

###  4.1 Fluid Limit

Consider the standard $\Pi$-model trained with a standard Stochastic Gradient
Descent (SGD). Denote by $\theta_{t}\in\Theta$ the current value of the
parameter and $\eta>0$ the learning rate. We have

$\displaystyle\theta_{k+1}\;=\;\theta_{k}-\eta\,\nabla_{\theta}\bigg{\\{}\frac{1}{|\mathcal{B}_{L}|}\sum_{i\in\mathcal{B}_{L}}\ell{\left(\;\mathcal{F}_{\theta_{k}}(x_{i}),\,y_{i}\;\right)}$
$\displaystyle+\frac{\lambda}{|\mathcal{B}_{L}|}\sum_{j\in\mathcal{B}_{L}}\Big{\|}\mathcal{F}_{\theta_{k}}(\mathscr{S}_{\omega}[x_{j}])-f_{j}\Big{\|}^{2}$
(3)
$\displaystyle+\frac{\lambda}{|\mathcal{B}_{U}|}\sum_{k\in\mathcal{B}_{U}}\Big{\|}\mathcal{F}_{\theta_{k}}(\mathscr{S}_{\omega}[x_{k}])-f_{k}\Big{\|}^{2}\bigg{\\}}$

for a parameter $\lambda>0$ that controls the trade-off between supervised and
consistency losses, as well as subsets $\mathcal{B}_{L}$ and $\mathcal{B}_{U}$
of labelled and unlabelled data samples, and
$f_{j}\equiv\mathcal{F}_{\theta\star}(x_{j})$ for
$\theta_{\star}\equiv\theta_{k}$ as discussed in Section 2. The right-hand-
side is an unbiased estimate of
$\eta\,\nabla_{\theta}\Big{[}\mathcal{L}_{L}({\theta_{k}})+\lambda\,\mathcal{R}({\theta_{k}})\Big{]}$
with variance of order $\mathcal{O}(\eta^{2})$, where the regularization term
$\mathcal{R}({\theta_{k}})$ is described in equation 2. It follows from
standard fluid limit approximations (Ethier & Kurtz, 2009)[Section 4.8] for
Markov processes that, under mild regularity and growth assumptions and as
$\eta\to 0$, the appropriately time-rescaled trajectory
$\\{\theta_{k}\\}_{k\geq 0}$ can be approximated by the trajectory of the
Ordinary Differential Equation (ODE).

###### Proposition 4.1

Let $\mathbf{D}([0,T],\mathbb{R}^{|\Theta|})$ be the usual space of càdlàg
$\mathbb{R}^{|\Theta|}$-valued functions on a bounded time interval $[0,T]$
endowed with the standard Skorohod topology. Consider the update equation 3
with learning rate $\eta>0$ and define the continuous time process
$\overline{\theta}^{\eta}(t)=\theta_{[t/\eta]}$. The sequence of processes
$\overline{\theta}^{\eta}\in\mathbf{D}([0,T],\mathbb{R}^{|\Theta|})$ converges
weakly in $\mathbf{D}([0,T],\mathbb{R}^{|\Theta|})$ and as $\eta\to 0$ to the
solution of the ordinary differential equation

$\displaystyle\dot{\overline{\theta}}_{t}\;=\;-\nabla\Big{(}\mathcal{L}(\overline{\theta}_{t})+\lambda\,\mathcal{R}(\overline{\theta}_{t})\Big{)}.$
(4)

The article (Tarvainen & Valpola, 2017) proposes the mean teacher model, an
averaging approach related to the standard Polyak-Ruppert averaging scheme
(Polyak, 1990; Polyak & Juditsky, 1992), which modifies the consistency
regularization term equation 2 by replacing the parameter $\theta_{\star}$ by
an exponential moving average (EMA). In practical terms, this simply means
that, instead of defining $f_{j}=\mathcal{F}_{\theta_{\star}}(x_{j})$, with
$\theta_{\star}=\theta_{k}$ in equation 3, one sets
$f_{j}=\mathcal{F}_{\theta_{\text{avg},k}}(x_{j})$ where the EMA process
$\\{\theta_{\text{avg},k}\\}_{k\geq 0}$ is defined through the recursion
$\theta_{\text{avg},k}=(1-\alpha\,\eta)\,\theta_{\text{avg},k-1}+\alpha\,\eta\,\theta_{k}$
where the coefficient $\alpha>0$ controls the time-scale of the averaging
process. The use of the EMA process $\\{\theta_{\text{avg},k}\\}_{k\geq 0}$
helps smoothing out the stochasticity of the process $\theta_{k}$. Similarly
to Proposition 4.1, as $\eta\to 0$, the joint process
$(\overline{\theta}^{\eta}_{t},\overline{\theta}^{\eta}_{\text{avg},t})\equiv(\theta^{\eta}_{[t/\eta]},\theta^{\eta}_{\text{avg},[t/\eta]})$
converges as $\eta\to 0$ to the solution of the following ordinary
differential equation

$\left\\{\begin{aligned}
\dot{\overline{\theta}}_{t}&=-\nabla\Big{(}\mathcal{L}(\overline{\theta}_{t})+\lambda\,\mathcal{R}(\overline{\theta}_{t},\overline{\theta}_{\text{avg},t})\Big{)}\\\
\dot{\overline{\theta}}_{\text{avg},t}&=-\alpha\,(\overline{\theta}_{\text{avg},t}-\overline{\theta}_{t})\end{aligned}\right.$
(5)

where the notation
$\mathcal{R}(\overline{\theta}_{t},\overline{\theta}_{\text{avg},t})$
designates the same quantity as the one described in equation 2, but with an
emphasis on the dependency on the EMA process. At convergence
$(\overline{\theta}_{t},\overline{\theta}_{\text{avg},t})\to(\overline{\theta}_{\infty},\overline{\theta}_{\text{avg},\infty})$,
one must necessarily have that
$\overline{\theta}_{\infty}=\overline{\theta}_{\text{avg},\infty}$, confirming
that, in the regime of small learning rate $\eta\to 0$, the Mean Teacher
method converges, albeit often more rapidly, towards the same solution as the
more standard $\Pi$-model. This indicates that the improved performances of
the Mean Teacher approach sometimes reported in the literature are either not
statistically meaningful, or due to poorly executed comparisons, or due to
mechanisms not captured by the $\eta\to 0$ asymptotic. Indeed, several
recently proposed consistency based SSL algorithms (Berthelot et al., 2019;
Sohn et al., 2020; Xie et al., 2019) achieve state-of-the-art performance
across diverse datasets without employing any exponential averaging processes.
These results are achieved by leveraging more sophisticated data augmentation
schemes such as Rand-Augment (Cubuk et al., 2019b) , Back Translation (Artetxe
et al., 2017) or Mixup (Zhang et al., 2017).

###  4.2 Minimizers are harmonic functions

To understand better the properties of the solutions, we consider a simplified
setting further exploited in Section 4.3. Assume that
$\mathcal{F}:\mathcal{X}\equiv\mathbb{R}^{D}\to\mathbb{R}$ and
$\mathcal{Y}\equiv\mathbb{R}$ and that, for every
$y_{i}\in\mathcal{Y}\equiv\mathbb{R}$, the loss function
$f\mapsto\ell(f,y_{i})$ is uniquely minimized at $f=y_{i}$. We further assume
that the data-manifold $\mathcal{M}\subset\mathbb{R}^{D}$ can be globally
parametrized by a smooth and bijective mapping
$\Phi:\mathbb{R}^{d}\to\mathcal{M}\subset\mathbb{R}^{D}$. Similarly to the
Section 2, we consider a data-augmentation scheme that can be described as
$\mathscr{S}_{\varepsilon\omega}(x)=\Phi(z+\varepsilon\omega)$ for
$z=\Phi^{-1}(x)$ and a sample $\omega$ from a $\mathbb{R}^{d}$-valued centred
and isotropic Gaussian distribution. We consider a finite set of labelled
samples $\\{x_{i},y_{i}\\}_{i\in\mathbf{I}_{L}}$, with $x_{i}=\Phi(z_{i})$ and
$z_{i}\in\mathbb{R}^{d}$ for $i\in\mathbf{I}_{L}$. We choose to model the
large number of unlabelled data samples as a continuum distributed on the data
manifold $\mathcal{M}$ as the push-forward measure $\Phi_{\sharp}\mu(dz)$ of a
probability distribution $\mu(dz)$ whose support is $\mathbb{R}^{d}$ through
the mapping $\Phi$. This means that an empirical average of the type
$(1/|\mathcal{D}_{U}|)\,\sum_{i\in\mathbf{I}_{u}}\varphi(x_{i})$ can be
replaced by $\int\varphi[\Phi(z)]\,\mu(dz)$. We investigate the regime
$\varepsilon\to 0$ and, similarly to Section 2, the minimization of the
consistency-regularized objective

$\displaystyle\mathcal{L}_{L}(\theta)$
$\displaystyle+\frac{\lambda}{\varepsilon^{2}}\int_{\mathbb{R}^{d}}\mathbb{E}_{\omega}{\left\\{\big{\|}\mathcal{F}_{\theta}[\mathscr{S}_{\varepsilon\omega}(\Phi(z))]-\mathcal{F}_{\theta}(\Phi(z))\big{\|}^{2}\right\\}}\,\mu(dz).$
(6)

For notational convenience, set
$f_{\theta}\equiv\mathcal{F}_{\theta}\circ\Phi$. Since
$\mathscr{S}_{\varepsilon\omega}[\Phi(z)]=\Phi(z+\varepsilon\,\omega)$, as
$\varepsilon\to 0$ the quantity
$\frac{1}{\varepsilon^{2}}\mathbb{E}_{\omega}{\left\\{\big{\|}\mathcal{F}_{\theta}[\mathscr{S}_{\varepsilon\omega}(\Phi(z))]-\mathcal{F}_{\theta}(\Phi(z))\big{\|}^{2}\right\\}}$
converges to $\|\nabla_{z}f_{\theta}\|^{2}$ and the objective function
equation 6 approaches the quantity

$\text{G}(f_{\theta})\;\equiv\;\frac{1}{|\mathcal{D}_{L}|}\sum_{i\in\mathbf{I}_{L}}\ell(f_{\theta}(z_{i}),y_{i})+\lambda\int_{\mathbb{R}^{d}}\,\|\nabla_{z}f_{\theta}(z)\|^{2}\,\mu(dz).$
(7)

A minimizer $f:\mathbb{R}^{d}\to\mathbb{R}$ of the functional G that is
consistent with the labelled data, i.e. $f(z_{i})=y_{i}$ for
$i\in\mathbf{I}_{L}$, is a minimizer of the energy functional
$f\mapsto\int_{\mathbb{R}^{d}}\,\|\nabla_{z}f_{\theta}(z)\|^{2}\,\mu(dz)$
subject to the constraints $f(z_{i})=y_{i}$. It is the variational formulation
of the Poisson equation

$\left\\{\begin{aligned} \Delta f(z)&=0\qquad\textrm{for}\quad
z\in\mathbb{R}^{d}\setminus\\{z_{i}\\}_{i\in\mathbf{I}_{L}}\\\
f(z_{i})&=y_{i}\qquad\textrm{for}\quad
i\in\mathbf{I}_{L}.\end{aligned}\right.$ (8)

Note that the solution does not depend on the regularization parameter
$\lambda$ in the regime of $\varepsilon\to 0$: this indicates, as will be
discussed in Section 4.3 in detail, that the generalization properties of
consistency-based SSL methods will typically be insensitive to this parameter,
in the regime of small data-augmentation at least. Furthermore, equation 8
shows that consistency-based SSL methods are indeed based on the same
principles as more standard graph-based approaches such as Label Propagation
(Zhu & Ghahramani, 2002): solutions are gradient/Laplacian penalized
interpolating functions. In Figure 2, we consider the case where $D=d=2$ with
trivial mapping $\Phi(x)=x$. We consider labelled data situated on the right
(resp. left) boundary of the unit square and corresponding to the label $y=0$
(resp. $y=1$). For simplicity, we choose the loss function
$\ell(f,y)=\frac{1}{2}\,(f-y)^{2}$ and parametrize $\mathcal{F}_{\theta}\equiv
f_{\theta}$ with a neural network with a single hidden layer with $N=100$
neurons. As expected, the $\Pi$-model converges to the solution to the Poisson
equation 8 in the unit square with boundary condition $f(u,v)=0$ for $u=0$ and
$f(u,v)=1$ for $u=1$.

Figure 2: Labelled data samples with class $y=0$ (green triangle) and $y=+1$
(red dot) are placed on the Left/Right boundary of the unit square. Unlabelled
data samples (blue stars) are uniformly placed within the unit square. We
consider a simple regression setting with loss function
$\ell(f,y)=\frac{1}{2}\,(f-y)^{2}$. Left: Randomly initialized neural network.
Middle: labelled/unlabelled data Right: Solution of $f$ obtained by training a
standard $\Pi$-model. It is the harmonic function $f(u,v)=u$, as described by
equation 8.

###  4.3 Generative model for Semi-Supervised Learning

As has been made clear throughout this text, SSL methods crucially rely on the
dependence structure of the data. The existence and exploitation of a much
lower-dimensional manifold $\mathcal{M}$ supporting the data-samples is
instrumental to this class of methods. Furthermore, the performance of
consistency-based SSL approaches is intimately related to the data-
augmentation schemes they are based upon. Consequently, in order to understand
the mechanisms that are at play when consistency-based SSL methods are used to
uncover the structures present in real datasets, it is important to build
simplified and tractable generative models of data that (1) respect these low-
dimensional structures and (2) allow the design of efficient data-augmentation
schemes. Several articles have investigated the influence of the dependence
structures that are present in the data on the learning algorithm (Bruna &
Mallat, 2013; Mossel, 2016). Here, we follow the Hidden Manifold Model (HMM)
framework proposed in (Goldt et al., 2019; Gerace et al., 2020) where the
authors describe a model of synthetic data concentrating near low-dimensional
structures and analyze the learning curve associated to a class of two-layered
neural networks.

Figure 3: Left: For a fixed data-augmentation scheme, generalization
properties for $\lambda$ spanning two orders of magnitude. Right: Influence of
the quantity of the data-augmentation of the generalization properties.

Low-dimensional structure: Similarly to Section 4.2, assume that the
$D$-dimensional data-samples $x_{i}\in\mathcal{X}$ can be expressed as
$x_{i}=\Phi(z_{i})\in\mathbb{R}^{D}$ for a fixed smooth mapping
$\Phi:\mathbb{R}^{d}\to\mathbb{R}^{D}$. In other words, the data-manifold
$\mathcal{M}$ is $d$-dimensional and the mapping $\Phi$ can be used to
parametrize it. The mapping $\Phi$ is chosen to be a neural network with a
single hidden layer with $H$ neurons, although other choices are indeed
possible. For $z=(z^{1},\ldots,z^{d})\in\mathbb{R}^{d}$, set $\Phi(z)=A^{1\to
2}\,\varphi(A^{0\to 1}z+b^{1})$ for matrices $A^{0\to 1}\in\mathbb{R}^{H,d}$
and $A^{1\to 2}\in\mathbb{R}^{D,H}$, bias vector $b^{1}\in\mathbb{R}^{H}$ and
non-linearity $\varphi:\mathbb{R}\to\mathbb{R}$ applied element-wise. In all
our experiments, we use the ELU non-linearity. We adopt the standard
normalization $A^{0\to 1}_{i,j}=w^{(1)}_{i,j}/\sqrt{d}$ and $A^{1\to
2}_{i,j}=w^{(2)}_{i,j}/\sqrt{H}$ for weights $w^{(k)}_{i,j}$ drawn i.i.d from
a centred Gaussian distribution with unit variance; this ensures that, if the
coordinate of the input vector $z\in\mathbb{R}^{d}$ are all of order
$\mathcal{O}(1)$, so are the coordinates of $x=\Phi(z)$.

Data-augmentation: consider a data sample $x_{i}\in\mathcal{M}$ on the data-
manifold. It can also be expressed as $x_{i}=\Phi(z_{i})$. We consider the
natural data-augmentation process which consists in setting
$\mathscr{S}_{\varepsilon\omega}(x_{i})=\Phi(z_{i}+\varepsilon\omega)$ for a
sample $\omega\in\mathbb{R}^{d}$ from an isotropic Gaussian distribution with
unit covariance and $\varepsilon>0$. Crucially, the data-augmentation scheme
respect the low-dimensional structure of the data: the perturbed sample
$\mathscr{S}_{\varepsilon\omega}(x_{i})$ belongs to the data-manifold
$\mathcal{M}$ for any perturbation vector $\varepsilon\,\omega$. Note that,
for any value of $\varepsilon$, the data-augmentation preserves the low-
dimensional manifold: perturbed samples
$\mathscr{S}_{\varepsilon\omega}(x_{i})$ exactly lie on the data-manifold. The
larger $\varepsilon$, the more efficient the data-augmentation scheme; this
property is important since it allows to study the influence of the amount of
data-augmentation.

Classification: we consider a balanced binary classification problem with
$|\mathcal{D}_{L}|\geq 2$ labelled training examples
$\\{x_{i},y_{i}\\}_{i\in\mathbf{I}_{L}}$ where $x_{i}=\Phi(z_{i})$ and
$y_{i}\in\mathcal{Y}\equiv\\{-1,+1\\}$. The sample $z_{i}\in\mathbb{R}^{d}$
corresponding to the positive (resp. negative) class are assumed to have been
drawn i.i.d from a Gaussian distribution with identity covariance matrix and
mean $\mu_{+}\in\mathbb{R}^{d}$ (resp. mean $\mu_{-}\in\mathbb{R}^{d}$). The
distance $\|\mu_{+}-\mu_{-}\|$ quantifies the hardness of the classification
task.

Neural architecture and optimization: Consider fitting a two-layered neural
network $\mathcal{F}_{\theta}:\mathbb{R}^{D}\to\mathbb{R}$ by minimising the
negative log-likelihood
$\mathcal{L}_{L}(\theta)\equiv(1/|\mathcal{D}_{L}|)\sum_{i}\ell[\mathcal{F}_{\theta}(x_{i}),y_{i}]$
where $\ell(f,y)=\log(1+\exp[-y\,f])$. We assume that there are
$|\mathcal{D}_{L}|=10$ labelled data pairs
$\\{x_{i},y_{i}\\}_{i=\mathbf{I}_{L}}$, as well as $|\mathcal{D}_{U}|=1000$
unlabelled data samples, that the ambient space has dimension $D=100$ and the
data manifold $\mathcal{M}$ has dimension $d=10$. The function $\Phi$ uses
$H=30$ neurons in its hidden layer. In all our experiments, we use a standard
Stochastic Gradient Descent (SGD) method with constant learning rate and
momentum $\beta=0.9$. For minimizing the consistency-based SSL objective
$\mathcal{L}_{L}(\theta)+\lambda\,\mathcal{R}(\theta)$, with regularization
$\mathcal{R}(\theta)$ given in equation 2, we use the standard strategy
(Tarvainen & Valpola, 2017) consisting in first minimizing the un-regularized
objective alone $\mathcal{L}_{L}$ for a few epochs in order for the function
$\mathcal{F}_{\theta}$ to be learned in the neighbourhood of the few labelled
data-samples before switching on the consistency-based regularization whose
role is to propagate the information contained in the labelled samples along
the data manifold $\mathcal{M}$.

Figure 4: Learning curve test (NLL) of the $\Pi$-model with $\lambda=10$ for
different “quality" of data-augmentation. The data manifold is of dimension
$d=10$ in an ambient space of dimension $D=100$. For $x_{i}=\Phi(z_{i})$ and
$1\leq k\leq d$, the data-augmentation scheme is implemented as
$\mathscr{S}_{\varepsilon\omega[k]}(x_{i})=\Phi(z_{i}+\varepsilon\,\omega[k])$
where $\omega[k]$ is a sample from a Gaussian distribution whose last $(d-k)$
coordinates are zero. In other words, the data-augmentation scheme only
explores $k$ dimensions out of the $d$ dimensions of the data-manifold. We use
$\varepsilon=0.3$ in all the experiments. Left: Learning curves (Test NLL) for
data-augmentation dimension $k\in[5,10]$ Right: Test NLL at epoch $N=200$ (see
left plot) for data-augmentation dimension $k\in[5,10]$.

Insensitivity to $\lambda$: Figure 3 (Left) shows that this method is
relatively insensitive to the parameter $\lambda$, as long as it is within
reasonable bounds. This phenomenon can be read from equation 8 that does not
depend on $\lambda$. Much larger or smaller values (not shown in Figure 3) of
$\lambda$ do lead, unsurprisingly, to convergence and stability issues.

Amount of Data-Augmentation: As is reported in many tasks Cubuk et al. (2018);
Zoph et al. (2019); Kostrikov et al. (2020), tuning the amount data-
augmentation in deep-learning applications is often a delicate exercise that
can greatly influence the resulting performances. Figure 3 (Right) reports the
generalization properties of the method for different amount of data-
augmentation. Too low an amount of data-augmentation (i.e. $\varepsilon=0.03$)
and the final performance is equivalent to the un-regularized method. Too
large an amount of data-augmentation (i.e. $\varepsilon=1.0$) also leads to
poor generalization properties. This is because the choice of
$\varepsilon=1.0$ corresponds to augmented samples that are very different
from the distribution of the training dataset (i.e. distributional shift),
although these samples are still supported by the data-manifold.

Figure 5: Mean-Teacher (MT) learning curves (Test NLL) for different values of
the exponential smoothing parameter $\beta_{\textrm{MT}}\in(0,1)$. For
$\beta_{\text{MT}}\in\\{0.9,0.95,0.99,0.995\\}$, the final test NLL obtained
through the MT approach is identical to the test NLL obtained through the
$\Pi$-model. In all the experiments, we used $\lambda=10$ and used SGD with
momentum $\beta=0.9$.

Quality of the Data-Augmentation: to study the influence of the quality of the
data-augmentation scheme, we consider a perturbation process implemented as
$\mathscr{S}_{\varepsilon\omega[k]}(x_{i})=\Phi(z_{i}+\omega[k])$ for
$x_{i}=\Phi(z_{i})$ where the noise term $\omega[k]$ is defined as follows.
For a data-augmentation dimension parameter $1\leq k\leq d$ we have
$\omega[k]=(\xi_{1},\ldots,\xi_{k},0,\ldots,0)$ for i.i.d standard Gaussian
samples $\xi_{1},\ldots,\xi_{k}\in\mathbb{R}$. This data-augmentation scheme
only explores the first $k$ dimensions of the $d$-dimensional data-manifold:
the lower $k$, the poorer the exploration of the data-manifold. As
demonstrated on Figure 4, lower quality data-augmentation schemes (i.e. lower
values of $k\in[0,d]$) hurt the generalization performance of the $\Pi$-model.

Mean-Teacher versus $\Pi$-model: we implemented the Mean-Teacher (MT) approach
with an exponential moving average (EMA) process
$\theta_{\text{avg},k}=\beta_{\textrm{MT}}\,\theta_{\text{avg},k-1}+(1-\beta_{\textrm{MT}})\,\theta_{k}$
for the MT parameter $\theta_{\text{avg},k}$ with different scales
$\beta_{\text{MT}}\in\\{0.9,0.95,0.99,0.995\\}$, as well as a $\Pi$-model
approach, with $\lambda=10$ and $\varepsilon=0.3$. Figure 5 shows, in
accordance with Section 4.1, that the different EMA schemes lead to
generalization performances similar to a standard $\Pi$-model.

##  5 Conclusion

Consistency-based SSL methods rely on a subtle trade-off between the
exploitation of the labelled samples and the discovery of the low-dimensional
data-manifold. The results presented in this article highlight the connections
with more standard methods such as Jacobian penalization and graph-based
approaches and emphasize the crucial role of the data-augmentation scheme. The
analysis of consistency-based SSL methods is still in its infancy and our
numerical simulations suggest that the variant of the Hidden Manifold Model
described in this text is a natural framework to make progress in this
direction.

## References

  * Advani & Saxe (2017) Madhu S Advani and Andrew M Saxe.  High-dimensional dynamics of generalization error in neural networks.  _arXiv preprint arXiv:1710.03667_ , 2017.
  * Artetxe et al. (2017) Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho.  Unsupervised neural machine translation.  _arXiv preprint arXiv:1710.11041_ , 2017.
  * Bachman et al. (2014) Philip Bachman, Ouais Alsharif, and Doina Precup.  Learning with pseudo-ensembles.  In _Advances in Neural Information Processing Systems_ , pp. 3365–3373, 2014.
  * Basri & Jacobs (2003) Ronen Basri and David W Jacobs.  Lambertian reflectance and linear subspaces.  _IEEE Transactions on Pattern Analysis & Machine Intelligence_, (2):218–233, 2003.
  * Belkin et al. (2006) Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani.  Manifold regularization: A geometric framework for learning from labeled and unlabeled examples.  _Journal of machine learning research_ , 7(Nov):2399–2434, 2006.
  * Berthelot et al. (2019) David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel.  Mixmatch: A holistic approach to semi-supervised learning.  In _Advances in Neural Information Processing Systems_ , pp. 5050–5060, 2019.
  * Bishop (1995) Chris M Bishop.  Training with noise is equivalent to tikhonov regularization.  _Neural computation_ , 7(1):108–116, 1995.
  * Blum & Chawla (2001) Avrim Blum and Shuchi Chawla.  Learning from labeled and unlabeled data using graph mincuts.  In _Proceedings of the Eighteenth International Conference on Machine Learning_ , pp. 19–26. Morgan Kaufmann Publishers Inc., 2001.
  * Bruna & Mallat (2013) Joan Bruna and Stéphane Mallat.  Invariant scattering convolution networks.  _IEEE transactions on pattern analysis and machine intelligence_ , 35(8):1872–1886, 2013.
  * Cayton (2005) Lawrence Cayton.  Algorithms for manifold learning.  _Univ. of California at San Diego Tech. Rep_ , 12(1-17):1, 2005.
  * Chapelle & Schölkopf (2002) Olivier Chapelle and Bernhard Schölkopf.  Incorporating invariances in non-linear support vector machines.  In _Advances in neural information processing systems_ , pp. 609–616, 2002.
  * Chapelle et al. (2009) Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien.  Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews].  _IEEE Transactions on Neural Networks_ , 20(3):542–542, 2009.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  _arXiv preprint arXiv:2002.05709_ , 2020.
  * Cubuk et al. (2018) Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le.  Autoaugment: Learning augmentation policies from data.  _arXiv preprint arXiv:1805.09501_ , 2018.
  * Cubuk et al. (2019a) Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le.  Autoaugment: Learning augmentation strategies from data.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 113–123, 2019a.
  * Cubuk et al. (2019b) Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le.  Randaugment: Practical data augmentation with no separate search.  _arXiv preprint arXiv:1909.13719_ , 2019b.
  * Devalla et al. (2018) Sripad Krishna Devalla, Prajwal K Renukanand, Bharathwaj K Sreedhar, Giridhar Subramanian, Liang Zhang, Shamira Perera, Jean-Martial Mari, Khai Sing Chin, Tin A Tun, Nicholas G Strouthidis, et al.  Drunet: a dilated-residual U-net deep learning network to segment optic nerve head tissues in optical coherence tomography images.  _Biomedical optics express_ , 9(7):3244–3265, 2018.
  * Dunlop et al. (2019) Matthew M Dunlop, Dejan Slepčev, Andrew M Stuart, and Matthew Thorpe.  Large data and zero noise limits of graph-based semi-supervised learning algorithms.  _Applied and Computational Harmonic Analysis_ , 2019.
  * Ethier & Kurtz (2009) Stewart N Ethier and Thomas G Kurtz.  _Markov processes: characterization and convergence_ , volume 282\.  John Wiley & Sons, 2009.
  * Gerace et al. (2020) Federica Gerace, Bruno Loureiro, Florent Krzakala, Marc Mezard, and Lenka Zdeborová.  Generalisation error in learning with random features and the hidden manifold model.  _arXiv preprint arXiv:2002.09339_ , 2020.
  * Goldt et al. (2019) Sebastian Goldt, Marc Mézard, Florent Krzakala, and Lenka Zdeborová.  Modelling the influence of data structure on learning in neural networks.  _arXiv preprint arXiv:1909.11500_ , 2019.
  * Grill et al. (2020) Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent: A new approach to self-supervised learning.  _arXiv preprint arXiv:2006.07733_ , 2020.
  * Haasdonk & Keysers (2002) Bernard Haasdonk and Daniel Keysers.  Tangent distance kernels for support vector machines.  In _Object recognition supported by user interaction for service robots_ , volume 2, pp. 864–868. IEEE, 2002.
  * Kostrikov et al. (2020) Ilya Kostrikov, Denis Yarats, and Rob Fergus.  Image augmentation is all you need: Regularizing deep reinforcement learning from pixels.  _arXiv preprint arXiv:2004.13649_ , 2020.
  * Laine & Aila (2016) Samuli Laine and Timo Aila.  Temporal ensembling for semi-supervised learning.  _arXiv preprint arXiv:1610.02242_ , 2016.
  * Lemley et al. (2017) Joseph Lemley, Shabab Bazrafkan, and Peter Corcoran.  Smart augmentation learning an optimal data augmentation strategy.  _IEEE Access_ , 5:5858–5869, 2017.
  * Luo et al. (2018) Yucen Luo, Jun Zhu, Mengxi Li, Yong Ren, and Bo Zhang.  Smooth neighbors on teacher graphs for semi-supervised learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 8896–8905, 2018.
  * Miyato et al. (2018) Takeru Miyato, Shin-ichi Maeda, Shin Ishii, and Masanori Koyama.  Virtual adversarial training: a regularization method for supervised and semi-supervised learning.  _IEEE transactions on pattern analysis and machine intelligence_ , 2018.
  * Montavon et al. (2012) Grégoire Montavon, Katja Hansen, Siamac Fazli, Matthias Rupp, Franziska Biegler, Andreas Ziehe, Alexandre Tkatchenko, Anatole V Lilienfeld, and Klaus-Robert Müller.  Learning invariant representations of molecules for atomization energy prediction.  In _Advances in Neural Information Processing Systems_ , pp. 440–448, 2012.
  * Mossel (2016) Elchanan Mossel.  Deep learning and hierarchal generative models.  _arXiv preprint arXiv:1612.09057_ , 2016.
  * Niyogi et al. (1998) Partha Niyogi, Federico Girosi, and Tomaso Poggio.  Incorporating prior information in machine learning by creating virtual examples.  _Proceedings of the IEEE_ , 86(11):2196–2209, 1998.
  * Oliver et al. (2018) Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow.  Realistic evaluation of deep semi-supervised learning algorithms.  In _Advances in Neural Information Processing Systems_ , pp. 3235–3246, 2018.
  * Park et al. (2019) Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le.  Specaugment: A simple data augmentation method for automatic speech recognition.  _arXiv preprint arXiv:1904.08779_ , 2019.
  * Perez & Wang (2017) Luis Perez and Jason Wang.  The effectiveness of data augmentation in image classification using deep learning.  _arXiv preprint arXiv:1712.04621_ , 2017.
  * Peyré (2009) Gabriel Peyré.  Manifold models for signals and images.  _Computer Vision and Image Understanding_ , 113(2):249–260, 2009.
  * Polyak (1990) Boris T Polyak.  New stochastic approximation type procedures.  _Automat. i Telemekh_ , 7(98-107):2, 1990.
  * Polyak & Juditsky (1992) Boris T Polyak and Anatoli B Juditsky.  Acceleration of stochastic approximation by averaging.  _SIAM journal on control and optimization_ , 30(4):838–855, 1992.
  * Rifai et al. (2011a) Salah Rifai, Yann N Dauphin, Pascal Vincent, Yoshua Bengio, and Xavier Muller.  The manifold tangent classifier.  In _Advances in Neural Information Processing Systems_ , pp. 2294–2302, 2011a.
  * Rifai et al. (2011b) Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio.  Contractive auto-encoders: Explicit invariance during feature extraction.  In _Proceedings of the 28th International Conference on International Conference on Machine Learning_ , pp. 833–840. Omnipress, 2011b.
  * Robbins & Monro (1951) Herbert Robbins and Sutton Monro.  A stochastic approximation method.  _The annals of mathematical statistics_ , pp. 400–407, 1951.
  * Sajjadi et al. (2016) Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen.  Regularization with stochastic transformations and perturbations for deep semi-supervised learning.  In _Advances in Neural Information Processing Systems_ , pp. 1163–1171, 2016.
  * Saxe et al. (2013) Andrew M Saxe, James L McClelland, and Surya Ganguli.  Exact solutions to the nonlinear dynamics of learning in deep linear neural networks.  _arXiv preprint arXiv:1312.6120_ , 2013.
  * Saxe et al. (2018) Andrew Michael Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan Daniel Tracey, and David Daniel Cox.  On the information bottleneck theory of deep learning.  2018\.
  * Shwartz-Ziv & Tishby (2017) Ravid Shwartz-Ziv and Naftali Tishby.  Opening the black box of deep neural networks via information.  _arXiv preprint arXiv:1703.00810_ , 2017.
  * Simard et al. (1998) Patrice Y Simard, Yann A LeCun, John S Denker, and Bernard Victorri.  Transformation invariance in pattern recognition—tangent distance and tangent propagation.  In _Neural networks: tricks of the trade_ , pp. 239–274. Springer, 1998.
  * Sohn et al. (2020) Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  _arXiv preprint arXiv:2001.07685_ , 2020.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  _The Journal of Machine Learning Research_ , 15(1):1929–1958, 2014.
  * Tarvainen & Valpola (2017) Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  In _Advances in neural information processing systems_ , pp. 1195–1204, 2017.
  * Tishby & Zaslavsky (2015) Naftali Tishby and Noga Zaslavsky.  Deep learning and the information bottleneck principle.  In _2015 IEEE Information Theory Workshop (ITW)_ , pp. 1–5. IEEE, 2015.
  * Turk & Pentland (1991) Matthew Turk and Alex Pentland.  Eigenfaces for recognition.  _Journal of cognitive neuroscience_ , 3(1):71–86, 1991.
  * Xie et al. (2019) Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le.  Unsupervised data augmentation for consistency training.  2019\.
  * Zhang et al. (2017) Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  _arXiv preprint arXiv:1710.09412_ , 2017.
  * Zhu & Ghahramani (2002) Xiaojin Zhu and Zoubin Ghahramani.  Learning from labeled and unlabeled data with label propagation.  Technical report, Citeseer, 2002.
  * Zhu et al. (2003) Xiaojin Zhu, Zoubin Ghahramani, and John D Lafferty.  Semi-supervised learning using gaussian fields and harmonic functions.  In _Proceedings of the 20th International conference on Machine learning (ICML-03)_ , pp. 912–919, 2003.
  * Zhu (2005) Xiaojin Jerry Zhu.  Semi-supervised learning literature survey.  Technical report, University of Wisconsin-Madison Department of Computer Sciences, 2005.
  * Zoph et al. (2019) Barret Zoph, Ekin D Cubuk, Golnaz Ghiasi, Tsung-Yi Lin, Jonathon Shlens, and Quoc V Le.  Learning data augmentation strategies for object detection.  _arXiv preprint arXiv:1906.11172_ , 2019.