# Enhancing Mixup-based Semi-Supervised Learning with Explicit Lipschitz
Regularization

Prashnna Kumar Gyawali  Sandesh Ghimire  Linwei Wang

###### Abstract

The success of deep learning relies on the availability of large-scale
annotated data sets, the acquisition of which can be costly requiring expert
domain knowledge. Semi-supervised learning (SSL) mitigates this challenge by
exploiting the behavior of the neural function on large unlabeled data. The
smoothness of the neural function is a commonly used assumption exploited in
SSL. A successful example is the adoption of mixup strategy in SSL that
enforces the global smoothness of the neural function by encouraging it to
behave linearly when interpolating between training examples. Despite its
empirical success, however, the theoretical underpinning of how mixup
regularizes the neural function has not been fully understood. In this paper,
we offer a theoretically substantiated proposition that mixup improves the
smoothness of the neural function by bounding the Lipschitz constant of the
gradient function of the neural networks. We then propose that this can be
strengthened by simultaneously constraining the Lipschitz constant of the
neural function itself through adversarial Lipschitz regularization,
encouraging the neural function to behave linearly while also constraining the
slope of this linear function. On three benchmark data sets and one real-world
biomedical data set, we demonstrate that this combined regularization results
in improved generalization performance of SSL when learning from a small
amount of labeled data. We further demonstrate the robustness of the presented
method against single-step adversarial attacks. Our code is available at
https://github.com/Prasanna1991/Mixup-LR.

###### Index Terms:

Mixup, Smoothness, Lipschitz regularization.

##  I Introduction

Deep Learning has been an increasingly common choice of data analyses across
various domains. They have achieved strong performance when trained with a
large set of well-annotated data. However, the acquisition of such data sets
is expensive in many domains as the annotation requires expert knowledge [1].
In comparison, the collection of a large amount of data without any
annotation, i.e., unlabeled data set, is often less costly. This surplus of
unlabeled data can be exploited to benefit the learning from small labeled
data via semi-supervised learning (SSL).

Formally, in SSL, a data set $\mathcal{X}=\\{x_{1},x_{2},...,x_{n}\\}$ is
given among which only the first $m$ points are annotated
$\\{y_{1},...,y_{m}\\}\in\mathcal{Y}$, and the remaining points are unlabeled.
While learning the function $f:\mathcal{X}\rightarrow\mathcal{Y}$, SSL will
exploit the hidden relationship within the data to predict the labels of
unlabeled data points. An important assumption commonly exploited is the
smoothness of the neural function [2]. Generally, this can be loosely
categorized into two groups: local smoothness, and global smoothness [2]. As
illustrated in Fig. (1) on a classic two-moon toy problem, local assumption
regularizes outputs of nearby points to have the same label. This is
represented by various perturbation-based methods that constrain the output of
the neural function in the vicinity of available data points [3, 4], which
does not consider the connection between data points. Alternatively, global
assumption regularizes outputs of the points of the same structure (e.g., any
points on a single moon in 1), more fully utilizing the information in the
unlabeled data structures. This is represented by various graph-based methods
[2, 5], where the similarity of data points is defined by graph and outputs of
neural functions are smoothed for the graph structure.

Figure 1: The use of local and global smoothness in the classic two-moon toy
SSL problem.

More recently, the mixup regularizer [6], initially proposed for supervised
learning, has been applied to SSL and demonstrated state-of-the-art
performance [7, 8]. Mixup trains a deep learning model on linear interpolants
of inputs and labels. It has been considered as regularizing the global
smoothness of the function by filling the void between input samples and, in
specific, has been interpreted to be encouraging a linear behavior of the
underlying neural function when interpolating between training examples [6].
Despite the empirical success obtained in many variants of the mixup strategy
[9, 10], however, the theoretical underpinning of its regularization effect on
the the neural function has not been fully understood.

Outside the regime of SSL, different learning theories for generalization have
agreed that regularizing some notion of smoothness of the hypothesis class of
the function helps improve generalization [11]. One increasingly popular
approach to regularize the smoothness of the neural function, or to control
the complexity of the function, is in enforcing Lipschitz continuity of the
deep network [12, 13, 14]. This interest is particularly noticeable in the
generative modeling community to improve the stability of GANs, where
efficiently constraining the Lipschitz constant of the critic function is
fundamental due to the nature of the underlying optimization problem
(minimization of the Wasserstein distance between real and generated samples)
[13]. The Lipschitz continuity of a function (see Definition 1 for gradient
function) essentially bounds the rate of changes in the function output as a
result of the change in the input, preventing a function from changing steeply
over its input space. To enforce the Lipschitz continuity of a neural function
(or to constrain the Lipschitz constant of a neural function), several
techniques have been proposed, such as weight clipping [13], gradient penalty
[15], and adversarial Lipschitz regularization [16]. Beside generative models,
the Lipschitz continuity is also considered in several deep learning topics,
including robust learning [17], deep learning theory [18] and supervised
learning [19]. Despite these extensive efforts, the effect of Lipschitz
continuity for SSL and its relation with mixup-based regularization have
received limited attention.

Figure 2: Schematic of the mixup-based strategy for learning function (blue
line) with limited labeled data (green dots). Mixup promotes linearity between
the pair of data (black line) but does not bound the steepness of the slope.
We add Lipschitz regularization to constrain the slope of such function (green
line).

In this paper, we first offer a theoretically substantiated interpretation of
the regularization effect of mixup from the lens of Lipschitz constant. We
show that, by promoting linearity, mixup minimizes the Lipschitz constant of
the gradient function of the neural network, thereby enforcing the Lipschitz
smoothness of the neural function (Proposition 1). We then note that, while
this minimizes the rate of change of the gradient of the neural function, it
does not enforce the Lipschitz continuity of the neural function itself.
Intuitively, this means that while the mixup encourages the function to behave
linearly when interpolating, it does not bound or constrain the steepness of
the slope of this linear function as illustrated in Fig. 2. We therefore
present a new SSL strategy that combines mix-up training with Lipshitz
regularization (LR) to simultaneously constraint the Lipschitz constant of
both the neural function (through LR) and its gradient function (through
mixup). This not only encourages the neural function to behave linearly, but
also constrains the slope of this linear function as illustrated in Fig. 2. We
hypothesize that this combined regularization will further smooth the neural
function and result in improved generalization performance of SSL when
learning from a small amount of labeled data.

We test our hypothesis on three widely considered benchmark data sets
(CIFAR-10 [20], SVHN [21], and CIFAR-100 [20]) and a real world biomedical
data set (Skin Lesion images [22, 23]). We compare the performance of the
presented method with standard SSL methods, including the state-of-the-art
MixMatch [7], and demonstrate its improvement in generalization across all
these datasets. We further investigate the robustness of the presented method
to single-step adversarial attacks and demonstrate improved robustness.

In summary, the contribution of this work includes:

  * •

We establish the first connection between mixup-based regularization and the
smoothness of the neural function via Lipschitz smoothness.

  * •

We propose augmenting mixup-based approach with explicit Lipschitz
regularization on the neural function to improve the generalization of SSL
methods.

  * •

We demonstrated improved performance over the state-of-the-art methods in
three benchmark datasets and a real-world biomedical dataset.

  * •

We demonstrate improved robustness of the presented method to single-step
adversarial attacks.

##  II Related Work

Our work is generally related to two broad research topics of deep learning
which we discuss separately below.

###  II-A Semi-supervised learning

SSL has been extensively studied in machine learning. In recent times, with
deep learning, SSL has seen tremendous success. One of the standard approaches
of SSL algorithms is the application of consensus regularization. Toward this,
perturbation around input data points (or its latent space) [4, 3, 24, 1] have
been considered. For instance, in $\Pi$-Model [3, 24], consistency-based
regularization is applied on ensemble predictions obtained via techniques like
random data augmentation, and network dropout. On the other hand, virtual
adversarial training (VAT) [4] maintains similar consistency by forcing
predictions of different adversarially-perturbed inputs to be the same.
However, by considering perturbations around single data points, these
approaches regularize only the local smoothness of the network function.
Furthermore, it has been argued that such local perturbations would not fully
utilize the information in the unlabeled data structure [5].

For regularizing the global smoothness of the neural function, mostly graph-
based methods have been considered [2, 5]. These methods defined the
similarity of the data points on the graph and smoothed outputs of the neural
network for such graph structure. More recently, mixup regularizer [6] is
argued to regularize the global smoothness of the neural function and, thus,
help achieve generalization in SSL [10]. Initially, mixup was proposed for
improving generalization in supervised learning, demonstrating state-of-the-
art results in the corresponding benchmarks [6]. Later, mixup was extended to
the SSL problems, such as MixMatch [7], where a even larger margin of
improvements were obtained across many datasets. Different variants of mixup
have since been presented in the literature, such as mixing in both data and
latent space for further improving generalization in supervised learning [9]
and SSL [10]. However, despite such empirical success, the theoretical
underpinning of how mixup regularizes the neural function has not been fully
understood.

Motivated by the performance of a mixup-based strategy, in this paper, we
offer a theoretical insight of the regularization effect of mixup through the
lens of Lipschitz constant and, based on which, identify a complementary
improvement to improve mixup-based SSL.

###  II-B Lipschitz regularization

The Lipschitz constant of the network has been proposed as a candidate measure
for the Rademacher complexity (a measure of generalization) [18]. Different
Lipschitz regularizations for controlling Lipschitz constant have been
considered in various topics of machine learning. Lipschitz continuity is
commonly considered for robust learning, avoiding adversarial attacks [17],
and stabilization to train generative adversarial networks [13]. To enforce
the Lipschitz constraint, different implicit approaches like weight clipping
[13] and gradient penalty [15, 25] have been considered. These implicit
approaches approximate the constraint on Lipschitz constant by, typically,
penalizing the norm of the function gradient at certain input points [15]. An
explicit approach to Lipschitz regularization, on the other hand, attempts to
directly encourages Lipschitz continuity based on its definition, which was
argued to provides more control over the regularization effect [16]. In [16],
for instance, this was done by explicitly penalizing the violation of
Lipschitz constraint. Despite these extensive studies, however, the use of
Lipschitz regularization has not been considered for improving generalization
of SSL methods.

##  III Preliminaries

In this section, we first briefly discuss some preliminaries required for the
presented method in Section IV.

###  III-A Preliminary I: Mixup Regularization

Mixup [6] is a data-dependent regularization inspired by Vicinal Risk
Minimization (VRM) principle [26] that encourages the model $f$ to behave
linearly in-between training samples. Formally, the mixup produces virtual
feature-target vector:

$\displaystyle\tilde{x}$ $\displaystyle=\lambda x_{1}+(1-\lambda)x_{2},$
$\displaystyle\tilde{y}$ $\displaystyle=\lambda y_{1}+(1-\lambda)y_{2}$

where ($x_{1}$, $y_{1}$) and ($x_{2}$, $y_{2}$) are two feature-target vectors
drawn randomly from the training data and
$\lambda\sim\text{Beta}(\alpha,\alpha)\in[0,1]$. This mixup is used to
construct a virtual dataset $\mathcal{D}_{v}$ :=
$\\{(\tilde{x_{i}},\tilde{y_{i}})\\}^{m}_{i=1}$ which is then used to train
the network function $f$ by minimizing the loss value:

$\displaystyle\mathcal{L}_{mixup}=\frac{1}{m}\sum_{i}^{m}\ell(f(\tilde{x_{i}},\tilde{y_{i}}))$
(1)

In between the original feature-target pairs, the loss function
$\mathcal{L}(f)$ encourages the network function $f$ to behave linearly:

$\displaystyle f(\lambda x_{1}+(1-\lambda)x_{2})=\lambda
f(x_{1})+(1-\lambda)f(x_{2})$ (2)

###  III-B Preliminary II: Lipschitz Regularization

A general definition of the smallest Lipschitz constant K of a function
$f:\mathcal{X}\rightarrow\mathcal{Y}$ is:

$\displaystyle\left\lVert
f\right\rVert_{K}=\underset{x_{1},x_{2}\in\mathcal{X};x_{1}\neq
x_{2}}{\text{sup}}\frac{d_{Y}(f(x_{1}),f(x_{2}))}{d_{X}(x_{1},x_{2})}$ (3)

where the metric spaces $d_{X}$ and $d_{Y}$ are the domain and co-domain of
the function $f$, respectively. The properties of low Lipschitz constants for
deep networks are explored in [27], demonstrating that it improves
generalization. Recent literature in stabalizing GAN have proposed different
approaches for Lipschitz regularization. These regularizations can be
generally grouped into an implicit and explicit form of penalization for
violation of Lipschitz constraint. For instance, gradient penalty [15], an
implicit Lipschitz regularization, penalizes the norm of the gradient as

$\displaystyle\mathbb{E}_{x\sim\mathcal{X}}(\left\lVert\triangledown_{x}f(x)\right\rVert_{2}-1)^{2}$
(4)

and Lipschitz penalty [28, 16], an explicit regularization, penalizes the
violation of the Lipschitz constraint as

$\displaystyle\mathbb{E}_{x_{1},x_{2}\sim\mathcal{X}}(\frac{|f(x_{1})-f(x_{2})|}{\left\lVert
x_{1}-x_{2}\right\rVert_{2}}-1)^{2}$ (5)

In both case, the objective is to achieve 1-Lipschitz function for $f$, an
optimization requirement for Wassterstein-GANs.

##  IV Methodology

We consider a mapping function $f:\mathcal{X}\rightarrow\mathcal{Y}$,
approximated via a deep network. As has been empirically shown, the given
function $f$ achieves better generalization when trained with a mixup strategy
[6]. In IV-A, we first establish the theoretical connection between mixup and
Lipschitz regularization (of the gradient function of the neural network).
With this understanding, in IV-B, we propose a complementary improvement of
mixup by explicit Lipschitz regularization of the neural function via
adversarial Lipschitz regularization. Finally, in IV-C, we integrate these
ideas into a new SSL method.

###  IV-A Mixup bounds the Lipschitz constant of the gradient of the neural
function

To understand the role of mixup in regularizing the smoothness of the neural
function, we consider the definition of Lipschitz smoothness.

###### Definition 1

(Lipschitz Smoothness). A differential function
$f:\mathcal{X}\rightarrow\mathcal{Y}$ is Lipschitz smooth with constant $L>0$
if its derivatives are Lipschitz continuous:

$\displaystyle\left\lVert\bigtriangledown f(x_{1})-\bigtriangledown
f(x_{2})\right\rVert\leq L\left\lVert x_{1}-x_{2}\right\rVert,\quad\forall
x_{1},x_{2}\in\mathcal{X}.$

Loosely speaking, when the gradient of a function is Lipschitz continuous,
such function are considered to be smooth.

We now show that mixup regularization is a lower bound of the Lipschitz
constant of the gradient of the neural network.

###### Proposition 1

Let $f$ be the differential function with a Lipschitz continuous gradient over
$\mathbb{R}^{n}$ with constant $L$. This function, $f$, via mixup regularizer
[6], is encouraged toward convexity such that $f(\lambda
x_{1}+(1-\lambda)x_{2})=\lambda f(x_{1})+(1-\lambda)f(x_{2})$ with
$\lambda\in[0,1]$. Then, $f(\alpha x_{1}+(1-\alpha)x_{2})-(\alpha
f(x_{1})+(1-\alpha)f(x_{2}))\leq\frac{\alpha(1-\alpha)L}{2}\left\lVert
x_{1}-x_{2}\right\rVert^{2}$.

###### Proof:

With the definition of Lipschitz smoothness, for a differential function
$f:\mathcal{X}\rightarrow\mathbb{R}$, $\mathcal{X}\subset\mathbb{R}^{n}$, and
a constant $L$, we have,

$\displaystyle\left\lVert\bigtriangledown f(x_{1})-\bigtriangledown
f(x_{2})\right\rVert\leq L\left\lVert x_{1}-x_{2}\right\rVert,\quad\forall
x_{1},x_{2}\in\mathcal{X}.$ (6)

Note that this definition does not assume convexity of $f$. But when we assume
that the function $f$ is convex, then using Cauchy-Schwartz inequality, we
have equivalent condition as:

$\displaystyle(\bigtriangledown f(x_{1})-\bigtriangledown
f(x_{2}))^{T}(x_{1}-x_{2})\leq L\left\lVert x_{1}-x_{2}\right\rVert^{2},$
$\displaystyle\quad\forall x_{1},x_{2}\in\mathcal{X}.$ (7)

Similarly, for the convex function $f$, using monotonicity of gradient
equivalence, we have following condition:

$\displaystyle(\bigtriangledown f(x_{1})-\bigtriangledown
f(x_{2}))^{T}(x_{1}-x_{2})\geq 0,\quad\forall x_{1},x_{2}\in\mathcal{X}.$ (8)

Now, let us consider the function

$\displaystyle g(x)=\frac{L}{2}x^{T}x-f(x)$ (9)

Using (IV-A) and (8), we first establish that $g$ is convex. We apply $x_{1}$
and $x_{2}$ to $g$, take the derivative, and subtract the two result to get:

$\displaystyle\bigtriangledown g(x_{1})$ $\displaystyle-\bigtriangledown
g(x_{2})=Lx_{1}-\bigtriangledown f(x_{1})-Lx_{2}+\bigtriangledown f(x_{2})$
(10) $\displaystyle\bigtriangledown g(x_{1})$ $\displaystyle-\bigtriangledown
g(x_{2})=(Lx_{1}-Lx_{2}-(\bigtriangledown f(x_{1})-\bigtriangledown
f(x_{2})))$ $\displaystyle(\bigtriangledown g(x_{1})$
$\displaystyle-\bigtriangledown g(x_{2}))^{T}(x_{1}-x_{2})$
$\displaystyle=(L(x_{1}-x_{2})-(\bigtriangledown f(x_{1})-\bigtriangledown
f(x_{2})))^{T}(x_{1}-x_{2})$ $\displaystyle=L\left\lVert
x_{1}-x_{2}\right\rVert^{2})-(\bigtriangledown f(x_{1})-\bigtriangledown
f(x_{2}))^{T}(x_{1}-x_{2})$ $\displaystyle\geq 0$

This is equivalent to the convex form of (8), and hence $g$ is convex. Now,
when we expand $g$ according to the standard definition of convexity, we get:

$\displaystyle f(\alpha x_{1}+(1-\alpha)x_{2})-$ $\displaystyle(\alpha
f(x_{1})+(1-\alpha)f(x_{2}))$ (11)
$\displaystyle\leq\frac{\alpha(1-\alpha)L}{2}\left\lVert
x_{1}-x_{2}\right\rVert^{2}$

where the LHS of (11) is the minimization of mixup loss and we finished the
proof. ∎

Note that only the constant $L$ on the right-hand side is subject to change
during optimization for a given pair of data $x_{1}$ and $x_{2}$. As such,
this proposition implies that minimizing mixup loss controls the constant $L$
which can be considered as the Lipschitz constant of the gradient function of
the neural network, thereby making the function smoother. Establishing this
theoretically-substantiated connection between mixup and Lipschitz
regularization is the first contribution of this work.

###  IV-B Bounding the Lipschitz constant of the neural function by
adversarial Lipschitz regularization

As illustrated in Fig. 2 when regularizing the neural function to interpolate
linearly with mixup provides the smoothest possible function among possible
choices, it does not constrain the steepness of the slope of the linear
function. Therefore, in this section, we propose to augment mixup strategy
with an explicit Lipschitz regularization to bound the Lipschitz constant of
the neural function itself. In specific, we consider penalizing the violation
of Lipschitz constraint as:

$\displaystyle\mathcal{L}_{LP}=\Big{(}\frac{d_{Y}(f(x_{1}),f(x_{2}))}{d_{X}(x_{1},x_{2})}-\gamma)$
(12)

where $d_{X}$ and $d_{Y}$ are the metric for input and output space,
respectively. Note that we put $\gamma$ = 0 because, unlike the GAN setup, we
are not required to obtain 1-Lipschitz function, as shown in Eq. (5).

The direct implementation of Eq.(12) is not trivial, partly due to the
sampling strategy of the training pairs of $x_{1}$ and $x_{2}$. We follow the
adversarial Lipschitz regularization strategy presented in [16] that penalizes
Eq.(12) on a pair of data points that maximizes the Lipschitz ratio. In
specific, we first select the data point $x_{2}$ to be in the vicinity of the
training point $x_{1}$ such that $x_{2}$ = $x_{1}+r$:

$\displaystyle\left\lVert
f\right\rVert_{K}=\underset{x_{1},x_{1}+r\in\mathcal{X};x_{1}\neq
x_{2}}{\text{sup}}\frac{d_{Y}(f(x_{1}),f(x_{1}+r))}{d_{X}(x_{1},x_{1}+r)}$
(13)

where the mapping $f$ is $K$-Lipschitz if taking maximum over $r$ results in
value $K$ or smaller. Toward this, we define $r$ by finding the adversarial
perturbation that maximizes the Lipschitz ratio for the given $x_{1}$ as:

$\displaystyle r_{adv}=\underset{x_{1},x_{1}+r\in\mathcal{X};x_{1}\neq
x_{2}}{\text{arg max}}\frac{d_{Y}(f(x_{1}),f(x_{1}+r))}{d_{X}(x_{1},x_{1}+r)}$
(14)

and penalize the corresponding maximum violation of the Lipschitz constraint
as:

$\displaystyle\mathcal{L}_{ALP}=\Big{(}\frac{d_{Y}(f(x_{1}),f(x_{1}+r_{adv}))}{d_{X}(x_{1},x_{1}+r_{adv})}-\gamma)$
(15)

Since computing adversarial perturbation is a nonlinear optimization problem,
we followed a crude and cheap power iteration based approximation approach
similar to works in [4, 16]. In this iterative scheme, we approximate the
direction at $x_{1}$ that induces the largest change in the output in terms of
divergence $d_{Y}$.

Combining the mixup loss of Eq. (1) and ALR of Eq. (15), we obtain our
proposed loss function:

$\displaystyle\mathcal{L}=\mathcal{L}_{mixup}+\zeta\cdot\mathcal{L}_{ALP}$
(16)

where $\zeta$ controls the relative strength of explicit Lipschitz
regularization.

###  IV-C Integrating mixup and explicit Lipschitz regularization for SSL

In this section, we apply the presented combination of loss (Eq. 16) for SSL
setup. Toward this, we consider a data set
$\mathcal{X}=\\{x_{1},x_{2},...,x_{n}\\}$ among which only the $m$ points are
annotated with labels $\\{y_{1},...,y_{m}\\}\in\mathcal{L}$, and the remaining
points are unlabeled. We aim to learn parameters $\theta$ for the mapping
function $f:\mathcal{X}\rightarrow\mathcal{Y}$, approximated via a deep
network.

Similar to MixMatch algorithm [18], along the course of training, we first
guess and continuously update the labels for unlabeled data points. We augment
$P$ separate copies of unlabeled data batch $u_{b}$, and compute the average
of the model’s prediction as:

$q_{b}=\frac{1}{P}\sum_{p=1}^{P}f(u_{b,p};\theta)$ (17)

Note that the label guessing in this manner also regularize the model toward
consistency in a similar fashion to typical perturbation-based approaches, as
the data transformations (e.g., rotation, translation, etc.) are assumed to
leave class semantics unaffected.

While generating a guessed label, we sharpen the obtained labels to minimize
the entropy in our estimation. Entropy minimization is a traditional and
successful strategy in the SSL to enforce the classifier output to have low-
entropy predictions on unlabeled data [7, 4]. For the sharpening function, we
use the following operation:

$\text{Sharpen}(q_{b},\tau)_{i}:=q_{b_{i}}^{\frac{1}{\tau}}/\sum_{j=1}^{S}q_{b_{j}}^{\frac{1}{\tau}}$
(18)

where $S$ represents the number of classes in the output space, and $\tau$ is
the temperature hyperparameter for the categorical distribution. However, note
that such sharpening is not feasible in a multi-label classification scenario
[10].

We then use this sharpened guessed labels for unlabeled data points, and
ground truth labels for labeled data points to train the network using mixup
strategy. In each batch, we mix both labeled and unlabeled data points
together to ensure that the mixed data fairly represent the distribution of
both labeled and unlabeled data.

$\displaystyle\lambda$ $\displaystyle\sim\text{Beta}(\alpha,\alpha)$ (19)
$\displaystyle\lambda^{\prime}$ $\displaystyle=\text{max}(\lambda,1-\lambda)$
$\displaystyle\tilde{x}$
$\displaystyle=\lambda^{\prime}\mathbf{x}_{1}+(1-\lambda^{\prime})\mathbf{x}_{1}$
$\displaystyle\tilde{y}$
$\displaystyle=\lambda^{\prime}\mathbf{y}_{2}+(1-\lambda^{\prime})\mathbf{y}_{2}$

It is reasonable to expect that the actual labels in labeled data are more
reliable than guessed labels for unlabeled data, which motivates us to use
different loss functions for labeled and unlabeled data points. Since we mixed
them together, we use $\lambda^{\prime}=\text{max}(\lambda,1-\lambda)$ in
Eq.(19) to ensure $\tilde{x}$ is closer to $x_{1}$ than $x_{2}$: this
knowledge then allow us to apply labeled and unlabeled loss according to the
index of $x_{1}$. For data points in a batch $\mathcal{B}$ that are closer to
labeled data, we apply following supervised loss term:

$\displaystyle\mathcal{L}_{\mathcal{X}}=\underset{\mathcal{B}}{\sum}\ell(f(\tilde{x}),\tilde{y})$
(20)

For data points in $\mathcal{B}$ that are closer to unlabeled data, we apply
$L_{2}$ loss as it is considered to be less sensitive to incorrect
predictions:

$\displaystyle\mathcal{L}_{\mathcal{U}}=\underset{\mathcal{B}}{\sum}\left\lVert
f(\tilde{x})-\tilde{y}\right\rVert_{2}^{2}$ (21)

Finally, we combined ALP loss as defined in Eq. (15) with the mixup-based loss
terms of Eq. (20) and Eq. (21) as:

$\displaystyle\mathcal{L}=\underset{\text{mixup
loss}}{\underbrace{\mathcal{L}_{\mathcal{X}}+\lambda\cdot\mathcal{L}_{\mathcal{U}}}}+\underset{\text{Lipschitz
regularization}}{\underbrace{\zeta\cdot\mathcal{L}_{ALP}}}$ (22)

where $\lambda_{\mathcal{U}}$ is the weight term for the unsupervised loss,
and $\zeta$ is the weight term for the explicit Lipschitz regularization
presented in IV-B. We refer to the model trained in this manner as Mixup-LR
throughout the rest of the manuscript for brevity.

##  V Experiments

We test the effectiveness of the presented Mixup-LR on three standard SSL
benchmark datasets (CIFAR-10 [20], SVHN [21] and CIFAR-100 [20]) and a real-
world biomedical dataset (Skin Lesion images [22, 23]). We also consider the
robustness of the presented Mixup-LR against adversarial attacks (section
V-C).

###  V-A Implementation details

In all standard SSL benchmark experiments, we use the Wide ResNet-28 model
from [29], and for the biomedical dataset, we use the AlexNet model from [10].
Our implementation of the model and training procedure closely matches that of
[7]. For benchmark data sets, we follow modern standards in SSL and report the
median error rate of the last 20 checkpoints on all the unlabeled data points,
and on the biomedical dataset, we follow the classic approach and report the
result on test data by choosing the checkpoint with the lowest validation
error. In all experiments, we linearly ramp up $\lambda$ to its maximum value
over the training steps. We set $\zeta$ hyperparameter to 2 in all the cases,
and consider only 1 iteration to calculate adversarial perturbation $r_{adv}$.
Given the diversity of data sets considered, we leave other specific
implementation details to each subsection.

For comparison, we consider three existing SSL methods from [7]: $\Pi$-Model
[3, 24], Virtual Adversarial Training [4], and MixMatch [7]. The first two
methods represent SSL considering local smoothness, and MixMatch represents
SSL with global smoothness via a mixup strategy. Since MixMatch inspires the
presented Mixup-LR, we re-implemented MixMatch in the same codebase to ensure
a fair comparison. Furthermore, we trained each model on five random seeds and
reported the mean and standard deviation of the error rates.

Methods/Labels | 250 | 1000 | 4000
---|---|---|---
$\Pi$-Model [7] | 53.02 $\pm$ 2.05 | 31.53 $\pm$ 0.09 | 17.41 $\pm$ 0.37
VAT [7] | 36.03 $\pm$ 2.82 | 18.68 $\pm$ 0.40 | 11.05 $\pm$ 0.31
MixMatch [7] | 11.08 $\pm$ 0.87 | 7.75 $\pm$ 0.32 | 6.24 $\pm$ 0.06
MixMatch (ours) | 12.90 $\pm$ 2.10 | 8.73 $\pm$ 0.29 | 6.29 $\pm$ 0.11
Mixup-LR |  9.47 $\pm$ 0.99 |  7.59 $\pm$ 1.64 |  5.44 $\pm$ 0.06
TABLE I: Error rate (%) comparison of Mixup-LR to baseline methods on CIFAR10 for a varying number of labels.  Methods/Labels | 250 | 1000 | 4000
---|---|---|---
$\Pi$-Model [7] | 17.65 $\pm$ 0.27 | 8.60 $\pm$ 0.18 | 5.57 $\pm$ 0.14
VAT [7] | 8.41 $\pm$ 1.01 | 5.98 $\pm$ 0.21 | 4.20 $\pm$ 0.15
MixMatch [7] | 3.78 $\pm$ 0.26 | 3.27 $\pm$ 0.31 | 2.89 $\pm$ 0.06
MixMatch (ours) | 3.65 $\pm$ 0.25 | 3.26 $\pm$ 0.13 | 2.87 $\pm$ 0.05
Mixup-LR |  3.58 $\pm$ 0.30 |  3.09 $\pm$ 0.13 |  2.81 $\pm$ 0.04
TABLE II: Error rate (%) comparison of Mixup-LR to baseline methods on SVHN
for a varying number of labels.

###  V-B Generalization performance of SSL

####  V-B1 CIFAR-10

CIFAR-10 is the standard SSL benchmark datasets with 60000 data points divided
uniformly across ten labels. We evaluate the accuracy of each method
considered with a varying number of labeled examples (250, 1000, and 4000) on
all the unlabeled data sets. This means for the case of a labeled number of
250, we report the performance on the rest of the 59750 unlabeled samples. The
$\lambda$ value was set to 75. We present the results on Table I. As compared
with the perturbation-based approach ($\Pi$-Model and VAT), the mixup strategy
(via MixMatch) achieves better generalization throughout all the cases. By
augmenting the mixup strategy with explicit Lipschitz regularization, the
presented Mixup-LR further improves the generalization performance ($p<0.02$
for $l=250$ and $l=4000$, unpaired t-test). For instance, for the case of
labeled number of 250, the presented Mixup-LR reduces the mean error rate by
nearly 25%.

####  V-B2 SVHN

SVHN consists of 73257 samples divided across ten labels. Similar to CIFAR-10,
we evaluate the accuracy of each method considered with varying numbers of
labeled examples (250, 1000, and 4000). The $\lambda$ value was set to 250.
The obtained results are presented in Table II. Similar to CIFAR-10, the
presented Mixup-LR achieves better generalization across all the labeled
training setup compared to both the perturbation-based approaches ($\Pi$-Model
and VAT) and mixup-based approach (MixMatch, $p=0.04$ for $l=4000$, $p\approx
0.20$ for $l=250$ and $l=1000$, unpaired t-test).

Methods/Labels | 10000 | 15000
---|---|---
MixMatch | 31.67 $\pm$ 0.30 | 28.50 $\pm$ 0.11
Mixup-LR |  29.11 $\pm$ 0.10 |  26.21 $\pm$ 0.15
TABLE III: Error rate (%) comparison of Mixup-LR to MixMatch on CIFAR100 for a varying number of labels.  Methods/Labels | 600 | 1200
---|---|---
Supervised baseline | 0.80 $\pm$ 1.70 | 0.85 $\pm$ 1.06
MixMatch | 0.88 $\pm$ 1.21 | 0.89 $\pm$ 0.47
Mixup-LR |  0.89 $\pm$ 0.73 |  0.90 $\pm$ 0.66
TABLE IV: AUROC comparison of Mixup-LR to MixMatch on Skin Lesion dataset for
a varying number of labels.

####  V-B3 CIFAR-100

CIFAR-100 is similar to CIFAR-10 except that it has 100 classes containing 600
images each. The $\lambda$ value was set to 250. Note that due to the
increased complexity of the dataset, some prior works [7] have suggested the
use of a larger model (26 million parameter) instead of the base model
considered in this work (1.5 million parameter). As such, the presented
results might confound with results previously reported in the literature.
Thus, we have only considered the evaluation of the baseline MixMatch and
presented Mixup-LR on two different numbers of labeled examples (10000 and
15000). As shown in Table III, the presented Mixup-LR significantly improved
the performance compared to MixMatch in both cases ($p<0.0001$, unpaired
t-test)): the mean error rate for MixMatch was reduced by around 9% with the
presented Mixup-LR.

####  V-B4 Skin Lesion

ISIC 2018 skin data set comprises of 10015 dermoscopic images with labels for
seven different disease categories. To evaluate the presented method, we
created two sets of labeled training data (600 and 1200) considering the class
balance. Similar to CIFAR-100, we compared the presented Mixup-LR against the
baseline MixMatch. In this case, the $\lambda$ value was set to 50. Unlike
other cases, to maintain the standards of the dataset, we report the AUROC.
Since this biomedical dataset is relatively less considered in the SSL
literature, we also included the supervised baseline. The obtained results are
shown in Table IV, where, similar to other datasets, the presented Mixup-ALR
achieves better generalization compared to the MixMatch approach, and the
supervised baseline. The receiver operating characteristic (ROC) curves for
the corresponding labeled number of 600 and 1200 are respectively presented in
Fig. 3 and Fig. 4. In these figures, we randomly selected the model for
demonstrating the ROC comparison among the five random seed models.

Figure 3: ROC curves of the presented Mixup-LR compared to alternative models
for classification of dermoscopic images into seven different disease
categories when trained with 600 labels. Figure 4: ROC curves of the presented
Mixup-LR compared to alternative models for classification of dermoscopic
images into seven different disease categories when trained using 1200 labels.

####  V-B5 Ablation study

Here, we primarily study the effect of $\zeta$ hyperparameter in the presented
SSL method, using a labeled dataset of size 250 for the CIFAR-10 data set. The
$\zeta$ hyperparameter controls the effect of the presented regularization
with the mixup-based loss, as shown in Eq. (22). We conducted this study for
four different values (0, 1, 2, and 3), as shown in Table. V. While the
inclusion of the Lipschitz penalty improves upon the baseline method (_i.e.,_
$\zeta$=0 vs. rest), different hyperparameter values of $\zeta$ produce a
similar result with the best case at $\zeta$=2. We also experiment to
understand the effect of the quality of adversarial perturbation on the
presented loss. Toward this, we increase the iteration number to 2 and for a
single seed experiment, obtain the median accuracy as 88.42 on CIFAR-10 (250
labels) compared to 90.53 $\pm$ 0.99 with doing a single iteration. This shows
that the single iteration for calculating adversarial perturbation for the
Lipsthiz penalty is reasonable enough in these SSL problems corroborating with
the previous discussion in the literature [4, 16]. However, increasing the
iteration beyond two may further improve the performance of the presented
method.

Ablation | CIFAR-10 (250 labels)
---|---
$\zeta$ = 0 | 12.90 $\pm$ 2.10
$\zeta$ = 1 | 10.03 $\pm$ 0.63
$\zeta$ = 2 |  9.47 $\pm$ 0.99
$\zeta$ = 3 | 9.77 $\pm$ 0.58
TABLE V: Ablation study on $\zeta$ hyperparameter.

###  V-C Robustness

The neural network vulnerability to the adversarial examples is well-known
phenomena [30, 31]. We hypothesize that compared to the mixup-based approach,
our presented Mixup-LR is more robust against the adversarial attacks for two
reasons. First, Mixup-LR is Lipschitz regularized using adversarial samples,
making them less insensitive to similar adversarial perturbations. Second, the
Lipschitz penalty of Mixup-LR enforces local Lipschitzness in the classifier
function. Recent works have theoretically demonstrated that regularizing the
local Lipschitzness of the classifier function helps in achieving high clean
and robust accuracy [32]. To test this hypothesis, we consider the Fast
Gradient Sign Method (FGSM) [31], which constructs adversarial examples in one
single step. We consider two different pixel-wise perturbation amount (_i.e.,_
$\epsilon$) of 0.07 and 0.007. We evaluate this adversarial attack on the SSL
model trained with the CIFAR-10 dataset. In Table VI, we present the
performance of networks trained with Mixup-LR compared against MixMatch. As we
can see for all the models trained with different labeled numbers (250, 1000,
and 4000), Mixup-LR demonstrates better robustness compared to MixMatch.
However, note that the result of adversarial robustness might be biased since
Mixup-LR was already better in terms of clean accuracy (as seen in Table I).
To investigate this, we evaluate the drop in performance after an adversarial
attack. We present the result in Fig. 5 where the percentage drop of the
presented Mixup-LR (solid line) is less than the corresponding MixMatch model
(dotted line). This confirms that the presented Mixup-LR is robust compared to
the model trained with the Mixup strategy only.

| MixMatch | Mixup-LR
---|---|---
$\epsilon$ | 0.007 | 0.07 | 0.007 | 0.07
250 | 68.01 $\pm$ 10.03 | 75.26 $\pm$ 7.64 | 55.24 $\pm$ 4.79 | 63.67 $\pm$ 4.00
1000 | 53.68 $\pm$ 2.35 | 63.67 $\pm$ 1.48 | 47.72 $\pm$ 3.04 | 57.64 $\pm$ 3.16
4000 | 57.73 $\pm$ 1.80 | 67.55 $\pm$ 2.20 | 53.55 $\pm$ 6.08 | 61.69 $\pm$ 5.39
TABLE VI: Accuracy of unlabeled data points on white-box FGSM adversarial
examples on CIFAR-10. We compare the presented Mixup-LR against MixMatch for
two different $\epsilon$ for three different labeled cases. Figure 5:
Percentage drop for the presented Mixup-LR compared to MixMatch after
adversarial attack (FGSM) for two perturbation amount $\epsilon=0.07$ and
$\epsilon=0.007$.

##  VI Discussion and Future Work

While this research presents a novel SSL model (Mixup-LR) to improve the
state-of-the-art MixMatch method on benchmark datasets, there are certain
limitations of the current research which we discuss here.

First, the current approach to penalize the violation of the Lipschitz
constraint might be expensive as it requires 1 step of back-propagation for
each power iteration step while calculating the adversarial perturbation.
Although this extra computation is standard in Lipschitz regularization (e.g.,
Gradient Penalty [15]), there are recent works that have demonstrated the
efficacy of cheap techniques for obtaining adversarial examples [33]. As
future work, we will consider such methods to eliminate the overhead cost of
generating adversarial examples for Lipschitz regularization.

Second, in this work, by augmenting the Lipschitz regularization with a mixup-
based strategy, we control the Lipschitz constant of the deep neural network.
As future work, we want to empirically validate this proposition by estimating
the network’s Lipschitz constants. Although estimation of the Lipschitz
constant for deep networks often suffers from either lack of accuracy or poor
scalability, some of the recent works have demonstrated the accurate
performance [34]. We leave utilizing these latest findings of the literature
for the future.

##  VII Conclusion

We presented a novel SSL method, Mixup-LR, which combines a mixup-based
strategy with the explicit Lipschitz regularization. We first showed the
effect of mixup regularization in promoting smoothness, where the mixup
approach was found to bound only the Lipschitz constant of the gradient of the
neural function. As such, we augmented mixup with explicit Lipschitz
regularization to control the Lipschitz constant of the function itself. The
efficacy of the presented Mixup-LR was demonstrated on three SSL benchmark
data sets and one real-world clinical data set, through improvement over
state-of-the-art MixMatch model along with other standard SSL algorithms. We
also demonstrated the robustness of Mixup-LR against single-step adversarial
attacks.

## References

  * [1] Prashnna Kumar Gyawali, Zhiyuan Li, Sandesh Ghimire, and Linwei Wang,  “Semi-supervised learning by disentangling and self-ensembling over stochastic latent space,”  in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2019, pp. 766–774.
  * [2] Dengyong Zhou, Olivier Bousquet, Thomas N Lal, Jason Weston, and Bernhard Schölkopf,  “Learning with local and global consistency,”  in Advances in neural information processing systems, 2004, pp. 321–328.
  * [3] Samuli Laine and Timo Aila,  “Temporal ensembling for semi-supervised learning,”  in ICLR, 2017.
  * [4] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii,  “Virtual adversarial training: a regularization method for supervised and semi-supervised learning,”  IEEE transactions on pattern analysis and machine intelligence, vol. 41, no. 8, pp. 1979–1993, 2018.
  * [5] Yucen Luo, Jun Zhu, Mengxi Li, Yong Ren, and Bo Zhang,  “Smooth neighbors on teacher graphs for semi-supervised learning,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 8896–8905.
  * [6] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz,  “mixup: Beyond empirical risk minimization,”  arXiv preprint arXiv:1710.09412, 2017.
  * [7] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel,  “Mixmatch: A holistic approach to semi-supervised learning,”  in Advances in Neural Information Processing Systems, 2019, pp. 5050–5060.
  * [8] Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz,  “Interpolation consistency training for semi-supervised learning,”  in Proceedings of the 28th International Joint Conference on Artificial Intelligence. AAAI Press, 2019, pp. 3635–3641.
  * [9] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, Aaron Courville, David Lopez-Paz, and Yoshua Bengio,  “Manifold mixup: Better representations by interpolating hidden states,”  arXiv preprint arXiv:1806.05236, 2018.
  * [10] Prashnna Kumar Gyawali, Sandesh Ghimire, Pradeep Bajracharya, Zhiyuan Li, and Linwei Wang,  “Semi-supervised medical image classification with global latent mixing,”  arXiv preprint arXiv:2005.11217, 2020.
  * [11] Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio,  “Generalization in deep learning,”  arXiv preprint arXiv:1710.05468, 2017.
  * [12] Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael Cree,  “Regularisation of neural networks by enforcing lipschitz continuity,”  arXiv preprint arXiv:1804.04368, 2018.
  * [13] Martin Arjovsky, Soumith Chintala, and Léon Bottou,  “Wasserstein gan,”  arXiv preprint arXiv:1701.07875, 2017.
  * [14] Kavosh Asadi, Dipendra Misra, and Michael Littman,  “Lipschitz continuity in model-based reinforcement learning,”  in International Conference on Machine Learning, 2018, pp. 264–273.
  * [15] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville,  “Improved training of wasserstein gans,”  in Advances in neural information processing systems, 2017, pp. 5767–5777.
  * [16] Dávid Terjék,  “Virtual adversarial lipschitz regularization,”  arXiv preprint arXiv:1907.05681, 2019.
  * [17] Tsui-Wei Weng, Huan Zhang, Pin-Yu Chen, Jinfeng Yi, Dong Su, Yupeng Gao, Cho-Jui Hsieh, and Luca Daniel,  “Evaluating the robustness of neural networks: An extreme value theory approach,”  ICLR, 2018.
  * [18] Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky,  “Spectrally-normalized margin bounds for neural networks,”  in Advances in Neural Information Processing Systems, 2017, pp. 6240–6249.
  * [19] Ilsang Ohn and Yongdai Kim,  “Smooth function approximation by deep neural networks with general activation functions,”  Entropy, vol. 21, no. 7, pp. 627, 2019.
  * [20] Alex Krizhevsky, Geoffrey Hinton, et al.,  “Learning multiple layers of features from tiny images,”  2009\.
  * [21] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng,  “Reading digits in natural images with unsupervised feature learning,”  2011\.
  * [22] Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al.,  “Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic),”  arXiv preprint arXiv:1902.03368, 2019.
  * [23] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler,  “The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,”  Scientific data, vol. 5, pp. 180161, 2018.
  * [24] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen,  “Regularization with stochastic transformations and perturbations for deep semi-supervised learning,”  in Advances in neural information processing systems, 2016, pp. 1163–1171.
  * [25] Xiang Wei, Boqing Gong, Zixia Liu, Wei Lu, and Liqiang Wang,  “Improving the improved training of wasserstein gans: A consistency term and its dual effect,”  in International Conference on Learning Representation (ICLR), 2018\.
  * [26] Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik,  “Vicinal risk minimization,”  in Advances in neural information processing systems, 2001, pp. 416–422.
  * [27] Adam M Oberman and Jeff Calder,  “Lipschitz regularized deep neural networks converge and generalize,”  arXiv preprint arXiv:1808.09540, 2018.
  * [28] Henning Petzka, Asja Fischer, and Denis Lukovnicov,  “On the regularization of wasserstein gans,”  arXiv preprint arXiv:1709.08894, 2017.
  * [29] Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow,  “Realistic evaluation of deep semi-supervised learning algorithms,”  in Advances in Neural Information Processing Systems, 2018, pp. 3235–3246.
  * [30] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus,  “Intriguing properties of neural networks,”  arXiv preprint arXiv:1312.6199, 2013.
  * [31] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy,  “Explaining and harnessing adversarial examples,”  arXiv preprint arXiv:1412.6572, 2014.
  * [32] Yao-Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Ruslan Salakhutdinov, and Kamalika Chaudhuri,  “Adversarial robustness through local lipschitzness,”  arXiv preprint arXiv:2003.02460, 2020.
  * [33] Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein,  “Adversarial training for free!,”  in Advances in Neural Information Processing Systems, 2019, pp. 3353–3364.
  * [34] Mahyar Fazlyab, Alexander Robey, Hamed Hassani, Manfred Morari, and George Pappas,  “Efficient and accurate estimation of lipschitz constants for deep neural networks,”  in Advances in Neural Information Processing Systems, 2019, pp. 11423–11434.