# Beckman Defense

A V Subramanyam IIITD<EMAIL_ADDRESS>
###### Abstract

Optimal transport (OT) based distributional robust optimisation (DRO) has
received some traction in the recent past. However, it is at a nascent stage
but has a sound potential in robustifying the deep learning models.
Interestingly, OT barycenters demonstrate a good robustness against
adversarial attacks. Owing to the computationally expensive nature of OT
barycenters, they have not been investigated under DRO framework. In this
work, we propose a new barycenter, namely Beckman barycenter, which can be
computed efficiently and used for training the network to defend against
adversarial attacks in conjunction with adversarial training. We propose a
novel formulation of Beckman barycenter and analytically obtain the barycenter
using the marginals of the input image. We show that the Beckman barycenter
can be used to train adversarially trained networks to improve the robustness.
Our training is extremely efficient as it requires only a single epoch of
training. Elaborate experiments on CIFAR-10, CIFAR-100 and Tiny ImageNet
demonstrate that training an adversarially robust network with Beckman
barycenter can significantly increase the performance. Under auto attack, we
get a a maximum boost of 10% in CIFAR-10, 8.34% in CIFAR-100 and 11.51% in
Tiny ImageNet. Our code is available at https://github.com/Visual-Conception-
Group/test-barycentric-defense.

##  1 Introduction

Optimal mass transport (OT), originally proposed by Monge in his seminal work
Monge (1781), has gathered a widespread interest in the field of learning
representations. The original deterministic OT problem was later relaxed by
Kantorovich Kantorovich (1942) and considered a probabilistic transport
problem. This formulation seeks solution for the optimal transport plan which
can transport mass between two measures by incurring the minimum cost and is
solved using a linear program. The modern day OT is also attributed to the
phenomenal work of Kantorovich. Following the OT theory, barycenters in
Wasserstein space was proposed by Agueh and Carlier in their remarkable work
Agueh and Carlier (2011). Further, using entropic regularization Cuturi
(2013), a fast method of computing barycenters was proposed by Cuturi and
Doucet Cuturi and Doucet (2014). Recent works addresses the challenge of
computational complexity of barycenters using neural networks Lacombe et al.
(2021). In this work, we investigate the barycenters towards robust learning
of deep learning models.

Deep learning systems have shown impressive performance in various
applications. However, these systems are vulnerable to adversarial
perturbations Wong et al. (2020), Croce and Hein (2020), Xie et al. (2019). In
order to counter these attacks, several defense mechanisms have also been
proposed. In one of the early works, Szegedy et al. Szegedy et al. (2013)
formulated the adversarial attack as an optimization problem and obtained the
adversarial sample using L-BFGS. Several adversarial attacks have been
proposed since Szegedy’ work Goodfellow et al. (2014); Kurakin et al. (2016).
On the other hand, strong defense measures have been studied in Madry et al.
(2017), Theagarajan et al. (2019), Wong et al. (2020), Rebuffi et al. (2021).

Figure 1: Illustration: Classical defense methods use Adversarial Training
(AT) as a major defense technique. Our method obtains barycenter from rotated
inputs and uses them for training the model using a cross-entropy loss. During
inference time also we compute barycenter of the given sample. The dashed
boundary of barycenter indicates that the barycenter is close to input samples
in terms of appearance but there are some differences. In the computation of
barycenter of adversarial sample, the barycenter shows the changes in same
color as that of the background to imply that barycenter suppresses the
adversarial noise.

In the field of adversarial attacks and defense, $l_{p}$ space has been
extensively studied. However, only a few works investigate attacks under OT
framework Wong et al. (2019), Li et al. (2021). There are even fewer works
which investigate robustness using OT theory Kwon et al. (2020), Subramanyam
and Raj (2022). Distinct from these works, we first introduce Beckman
barycenter, a concept analogous to Wasserstein barycenter. We use proximal
operator methods to solve for the barycenter. The barycenters obtained from
the clean samples are used to train a pretrained adversarially robust network.
We note that in the absence of adversarial samples in the training, the model
would give a better clean accuracy but will suffer in terms of adversarial
accuracy. Therefore, we use a pre-trained adversarially robust network to
overcome this challenge. An abstract illustration of our method is given in
Figure 1.

Beckman barycenter is obtained from input marginals via a non-linear
interpolation. The input marginals are linearly transformed versions of the
input and thus interfere with the adversarial noise. Using these marginals the
barycenter generates a sample which is similar in appearance to the input and
is closer in terms of class label. Thus, the class label is preserved when the
input is a clean sample, whereas, the adversarial noise gets suppressed when
the input is an adversarial sample. Further, the network needs to be trained
with barycenter of clean samples so as to correctly classify them. However,
this training is cheap as a single epoch is sufficient. We prove our
hypothesis using extensive qualitative and quantitative experiments.

##  2 Related Works

Adversarial Attacks Given an adversarial sample $\mathbf{x}$ with label $y$, a
target network $f$ parameterized by $\theta$, the adversary tries to find
$\mathbf{x}_{adv}$ by adding an adversarial noise such that the prediction
$f_{\theta}(\mathbf{x}_{adv})\neq f_{\theta}(\mathbf{x})=y$. Some of the
robust attacks are iterative FGSM Kurakin et al. (2016), PGD Madry et al.
(2017), Carlini and Wagner attacks Carlini and Wagner (2017), Jacobian based
attack Papernot et al. (2016), physical attack Athalye Athalye et al. (2018),
and Autoattack Croce and Hein (2020). These attacks are primarily focused in
$l_{p}$ domain.

Adversarial Defense In response to adversarial attacks, several defenses have
been proposed. One of the best defense approach is adversarial training
Szegedy et al. (2013), Goodfellow et al. (2014), Moosavi-Dezfooli et al.
(2016). Madry et al. Madry et al. (2017) formally studied adversarial training
and proposed that such training allows network to defend well against first
order adversary. Adversarial logit pairing uses a pair of logits from clean
and adversarial examples to defend against adversarial samples Kannan et al.
(2018). TRADES Zhang et al. (2019) prove the bounds based on regularization
term which minimizes the difference in prediction between clean and
adversarial examples. In Wong et al. (2020), authors proposed to effectively
combine FGSM and random initialization to demonstrate better adversarial
training. RST Carmon et al. (2019) propose a self-training technique using
unlabelled samples to improve the robustness. Observing the correlation
between flatness of weight loss landscape and adversarial robustness, Wu et
al. proposed adversarial weight perturbation (AWP) to regularize the flatness
of weight loss Wu et al. (2020). On similar lines, Yu et al. (2022) propose a
criterion called Loss Stationary Condition (LSC) for constrained perturbation,
which regulates the weight perturbation to prevent overfitting. LBGAT Cui et
al. (2021) constrains the logits of a robust model, trained with adversarial
examples, to be similar to the logits of a clean model trained on natural
data.

While adversarial training uses all the samples, many techniques propose that
naively using adversarial samples in adversarial training is not efficient.
This primarily involves training the model with a weak attack first, and then
gradually increasing the strength of the adversary - CAT Cai et al. (2018),
DART Wang et al. (2019a), MART Wang et al. (2019b), FAT Zhang et al. (2020).
Aforementioned methods rely on pre-determined attack parameters for
adversarial sample generation. However, this restricts the model’s robustness.
To address this issue, LAS-AT Jia et al. (2022) propose a framework for
adversarial training that introduces the notion of learnable attack strategy.
It is composed of two components: a target network that uses adversarial
examples for training to improve robustness, and a strategy network that
produces attack strategies to control adversarial sample generation. In
similar spirit, A2 Xu et al. (2022) and Cheng et al. (2022) have also been
proposed. A classical review of defense methods can be obtained in Bai et al.
(2021).

In a parallel line of defense works, input purification has also been
explored. At the test time, these techniques try to remove the adversarial
noise Shi et al. (2021), TRADES${}_{\text{SSL}}$ Mao et al. (2021),
Hedge${}_{\text{RST}}$ Wu et al. (2021). Score based generative models such as
Yoon et al. (2021) and Nie et al. (2022) have also been used to purify the
images before sending them for classification.

Our work is inspired from two different theories, namely, OT barycenters and
distributional robust optimization. We discuss these theories in the
following.

Wasserstein Barycenter In the following we discuss Wasserstein distance and
barycenter. Given probability distributions,
$\boldsymbol{\mu}_{1},\boldsymbol{\mu}_{2}$ $\in$ $\Omega$, the Wasserstein
distance is defined as,

$\displaystyle\mathcal{W}(\boldsymbol{\mu}_{1},\boldsymbol{\mu}_{2})=\inf_{\Omega\times\Omega}c(x,y)\pi(x,y)dxdy,$
(1) $\displaystyle
s.t.\;\int_{\Omega}\pi(x,y)dx=\boldsymbol{\mu}_{1}(x),\int_{\Omega}\pi(x,y)dy=\boldsymbol{\mu}_{2}(y),$

where the cost matrix $c(x,y)=\lVert x-y\rVert_{1}$ and $\pi$ denotes the
transport plan. This is also known as Earth Mover’ Distance (EMD). This form
is also used to compute barycenter Cuturi and Peyré (2016) wherein the
summation of Wasserstein distance between the barycenter and each input
marginal is considered. However, barycenters are costly to compute and the
best known complexity scales exponentially with the number of marginals Fan et
al. (2022).

EMD can also be represented as dual of the dual of Eq 1 in variational form
popularly introduced by Beckman Beckmann (1952), Li et al. (2018), Lee et al.
(2020),

$\displaystyle\mathcal{W}(\boldsymbol{\mu}_{1},\boldsymbol{\mu}_{2})=\inf_{\mathbf{M}}\int_{\Omega}\lVert\mathbf{M}\rVert$
(2) $\displaystyle
s.t.\;\text{div}(\mathbf{M})+\boldsymbol{\mu}_{1}-\boldsymbol{\mu}_{2}=0$
$\displaystyle\mathbf{M}.\mathbf{n}=0\;\forall
x\in\partial\boldsymbol{\Omega};n\;\text{is normal
to}\;\partial\boldsymbol{\Omega}$

Under appropriate discretization,
$\mathbf{M}=(\mathbf{M}_{x},\mathbf{M}_{y})$,
$\mathbf{M}\in\mathcal{R}^{n\times 2}$ is flux vector satisfying zero flux
boundary conditions.
$\boldsymbol{\mu}_{1},\boldsymbol{\mu}_{2}\in\mathcal{R}^{n}$, and,

$\text{div}(\mathbf{M})=(\mathbf{M}_{x}[i,j]-\mathbf{M}_{x}[i-1,j])+(\mathbf{M}_{y}[i,j]-\mathbf{M}_{y}[i,j-1])$

and the zero-flux boundary conditions mean that
$\mathbf{M}_{x}[i,j]=\mathbf{M}_{y}[i,j]=0$ outside the boundary. Eq 2 is
favorable compared to Eq 1 as it reduces the complexity from
$\mathcal{O}(n^{2})$ to $\mathcal{O}(n)$ Li et al. (2018). Motivated by the
recent developments of OT barycenters, we make use of Eq 2 to propose Beckman
barycenter as they can be efficiently solved using well known techniques like
Goldstein and Osher (2009), Chambolle and Pock (2011).

DRO One of the influential works in DRO was proposed by Scarf Scarf (1957).
Following this work, significant research has been done in this field Ben-Tal
et al. (2009), Duchi et al. (2021), Staib and Jegelka (2017). DRO aims to
address the problem of uncertainty or shift in the data distribution that can
arise due to measurement errors and admits a solution for the worst case
scenario. Let $\mathcal{L}(\boldsymbol{\theta},\mathbf{x})$ be the loss
function where $\boldsymbol{\theta}$ are network parameters. Then, DRO solves
for,

$\inf_{\boldsymbol{\theta}}\sup_{Q\in\mathcal{Q}}\mathbb{E}_{\mathcal{Q}}\mathcal{L}(\boldsymbol{\theta},\mathbf{x})$
(3)

Here, $\mathcal{Q}$ is the distribution against which DRO minimizes the loss.
For instance, $\mathcal{Q}$ can be considered as a distribution set which
contains perturbations of input samples $\mathbf{x}$. Here we note that
adversarial training can be considered to be a specific instance of DRO
wherein the distribution $\mathcal{Q}$ is drawn from adversarial samples. In
our case, we consider the barycenters as the samples drawn from the
distribution $\mathcal{Q}$ and thus provide robustness against perturbed
samples.

###  2.1 Proposed Algorithm

In this work, we propose a novel Beckman barycenter formulation and derive the
barycenter analytically. We use the barycenter to demonstrate that it can be
applied for adversarial defense. We first obtain the barycenter using the
marginals from the given input image and then train the network using
barycenter.

While OT barycenters are a good choice for the distribution $\mathcal{Q}$ in
Eq 3, computing OT barycenter suffers from high complexity and exponentially
increases with the number marginals Fan et al. (2022). To counter this high
complexity challenge, we first discuss an analogous barycenter problem by
building upon the formulation given in Eq 2,

$\displaystyle\inf_{\begin{subarray}{c}\mathbf{M}_{1},\mathbf{M}_{2}\\\
\mathbf{r}_{1},\mathbf{r}_{2},\boldsymbol{\mu}\end{subarray}}\lVert\mathbf{M}_{1}\rVert_{2,1}+\lVert\mathbf{M}_{2}\rVert_{2,1}+\alpha(\lVert\mathbf{r}_{1}\rVert_{1}$
(4)
$\displaystyle+\lVert\mathbf{r}_{2}\rVert_{1})+\beta\lVert\boldsymbol{\mu}\rVert_{1}$
$\displaystyle
s.t.\;\text{div}(\mathbf{M}_{1})+\boldsymbol{\mu}_{1}-\boldsymbol{\mu}=\mathbf{r}_{1}$
$\displaystyle\text{div}(\mathbf{M}_{2})+\boldsymbol{\mu}_{2}-\boldsymbol{\mu}=\mathbf{r}_{2}$

where, $\mathbf{r}_{1},\mathbf{r}_{2},\boldsymbol{\mu}\in\mathcal{R}^{n}$. Our
formulation is loosely inspired from the Beckman OT formulation that is given
in Li et al. (2018), Lee et al. (2020). There are notable changes in Eq 4 from
Eq 2. First we solve for Beckman barycenter $\boldsymbol{\mu}$ in addition to
other variables. Similar to Wasserstein barycenter which acts as a
representative of marginals using Wasserstein metric, the Beckman barycenter
$\boldsymbol{\mu}$ minimizes the flux with respect to input marginals
$\boldsymbol{\mu}_{1}$ and $\boldsymbol{\mu}_{2}$. In our experiments, these
marginals are obtained by rotating the input image with $\pm 4^{\circ}$.
Second, the variables $\mathbf{r}_{1}$ and $\mathbf{r}_{2}$ allow the mass to
be created or destroyed Lee et al. (2020) and the regularization over
$\mathbf{r}_{1}$, $\mathbf{r}_{2}$ and $\boldsymbol{\mu}$ ensure that these
variable do not take arbitrarily large values. Third, Eq 4 can be easily
converted to Lagrange formulation and solved in linear time using primal-dual
method of Chambolle and Pock Chambolle and Pock (2011).

In order to make the objective strongly convex, we first apply proximal
operators. The $l_{2}$ regularizer makes the objective strongly convex. Using
the proximal operator,

$\displaystyle\inf_{\begin{subarray}{c}\mathbf{M}_{1},\mathbf{M}_{2},\mathbf{r}_{1}\\\
\mathbf{r}_{2},\boldsymbol{\mu}_{1}^{\prime},\boldsymbol{\mu}_{2}^{\prime},\boldsymbol{\mu}\end{subarray}}\lVert\mathbf{M}_{1}\rVert_{2,1}+\lVert\mathbf{M}_{2}\rVert_{2,1}+\alpha(\lVert\boldsymbol{r}_{1}\rVert_{1}$
(5)
$\displaystyle+\lVert\boldsymbol{r}_{2}\rVert_{1})+\frac{1}{2\rho}(\lVert\boldsymbol{\mu}_{1}^{\prime}-\boldsymbol{\mu}_{1}\rVert_{2}+\lVert\boldsymbol{\mu}_{2}^{\prime}-\boldsymbol{\mu}_{2}\rVert_{2})+\beta\lVert\boldsymbol{\mu}\rVert_{1}$
$\displaystyle
s.t.\;\text{div}(\mathbf{M}_{1})+\boldsymbol{\mu}_{1}^{\prime}-\boldsymbol{\mu}=\boldsymbol{r}_{1}$
$\displaystyle\text{div}(\mathbf{M}_{2})+\boldsymbol{\mu}_{2}^{\prime}-\boldsymbol{\mu}=\boldsymbol{r}_{2}$

The Lagrangian of Eq 5 is given as,

$\displaystyle\inf_{\begin{subarray}{c}\mathbf{M}_{1},\mathbf{M}_{2},\mathbf{r}_{1}\\\
\mathbf{r}_{2},\boldsymbol{\mu}_{1}^{\prime},\boldsymbol{\mu}_{2}^{\prime},\boldsymbol{\mu}\end{subarray}}\lVert\mathbf{M}_{1}\rVert_{2,1}+\lVert\mathbf{M}_{2}\rVert_{2,1}+\alpha(\lVert\boldsymbol{r}_{1}\rVert_{1}$
(6)
$\displaystyle+\lVert\boldsymbol{r}_{2}\rVert_{1})+\frac{1}{2\rho}(\lVert\boldsymbol{\mu}_{1}^{\prime}-\boldsymbol{\mu}_{1}\rVert_{2}+\lVert\boldsymbol{\mu}_{2}^{\prime}-\boldsymbol{\mu}_{2}\rVert_{2})+\beta\lVert\boldsymbol{\mu}\rVert_{1}$
$\displaystyle+\sum_{i}\langle\boldsymbol{\lambda}_{i},\text{div}(\mathbf{M}_{i})+\boldsymbol{\mu}_{i}^{\prime}-\boldsymbol{\mu}-\boldsymbol{r}_{i}\rangle$

Eq 6 can be solved using first-order primal dual method of Chambolle and Pock
Chambolle and Pock (2011)111We use similar notations to that of Li et al.
(2018), Chambolle and Pock (2011) for consistency and simplicity..

$\displaystyle\mathbf{M}_{i}^{t+1}\leftarrow\operatorname*{arg\,min}_{\mathbf{M}_{i}}\lVert\mathbf{M}_{i}\rVert_{2,1}+\langle\boldsymbol{\lambda}_{i},\text{div}(\mathbf{M}_{i})+\boldsymbol{\mu}_{i}^{\prime}-$
$\displaystyle\boldsymbol{\mu}-\mathbf{r}_{i}\rangle+\frac{1}{2\tau_{1}}\lVert\mathbf{M}_{i}-\mathbf{M}_{i}^{t}\rVert_{2}\;\;\;\;\forall
i=\\{1,2\\}$
$\displaystyle{\boldsymbol{\mu}^{\prime}_{i}}^{t+1}\leftarrow\operatorname*{arg\,min}_{{\boldsymbol{\mu}^{\prime}_{i}}}\frac{1}{2\tau_{1}}(\lVert\boldsymbol{\mu}_{i}^{\prime}-\boldsymbol{\mu}_{i}\rVert_{2})+\langle\boldsymbol{\lambda}_{i},\boldsymbol{\mu}_{i}^{\prime}\rangle$
$\displaystyle+\frac{1}{2\tau_{1}}\lVert\boldsymbol{\mu}_{i}^{\prime}-{\boldsymbol{\mu}^{\prime}_{i}}^{t}\rVert_{2}$
$\displaystyle\mathbf{r}_{i}^{t+1}\leftarrow\operatorname*{arg\,min}_{\mathbf{r}_{i}}\alpha\lVert\boldsymbol{r}_{i}\rVert_{1}+\langle\boldsymbol{\lambda}_{i}^{t},-\mathbf{r}_{i}\rangle+\frac{1}{2\tau_{1}}\lVert\boldsymbol{r}_{i}-\boldsymbol{r}_{i}^{t}\rVert_{2}$
$\displaystyle\boldsymbol{\mu}^{t+1}\leftarrow\operatorname*{arg\,min}_{\boldsymbol{\mu}}\lVert\boldsymbol{\mu}\rVert_{1}+\langle\boldsymbol{\lambda}_{i}^{t},-\boldsymbol{\mu}\rangle+\frac{1}{2\tau_{1}}\lVert\boldsymbol{\mu}-\boldsymbol{\mu}^{t}\rVert_{2}$
$\displaystyle\boldsymbol{\lambda}_{i}^{t+1}\leftarrow\operatorname*{arg\,max}_{\boldsymbol{\lambda}_{i}}\langle\boldsymbol{\lambda}_{i},\boldsymbol{\kappa}^{t+1}\rangle-\frac{1}{2\tau_{2}}\lVert\boldsymbol{\lambda}_{i}-\boldsymbol{\lambda}_{i}^{t}\rVert_{2},$

where,
$\boldsymbol{\kappa}^{t+1}=2(\text{div}(\mathbf{M}_{i})^{t+1}+{\boldsymbol{\mu}^{\prime}_{i}}^{t+1}-\boldsymbol{r}_{i}^{t+1})-(\text{div}(\mathbf{M}_{i})^{t}+{\boldsymbol{\mu}^{\prime}_{i}}^{t}-\boldsymbol{r}_{i}^{t})$.

We now discuss the solution of each individual optimization.

Solving for $\mathbf{M}_{i}$: The rows $\mathbf{m}_{ij}$ of $\mathbf{M}_{i}$
can be expressed and solved using $l_{21}$ norm shrinkage operator,

$\mathbf{m}_{ij}^{t+1}\leftarrow\text{shrink}_{\tau_{1}}^{l_{2}}(\mathbf{m}_{ij}^{t}-\tau_{1}\text{div}^{*}(\boldsymbol{\lambda}_{i}^{t})_{j})$
(7)

Here, $\text{div}^{*}$ denotes the adjoint of div operator, and
$\text{shrink}_{\tau_{1}}^{l_{2}}{\boldsymbol{\eta}}=\max(\lVert\boldsymbol{\eta}\rVert_{2}-\tau_{1},0)\odot\frac{\boldsymbol{\eta}}{(\lVert\boldsymbol{\eta}\rVert_{2})}$.
“$\odot$” denotes the Hadamard product.

Solving for $\boldsymbol{\mu}_{i}^{\prime}$:

$\centering{\boldsymbol{\mu}^{\prime}_{i}}^{t+1}\leftarrow\max\\{0,\frac{\rho\tau_{1}}{1+\rho\tau_{1}}\boldsymbol{\mu}_{i}^{\prime}+\frac{1}{1+\rho\tau_{1}}({\boldsymbol{\mu}_{i}^{\prime}}^{t}-\tau_{1}\boldsymbol{\lambda}_{i}^{t})\\},\@add@centering$
(8)

Solving for $\mathbf{r}_{i}$: We use an $l_{1}$ shrinkage operator.

$\mathbf{r}_{i}^{t+1}\leftarrow\text{shrink}_{\alpha\tau_{1}}^{l_{1}}(\mathbf{r}_{i}^{t}+\tau_{1}\boldsymbol{\lambda}_{i}^{t})$
(9)

Here,
$\text{shrink}_{\alpha\tau_{1}}^{l_{1}}(\boldsymbol{\eta})=\text{sign}(\boldsymbol{\eta})\odot\max(\lVert\boldsymbol{\eta}\rVert-\alpha\tau_{1},0)$.

Solving for barycenter $\boldsymbol{\mu}$:

$\boldsymbol{\mu}^{t+1}\leftarrow\text{shrink}_{\beta\tau_{1}}^{l_{1}}(\boldsymbol{\mu}^{t}+\tau_{1}(\boldsymbol{\lambda}_{1}^{t}+\boldsymbol{\lambda}_{2}^{t}))$
(10)

Solving for $\boldsymbol{\lambda}$:

$\boldsymbol{\lambda}_{i}^{t+1}\leftarrow\boldsymbol{\lambda}_{i}^{t}+\tau_{2}\boldsymbol{\kappa}^{t+1}$
(11)

###  2.2 Toy example

We demonstrate the barycenter computation using a Gaussian image in Figure 2.
The barycenter of clean samples, sample with random noise and adversarial
sample are shown. As we see, for the clean case the barycenter is very similar
to that of the original image. In the second column where random noise is
added, the barycenter reduces the noise. Similar effect is also seen for the
case where adversarial noise is present. This indicates that non-linear
interpolation of barycenter suppresses the adversarial noise.

(a) Clean

(b) Random

(c) Adversarial

Figure 2: Top: Clean image, noisy image, adversarial image. Bottom: Barycenter
of clean image, noisy image, adversarial image.

###  2.3 Training

Let a model be given by $f_{\boldsymbol{\theta}}$, the barycenter of clean
samples be denoted by $\mathbf{x}$ and its labels as $y$. We then optimize the
following loss

$\operatorname*{arg\,min}_{\boldsymbol{\theta}}\frac{1}{n}\sum_{i=1}^{n}\mathcal{L}_{CE}(f_{\boldsymbol{\theta}}(\mathbf{x}_{i}),\mathbf{y}_{i})$

where $\mathcal{L}_{CE}$ is the cross-entropy loss. We would like to emphasize
that we do not perform adversarial training. Instead we use an adversarially
pretrained model. Thus, $f_{\boldsymbol{\theta}}$ is an adversarial robust
model and our training further enhances the robustness. We also note that this
optimization falls under DRO as the samples used are barycenters which belong
to the distribution $\mathcal{Q}$.

###  2.4 Theoretical analysis

We first present a convergence analysis of Eq 6.

Theorem 1. Let
$\tau_{1}\tau_{2}(\lambda_{\max}(\boldsymbol{\nabla}^{2})+3)<1$, where
$\lambda_{max}(\boldsymbol{\nabla}^{2})$ denotes the largest eigenvalue of
discrete Laplacian operator
$\boldsymbol{\nabla}^{2}=\mathbf{D}\mathbf{D}^{\top}$, where $\mathbf{D}$ is
the matrix representing div operator. Then, the iterations
$\mathbf{M}^{t}_{i},{\boldsymbol{\mu}^{\prime}}^{t}_{i},\boldsymbol{\mu}^{t},\mathbf{r}^{t}_{i},\boldsymbol{\lambda}^{t}_{i}$
converge to the saddle point solution of the Lagrangian
$\mathbf{M}^{*}_{i},\boldsymbol{\mu}^{*}_{i},\boldsymbol{\mu}^{*},\mathbf{r}^{*}_{i},\boldsymbol{\lambda}^{*}_{i}$.

Proof: Let
$\mathbf{u}=\\{\mathbf{M}_{1},\mathbf{M}_{2},\boldsymbol{\mu}_{2},\boldsymbol{\mu}_{2},\boldsymbol{\mu},\mathbf{r}\\}$.
Then, we write Eq 6 as

$\mathcal{L}(\mathbf{u},\boldsymbol{\lambda})=\mathcal{G}(\mathbf{u})+\langle\boldsymbol{\lambda},\tilde{\mathbf{K}}\mathbf{b}\rangle$

where
$\boldsymbol{\lambda}=[\boldsymbol{\lambda}_{1};\boldsymbol{\lambda}_{2}]$,
$\mathbf{K}=[\mathbf{D},\mathbf{I},\mathbf{-I},\mathbf{-I}]$,
$\tilde{\mathbf{K}}=[\mathbf{K},\mathbf{0};\mathbf{0},\mathbf{K}]$,
$\mathbf{b}=[\mathbf{b}_{1};\mathbf{b}_{2}],\mathbf{b}_{1}=[\text{vec}(\mathbf{M}_{1});\boldsymbol{\mu}^{\prime}_{1};\boldsymbol{\mu};\boldsymbol{r}_{1}];\mathbf{b}_{2}=[\text{vec}(\mathbf{M}_{2});\boldsymbol{\mu}^{\prime}_{2};\boldsymbol{\mu};\boldsymbol{r}_{2}]$.
The function
$\mathcal{G}=\lVert\mathbf{M}_{1}\rVert_{2,1}+\lVert\mathbf{M}_{2}\rVert_{2,1}+\alpha(\lVert\boldsymbol{r}_{1}\rVert_{1}+\lVert\boldsymbol{r}_{2}\rVert_{1})+\frac{1}{2\rho}(\lVert\boldsymbol{\mu}_{1}^{\prime}-\boldsymbol{\mu}_{1}\rVert_{2}+\lVert\boldsymbol{\mu}_{2}^{\prime}-\boldsymbol{\mu}_{2}\rVert_{2})+\beta\lVert\boldsymbol{\mu}\rVert_{1}$
is convex and $\tilde{\mathbf{K}}$ is a linear operator. These conditions
satisfy Theorem 1 of Chambolle and Pock (2011). If
$\lambda_{max}(\boldsymbol{\nabla}^{2})$ is the max eigenvalue of
$\mathbf{D}\mathbf{D}^{\top}$, then the max eigenvalue of
$[\mathbf{D},\pm\mathbf{I}][\mathbf{D},\pm\mathbf{I}]^{\top}$ is
$\lambda_{\max}(\boldsymbol{\nabla}^{2})+1$. Similarly, for
$\mathbf{K}\mathbf{K}^{\top}$, it is
$\lambda_{\max}(\boldsymbol{\nabla}^{2})+3$. Since $\tilde{\mathbf{K}}$ is
obtained from $\mathbf{K}$ by padding zeros only, $\tilde{\mathbf{K}}$ has the
same max eigenvalue as that of $\mathbf{K}$. Further, since
$\lVert\tilde{\mathbf{K}}\tilde{\mathbf{K}}^{\top}\rVert^{2}_{2}\geq\lambda_{\max}(\tilde{\mathbf{K}}\tilde{\mathbf{K}}^{\top})=\lambda_{\max}(\boldsymbol{\nabla}^{2})+3$,
we can also write the convergence criteria as
$\tau_{1}\tau_{2}\lVert\tilde{\mathbf{K}}\tilde{\mathbf{K}}^{\top}\rVert^{2}_{2}<1$.

Since we solve for the Lagrangian dual function, we analyse the primal dual
gap which is given as Jacobs et al. (2019)

$\mathbb{G}(\mathbf{u},\boldsymbol{\lambda})={\sup_{\lVert\boldsymbol{\lambda}^{\prime}-\boldsymbol{\lambda}_{0}\rVert\leq
R_{1}}}\mathcal{L}(\mathbf{u},\boldsymbol{\lambda}^{\prime})-{\inf_{\lVert\mathbf{u}^{\prime}-\mathbf{u}_{0}\rVert\leq
R_{2}}}\mathcal{L}(\mathbf{u}^{\prime},\boldsymbol{\lambda})$

Theorem 2. Suppose the step sizes $\tau_{1}$ and $\tau_{2}$ satisfy
$\tau_{1}\tau_{2}\lVert\tilde{\mathbf{K}}\tilde{\mathbf{K}}^{\top}\rVert^{2}_{2}<1$.
Let $\mathbf{u}^{N}=\frac{1}{N}\sum^{N}_{n=1}\mathbf{u}_{n}$ and
$\boldsymbol{\lambda}^{N}=\frac{1}{N}\sum^{N}_{n=1}\boldsymbol{\lambda}_{n}$,
where $\mathbf{u}_{n}$ and $\boldsymbol{\lambda}_{n}$ are sequences generated
from Eqns 7 \- 11. Then after $N$ iterations, we have,

$\mathbb{G}(\mathbf{u},\mathbf{\boldsymbol{\lambda}})\leq{\sup_{\mathbf{u},\mathbf{\boldsymbol{\lambda}}}}\frac{1}{2N}\Bigg{(}\frac{\lVert\mathbf{u}-\mathbf{u}_{0}\rVert_{2}}{\tau_{1}}+\frac{\lVert\mathbf{\boldsymbol{\lambda}}-\mathbf{\boldsymbol{\lambda}}_{0}\rVert_{2}}{\tau_{2}}\Bigg{)}$

This rate is similar to convergence rates in gradient descent and shows that
the gap converges with rate $\mathcal{O}(1/N)$. For brevity, we omit the proof
and it can be derived as an extension of Theorem 1 Chambolle and Pock (2011).

###  2.5 Mutual Information

In order to understand the underlying reason behind the performance of our
method, we provide more insights using mutual information (MI). We first note
that the MI between two random variables is given by
$\mathcal{I}(X,y)=\mathcal{H}(\mathcal{P}(y))-\underset{\mathcal{P}(x)}{\mathbb{E}}[\mathcal{H}(\mathcal{P}(y|X))]$.
In our case, we take the random variables as model parameters
$\boldsymbol{\theta}$ and softmax output $\mathbf{y}$. Then, given a sample
$\mathbf{x}$ and dataset $\mathcal{D}$,

$\mathcal{I}(\boldsymbol{\theta},\mathbf{y}|\mathcal{D},\mathbf{x})=\mathcal{H}(p(\mathbf{y}|\mathbf{x},\mathcal{D}))-\underset{p(\boldsymbol{\theta}|\mathcal{D})}{\mathbb{E}}\mathcal{H}(p(\mathbf{y}|\mathbf{x},\boldsymbol{\theta}))$
(12)

Eq 12 measures the information shared between $\boldsymbol{\theta}$ and
$\mathbf{y}$. A tractable way of computing
$\mathcal{I}(\boldsymbol{\theta},\mathbf{y}|\mathcal{D},\mathbf{x})$ is given
in Smith and Gal (2018), Houlsby et al. (2011).

${I}(\boldsymbol{\theta},\mathbf{y}|D,\mathbf{x})=\frac{1}{C}\sum^{C}_{j=1}\frac{1}{n}\sum^{n}_{i=1}(p_{ij}-\hat{\mathbf{p}})^{2}$
(13)

where, $\hat{\mathbf{p}}\in[0,1]^{C}$ is computed as the mean of all softmax
probabilities, $C$ is the number of classes, $\mathbf{p}_{i}\in[0,1]^{C}$,
$p_{ij}\in[0,1]$ denotes the softmax probability for a particular class $j$. A
higher ${\mathcal{I}}$ indicates that knowing $\boldsymbol{\theta}$ (or
$\mathbf{y}$) gives a higher information about $\mathbf{y}$ (or
$\boldsymbol{\theta}$). In other words, the model will perform better if the
mutual information is high.

In addition, we also compute MI between the predictions for the following two
cases - (i) clean test set and adversarial test set, and (ii) barycenter of
clean test set and barycenter of adversarial test set using Ji et al. (2019).
Given a model $f$ parameterised by $\boldsymbol{\theta}$, clean sample
$\mathbf{x}_{i}$ and its adversarial counterpart $\mathbf{x}^{\prime}_{i}$,
the joint probability distribution between natural and adversarial samples is
given by the following $C\times C$ matrix,

$\mathcal{I}(f(\mathbf{x}_{i},\boldsymbol{\theta}),f(\mathbf{x}^{\prime},\boldsymbol{\theta}))=\sum^{C}_{y=1}\sum^{C}_{y^{\prime}=1}\mathcal{P}_{yy^{\prime}}\ln\frac{\mathcal{P}_{yy^{\prime}}}{\mathcal{P}_{y}\mathcal{P}_{y^{\prime}}}$
(14)

where, $\mathcal{P}_{yy^{\prime}}$ is given as,

$\mathcal{P}_{yy^{\prime}}=\frac{1}{n}\sum^{n}_{i=1}f(\mathbf{x}_{i},\boldsymbol{\theta})f(\mathbf{x}^{\prime}_{i},\boldsymbol{\theta})^{\top}$
(15)

and the marginals $\mathcal{P}_{y},\mathcal{P}_{y^{\prime}}$ are obtained by
row and column sum of $\mathcal{P}_{yy^{\prime}}$. A higher value of
$\mathcal{I}(.,.)$ indicates that knowing about clean samples gives a higher
amount of information about the adversarial samples.

##  3 Experiments

We present elaborate experimental results on CIFAR-10, CIFAR-100 and Tiny
ImageNet. We use strong baselines of LAS Jia et al. (2022), LBGAT Cui et al.
(2021), PGD-AT Madry et al. (2017), TRADES Zhang et al. (2019), RST Carmon et
al. (2019). We compare against several popular adversarial training models,
MART Wang et al. (2019b), AWP-A2 Xu et al. (2022), RST-RWT Yu et al. (2022),
TRADES${}_{\text{AWP}}$ Wu et al. (2020), AWP Wu et al. (2020),
LAS${}_{\text{AT}}$, LAS${}_{\text{TRADES}}$, LAS${}_{\text{AWP}}$ Jia et al.
(2022). We also compare with adaptive test time defenses
Hedge${}_{\text{RST}}$ Wu et al. (2021) and TRADES${}_{\text{SSL}}$ Mao et al.
(2021). In the Tables, we use “$+$B” to indicate the results obtained using
our approach.

###  3.1 Implementation details

In case of CIFAR10 and CIFAR100, WideResNet34-10 is used and for Tiny ImageNet
PreActResnet18 is used. Additionally, we evaluate on CIFAR-10 with
WideResNet28-10, WideResNet32-10, WideResNet70-16 and on CIFAR-100 with
WideResNet34-20. We use these models for a fair comparison with existing works
as these models are widely used for adversarial defense evaluation. We
evaluate against different attacks namely FGSM, PGD-10, PGD-20, CW, and AA
using $l_{\infty}$ attack with $\epsilon=8/255$. Our evaluation protocols are
similar to the protocols given in Zhang et al. (2019), Jia et al. (2022). We
would like to emphasize that we use the checkpoints from the baseline models
and perform a single epoch training using clean barycenters. Upon increasing
the number of epochs, the clean accuracy improves, however, the adversarial
accuracy becomes comparable to that of baseline and further increasing epochs
leads to subsequent drop in accuracy against adversarial samples. We use SGD
optimizer with a learning rate of 1e-4, momentum = 0.9 without any weight
decay.

In order to compute the barycenter, we set
$\rho=5e-1,\tau_{1}=1e-1,\tau_{2}=\alpha=\beta=1$ and iterations is set to
200. While one can also attack the barycenter, we give experiments for the
case where the clean image is attacked. This is because the barycenter itself
lies at an $\epsilon$ which is greater than attacker’ budget. Thus attacking
barycenter has little incentive as in that case the attacked image will lie at
an $\epsilon$ outside the given $\epsilon=8/255$ for the $l_{\infty}$ attack.

###  3.2 Comparison on CIFAR-10

In Table 1, we observe that clean performance is better for the models trained
with barycenters - TRADES${}_{\text{AWP}}+$B, LBGAT$+$B,
LAS${}_{\text{TRADES}}+$B and RST$+$B. Amongst WRN-28-10 models, RST has the
best clean performance and our method enhances it by 1%. In PGD-10, there is a
rise of 2.57%. In case of AA, there is a boost of 6.49%.

In case of WRN-34-10, LAS${}_{\text{AT}}+$B shows a huge boost of 10% under
AA. Further, LAS${}_{\text{AWP}}+$B shows the best performance under PGD-10,
PGD-20 and CW attack amongst WRN-34-10 models. Under AA it shows an
improvement of 7.71%.

Comparison with Adversarial Purification models Our RST$+$B model outperforms
Hedge${}^{*}_{\text{RST}}$ under all the cases. Against AA, our approach gives
3.1% higher accuracy compared to Hedge${}^{*}_{\text{RST}}$. We also see that
compared to TRADES${}_{\text{SSL}}$, TRADES${}_{\text{AWP}}$ has a better
performance.

Table 1: CIFAR-10. ∗ indicates that the model uses WRN-28-10. Bold font is used to indicate the best performance amongst WRN-34-10 and Red color font is used to indicate the best performance amongst WRN-28-10. Method | Clean | PGD10 | PGD20 | CW | AA
---|---|---|---|---|---
Adversarial Training
PGD-AT | 85.17 | 56.07 | 55.08 | 53.91 | 51.69
TRADES | 85.72 | 56.75 | 56.10 | 53.87 | 53.40
MART | 84.17 | 58.98 | 58.56 | 54.58 | 51.10
AWP-A2 | 87.54 | - | 59.50 | 57.42 | 54.86
RST-RWT∗ | 88.87 | - | 64.11 | 62.03 | 60.36
Adversarial Purification
TRADES${}_{\text{SSL}}$ | 82.12 | - | - | - | 60.67
Hedge${}^{*}_{\text{RST}}$ | 88.64 | - | - | 73.89 | 63.10
Adversarial and Barycentric Training
TRADES${}_{\text{AWP}}$ | 85.36 | 59.58 | 59.25 | 57.07 | 56.17
$+$B | 87.32 | 62.60 | 62.32 | 75.85 | 65.32
LBGAT | 88.22 | 56.25 | 54.60 | 54.29 | 52.23
$+$B | 88.38 | 59.28 | 58.43 | 74.61 | 61.22
LAS${}_{\text{AT}}$ | 86.23 | 57.11 | 56.41 | 55.54 | 53.58
$+$B | 86.21 | 61.08 | 60.64 | 74.09 | 63.59
LAS${}_{\text{TRADES}}$ | 85.24 | 57.66 | 57.07 | 55.45 | 54.15
$+$B | 86.15 | 60.32 | 60.03 | 73.75 | 63.43
LAS${}_{\text{AWP}}$ | 87.74 | 61.09 | 60.16 | 58.22 | 55.52
$+$B | 87.45 | 63.66 | 61.16 | 74.81 | 63.23
RST∗ | 89.69 | 63.48 | 62.51 | 61.06 | 59.71
$+$B∗ | 90.68 | 65.12 | 64.38 | 77.08 | 66.20

###  3.3 Comparison on CIFAR-100

In Table 2, we observe that our method gives a significant boost under all the
cases. In case of strong baseline LAS${}_{\text{AWP}}$, our method increases
the performance by 0.85% under clean accuracy. For PGD-20, there is a rise of
0.91%. In case of CW, there is an increase of 18.35%. In other models such as
LBGAT, we see a rise of 5.4% in clean accuracy.

Table 2: CIFAR-100 WRN-34-10. Method | Clean | PGD-10 | PGD-20 | CW
---|---|---|---|---
PGD-AT | 60.89 | 32.19 | 31.69 | 30.10
TRADES | 58.61 | 29.20 | 28.66 | 27.05
TRADES${}_{\text{AWP}}$ | 60.17 | 33.81 | 33.6 | 57.07
$+$B | 63.67 | 36.34 | 36.15 | 51.92
LBGAT | 60.64 | 35.13 | 34.53 | 30.65
$+$B | 66.04 | 36.29 | 36.01 | 52.92
LAS${}_{\text{AT}}$ | 61.8 | 33.27 | 32.83 | 31.12
$+$B | 62.45 | 36.60 | 36.17 | 49.60
LAS${}_{\text{TRADES}}$ | 60.62 | 32.82 | 32.51 | 29.51
$+$B | 62.58 | 35.22 | 34.96 | 50.99
LAS${}_{\text{AWP}}$ | 64.89 | 37.11 | 36.36 | 33.92
$+$B | 65.50 | 37.55 | 37.27 | 52.27

###  3.4 Comparison on Tiny ImageNet

We present the results in Table 3. In comparison to baselines, our method
shows significant improvement in all cases. For LAS${}_{\text{AWP}}$, our
method improves the performance under clean samples by 1.65%. In case of
PGD-50, our method shows a rise of 1.28%, and in case of CW attack, our method
almost doubles the accuracy. Under AA, LAS${}_{\text{TRADES}}$ observes a
maximum performance rise by 11.51%.

Table 3: Tiny ImageNet PreActResNet18. Method | Clean | PGD-50 | CW | AA
---|---|---|---|---
LAS${}_{\text{AT}}$ | 44.86 | 22.16 | 18.54 | 16.74
$+$B | 45.12 | 24.54 | 37.14 | 27.78
LAS${}_{\text{TRADES}}$ | 41.38 | 18.36 | 14.50 | 14.08
$+$B | 43.07 | 19.25 | 35.13 | 25.59
LAS${}_{\text{AWP}}$ | 45.26 | 23.42 | 19.88 | 18.42
$+$B | 46.91 | 24.70 | 37.93 | 27.00

###  3.5 Comparison with Curriculum based AT

In Table 4, we compare against curriculum based AT methods like CAT Cai et al.
(2018), FAT Zhang et al. (2020) and DART Wang et al. (2019a). Under FGSM,
PGD-20 and CW, our model shows a huge improvement. In FGSM, our model gives a
boost of 4.46% over FAT. In PGD-20 the boost is 10.89% and in CW there is a
rise of 24.54%. In case of clean samples, we see that the accuracy of FAT$+$B
compared to FAT is less. This may be due to the fact that FAT employs
curriculum learning in the training whereas our method does not use curriculum
learning.

Table 4: CIFAR-10 WRN-32-10. Method | Clean | FGSM | PGD-20 | CW
---|---|---|---|---
CAT | 77.43 | 57.17 | 46.06 | 42.48
DART | 85.03 | 63.53 | 48.70 | 47.27
FAT | 89.34 | 65.52 | 46.13 | 46.82
$+$B | 84.59 | 69.98 | 57.02 | 71.36

###  3.6 Analysis using Deeper and Wider Models

We use WRN-70-16 and WRN-34-20 to analyse the effect when the models get
deeper and wider. In particular, for clean samples, we can observe that the
deeper and wider models give a better boost. In CIFAR-10, WRN-70-16 gives
88.87% for clean samples which is 3.21% better than LAS${}_{\text{AWP}}$
model’ 85.66%. In contrast, for WRN-34-10, our method gives accuracy similar
to that of LAS${}_{\text{AWP}}$.

In CIFAR-100, our method boosts the performance by 8.34% under AA. In other
cases also we see that the barycenters improve the performance by a
significant margin.

Table 5: CIFAR-10 (C-10) WRN-70-16 and CIFAR-100 (C-100) WRN-34-20. Dataset | Method | Clean | FGSM | CW | AA
---|---|---|---|---|---
C-10 | LAS${}_{\text{AWP}}$ | 85.66 | 70.25 | 58.44 | 57.61
| +B | 88.87 | 74.04 | 75.40 | 62.54
C-100 | LBGAT | 62.55 | 43.16 | 31.72 | 31.92
| +B | 66.86 | 50.92 | 54.19 | 40.26

###  3.7 TSNE

In Figure 3, we show the tsne plots for MNIST testset with classes 0 and 1.
Here, we use a weak MNIST model which has only two dimensions before the
classification layer. We deliberately chose a weak model so that we can easily
show the effect in low dimensions. Though higher dimensions could be taken,
the effect cannot be easily seen due to a highly non-linear transformation
from high to low dimension of tsne. We can see that the two clusters yellow
and purple are well separated for clean and barycenters of clean images. In
case of adversarial samples, the points overlap on each other. However, when
we take barycenter of adversarial samples, we again see that the clusters are
well separated, similar to the case of clean images. Thus, it is evident that
the barycenter nullifies the effect of adversarial noise.

(a) Clean

(b) Barycenter

(c) Attacked

(d) Adv.+Bary.

Figure 3: Left to right: Plot of 2D features of Clean image, Barycenter of
clean image, Attacked image, Barycenter of adversarial image. MNIST model
obtains 51% accuracy and has only 2D feature vector before the classification
layer.

###  3.8 Mutual Information

In Table 6, we present the study of mutual information. We use
LAS${}_{\text{AT}}$ and LAS${}_{\text{TRADES}}$ on CIFAR-10. The MI is
computed using Eq 12 and Eq 14. Here we note that the MI for
LAS${}_{\text{AT}}+$B is more for training set compared to that of
LAS${}_{\text{AT}}$. This indicates that the information available about the
labels given the model parameters is high and in turn gives a better clean
accuracy. In case of adversarial samples too, we see that the MI is higher for
our case. This indicates that the model has better prediction for these
samples. Further, the MI for test set is smaller compared to training set
which is expected as the model carries more information about train set
compared to test set.

Table 6: CIFAR-10 WRN-34-10. Method | Train | Test | FGSM | CW
---|---|---|---|---
LAS${}_{\text{AT}}$ | 0.029 | 0.026 | 0.020 | 0.019
+B | 0.034 | 0.029 | 0.023 | 0.022
LAS${}_{\text{TRADES}}$ | 0.048 | 0.040 | 0.033 | 0.032
+B | 0.054 | 0.045 | 0.037 | 0.036

In Table 7, we present the results obtained using Eq 14. We observe that for
the model trained with barycenter, the MI is higher between the barycenter of
clean and adversarial samples. Thus, the model does better on barycenter of
adversarial samples compared to baseline LAS${}_{\text{AT}}$ and
TRADES${}_{\text{AWP}}$. This is consistent across FGSM, PGD-10 and CW
attacks.

Table 7: Mutual Information for CIFAR-10 WRN-34-10. Method | FGSM | PGD-10 | CW
---|---|---|---
LAS${}_{\text{AT}}$ | 0.218 | 0.198 | 0.203
+B | 0.275 | 0.241 | 0.264
LAS${}_{\text{TRADES}}$ | 0.576 | 0.554 | 0.563
+B | 0.629 | 0.572 | 0.618

###  3.9 Sensitivity to Barycenter Parameters

In Figure 4, we demonstrate the sensitivity to different parameters involved
in the computation of barycenter. In the top row, we fix the number of
iterations to 200 and $\tau_{1}=1e-1$. Here we observe that increasing
$\tau_{2}$ makes the barycenter brighter. In the second row, increasing
$\tau_{1}$ makes the barycenter darker. Decreasing iterations has a similar
effect in the last row. We see that unless there is a change of order of
magnitude, the appearance does not substantially change. Thus, our proposed
Beckman barycenter is robust with respect to the parameter settings.

Figure 4: Top row: Blue boundary represent the given image. Barycenter for
iterations = 200, $\tau_{1}=1e-1$, $\tau_{2}$ = 1, 1e-1, 1e-2, 1e-3. Second:
barycenter for iterations = 200, $\tau_{1}$=1, 1e-1, 1e-2, 1e-3, $\tau_{2}=1$.
Third: iterations = 200, 100, 50, 10, $\tau_{1}$=1e-1, $\tau_{2}$=1. The red
boundary indicates the images obtained from default settings of the parameter
which are used for all experiments.

The Figure 5 demonstrates a visualization of clean image, its barycenter,
attacked image and its barycenter. The difference between them is
imperceptible except for the black portion in the border for barycenters. This
arises due to interpolation of the images from rotation.

(a) Clean

(b) Barycenter

(c) Adversarial

(d) Adv.+Bary.

Figure 5: Illustration of barycenters from Tiny ImageNet.

##  4 Conclusion

In this work we introduce Beckman barycenter analogous to Wasserstein
barycenter. We use the Beckman OT formulation and analytically solve for the
barycenter. Defining the baycenter using Beckman OT also has the advantage
that the computational tools to obtain barycenter are well known and
efficient. This overcomes the complexity in solving Wasserstein barycenters.
Further, we show that barycenter can be used for enhancing the performance of
adversarially trained models. Our training is very efficient as we only need a
single epoch. Experimental analysis demonstrates state-of-art results against
different attacks.

## References

  * Agueh and Carlier [2011] Martial Agueh and Guillaume Carlier.  Barycenters in the wasserstein space.  SIAM Journal on Mathematical Analysis, 43(2):904–924, 2011.
  * Athalye et al. [2018] Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok.  Synthesizing robust adversarial examples.  In ICML, pages 284–293. PMLR, 2018.
  * Bai et al. [2021] Tao Bai, Jinqi Luo, Jun Zhao, Bihan Wen, and Qian Wang.  Recent advances in adversarial training for adversarial robustness.  In IJCAI, 2021.
  * Beckmann [1952] Martin Beckmann.  A continuous model of transportation.  Econometrica: Journal of the Econometric Society, pages 643–660, 1952.
  * Ben-Tal et al. [2009] Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Nemirovski.  Robust optimization, volume 28.  Princeton university press, 2009.
  * Cai et al. [2018] Qi-Zhi Cai, Min Du, Chang Liu, and Dawn Song.  Curriculum adversarial training.  arXiv preprint arXiv:1805.04807, 2018.
  * Carlini and Wagner [2017] Nicholas Carlini and David Wagner.  Towards evaluating the robustness of neural networks.  In Symposium on security and privacy, pages 39–57, 2017.
  * Carmon et al. [2019] Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang.  Unlabeled data improves adversarial robustness.  NeurIPS, 2019.
  * Chambolle and Pock [2011] Antonin Chambolle and Thomas Pock.  A first-order primal-dual algorithm for convex problems with applications to imaging.  Journal of mathematical imaging and vision, 40(1):120–145, 2011\.
  * Cheng et al. [2022] Minhao Cheng, Qi Lei, Pin-Yu Chen, Inderjit Dhillon, and Cho-Jui Hsieh.  Cat: Customized adversarial training for improved robustness.  In IJCAI, 2022.
  * Croce and Hein [2020] Francesco Croce and Matthias Hein.  Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks.  In ICML, pages 2206–2216. PMLR, 2020.
  * Cui et al. [2021] Jiequan Cui, Shu Liu, Liwei Wang, and Jiaya Jia.  Learnable boundary guided adversarial training.  In ICCV, pages 15721–15730, 2021.
  * Cuturi and Doucet [2014] Marco Cuturi and Arnaud Doucet.  Fast computation of wasserstein barycenters.  In ICML, pages 685–693. PMLR, 2014.
  * Cuturi and Peyré [2016] Marco Cuturi and Gabriel Peyré.  A smoothed dual approach for variational wasserstein problems.  SIAM Journal on Imaging Sciences, 9(1):320–343, 2016.
  * Cuturi [2013] Marco Cuturi.  Sinkhorn distances: Lightspeed computation of optimal transport.  NeurIPS, 26:2292–2300, 2013.
  * Duchi et al. [2021] John C Duchi, Peter W Glynn, and Hongseok Namkoong.  Statistics of robust optimization: A generalized empirical likelihood approach.  Mathematics of Operations Research, 46(3):946–969, 2021.
  * Fan et al. [2022] Jiaojiao Fan, Isabel Haasler, Johan Karlsson, and Yongxin Chen.  On the complexity of the optimal transport problem with graph-structured cost.  In AISTATS, pages 9147–9165. PMLR, 2022.
  * Goldstein and Osher [2009] Tom Goldstein and Stanley Osher.  The split bregman method for l1-regularized problems.  SIAM journal on imaging sciences, 2(2):323–343, 2009.
  * Goodfellow et al. [2014] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  arXiv preprint arXiv:1412.6572, 2014.
  * Houlsby et al. [2011] Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel.  Bayesian active learning for classification and preference learning.  arXiv preprint arXiv:1112.5745, 2011.
  * Jacobs et al. [2019] Matt Jacobs, Flavien Léger, Wuchen Li, and Stanley Osher.  Solving large-scale optimization problems with a convergence rate independent of grid size.  SIAM Journal on Numerical Analysis, 57(3):1100–1123, 2019.
  * Ji et al. [2019] Xu Ji, Joao F Henriques, and Andrea Vedaldi.  Invariant information clustering for unsupervised image classification and segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9865–9874, 2019.
  * Jia et al. [2022] Xiaojun Jia, Yong Zhang, Baoyuan Wu, Ke Ma, Jue Wang, and Xiaochun Cao.  Las-at: Adversarial training with learnable attack strategy.  In CVPR, pages 13398–13408, 2022.
  * Kannan et al. [2018] Harini Kannan, Alexey Kurakin, and Ian Goodfellow.  Adversarial logit pairing.  NIPS, 2018.
  * Kantorovich [1942] L Kantorovich.  On the transfer of masses (in russian).  In Doklady Akademii Nauk, volume 37, pages 227–229, 1942.
  * Kurakin et al. [2016] Alexey Kurakin, Ian Goodfellow, and Samy Bengio.  Adversarial machine learning at scale.  arXiv preprint arXiv:1611.01236, 2016.
  * Kwon et al. [2020] Yongchan Kwon, Wonyoung Kim, Joong-Ho Won, and Myunghee Cho Paik.  Principled learning method for wasserstein distributionally robust optimization with local perturbations.  In International Conference on Machine Learning, pages 5567–5576. PMLR, 2020.
  * Lacombe et al. [2021] Julien Lacombe, Julie Digne, Nicolas Courty, and Nicolas Bonneel.  Learning to generate wasserstein barycenters.  arXiv preprint arXiv:2102.12178, 2021.
  * Lee et al. [2020] John Lee, Nicholas P Bertrand, and Christopher J Rozell.  Unbalanced optimal transport regularization for imaging problems.  IEEE Transactions on Computational Imaging, 6:1219–1232, 2020.
  * Li et al. [2018] Wuchen Li, Ernest K Ryu, Stanley Osher, Wotao Yin, and Wilfrid Gangbo.  A parallel method for earth mover’s distance.  Journal of Scientific Computing, 75(1):182–197, 2018.
  * Li et al. [2021] Jincheng Li, Jiezhang Cao, Shuhai Zhang, Yanwu Xu, Jian Chen, and Mingkui Tan.  Internal wasserstein distance for adversarial attack and defense.  arXiv preprint arXiv:2103.07598, 2021.
  * Madry et al. [2017] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  arXiv preprint arXiv:1706.06083, 2017.
  * Mao et al. [2021] Chengzhi Mao, Mia Chiquier, Hao Wang, Junfeng Yang, and Carl Vondrick.  Adversarial attacks are reversible with natural supervision.  In ICCV, pages 661–671, 2021.
  * Monge [1781] Gaspard Monge.  Memoir on the theory of cuttings and embankments.  Histoire de l’Acad’e mie Royale des Sciences de Paris, 1781\.
  * Moosavi-Dezfooli et al. [2016] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard.  Deepfool: a simple and accurate method to fool deep neural networks.  In CVPR, pages 2574–2582, 2016.
  * Nie et al. [2022] Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar.  Diffusion models for adversarial purification.  ICML, 2022.
  * Papernot et al. [2016] Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami.  The limitations of deep learning in adversarial settings.  In European symposium on security and privacy, pages 372–387, 2016\.
  * Rebuffi et al. [2021] Sylvestre-Alvise Rebuffi, Sven Gowal, Dan A Calian, Florian Stimberg, Olivia Wiles, and Timothy Mann.  Fixing data augmentation to improve adversarial robustness.  arXiv preprint arXiv:2103.01946, 2021.
  * Scarf [1957] Herbert E Scarf.  A min-max solution of an inventory problem.  Rand Corporation Santa Monica, 1957.
  * Shi et al. [2021] Changhao Shi, Chester Holtz, and Gal Mishne.  Online adversarial purification based on self-supervision.  ICLR, 2021.
  * Smith and Gal [2018] Lewis Smith and Yarin Gal.  Understanding measures of uncertainty for adversarial example detection.  arXiv preprint arXiv:1803.08533, 2018.
  * Staib and Jegelka [2017] Matthew Staib and Stefanie Jegelka.  Distributionally robust deep learning as a generalization of adversarial training.  In NIPS workshop on Machine Learning and Computer Security, volume 3, page 4, 2017.
  * Subramanyam and Raj [2022] AV Subramanyam and Abhigyan Raj.  Barycentric defense.  In ICIP, pages 2276–2280, 2022.
  * Szegedy et al. [2013] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  arXiv preprint arXiv:1312.6199, 2013.
  * Theagarajan et al. [2019] Rajkumar Theagarajan, Ming Chen, Bir Bhanu, and Jing Zhang.  Shieldnets: Defending against adversarial attacks using probabilistic adversarial robustness.  In CVPR, pages 6988–6996, 2019.
  * Wang et al. [2019a] Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu.  On the convergence and robustness of adversarial training.  In International Conference on Machine Learning, 2019.
  * Wang et al. [2019b] Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu.  Improving adversarial robustness requires revisiting misclassified examples.  In International Conference on Learning Representations, 2019.
  * Wong et al. [2019] Eric Wong, Frank Schmidt, and Zico Kolter.  Wasserstein adversarial examples via projected sinkhorn iterations.  In ICML, pages 6808–6817. PMLR, 2019.
  * Wong et al. [2020] Eric Wong, Leslie Rice, and J Zico Kolter.  Fast is better than free: Revisiting adversarial training.  ICLR, 2020.
  * Wu et al. [2020] Dongxian Wu, Shu-Tao Xia, and Yisen Wang.  Adversarial weight perturbation helps robust generalization.  NeurIPS, 33:2958–2969, 2020.
  * Wu et al. [2021] Boxi Wu, Heng Pan, Li Shen, Jindong Gu, Shuai Zhao, Zhifeng Li, Deng Cai, Xiaofei He, and Wei Liu.  Attacking adversarial attacks as a defense.  arXiv preprint arXiv:2106.04938, 2021.
  * Xie et al. [2019] Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L Yuille.  Improving transferability of adversarial examples with input diversity.  In CVPR, pages 2730–2739, 2019.
  * Xu et al. [2022] Zhuoer Xu, Guanghui Zhu, Changhua Meng, Shiwen Cui, Zhenzhe Ying, Weiqiang Wang, Ming Gu, and Yihua Huang.  A2: Efficient automated attacker for boosting adversarial training.  arXiv preprint arXiv:2210.03543, 2022.
  * Yoon et al. [2021] Jongmin Yoon, Sung Ju Hwang, and Juho Lee.  Adversarial purification with score-based generative models.  In ICML, 2021.
  * Yu et al. [2022] Chaojian Yu, Bo Han, Mingming Gong, Li Shen, Shiming Ge, Bo Du, and Tongliang Liu.  Robust weight perturbation for adversarial training.  In IJCAI, 2022.
  * Zhang et al. [2019] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan.  Theoretically principled trade-off between robustness and accuracy.  In ICML, pages 7472–7482. PMLR, 2019.
  * Zhang et al. [2020] Jingfeng Zhang, Xilie Xu, Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan Kankanhalli.  Attacks which do not kill training make adversarial learning stronger.  In ICML, pages 11278–11287. PMLR, 2020.