# Language Guided Adversarial Purification

###### Abstract

Adversarial purification using generative models demonstrates strong
adversarial defense performance. These methods are classifier and attack-
agnostic, making them versatile but often computationally intensive. Recent
strides in diffusion and score networks have improved image generation and, by
extension, adversarial purification. Another highly efficient class of
adversarial defense methods known as adversarial training requires specific
knowledge of attack vectors, forcing them to be trained extensively on
adversarial examples. To overcome these limitations, we introduce a new
framework, namely Language Guided Adversarial Purification (LGAP), utilizing
pre-trained diffusion models and caption generators to defend against
adversarial attacks. Given an input image, our method first generates a
caption, which is then used to guide the adversarial purification process
through a diffusion network. Our approach has been evaluated against strong
adversarial attacks, proving its effectiveness in enhancing adversarial
robustness. Our results indicate that LGAP outperforms most existing
adversarial defense techniques without requiring specialized network training.
This underscores the generalizability of models trained on large datasets,
highlighting a promising direction for further research.

Index Terms—  Adversarial purification, Language guidance, Diffusion

##  1 Introduction

The use of deep neural networks, especially within the realm of computer
vision, has ushered in transformative advancements in various applications.
Despite these strides, a consistent vulnerability is the susceptibility of
such models to adversarial perturbations [1]. These perturbations, often
imperceptible, can fool even the most sophisticated neural networks, causing
them to misclassify inputs. Addressing this alarming vulnerability has become
a research imperative, leading to a rapidly growing body of literature
dedicated to understanding and defending against these adversarial threats [2,
3].

Fig. 1: Illustration of LGAP. A pre-trained image-captioning model (BLIP)
generates captions for input images, providing a textual representation of the
visual content. Leveraging the generated captions, purified images are created
via the diffusion model. The red dashed lines represent the adversarial image
input, while the green dotted lines indicate the resulting purified image.

Historically, adversarial training, introduced by Goodfellow et al. [1], has
been posited as an effective defense strategy. This approach, which integrates
adversarial examples into the training phase, aims to strengthen models
against specific adversarial attacks. However, its efficacy is often limited
to the spectrum of attacks encountered during training, thereby leaving models
vulnerable to novel adversarial strategies. This constraint underscores the
necessity for alternative defensive paradigms.

Given their inherent capability to generate or transform data, generative
models have recently been explored as potential tools for adversarial
purification [4, 5, 6]. Within this domain, diffusion models have emerged as
particularly promising candidates. Recent studies, as exemplified by Nie et
al. [7] and Carlini et al. [8], have harnessed the potential of score-based
and diffusion models towards purification of adversarial samples.

Primarily, the adversarial purification techniques have focussed only on the
image modality, despite promising performance of diffusion models in multi-
modal tasks such as text-to-image generation [9]. Thus, in our work, we
investigate the impact of language towards the robustness of vision models.
Our research focuses on defensive strategy based on vision and language models
trained on large datasets. By leveraging the capabilities of such models
trained jointly on language and vision tasks, we propose a novel framework of
Language Guided Adversarial Purification (LGAP), as illustrated in Figure 1.
This novel framework, which seamlessly integrates a caption generator and a
pre-trained diffusion model with a classifier, leverages the inherent
generalisability of these models to purify an adversarial input. To the best
of our knowledge, language based adversarial purification has not been
addressed in the literature.

We conduct elaborate empirical evaluations across benchmark datasets,
including ImageNet [10], CIFAR-10 [11] and CIFAR-100 [11]. The results of
evaluation against $L_{\infty}$ norm attacks corroborate the robustness of our
framework. Notably, for the ImageNet, our method reveals better performance
compared to previous techniques.

##  2 Related Works

Diffusion models in image generation: The landscape of image generation has
been revolutionized by diffusion models. Rooted in the foundational works of
Sohl-Dickstein et al. [12] and later extended by Song et al. [13] and Ho et
al. [14], these models have exhibited unparalleled prowess in generating high-
quality image samples. Song et al. [15] further advanced this domain by
combining generative learning mechanisms with stochastic differential
equations, thereby broadening the horizon of diffusion models.

Language-image pretraining: A significant milestone in deep learning,
language-image pretraining bridges the gap between textual and visual data.
Pioneering models such as CLIP [16] and BLIP [17] have leveraged vast amounts
of text and image data to jointly train vision and language models,
demonstrating tremendous progress in multi-modal tasks.

Adversarial training: The foundational work of Madry et al. [2] established
adversarial training as a robust method for safeguarding neural networks from
known adversarial attacks. While the effectiveness of the method is well-
recognized, its scalability and adaptability have been enhanced through
inspirations from metric learning [18] and self-supervised paradigms [19].
However, the computational demands of adversarial training has spurred
research into more efficient training methods [20, 21].

Adversarial purification: Generative models have emerged as a pioneer in the
adversarial purification realm. Initial endeavors, such as those by Samangouei
et al. [4], harnessed GANs for purification. Subsequent innovations leaned on
energy-based models (EBMs) to refine the purification process using Langevin
dynamics [22]. Notably, the intersection of score networks and diffusion
models with adversarial purification has been explored recently, with
promising results against benchmark adversarial attacks [6, 7].

##  3 Proposed Method

We propose a novel defense strategy against adversarial attacks on
classification models by leveraging language guidance in diffusion models for
adversarial purification. For a clean sample $\mathbf{x}$ with label $y$, and
a target neural network $f_{\boldsymbol{\theta}}$, the adversary aims to
produce $\mathbf{x}_{\text{adv}}$ by introducing adversarial perturbations.
This results in a prediction $f_{\boldsymbol{\theta}}(\mathbf{x}_{adv})$ that
differs from the original prediction $f_{\boldsymbol{\theta}}(\mathbf{x})=y$.
The underlying premise of the proposed method is to preprocess the input
$\mathbf{x}$ through a diffusion model conditioned on a caption to remove any
adversarial perturbations before feeding it to $f_{\boldsymbol{\theta}}$. We
first discuss the caption generation followed by purification using diffusion
model.

###  3.1 Image captioning

For image captioning, we use a caption generator from BLIP [17]. BLIP has a
multi-modal encoder-decoder architecture which consists of three major
components a unimodal encoder for generating image and text embeddings, an
image-grounded text encoder that computes cross attention and self-attention
between the two encodings to give a multimodal representation of image text
pair, and an image-grounded text decoder that uses casual self-attention to
give the text caption. We use the unimodal encoder and image-grounded text
decoder to generate the captions. Given an input $\mathbf{x}$, the captions
are generated as,

$\text{{Caption}}_{\text{BLIP}}=\text{{BLIP}}(\mathbf{x}).$

We show some sample captions in Figure 2. We can see that the captions for the
clean samples (top row) contains the true label. In the second row,
adversarial samples are given and the classifier’ prediction is incorrect.
Here, truck is classified as ship. However, the BLIP caption still contains
the true label truck, though the caption is not the same as that of clean
sample. Thus, using these captions can condition the diffusion models with
true semantics which can enhance purification of the adversarial images. Next,
we discuss the diffusion based purification.

###  3.2 Diffusion purification process

Latent diffusion process In a standard diffusion model [14], the diffusion
process can be defined as:

${\bf{x}_{t}}=\sqrt{1-\beta_{t}}\cdot{\bf{x}_{t-1}}+\sqrt{\beta_{t}}\cdot\boldsymbol{\epsilon}_{t}$

where $\beta_{t}\in(0,1)$ is the variance schedule, ${\bf{x}_{t}}$ is the
noisy sample, and $\boldsymbol{\epsilon}_{t}$ is the noise at time step $t$.
In Latent Diffusion Models [9], this process is applied in latent space:

$\displaystyle\mathbf{z}_{0}=\mathcal{E}(\mathbf{x})$
$\displaystyle{\bf{z}_{t}}=\sqrt{1-\beta_{t}}\cdot{\bf{z}_{t-1}}+\sqrt{\beta_{t}}\cdot{\boldsymbol{\epsilon}_{t}}$

where $\mathbf{z}_{0}$ is the latent vector obtained from the encoder
$\mathcal{E}$ and $\mathbf{z}_{t}$ is the noisy latent vector at time step
$t$.

Reverse process in latent space In the reverse process, the aim is to recover
$\mathbf{z}_{0}$ from $\mathbf{z}_{T}$ given a sequence of noise terms
$\boldsymbol{\epsilon}_{t}$. Mathematically, this is defined as:

${\mathbf{z}_{t}}=g_{\theta}({\mathbf{z}_{t+1}},t,{\boldsymbol{\epsilon}_{t}})$

where ${g_{\theta}}$ is a parameterized model. Additionally, ${g_{\theta}}$ is
conditioned on text by augmenting ${g_{\theta}}$ architecture with cross
attention layers. Since our goal is to leverage the BLIP generated captions,
we condition the diffusion model as:

${\mathbf{z}_{t}}=g_{\theta}({\mathbf{z}_{t+1}},t,{\boldsymbol{\epsilon}_{t}},\mathbf{C})$

where $\mathbf{C}={\tau_{\theta}}(\text{{Caption}}_{\text{BLIP}})$, and
$\tau_{\theta}$ is text encoder. Since BLIP is a powerful model, the
likelihood that it correctly identifies the image is high. This gives a better
guidance to diffusion model compared to image-only case.

Final image reconstruction and training Finally, the reconstructed image
$\hat{\mathbf{x}}$ can be obtained from the reconstructed latent
representation ${\mathbf{z}}_{0}$ as,
$\hat{\mathbf{x}}=\mathcal{D}({\mathbf{z}}_{0})$, where $\mathcal{D}$ is the
decoder. Given model $f_{\boldsymbol{\theta}}$, clean image $\mathbf{x}$, its
corresponding pre-processed sample $\hat{\mathbf{x}}$ and labels $y$, we
optimize for,
$\arg\min_{\boldsymbol{\theta}}\frac{1}{n}\sum_{i=1}^{n}\mathcal{L}_{CE}(f_{\boldsymbol{\theta}}(\hat{\mathbf{x}}_{i}),{y}_{i})$,
where $\mathcal{L}_{CE}$ is the cross-entropy loss and $n$ is the number of
samples.

In contrast to adversarial training of several epochs with adversarial
samples, we only need a few epochs of finetuning with pre-processed clean
samples. Further, compared to score or diffusion-based purification, which
extensively trains these models, we only need minimal training of the
classifier.

##  4 Experiments and Results

###  4.1 Experimental settings

Datasets and network architectures: Our experimental evaluation involves three
datasets, namely CIFAR-10 [11], CIFA-100 [11] and ImageNet [10]. We utilize
the base models from RobustBench [23] model zoo for CIFAR-10 and ImageNet. For
CIFAR-100 we train the model following Yoon et al. [6]. We compare our
approach against other adversarial purification strategies on CIFAR-10,
adhering to their experimental configurations. We also evaluate our method
against preprocessor blind attacks on ImageNet. Regarding classifier
architectures, we opt for two prevalent models: ResNet-50 [24] for ImageNet
and WideResNet-28-10 [25] for CIFAR-10 and CIFAR-100. We fine-tune the
WideResNet on images generated from the diffusion network for 15 epochs. We
utilize Adam optimizer with a $10^{-3}$ learning rate. For generating
captions, we use pre-trained BLIP [17] with default hyperparameters, and for
the diffusion process, we use a pre-trained latent diffusion model from [9]
with default parameters except for the noise parameter t. We set t to 0.5 for
CIFAR-10, and CIFAR-100 and 0.1 for ImageNet. We will be releasing the code
soon.

Fig. 2: Purified samples given by LGAP. The first, second, and third rows contain clean, adversarial, and purified samples. The BLIP generated captions are given on the right, and the predicted label is on top of the image. Methods | Accuracy(%) | Architecture
---|---|---
| Natural | Robust |
Raw WideResNet | 95.80 | 0.00 | WRN-28-10
Adv. purification methods |  |  |
LGAP | 90.03 | 71.68 | WRN-28-10
Yoon et al. [6]* |  |  |
($\sigma$ = 0.1)* | 93.09 | 85.45 | WRN-28-10
($\sigma$ = 0.25)* | 86.14 | 80.24 | WRN-28-10
Hill et al. [26]* | 84.12 | 78.91 | WRN-28-10
Shi et al. [5]* | 96.93 | 63.10 | WRN-28-10
Du et al. [27]* | 48.7 | 37.5 | WRN-28-10
Grathwohl et al. [22]* | 75.5 | 23.8 | WRN-28-10
Song et al. [3]* |  |  |
Natural + PixelCNN | 82 | 61 | ResNet-62
AT + PixelCNN | 90 | 70 | ResNet-62
Adv training methods |  |  |
Madry et al. [2]* | 87.3 | 70.2 | ResNet-56
Dong et al. [28]* | 84.98 | 51.29 | ResNet-18
Table 1: Results for preprocessor blind PGD attack for CIFAR-10, within an
$L_{\infty}$ $\epsilon$-ball, where $\epsilon$ = 8/255. Data sourced from
existing literature is indicated by an asterisk *.

Adversarial attacks: We test our algorithm against preprocessor blind PGD
attacks, in which the adversary has complete visibility into the classifier
but is uninformed about the purification model. We also evaluate our algorithm
against strong adaptive attack, which involves more complex scenarios due to
our purification algorithm’s iterative nature through neural networks,
potentially leading to obfuscated gradients. To rigorously test our defense
mechanism, we use potent adaptive attacks, such as Backward Pass
Differentiable Approximation (BPDA) [29] and its variations. We experiment
with the basic form of BPDA, where the purification function is approximated
as the identity function. We further validate its robustness using Expectation
Over Time (EOT) attacks [29].

Method | Accuracy
---|---
| Natural | Robust
LGAP | 58.71 | 39.82
Yoon et al.[6]* | 77.83 | 43.21
Adversarial training methods |  |
Madry et al.[2]* | 59.58 | 25.47
Li et al.[30]* | 61.01 | 28.88
Table 2: Preprocessor blind PGD attack for CIFAR-100, $\epsilon$ = $8$/$255$. Data sourced from existing literature is indicated by an asterisk *. Methods | Accuracy(%) | Architecture
---|---|---
Attacks | Natural | Robust |
Adv. Purification Methods |  |  |
LGAP | 90.30 |  |
BPDA 40+EOT |  | 44.96 | WRN-28-10
BPDA |  | 62.63 | WRN-28-10
Yoon et al. [6] ($\sigma$ = 0.25)* |  |  |
BPDA 40+EOT | 86.14 | 70.01 | WRN-28-10
Yoon et al. [6] ($\sigma$ = 0.0)* |  |  |
BPDA | 90.60 | 76.87 | WRN-28-10
Hill et al. [26]* |  |  |
BPDA 50+EOT | 84.12 | 54.9 | WRN-28-10
Song et al. [3]* |  |  |
BPDA | 95.00 | 9 | ResNet-62
Yang et al. [31]* |  |  |
BPDA 1000 | 94.8 | 40.8 | ResNet-18
(+AT. p = 0.4 ->0.6) | 88.7 | 55.1 | WRN-28-10
(+AT. p = 0.6 ->0.8) | 91 | 52.9 | WRN-28-10
Approx. Input | 89.4 | 41.5 | ResNet-18
Shi et al. [5]* |  |  |
Classifier PGD 20 | 91.89 | 53.58 | WRN-28-10
Adv. training methods |  |  |
Madry et al. [2]* | 87.3 | 45.8 | ResNet-18
Carmon et al. [32]* | 89.67 | 63.1 | WRN-28-10
Table 3: Adaptive attacks for CIFAR-10, $\epsilon$ = 8/255.

###  4.2 Comparison with state of the art

The results for preprocessor blind setup shown in Table 1 on CIFAR10 show that
our method gives better robust performance than most previous methods,
specifically adversarial training methods, while maintaining comparable
performance on natural images. Our method achieves a robust accuracy of
71.68%, which clearly outperforms seven out of ten methods including two
adversarial defense methods and five adversarial purification methods. A
snapshot of adversarial samples and their corresponding purified images is
given in Figure 2.

We further extend our evaluation to the CIFAR-100 dataset, with the robust
performance comparisons listed in Table 2. Unlike other methods, such as the
one by Yoon et al., which demands training a score network and noise parameter
tuning, our method, LGAP delivers competitive results with substantially lower
computational overhead.

Table 3 shows robust accuracy of our method against BPDA attack for CIFAR-10.
Our method outperforms most previous techniques of adversarial purification
and adversarial training. The gap in accuracy between our method and some
recent techniques remains owing to other methods training the purification
model on CIFAR10. Yoon et al. and Hill et al. which show better robust
performance, train diffusion and EBM networks on CIFAR-10 for 200,000
iterations [6, 26]. Whereas our method requires no such training.

Method | Accuracy | Architecture
---|---|---
Attacks | Natural | Robust |
Undefended | 76.76 | 0 | ResNet-50
LGAP | 69.09 |  | ResNet-50
AA |  | 57.12 |
BPDA-40 |  | 45.31 |
PGD-10 |  | 52.73 |
Adv training methods |  |  |
Salman et al. [33]* | 64.02 |  | ResNet-50
AA |  | 34.96 |
Wong et al. [21]* | 55.62 |  | ResNet-50
AA |  | 26.24 |
Table 4: Preprocessor blind attacks for ImageNet, $\epsilon$ = $4$/$255$.

Table 4 shows the robust performance of our method for ImageNet. Due to the
high computational cost of some attacks, we evaluate on a fixed set of 2048 as
robust accuracy does not change much on the sampled subset compared to the
whole subset [7]. We can see that even against strong adaptive attack such as
BPDA-40, LGAP attains an accuracy of 45.31% demonstrating the efficacy of the
proposed method. The enhanced performance of the method can be attributed to
the diffusion model trained on ImageNet. Similarly, a diffusion model trained
on CIFAR-10 is expected to yield improved results when applied to CIFAR-10
classification.

##  5 Conclusion

Our method addressed key limitations in adversarial defense by introducing a
language-guided purification approach. Unlike traditional methods, which
require extensive computational resources and specific attack knowledge, our
method leverages pre-trained diffusion models and caption generators. This
reduces computational overhead and enhances scalability. Empirical tests show
our approach is robust, outperforming conventional methods in several metrics,
despite trailing some diffusion-based methods. Notably, this performance is
achieved with minimal training and do not require adversarial samples or
training the score or diffusion networks, thus broadening the method’s
applicability and setting a new efficiency standard. Our method underscores
the generalizability of deep learning models trained on large datasets,
pointing to avenues for future research, especially in model generalizability.

## References

  * [1] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy,  “Explaining and harnessing adversarial examples,”  in ICLR, 2015.
  * [2] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu,  “Towards deep learning models resistant to adversarial attacks,”  in ICLR, 2018.
  * [3] Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman,  “Pixeldefend: Leveraging generative models to understand and defend against adversarial examples,”  in ICLR, 2018.
  * [4] Pouya Samangouei, Maya Kabkab, and Rama Chellappa,  “Defense-gan: Protecting classifiers against adversarial attacks using generative models,”  in ICLR, 2018.
  * [5] Changhao Shi, Chester Holtz, and Gal Mishne,  “Online adversarial purification based on self-supervised learning,”  in ICLR, 2020.
  * [6] Jongmin Yoon, Sung Ju Hwang, and Juho Lee,  “Adversarial purification with score-based generative models,”  in ICML, 2021.
  * [7] Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Animashree Anandkumar,  “Diffusion models for adversarial purification,”  in ICML, 2022.
  * [8] Nicholas Carlini, Florian Tramer, Krishnamurthy Dj Dvijotham, Leslie Rice, Mingjie Sun, and J Zico Kolter,  “(certified!!) adversarial robustness for free!,”  in ICLR, 2022.
  * [9] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer,  “High-resolution image synthesis with latent diffusion models,”  in CVPR, 2022.
  * [10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei,  “Imagenet: A large-scale hierarchical image database,”  in CVPR, 2009.
  * [11] Alex Krizhevsky, Geoffrey Hinton, et al.,  “Learning multiple layers of features from tiny images,”  2009\.
  * [12] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli,  “Deep unsupervised learning using nonequilibrium thermodynamics,”  in ICML, 2015, pp. 2256–2265.
  * [13] Yang Song and Stefano Ermon,  “Generative modeling by estimating gradients of the data distribution,”  NeurIPS, vol. 32, 2019.
  * [14] Jonathan Ho, Ajay Jain, and Pieter Abbeel,  “Denoising diffusion probabilistic models,”  NeurIPS, vol. 33, pp. 6840–6851, 2020.
  * [15] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole,  “Score-based generative modeling through stochastic differential equations,”  in ICLR, 2020.
  * [16] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.,  “Learning transferable visual models from natural language supervision,”  in ICML, 2021.
  * [17] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi,  “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,”  in ICML, 2022.
  * [18] Chengzhi Mao, Ziyuan Zhong, Junfeng Yang, Carl Vondrick, and Baishakhi Ray,  “Metric learning for adversarial robustness,”  NeurIPS, vol. 32, 2019.
  * [19] Kejiang Chen, Yuefeng Chen, Hang Zhou, Xiaofeng Mao, Yuhong Li, Yuan He, Hui Xue, Weiming Zhang, and Nenghai Yu,  “Self-supervised adversarial training,”  in ICASSP, 2020.
  * [20] Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein,  “Adversarial training for free!,”  NeurIPS, vol. 32, 2019.
  * [21] Eric Wong, Leslie Rice, and J Zico Kolter,  “Fast is better than free: Revisiting adversarial training,”  in ICLR, 2019.
  * [22] Will Grathwohl, Kuan-Chieh Wang, Joern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky,  “Your classifier is secretly an energy based model and you should treat it like one,”  in ICLR, 2019.
  * [23] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein,  “Robustbench: a standardized adversarial robustness benchmark,”  arXiv preprint arXiv:2010.09670, 2020.
  * [24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in CVPR, 2016.
  * [25] Sergey Zagoruyko and Nikos Komodakis,  “Wide residual networks,”  arXiv preprint arXiv:1605.07146, 2016.
  * [26] Mitch Hill, Jonathan Mitchell, and Song-Chun Zhu,  “Stochastic security: Adversarial defense using long-run dynamics of energy-based models,”  in ICLR, 2021.
  * [27] Yilun Du and Igor Mordatch,  “Implicit generation and modeling with energy based models,”  in NeurIPS, 2019.
  * [28] Junhao Dong, Seyed-Mohsen Moosavi-Dezfooli, Jianhuang Lai, and Xiaohua Xie,  “The enemy of my enemy is my friend: Exploring inverse adversaries for improving adversarial training,”  in CVPR, 2023.
  * [29] Anish Athalye, Nicholas Carlini, and David Wagner,  “Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples,”  in ICML, 2018.
  * [30] Xiang Li and Shihao Ji,  “Defense-VAE: A fast and accurate defense against adversarial attacks,”  in Machine Learning and Knowledge Discovery in Databases, Peggy Cellier and Kurt Driessens, Eds. pp. 191–207, Springer International Publishing.
  * [31] Yuzhe Yang, Guo Zhang, Dina Katabi, and Zhi Xu,  “Me-net: Towards effective adversarial robustness with matrix estimation,”  in ICML, 2019.
  * [32] Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang,  “Unlabeled data improves adversarial robustness,”  NeurIPS, 2019.
  * [33] Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry,  “Do adversarially robust imagenet models transfer better?,”  NeurIPS, 2020.