# Strong Data Augmentation Sanitizes Poisoning and Backdoor Attacks Without an
Accuracy Tradeoff

###### Abstract

Data poisoning and backdoor attacks manipulate victim models by maliciously
modifying training data. In light of this growing threat, a recent survey of
industry professionals revealed heightened fear in the private sector
regarding data poisoning. Many previous defenses against poisoning either fail
in the face of increasingly strong attacks, or they significantly degrade
performance. However, we find that strong data augmentations, such as mixup
and CutMix, can significantly diminish the threat of poisoning and backdoor
attacks without trading off performance. We further verify the effectiveness
of this simple defense against adaptive poisoning methods, and we compare to
baselines including the popular differentially private SGD (DP-SGD) defense.
In the context of backdoors, CutMix greatly mitigates the attack while
simultaneously increasing validation accuracy by 9%.

Index Terms—  Data Poisoning, Backdoor Attacks, Adversarial Attacks,
Differential Privacy, Data Augmentation

##  1 Introduction

Machine learning models have demonstrated tremendous success in many domains
including mobile image processing [1], targeted ad placement [2], and security
services [3]. The growing availability of vast datasets has aided in this
recent success. Practitioners often rely upon data scraped from the web or
sourced from a third party [4, 5], where the security of such data can be
compromised by malicious actors. Data Poisoning attacks pose a specific threat
in which an attacker modifies a victim’s training data to achieve goals such
as targeted misclassification or performance degradation. Basic data poisoning
schemes implement backdoor triggers in training data, whereas recent works
have also demonstrated that data poisoning schemes can successfully attack
deep learning models trained on industrial-scale datasets [6, 7] without
perceptible modifications. The seriousness of these threats is acknowledged by
industrial practitioners, who recently ranked poisoning as the most worrisome
threat to their interests, in a recent study [8]. Furthermore, defenses
designed for older, less powerful poisoning strategies work by filtering out
poisons based on feature anomalies [9] but fail when models are trained from
scratch on poisoned data [10, 7]. Currently, the only method to prevent state-
of-the-art targeted poisoning relies upon differentially private SGD (DP-SGD)
and leads to a significant drop in validation accuracy [7, 11, 12].

On the other hand, data augmentation has been a boon to practitioners, aiding
in state-of-the-art performance on a variety of tasks [13, 14]. Data
augmentation can be used in many regimes, including settings where data is
sparse, to improve generalization [15]. Simple augmentations include random
crops or horizontal flips. Recently, more sophisticated augmentation schemes
have emerged that improve model performance: mixup takes pairwise convex
combinations of randomly sampled training data and uses the corresponding
convex combinations of labels. Not only does this prevent memorization of
corrupt labels and provide robustness to adversarial examples, but it has also
been shown to improve generalization [13]. Another augmentation technique,
cutout, randomly erases patches of training data [16], whereas CutMix instead
combines pairwise randomly sampled training data by taking random patches from
one image and overlaying these patches onto other images [17]. The labels are
then mixed proportionally to the area of these patches. CutMix enhances model
robustness, achieves better test accuracy, and improves localization ability
by encouraging the network to correctly classify images from a partial view.
Finally, MaxUp applies a set of data augmentation techniques (basic or
complex) to the training data and chooses the augmentation method and
parameters that achieve the worst model performance of all the techniques
[14]. By training against the most “difficult” data augmentation, MaxUp is
able to improve generalization and, in some cases, adversarial robustness.

We investigate the effects of multiple augmentation strategies on data
poisoning attacks. We find that these modern data augmentation strategies are
often more effective than previous more cumbersome defenses against poisoning
while also not sacrificing significant natural validation accuracy. Our data
augmentation defense is additionally convenient for practitioners as it
involves only a small and easy-to-implement change to standard training
pipelines.

We empirically analyze this defense both in the setting of a simple backdoor
trigger attack and in the setting of a modern imperceptible targeted data
poisoning attack [7].

##  2 Threat Model

There are many different flavors of poisoning attacks. In this paper, we focus
on two attacks from both sides of the spectrum: A simple and robust backdoor
trigger attack and a modern poisoning attack that is targeted and also clean-
label. Backdoor trigger attacks insert a specific trigger pattern (usually a
small patch, but sometimes an additive perturbation or non-rectangular symbol)
into training data. If this pattern is then added to images at test time, the
network will misclassify the test image, assigning the label that was placed
on the training images poisoned at train time. In contrast, targeted attacks
are those in which the attacker wishes to modify the victim model to
specifically misclassify a set of target images at inference time. Such
attacks are often clean-label, meaning the modified training images retain
their semantic content and are labeled correctly. Such attacks are
optimization-based, finding the most effective perturbation of training data
using gradient descent. This can make the attack especially hard to detect for
sanitization-based defenses because it does not significantly degrade clean
accuracy [6, 7]. We focus on both backdoor and clean-label attacks because of
their renewed and recent interest in the community and because they cover the
poisoning literature from two sides.

Any poisoning threat model can be formally described as a bilevel problem. Let
$F(x,\theta)$ be a neural network taking inputs $x\in{\mathbb{R}}^{n}$ with
parameters $\theta\in\mathbb{R}^{p}$. The attacker is allowed to modify $P$
samples out of $N$ total samples (where $P\ll N$) by adding perturbation
$\Delta_{i}$ to the $i^{th}$ training image. The perturbation is constrained
in the $\ell_{0}$ norm for the patch-based/backdoor trigger attack, or the
$\ell_{\infty}$-norm in the case of optimization-based attacks we consider.

Attackers wish to find $\Delta$ so that a set of $T$ target samples
$(x^{t}_{i},y^{t}_{i})_{i=1}^{T}$ are classified with the new, incorrect,
adversarial labels $y^{\text{adv}}_{i}$ after training by minimizing loss
function $\mathcal{L}$:

$\displaystyle\min_{\Delta\in\mathcal{C}}\sum_{i=1}^{T}\mathcal{L}\left(F(x^{t}_{i},\theta(\Delta)),y^{\text{adv}}_{i}\right)$
(1) $\displaystyle\operatorname*{\quad\textnormal{ s.t.
}}\theta(\Delta)\in\operatorname*{arg\,min}_{\theta}\frac{1}{N}\sum_{i=1}^{N}\mathcal{L}(F(x_{i}+\Delta_{i},\theta),y_{i}).$
(2)

In this framework, we can understand backdoor attacks as choosing the optimal
$\Delta$ directly based on a given rule (here via patch insertion onto
training images with the target label $y_{i}^{\text{adv}}$), whereas
optimization-based methods such as [7] optimize some approximation of the
(intractable) full bilevel optimization problem. Witches’ Brew [7]
approximately optimizes $\Delta$ by modifying training data so the gradient of
the training objective is aligned with the gradient of the adversarial loss
$\mathcal{L}\left(F(x^{t}_{i},\theta(\Delta)),y^{\text{adv}}_{i}\right)$,
using optimization methods based on adversarial literature [18, 19, 20].

Both backdoor attacks and targeted data poisoning attacks rely upon the
expressiveness of modern deep networks trained from scratch in order to
“gerrymander” the network’s decision boundary around specific target images
[7] — they behave normally on validation data and the chosen target is often
made into a class outlier that is surgically labeled in an adversarial way
using an erratic decision boundary. As a result, one can assume changes in the
training procedure, such as strong data augmentation, could have a significant
impact on the success of poisoning methods by imposing regularity on the
decision boundary and preventing target images from being gerrymandered into
the wrong class. Note the sensitivity of poisoning to changes in training has
been confirmed by defenses involving gradient clipping/noising, which is the
only defense shown in [7, 12] to degrade poisoning success. However, this
defense ultimately proves impractical because of the decreased natural
accuracy that comes with robustness to poisoning. Thus, we aim to bridge this
gap and develop small changes in training via data augmentation that defend
against data poisoning without impeding normal training.

##  3 Method

Mixup can be interpreted as a method for convexifying class regions in the
input space [21]. By enforcing that convex combinations of training points are
assigned convex combinations of the labels, this augmentation method
regularizes class boundaries, and removes small non-convex regions. In
particular, we are motivated by the idea of using mixup to promote the removal
of small “gerrymandered” regions in input space in which a target/poisoned
data instance is assigned an adversarial label while being surrounded by (non-
poisoned) instances with different labels.

In our experiments, we use a generalization of the mixup process from [21] for
mixture width $k$. Instead of a Beta distribution, convex coefficients are
drawn from a Dirichlet distribution $\text{Dir}[\alpha,\dots,\alpha]$ of order
$k$ with interpolation parameter $\alpha=1.$

| Poison Success (100%) | Validation Accuracy (100%) | Poison Success (10%) | Validation Accuracy (10%)
---|---|---|---|---
Baseline | 100% | 85% | 57% | 94%
mixup | 100% | 85% | 42% | 95%
CutMix | 36% | 94% | 23% | 95%
Table 1: Poison success and (clean) validation accuracy in the from-scratch
setting against backdoor attacks. The first two columns correspond to
poisoning all images in the target class, the last two columns correspond to
poisoning 10% of images in the target class. All values are averaged over 4
runs.

We implement CutMix [22] as follows: Let
$\mathcal{D}=\\{(x_{i},y_{i})\\}_{i=1}^{n}$ be the training dataset with
$x_{i}\in\mathbb{R}^{w\times h\times c}$. Let $(x_{i},y_{i})$ and
$(x_{j},y_{j})$ be two randomly sampled feature-target pairs. We randomly
generate a box $M\in\\{0,1\\}^{w\times h}$ that indicates the pixels to be
cut/pasted. All values of $M$ are defined to be one except in a box centered
at $(r_{x},r_{y})$ where they are zero. To obtain the box location, we
randomly sample the center $r_{x}\sim\text{Unif}(0,w)$ and
$r_{y}\sim\text{Unif}(0,h)$ for the first and second axes respectively. As in
mixup, we use a coefficient $\lambda\sim\text{Dir}[1,1]$ to determine the
relative contribution from each of the two randomly sampled data points. i.e.
$r_{w}=w\sqrt{1-\lambda}$ and $r_{h}=h\sqrt{1-\lambda}$ give the width and
height for the patch of zeros in $M$. The augmented image for CutMix becomes
$\tilde{x}=M\bigodot x_{i}+(1-M)\bigodot x_{j}$ with label $\tilde{y}=\lambda
y_{i}+(1-\lambda)y_{j}$ obtained by mixing initial labels proportionally to
the size of $M$. The binary operation $\bigodot$ represents element-wise
multiplication.

Cutout is similar, but the patch remains black. The cutout data point is given
by $\tilde{x}=M\bigodot x_{i}$ with label $y_{i}$.

The procedure for MaxUp is taken from [14]: For each data point $x_{i}$ in the
original training set $\mathcal{D}=\\{(x_{i},y_{i})\\}_{i=1}^{n}$, a set
$\\{\tilde{x}_{i,j}\\}_{j=1}^{m}$ of augmented data points are produced.
Learning is defined according to a modified ERM,

$\min_{\theta}\mathbb{E}_{x\sim\mathcal{D}}[\max_{j}\mathcal{L}(\tilde{x}_{i,j},\theta)].$

Fig. 1: An illustration of CutMix with poisoned data (truck) and non-poisoned
data (deer). CutMix chops up the patch, lessening its visual impact in
training data.

##  4 Experiments

| Basic Attack (%) | Adaptive Attack (%) | Validation Accuracy (%)
---|---|---|---
Baseline | 90.00 | 90.00 | 92.08
DP-SGD, $0.01$ | 77.00 | 86.00 | 91.33
DP-SGD, $0.05$ | 1.00 | 40.00 | 81.62
mixup | 45.00 | 72.00 | 91.50
mixup (4-way) | 5.00 | 55.00 | 87.45
CutMix | 75.00 | 60.00 | 91.62
cutout | 60.00 | 81.25 | 91.64
MaxUp-cutout | 5.00 | 20.00 | 86.05
Table 2: Poison success and (clean) validation error results for the from-
scratch setting against Witches’ Brew. All values are averaged over 20 runs.
Lower values in the first two columns are better. Higher numbers in the last
column are better.

###  4.1 Defending Against Backdoor Trigger Attacks

We first demonstrate the effectiveness of data augmentation at mitigating
backdoor attacks while at the same time increasing test accuracy. To establish
baselines for backdoor attacks, we train a ResNet-18 [23] on the CIFAR-10
dataset consisting of $50,000$ images in $10$ balanced classes [24]. We insert
triggers into the training set by adding $4\times 4$ patches to training
images in a randomly selected _target class_. We then evaluate on patched
images from a new _base class_ to see if the patch triggers images to be
misclassified into the target class. We perform this experiment in two
different settings: when all training images are patched in the base class and
only 10% of training images are patched in this class. These two scenarios
reflect varying access an adversary might have to their victim’s training
data.

We report the success of this attack at causing base images to be
misclassified with the target label in Table 1. Although mixup data
augmentation does not defend against the backdoor attack, CutMix dramatically
reduces the success rate of the poison attack from $100\%$ to $36\%$ while
simultaneously increasing validation accuracy by $9\%$. One explanation for
the ineffectiveness of mixup in this domain is that, under this strategy, the
base class can still be associated with the patch. On the other hand, CutMix
randomly replaces parts of the images and therefore may cut patches apart.
Moreover, CutMix improves clean validation accuracy since the network learns
relevant features from the target class rather than simply relying on the
patch.

Fig. 2: Under mixup, perturbed images are averaged with clean images.

###  4.2 Defending Against Targeted Poisoning Attacks

Fig. 3: Poison success vs. (clean) validation accuracy for non-adaptive
attacks (left) and adaptive attacks (right).

We now evaluate data augmentation as a defense against targeted poisoning
attacks. To this end, we use the state-of-the-art method, Witches’ Brew [7],
as well as an “adaptive attack” version in which attacks are generated on
networks trained with the same data augmentations as the victim. We train a
ResNet-18 on CIFAR-10, and we consider a threat model with $\ell_{\infty}$
bound $16/255$ and a budget of $1\%$. For all experiments, we consider a
baseline model trained with random horizontal flips and random crops. We then
compare modern data augmentation against differential-privacy based defenses
with a special focus on practical clean validation accuracy, that is “normal”
usability of the model. We conduct and average success over 20 runs, where
each run consists of a randomly chosen target image and a randomly chosen
adversarial label. In each run, a new model is trained from scratch on the
poisoned training set and evaluated on the target image. The lower the
attack’s success rate, the more effective the defense.

Table 2 shows that we can defend using differentially private SGD (DP-SGD)
[25] with sufficient amounts of Gaussian noise added to all training gradients
— but this comes at a tremendous costs in validation accuracy. In this setting
of an optimization-based attack, we also have to contend with another factor:
adaptive attacks. An adaptive attack threat model assumes that the attacker is
aware of the defense and can optimize their attack w.r.t to this defense. This
attack is highlighted in the second column of Table 2. In this work we adapt
WitchesBrew to advanced data augmentation by incorporating the data
augmentation into the training phase of the clean model used in the attack.
This way poisoned data is created based on a model trained to be invariant to
these augmentations. We adapt differential privacy as in [7] by a straight-
through estimate, finding it to be mitigated especially well by an adaptive
attack.

In contrast to the loss of validation accuracy incurred by DP-SGD defenses,
the defenses via data augmentation (lower rows in Table 2, blue dots in Figure
3) lose almost no validation accuracy. Nonetheless, they reduce poison success
by, for example, up to $60\%$ in the case of MaxUp based on four cutouts. This
improvement supports the conclusion of [14] that MaxUp implicitly imposes a
gradient penalty, thus smoothing decision boundaries. Also notable is that for
this case of optimization-based poisoning attacks, mixup seems to be the
optimal data augmentation among those tried which do not degrade validation
accuracy. We conjecture that this is due to the implicit linearity enforced
between data points by mixup optimization, which makes it harder for the
poisoning scheme to successfully generate outliers. We can even further
increase the defensive capabilities of mixup by considering a four-fold
mixture of images instead of the two-fold mixtures discussed in [21]. This
defense is even stronger than vanilla mixup in both cases, but the four-fold
mixtures of images leads to data modifications so stark that they negatively
influence validation accuracy.

In comparison to the backdoor trigger attacks, CutMix is less effective
against the non-adaptive attack. However, we also find that for the adaptive
setting, CutMix improves upon mixup. This is likely because the entire image
perturbation appears in the mixed image when mixup is used, enabling reliable
poisoning. In contrast, an adaptive attack on CutMix cannot a priori know the
location of the cut patch, hence the attack is impeded even if it is known
that CutMix is used.

##  5 Conclusions

Data poisoning attacks are increasingly threatening to machine learning
practitioners. By and large, defenses have not kept pace with rapidly improved
attacks, and these defenses can be impractical to implement. We demonstrate
that modern data augmentation schemes can adjust training behavior of neural
networks to mitigate from-scratch poisoning while maintaining natural accuracy
for both backdoor triggers and optimization-based attacks. We think these
results suggest the possibility of specially-designed augmentations for poison
defense, and we think this may be a fruitful direction for future research.

## References

  * [1] Eli Schwartz, Raja Giryes, and Alex M. Bronstein,  “Deepisp: Toward learning an end-to-end image processing pipeline,”  IEEE Transactions on Image Processing, vol. 28, no. 2, pp. 912–923, Feb 2019.
  * [2] Minghong Fang, Guolei Yang, Neil Zhenqiang Gong, and Jia Liu,  “Poisoning Attacks to Graph-Based Recommender Systems,”  in Proceedings of the 34th Annual Computer Security Applications Conference, San Juan, PR, USA, Dec. 2018, ACSAC ’18, pp. 381–392, Association for Computing Machinery.
  * [3] Giulio Lovisotto, Simon Eberz, and Ivan Martinovic,  “Biometric Backdoors: A Poisoning Attack Against Unsupervised Template Updating,”  ArXiv190509162 Cs, May 2019.
  * [4] Yaniv Taigman, Ming Yang, Marc’Aurelio Ranzato, and Lior Wolf,  “DeepFace: Closing the Gap to Human-Level Performance in Face Verification,”  in 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, June 2014, pp. 1701–1708, IEEE.
  * [5] Nicolas Papernot,  “A Marauder’s Map of Security and Privacy in Machine Learning,”  ArXiv181101134 Cs, Nov. 2018.
  * [6] W Ronny Huang, Jonas Geiping, Liam Fowl, Gavin Taylor, and Tom Goldstein,  “Metapoison: Practical general-purpose clean-label data poisoning,”  arXiv preprint arXiv:2004.00225, 2020.
  * [7] Jonas Geiping, Liam Fowl, W Ronny Huang, Wojciech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein,  “Witches’ brew: Industrial scale data poisoning via gradient matching,”  arXiv preprint arXiv:2009.02276, 2020.
  * [8] Ram Shankar Siva Kumar, Magnus Nyström, John Lambert, Andrew Marshall, Mario Goertzel, Andi Comissoneru, Matt Swann, and Sharon Xia,  “Adversarial Machine Learning – Industry Perspectives,”  ArXiv200205646 Cs Stat, May 2020.
  * [9] Benjamin I.P. Rubinstein, Blaine Nelson, Ling Huang, Anthony D. Joseph, Shing-hon Lau, Satish Rao, Nina Taft, and J. D. Tygar,  “ANTIDOTE: Understanding and Defending Against Poisoning of Anomaly Detectors,”  in Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement, New York, NY, USA, 2009, IMC ’09, pp. 1–14, ACM.
  * [10] Neehar Peri, Neal Gupta, W. Ronny Huang, Liam Fowl, Chen Zhu, Soheil Feizi, Tom Goldstein, and John P. Dickerson,  “Deep k-nn defense against clean-label data poisoning attacks,” 2019\.
  * [11] Avi Schwarzschild, Micah Goldblum, Arjun Gupta, John P Dickerson, and Tom Goldstein,  “Just how toxic is data poisoning? a unified benchmark for backdoor and data poisoning attacks,”  arXiv preprint arXiv:2006.12557, 2020.
  * [12] Sanghyun Hong, Varun Chandrasekaran, Yiğitcan Kaya, Tudor Dumitraş, and Nicolas Papernot,  “On the Effectiveness of Mitigating Data Poisoning Attacks with Gradient Shaping,”  ArXiv200211497 Cs, Feb. 2020.
  * [13] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz,  “mixup: Beyond empirical risk minimization,”  arXiv preprint arXiv:1710.09412, 2017.
  * [14] Chengyue Gong, Tongzheng Ren, Mao Ye, and Qiang Liu,  “Maxup: A simple way to improve generalization of neural network training,”  arXiv preprint arXiv:2002.09024, 2020.
  * [15] Renkun Ni, Micah Goldblum, Amr Sharaf, Kezhi Kong, and Tom Goldstein,  “Data augmentation for meta-learning,” 2020.
  * [16] Terrance DeVries and Graham W Taylor,  “Improved regularization of convolutional neural networks with cutout,”  arXiv preprint arXiv:1708.04552, 2017.
  * [17] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo,  “Cutmix: Regularization strategy to train strong classifiers with localizable features,”  in Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 6023–6032.
  * [18] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu,  “Towards Deep Learning Models Resistant to Adversarial Attacks,”  ArXiv170606083 Cs Stat, June 2017.
  * [19] Ping-Yeh Chiang, Jonas Geiping, Micah Goldblum, Tom Goldstein, Renkun Ni, Steven Reich, and Ali Shafahi,  “Witchcraft: Efficient pgd attacks with random step size,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 3747–3751.
  * [20] Ahmed Abdelkader, Michael J Curry, Liam Fowl, Tom Goldstein, Avi Schwarzschild, Manli Shu, Christoph Studer, and Chen Zhu,  “Headless horseman: Adversarial attacks on transfer learning models,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 3087–3091.
  * [21] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz,  “Mixup: Beyond Empirical Risk Minimization,”  ArXiv171009412 Cs Stat, Oct. 2017.
  * [22] Terrance DeVries and Graham W. Taylor,  “Improved Regularization of Convolutional Neural Networks with Cutout,”  ArXiv170804552 Cs, Aug. 2017.
  * [23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep Residual Learning for Image Recognition,”  ArXiv151203385 Cs, Dec. 2015.
  * [24] Alex Krizhevsky, Geoffrey Hinton, et al.,  “Learning multiple layers of features from tiny images,”  2009\.
  * [25] Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang,  “Deep Learning with Differential Privacy,”  in Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, Oct. 2016, CCS ’16, pp. 308–318, Association for Computing Machinery.