# Limits of Machine Learning
for Automatic Vulnerability Detection

Niklas Risse  Max-Planck-Institute for Security and PrivacyBochumGermany and
Marcel Böhme  Max-Planck-Institute for Security and PrivacyBochumGermany

###### Abstract.

Recent results of machine learning for automatic vulnerability detection have
been very promising indeed: Given only the source code of a function $f$,
models trained by machine learning techniques can decide if $f$ contains a
security flaw with up to 70% accuracy.

But how do we know that these results are general and not specific to the
datasets? To study this question, researchers proposed to _amplify_ the
testing set by injecting semantic preserving changes and found that the
model’s accuracy significantly drops. In other words, the model uses _some_
unrelated features during classification. In order to increase the robustness
of the model, researchers proposed to train on amplified training data, and
indeed model accuracy increased to previous levels.

In this paper, we replicate and continue this investigation, and provide an
actionable model benchmarking methodology to help researchers better evaluate
advances in machine learning for vulnerability detection. Specifically, we
propose (i) a cross validation algorithm, where a semantic preserving
transformation is applied during the amplification of either the training set
or the testing set, and (ii) the amplification of the testing set with code
snippets where the vulnerabilities are fixed. Using 11 transformations, 3 ML
techniques, and 2 datasets, we find that the improved robustness only applies
to the specific transformations used during training data amplification. In
other words, the robustified models still rely on unrelated features for
predicting the vulnerabilities in the testing data. Additionally, we find that
the trained models are unable to generalize to the modified setting which
requires to distinguish vulnerable functions from their patches.

##  1\. Introduction

Recently a number of different publications have reported high scores on
vulnerability detection benchmarks using machine learning (ML) techniques
(Feng et al., 2020; Phan et al., 2021; Hanif and Maffeis, 2022; Ahmad et al.,
2021a; Fu and Tantithamthavorn, 2022; Buratti et al., 2020). The resulting
models seem to outperform methods from other research directions (e.g. static
analysis), even without requiring any hard-coded knowledge of program
semantics or computational models. So, does this mean that the problem of
detecting security vulnerabilities in software is solved? Are these models
actually able to detect security vulnerabilities, or do the reported scores
provide a false sense of security?

In order to be safely applied to real-world software systems, ML techniques
for vulnerability detection need to be reliable and trustworthy. This is why
researchers have tried to explore the capabilities and limits of machine
learning techniques in ways that go beyond simple evaluations on benchmark
testing sets.

One approach to finding these limits is to create situations in which ML
techniques fail to meet expectations. For example, it is possible to apply
small semantic preserving changes to the input programs of a state-of-the-art
model and then measure, whether the model changes its predictions and whether
it still performs well. Examples for such transformations are identifier
renaming (Yefet et al., 2020; Henkel et al., 2022; Zhang et al., 2020; Yang et
al., 2022; Zhang et al., 2022), insertion of unexecuted statements (Yefet et
al., 2020; Zhang et al., 2022; Henkel et al., 2022; Srikant et al., 2021) or
replacement of code elements with equivalent elements (Li et al., 2022a;
Applis et al., 2021). The impact of applying these transformations to testing
data has been explored for many different tasks in software engineering, and
the results seems to be clear: ML techniques lack robustness against semantic
preserving transformations (Yang et al., 2022; Rabin et al., 2021; Applis et
al., 2021; Henkel et al., 2022; Srikant et al., 2021; Yefet et al., 2020;
Bielik and Vechev, 2020; Zhang et al., 2020; Li et al., 2022b; Zhang et al.,
2022).

A common strategy to address the robustness problem is training data
amplification; applying the same or similar transformations to the training
dataset. Many of the works that reported the lack of robustness of ML models
when trained on untransformed data also investigated training data
amplification using their respective methods (Yang et al., 2022; Henkel et
al., 2022; Srikant et al., 2021; Yefet et al., 2020; Bielik and Vechev, 2020;
Zhang et al., 2020; Li et al., 2022b; Zhang et al., 2022). They found a
restoration or at least improvement towards the initial high performance. But
does training data amplification actually improve the ability of these models
to detect vulnerabilities, or are they just over-fitting to a different set of
data?

We contribute to answering this question by proposing a general benchmarking
methodology that can be used to evaluate the capabilities of machine learning
models for vulnerability detection by using data amplification. First, we
propose Algorithm 1, in which a selected semantic preserving transformation is
applied to the training dataset of a model, and a _different_ transformation
is applied to the testing dataset. When repeated for all possible pairs out of
a set of transformations, the resulting scores provide a measure of over-
fitting to the specific semantic preserving transformations that were used
during training data amplification. Second, we propose Algorithm 2, in which a
target model that was trained on an amplified training dataset is evaluated on
a testing dataset, that contains both vulnerable programs and their respective
fixes. The results provide a measure of the model’s ability to generalize to a
slightly modified vulnerability detection setting and the effects of training
data amplification on this ability.

In addition to the general methodology, we present the results of an empirical
study, in which we implement and validate the proposed algorithms on three
state-of-the-art ML techniques for vulnerability detection. For Algorithm 1,
we implemented 11 different semantic preserving transformations (see Table 1)
for data amplification and evaluated the trained models using two different
vulnerability detection datasets. As expected, we find a strong benefit of
training data amplification (59.8% and 54.1% average restoration of
performance for the two datasets) when the transformations applied to training
and testing dataset are the same. However, we find no improvement in
performance when the transformations applied to training and testing dataset
are different. In fact, we even find an additional 35.7% and 97.4% average
decrease in performance for the two datasets. In other words, state-of-the-art
ML techniques severely over-fit to the specific label-unrelated features
introduced by training data amplification. The improved robustness only
applies to the specific type of transformations used during training.

For Algorithm 2, we introduce a new dataset, which we call _VulnPatchPairs_.
VulnPatchPairs contains 26.2k C functions and is derived from the
CodeXGLUE/Devign vulnerability detection dataset (Zhou et al., 2019). Exactly
half (13.1k) of the 26.2k functions in VulnPatchPairs contain vulnerabilities.
The other 13.1k functions are patched versions of the first 13.1k functions
(see Section 4.3 for details). Again, we use the 11 transformations that we
implemented and three ML techniques to train $11\cdot 3=33$ models, and
evaluate each of them on the testing subset of VulnPatchPairs. To our
surprise, no trained model is able to distinguish between the vulnerable
functions and their fixed counterparts. On average, the performance is only
1.8% better than random guessing. The trained models are unable to generalize
from a standard vulnerability detection dataset to the slightly modified
setting.

In summary, this paper makes the following contributions:

  * $\star$

We present a general methodology that can be used to evaluate ML models for
vulnerability detection using data amplification.

  * $\star$

We show empirically, that the robustness gained by data amplification only
applies to the specific transformations used during training, and that
robustified models over-fit to the unrelated features introduced by semantic
preserving transformations.

  * $\star$

We introduce _VulnPatchPairs_ , a new dataset that contains vulnerable C
function and the corresponding patched versions of the same functions.

  * $\star$

We demonstrate, that three state-of-the-art ML techniques for vulnerability
detection are not able to distinguish between the vulnerable and patched
functions in VulnPatchPairs and that training data amplification leads to no
significant improvement.

  * $\star$

We publish all of our datasets, code, and results for reproducibility. They
are available at
https://github.com/niklasrisse/LimitsOfML4Vuln.

##  2\. Related Work

One of the main tools to study the robustness of deep learning based
vulnerability detection systems are semantic preserving transformations of
code. Previous work (Yang et al., 2022; Rabin et al., 2021; Applis et al.,
2021; Henkel et al., 2022; Srikant et al., 2021; Yefet et al., 2020; Bielik
and Vechev, 2020; Zhang et al., 2020; Li et al., 2022b; Zhang et al., 2022; Li
et al., 2021, 2022a) proposed methods to generate semantic preserving
transformations for source code datasets. The effects of training data
amplification using semantic preserving transformations of code have been
studied for many software engineering tasks, such as code summarization
(Henkel et al., 2022; Srikant et al., 2021; Yefet et al., 2020), type
prediction (Bielik and Vechev, 2020), code functionality classification (Zhang
et al., 2020) and vulnerability detection (Li et al., 2022b, 2021; Zhang et
al., 2022; Yang et al., 2022). A common finding in all of these publications
is that training data amplification using a specific type of semantic
preserving transformation leads to improved performance on testing sets that
have been amplified the same way.

Some of the works (Henkel et al., 2022; Yang et al., 2022; Bielik and Vechev,
2020; Srikant et al., 2021) go a step further; they evaluate the effects of
amplifying the training dataset using their proposed method on a testing
dataset that has been amplified using a slightly different but related type of
transformation. For example, Henkel et al. (Henkel et al., 2022) apply their
gradient-based resolution of transformation sketches to training data, e.g.
they select a local variable for renaming and determine the name in such a way
that the loss of the target model is maximized, and evaluate their method on a
testing dataset where variables are randomly renamed. Similarly, Yang et al.
(Yang et al., 2022) apply their method for variable renaming to a training
dataset and evaluate using the method proposed by Zhang et al. (Zhang et al.,
2020), which also renames variables but determines the new names in a
different way. All of these works find an improved robustness when the
training dataset is amplified in a similar way than the testing dataset.

A recent publication (Zhang et al., 2022) investigates the benefits of data
amplification when the transformations applied to training and testing dataset
are of completely different type. The authors propose two methods, one for
identifier renaming and one for statement insertion, and explore robustness
when using one for training and the other for testing data amplification. They
find lower performance compared to when training and testing dataset were
amplified using the same type of transformation. However, two questions remain
unanswered in this work: Is this preliminary result specific to the two
transformations proposed by the authors or does it hold up in general, i.e.
across different transformations, models and datasets? What are the effects of
training data amplification on the general capabilities of the model to detect
vulnerabilities? Our work aims to answer both open questions by proposing a
methodology that generalizes the initial result proposed by Zhang et al.
(Zhang et al., 2022), and by carrying out a thorough empirical study that
considers a diverse set of 11 different transformations, three state-of-the-
art ML techniques and two high-quality datasets.

Some of the semantic preserving transformations proposed in the literature
(Yefet et al., 2020; Henkel et al., 2022; Srikant et al., 2021) are computed
during the training of the respective models with the goal to optimize an
additional training objective, which is designed to produce models that are
robust against small transformations of their input (adversarial training).
The difference of adversarial training to standard training data amplification
is, that in adversarial training the transformations are computed during
training (instead of before the training) and in such a way, that the loss of
the model for each training sample is maximized. We use both simple semantic
preserving transformations and adversarial training in our experiments.

In order to evaluate the general capabilities of ML techniques for
vulnerability detection and the benefits of data amplification on them, we
collected a new dataset (VulnPatchPairs), which contains both vulnerable
functions and their respective patches. The collection of a pairwise
vulnerability-patch dataset has been proposed by previous work (Ponta et al.,
2019; Bhandari et al., 2021; Bui et al., 2022), e.g. for the research field of
automated fixing. However, to the best of our knowledge, we are the first use
such a dataset to evaluate the general capabilities of ML techniques for
vulnerability detection and the effects of data amplification using semantic
preserving transformations.

##  3\. Methodology

We propose a novel benchmarking methodology to help researchers better
evaluate advances in machine learning for vulnerability detection. The
methodology consists of two parts, Algorithm 1 (A1) and Algorithm 2 (A2).

###  3.1. Label-aware Transformations of Code

A central component of both algorithms are label-aware transformations of
code, and the expectations for vulnerability detection models that emerge from
using them for data amplification.

A code transformation $t:\mathcal{C}\to\mathcal{C}$ is a function that maps
from and to a space $\mathcal{C}$, which represents all possible code snippets
$c\in\mathcal{C}$ in a given programming language. Lets assume we have an
oracle function $g:\mathcal{C}\to\\{0,1\\}$, which maps from the space of code
snippets $\mathcal{C}$ to either 0 or 1. The oracle function $g$ represents
the ground truth, i.e. it shows whether a code snippet $c$ does (1) or does
not (0) contain a security vulnerability. For a given code snippet dataset
$CD\subset\mathcal{C}$, a code transformation $t$ can be characterized by its
effect on $g(t(c))$ $\forall$ $c\in CD$:

Algorithm 1 Detecting Over-fitting to Code Changes

0: Semantic Preserving Transformations $T:=\\{t_{1},...,t_{N}\\}$
Training Dataset Tr
Testing Dataset Te
Machine Learning Technique MLT
Performance Metric M

1: $f[Tr]=$_train_model_ $(Tr,MLT)$

2: $s[f[Tr],Te]=$_evaluate_model_ $(f[Tr],Te,M)$

3: for each $t_{k}\in T$ do

4: $Te_{k}=t_{k}(Te)$  _// testing data amplification_

5: $s[f[Tr],Te_{k}]=$_evaluate_model_ $(f[Tr],Te_{k},M)$

6: $e[Tr,Te_{k}]=s[f[Tr],Te_{k}]-s[f[Tr],Te]$

7: $Tr_{k}=t_{k}(Tr)$  _// training data amplification_

8: $f[Tr_{k}]=$_train_model_ $(Tr_{k},MLT)$

9: $s[f[Tr_{k}],Te_{k}]=$_evaluate_model_ $(f[Tr_{k}],Te_{k},M)$

10: $e[Tr_{k},Te_{k}]=s[f[Tr_{k}],Te_{k}]-s[f[Tr],Te]$

11: for each $t_{j\neq k}\in T$ do

12: $Te_{j}=t_{j}(Te)$  _// testing data amplification_

13: $s[f[Tr_{k}],Te_{j}]=$_evaluate_model_ $(f[Tr_{k}],Te_{j},M)$

14: $e[Tr_{k},Te_{j}]=s[f[Tr_{k}],Te_{j}]-s[f[Tr],Te]$

15: end for

16: end for

16: $o_{1}=(\sum_{k}e[Tr,Te_{k}])/N$
$o_{2}=(\sum_{k}e[Tr_{k},Te_{k}])/N$
$o_{3}=(\sum_{k}\sum_{j\neq k}e[Tr_{k},Te_{j}])/(N(N-1))$

Semantic Preserving Transformation. We call a transformation $t_{p}$ semantic
preserving w.r.t CD, if the changes introduced by applying it do not affect
the ground truth vulnerability label, $g(c)=g(t_{p}(c))$ $\forall$ $c\in CD$.
Figure 1 shows a toy example of a semantic preserving transformation applied
to a simple code snippet.

Label Inverting Transformation. We call a transformation $t_{d}$ label
inverting w.r.t CD, if the changes introduced by applying it change the ground
truth vulnerability label, $g(c)\neq g(t_{d}(c))$ $\forall$ $c\in CD$. In
other words, a label inverting transformation either adds or removes a
vulnerability from a code snippet.

In general, we expect a vulnerability detection model to correctly predict,
whether a given code snippet contains a security vulnerability, independent of
any semantic preserving or label inverting transformations that have been
previously applied to the code snippet. Specifically, we can formulate the
following expectations:

  1. (1)

If we change a code snippet without affecting the vulnerability label
(semantic preserving transformation), we expect a vulnerability detection tool
to compute the same correct prediction as before applying the change.

  2. (2)

If we add or remove a vulnerability from a code snippet (label inverting
transformation), we expect a vulnerability detection tool to still deliver a
correct prediction, or i.e. we expect it to change its prediction with the
ground truth label of the code snippet.

In the following sections we present two algorithms, which allow to evaluate
machine learning techniques for vulnerability detection using these two
expectations.

###  3.2. A1: Detecting Over-fitting to Code Changes

The goal of Algorithm 1 is to measure, whether the robustness gained by data
amplification generalizes beyond the specific amplifications applied during
training, and whether machine learning techniques for vulnerability detection
still over-fit to amplifications of their training data that are unrelated to
the respective vulnerability labels.

What are the inputs? The inputs of Algorithm 1 are a set of different semantic
preserving transformations $T:=\\{t_{1},...,t_{N}\\}$, a training dataset
$Tr$, a testing dataset $Te$, a machine learning technique for vulnerability
detection MLT, and a performance metric M. The training and testing datasets
Tr and Te consist of code-label pairs $(c_{i},v_{i})$, with
$c_{i}\in\mathcal{C}$ representing code snippets and $v_{i}\in\\{0,1\\}$
representing labels that indicate the absence (0) or presence (1) of security
vulnerabilities in the respective code snippets. The machine learning
technique MLT can utilize the training dataset Tr to train a model
$f:\mathcal{C}\to\\{0,1\\}$, which maps from the space of code snippets
$\mathcal{C}$ to either 1 (vulnerability) or 0 (no vulnerability). The
performance metric M quantifies and aggregates the performance of a trained
machine learning model $f$ on a testing dataset Te into a single number
between 0 (bad) and 1 (perfect).

What is computed? Algorithm 1 computes the average effects of (a) amplifying
the testing data of the selected ML technique using transformations $t_{k}\in
T$ (Output $o_{1}$), (b) using the same transformations to amplify the
training data (Output $o_{2}$), and (c) using different transformations to
amplify the training data (Output $o_{3}$).

In lines 4-6, Algorithm 1 computes the effect of amplifying the testing
dataset Te with the transformation $t_{k}$ on the performance of the model
$f[Tr]$. The result is $e[Tr,Te_{k}]$, the absolute difference between the
scores of $f[Tr]$ on the clean testing dataset $Te$ and the amplified testing
dataset $Te_{k}$. In other words, $e[Tr,Te_{k}]$ quantifies how many points in
score are lost if we amplify the testing dataset with transformation $t_{k}$.
The output $o_{1}$ aggregates this intermediate result over all
transformations $t_{k}\in T$.

⬇

1#include <stdio.h>

2int main() {

3 printf("Hello, World!");

4

5 return 0;

6}

(a) Code Snippet

⬇

1#include <stdio.h>

2int main() {

3 printf("Hello, World!");

⬇

4#include <stdio.h>

⬇

5#include <stdio.h>

6int main() {

7 printf("Hello, World!");

8 char x;

9 return 0;

10}

(b) Transformed Code Snippet

Figure 1. Example of a semantic preserving transformation. The change (orange)
has no effect on the vulnerability label (no vulnerability).

In lines 7-10, Algorithm 1 goes a step further and computes the effect of both
amplifying the training dataset $Tr$ and the testing dataset $Te$ using the
same transformation $t_{k}$. The result is $e[Tr_{k},Te_{k}]$, the absolute
difference between scores of $f[Tr_{k}]$ on the amplified testing $Te_{k}$ and
$f[Tr]$ on the testing dataset $Te$. In other words, $e[Tr_{k},Te_{k}]$
quantifies how many points in score are lost if we amplify both the training
and the testing dataset with transformation $t_{k}$. The output $o_{2}$
aggregates this intermediate result over all transformations $t_{k}\in T$.

In lines 12-14, the algorithm computes the effect of amplifying the testing
dataset using a _different_ transformation $t_{j}$ than for the training
dataset. The result is $e[Tr_{k},Te_{j}]$, the absolute difference between
scores of $f[Tr]$ on the testing dataset $Te$ and $f[Tr_{k}]$ on the amplified
testing $Te_{j}$. In other words, $e[Tr_{k},Te_{j}]$ quantifies how many
points in score are lost if we amplify the training and the testing dataset
with different transformations $t_{k}$ and $t_{j}$. The output $o_{3}$
aggregates this intermediate result over all transformations $t_{k}\in
T,t_{j\neq k}\in T$.

Algorithm 2 Distinguish between Vulnerability and Patch

0: Semantic Preserving Transformations $T:=\\{t_{1},...,t_{N}\\}$
Training Dataset Tr
Testing Dataset Te
Vulnerability-Patch Testing Dataset VPT
Machine Learning Technique MLT
Performance Metric M

1: $f[Tr]=$_train_model_ $(Tr,MLT)$

2: $s[f[Tr],VPT]=$_evaluate_model_ $(f[Tr],VPT,M)$

3: for each $t_{k}\in T$ do

4: $Tr_{k}=t_{k}(Tr)$  _// training data amplification_

5: $f[Tr_{k}]=$_train_model_ $(Tr_{k},MLT)$

6: $s[f[Tr_{k}],VPT]=$_evaluate_model_ $(f[Tr_{k}],VPT,M)$

7: $e[Tr_{k},VPT]=s[f[Tr_{k}],VPT]-s[f[Tr],VPT]$

8: end for

8: $o_{1}=s[f[Tr],VPT]$
$o_{2}=(\sum_{k}s[f[Tr_{k}],VPT])/N$
$o_{3}=(\sum_{k}e[Tr_{k},VPT])/N$

How can the results be used? Using this algorithm, researchers can effectively
evaluate new machine learning techniques for vulnerability detection.
Specifically, for a selected technique researchers can answer the following
questions:

  1. (1)

How much does the performance of my technique decrease if we amplify the code
snippets for testing without affecting the vulnerability labels? Answer: On
average, the performance does change by $o_{1}$ points.

  2. (2)

How much performance of my technique can be restored, if we amplify the
training code snippets in a similar way than the testing code snippets?
Answer: On average, $o_{2}-o_{1}$ of the initial decrease can be restored.

  3. (3)

How much performance of my technique can be restored, if we amplify the
training code snippets in a different way than the testing code snippets?
Answer: On average, $o_{3}-o_{1}$ of the initial decrease can be restored.

  4. (4)

Does my technique over-fit to specific amplifications of the training data
that are unrelated to the respective vulnerability labels? Answer: If
$o_{2}>>o_{3}$: Yes, otherwise No.

###  3.3. A2: Distinguish between Vulnerability and Patch

The goal of Algorithm 2 is to evaluate, whether machine learning techniques
for vulnerability detection are able to generalize from their typical training
data to a modified setting, which requires to distinguish security
vulnerabilities from their patches.

What are the inputs? In addition to the inputs of Algorithm 1, Algorithm 2
requires a special Vulnerability-Patch testing dataset VPT. VPT also consists
of code snippets $c_{i}\in\mathcal{C}$ and vulnerability labels
$v_{i}\in\\{0,1\\}$, but for every code snippet $c_{j}$ with label $v_{j}=0$,
it also contains a snippet $c_{l\neq j}$ with $v_{l}=1$ which represents the
patched version of $c_{j}$. The relationship between a code snippet and its
patched version can be characterized as a label inverting transformation
$t_{PATCH}:\mathcal{C}\to\mathcal{C}$, which maps code snippets $c_{j}$ to
their patched versions $c_{l}$.

What is computed? Algorithm 2 uses the Vulnerability-Patch testing dataset VPT
to quantify the ability of the selected ML technique to generalize to the
modified setting of distinguishing between vulnerabilities and their patches.
Specifically, the algorithm computes this for (a) a model, that was trained on
the standard training dataset $Tr$ (Output $o_{1}$), and (b) for models that
were trained on amplified data. For the models trained on amplified data, it
computes both the average scores (Output $o_{2}$), and the average improvement
compared to the model that was trained on unamplified data (Output $o_{3}$).

In the first two lines of the algorithm, a machine learning model $f[Tr]$ is
trained on the training dataset $Tr$ (line 1) and evaluated on the
Vulnerability-Patch testing dataset VPT (line 2), resulting in the score
$s[f[Tr],VPT]$. The score $s[f[Tr],VPT]$ quantifies the ability of $f[Tr]$ to
distinguish between the vulnerable code snippets and their patches contained
in $VPT$.

After that, the algorithm computes the effects of amplifying the training
dataset $Tr$ with the different transformations $t_{k}$ on the performance of
the model on the Vulnerability-Patch testing dataset VPT. The result is
$e[Tr_{k},VPT])$, the absolute difference in performance of $f[Tr_{k}]$ and
$f[Tr]$ on the testing dataset VPT. In other words, $e[Tr_{k},VPT]$ quantifies
how many points in score are gained, if we amplify the training dataset with
transformation $t_{k}$. The output $o_{3}$ aggregates this intermediate result
over all transformations $t_{k}\in T$.

How can the results be used? Using Algorithm 2, researchers can effectively
evaluate whether the high scores of ML techniques for vulnerability detection
are specific to the testing datasets on which they were computed, or if they
generalize to a slightly modified vulnerability detection setting.
Specifically, for a selected technique researchers can answer the following
questions:

  1. (1)

Does the performance of my model generalize to a modified setting, which
requires to distinguish vulnerabilities from their patches? Answer: The model
can distinguish between vulnerabilities and their patches with performance
$o_{1}$.

  2. (2)

Does training data amplification using semantic preserving transformation help
my model to generalize to a modified vulnerability detection setting? Answer:
On average, $o_{3}$ score points are gained by training data amplification.

Table 1. The semantic preserving transformations, that we used in our experiments. Identifier | Type |  Description
---|---|---
$t_{1}$ | Identifier Renaming |  Rename all function parameters to a random token.
$t_{2}$ | Statement Reordering |  Reorder all function parameters.
$t_{3}$ | Identifier Renaming |  Rename the function.
$t_{4}$ | Statement Insertion |  Insert unexecuted code.
$t_{5}$ | Statement Insertion |  Insert comment.
$t_{6}$ | Statement Reordering |  Move the code of the function into a separate function.
$t_{7}$ | Statement Insertion |  Insert white space.
$t_{8}$ | Statement Insertion |  Define additional void function and call it from the function.
$t_{9}$ | Statement Removal |  Remove all comments.
$t_{10}$ | Statement Insertion |  Add code from training set as comment.
$t_{11}$ | All |  Random selection of $t_{1}$ to $t_{10}$.
$ADV$ | Statement Insertion |  Add adversarially optimized variable declaration (Adversarial training) (Yefet et al., 2020).

##  4\. Experimental Setup

###  4.1. Research Questions

Our objective is to validate empirically, whether the two proposed algorithms
can be used to evaluate state-of-the-art machine learning techniques for
vulnerability detection. Specifically, we aim to answer the following research
questions.

RQ.1 How is the performance of ML techniques affected, if we amplify the input
code snippets without affecting the vulnerability labels? (a) Can we measure a
decrease in performance, if we amplify the testing data of ML techniques using
semantic preserving transformations? (b) Does training data amplification
using the same transformations restore the initial performance?

RQ.2 Do ML techniques over-fit to specific amplifications of their training
data that do not affect the respective vulnerability labels? (a) Can we still
restore the performance, if we amplify the training dataset with a different
semantic preserving transformation than the testing dataset? (b) Can we
restore the performance, if we replace the simple semantic preserving
transformations with adversarial training?

RQ.3 Are the high scores of ML techniques specific to benchmark datasets or do
they generalize to a modified vulnerability detection setting? (a) Are state-
of-the-art machine learning techniques able to distinguish between vulnerable
functions and their patches? (b) Does training data amplification using
semantic preserving transformations have a positive effect on this ability?

###  4.2. Semantic Preserving Transformations

One of the central components of algorithms 1 and 2 are semantic preserving
transformations.

The most common semantic preserving transformations that are used in the
literature to investigate the robustness of machine learning techniques for
source code related tasks are identifier renaming (Yefet et al., 2020; Henkel
et al., 2022; Zhang et al., 2020; Srikant et al., 2021; Yang et al., 2022;
Zhang et al., 2022), insertion of unexecuted statements (Yefet et al., 2020;
Zhang et al., 2022; Henkel et al., 2022; Srikant et al., 2021), replacement of
statements with equivalent statements (Li et al., 2022a), reordering of
unrelated statements (Orvalho et al., 2022), deletion of unexecuted statements
(e.g. comments) (Li et al., 2022a), or combinations of the before mentioned
(Henkel et al., 2022; Srikant et al., 2021; Zhang et al., 2022).

Table 1 shows the 11 semantic preserving transformations that we implemented
for the experiments presented in this paper. We tried to cover all types of
transformations commonly used in the literature. The table lists all
transformations, categorizes them by type and provides short descriptions for
each of them. To investigate the effects of adversarial training for RQ.2 (see
Section 5), we re-implemented the adversarial training method proposed by
Yefet et al. (Yefet et al., 2020) as one of our semantic preserving
transformations (see $ADV$ in Table 1).

###  4.3. Vulnerability Detection Datasets

We use two publicly available vulnerability detection datasets for our
experiments.

CodeXGLUE/Devign. CodeXGLUE is a machine learning benchmark for code
understanding and generation (Lu et al., 2021). It consists of several
datasets for different source code related tasks. In our experiments, we only
use the dataset for vulnerability detection, which is based on the Devign
dataset (Zhou et al., 2019). Throughout this paper, we refer to this dataset
as the _CodeXGLUE/Devign dataset_ or just as the CodeXGLUE dataset. The
CodeXGLUE dataset contains 26.4k C functions, from which 45.6% contain
vulnerabilities, i.e. the dataset is fairly balanced. The types of the
vulnerabilities were not formally classified, but based of the collection
process the authors found most vulnerabilities in the dataset to be memory-
related, e.g. memory leaks, buffer overflows, memory corruption or crashes.
The authors of the CodeXGLUE benchmark also maintain a leaderboard (Microsoft,
2021), which tracks the performance of popular machine learning techniques on
the different datasets of the benchmark.

VulDeePecker. The other vulnerability detection dataset that we use in this
paper is the Code Gadget Database, that was introduced with the VulDeePecker
bug detector (Li et al., 2018). We refer to this dataset as the _VulDeePecker
dataset_. The original dataset contains 61.6k C/C++ code samples derived from
the NVD (Booth et al., 2013) and the SARD (Black, 2018), from which 17.7k
contain vulnerabilities, mainly buffer (CWE-119) and resource management
errors (CWE-399).

###  4.4. New Dataset: VulnPatchPairs

In order to investigate the ability of machine learning techniques to
distinguish between vulnerabilities and their patches (RQ3), we collected a
new dataset, which we call _VulnPatchPairs_.

VulnPatchPairs is an extension of the CodeXGLUE/Devign vulnerability detection
dataset (Zhou et al., 2019), which consists of C functions from two popular
open source repositories, FFmpeg (FFmpeg, 2023) and Qemu (Qemu, 2023). The
creators of the dataset describe the collection process in their original
publication (Zhou et al., 2019). As a first step, they filtered the selected
repositories for security related commits using a list of keywords. Then, they
invested 600 work-hours of a four-person team of security researchers to
classify the security related commits into vulnerability-fix commits (VFCs)
and non vulnerability-fix commits (non-VFCs), and extracted the respective
functions before the commits were applied as vulnerable (before VFCs) and non-
vulnerable (before non VFCs) functions. The actual patched versions of the
functions (after the VFCs were applied) are not part of their original
dataset. However, for each function of their dataset, the authors released the
respective commit id from the two open-source repositories as additional
information. We used this information to extract the actual patched versions
of the vulnerable functions in the CodeXGLUE dataset from the FFmpeg and Qemu
repositories, and created a new dataset: VulnPatchPairs. We verified
VulnPatchPairs by randomly selecting approx. 100 vulnerability-patch pairs
from the dataset and checking, whether they actually contain vulnerabilities
and their patches.

In total, VulnPatchPairs consists of 26.2k C functions from the two open
source repositories FFmpeg and Qemu. Exactly half (13.1k) of the 26.2k
functions contain security vulnerabilities, and were copied from the
CodeXGLUE/Devign vulnerability detection dataset. The other 13.1k are the
respective patches of the vulnerable functions, which we extracted from the
open source repositories. We publish the complete dataset as supplementary
material in a publicly available GitHub repository (see Section 4.7).

###  4.5. Machine Learning Techniques

(a) Test set accuracy over ten training epochs of different models trained
with VulBERTa on the CodeXGLUE/Devign dataset. Amplifying the testing set $Te$
with transformation $t_{10}$ (green) decreases the accuracy, but applying the
same transformation also to the training dataset $Tr$ (purple) restores the
accuracy back to previous levels. (b) Extension of the results in Figure 2a,
for all transformations $t_{k}\in T$, and for all three ML techniques. The
boxplots represent distributions of the resulting accuracies. Amplifying the
testing set $Te$ with transformations $t_{k}\in T$ (green boxplots) decreases
the accuracy, but applying the same transformation also to the training
dataset $Tr$ (purple boxplots) restores the accuracy back to previous levels.

Figure 2. Effects of amplifying the testing data and the training data using
the same semantic preserving transformations.

We selected three state-of-the-art machine learning techniques for our
experiments.

Selection Criteria. In order to select techniques which represent the state-
of-the-art of machine learning for vulnerability detection, we chose the Top-3
techniques from the CodeXGLUE leaderboard (Microsoft, 2021) for which the
authors provide open-source implementations. Measured by citations111Source:
https://api.semanticscholar.org/CorpusID:231855531 (289) and GitHub
Stars222Source: https://github.com/microsoft/CodeXGLUE (972), CodeXGLUE is the
most well known benchmark for source code related machine learning techniques.
The vulnerability detection dataset of the benchmark (Zhou et al., 2019) is
also highly cited (253 citations333Source:
https://api.semanticscholar.org/CorpusID:202539112) and widely used for
evaluating machine learning techniques for automatic vulnerability detection.

Selected Techniques. Based on the described criteria, we selected CoTexT (Phan
et al., 2021), VulBERTa (Hanif and Maffeis, 2022) and PLBart (Ahmad et al.,
2021b) for our experiments. At submission time of this paper, the three
techniques hold rank 1 (CoTexT), rank 5 (VulBERTa), and rank 9 (PLBart) on the
CodeXGLUE leaderboard for vulnerability detection (Microsoft, 2021). The
authors of all three techniques provide publicly available implementations of
their techniques (ML4CSec research group from the Department of Computing,
Imperial College London, 2022a; Phan, 2023; Wasi, 2021).

###  4.6. Model Training Pipeline

We used a similar training setup for all model instances that we trained for
our experiments.

Data split. For the CodeXGLUE/Devign dataset we used the
train-/validation-/testing dataset split provided by the authors of the
benchmark (Lu et al., 2021), which resulted in a training dataset with 21k
functions, a validation dataset with 2.7k functions, and a testing dataset
with 2.7k functions. For the VulDeePecker dataset we used the split provided
by the Hanif et al. (Hanif and Maffeis, 2022), which resulted in a training
dataset with 128.1k functions, a validation dataset with 16k functions, and a
testing dataset with 16k functions. The split for VulnPatchPairs is derived
from the split of CodeXGLUE, such that _all and only_ vulnerable functions in
training, validation, and testing sets, respectively, of CodeXGLUE where taken
as training, validation, and testing sets of VulnPatchPairs, augmented by
their corresponding patches.

Pre-processing. We employed several pre-processing steps for the models
trained with CoTexT. Since some special characters (e.g. ”#”) are out-of-
vocabulary for the tokenizer of CoTexT, the authors replaced them with natural
language tokens (e.g. ”SHARP_TOKEN”). We implemented the same mapping as the
authors of CoTexT (Phan, 2023) to address this issue for all three datasets.
We also pre-pended the string ”defect detection: ” to each code snippet, since
the authors used this approach to distinguish between the different tasks in
their multi-task setup. Furthermore, we mapped all the labels of our datasets
to either the tokens ”positive” (vulnerable) or ”negative” (not vulnerable) to
match the text-to-text architecture of CoTexT. We verified our pre-processing
by successfully reproducing the original scores reported by Phan et al. (Phan
et al., 2021). For the models trained with PLBart and VulBERTa, we did not
employ additional technique-specific pre-processing steps.

For the VulDeePecker dataset, we removed all duplicate functions and also
replaced all label revealing tokens (e.g. comment with token ”bad” above a
vulnerable function) that we found by manual inspection of the dataset with
randomly selected tokens. For the CodeXGLUE and VulnPatchPairs datasets, we
applied no additional pre-processing steps.

Hyperparameters. For all three ML techniques, we used the pre-trained models
provided by the respective authors as starting points for our experiments.
Similar to Hanif et al. (Hanif and Maffeis, 2022), we noticed a relatively
quick convergence of our performance metrics in our initial experiments (after
2-7 epochs), which is why we trained each model instance for 10 epochs. After
each epoch, we collected the average loss and several performance metrics both
for the training dataset and the testing dataset. For all hyperparameters
except the batch size for CoTexT, we used the values that were reported in the
original papers (Hanif and Maffeis, 2022; Phan et al., 2021; Ahmad et al.,
2021b). Due to limited VRAM of our hardware, we had to change the batch size
for CoTexT from 128 to 8. To make sure that this change had no negative impact
on the performance of CoTexT, we verified that we can still reproduce the
reported scores of the original publication (Phan et al., 2021). Consult our
published training scripts (see Section 4.7) for the complete list of
hyperparameter values that we used.

Tokenization. For all three ML techniques, we used the pre-configured
tokenizers that the authors provide in their open-source implementations
(ML4CSec research group from the Department of Computing, Imperial College
London, 2022a; Phan, 2023; Wasi, 2021).

Performance metrics. We tracked and quantified the performance of our trained
models on the selected testing datasets using several commonly used
performance metrics. We report two metrics in this paper: Accuracy and
F1-score.

For CodeXGLUE/Devign as a balanced dataset (45.6% vulnerable functions), we
use accuracy as the main performance metric. Accuracy is defined as the
percentage of correct predictions for a particular testing dataset, and is the
most popular metric for binary classification tasks in machine learning. Both
in the CodeXGLUE benchmark (Lu et al., 2021) and on the leaderboard
(Microsoft, 2021) accuracy is used as main performance measure.

For VulDeePecker as a relatively imbalanced dataset (28.7% vulnerable
functions) we use the F1-score as the main performance metric. The F1-score is
defined as the harmonic mean of precision and recall, and is most suitable
when the positive class (in our case vulnerable functions) is the minority
class in an imbalanced dataset.

Hardware. We used a setup of five Nvidia A100 GPUs, each equipped with 40 GB
HBM2 memory. One complete run of all our experiments takes approx. 45 days on
a single A100 GPU, which equals approx. $45/5=9$ days when all five GPUs are
utilized.

###  4.7. Reproducibility

To facilitate open science and rigorous reproducibility, we publish all of our
data, code and results. At the time of submission, they are available at the
following URL.

  * $\star$

https://github.com/niklasrisse/LimitsOfML4Vuln

In addition to all of the data and results, the repository contains Python
scripts with deterministic settings, so that our experimental results can be
exactly reproduced.

##  5\. Experimental Results

### RQ.1 Testing- and Training Data Amplification

We investigate, whether (a) testing data amplification using semantic
preserving transformation decreases the performance of state-of-the-art ML
techniques for vulnerability detection, and (b) whether training data
amplification using the same transformations restores the performance towards
previous levels.

Methodology. We used Algorithm 1 to investigate both questions. We ran the
algorithm for each ML technique and dataset separately and measured the
outcomes using the respective preferred performance metrics (see Section 4.6).
We did not only record the outcomes after completing the full training of the
respective models, but also after each training epoch in order to observe the
progression of the learning process.

Results. Figure 2a shows the test set accuracy of different VulBERTa models
measured after each of the ten training epochs. We can observe, that
amplifying the testing dataset $Te$ with transformation $t_{10}$ leads to a
substantial drop in accuracy, represented by the gap between the green and red
graphs in the figure. We can also observe, that amplifying the training
dataset $Tr$ with the same transformation $t_{10}$ as the testing dataset,
restores the accuracy back to previous levels (purple graph).

Figure 2b extends the results of Figure 2a to all semantic preserving
transformations $t_{k}\in T$, and to all three ML techniques. Instead of
showing the accuracy for each training epoch, we only use the maximum accuracy
across the full training. Across all three ML techniques we can observe, that
amplifying the testing dataset $Te$ with transformations $t_{k}\in T$
(represented by the green boxplots), on average, leads to a drop in accuracy
compared to evaluations on the standard testing dataset $Te$ (represented by
the horizontal lines). In parallel to Figure 2a we can also observe that, on
average, training data amplification using the same transformation as for
testing data amplification leads to a restoration of the observed drops in
accuracy (represented by the purple boxplots).

Table 2. Algorithm 1 Results: Average effects of amplifying only the testing data ($o_{1}$), training and testing data using the same ($o_{2}$), or a different transformation ($o_{3}$). Dataset | Technique | $s[Tr,Te]$ | $o_{1}=\sum_{k}\frac{e[Tr,Te_{k}]}{N}$ | $o_{2}=\sum_{k}\frac{e[Tr_{k},Te_{k}]}{N}$ | $o_{3}=\sum_{k}\sum_{j\neq k}\frac{e[Tr_{k},Te_{j}]}{(N(N-1))}$
---|---|---|---|---|---
CodeXGLUE | VulBERTa | 0.639 |  -0.017 $\downarrow$ |  -0.004 $\uparrow$ |  -0.025 $\downarrow$
CodeXGLUE | CoTexT | 0.673 |  -0.022 $\downarrow$ |  -0.014 $\uparrow$ |  -0.030 $\downarrow$
CodeXGLUE | PLBart | 0.633 |  -0.021 $\downarrow$ |  -0.007 $\uparrow$ |  -0.026 $\downarrow$
|  |  |  -0.020 $\downarrow$ |  -0.008 $\uparrow$ |  -0.027 $\downarrow$
VulDeePecker | VulBERTa | 0.875 |  -0.067 $\downarrow$ |  -0.011 $\uparrow$ |  -0.057 $\downarrow$
VulDeePecker | CoTexT | 0.873 |  -0.007 $\downarrow$ |  -0.005 $\uparrow$ |  -0.018 $\downarrow$
VulDeePecker | PLBart | 0.865 |  -0.014 $\downarrow$ |  -0.007 $\uparrow$ |  -0.035 $\downarrow$
|  |  |  -0.030 $\downarrow$ |  -0.007 $\uparrow$ |  -0.037 $\downarrow$

Table 2 summarizes the outputs of Algorithm 1 for all three ML techniques and
both datasets. Specifically, it shows the average recorded drops in score,
when only the testing dataset was amplified (fourth column, $o_{1}$), the
training and testing datasets were amplified using the same transformation
(fifth column, $o_{2}$), and the training and testing datasets were amplified
using a different transformation (sixth column, $o_{3}$). We observe that, on
average, amplifying the testing dataset leads to a drop in score ($-0.020$ for
CodeXGLUE, $-0.030$ for VulDeePecker), and amplifying the training dataset
using the same transformation restores that decrease towards previous levels.
On average, approx. 59.8% (CodeXGLUE) and 54.1% (VulDeePecker) of the lost
score is restored.

_Across 2 datasets, 3 ML techniques, and 11 transformations, on average, (a)
testing data amplification using semantic preserving transformations leads to
a drop in score (CodeXGLUE: $-0.020$, VulDeePecker: $-0.030$), and (b)
training data amplification using the same transformations restores 59.8%
(CodeXGLUE) and 54.1% (VulDeePecker) of the lost performance._

RQ.1 has already been studied in the literature, for many different ML
techniques, datasets, and tasks (Henkel et al., 2022; Srikant et al., 2021;
Yefet et al., 2020; Li et al., 2022b, 2021; Zhang et al., 2022; Yang et al.,
2022; Zhang et al., 2020; Bielik and Vechev, 2020). Based on our evidence, we
can approve the findings of the literature.

### RQ.2 Over-fitting to Specific Transformations

We investigate, whether (a) the performance of ML techniques can still be
restored if we amplify the training dataset with a different semantic
preserving transformation than the testing dataset, and (b) whether the
performance can be restored if we replace the training data amplification with
adversarial training.

Methodology. We use Algorithm 1 to investigate RQ.2(a), with the same setup as
for RQ.1. To investigate research question RQ.2 (b), we use an altered version
of Algorithm 1, in which we train an additional model with adversarial
training using the method proposed by Yefet et al. (Yefet et al., 2020) (see
Section 4.2) after the standard training in line 1 of the algorithm. We denote
this adversarially trained model by $f[Tr_{ADV}]$.

Results. Figure 3a is similar to Figure 2a, it also shows the test set
accuracies of different VulBERTa models measured after each of the ten
training epochs. In addition to the results displayed in Figure 2a, Figure 3a
shows the accuracies (gray lines) of models trained on data that was amplified
using all transformations except $t_{10}$, which was used to amplify the
testing dataset. We observe, that amplifying the training dataset $Tr$ with
different transformations $t_{k\neq 10}$ as the testing dataset does not
restore the accuracy to previous levels.

Figure 3b visualizes the same results for all semantic preserving
transformations $t_{k}\in T$ and for all three ML techniques. Again, the green
and the purple boxplots represent the distributions of accuracies, when either
only the testing dataset (green) or training and testing datasets were
amplified using the same transformation (purple). The yellow boxplots
represent the distribution of accuracies achieved by models that were trained
on data, which was amplified using a different transformation than for the
testing data. Across all three ML techniques we observe that, on average,
amplifying the training dataset $Tr$ with a different transformations
$t_{k\neq j}$ as the testing dataset does not restore the accuracy back to
previous levels.

Figure 3c visualizes the same results as Figure 3b, but using the VulDeePecker
dataset. In this case, the y-axis measures the F1-score, since this is the
preferred evaluation metric for the VulDeePecker dataset. Again, we observe
that, on average, amplifying the training dataset $Tr$ with a different
transformations $t_{k\neq j}$ as the testing dataset does not restore the
accuracy back to previous levels.

(a) Test set accuracy over ten training epochs of different models trained
with VulBERTa on the CodeXGLUE/Devign dataset. Amplifying the training set
$Tr$ with different transformations $t_{k\neq 10}$ than the testing dataset
(gray lines) does not restore the accuracy back to previous levels. (b)
Extension of the results in Figure 3a, for all transformations $t_{k}\in T$
and for all three ML techniques. The boxplots represent distributions of the
resulting accuracies. Amplifying the training set $Tr$ with different
transformations $t_{k}\neq j$ than the testing dataset (yellow boxplots) does
not restore the accuracy back to previous levels.  (c) Same setup as for
Figure 3b, but using the VulDeePecker dataset. The boxplots represent
distributions of the resulting F1-scores. Amplifying the training set $Tr$
with different transformations $t_{k}\neq j$ than the testing dataset (yellow
boxplots) does not restore the F1-score back to previous levels. (d) Similar
setup as for Figure 3b, but using adversarial training to amplify the training
dataset. Amplifying the training set $Tr$ with adversarial training (pink
boxplots) does not restore the accuracy back to previous levels.

Figure 3. Effects of amplifying the training data with different semantic
preserving transformations than the testing data.

In addition to the results for RQ.1, Table 2 also shows the average recorded
drops in score, when the training and testing datasets were amplified using
different transformations (sixth column, $o_{3}$). We observe that, on
average, the score drops by 0.027 (CodeXGLUE) and 0.037 (VulDeePecker). Across
the three techniques, the decrease is on average 35.7% (CodeXGLUE) and 97.4%
(VulDeePecker) stronger than for training on unamplified data. In other words,
amplifying the training dataset using a different transformation than for the
testing dataset did not restore the score towards previous levels, but instead
decreased it even further.

Figure 3d shows the results of extending Algorithm 1 to adversarial training.
In parallel to Figure 3b and Figure 3c, the green and the purple boxplots
represent the distributions of accuracies, when either only the testing
dataset (green) or training and testing datasets were amplified using the same
transformation (purple). The pink boxplots represent the distribution of
accuracies achieved by adversarial training for testing datasets amplified
using the different transformations $t_{k}\in T$. We observe, that adversarial
training also does not restore the accuracy back to previous levels. Across
all transformations $t_{k}\in T$, on average, adversarial training leads to
0.002 (VulBERTa), -0.014 (CoTexT), and 0.0 (PLBart) improvement in accuracy
compared to unamplified training.

_(a) Across 2 datasets, 3 ML techniques, and 11 transformations, amplifying
the training dataset using a different transformation than for the testing
dataset does not restore the performance back to previous levels. Instead of
restoring, on average, the performance even drops by 35.7% (CodeXGLUE) and
97.4% (VulDeePecker) more compared to using standard training data. In other
words, the ML techniques over-fit to the label-unrelated features introduced
by semantic preserving transformations during training. (b) Replacing training
data amplification with adversarial training also does not restore the
performance back to previous levels._

In summary, we can observe that across the tested ML techniques,
transformations and datasets, training data amplification does only restore
the performance back to previous levels when the testing dataset is amplified
in a similar way than the training dataset.

_The robustness gained by data amplification only applies to the specific
transformations used during training of the model._

### RQ.3 Generalization to VulnPatchPairs

We investigate, whether (a) ML techniques are able to generalize from typical
vulnerability detection training datasets to a modified setting, in which they
have to distinguish between vulnerabilities and their patches, and (b) whether
training data amplification leads to an improvement in performance in this
modified setting.

Methodology. We used Algorithm 2 to investigate both questions. We ran the
algorithm for all three ML techniques seperately, and used the training subset
of the CodeXGLUE/Devign dataset (see Section 4.3) as training dataset Tr. We
used the testing subset of VulnPatchPairs (see Section 4.4) as the
vulnerability-patch testing dataset VPT.

Results. Table 3 shows the computed outputs of running Algorithm 2.
Specifically, it shows the accuracy of different models evaluated on the
standard CodeXGLUE/Devign testing dataset Te (second column, $s[f[Tr],Te]$),
evaluated on the vulnerability-patch testing dataset VPT (third column,
$o_{1}$), and the average accuracy of models trained on amplified data and
evaluated on the vulnerability-patch testing dataset VPT (fourth column,
$o_{2}$). We observe, that the accuracy of all three ML techniques evaluated
on the vulnerability-patch testing dataset VPT ($o_{1}$) is far worse than the
accuracy on the standard CodeXGLUE/Devign testing dataset Te ($s[f[Tr],Te]$).
Since VPT is perfectly balanced (50% vulnerable, 50% not vulnerable), a random
guesser (coin flip) would be expected to achieve an accuracy of 0.5. On
average, the three ML techniques achieve an accuracy of 0.518, which is only
0.018 higher than the expected accuracy of a random guesser.

From the fourth and the fifth columns of Table 3 ($o_{2}$ and $o_{3}$) we can
also observe that, on average, training data amplification leads to no
improvement of the accuracy compared to the models trained on unamplified
training data.

_(a) All three ML techniques are not able to distinguish between
vulnerabilities and their patches. On average, the performance is only 0.018
better than the expected accuracy of a random guesser. In other words, the ML
techniques are unable generalize from their training data to a slightly
modified vulnerability detection setting. (b) Training data amplification has
no effect on the ability of the models to generalize to the modified setting._

##  6\. Threats to Validity

Table 3. Algorithm 2 Results: Accuracy on VulnPatchPairs ($o_{1}$) and the effects of training data amplification ($o_{2}$, $o_{3}$). |  | $o_{1}=$ | $o_{2}=$ | $o_{3}=$
---|---|---|---|---
Technique | $s[f[Tr],Te]$ | $s[f[Tr],VPT]$ | $\frac{\sum_{k}s[f[Tr_{k}],VPT]}{N}$ | $\frac{\sum_{k}e[f[Tr_{k}],VPT]}{N}$
VulBERTa | 0.639 | 0.527 | 0.522 | -0.005
CoTexT | 0.673 | 0.503 | 0.501 | -0.001
PLBart | 0.633 | 0.524 | 0.528 | 0.004
| 0.648 | 0.518 | 0.517 | -0.001

As for any empirical study, there are various threats to the validity of our
results and conclusions.

Internal validity. There are several potential sources of systematic error in
our experiments.

A common source of systematic error in empirical studies on machine learning
techniques is hyperparameter selection. Given a particular desired outcome,
hyperparameters can be optimized to move the result in the desired direction.
We tried to minimize this risk by taking the values for hyperparameters
provided by the authors of the chosen ML techniques.

Another potential source of systematic error is the training-/testing dataset
split. Similar to hyperparameter selection, dataset split can also be varied
to change a result in a desired direction. We tried to avoid this risk by
taking the provided splits of the CodeXGLUE benchmark (Lu et al., 2021) and by
Hanif et al. (ML4CSec research group from the Department of Computing,
Imperial College London, 2022b).

Even the choice of random seed can be a source of systematic error in machine
learning training pipelines. We addressed this by randomly selecting and then
fixing random seeds prior to carrying out the particular experiments.

External validity. The degree to which our results generalize to other machine
learning techniques, vulnerability detection datasets, semantic preserving
transformations, and performance metrics, are concerns of external validity.
We tried to minimize the risk attached to these concerns by evaluating a wide
set of three state-of-the-art techniques, two datasets, two performance
metrics, and 11 semantic preserving transformations. However, all evaluated ML
techniques happen to be token-based large language models (LLMs). As our
selection criterion, we defined the top-performing ML techniques on the most
widely known ML vulnerability detection benchmark CodeXGLUE (Lu et al., 2021;
Microsoft, 2021) that are available as open source. This gave us ranks 1, 5,
and 9 of the leaderboard, all of which are token-based LLMs. In fact, 8 of the
Top-10 solutions on the leaderboard are token-based LLMs. Investigating,
whether our results generalize to other types of techniques, e.g. to graph-
based neural networks or path-based sequential models, would be an interesting
avenue for future work. We tried to keep both Algorithm 1 and Algorithm 2 as
general as possible, so that they can easily be adapted to other techniques,
datasets, transformations, and metrics.

The concern of generalization is also valid for our results on adversarial
training (see Figure 3d), which we computed by using only one particular
technique (Yefet et al., 2020) to compute the adversarial transformations.

Simplicity of Transformations. Some of the semantic preserving transformations
that we used (see Table 1) could be easily addressed by adding additional data
pre-processing (e.g. mapping identifiers to standardized names). However, the
specific transformations that we implemented are merely a tool to demonstrate,
that the robustness gained by training data amplification only applies to the
specific transformations used for training and that the techniques that we
investigated over-fit to the label-unrelated features introduced by these
transformations. For a new technique, they could be replaced by a different
set of transformations.

Figure 4. Naturalness of the semantic preserving transformations, that we used
in our experiments. Lower cross entropy means higher naturalness. All
transformations except $t_{1}$ (identifier renaming) lead to lower or equal
cross entropy than no transformation (None).

Naturalness. Since vulnerability detection techniques are ultimately designed
to be applied to real-world code, we also need to ensure that our
transformations lead to code snippets which could occur in the real world, or
i.e., lead to _natural_ code. We measured the naturalness of our
transformations using the method introduced by Hindle et al. (Hindle et al.,
2012) and present the results in Figure 4. Following the method of Hindle et
al., we first trained a 2-gram markov model on the CodeXGLUE training dataset.
Using the trained model, we can compute the _cross entropy_ of a given code
snippet, which represents how surprising or _unnatural_ the code snippet is
relative to the code snippets observed in the training dataset (for a detailed
explanation consult the work of Hindle et al. (Hindle et al., 2012)). Using
this approach, we computed the cross entropy for all code snippets in the
CodeXGLUE testing dataset (green boxplot), and for transformed versions of the
CodeXGLUE testing dataset (purple boxplots). The green line represents the
average cross entropy for all code snippets in the untransformed CodeXGLUE
testing dataset. We can observe, that for all transformations except $t_{1}$
(identifier renaming), the cross entropy is similar or lower than for the
untransformed dataset. In other words, all transformations except $t_{1}$
(identifier renaming) lead to code snippets that are similar in naturalness
compared to the real-world code of the CodeXGLUE testing dataset.

Class balance. He et al. (He et al., 2022) have shown that learning-based
vulnerability detection techniques trained on fairly balanced datasets (such
as CodeXGLUE and VulDeePecker) often fail to generalize to real-world code
repositories, which usually contain a much smaller ratio of security
vulnerabilities (Karampatsis and Sutton, 2020). However, measured by
citations, these balanced datasets are still by far the most popular datasets
to evaluate machine learning techniques for vulnerability detection. To our
current knowledge, there is no vulnerability detection dataset with sufficient
size (more than 10k code snippets), high quality labels (provided by security
experts), and a realistic distribution of vulnerable to non-vulnerable code
snippets that is widely used in the research community to evaluate machine
learning techniques (at least 50 citations). This is why we decided to focus
our experiments on the CodeXGLUE and VulDeePecker datasets, even though they
do not reflect a realistic class distribution.

##  7\. Discussion and Future Work

In summary, our results demonstrate that the robustness gained by data
amplification only applies to the specific transformations used during
training of the model, that state-of-the-art ML techniques still over-fit to
the label-unrelated features introduced by semantic preserving transformations
during training, and that they are not able to generalize to a modified
setting, in which they have to distinguish between vulnerabilities and their
patches.

Over-fitting of learned models is a well-known problem in the machine learning
research field (Ying, 2019; Dietterich, 1995). The degree of over-fitting for
a specific machine learning model is usually measured as the gap in score
quantified by a chosen performance metric between the training dataset and a
held-out test dataset. Our proposed Algorithm 1 is a novel way to measure
over-fitting of ML techniques for vulnerability detection, that goes beyond
the performance gap between training and testing datasets in a standard
evaluation setup, and can even detect over-fitting if there is no gap in the
standard setup at all. There are several common strategies to reduce over-
fitting in the standard evaluation setup, e.g. early-stopping, dropout or
large pre-training datasets (Ying, 2019), which are already integrated in the
ML techniques that we used in our experiments. However, our experiments
demonstrate that the techniques are still severely over-fitting to label-
unrelated features introduced by semantic preserving transformations during
training data amplification. Finding ways to robustify ML techniques without
or with minimal over-fitting will be a central challenge of the machine
learning for vulnerability detection research area. We hope that our proposed
algorithms can be used to understand the problem, to develop new approaches
and to track the progress in this direction.

Generalization. The results for RQ.3 (see Section 5) reveal, that state-of-
the-art ML techniques for vulnerability detection lack the ability to
generalize from their training data to a modified setting, which requires to
distinguish between vulnerabilities and their patches. Since we can not assume
that real-world software systems would be similar to the training data of
these techniques, the ability to generalize to modified settings would be
required for these techniques to be safely integrated in real software
engineering environments.

The ability of a ML technique to generalize to testing data that is
differently distributed than the training data is also called _out-of-
distribution generalization_ , and the lack of it for machine learning
techniques has been recently identified (e.g. in the computer vision domain
(Hendrycks et al., 2021; Shen et al., 2021)). Our proposed Algorithm 2 can be
seen as a tool to measure out-of-distribution generalization for the domain of
automatic vulnerability detection. It would be an interesting avenue for
future work to try approaches that have been used to address out-of-
distribution generalization in other domains (e.g. causal representation
learning (Schölkopf et al., 2021)) on the task of automatic vulnerability
detection, and measure the success using our Algorithm 2.

## References

  * (1)
  * Ahmad et al. (2021a) Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021a.  Unified Pre-training for Program Understanding and Generation. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Online, 2655–2668.  https://www.aclweb.org/anthology/2021.naacl-main.211
  * Ahmad et al. (2021b) Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021b.  Unified Pre-training for Program Understanding and Generation. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Online, 2655–2668.  https://www.aclweb.org/anthology/2021.naacl-main.211
  * Applis et al. (2021) Leonhard Applis, Annibale Panichella, and Arie van Deursen. 2021\.  Assessing Robustness of ML-Based Program Analysis Tools using Metamorphic Program Transformations. In _2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. 1377–1381.  https://doi.org/10.1109/ASE51524.2021.9678706
  * Bhandari et al. (2021) Guru Bhandari, Amara Naseer, and Leon Moonen. 2021\.  CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software. In _Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering_ (Athens, Greece) _(PROMISE 2021)_. Association for Computing Machinery, New York, NY, USA, 30–39.  https://doi.org/10.1145/3475960.3475985
  * Bielik and Vechev (2020) Pavol Bielik and Martin Vechev. 2020.  Adversarial Robustness for Code. In _Proceedings of the 37th International Conference on Machine Learning_ _(ICML’20)_. JMLR.org, Article 84, 12 pages.
  * Black (2018) Paul Black. 2018\.  A Software Assurance Reference Dataset: Thousands of Programs With Known Bugs.  https://doi.org/10.6028/jres.123.005
  * Booth et al. (2013) Harold Booth, Doug Rike, and Gregory Witte. 2013.  The National Vulnerability Database (NVD): Overview.  https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=915172
  * Bui et al. (2022) Quang-Cuong Bui, Riccardo Scandariato, and Nicolás E. Díaz Ferreyra. 2022\.  Vul4J: A Dataset of Reproducible Java Vulnerabilities Geared towards the Study of Program Repair Techniques. In _Proceedings of the 19th International Conference on Mining Software Repositories_ (Pittsburgh, Pennsylvania) _(MSR ’22)_. Association for Computing Machinery, New York, NY, USA, 464–468.  https://doi.org/10.1145/3524842.3528482
  * Buratti et al. (2020) Luca Buratti, Saurabh Pujar, Mihaela Bornea, Scott McCarley, Yunhui Zheng, Gaetano Rossiello, Alessandro Morari, Jim Laredo, Veronika Thost, Yufan Zhuang, et al. 2020\.  Exploring software naturalness through neural language models.  _arXiv preprint arXiv:2006.12641_ (2020).
  * Dietterich (1995) Tom Dietterich. 1995\.  Overfitting and Undercomputing in Machine Learning.  _ACM Comput. Surv._ 27, 3 (sep 1995), 326–327.  https://doi.org/10.1145/212094.212114
  * Feng et al. (2020) Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020.  CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. Association for Computational Linguistics, Online, 1536–1547.  https://doi.org/10.18653/v1/2020.findings-emnlp.139
  * FFmpeg (2023) FFmpeg 2023.  _FFmpeg GitHub repository_.  Retrieved March 8, 2023 from https://github.com/FFmpeg/FFmpeg
  * Fu and Tantithamthavorn (2022) Michael Fu and Chakkrit Tantithamthavorn. 2022.  LineVul: A Transformer-based Line-Level Vulnerability Prediction. In _2022 IEEE/ACM 19th International Conference on Mining Software Repositories (MSR)_. IEEE.
  * Hanif and Maffeis (2022) Hazim Hanif and Sergio Maffeis. 2022.  VulBERTa: Simplified Source Code Pre-Training for Vulnerability Detection. In _2022 International Joint Conference on Neural Networks (IJCNN)_. 1–8.  https://doi.org/10.1109/IJCNN55064.2022.9892280
  * He et al. (2022) Jingxuan He, Luca Beurer-Kellner, and Martin Vechev. 2022\.  On Distribution Shift in Learning-based Bug Detectors. In _Proceedings of the 39th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 162)_ , Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds.). PMLR, 8559–8580.  https://proceedings.mlr.press/v162/he22a.html
  * Hendrycks et al. (2021) Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. 2021.  The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 8340–8349.
  * Henkel et al. (2022) Jordan Henkel, Goutham Ramakrishnan, Zi Wang, Aws Albarghouthi, Somesh Jha, and Thomas Reps. 2022\.  Semantic Robustness of Models of Source Code. In _2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)_. IEEE.  https://doi.org/10.1109/saner53432.2022.00070
  * Hindle et al. (2012) Abram Hindle, Earl T. Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu. 2012.  On the naturalness of software. In _2012 34th International Conference on Software Engineering (ICSE)_. 837–847.  https://doi.org/10.1109/ICSE.2012.6227135
  * Karampatsis and Sutton (2020) Rafael-Michael Karampatsis and Charles Sutton. 2020.  How Often Do Single-Statement Bugs Occur? The ManySStuBs4J Dataset. In _2020 IEEE/ACM 17th International Conference on Mining Software Repositories (MSR)_. 573–577.  https://doi.org/10.1145/3379597.3387491
  * Li et al. (2022a) Yaoxian Li, Shiyi Qi, Cuiyun Gao, Yun Peng, David Lo, Zenglin Xu, and Michael R. Lyu. 2022a.  A Closer Look into Transformer-Based Code Intelligence Through Code Transformation: Challenges and Opportunities.  https://doi.org/10.48550/ARXIV.2207.04285
  * Li et al. (2022b) Yiyang Li, Hongqiu Wu, and Hai Zhao. 2022b.  Semantic-Preserving Adversarial Code Comprehension. In _Proceedings of the 29th International Conference on Computational Linguistics_. International Committee on Computational Linguistics, Gyeongju, Republic of Korea, 3017–3028.  https://aclanthology.org/2022.coling-1.267
  * Li et al. (2021) Zhen Li, Jing Tang, Deqing Zou, Qian Chen, Shouhuai Xu, Chao Zhang, Yichen Li, and Hai Jin. 2021\.  Towards Making Deep Learning-based Vulnerability Detectors Robust.  https://doi.org/10.48550/ARXIV.2108.00669
  * Li et al. (2018) Zhen Li, Deqing Zou, Shouhuai Xu, Xinyu Ou, Hai Jin, Sujuan Wang, Zhijun Deng, and Yuyi Zhong. 2018\.  VulDeePecker: A Deep Learning-Based System for Vulnerability Detection. In _Proceedings 2018 Network and Distributed System Security Symposium_. Internet Society.  https://doi.org/10.14722/ndss.2018.23158
  * Lu et al. (2021) Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, MING GONG, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie LIU. 2021\.  CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation. In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks_ , J. Vanschoren and S. Yeung (Eds.), Vol. 1.  https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/file/c16a5320fa475530d9583c34fd356ef5-Paper-round1.pdf
  * Microsoft (2021) Microsoft 2021.  _CodeXGLUE leaderboards_.  Retrieved March 8, 2023 from https://microsoft.github.io/CodeXGLUE/#LB-DefectDetection
  * ML4CSec research group from the Department of Computing, Imperial College London (2022a) ML4CSec research group from the Department of Computing, Imperial College London 2022a.  _VulBERTa GitHub repository_.  Retrieved March 8, 2023 from https://github.com/ICL-ml4csec/VulBERTa
  * ML4CSec research group from the Department of Computing, Imperial College London (2022b) ML4CSec research group from the Department of Computing, Imperial College London 2022b.  _VulDeePecker Function-level dataset_.  Retrieved March 8, 2023 from https://github.com/ICL-ml4csec/VulBERTa/tree/main/data
  * Orvalho et al. (2022) Pedro Orvalho, Mikoláš Janota, and Vasco Manquinho. 2022.  MultIPAs: applying program transformations to introductory programming assignments for data augmentation. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 1657–1661.
  * Phan (2023) Long Phan. 2023\.  _CoTexT GitHub repository_.  Retrieved March 8, 2023 from https://github.com/justinphan3110/CoTexT
  * Phan et al. (2021) Long Phan, Hieu Tran, Daniel Le, Hieu Nguyen, James Annibal, Alec Peltekian, and Yanfang Ye. 2021.  CoTexT: Multi-task Learning with Code-Text Transformer. In _Proceedings of the 1st Workshop on Natural Language Processing for Programming (NLP4Prog 2021)_. Association for Computational Linguistics, Online, 40–47.  https://doi.org/10.18653/v1/2021.nlp4prog-1.5
  * Ponta et al. (2019) Serena E. Ponta, Henrik Plate, Antonino Sabetta, Michele Bezzi, and Cédric Dangremont. 2019\.  A Manually-Curated Dataset of Fixes to Vulnerabilities of Open-Source Software. In _Proceedings of the 16th International Conference on Mining Software Repositories_ (Montreal, Quebec, Canada) _(MSR ’19)_. IEEE Press, 383–387.  https://doi.org/10.1109/MSR.2019.00064
  * Qemu (2023) Qemu 2023.  _Qemu GitHub repository_.  Retrieved March 8, 2023 from https://github.com/qemu/qemu
  * Rabin et al. (2021) Md Rafiqul Islam Rabin, Nghi D.Q. Bui, Ke Wang, Yijun Yu, Lingxiao Jiang, and Mohammad Amin Alipour. 2021.  On the generalizability of Neural Program Models with respect to semantic-preserving program transformations.  _Information and Software Technology_ 135 (jul 2021), 106552\.  https://doi.org/10.1016/j.infsof.2021.106552
  * Schölkopf et al. (2021) B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. 2021.  Toward Causal Representation Learning.  _Proc. IEEE_ 109, 5 (2021), 612–634.  https://doi.org/10.1109/JPROC.2021.3058954
  * Shen et al. (2021) Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. 2021.  Towards Out-Of-Distribution Generalization: A Survey.  https://doi.org/10.48550/ARXIV.2108.13624
  * Srikant et al. (2021) Shashank Srikant, Sijia Liu, Tamara Mitrovska, Shiyu Chang, Quanfu Fan, Gaoyuan Zhang, and Una-May O’Reilly. 2021.  Generating Adversarial Computer Programs using Optimized Obfuscations. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=PH5PH9ZO_4
  * Wasi (2021) Ahmad Wasi. 2021\.  _PLBart GitHub repository_.  Retrieved March 8, 2023 from https://github.com/wasiahmad/PLBART
  * Yang et al. (2022) Zhou Yang, Jieke Shi, Junda He, and David Lo. 2022\.  Natural Attack for Pre-Trained Models of Code. In _Proceedings of the 44th International Conference on Software Engineering_ (Pittsburgh, Pennsylvania) _(ICSE ’22)_. Association for Computing Machinery, New York, NY, USA, 1482–1493.  https://doi.org/10.1145/3510003.3510146
  * Yefet et al. (2020) Noam Yefet, Uri Alon, and Eran Yahav. 2020.  Adversarial Examples for Models of Code.  _Proc. ACM Program. Lang._ 4, OOPSLA, Article 162 (nov 2020), 30 pages.  https://doi.org/10.1145/3428230
  * Ying (2019) Xue Ying. 2019.  An Overview of Overfitting and its Solutions.  _Journal of Physics: Conference Series_ 1168, 2 (feb 2019), 022022.  https://doi.org/10.1088/1742-6596/1168/2/022022
  * Zhang et al. (2022) Huangzhao Zhang, Zhiyi Fu, Ge Li, Lei Ma, Zhehao Zhao, Hua’an Yang, Yizhe Sun, Yang Liu, and Zhi Jin. 2022.  Towards Robustness of Deep Program Processing Models—Detection, Estimation, and Enhancement.  _ACM Trans. Softw. Eng. Methodol._ 31, 3, Article 50 (apr 2022), 40 pages.  https://doi.org/10.1145/3511887
  * Zhang et al. (2020) Huangzhao Zhang, Zhuo Li, Ge Li, L. Ma, Yang Liu, and Zhi Jin. 2020.  Generating Adversarial Examples for Holding Robustness of Source Code Processing Models. In _AAAI Conference on Artificial Intelligence_.
  * Zhou et al. (2019) Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. 2019.  _Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks_.  Curran Associates Inc., Red Hook, NY, USA.


# Using Multiple Self-Supervised Tasks Improves Model Robustness

Matthew Lawhon, Chengzhi Mao & Junfeng Yang
Department of Computer Science
Columbia University
New York, NY 10027, USA
<EMAIL_ADDRESS>{mcz<EMAIL_ADDRESS>
###### Abstract

Deep networks achieve state-of-the-art performance on computer vision tasks,
yet they fail under adversarial attacks that are imperceptible to humans. In
this paper, we propose a novel defense that can dynamically adapt the input
using the intrinsic structure from multiple self-supervised tasks. By
simultaneously using many self-supervised tasks, our defense avoids over-
fitting the adapted image to one specific self-supervised task and restores
more intrinsic structure in the image compared to a single self-supervised
task approach. Our approach further improves robustness and clean accuracy
significantly compared to the state-of-the-art single task self-supervised
defense. Our work is the first to connect multiple self-supervised tasks to
robustness, and suggests that we can achieve better robustness with more
intrinsic signal from visual data.

##  1 Introduction

Deep learning architectures achieve state-of-the-art and often superhuman
performance across a wide variety of vision tasks (Croce & Hein, 2020).
Despite this, as first noticed in 2014, they remain vulnerable to adversarial
attacks: visually imperceptible perturbations that cause easily classifiable
images to be misclassified with high confidence by otherwise state-of-the-art
machine learning algorithms (Szegedy et al., 2014). This results in
unpredictable behavior in edge-cases, contrived examples and examples
unrepresented in training data. The inability to address this sufficiently is
a leading hurdle to deploying deep learning solutions to human safety and
well-being critical applications like autonomous transportation and health-
care.

Though there is a large line of research into adversarial training, how we can
train networks to resist adversarial attacks, training time defenses can be
very computationally expensive and it is difficult to provide guarantees for
all possible attack methods (Tramèr et al., 2018). Empirically, unrestricted
white box attacks, in which an adversary has complete, unrestricted access to
the network it is trying to corrupt, have been found very difficult to resist
via adversarial training. Mao et al. (2021) propose to adapt to adversarial
attacks at test time by restoring the performance of a selected self-
supervised task, however, the adaption method significantly reduces the clean
accuracy after adaptation because it over-fits to a single self-supervised
task.

Figure 1: Given a possibly attacked image, in our approach we find some small
reverse perturbation to minimize multiple self-supervised learning task losses
and recover structure in the image. Adding this reverse perturbation to our
image, we are able to improve classification accuracy on attacked images with
minimal disruption to classification accuracy on unattacked images.

In this paper, we propose to mitigate the over-fitting problem present in Mao
et al. (2021)’s reversal method by using multi-task learning. Our key insight
is that different self-supervised tasks, none of which require labels, capture
different aspects of the intrinsic structure of images. By restoring the
performance of attacked images on multiple self-supervised tasks, we can
recover a larger set of features that have been corrupted by an adversarial
attack. In addition, Mao et al. (2020) show that it is harder to
simultaneously attack multiple tasks, suggesting that this methodology is also
resistant to an adaptive attacker who has full access to our reversal
methodology and attempts to optimize their attack knowing our reversal
methodology in place.

Using three self-supervised tasks, our approach yields statistically
significant improvement upon the state-of-the-art defense. On the CIFAR-10
dataset using Carmon et al. (2019)’s robustly trained baseline model, we
achieved a 1.1% improvement in classification accuracy of unattacked images,
and a 0.3% improvement in classification accuracy of PGD attacked images
compared with the state-of-the-art results from Mao et al. (2021). Our work
suggests that deep computer vision models’ robustness can be improved by
ensuring that they leverage the rich intrinsic structure of image data.

##  2 Related Work

###  2.1 Adversarial Attacks

Early work in this field demonstrated the susceptibility of neural networks
trained to solve computer vision tasks to human-imperceptible amounts of noise
that would result in high-confidence misclassifications (Szegedy et al.,
2014). Notable early adversarial attack methods include the Fast Gradient Sign
Method (FGSM) from Goodfellow et al. (2015), and Projected Gradient Descent
(PGD) from Madry et al. (2018), in which the attacker attempts to maximize the
loss function of a classification network within the local
$\epsilon$-neighborhood of a particular example. In this paper, we assume
white box access to the network and reversal procedures, and attempt to
optimize attacker performance using the same minimax optimization setup
presented initially by Mao et al. (2021).

###  2.2 Self-supervised Learning

Image data contains rich intrinsic structure that can be used in learning
image representations. In Self-supervised learning for images we use deep
architectures to learn unsupervised tasks including canonical examples like
rotation prediction, inpainting and contrastive predictive coding (Gidaris et
al., 2018; Pathak et al., 2016; Chen et al., 2020). Mao et al. (2021)’s first
noted the transferability of adversarial attacks on classification, a
supervised task, to contrastive learning. While we can’t repair an attacked
image by minimizing the classification loss since we don’t have a ground truth
label, we can still minimize loss for self-supervised tasks. They show the
effectiveness of reversing adversarial attacks by minimizing contrastive loss
thereby implicitly strengthening the inherent structure in the image.

###  2.3 Multitask Learning

In multitask learning, we attempt to learn multiple related tasks at once
using a shared architecture. Heuristically, this leverages the fact that much
of the representational information needed to solve related tasks is shared
(Caruana, 1997). Mao et al. (2020) provide theoretical and empirical results
concerning multitask learning’s ability to enhance robustness of single-task
and multi-task attacks. In this work, we incorporate these theoretical and
empirical observations by leveraging multiple self-supervised tasks to repair
potentially attacked images, instead of one.

##  3 Our Approach

In our approach we improve Mao et al. (2021)’s results by migrating their
self-supervised learning based image repair to a multi-task learning approach
using multiple self-supervised tasks. In this paper we experimented with
SimCLR, Inpainting and Rotation prediction tasks, but note that this is a
somewhat arbitrary and preliminary choice that would benefit from additional
investigation in future work (Gidaris et al., 2018; Pathak et al., 2016; Chen
et al., 2020).

###  3.1 Attack Model

We use a standard attack model in which for a given image ${\bm{x}}$,
classifier $F$ and its loss function $\mathcal{L}_{c}$ (we use cross-entropy
loss, defined as $\mathcal{L}_{c}({\bm{x}},y)=H(F({\bm{x}}),y)$), norm
parameter $p$ ($\infty$ here) and small $\epsilon$, the attacker searches for
an adversarial perturbation ${\bm{x}}_{a}$ where

${\bm{x}}_{a}=\arg\max_{{\bm{x}}_{a}}\mathcal{L}_{c}({\bm{x}}+{\bm{x}}_{a},y):\|{\bm{x}}_{a}\|_{p}<\epsilon$

###  3.2 Reverse Model

Algorithm 1 Multi-task Learning Reverse Attack

1:Potentially attacked image $x$, step size $\eta$, number of iterations $K$,
a classifier $F$, reverse attack bound $\epsilon_{v}$, contrastive loss
$\mathcal{L}_{s}$, roation loss $\mathcal{L}_{r}$ and inpainting loss
$\mathcal{L}_{i}$.

2:Class prediction $\hat{y}$

3:$x^{\prime}\leftarrow x+n$, where $n$ is the initial random noise

4:for $k=1,...,K$ do

5:
$L=\mathcal{L}_{s}({\bm{x}})+\mathcal{L}_{r}({\bm{x}})+\mathcal{L}_{i}({\bm{x}})$

6:
${\bm{x}}^{\prime}\leftarrow{\bm{x}}^{\prime}-\eta(\mathcal{L}_{s}({\bm{x}})/L)\nabla_{{\bm{x}}}\mathcal{L}_{s}({\bm{x}})$

7: if $k\mod 2=0$ then

8:
${\bm{x}}^{\prime}\leftarrow{\bm{x}}^{\prime}-\eta(\mathcal{L}_{r}({\bm{x}})/L)\nabla_{{\bm{x}}}\mathcal{L}_{r}({\bm{x}})$

9: else

10:
${\bm{x}}^{\prime}\leftarrow{\bm{x}}^{\prime}-\eta(\mathcal{L}_{i}({\bm{x}})/L)\nabla_{{\bm{x}}}\mathcal{L}_{i}({\bm{x}})$

11: end if

12:
${\bm{x}}^{\prime}\leftarrow\Pi_{({\bm{x}},\epsilon_{v})}{\bm{x}}^{\prime}$
which projects the image back into the bounded region.

13: ${\bm{x}}\leftarrow{\bm{x}}^{\prime}$

14:end for

15:Predict the final output by $\hat{y}=F({\bm{x}}^{\prime})$

We observe that often ${\bm{x}}_{a}$ is designed in such a way as to disrupt
the inherent structure of the resultant image ${\bm{x}}+{\bm{x}}_{a}$. We can
thus seek to repair this inherent structure by minimizing the loss associated
with our multiple self-supervised task loss function
$\mathcal{L}_{m}({\bm{x}})$, defined as a weighted sum of three self-
supervised task loss functions
$\mathcal{L}_{s}({\bm{x}}),\mathcal{L}_{i}({\bm{x}}),\mathcal{L}_{r}({\bm{x}})$,
explained in detail in the following sections. To reverse an attack for a
given input image ${\bm{x}}^{\prime}$ (which may or may not have been
attacked), classifier $F$, norm parameter $p$ ($\infty$ here) and small
$\epsilon_{r}$, we find reverse vector ${\bm{r}}$,

${\bm{r}}=\arg\min_{{\bm{r}}}\mathcal{L}_{m}({\bm{x}}^{\prime}+{\bm{r}}):\|{\bm{r}}\|_{p}<\epsilon$

After finding a minimal ${\bm{r}}$ via PGD we can recover robust
classifications by classifying on ${\bm{x}}^{\prime}+{\bm{r}}$ (Madry et al.,
2018). See 1 for the full reversal algorithm. Because
$\mathcal{L}_{m}({\bm{x}})$ is a sum of multiple objective functions, a
whitebox attack will have to balance objectives, thus reducing its
effectiveness as an attacker (Mao et al., 2020). As seen in the equation
above, this approach to finding ${\bm{r}}$ is independent of $F$, and is thus
compatible with any classification architecture or supervised task.

Figure 2: On the left see the rotation task, in which a network predicts the
rotation of an image (lower left - 90 degree rotation, upper left - no
rotation) (Gidaris et al., 2018). In the middle see the inpainting task in
which given the upper middle image, a network attempts to predict the lower
middle image (Pathak et al., 2016). On the right see the SimCLR task, in which
a network tries to map augmented images (Crop, gray-scale and rotation shown
here) from the same class closer to one another in a latent space than images
from different classes (Chen et al., 2020)

####  3.2.1 Contrastive Loss: $\mathcal{L}_{s}$

In the SimCLR contrastive learning task introduced by Chen et al. (2020), we
train a ResNet branch to minimize the latent space distance to copies of the
same image under various transformations, and maximize the distance of non-
matching image pairs, thus learning an augmentation-invariant representation
of images (He et al., 2016). More formally, we define our contrastive loss for
an image ${\bm{x}}$ as

$\mathcal{L}_{s}({\bm{x}})=-\mathbb{E}_{i,j}\left[y_{i,j}\log\left(\frac{\exp({\bm{z}}_{i}{\bm{z}}_{j}^{T}/\tau)}{\sum_{k}\exp({\bm{z}}_{i}{\bm{z}}_{k}^{T}/\tau)}\right)\right]$

Where ${\bm{z}}_{i}$ is a possible result from transforming
${\bm{x}}$,${\bm{z}}_{j}$ is a random transformed image, and $y_{i,j}$ is 1 if
${\bm{z}}_{i}$ and ${\bm{z}}_{j}$ originate from the same source image
${\bm{x}}$, and 0 otherwise. $\tau$ is a hyperparameter. For our
augmentations, we sequentially applied random cropping and color jittering,
and also applied a horizontal flip and/or grayscale filter at random to each
image. See 2 for an example.

####  3.2.2 Inpainting Task Loss: $\mathcal{L}_{i}$

In the inpainting task, we train an encoder-decoder deep network to fill in
the missing centers of images, using encoders and decoder structures derived
from the AlexNet architecture (Krizhevsky et al., 2012; Pathak et al., 2016).
For a given whole image ${\bm{x}}$, and context encoder-decoder $F$, we denote
the output of $F$ on ${\bm{x}}$ as $F({\bm{x}})$. We define $\hat{M}$ to be a
binary mask indicating dropped pixels via 1 and 0 otherwise. Letting $\odot$
define the element-wise product operation, we define the inpainting task loss
on a given image ${\bm{x}}$ as

$\mathcal{L}_{i}({\bm{x}})=\|\hat{M}\odot({\bm{x}}-F((1-\hat{M})\odot{\bm{x}}))\|_{2}^{2}$

Intuitively speaking, we may interpret this as the $L_{2}$ norm of the
difference between the true inpainted section of ${\bm{x}}$, and the predicted
inpainted section of the image returned by $F((1-\hat{M})\odot{\bm{x}})$
(predicted using all pixels but the pixels masked by $\hat{M}$, thus giving
the $(1-\hat{M})$ term). See 2 for an example.

####  3.2.3 Rotation Task Loss: $\mathcal{L}_{r}$

In the image rotation task, we train a deep convolutional neural network to
predict the rotation angle of an input image ${\bm{x}}$. We use the
experimental setup given in Gidaris et al. (2018). Given a convolutional
neural network $F$ with learnable parameters $\theta$ designed to predict
image rotations, where $F^{k}({\bm{x}})$ denotes the probability of ${\bm{x}}$
having been transformed by rotation labeled $k$, and ${\bm{x}}^{k}$ denotes
${\bm{x}}$ transformed by rotation labeled $k$ we define the loss:

$\mathcal{L}_{r}({\bm{x}},\theta)=-\frac{1}{K}\sum_{k=1}^{K}\log(F^{k}({\bm{x}}^{k}|\theta))$

Intuitively, we treat this as a classification problem with cross-entropy loss
and take the average loss across a set of K different rotations, trying to
maximize the probability of the correct rotation for each $k$. Minimizing
$\mathcal{L}_{r}({\bm{x}},\theta)$ pushes
$F^{k}({\bm{x}}^{k}|\theta)\rightarrow 1$ as desired. See 2 for an example.

##  4 Results

Classification Accuracy
---
Reversal Method |  Attacked Input |  Clean Input
None |  63.9% |  89.7%
SSL |  65.3% |  86.6%
MTL |  65.6% |  87.7%
Table 1: Using Carmon et al. (2019)’s robustly trained baseline model on
CIFAR-10, we see a 1.1% improvement in classification accuracy of clean
images, and a 0.3% improvement in classification accuracy of PGD attacked
images compared with the state-of-the-art results from Mao et al. (2021)

In line with the literature, we set our attack bounds $\epsilon=8$ (Madry et
al., 2018). In configuring reversal parameters we continue Mao et al. (2021)’s
experiments and similarly find that the defense aware attacker can do no
better than standard PGD attack, and that a reversal with $\epsilon=8$
maximizes attacked input classification accuracy. Our results can are listed
in 1 and our code can be found at
https://github.com/mattlawhon/SelfSupDefense/tree/one-by-one.

###  4.1 Analysis

Using three self-supervised tasks, our approach yields significant improvement
upon the state-of-the-art defense. On the CIFAR-10 dataset using Carmon et al.
(2019)’s robustly trained baseline model, we achieved a 1.1% improvement in
classification accuracy of unattacked images, and a 0.3% improvement in
classification accuracy of PGD attacked images compared with the state-of-the-
art results from Mao et al. (2021). Given an observed standard error of at
most 0.4% in the estimation of the approaches’ true accuracy, this indicates
statistically significant improvement in the classification accuracy of clean
input with over 95% confidence. Our work suggests that deep computer vision
models’ robustness can be improved by ensuring that they leverage the rich
intrinsic structure of image data.

We note that this procedure increases computational cost because it requires
the calculation of two gradients rather than one per iteration of the PGD
reversal procedure. Further, we use different backbones for each self-
supervised task, which improves accuracy of each task at the cost of
performance. This may very well be an acceptable trade-off for better
performance on clean input, though exploring this trade-off is an interesting
direction for future research.

##  5 Conclusions

Though we show statistically significant improvements in overfitting over
baseline, there is much additional work to be done in this domain because the
scope of the experiments presented here is limited. Particularly exciting
avenues for expanding this body of work include: experimenting with other
self-supervised tasks, different datasets and adding a shared backbone for all
self-supervised tasks for a true multi-task learning approach. Further, we
note that this approach broadly furthers the finding that both multi-task
learning and self-supervised learning hold promise to increasing adversarial
robustness.

###  5.1 Acknowledgements

We would like to thank Gustave Ducrest for the productive discussions.

## References

  * Carmon et al. (2019) Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang.  Unlabeled data improves adversarial robustness.  In _Advances in Neural Information Processing Systems_ , volume 32, 2019.
  * Caruana (1997) Rich Caruana.  Multitask learning.  _Machine learning_ , 28(1):41–75, 1997.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pp. 1597–1607. PMLR, 2020.
  * Croce & Hein (2020) Francesco Croce and Matthias Hein.  Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks.  In _International conference on machine learning_ , pp. 2206–2216. PMLR, 2020.
  * Gidaris et al. (2018) Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  In _International Conference on Learning Representations (ICLR)_ , 2018.
  * Goodfellow et al. (2015) Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  In _International Conference on Learning Representations (ICLR)_ , 2015.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In _Advances in Neural Information Processing Systems_ , volume 25, 2012.
  * Madry et al. (2018) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  In _International Conference on Learning Representations_ , 2018.
  * Mao et al. (2020) Chengzhi Mao, Amogh Gupta, Vikram Nitin, Baishakhi Ray, Shuran Song, Junfeng Yang, and Carl Vondrick.  Multitask learning strengthens adversarial robustness.  In _European Conference on Computer Vision_ , pp. 158–174. Springer, 2020.
  * Mao et al. (2021) Chengzhi Mao, Mia Chiquier, Hao Wang, Junfeng Yang, and Carl Vondrick.  Adversarial attacks are reversible with natural supervision.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 661–671, 2021.
  * Pathak et al. (2016) Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 2536–2544, 2016.
  * Szegedy et al. (2014) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  In _International Conference on Learning Representations (ICLR)_ , 2014.
  * Tramèr et al. (2018) Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel.  Ensemble adversarial training: Attacks and defenses.  In _International Conference on Learning Representations (ICLR)_ , 2018.