On Distribution Shift in Learning-based Bug Detectors

Jingxuan Heeth
Luca Beurer-Kellnereth
Martin Vecheveth

ethDepartment of Computer Science, ETH Zurich, Switzerland

Jingxuan<EMAIL_ADDRESS>
Machine Learning, ICML

Deep learning has recently achieved initial success in program analysis tasks such as bug detection. Lacking real bugs, most existing works construct training and test data by injecting synthetic bugs into correct programs. Despite achieving high test accuracy (e.g., 90%), the resulting bug detectors are found to be surprisingly unusable in practice, i.e., 10% precision when used to scan real software repositories. In this work, we argue that this massive performance difference is caused by a distribution shift, i.e., a fundamental mismatch between the real bug distribution and the synthetic bug distribution used to train and evaluate the detectors. To address this key challenge, we propose to train a bug detector in two phases, first on a synthetic bug distribution to adapt the model to the bug detection domain, and then on a real bug distribution to drive the model towards the real distribution. During these two phases, we leverage a multi-task hierarchy, focal loss, and contrastive learning to further boost performance. We evaluate our approach extensively on three widely studied bug types, for which we construct new datasets carefully designed to capture the real bug distribution. The results demonstrate that our approach is practically effective and successfully mitigates the distribution shift: our learned detectors are highly performant on both our test set and the latest version of open source repositories. Our code, datasets, and models are publicly available at <https://github.com/eth-sri/learning-real-bug-detector>.

§ INTRODUCTION

The increasing amount of open source programs and advances in neural code models have stimulated the initial success of deep learning-based bug detectors [Vasic et al., 2019, Hellendoorn et al., 2020, Allamanis et al., 2021, Kanade et al., 2020, Chen et al., 2021]. These detectors can discover hard-to-spot bugs such as variable misuses [Allamanis et al., 2018] and wrong binary operators [Pradel & Sen, 2018], issues that greatly impair software reliability [Rice et al., 2017, Karampatsis & Sutton, 2020] and cannot be handled by traditional formal reasoning techniques.

Lacking real bugs, existing works build training sets by injecting few synthetic bugs, , one [Kanade et al., 2020], three [Hellendoorn et al., 2020], or five [Allamanis et al., 2021], into each correct program. The learned bug detectors then achieve high accuracy on test sets created in the same way as the training set. However, when scanning real-world software repositories, these detectors were found to be highly imprecise and practically ineffective, achieving only 2% precision [Allamanis et al., 2021] or 10% precision [He et al., 2021]. The key question then is: what is the root cause for this massive drop in performance?

Unveiling Distribution Shifts
We argue that the root cause is distribution shift [Koh et al., 2021], a fundamental mismatch between the real bug distribution found in public code repositories and the synthetic bug distribution used to train and evaluate existing detectors. Concretely, real bugs are known to be different from synthetic ones [Yasunaga & Liang, 2021], and further, correct programs outnumber buggy ones in practice, , around 1:2000 as reported in [Karampatsis & Sutton, 2020]. This means that the real bug distribution inherently exhibits extreme data imbalance.

<ref> reproduces the performance drop, showing that existing detectors indeed fail to capture these two key factors. As in [Kanade et al., 2020], we fine-tune a classifier based on  for variable misuse bugs, using a balanced dataset with randomly injected synthetic bugs. Non-surprisingly, the fine-tuned model is close-to-perfect on test set I created in the same way as the fine-tuning set (top-left of <ref>). Then, we replace the synthetic bugs in test set I with real bugs extracted from GitHub to create test set II (top-mid of <ref>). The precision and recall drop by 7% and 56%, respectively, meaning that the model is significantly worse at finding real bugs. Next, we evaluate the classifier on test set III created by adding a large amount of non-buggy code to test set II so to mimic the real-world data imbalance. The model achieves a precision of only 3% (top-right of <ref>). A similar performance loss occurs with graph neural networks (s) [Allamanis et al., 2018], trained on either the dataset for fine-tuning the previous  model (mid row of <ref>) or another balanced dataset where synthetic bugs are injected by  [Allamanis et al., 2021], a learned bug selector (bottom row of <ref>).

This Work: Alleviating Distribution Shifts
In this work, we aim to alleviate such a distribution shift and learn bug detectors capturing the real bug distribution. To achieve this goal, we propose to train bug detectors in two phases: (1) on a balanced dataset with synthetic bugs, similarly to existing works, and then (2) on a dataset that captures data imbalance and contains a small number of real bugs, possible to be extracted from GitHub commits [Allamanis et al., 2021] or industry bug archives [Rice et al., 2017]. In the first phase, the model deals with a relatively easier and larger training dataset. It quickly learns relevant features and captures the synthetic bug distribution. The second training dataset is significantly more difficult to learn from due to only a small number of positive samples and the extreme data imbalance. However, with the warm-up from the first training phase, the model can catch new learning signals and adapt to the real bug distribution. Such a two-phase learning process resembles the pre-training and fine-tuning scheme of large language models [Devlin et al., 2019] and self-supervised learning [Jing & Tian, 2021]. The two phases are indeed both necessary: as we show in <ref>, without either of the two phases or mixing them into one, the learned detectors achieve sub-optimal bug detection performance.

To boost performance, our learning framework also leverages additional components such as task hierarchy [Søgaard & Goldberg, 2016], focal loss [Lin et al., 2017], and a contrastive loss term to differentiate buggy/non-buggy pairs.

Datasets, Evaluation, and Effectiveness
In our work, we construct datasets capturing the real bug distribution. To the best of our knowledge, these datasets are among the ones containing the largest number of real bugs (, 1.7x of PyPIBugs [Allamanis et al., 2021]) and are the first to capture data imbalance, for the bug types we handle. We use half of these datasets for our second phase of training and the other two quarters for validation and testing, respectively. Our extensive evaluation shows that our method is practically effective: it yields highly performant bug detectors that achieve matched precision (or the precision gap is greatly reduced) on our constructed test set and the latest version of open source repositories. This demonstrates that our approach successfully mitigates the challenge of distribution shift and our dataset is suitable for evaluating practical usefulness of bug detectors.

§ BACKGROUND

We now define the bug types our work handles and describe the state-of-the-art pointer models for detecting them.

Detecting Token-based Bugs
We focus on token-based bugs caused by misuses of one or a few program tokens. One example is  where a variable use is wrong and should be replaced by another variable defined in the same scope. Formally, we model a program $p$ as a sequence of $n$ tokens $T=\langle t_1, t_2, \ldots, t_n \rangle$. Fully handling a specific type of token-based bug in $p$ involves three tasks:

* Classification: classify if $p$ is buggy or not.

* Localization: if $p$ is buggy, locate the bug.

* Repair: if $p$ is buggy, repair the located bug.

Note that these three tasks form a dependency where the later task depends on the prior tasks. To complete the three tasks, we first extract $Loc \subseteq T$, a set of candidate tokens from $T$ where a bug can be located. If $Loc = \varnothing$, $p$ is non-buggy, Otherwise, we say that $p$ is eligible for bug detection and proceed with the classification task. We assign a value in $\{\pm 1\}$ to $p$, where $-1$ (, $1$) means that $p$ is non-buggy (, buggy). If $1$ is assigned to $p$, we continue with the localization and repair tasks. For localization, we identify a bug location token $t_{loc} \in Loc$. For repair, we apply simple rules (see <ref>) to extract $Rep$, a set of candidate tokens that can be used to repair the bug located at $t_{loc}$ and find a token $t_{rep} \in Rep$ as the final repair token. $Loc$ and $Rep$ are defined based on the specific type of bug to detect. For example with , $Loc$ is the set of variable uses in $p$ and $Rep$ is the set of variables defined in the scope of the wrong variable use. The above definition is general and applies to the three popular types of token-based bugs handled in this work: , wrong binary operator (), and argument swapping (). These bugs were initially studied in [Allamanis et al., 2018, Pradel & Sen, 2018]. We provide examples of those bugs in <ref> and describe them in detail in <ref>.

Existing Pointer Models for Token-based Bugs
State-of-the-art networks for handling token-based bugs follow the design of pointer models [Vasic et al., 2019], which identify bug locations and repair tokens based on predicted pointer vectors. Given the program tokens $T$, pointer models first apply an embedding method $\phi$ to convert each token $t_i$ into an $m$-dimensional feature vector $h_i \in \mathbb{R}^m$:

\begin{equation*}
[h_1, \ldots, h_n] = \phi(\langle t_1, \ldots, t_n \rangle).
\end{equation*}

Existing works instantiate $\phi$ as GNNs and GREATs [Hellendoorn et al., 2020], LSTMs [Vasic et al., 2019], or BERT [Kanade et al., 2020]. Then, a feedforward network $\pi^{loc}$ is applied on the feature vectors to obtain a probability vector $P^{loc} = [p^{loc}_1, \ldots, p^{loc}_n]$ pointing to the bug location. The repair probabilities $P^{rep} = [p^{rep}_1, \ldots, p^{rep}_l]$ are computed in a similar way with another feedforward network $\pi^{rep}$. We omit the steps for computing $P^{loc}$ and $P^{rep}$ for brevity and elaborate on them in <ref>.

Importantly, in existing pointer models, classification is done jointly with localization. That is, each program has a special  location (typically the first token). When the localization result points to , the classification result is $-1$. Otherwise, the localization result points to a bug location and the classification result is $1$.

Training Existing Pointer Models
For training, two masks, $C^{loc}$ and $C^{rep}$, are required as ground truth labels. $C^{loc}$ sets the index of the correct bug location as $1$ and other indices to $0$. Similarly, $C^{rep}$ sets the indices of the correct repair tokens as $1$ and other indices to $0$. The localization and repair losses are:

\begin{align*}
L^{loc} &= -\sum\nolimits_{i} p^{loc}_i \times C^{loc}[i], \\
L^{rep} &= -\sum\nolimits_{i} p^{rep}_i \times C^{rep}[i].
\end{align*}

The additive loss $L = L^{loc} + L^{rep}$ is optimized. In <ref>, we introduce additional loss terms to $L$.

§ LEARNING DISTRIBUTION-AWARE BUG DETECTORS

Building on the pointer models discussed in <ref>, we now present our framework for learning bug detectors capable of capturing the real bug distribution.

§.§ Network Architecture with Multi-Task Hierarchy

We first describe architectural changes to the pointer model.

Adding Classification Head
In our early experiments, we found that a drawback of pointer models is mixing the classification and localization results in one pointer vector. As a result, the model can be confused by the two tasks. We propose to perform the two tasks individually by adding a binary classification head: We treat the first token $t_1$ as the classification token $t_{[cls]}$ and apply a feedforward network $\pi^{cls}: \mathbb{R}^{m} \rightarrow \mathbb{R}^2$ over its feature vector $h_{[cls]}$ to compute the classification probabilities $p^{cls}_{-1}$ and $p^{cls}_{1}$:

\begin{equation*}
[p^{cls}_{-1},\; p^{cls}_{1}] = \softmax(\pi^{cls}(h_{[cls]})).
\end{equation*}

Task Hierarchy
To exploit the inter-dependence of the $cls$, $loc$, and $rep$ tasks, we formulate a task hierarchy for the pointer model. This allows the corresponding components to reinforce each other and improve overall performance. Task hierarchies are a popular multi-task learning technique [Zhang & Yang, 2021] and are effective in natural language [Søgaard & Goldberg, 2016] and computer vision [Guo et al., 2018]. To the best of our knowledge, this work is the first to apply a task hierarchy on code tasks.

Using a task hierarchy, we process each task one by one following a specific order instead of addressing all tasks in the same layer. Formally, to encode a task hierarchy, we consider the feature embedding function $\phi$ to consist of $k$ feature transformation layers $B_1, \ldots, B_k$, which is a standard design of existing pointer models [Hellendoorn et al., 2020, Allamanis et al., 2021, Kanade et al., 2020]:

\begin{equation*}
\phi(T) =  (B_k \circ B_{k-1} \circ \ldots \circ B_1)(T).
\end{equation*}

We order our tasks $cls$, $loc$ and $rep$, and apply their feedforward networks separately on the last three feature transformation layers. <ref> shows a task hierarchy with the order $[loc, cls, rep]$ and compares it with feature sharing.

§.§ Imbalanced Classification with Focal Loss

To handle the extreme data imbalance in practical bug classification, we leverage the focal loss from imbalanced learning in computer vision and natural language processing [Lin et al., 2017, Li et al., 2020]. Focal loss is defined as:

\begin{equation*}
L^{cls} = \text{FL}([p^{cls}_{-1},\; p^{cls}_{1}], y) = -(1-p^{cls}_y)^{\gamma}\log(p^{cls}_y),
\end{equation*}

where $y$ is the ground truth label. Compared with the standard cross entropy loss, focal loss adds an adjusting factor $(1-p^{cls}_y)^{\gamma}$ serving as an importance weight for the current sample. When the model has high confidence with large $p^{cls}_y$, the adjusting factor becomes exponentially small. This helps the model put less attention on the large volume of easy, negative samples and focus on hard samples which the model is unsure about. $\gamma$ is a tunable parameter that we set to $2$ according to [Lin et al., 2017].

§.§ Two-phase Training

Bug detectors are ideally trained on a dataset containing a large number of real bugs as encountered in practice. However, since bugs are scarce, the ideal dataset does not exist and is hard to obtain with either manual or automatic approaches [He et al., 2021]. Only a small number of real bugs can be extracted from GitHub commits [Allamanis et al., 2021, Karampatsis & Sutton, 2020] or industry bug archives [Rice et al., 2017], which are not sufficient for data intensive deep models. Our two-phase training overcomes this dilemma by utilizing both synthetic and real bugs.

Phase 1: Training with large amounts of synthetic bugs
In the first phase, we train the model using a dataset containing a large number of synthetic bugs and their correct version. Even though learning from such a dataset does not yield a final model that captures the real bug distribution, it drives the model to learn relevant features for bug detection and paves the way for the second phase. We create this dataset following existing work [Kanade et al., 2020]: we obtain a large set of open source programs $P$, inject synthetic bugs into each program $p \in P$, and create a buggy program $p'$, which results in an $1:1$ balanced dataset.

Since $p$ and $p'$ only differ in one or a few tokens, the model sometimes struggles to distinguish between the two, which impairs classification performance. We alleviate this issue by forcing the model to produce distant classification feature embeddings $h_{[cls]}$ for $p$ and $h_{[cls]}'$ for $p'$. This is achieved by a contrastive loss term measuring the cosine similarity of $h_{[cls]}$ and $h_{[cls]}'$:

\begin{equation*}
L^{contrastive} = \cos(h_{[cls]}, h'_{[cls]}).
\end{equation*}

The final loss in the first training phase is the addition of four loss terms: $L^{cls}$, $L^{loc}$, $L^{rep}$, and $\beta L^{contrastive}$, where $\beta$ is the tunable weight of the contrastive loss.

Phase 2: Training with real bugs and data imbalance
The second training phase provides additional supervision to drive the model trained in phase 1 to the real bug distribution. To achieve this, we leverage a training dataset that mimics the real bug distribution. The dataset contains a small number of real bugs (typically hundreds) extracted from GitHub commits, which helps the model to adapt from synthetic bugs to real ones. Moreover, we add a large amount of non-buggy samples (, hundreds of thousands) to mimic the real data imbalance. For more details on how we construct this dataset, please see <ref>. The loss for the second training phase is the sum of the three task losses $L^{cls}$, $L^{loc}$, and $L^{rep}$.

§ IMPLEMENTATION AND DATASET CONSTRUCTION

In this section, we discuss the implementation of our learning framework and our dataset construction procedure.

We implement our framework on top of  [Kanade et al., 2020], a BERT-like model [Devlin et al., 2019] pretrained on source code. Still, our framework is general and can be applied to any existing pointer models (as we show in <ref>, our two-phase training brings improvement to the  model in [Allamanis et al., 2021]). We chose  mainly because, according to [Chen et al., 2021], it achieves top-level performance on many programming tasks, including detecting synthetic  and  bugs. Moreover, the reproducibility package provided by the  authors is of high quality and easy to extend. We present implementation details in <ref>.

Dataset Construction
We focus on Python code. <ref> shows our dataset construction process. Careful deduplication was applied throughout the entire process [Allamanis, 2019]. After construction, we obtain a balanced dataset with synthetic bugs, called , used for the first training phase. Moreover, we obtain an imbalanced dataset with real bugs, which is randomly split into  (used for the second training phase),  (used as the validation set), and  (used as the blind test set). The split ratio is 0.5:0.25:0.25. Instead of splitting by files, we split the dataset by repositories. This prevents distributing files from the same repositories into different splits and requires generalization across codebases [Koh et al., 2021]. Note that we do not evaluate on synthetic bugs as it does not reflect the practical usage. The statistics of the constructed datasets are given in <ref>.

In <ref>, we start the construction with a set of open source repositories (ETH Py150 Open [eth, 2022, Raychev et al., 2016] for  and , and on top of that 894 additional repositories for  to collect enough real bugs). We go over the commit history of the repositories and extract real bugs that align with the bug-inducing rewrite rules of [Allamanis et al., 2021] applied to both versions of a changed file. The repositories are then split into two sets depending on whether any real bug is found. To construct , we extract $\sim$150k eligible functions as non-buggy samples for each bug type from the repositories in which no real bugs were found ([draw=drawgray, line width=0.75pt, pattern color=mypurple, pattern=north west hatch, hatch distance=4.5pt, hatch thickness=1.401pt] (0, 0) rectangle (0.4, -0.25);). Then, we inject bugs into [draw=drawgray, line width=0.75pt, pattern color=mypurple, pattern=north west hatch, hatch distance=4.5pt, hatch thickness=1.401pt] (0, 0) rectangle (0.4, -0.25); to create synthetic buggy samples ([draw=drawgray, line width=0.75pt, pattern color=myyellow, pattern=north west hatch, hatch distance=4.5pt, hatch thickness=1.4pt] (0, 0) rectangle (0.4, -0.25);). The bugs are selected uniformly at random among all bug candidates. We leave the use of more advanced learning methods for bug selection [Patra & Pradel, 2021, Yasunaga & Liang, 2021] as future work.
To construct , we combine the found real bugs ([draw=drawgray, line width=0.75pt, pattern color=myyellow, pattern=north east hatch, hatch distance=4.5pt, hatch thickness=1.4pt] (0, 0) rectangle (0.4, -0.25);) and other eligible functions from the same repositories, which serve as non-buggy samples ([draw=drawgray, line width=0.75pt, pattern color=mypurple, pattern=north east hatch, hatch distance=4.5pt, hatch thickness=1.401pt] (0, 0) rectangle (0.4, -0.25);). Since the number of eligible functions is significantly larger than the number of real bugs, real data imbalance is preserved. Finally, we perform random splitting to obtain datasets as discussed before.

§ EXPERIMENTAL EVALUATION

In this section, we present an extensive evaluation of our framework. We first describe our experimental setup.

Training and Model
We perform training per bug type because the three bug types have different characteristics. We tried a number of training configurations and found that our full method with all the techniques described in <ref> performed the best on the validation set. For , , and , the best orders for the task hierarchy are $[cls, loc, rep]$, $[rep, loc, cls]$, and $[loc, cls, rep]$, respectively. The best $\beta$ for the contrastive loss are 0.5, 4, and 0.5, respectively. We provide training and model details in <ref>.

Testing and Metrics
We perform testing on the  dataset. For space reasons, we only discuss the testing results for  and  in this section. We show the results for  in <ref>.

Instead of accuracy [Allamanis et al., 2021, Hellendoorn et al., 2020, Vasic et al., 2019], we use precision and recall that are known to be better suited for data imbalance settings [wik, 2022, Saito & Rehmsmeier, 2015]. They are computed per evaluation target tgt as follows:

\begin{equation*}
\text{P}^{\texttt{tgt}} = \frac{tp^{\texttt{tgt}}}{tp^{\texttt{tgt}} + fp^{\texttt{tgt}}}, \quad\quad\quad \text{R}^{\texttt{tgt}} = \frac{tp^{\texttt{tgt}}}{\small \#\texttt{buggy}},
\end{equation*}

where $\#\texttt{buggy}$ is the number of samples labeled as buggy. When $tp^{\texttt{tgt}} + fp^{\texttt{tgt}} = 0$, we assign $\text{P}^{\texttt{tgt}} = 0$. We consider three evaluation targets cls, cls-loc, and cls-loc-rep:

* cls: binary classification. $tp^{\texttt{cls}}$ means that the classification prediction and the ground truth are both buggy. A sample is an $fp^{\texttt{cls}}$ when the classification result is buggy but the ground truth is non-buggy.

* cls-loc: joint classification and localization. A sample is a $tp^{\texttt{cls-loc}}$ when it is a $tp^{\texttt{cls}}$ and the localization token is correctly predicted. A sample is an $fp^{\texttt{cls-loc}}$ when it is a $fp^{\texttt{cls}}$ or a $tp^{\texttt{cls}}$ with wrong localization result.

* cls-loc-rep: joint classification, localization, and repair. A sample is a $tp^{\texttt{cls-loc-rep}}$ when it is a $tp^{\texttt{cls-loc}}$ and the repair token is correctly predicted. A sample is an $fp^{\texttt{cls-loc-rep}}$ when it is a $fp^{\texttt{cls-loc}}$ or a $tp^{\texttt{cls-loc}}$ with wrong repair result.

The dependency between classification, localization, and repair determines how a bug detector is used in practice. That is, the users will look at the localization result only when the classification result is buggy. And the repair result is worth checking only when the classification returns buggy and the localization result is correct. Our metrics conform to this dependency by ensuring that the performance of the later target is bounded by the previous target.

During model comparison, we first compare precision and recall without manual tuning of the thresholds to purely assess learnability. We prefer higher precision when the recall is comparable. This is because high precision reduces the burden of manual inspection to rule out false positives. When it is necessary to compare the full bug detection ability, we plot precision-recall curves by varying classification thresholds and compare average precision (AP).

§.§ Evaluation Results on Two-phase Training

We present the evaluation results on our two-phase training.

Changing Training Phases
We create four baselines by only changing the training phases: (i) Only Synthetic: training only on ; (ii) Only Real: training only on ; (iii) Mix: combining  and  into a single training phase; (iv) Two Synthetic: two-phase training, first on  and then on a new training set constructed by replacing the real bugs in  with synthetic ones while maintaining the imbalance. They are compared with Our Full Method in <ref>. We make the following observations:

* Unable to capture data imbalance, Only Synthetic is extremely imprecise (, 10% precision), matching the results from previous works [Allamanis et al., 2021, He et al., 2021]. Such imprecise detectors will flood users with false positives and are practically useless.

* For , Only Real classifies all test samples as non-buggy. For , Only Real has significantly lower recall than Our Full Method. These results show that our first training phase can make the learning stable or improve model performance.

* Mix does not perform well even if it is trained on real bugs. This is because, in the mixed dataset, synthetic bugs outnumber real bugs. As a result, the model does not receive enough learning signals from real bugs.

* By capturing data imbalance, Two Synthetic is more precise than Only Synthetic but sacrifices recall. Moreover, Our Full Method reaches significantly higher precision and recall than Two Synthetic, showing the importance of real bugs in training.

A precision-recall trade-off exists between Only Synthetic, Mix, and Our Full Method. To fully compare their bug classification capability, we plot their precision-recall curves and AP in <ref>. The results show that Our Full Method significantly outperforms Only Synthetic and Mix with 3-4x higher AP. This means that our two-phase training helps the model generalize better to the real bug distribution.

Varying Data Imbalance Ratio
To show how the data imbalance in the second training phase helps the model training, we vary the number of non-buggy training samples and keep the buggy training samples the same, resulting in different non-buggy/buggy ratios (20-6 and the original ratio). The results are plotted in <ref>, showing that the non-buggy/buggy ratio affects the precision-recall trade-off. Moreover, AP increases with data imbalance ratio. From 1:1 to the original ratio, AP increases from 7.40 to 20.96 for  and from 19.84 to 39.85 for .

Varying Amount of Training Data
We also vary the size of our training sets  and . In each experiment, we subsample one training set (with percentage 0, 2, 4, 8, 16, 32, and 64) and fully use the other training set. The subsampling is done by repositories. The results are plotted in <ref>. A general observation is that more training data, in either  or , improves AP. More data in  increases R. For , the model starts to classify samples as buggy only when given a sufficient amount of data from . For , the amount of data in  does not affect the precision. For , we can make consistent observations across  and : first, more data in  improves P; second, as the amount of data in  increases from 0, the R first decreases but then starts increasing from around 16%-32%.

Applying Two-phase Training to Existing Methods
Next, we demonstrate that our two-phase training method can benefit other methods. We consider , a learned bug selector for injecting synthetic bugs, and its  implementation [Allamanis et al., 2021]. We train four  models with different training phases:

* Only Synthetic: train only on .

* Synthetic + Real: two-phase training same as ours, first on  and then on .

* Only : train only on a balanced dataset where bugs are created by .

*  + Real: two-phase training, first on a balanced dataset created by  and then on .

In <ref>, we show the results of the trained variants together with Our Model (Synthetic + Real) which corresponds to Our Full Method. Comparing models trained with Synthetic + Real, we can see that Our Model clearly outperforms  with significantly higher precision and recall. This is likely because Our Model starts from a  model pretrained on a large corpus of code. Moreover, compared with Only Synthetic,  trained with Synthetic + Real achieves significantly higher precision. The same phenomenon also applies when the first training phase is done with . We provide precision-recall curves and AP of the trained variants in <ref>, showing that our second training phase can help , trained with either Only Synthetic or Only , achieve higher AP, especially for  bugs.

§.§ Evaluation Results on Other Techniques

We show the effectiveness of our other techniques with four baselines listed as follows. Each baseline excludes one technique as described below and keeps the other techniques the same as Our Full Method:

* No cls Head: no classification head. Classification and localization are done jointly like existing pointer models.

* No Hierarchy: no task hierarchy. All tasks are performed after the last feature transformation layer.

* No Focal Loss: use cross entropy loss for classification.

* No Contrastive: no contrastive learning with $\beta = 0$.

The above baselines have similar-level recall but noticeably lower precision than Our Full Method. This means all the evaluated techniques contribute to the high performance of Our Full Method. The classification head and task hierarchy play a major role for . We provide results with different task orders and $\beta$ values in <ref>.

§.§ Scanning Latest Open Source Repositories

In an even more practical setting, we evaluate our method on the task of scanning the latest version of open source repositories. To achieve this, we obtain 1118 (, 2339) GitHub repositories for  and  (, ). Those repositories do not overlap with the ones used to construct  and . We apply our full method on all eligible functions in the repositories, without any sample filtering or threshold tuning, and deduplicate the reported warnings together with the extracted real bugs. This results in 427 warnings for , 2102 for , and 203 for .

For each bug type, we manually investigate $100$ randomly sampled warnings and, following [Pradel & Sen, 2018], categorize them into (i) Bugs: warnings that cause wrong program behaviors, errors, or crashes; (ii) Code Quality Issues: warnings that are not bugs but impair code quality (, unused variables), or do not conform to Python coding conventions, and therefore should be raised and fixed; (iii) False Positives: the rest. To reduce human bias, two authors independently assessed the warnings and discussed differing opinions to reach an agreement. We show the inspection statistics in <ref> and present case studies in <ref>. Moreover, we report a number of bugs to the developers and the links to the bug reports are provided in <ref>.

For , most code quality issues are unused variables. For , most warnings are related to ==, !=, is, or is not. Our detector flags the use of == and != for comparing with None, and the use of is (, is not) for equality (, inequality) check with primitive types. Those behaviors, categorized by us as code quality issues, do not conform to Python coding conventions and even cause bugs in rare cases [sof, 2022]. Our model learns to detect them because such samples exist as real bugs in . Depending on the demand on code quality, users might want to turn off such warnings. We simulate this case by filtering out those behaviors and inspect another 100 random warnings from the 255 warnings after filtering. The results are shown in row wrong-binop-filter of <ref>. The bug ratio becomes significantly higher than the original version. For , our model mostly detects bugs with Python standard library functions, such as isinstance and super, or APIs of popular libraries such as TensorFlow. Most false positives are reported on repository-specific functions not seen during training.

The inspection results demonstrate that our detectors are performant and useful in practice. Counting both bugs and code quality issues as true positives, the precision either matches the evaluation results with  ( and ) or the performance gap discussed in <ref> is greatly reduced (). This demonstrates that our method is able to handle the real bug distribution and our dataset can be reliably used for measuring the practical effectiveness of bug detectors.

§ RELATED WORK

We now discuss works most closely related to ours.

Machine Learning for Bug Detection
s [Allamanis et al., 2018], LSTMs [Vasic et al., 2019], and  [Hellendoorn et al., 2020] are used to detect  bugs. Deepbugs [Pradel & Sen, 2018] learns classifiers based on code embeddings to detect , , and incorrect operands bugs. Hoppity [Dinella et al., 2020] learns to perform graph transformations representing small code edits. A number of models are proposed to handle multiple coding tasks including bug detection. This includes PLUR [Chen et al., 2021], a unified graph-based framework for code understanding, and pre-trained code models such as  [Kanade et al., 2020] and CodeBert [Feng et al., 2020].

The above works mainly use datasets with synthetic bugs to train and evaluate the learned detectors. Some spend efforts on evaluation with real bugs but none of them completely capture the real bug distribution: the authors of [Vasic et al., 2019] and [Hellendoorn et al., 2020] evaluate their models on a small set of paired code changes from GitHub (, buggy/non-buggy ratio is $1$:$1$). The PyPIBugs [Allamanis et al., 2021] and ManySStuBs4J [Karampatsis & Sutton, 2020] datasets use real bugs from GitHub commits but do not contain non-buggy samples. Hoppity [Dinella et al., 2020] is trained and evaluated on small code edits in GitHub commits, which are not necessarily bugs and can be refactoring, version changes, and other code changes [Berabi et al., 2021]. Compared with the above datasets, our datasets with real bugs are the closest to the real bug distribution so far.

Other works focus on complex bugs such as security vulnerabilities [Li et al., 2018, Zhou et al., 2019, Chen et al., 2022]. We believe that the characteristics of bugs discussed in our work are general and extensible to complex bugs.

Distribution Shift in Bug Detection and Repair
A few works try to create realistic bugs for training bug detectors or fixers.  [Allamanis et al., 2021] jointly learns a bug selector with the detector to create bugs for training. Since no real bugs are involved in the training process, it is unclear if the learned selector actually constructs realistic bugs. Based on code embeddings, SemSeed [Patra & Pradel, 2021] learns manually defined bug patterns from real bugs, to create new, realistic bugs, which can be used to train bug detectors. Unlike SemSeed, our bug detectors learn directly from real bugs which avoids one level of information loss. BIFI [Yasunaga & Liang, 2021] jointly learns a breaker for injecting errors to code and a fixer for fixing errors. Focusing on fixing parsing and compilation errors, BIFI assumes a perfect external error classifier (, AST parsers and compilers), while our work learns a classifier for software bugs. Namer [He et al., 2021] proposes a similar two-step learning recipe for finding naming issues. Different from our work, Namer relies on manually defined patterns and does not benefit from training with synthetic bugs.

Neural Models of Code
Apart from bug detection, neural models are adopted for a number of other code tasks including method name suggestion [Alon et al., 2019, Allamanis et al., 2016, Zügner et al., 2021], type inference [Wei et al., 2020, Allamanis et al., 2020], code editing [Brody et al., 2020, Yin et al., 2019], and program synthesis [Alon et al., 2020, Brockschmidt et al., 2019, Mukherjee et al., 2021]. More recently, large language models are used to generate real-world code [Austin et al., 2021, Chen et al., 2021].

Code Rewriting for Data Augmentation
Semantics-preserving code rewriting is used for producing programs, , for adversarial training of type inference models [Bielik & Vechev, 2020] and contrastive learning of code clone detectors [Jain et al., 2021]. The rewritten and the original programs are considered to be similar. In the setting of bug detection, however, the programs created by bug-injection rules and the original ones should be considered by the model to be distinct [Patra & Pradel, 2021, Allamanis et al., 2021], which is captured by our contrastive loss.

§ CONCLUSION

In this work, we revealed a fundamental mismatch between the real bug distribution and the synthetic bug distribution used to train and evaluate existing learning-based bug detectors. To mitigate this distribution shift, we proposed a two-phase learning method combined with a task hierarchy, focal loss, and contrastive learning. Our evaluation demonstrates that the method yields bug detectors able to capture the real bug distribution. We believe that our work is an important step towards understanding the complex nature of bug detection and learning practically useful bug detectors.

[eth, 2022]
ETH Py150 Open Corpus, 2022.
URL <https://github.com/google-research-datasets/eth_py150_open>.

[sof, 2022]
What is the difference between "is None" and "== None", 2022.

[wik, 2022]
Wikipedia - Precision and Recall for Imbalanced Data, 2022.

[Allamanis, 2019]
Allamanis, M.
The adverse effects of code duplication in machine learning models of
In Masuhara, H. and Petricek, T. (eds.), Onward!, 2019.
URL <https://doi.org/10.1145/3359591.3359735>.

[Allamanis et al., 2016]
Allamanis, M., Peng, H., and Sutton, C.
A convolutional attention network for extreme summarization of source
In ICML, 2016.
URL <http://proceedings.mlr.press/v48/allamanis16.html>.

[Allamanis et al., 2018]
Allamanis, M., Brockschmidt, M., and Khademi, M.
Learning to represent programs with graphs.
In ICLR, 2018.
URL <https://openreview.net/forum?id=BJOFETxR->.

[Allamanis et al., 2020]
Allamanis, M., Barr, E. T., Ducousso, S., and Gao, Z.
Typilus: neural type hints.
In PLDI, 2020.
URL <https://doi.org/10.1145/3385412.3385997>.

[Allamanis et al., 2021]
Allamanis, M., Jackson-Flux, H., and Brockschmidt, M.
Self-supervised bug detection and repair.
In NeurIPS, 2021.
URL <https://arxiv.org/abs/2105.12787>.

[Alon et al., 2019]
Alon, U., Zilberstein, M., Levy, O., and Yahav, E.
code2vec: learning distributed representations of code.
Proc. ACM Program. Lang., 30 (POPL):0
40:1–40:29, 2019.
URL <https://doi.org/10.1145/3290353>.

[Alon et al., 2020]
Alon, U., Sadaka, R., Levy, O., and Yahav, E.
Structural language models of code.
In ICML, Proceedings of Machine Learning Research, 2020.
URL <http://proceedings.mlr.press/v119/alon20a.html>.

[Austin et al., 2021]
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang,
E., Cai, C. J., Terry, M., Le, Q. V., and Sutton, C.
Program synthesis with large language models.
CoRR, abs/2108.07732, 2021.
URL <https://arxiv.org/abs/2108.07732>.

[Berabi et al., 2021]
Berabi, B., He, J., Raychev, V., and Vechev, M.
Tfix: Learning to fix coding errors with a text-to-text transformer.
In ICML, 2021.
URL <http://proceedings.mlr.press/v139/berabi21a.html>.

[Bielik & Vechev, 2020]
Bielik, P. and Vechev, M.
Adversarial robustness for code.
In ICML, 2020.
URL <http://proceedings.mlr.press/v119/bielik20a.html>.

[Brockschmidt et al., 2019]
Brockschmidt, M., Allamanis, M., Gaunt, A. L., and Polozov, O.
Generative code modeling with graphs.
In ICLR, 2019.
URL <https://openreview.net/forum?id=Bke4KsA5FX>.

[Brody et al., 2020]
Brody, S., Alon, U., and Yahav, E.
A structural model for contextual code changes.
Proc. ACM Program. Lang., 40 (OOPSLA):0
215:1–215:28, 2020.
URL <https://doi.org/10.1145/3428283>.

[Chen et al., 2021]
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J.,
Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger,
G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S.,
Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C.,
Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes,
E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang,
J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr,
A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight,
M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei,
D., McCandlish, S., Sutskever, I., and Zaremba, W.
Evaluating large language models trained on code.
CoRR, abs/2107.03374, 2021a.
URL <https://arxiv.org/abs/2107.03374>.

[Chen et al., 2021]
Chen, Z., Hellendoorn, V. J., Lamblin, P., Maniatis, P., Manzagol, P.-A.,
Tarlow, D., and Moitra, S.
Plur: A unifying, graph-based view of program learning,
understanding, and repair.
In NeurIPS, 2021b.
URL <https://research.google/pubs/pub50846/>.

[Chen et al., 2022]
Chen, Z., Kommrusch, S. J., and Monperrus, M.
Neural transfer learning for repairing security vulnerabilities in c
IEEE Transactions on Software Engineering, 2022.
URL <https://ieeexplore.ieee.org/document/9699412>.

[Devlin et al., 2019]
Devlin, J., Chang, M., Lee, K., and Toutanova, K.
BERT: pre-training of deep bidirectional transformers for language
In NAACL, 2019.
URL <https://doi.org/10.18653/v1/n19-1423>.

[Dinella et al., 2020]
Dinella, E., Dai, H., Li, Z., Naik, M., Song, L., and Wang, K.
Hoppity: Learning graph transformations to detect and fix bugs in
In ICLR, 2020.
URL <https://openreview.net/forum?id=SJeqs6EFvB>.

[Feng et al., 2020]
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B.,
Liu, T., Jiang, D., and Zhou, M.
Codebert: A pre-trained model for programming and natural
In Findings of EMNLP, 2020.
URL <https://doi.org/10.18653/v1/2020.findings-emnlp.139>.

[Guo et al., 2018]
Guo, M., Haque, A., Huang, D., Yeung, S., and Fei-Fei, L.
Dynamic task prioritization for multitask learning.
In ECCV, 2018.
URL <https://doi.org/10.1007/978-3-030-01270-0_17>.

[He et al., 2021]
He, J., Lee, C., Raychev, V., and Vechev, M.
Learning to find naming issues with big code and small supervision.
In PLDI, 2021.
URL <https://doi.org/10.1145/3453483.3454045>.

[Hellendoorn et al., 2020]
Hellendoorn, V. J., Sutton, C., Singh, R., Maniatis, P., and Bieber, D.
Global relational models of source code.
In ICLR, 2020.
URL <https://openreview.net/forum?id=B1lnbRNtwr>.

[Jain et al., 2021]
Jain, P., Jain, A., Zhang, T., Abbeel, P., Gonzalez, J., and Stoica, I.
Contrastive code representation learning.
In EMNLP, 2021.
URL <https://doi.org/10.18653/v1/2021.emnlp-main.482>.

[Jing & Tian, 2021]
Jing, L. and Tian, Y.
Self-supervised visual feature learning with deep neural networks:
A survey.
TPAMI, 430 (11):0 4037–4058, 2021.
URL <https://doi.org/10.1109/TPAMI.2020.2992393>.

[Kanade et al., 2020]
Kanade, A., Maniatis, P., Balakrishnan, G., and Shi, K.
Learning and evaluating contextual embedding of source code.
In ICML, 2020.
URL <http://proceedings.mlr.press/v119/kanade20a.html>.

[Karampatsis & Sutton, 2020]
Karampatsis, R. and Sutton, C.
How often do single-statement bugs occur?: The manysstubs4j dataset.
In MSR, 2020.
URL <https://doi.org/10.1145/3379597.3387491>.

[Koh et al., 2021]
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A.,
Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., Lee, T., David, E., Stavness,
I., Guo, W., Earnshaw, B., Haque, I., Beery, S. M., Leskovec, J., Kundaje,
A., Pierson, E., Levine, S., Finn, C., and Liang, P.
WILDS: A benchmark of in-the-wild distribution shifts.
In ICML, 2021.
URL <http://proceedings.mlr.press/v139/koh21a.html>.

[Li et al., 2020]
Li, X., Sun, X., Meng, Y., Liang, J., Wu, F., and Li, J.
Dice loss for data-imbalanced NLP tasks.
In ACL, 2020.
URL <https://doi.org/10.18653/v1/2020.acl-main.45>.

[Li et al., 2018]
Li, Z., Zou, D., Xu, S., Ou, X., Jin, H., Wang, S., Deng, Z., and Zhong, Y.
Vuldeepecker: A deep learning-based system for vulnerability
In NDSS, 2018.

[Lin et al., 2017]
Lin, T., Goyal, P., Girshick, R. B., He, K., and Dollár, P.
Focal loss for dense object detection.
In ICCV, 2017.
URL <https://doi.org/10.1109/ICCV.2017.324>.

[Mukherjee et al., 2021]
Mukherjee, R., Wen, Y., Chaudhari, D., Reps, T. W., Chaudhuri, S., and
Jermaine, C.
Neural program generation modulo static analysis.
URL <https://arxiv.org/abs/2111.01633>.

[Paszke et al., 2019]
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen,
T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang,
E. Z., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B.,
Fang, L., Bai, J., and Chintala, S.
Pytorch: An imperative style, high-performance deep learning library.
In NeurIPS 2019, 2019.
URL <https://arxiv.org/abs/1912.01703>.

[Patra & Pradel, 2021]
Patra, J. and Pradel, M.
Semantic bug seeding: a learning-based approach for creating
realistic bugs.
In ESEC/FSE, 2021.
URL <https://doi.org/10.1145/3468264.3468623>.

[Pradel & Sen, 2018]
Pradel, M. and Sen, K.
Deepbugs: a learning approach to name-based bug detection.
Proc. ACM Program. Lang., 20 (OOPSLA):0
147:1–147:25, 2018.
URL <https://doi.org/10.1145/3276517>.

[Raychev et al., 2016]
Raychev, V., Bielik, P., and Vechev, M.
Probabilistic model for code with decision trees.
In OOPSLA, 2016.
URL <https://doi.org/10.1145/2983990.2984041>.

[Rice et al., 2017]
Rice, A., Aftandilian, E., Jaspan, C., Johnston, E., Pradel, M., and
Arroyo-Paredes, Y.
Detecting argument selection defects.
Proc. ACM Program. Lang., 10 (OOPSLA):0
104:1–104:22, 2017.
URL <https://doi.org/10.1145/3133928>.

[Saito & Rehmsmeier, 2015]
Saito, T. and Rehmsmeier, M.
The precision-recall plot is more informative than the roc plot when
evaluating binary classifiers on imbalanced datasets.
PloS one, 100 (3):0 e0118432, 2015.
URL <https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4349800/>.

[Søgaard & Goldberg, 2016]
Søgaard, A. and Goldberg, Y.
Deep multi-task learning with low level tasks supervised at lower
In ACL, 2016.
URL <https://doi.org/10.18653/v1/p16-2038>.

[Vasic et al., 2019]
Vasic, M., Kanade, A., Maniatis, P., Bieber, D., and Singh, R.
Neural program repair by jointly learning to localize and repair.
In ICLR, 2019.
URL <https://openreview.net/forum?id=ByloJ20qtm>.

[Wei et al., 2020]
Wei, J., Goyal, M., Durrett, G., and Dillig, I.
Lambdanet: Probabilistic type inference using graph neural networks.
In ICLR, 2020.
URL <https://openreview.net/forum?id=Hkx6hANtwH>.

[Wolf et al., 2019]
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P.,
Rault, T., Louf, R., Funtowicz, M., and Brew, J.
Huggingface's transformers: State-of-the-art natural language
CoRR, abs/1910.03771, 2019.
URL <http://arxiv.org/abs/1910.03771>.

[Yasunaga & Liang, 2021]
Yasunaga, M. and Liang, P.
Break-it-fix-it: Unsupervised learning for program repair.
In ICML, 2021.
URL <http://proceedings.mlr.press/v139/yasunaga21a.html>.

[Yin et al., 2019]
Yin, P., Neubig, G., Allamanis, M., Brockschmidt, M., and Gaunt, A. L.
Learning to represent edits.
In ICLR, 2019.
URL <https://openreview.net/forum?id=BJl6AjC5F7>.

[Zhang & Yang, 2021]
Zhang, Y. and Yang, Q.
A survey on multi-task learning.
IEEE Transactions on Knowledge and Data Engineering, 2021.

[Zhou et al., 2019]
Zhou, Y., Liu, S., Siow, J. K., Du, X., and Liu, Y.
Devign: Effective vulnerability identification by learning
comprehensive program semantics via graph neural networks.
In NeurIPS, 2019.

[Zügner et al., 2021]
Zügner, D., Kirschstein, T., Catasta, M., Leskovec, J., and
Günnemann, S.
Language-agnostic representation learning of source code from
structure and context.
In ICLR, 2021.
URL <https://openreview.net/forum?id=Xh5eMZVONGF>.

§ COMPUTING LOCALIZATION AND REPAIR PROBABILITIES

Now we discuss how pointer models compute the localization probabilities $P^{loc} = [p^{loc}_1, \ldots, p^{loc}_n]$ and the repair probabilities $P^{rep} = [p^{rep}_1, \ldots, p^{rep}_l]$. Given the feature embeddings $[h_1, \ldots, h_n]$ for program tokens $T=\langle t_1, t_2, \ldots, t_n \rangle$, pointer models first compute a score vector $S^{loc} = [s^{loc}_1, \ldots, s^{loc}_n]$ where each score $s^{loc}_i$ reflects the likelihood of token $t_i$ to be the bug location. If $t_i \in Loc$, , token $t_i$ is a candidate bug location, a feedforward network $\pi^{loc}: \mathbb{R}^m \rightarrow \mathbb{R}$ is applied on the feature vector $h_i$ to compute $s^{loc}_i$. Otherwise, it is unlikely that $t_i$ is the bug location so a minus infinity score is assigned. Formally, $s^{loc}_i$ is computed as follows:

\begin{equation*}
s^{loc}_i =
\begin{cases}
\pi^{loc}(h_i) & \text{if } M^{loc}[i] = 1, \\
-\text{inf} & \text{otherwise},
\end{cases}
\end{equation*}

where $M^{loc}$ is the localization candidate mask:

\begin{equation*}
M^{loc}[i] =
\begin{cases}
1 & \text{if } t_i \in Loc, \\
0 & \text{otherwise}.
\end{cases}
\end{equation*}

$S^{loc}$ is then normalized to localization probabilities with the softmax function: $P^{loc} = [p^{loc}_1, \ldots, p^{loc}_n] = \softmax(S^{loc})$.

Depending on the bug type, the set of repair tokens, $Rep$, can be drawn from $T$ (, for  and ) or fixed (, for ). For the former case, $P^{rep}$ is computed in the same way as computing $P^{loc}$, except that another feedforward network, $\pi^{rep}$, and the repair candidate mask, $M^{rep}$, are used instead of $\pi^{loc}$ and $M^{loc}$. When $Rep$ is a fixed set of $l$ tokens, the repair prediction is basically an $l$-class classification problem. We treat the first token $t_1$ of $T$ as the repair token $t_{[rep]}$ and apply $\pi^{rep}: \mathbb{R}^m \rightarrow \mathbb{R}^l$ over its feature vector $h_{[rep]}$ to compute scores $S = \pi^{rep}(h_{[rep]})$. Then, the final repair score is set to $S[i]$ if $M^{rep}$ indicates that the $i$-th repair token is valid, or to minus infinity otherwise. Overall, the repair scores $S^{rep} = [s^{rep}_1, \ldots, s^{rep}_l]$ are computed as follows:

\begin{equation*}
s^{rep}_i =
\begin{cases}
S[i] & \text{if } M^{rep}[i] = 1, \\
-\text{inf} & \text{otherwise},
\end{cases}
\end{equation*}

$S^{rep}$ is then normalized to repair probabilities with the softmax function: $P^{rep} = [p^{rep}_1, \ldots, p^{loc}_l] = \softmax(S^{rep})$.

§ IMPLEMENTATION, MODEL AND TRAINING DETAILS

In this section, we provide details in implementation, models, and training.

Constructing the Test Sets in <ref>
In <ref>, we mention three test sets used to reveal distribution shift in existing learning-based  detectors. Test set I is a balanced dataset with synthetic bugs, created by randomly selecting 336 non-buggy samples from  and injecting one synthetic bug into each non-buggy sample. Test set II is a balanced dataset with real bugs, created by replacing the synthetic bugs in the first test set by the 336 real bugs in . Test set III is .

Three Bug Types Handled by Our Work
The definition of  (,  and ) can be found in [Allamanis et al., 2018, Vasic et al., 2019] (, in [Pradel & Sen, 2018]). To determine $Loc$ and $Rep$, we mainly follow [Allamanis et al., 2021, Kanade et al., 2020] and add small adjustments to capture more real bugs:

* : we include all appearances of all local variables in $Loc$, as long as the appearance is not in a function definition and the variable has been defined before the appearance. When constructing $Rep$ for each bug location variable, we include all local variable definitions that can be found in the scope of the bug location variable, except for the ones that define the bug location variable itself.

* : we deal with three sets of binary operators: arithmetics {+, *, -, /, %}, comparisons {==, !=, is, is not, , =, , =, in, not in}, and booleans {and, or}. If a binary operator belongs to any of the three sets, it is added to $Loc$. The set that the operator belongs to, excluding the operator itself, is treated as $Rep$. The repair candidate mask $M^{rep}$ is of size 17, , it includes all the operators in the three sets. $M^{rep}$ sets the operators in $Rep$ to $1$ and the other operators to $0$.

* : We handle most function arguments but exclude keyworded and variable-length arguments that are less likely to be mistaken. In contrast to the other bug types, we also support the swapping of arguments that consist of more than a single token (., an expression), by simply marking the first token as the bug location or the repair token. Moreover, we consider only functions that have two or more handled arguments. We put all candidate arguments in $Loc$. For each argument in $Loc$, $Rep$ is the other candidate arguments used in the same function.

For bug injection and real bug extraction, we apply the bug-inducing rewriting rules in [Allamanis et al., 2021] given the definitions of $Loc$ and $Rep$ above.

Implementation with
Here, we describe the implementation of our techniques with .  tokenizes the input program into a sequence of sub-tokens. When constructing the masks $M^{\text{loc}}$, $C^{\text{loc}}$, $M^{\text{rep}}$, and $C^{\text{rep}}$ from $Loc$ and $Rep$, we set the first sub-token of each token to $1$. As standard with BERT-like models [Devlin et al., 2019], the first sub-token of the input sequence to  is always [CLS] used as aggregate sequence representation for classification tasks. We also use this token and its corresponding feature embedding for bug classification (all three tasks) and repair (only ).  consists of a sequence of BERT layers and thus naturally aligns with our task hierarchy. Our two-phase training is technically a two-phase fine-tuning procedure when applied to pre-trained models like .  requires the input sequence to be of fixed length, meaning that shorted sequences will be padded and longer sequences will be truncated. We chose length 512 due to contraints on hardware:  is demanding in terms of GPU memory and longer lengths caused out-of-memory errors on our machines. When extracting real bugs and injecting bugs into open source programs, we only consider bugs for which the bug location and at least one correct repair token are within the fixed length. This includes most real bugs we found.

Model Details
is a BERT-Large model with 24 hidden layers, 16 attention heads, 1024 hidden units, and in total  340M parameters. Our classification head $\pi^{cls}$ is a two-layer feedforward network. The localization head $\pi^{loc}$ is just a linear layer. The repair head $\pi^{rep}$ is a linear layer for  and  and a two-layer feedforward network for . The size of the hidden layers is 1024 for all task heads. The implementation of our model is based on Hugging Face [Wolf et al., 2019] and PyTorch [Paszke et al., 2019].

Training Details
Our experiments were done on servers with NVIDIA RTX 2080 Ti and NVIDIA TITAN X GPUs. As described in <ref>, our training procedure consists of two phases. In the first phase, we load a pretrained  model provided by the authors [Kanade et al., 2020] and fine-tune it with . In the second phase, we load the model trained from the first phase and perform fresh fine-tuning with . The number of epochs, learning rate, and the time cost of the two training phases are shown in <ref>. Both training phases require at most two epochs to achieve good performance, highlighting the power of pretrained models to quickly adapt to new tasks and data distributions. In each batch, we feed two samples into the model as larger batch size will cause out-of-memory errors.

For fair comparison, when creating synthetic bugs with , we do not perform their data augmentation rewrite rules for all models. Those rules apply to all models and can be equally beneficial. When training  models with  and , we follow [Allamanis et al., 2021] to use early stopping over . When training with , we use 80 meta-epochs, 5k samples (buggy/non-buggy raito 1:1) per meta-epoch, and 40 model training epochs within one meta-epoch. This amounts to a total of around $6$ days of training time for .

§ MORE EVALUATION RESULTS

In this section, we present additional evaluation results.

Evaluation Results for
We repeat the experiments in <ref> for . The results are shown in <ref>. Most observations that we can make from those results are similar to what we discussed in <ref> for  and . We highlight two difference: first, Our Full Method does not have a clear advantage over Only Synthetic and Mix in terms of AP (see <ref>); second, the data imbalance and the amount of training data do not clearly improve the AP (see <ref>). These different points are likely due to the distinct characteristics of  bugs. We leave it as an interesting future work item to further improve the performance of  detectors.

Parameter Selection for Task Hierarchy and Contrastive Learning
In <ref> (, <ref>), we show the model performance by only changing weight $\beta$ of the contrastive loss (, the task order in our task hierarchy). For  and , Our Full Method (highlighted with $\star$) performs the best among all the configurations. In terms of $\beta$ and , Our Full Method ($\beta=0.5$) is less precise but has significantly higher recall than $\beta=8$. For , Our Full Method performs the best on the validation set but not on the test set.

§ CASE STUDIES OF INSPECTED WARNINGS

§ BUG REPORTS TO THE DEVELOPERS

We report a number of bugs found during our manual inspection as pull requests to the developers. For forked repositories, we trace the buggy code in the original repository. If the original repository has the same code, we create a bug report in the original repository. Otherwise, we do not report the bug. We also found that 7 bugs are already fixed in the latest version of the repository. The links to the pull requests are listed below. We also mark the pull requests for which we received a confirmation from the developers before the deadline for the final version of this paper (two days after we reported them).

(merged) <https://github.com/numpy/numpy/pull/21764>

(merged) <https://github.com/frappe/erpnext/pull/31372>

(merged) <https://github.com/spirali/kaira/pull/31>

(merged) <https://github.com/pyro-ppl/pyro/pull/3107>

(merged) <https://github.com/nest/nestml/pull/789>

(merged) <https://github.com/cupy/cupy/pull/6786>

(merged) <https://github.com/funkring/fdoo/pull/14>

(confirmed) <https://github.com/apache/airflow/pull/24472>

(merged) <https://github.com/python-pillow/Pillow/pull/6370>

(merged) <https://github.com/funkring/fdoo/pull/15>

(false positive) <https://github.com/kovidgoyal/calibre/pull/1658>

(merged) <https://github.com/clinton-hall/nzbToMedia/pull/1889>

(merged) <https://github.com/IronLanguages/ironpython3/pull/1495>

(false positive) <https://github.com/python/cpython/pull/93869>