# CodeQueries: A Dataset of Semantic Queries over Code

Surya Prakash Sahu  Indian Institute of ScienceIndia<EMAIL_ADDRESS>,
Madhurima Mandal  Indian Institute of ScienceIndia ,  Shikhar Bharadwaj
Indian Institute of ScienceIndia ,  Aditya Kanade  Microsoft ResearchIndia
<EMAIL_ADDRESS>,  Petros Maniatis  Google Research, Brain TeamUSA
and  Shirish Shevade  Indian Institute of ScienceIndia<EMAIL_ADDRESS>
(2018)

###### Abstract.

During software development, developers need to identify code spans with
certain semantic properties. Due to upfront costs involved in using
specialized program-analysis frameworks, they resort to manual inspection,
which can be laborious and difficult to scale. We propose an alternative in
the form of program analysis by example, in which the developers provide
labeled examples and leverage a neural model to identify similar cases in
other, unseen code. We formulate this as a novel task of answering semantic
queries over code and prepare a challenging dataset, called CodeQueries, based
on real-world semantic queries over code.

We build upon state-of-the-art pre-trained models of code to predict the
answer and supporting-fact spans. Given a query and code, only some of the
code may be relevant to answer the query. We first experiment under an ideal
setting where only the relevant code is given to the model and show that our
models do well. We then experiment under four pragmatic considerations: (1)
scaling to large-size code containing possibly irrelevant code, (2) learning
from a limited number of examples, (3) robustness to minor syntax errors in
code and (4) robustness to noisy labeling. Our results show that while a
neural model can be resilient to minor syntax errors in code and noisy
labeling, increasing size of code, presence of code that is not relevant to
the query, and reduced number of training examples limit the model
performance. We propose a two-step procedure to improve the ability to handle
large-size code containing possibly irrelevant code. We have released our data
and models111https://github.com/thepurpleowl/codequeries-benchmark to
facilitate future work on the proposed problem of answering semantic queries
over code.

Code understanding, neural modeling, extractive question-answering, developer
productivity

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Software and its engineering
Software post-development issues††ccs: Computing methodologies Multi-task
learning

##  1\. Introduction

Extractive question-answering in natural-language settings is a venerable
domain of NLP, requiring detailed reasoning about a single reasoning step
(“single hop” (DBLP:conf/emnlp/RajpurkarZLL16)) or multiple reasoning steps
(“multi-hop” (DBLP:conf/emnlp/Yang0ZBCSM18)). In the context of programming
languages, neural question answering has not grown to similar complexity:
tasks are either binary yes/no questions (DBLP:conf/acl/HuangTSG0J0D20) or
range over a localized context (e.g., a source-code method)
(DBLP:conf/wcre/BansalEWM21; DBLP:conf/emnlp/Liu021). PM: Does this still
hold?

Recent results show promise towards neural program analyses around complex
concepts such as program invariants (DBLP:conf/nips/SiDRNS18; daikonml),
inter-procedural properties (DBLP:conf/icml/CumminsFBHOL21), and even evidence
of deeper semantic meaning (semanticmeaning). However, it is relatively
challenging to find semantically rich question-answering benchmarks about
code, especially for questions with large scope (entire files) and high
complexity (e.g., multi-hop reasoning). Also, given the criticality of program
analysis, it is relatively hard to judge neural approaches not only on the
answer to questions, but also on the reasoning or evidence for that answer.

In this work, we set out to build such a benchmark for extractive question
answering about code semantics, taking special care to consider large context
and supporting facts, with the intent of helping future neural techniques be
evaluated not only on the accuracy of their answers, but also on their ability
to justify those answers. To produce this benchmark, we used a (symbolic)
program analysis tool and a suite of pre-existing analyses (CodeQL
(DBLP:conf/ecoop/AvgustinovMJS16)), run on a Python code corpus, resulting on
a large number of rich, challenging analysis examples for a diverse set of
concepts.

⬇

1class TCPServer:

2 def __init__(self, service, …): …

3

4 def serve(self, address): …

5

6 # Supporting Fact 1

7 def acceptConnection(self, conn) : …

8 def acceptConnection(self, conn):

9 def handleConnection(self, conn): …

10

11class ThreadingMixin:

12 # Supporting Fact 2

13 def acceptConnection(self, conn) : …

14 t = Thread(target=self.handleConnection, args=(conn,))

15# Answer Span

16 class ThreadedTCPServer (

17 ThreadingMixin, TCPServer) :

18 pass

Figure 1. Example code annotated with the answer and supporting-fact spans for
the conflicting-attributes query.

Figure 2. Methodology for preparing the CodeQueries dataset. All source-code
files are analyzed against each of the 52 CodeQL queries to gather multiple
positive and negative examples for that query. We derive answer spans,
supporting-fact spans and code relevant for answering the query for each
example. The details are discussed in Section 3.

Figure 1 shows an illustrative example. It consists of a compressed view of
some Python code. In the example, the subclass ThreadedTCPServer inherits from
the two base classes ThreadingMixin and TCPServer, both of which define method
acceptConnection. Since both superclasses define the same method, there is a
conflict in resolving method acceptConnection invoked on instances of
ThreadedTCPServer.

To assess the value of this dataset, we consider various baseline neural
approaches, varying in the amount of model customization, representational
customization (e.g., using annotations versus raw text), and the presence of
supporting facts. Specifically, we study the ability of a Large Language Model
(GPT3.5) that has seen extensive natural language and code to tackle the
dataset task without the benefit of fine-tuning, with various amounts of
prompting. We also study a much smaller but more custom model, fine-tuned from
CuBERT (DBLP:conf/icml/KanadeMBS20).

We find that …

We have released our data and models to facilitate future work on the proposed
problem of answering semantic queries over code at
https://github.com/thepurpleowl/codequeries-benchmark.

##  2\. Related Work

###  2.1. Learning-based Program Analysis

Use of program analysis helps improve software quality. However, implementing
analysis algorithms requires expertise and effort. There is increasing
interest in using machine learning for program analysis. Recent work in this
direction includes learning loop and other invariants
(DBLP:conf/nips/SiDRNS18; daikonml), rules for static analysis
(DBLP:conf/cav/BielikRV17), intra- and inter-procedural data flow analysis
(DBLP:conf/icml/CumminsFBHOL21), specification inference
(10.1145/3296979.3192383; DBLP:conf/pldi/ChibotaruBRV19), reverse engineering
(nero), and type inference (10.1145/3236024.3236051;
DBLP:journals/corr/abs-2004-00348; DBLP:conf/sigsoft/PradelGL020;
DBLP:conf/iclr/WeiGDD20; DBLP:journals/corr/abs-2101-04470; peng2022static).
These techniques target specific analysis problems with specialized program
representations or learning methods. Our work targets semantic queries over
code and presents a uniform extractive question-answering setup for them. Our
queries cover diverse program analyses involving forms of type checking,
control-flow and data-flow analyses, and many other checks (see Appendix B for
the list of queries). In another line of work, codetrek; codetrek-walks
advocate the use of relational representations of code, as used in CodeQL, in
neural modeling and use them on classification tasks.

GitHub has recently launched an experimental feature222
https://github.blog/2022-02-17-code-scanning-finds-vulnerabilities-using-
machine-learning/ that uses machine learning to classify JavaScript and
TypeScript code with regards to four common vulnerabilities. Similar to our
work, they built the training set using pre-existing CodeQL queries (written
to detect those vulnerabilities). They expect a classifier to catch cases
missed by incomplete or stale CodeQL queries. They use relational
representation of code built by the CodeQL engine and take help from human
experts in feature selection333 https://github.blog/2022-02-17-leveraging-
machine-learning-find-security-vulnerabilities/ . In contrast, we learn
directly on source code. They perform binary classification to surface
security alerts, whereas our goal is extractive program analysis to aid
developers in code understanding. Despite these technical differences, we
share the motivation that machine learning can be used to ease the burden of
manually writing or maintaining detailed, formal analyses.

###  2.2. Natural-language Questions and Queries about Code

CoSQA (DBLP:conf/acl/HuangTSG0J0D20) includes yes/no questions to determine
whether a web search query and a method match. DBLP:conf/wcre/BansalEWM21 and
CodeQA (DBLP:conf/emnlp/Liu021) are two recent works on question-answering
over code. Both consider a method as the code context, and programmatically
extract question-answer pairs specific to the method from the method body and
comments. DBLP:conf/wcre/BansalEWM21 generate questions about method
signatures (e.g., what are parameter and return types), (mis)matches between a
function and a docstring, and natural-language function summaries. CodeQA is
generated from code comments using rule-based templates. The answers are
natural-language sentences extracted from code comments using NLP techniques.
The code in our case can be larger, encompassing multiple methods and classes;
queries are about semantic aspects of code and may require long chains of
reasoning; and answers are spans over code. CS1QA (lee2022cs1qa) is a dataset
of question-answering in an introductory programming course and proposes
classification of the question into pre-defined types, identification of
relevant source-code lines and retrieval of related QAs. In an orthogonal
direction, natural language queries have been used for code retrieval
(DBLP:conf/icse/GuZ018; DBLP:journals/corr/abs-1909-09436;
DBLP:conf/sigsoft/CambroneroLKS019; DBLP:journals/corr/abs-2008-12193;
gu2021cradle).

###  2.3. Question-answering over Text

Various datasets for extractive question-answering over text requiring single-
hop (DBLP:conf/emnlp/RajpurkarZLL16) and multi-hop
(DBLP:conf/emnlp/Yang0ZBCSM18) reasoning have been proposed. Our dataset
consists of queries requiring single- and multi-hop reasoning over code. Along
the lines of prior work (clark2017simple; DBLP:conf/acl/RajpurkarJL18), we
include negative examples in which the queries cannot be answered with the
given context, though the context contains plausible answers
(DBLP:conf/emnlp/Yang0ZBCSM18). For improving explainability, we also include
in our dataset and models prediction of supporting facts
(DBLP:conf/emnlp/Yang0ZBCSM18); supporting-fact supervision might also be
helpful in alternative chain-of-thought methodologies (chain-of-thought;
yao2023react). We experiment on large-size code which may contain parts that
are not relevant to the query. This is analogous to distractor paragraphs
(DBLP:conf/emnlp/Yang0ZBCSM18) and requires the models to deal with spurious
information.

##  3\. Dataset Preparation

###  3.1. Query Evaluation

To prepare the CodeQueries dataset, we evaluated the queries from a standard
suite of CodeQL (querySuite) on the open, redistributable subset
(DBLP:conf/icml/KanadeMBS20) of the ETH Py150 dataset of Python programs
(raychev2016probabilistic) (the ETH Py150 Open dataset). These queries are
written by experts and identify coding issues pertaining to correctness,
reliability, maintainability and security of code. We evaluated each query on
individual Python files (Figure 2). To get a reasonable number of positive
examples for each query, we selected queries with at least 50 answer spans in
the training split of the ETH Py150 Open dataset. This gave us a suite of 52
queries and corresponding positive examples. The query definitions build upon
specialized CodeQL libraries and have 17–689 lines of code with an average of
61 lines. We inspected the definition of a query to check whether answering it
requires a single reasoning step or multiple reasoning steps, and classified
the query accordingly as a _single-hop_ or _multi-hop_ query. Out of the 52
queries, 15 are multi-hop and 37 are single-hop. We refer to them as _positive
queries_. To obtain negative examples, we derived negated versions of these
positive queries called _negative queries_ (Section A.2). The negative
examples are obtained by evaluating the negative queries on Python files.

We identify the answer and supporting-fact spans from the results produced by
the CodeQL engine for each of the positive queries. These spans come from a
wide variety of syntactic patterns, making it non-trivial for a model to
identify the right candidates for answering the queries. In all, there are 42
different syntactic patterns of spans such as class declarations, with
statements, and list comprehensions. We give the statistics of syntactic
patterns of spans in Appendix C. Note that negative examples do not have
answer or supporting-fact spans.

###  3.2. Deriving Relevant Code Blocks and Labeled Examples

A CodeQL query produces answers based only on specific parts of code within a
file, e.g., a set of classes within the file or a set of methods within a
class in the file. We inspect the query definitions and automate extraction of
the query-relevant parts from a file. Given the query results, we
programmatically obtain the code blocks needed for arriving at the same
results for the query (see the details below). We call them _relevant code
blocks_.

We represent a _labeled example_ in our dataset as a tuple $(Q,C,A,SF)$ where
$Q$ is the unique identifier of a query, $C$ is a set of relevant code blocks
in a file over which the query is evaluated, $A$ is the set of answer spans
over $C$ and $SF$ is the set of supporting-fact spans over $C$. Each span is a
tuple $(b,i,j)$ which identifies the code block $b\in C$ and the start and end
indexes, $i$ and $j$ respectively, over tokens in $b$. By definition, the sets
$A$ and $SF$ are empty for negative examples. The query identifier $Q$ is used
for uniquely identifying which query among the 52 queries is being applied on
$C$. The formal query definitions are used only for preparing the dataset.

Details. We explain the procedure to derive labeled examples starting from the
results returned by the CodeQL engine. Given the answer and supporting-fact
spans of a single-hop query in a file, we select the method to which the spans
belong as the code to be queried upon. The locations of the spans within the
method are treated as the ground truth. A class may contain class-level
statements that do not fall within any method, e.g., declarations of class
variables. If the spans fall within class-level statements, we collate all the
class-level statements together as the code to be queried. Similarly, a file
may contain file-level statements that do not fall within any class or method,
e.g., the import statements. We collate the file-level statements together to
form an example for a single-hop query if the spans fall within file-level
statements. We generically refer to the statements of a method (including the
method declaration), the class-level statements within a class (including the
class declaration), or the file-level statements within a file as a _code
block_. Single-hop queries involve analyzing individual code blocks in
isolation.

A multi-hop query analyzes multiple code blocks to produce an answer. However,
the result returned by the CodeQL engine does not explicitly identify which
code blocks were required for the analysis. We therefore inspect the CodeQL
query definitions to identify the steps used by the query to select code
blocks for analysis. For example, the conflicting-attributes query selects a
class and all of its superclass(es). Another query may analyze code within
only a single class. Given the answer and supporting-fact spans for a query,
we programmatically obtain the set of code blocks needed for arriving at the
same results for the given query. We implement this procedure using the tree-
sitter parsing library (treeSitter). A labeled example for a multi-hop query
also identifies the relevant code blocks needed to answer the query.

As discussed in Section A.2, we create negative queries to obtain negative
examples. The procedure to derive negative examples from the results of
negative queries is the same as described above. However, the resulting
examples are labeled with an empty set of spans as ground truth, meaning that
a model should not return any spans on the corresponding code with respect to
the original (positive) query.

| Training | Validation | Test
---|---|---|---
Files | 74,749 | 8,302 | 41,457
1-hop positive examples | 29,202 | 3,216 | 16,019
N-hop positive examples | 13,893 | 1,528 | 7,745
1-hop negative examples | 40,111 | 4,244 | 22,346
N-hop negative examples | 19,756 | 2,195 | 11,091
Total examples | 102,962 | 11,183 | 57,201

Table 1. Dataset statistics.

Figure 3. The span prediction setup.

###  3.3. Dataset Statistics

Table 1 gives the statistics of the CodeQueries dataset according to the
splits of the ETH Py150 Open dataset. We place the examples derived from a
Python file in the same split as the file. We train the neural models on the
train split comprising examples from the 52 queries and evaluate them to
answer the test split examples of those queries. The table also gives the
number of positive and negative examples for single-hop and multi-hop queries
(see Appendix B for detailed query-wise statistics).

##  4\. Modeling and Metrics

###  4.1. Span Prediction Model

Input representation and encoding. Given a query identifier $Q$ and a set of
code blocks $C$, we prepare an input sequence by concatenating $Q$ and the
code blocks in $C$. The code blocks are ordered by their order of appearance
in the code files they are extracted from. They are separated by a special
[SEP] token and the entire sequence is preceded with the [CLS] token, similar
to BERT (DBLP:conf/naacl/DevlinCLT19). We use the pre-trained CuBERT
(DBLP:conf/icml/KanadeMBS20) and CodeBERT (DBLP:conf/emnlp/FengGTDFGS0LJZ20)
models as encoders of input sequences. These models use subword vocabularies.
Therefore, the input sequences are represented as sequences of subword tokens
from the respective vocabularies.

We also use the GraphCodeBERT model (guo2020graphcodebert), which additionally
embeds data-flow information about code. Our code representation is the same
as GraphCodeBERT: code blocks are concatenated and separated by a special
delimiter, followed by the sequence of data-flow graph nodes for each of the
code blocks; data-flow and variable-alignment edges are represented as
attention masks over node sequences and code blocks.

Output representation and span prediction layer. Let $\\{B,I,O\\}$
respectively indicate Begin, Inside and Outside labels (DBLP:conf/acl-
vlc/RamshawM95). An answer span is represented by a sequence of labels such
that the first token of the answer span is labeled by a $B$ and all the other
tokens in the span are labeled by $I$’s. We use an analogous encoding for
supporting-fact spans, but we use the $F$ label instead of $B$ to distinguish
facts from answers. The use of a distinct label ($B$ or $F$) for the first
token of a span helps distinguish the spans even if they are contiguous to
each other in the code. Any token that does not belong to either kind of span
is labeled by an $O$. This allows us to represent multiple answer or
supporting-fact spans for the given code blocks in a single sequence over
$\\{B,I,O,F\\}$ labels. Note that this does not allow overlap between spans
(e.g., between an answer and a supporting fact), which we have empirically
found not to be a problem in our dataset.

The span prediction layer consists of a token classifier that performs a four-
way classification over the labels $\\{B,I,O,F\\}$. It is applied to the
encoding of every code token in the last layer of the encoder. We finetune the
model by minimizing the cross-entropy loss. Note that in the case of negative
examples, all tokens are to be classified as $O$. Figure 3 shows the setup for
span prediction. The symbols $\text{Q}_{i}$ and $\text{C}_{j}$ denote subword
tokens of the query identifier and code, respectively. For simplicity, we do
not explicitly show the special delimiter tokens such as [CLS] and the data-
flow information encoded in the GraphCodeBERT based model.

###  4.2. Two-step Procedure of Relevance Classification and Span Prediction

Figure 4. Two-step procedure to handle large-size code containing possibly
irrelevant code blocks.

As discussed in Section 3.2, we identify the relevant code blocks
programmatically using the CodeQL results during data preparation. We shall
use this information to simulate an ideal setting for evaluating our models
(Section 5.2). However, in practice given new code, the information about
which code blocks are relevant for answering a query is unknown, and a
developer may provide large-size code (e.g., an entire file) which could
contain large chunks of code irrelevant to the query. The presence of
irrelevant code may confuse the span prediction model. Further, large code
could overflow the maximum input sequence length the model can handle and the
model could lose critical information when the input is trimmed to fit its
input constraint. We devise a two-step procedure (see Figure 4) to deal with
this:

  1. (1)

Step 1: We first apply a _relevance classifier_ to every block in the given
code and select code blocks that are likely to be relevant for answering a
given query.

  2. (2)

Step 2: We then apply the span prediction model (Section 4.1) to the set of
selected code blocks to predict answer and supporting-fact spans.

Training relevance classifier. Recall that we evaluate CodeQL queries on
individual Python files to create the labeled dataset. Let $F$ be a file and
$R$ be the set of code blocks in $F$ that are relevant for a query $Q$. We
treat other blocks in $F$ as irrelevant. We form a labeled example for
training the relevance classifier as a tuple $(Q,b,Label)$ where $Label$ is
$True$ if the block $b\in R$ otherwise $False$. We concatenate $Q$ and the
contents of $b$ to create the input sequence for a model. We separate the two
by [SEP] and prepend the whole sequence by the [CLS] token. A binary
classification layer is applied on the encoding of the [CLS] token. We
finetune pre-trained models of code on these sequences using the binary cross-
entropy loss over the ground-truth labels. Note that these relevance
classifiers are distinct from the span prediction models.

Using relevance classifier. At inference time, given a query identified by $Q$
and a file comprising code blocks $\\{b_{1},\ldots,b_{n}\\}$, we generate a
set of $n$ examples by concatenating $Q$ and the contents of each of $b_{i}$.
The relevance classifier is applied on each of these examples and all blocks
classified as relevant are selected. The selected blocks and the query are
passed to the span prediction model as shown in Figure 4.

###  4.3. Evaluation Metrics

We measure the performance of a span prediction model as the percentage of
examples for which the set of predicted answer spans is same as the set of
ground-truth answer spans, and the set of predicted supporting-fact spans is
same as the set of ground-truth supporting-fact spans. We call this metric the
_exact match_. For a relevance classification model, we measure the usual
classification metrics: accuracy, precision, and recall.

##  5\. Experimental Results

###  5.1. Training Setup

Hyper-parameter | Experimented Values | Best Value
---|---|---
Number of layers | 3-6 | 3
Attention heads | {8, 16} | 8
Embedding dimension | {512, 1024} | 512
Hidden dimension | {1024, 2048} | 2048
Table 2. Hyper-parameter sweep for the baseline Transformer model.

Model configurations. We use state-of-the-art pre-trained encoders of code:
CuBERT (DBLP:conf/icml/KanadeMBS20), CodeBERT
(DBLP:conf/emnlp/FengGTDFGS0LJZ20) and GraphCodeBERT (guo2020graphcodebert).
These models come with different input size restrictions. For CuBERT and
CodeBERT, checkpoints are available for input length of 512. For CuBERT, a
checkpoint for input length of 1024 is also available. The GraphCodeBERT model
allocates input length of 512 for code tokens and 128 for data-flow graph
nodes. We use all these available checkpoints for experimentation. We omit the
architectural details of these pre-trained models and refer the reader to the
respective papers for the details. The inputs are tokenized with the
respective subword vocabularies, and then are pruned or padded to match the
model’s expected sequence length.

As a non-pre-trained baseline, we train a Transformer encoder with input
length of 1024 from scratch. We used the CuBERT vocabulary for this
Transformer encoder but trained the token embeddings in an end-to-end manner.
Table 2 gives the hyper-parameters and the best configuration according to
exact match on the validation set.

Span prediction. The token encodings from the final hidden layer of an encoder
are passed through a dropout layer with dropout probability 0.1 followed by a
classification layer. Each model is trained jointly on the training split of
the complete dataset spanning all 52 queries. We initially experimented with
up to 10 epochs and learning rates in the order of $10^{-5}$ and $10^{-6}$ for
these models. We observed that the models reached minimum validation loss with
the following configurations and used them: Finetuning is performed for 5
epochs for the 512-length models and for 3 epochs for the 1024-length models.
The learning rate of $3\times 10^{-5}$ is used for CuBERT and CodeBERT, and
$5\times 10^{-5}$ is used for GraphCodeBERT. The non-pre-trained baseline
Transformer model is trained for 40 epochs with the learning rate of $3\times
10^{-5}$ . The checkpoints are selected by least validation loss.

Based on the memory constraints, we used batch sizes of 4 and 16 for sequence
lengths 1024 and 512 respectively. All the models are trained by minimizing
the cross-entropy loss using the AdamW optimizer
(DBLP:journals/corr/abs-1711-05101) and linear scheduling without any warmup.
We finetuned the CuBERT 1024 model on specialized datasets for ablation of
supporting facts and query-specific models (Section 5.2) and under limited
number of per-query examples (Section 5.3). We used the same hyper-parameters
for these experiments as stated for the CuBERT 1024 model above.

Relevance classification. For relevance classification, we finetuned the pre-
trained CuBERT model with input length limit of 512. The pooled output is
passed through a dropout layer with dropout probability of 0.1 and a 2 layer
classifier with hidden-dimension of 2048. We finetuned it for 5 epochs with
learning rate of $3\times 10^{-6}$ and weighted crossentropy (with weights 1/2
for irrelevant/relevant class) loss. The best checkpoint is decided based on
least validation loss.

System configuration. All experiments were performed on a 64 bit Debian system
with an NVIDIA Tesla A100 GPU having 40GB GPU memory and 85GB RAM.

Models | All | Positive | Negative
---|---|---|---
Transformer | 66.38 | 22.50 | 97.57
CuBERT | 81.76 | 59.77 | 97.38
CodeBERT | 82.13 | 62.67 | 95.96
GraphCodeBERT | 82.31 | 61.08 | 97.40
CuBERT-1K | 86.70 | 72.51 | 96.79

Table 3. Exact-match results in the ideal setting.

Figure 5. Span-wise distribution of exact match.

###  5.2. Experimentation under an Ideal Setting with only Relevant Code

We now evaluate our models on test split examples containing only relevant
code; this is the _ideal_ evaluation setting since the models do not have to
deal with irrelevant code.

Exact-match results. Table 3 reports the exact-match metrics for all the
models: Column _All_ refers to results on all (both positive and negative)
examples, _Positive_ and _Negative_ refer to results only on positive or
negative examples. CuBERT and CuBERT-1K refer to the 512 and 1024 length
models respectively. The CuBERT-1K model achieves the maximum exact match of
86.70% on all examples. We use the _best performing model_ , CuBERT-1K, to
further characterize the dataset and assess model performance.

Table 4. Query-wise results for the multi-hop queries. |  |  |  |
---|---|---|---|---
|  | Exact Match
Index |  Query Name |  All
Examples |  Positive |  Negative
Q1 |  Unused import |  78.30 |  52.10 |  95.35
Q2 |  Missing call to __init__ during object initialization |  88.06 |  85.45 |  90.67
Q3 |  Use of the return value of a procedure |  75.32 |  50.34 |  96.49
Q4 |  __eq__ not overridden when adding attributes |  56.18 |  63.90 |  11.90
Q5 |  Wrong number of arguments in a call |  48.97 |  14.81 |  76.30
Q6 |  Comparison using is when operands support __eq__ |  82.11 |  64.63 |  95.37
Q7 |  Signature mismatch in overriding method |  51.58 |  2.97 |  92.50
Q8 |  Non-callable called |  59.21 |  8.82 |  100.00
Q9 |  __init__ method calls overridden method |  34.27 |  2.82 |  65.28
Q10 |  Conflicting attributes in base classes |  47.46 |  16.95 |  77.97
Q11 |  __iter__ method returns a non-iterator |  79.25 |  96.77 |  54.55
Q12 |  Flask app is run in debug mode |  97.47 |  97.50 |  97.44
Q13 |  Inconsistent equality and hashing |  64.06 |  71.88 |  56.25
Q14 |  Wrong number of arguments in a class instantiation |  48.61 |  23.53 |  71.05
Q15 |  Incomplete ordering |  63.79 |  48.28 |  79.31
Aggregate |  76.82 |  52.19 |  94.01

All the models have excellent exact-match accuracy for the negative examples;
meaning that they are successful in identifying unanswerable contexts. On the
positive examples, the finetuned models achieve accuracy in the range of
59.77–72.51%. Predicting spans for positive examples requires accurately
identifying both the beginning token and all the other tokens that form the
span, whereas for a negative example it suffices to predict that no token
belongs to a span. We believe that the relative gap in the performance of the
models between positive and negative examples stems from this difference. The
significant gap between the baseline Transformer model, which is trained from
scratch, and the finetuned models shows that pre-training provides a clear
advantage on this dataset.

Query-wise analysis. We carried out query-wise analysis of the predictions of
the CuBERT-1K model. Tables LABEL:tab:multihop-results and
LABEL:tab:singlehop-results give the query-wise results for the multi-hop and
single-hop queries respectively, with minimum and maximum results highlighted
in bold face. In addition, detailed statistics for the individual queries in
terms number of examples, average number of tokens, average number of spans,
etc. are provided in Appendix B.

Among the multi-hop queries: On positive examples, the model struggles the
most on the top-2 queries (Q9 and Q7) by the average number of tokens in
examples (see Table LABEL:tab:querywise-multihop in Appendix B); and the query
Q12 has the highest exact match due to the simplicity of the query. On
negative examples, the query Q8 works best at the cost of positive examples;
and the worst performing query Q4 has the smallest number of negative examples
(see Table LABEL:tab:querywise-multihop in Appendix B).

Among the single-hop queries, there are 6 queries with fewer than 100 positive
examples (see Table LABEL:tab:querywise-singlehop in Appendix B). Of these,
except for the query Q49, all others are among the worst performing. The
queries Q36 and Q38 are simple and are the best performing on positive
examples. On negative examples, the model achieves very high exact match
($>$95%) for several queries; and the query Q42 has the lowest exact match.

Table 5. Query-wise results for the single-hop queries. |  |  |  |
---|---|---|---|---
|  | Exact Match
Index |  Query Name |  All
Examples |  Positive |  Negative
Q16 |  Unused local variable |  91.78 |  82.90 |  98.34
Q17 |  Except block handles BaseException |  96.56 |  92.71 |  99.82
Q18 |  Imprecise assert |  98.64 |  96.25 |  99.95
Q19 |  Variable defined multiple times |  78.56 |  50.13 |  95.44
Q20 |  Testing equality to None |  97.31 |  93.82 |  100.00
Q21 |  Unreachable code |  78.07 |  56.04 |  96.68
Q22 |  First parameter of a method is not named self |  99.93 |  99.86 |  100.00
Q23 |  Unnecessary pass |  95.79 |  91.28 |  99.39
Q24 |  Module is imported with import and import from |  79.85 |  67.67 |  91.24
Q25 |  Module is imported more than once |  72.18 |  50.58 |  91.62
Q26 |  Comparison of constants |  93.79 |  42.86 |  95.84
Q27 |  Implicit string concatenation in a list |  83.17 |  62.60 |  98.31
Q28 |  Suspicious unused loop iteration variable |  86.79 |  79.38 |  93.04
Q29 |  Duplicate key in dict literal |  59.69 |  10.71 |  97.26
Q30 |  Unnecessary else clause in loop |  86.92 |  74.53 |  99.07
Q31 |  First argument to super() is not enclosing class |  95.65 |  92.98 |  98.28
Q32 |  Redundant assignment |  91.24 |  83.53 |  97.25
Q33 |  An assert statement has a side-effect |  84.52 |  62.90 |  98.92
Q34 |  Nested loops with same variable |  53.53 |  31.71 |  73.86
Q35 |  Import of deprecated module |  72.63 |  42.05 |  99.02
Q36 |  NotImplemented is not an Exception |  100.00 |  100.00 |  100.00
Q37 |  Redundant comparison |  56.91 |  12.99 |  89.42
Q38 |  Deprecated slice method |  100.00 |  100.00 |  100.00
Q39 |  Constant in conditional expression or statement |  88.31 |  67.92 |  99.01
Q40 |  Comparison of identical values |  83.46 |  58.00 |  98.80
Q41 |  import $\star$ may pollute namespace |  93.02 |  87.50 |  98.46
Q42 |  Unnecessary delete statement in function |  72.63 |  77.55 |  67.39
Q43 |  Illegal raise |  86.73 |  85.11 |  88.24
Q44 |  Insecure temporary file |  95.65 |  90.91 |  100.00
Q45 |  Modification of parameter with default |  60.00 |  9.09 |  92.31
Q46 |  Should use a with statement |  95.00 |  92.50 |  97.50
Q47 |  Special method has incorrect signature |  82.43 |  78.38 |  86.49
Q48 |  Non-standard exception raised in special method |  95.31 |  96.88 |  93.75
Q49 |  Use of global at module level |  92.06 |  79.17 |  100.00
Q50 |  Modification of dictionary returned by locals() |  75.86 |  47.83 |  94.29
Q51 |  Incomplete URL substring sanitization |  76.71 |  39.29 |  100.00
Q52 |  Unguarded next in generator |  42.86 |  13.04 |  69.23
Aggregate |  91.55 |  82.34 |  98.16

| Answer Match
---|---
Setting | All | Positive | Negative
With supporting facts | 87.41 | 73.76 | 97.11
Without supporting facts | 86.96 | 73.67 | 96.41
Table 6.  Results of ablation with respect to supporting facts. “Answer Match"
is exact match but only on answer spans.

Setting | All | Positive | Negative
---|---|---|---
Original IDs | 86.70 | 72.51 | 96.79
Modified IDs | 86.30 | 71.76 | 96.62

Table 7. Comparison of difference choices of query identifiers. Original IDs
are query names and modified IDs are selected from
$Q_{0},Q_{1},\ldots,Q_{51}$.

Effect of number of spans. Figure 5 is a Radar chart of span-wise distribution
of exact match for the CuBERT-1K model. The number of spans per example range
from $0$ to $>20$. The number in parentheses against a span label is the count
of examples in the test split with those many spans, e.g., 33,437 negative
examples with “0-Span". The concentric circles indicate exact match in the
range of $[0,1]$ in steps of $0.2$. The exact match of the predicted spans is
shaded in light Red color. Both the number of examples and the exact match
decrease with increasing number of spans.

Ablation with respect to supporting facts. We perform an ablation study to
investigate the effect of the presence of supporting facts in the data. We
finetune CuBERT-1K with only answer spans in targets. We compare it against
the CuBERT-1K model from Table 3, which is finetuned on both answer and
supporting-fact spans. We compute exact match with respect to answer spans
only and refer to it as _answer match_. As seen in Table 5.2, the model
finetuned without supporting facts achieves 86.96% exact match on all
examples, whereas the model finetuned with supporting facts achieves exact
match of 87.41%. Thus, training with supporting facts provides additional
supervision and helps with slightly improved exact match with respect to
answer spans.

Providing annotations for both answer spans and supporting facts for training
could be laborious for a developer. This ablation demonstrates that a model
could learn to generate correct answers even if the developer does not provide
supporting facts. Of course, if the developer provides supporting facts, it
not only helps improve the model’s accuracy on answers but also helps in
interpretability of the model’s predictions as the developer can inspect
supporting facts reported by the model on new code.

Figure 6.  Comparison of training on all queries versus individual queries in
terms of answer match. The Orange bar plot shows the answer match (scaled to
[0,1]) of the multi-query model (trained on all queries) on the test splits of
the 52 queries arranged on X-axis in the decreasing order of the number of
training examples. The line plot shows the answer match of the 52 query-
specific models on their test sets. As seen, the multi-query model outperforms
the query-specific models on 46 of 52 queries, showing significant positive
transfer across queries. In the line plot, the 6 queries on which the query-
specific models perform better than the multi-query model are marked by
$\bullet$ and the 46 queries on which the multi-query model performs better
are marked by x.

Training on all queries versus individual queries. We investigate the effect
of training a single model for all queries (our default choice) versus
training a separate model for each query. We finetune the CuBERT-1K model for
each of the 52 queries separately, i.e, we group the training examples by
queries and train 52 models (one model on each of these groups). We call these
models _query-specific models_. We finetune a single model, called the _multi-
query model_ , on the training examples of all queries. We group the test
examples by queries and compare the performance of the 52 query-specific
models, against the multi-query model, on their respective test examples. For
queries with small number of examples, predicting both answer and supporting
facts is harder than predicting only the answer spans. We therefore train the
models for answer span prediction and compare them over the test set in terms
of exact match over answer spans (answer match).

Figure 6 shows the performance of these models. The queries are arranged on
X-axis in decreasing order of training-example count. As the number of
training examples decreases, the performance of the multi-query model is much
better than the performance of the query-specific models. The average number
of training examples when the multi-query model is better performing than the
query-specific models is 434, and the least number of training examples in a
query where a query-specific model is better performing than the multi-query
model is 842. This indicates that training on multiple queries increases the
model performance as compared to training on individual queries due to useful
transfer learning across queries. The number of queries among the 52 queries
where the multi-query model performs better than the query-specific models is
46 (these queries are marked by x) as compared to 6 queries where training
individually is better (these queries are marked by $\bullet$). This shows
that multi-query learning is better performing and also convenient as a single
model can answer several queries, i.e, a single model is effectively
performing several program analysis tasks.

Choice of query identifier. As discussed in Section 4.1, each example in our
dataset is a pair $(Q,C)$ where $Q$ is a query identifier used for
distinguishing among the 52 queries. Each of our queries is associated with a
short natural-language phrase written by the query authors, termed the query
name (see the query names in Tables LABEL:tab:multihop-results and
LABEL:tab:singlehop-results). We use the query names as query identifiers in
our experiments. However, the exact choice of query identifier is not central
and a developer can choose other identifiers for the queries. Here, we conduct
an experiment in which we use $Q0,Q1,\ldots,Q51$ as the query identifiers
instead of the query names. Table 7 reports the results of this variation for
the CuBERT-1K model as “Modified IDs”. For reference, we have copied the
results from Table 3 as “Original IDs". For fair comparison, we use the exact
same token sequences for code across the two experiments and change only the
query tokens. We notice only a slight degradation in performance when modified
IDs are used.

Setting | All | Positive | Negative
---|---|---|---
Prefix | 72.28 | 36.60 | 93.80
Sliding window | 73.03 | 51.91 | 85.75
Two-step | 80.13 | 52.61 | 96.73
File-level ideal | 82.47 | 59.60 | 96.26

Table 8. Exact-match results on large-size code.

Figure 7. Learning from limited examples per query.

###  5.3. Experimentation under Pragmatic Considerations

Scaling to large-size code. Section 5.2 evaluates the models in the ideal,
unrealistic setting where the model is shown _only relevant code_. However, a
developer may provide large-size code (e.g., entire files) during inference
which could contain code not relevant to the query. To evaluate this pragmatic
setting, we prepare a _file-level test dataset_ from the test split of the ETH
Py150 Open dataset. Note that the CodeQL engine may return multiple answers
for a given query on code within a file. For preparing CodeQueries, they are
treated as distinct examples with their own distinct sets of relevant code
blocks. We now drop this distinction, and consider all answer and supporting-
fact spans for a query over a file together as ground truth for a single
example comprising the entire file. This dataset is substantially more
challenging because of large size and more spans per example. This file-level
dataset contains 44,423 test examples, among those, 16,712 are positive (with
9,408 being single-hop and 7,304 being multi-hop) and the remaining 27,711 are
negative (with 16,710 being single-hop and 11,001 being multi-hop). Out of
these, 75.33% examples have more than 1024 tokens (the input size limit of our
largest model, CuBERT-1K) and the average number of tokens is 5407. Note that
even if an example has less than 1024 tokens, it may contain code irrelevant
for the query unlike in the ideal setting of Section 5.2.

We evaluate the two-step procedure from Section 4.2 on the file-level test
dataset against _two heuristic baseline procedures_ : (1)  _Prefix_ : Take the
maximum sized prefix of the file that the model can handle as input. (2)
_Sliding window_ : Take the maximum sized non-overlapping chunks of the file
w.r.t. the input size of the model, perform inference on them independently
and combine (union) the results. We also consider an analog of the ideal,
unrealistic setting at the file level, which we call (3)  _file-level ideal_ :
this takes the union of oracle-provided relevant code blocks for a query from
a file across all ground-truth spans in it.

For the two-step procedure, we train a relevance classification model as
outlined in Section 4.2 and use it to first select a set of blocks from a file
for a given query. The span prediction model is then applied on the selected
set of blocks to answer the query (see Figure 4). We use the best performing
span-prediction model from Section 5.2 (the CuBERT-1K model) as it is _without
additional training_.

Table 8 shows the exact match results for all the procedures. The performance
of the span prediction model in the file-level ideal setting is provided for
reference. Because the model is supplied only relevant blocks, its exact match
is higher than the three procedures. However, its exact match on positive
examples is 59.60%. The average number of code tokens per example in the ideal
setting of Section 5.2 is 1297, whereas due to the file-level scope, the
average number of code tokens in the examples of the _file-level_ ideal
setting is 1660. Thus, the increase in size of code adversely affects the
model performance. Further, the average number of ground-truth spans on
positive examples go up from 1.83 in the ideal setting of Section 5.2 to 2.57
in the file-level dataset, making the task of achieving exact match w.r.t. all
the spans more difficult.

The two-step procedure outperforms both the heuristic procedures, but is
understandably inferior to the file-level ideal setting. Relevance
classification achieves accuracy, precision and recall of 96.38, 95.73 and
90.10 respectively. Thus, the classifier may include spurious code blocks
(false positives) or filter out relevant code blocks (false negatives). This,
in addition to the large size of examples and more spans, limits its exact
match on positive examples to only 52.61%. The performance of all procedures
on negative examples remains, unsurprisingly, high. In sliding-window, the
model sees code selected based on number of tokens from the beginning of the
file. This results in arbitrary splits, causing the model to predict spurious
spans, and dragging down exact match on negative examples.

Learning from limited number of examples. In practice, a developer may have a
limited number of labeled examples. We now assess the ability of the CuBERT-1K
model to answer the queries when trained with decreasing budget. We finetune
the model on all the queries but restrict the number of examples per query to
at most 512, 256, 128 or 64. We select an equal number of positive and
negative examples within the restricted budget. Figure 7 compares models
trained with the different training budgets on the entire test split of
CodeQueries. We have repeated the experiments three times for the restricted
budgets and the variance is shown in the figure. The exact match over all
examples drops from 86.70% to 66.29% (mean over three experiments) when we go
from training with all available examples to 64 examples per query. This is
mainly due to the drop in performance on the positive examples. As the
occurrences of the Outside label far outnumber other labels, the models
gravitate towards predicting the label $O$ for all tokens and find it
difficult to accurately predict the spans in positive examples. This is seen
in the consistently better performance on the negative examples even with
decreasing budget.

Test Data | All | Positive | Negative
---|---|---|---
Original | 86.70 | 72.51 | 96.79
Perturbed | 83.29 | 67.16 | 94.59
Table 9. Results of robustness to minor syntax errors.

Setting | All | Positive | Negative
---|---|---|---
Original | 86.70 | 72.51 | 96.79
Inaccurate spans | 86.31 | 71.99 | 96.47
Incomplete spans | 85.04 | 68.73 | 96.64
Incorrect spans | 84.93 | 68.89 | 96.33

Table 10. Robustness to mislabled spans.

Robustness to minor syntax errors. Symbolic techniques, like CodeQL, build
specialized representations of code for analysis and fail in the presence of
even minor syntax errors in code. We may be able to apply neural networks that
require only tokenization on such code. To validate robustness of neural
models to minor errors, we consider representative errors committed by
developers, such as improper indentation, omission of curly braces, and
absence of keywords and operators, as possible code perturbations. We perform
these perturbations on the examples in the test split of CodeQueries to form
perturbed examples and evaluate the CuBERT-1K model from Section 5.2 on them
_without additional training_.

For each example, we sample up to three lines from the code and apply a
perturbation to each. To maintain correspondence to ground truth, we do not
perturb the ground truth spans. We bias sampling towards lines in a prefix of
the code, so that perturbed tokens are not pruned away when examples are
pruned to length of 1024. We discard around 3.5% examples for which we could
not get perturbed code that tokenizes within a small, fixed sampling budget.
The CuBERT-1K model achieved 83.29% exact match on all perturbed examples. The
same model has achieved 86.70% exact match on the original, unperturbed
dataset (see Table 3). The full results are available in Table 5.3. We leave
training on perturbed examples (jain2020contrastive; pybuglab) to future work.

Robustness to mislabeled spans. We have conducted the experiments using
perfect, ground truth span labels. In practice, developers may make mistakes
in labeling. In order to evaluate robustness of the neural models to
mislabeled spans, we devise three settings to corrupt the training and
validation sets: (1)  _Inaccurate spans_ : We select up to 3 spans in 10% of
positive examples. For each span, we change the indexes of beginning and end
tokens of the span by a value in {-2, -1, +1, +2}. This simulates the
situation in which the developer selects some spurious tokens or leaves out
some required tokens in the vicinity of the correct span. (2)  _Incomplete
spans_ : We drop up to 3 spans in 10% of positive examples. This simulates the
situation in which the developer does not identify all the required spans. (3)
_Incorrect spans_ : In addition to the incomplete-spans setting, we also
select up to 3 plausible spans (obtained by running negative queries) in 10%
of negative examples and include them as labels. This simulates the situation
in which the developer not only forgets to identify actual spans (as in the
incomplete-spans setting) but also incorrectly marks some spurious spans as
answer spans, in effect incorrectly turning negative examples into positive
examples. Except for the examples affected by mislabeling, all other examples
remain unchanged in the training and validation sets.

We finetune the CuBERT-1K model separately for each of these settings and
evaluate them on the test split in Section 5.2 which has ground truth labels.
As seen in Table 10, the models are reasonably robust to mislabeled spans. The
maximum degradation in exact match over positive examples is -3.78% in the
case of incomplete spans. The models do not suffer much degradation over
negative examples.

##  6\. Discussion

Our dataset comprises a diverse set of 52 queries that developers have posed
over code. Each of these queries is a unique program analysis task on its own
and in the traditional program analysis setting, requires implementing a
specialized algorithm in a framework like CodeQL. Our experimental results
demonstrate that the SOTA pre-trained models can be finetuned to answer these
queries without any additional program analysis information and in fact, a
single model can be trained jointly across all these queries. Our models do
quite well in the idealized setting where only relevant code blocks are given
to the model.

Our investigations into pragmatic considerations offer further insights into
the strengths and weaknesses of the neural models. The neural models continue
to perform well in the presence of noisy labeling and in the presence of minor
syntax errors in code. The presence of the latter makes traditional program
analysis frameworks unusable since they need code to compile and build before
performing any analysis. As the size of the code increases, the model
performance suffers. The two-step procedure helps improve this by filtering
out irrelevant code blocks. Though the current SOTA pre-trained models
demonstrate the ability to assist in answering the semantic queries, the
subtle and complex nature of the semantic queries still requires them to be
provided with a few hundred examples. As we reduce the number of labeled
examples further, the model performance degrades (Figure 7). This provides
evidence that the popular pre-training objectives in the literature
(DBLP:conf/icml/KanadeMBS20; DBLP:conf/emnlp/FengGTDFGS0LJZ20;
guo2020graphcodebert) are not strong enough to offer data-efficient solution
to semantically richer tasks such as ours. Thus, beyond the practical
motivation of assisting developers, CodeQueries can serve as a challenge
benchmark to measure progress on neural code understanding, much in the same
way that QA datasets (DBLP:conf/emnlp/RajpurkarZLL16;
DBLP:conf/emnlp/Yang0ZBCSM18) have served to measure progress on natural
language understanding.

We now discuss some concrete examples towards qualitative study of the model’s
performance.

⬇

1 def test_jtheta_issue_79():

2 # near the circle of covergence q = 1 the convergence slows

3 # down; for q > Q_LIM the theta functions raise ValueError

4 mp.dps = 30

5 mp.dps += 30

6 q = mpf(6)/10 - one/10**6 - mpf(8)/10 * j

7 mp.dps -= 30

8 # Mathematica run first

9 # N[EllipticTheta[3, 1, 6/10 - 10^-6 - 8/10*I], 2000]

10 # then it works:

11 # N[EllipticTheta[3, 1, 6/10 - 10^-6 - 8/10*I], 30]

12 res = mpf(’32.0031009628901652627099524264’) + \

13 mpf(’16.6153027998236087899308935624’) * j

14 result = jtheta(3, 1, q)

15 # check that for abs(q) > Q_LIM a ValueError exception is raised

16 mp.dps += 30

17 q = mpf(6)/10 - one/10**7 - mpf(8)/10 * j

18 mp.dps -= 30

19 try:

20 result = jtheta(3, 1, q)

21 except ValueError:

22 pass

23 else:

24 assert(False)

25

26 # bug reported in issue 79

27 mp.dps = 100

28 ...result = testdir.runpytest("--tb=short")

Figure 8. A negative example for the query “An assert statement has a side-
effect.”. It has no assert statement which has a side-effect, but the model
incorrectly predicts Line 24 as an answer span.

###  6.1. Examples of Successful and Unsuccessful Span Predictions

We now present examples of both successful and unsuccessful predictions of the
CuBERT-1K model for the query “An assert statement has a side-effect.”. Figure
14 is a negative example where there are four assert statements, but none of
them cause a side-effect. The model does not predict any answer spans as the
assert statements are simple enough to deduce absence of side-effect. This is
a case of _successful prediction on a negative example_ (true negative).
Figure 8444Part of fredrik-johansson/mpmath/mpmath/tests/test_elliptic.py file
in the ETH Py150 Open dataset shows _a negative example for which the model is
unsuccessful_ (false positive). It incorrectly predicts Line 24 as an answer
span, possibly because of the peculiar numerical expressions in the code.

Figure 9555Part of kvesteri/flask-storage/tests/test_mock.py file in the ETH
Py150 Open dataset. is _a positive example where the model successfully
predicted the answer span_ (true positive). The presence of the open method
might have helped the model to identify the side-effect. Figure 10666Part of
getsentry/raven-python/tests/functional/tests.py file in the ETH Py150 Open
dataset. shows _a positive example where the model fails to predict the
correct span_ (false negative). The complex, multi-line expression in the
statement might have made it difficult for the model to predict the correct
answer span.

⬇

1 def test_reads_file_object_and_saves_in_dict(self):

2 storage = MockStorage()

3 io = StringIO()

4 io.write(’file contents’)

5 storage.save(’key’, io)

6 assert storage.open(’key’).read() == ’file contents’ result =
testdir.runpytest("--tb=short")

Figure 9. A positive example for the query “An assert statement has a side-
effect.” where the model predicts the ground-truth span (highlighted in Red)
successfully.

⬇

1 def test_absolute_import(self):

2 string = ’from __future__ import absolute_import’

3 kwargs = {

4 ’stdout’: open(’/dev/null’, ’a’),

5 ’stderr’: open(’/dev/null’, ’a’),

6 }

7 for filename in find_files(ROOT, ’*.py’):

8 assert not call([’grep’, string, filename], **kwargs), \

9 "Missing %r in %s" % (string, filename[len(ROOT) - 5:]) result =
testdir.runpytest("--tb=short")

Figure 10. A positive example for the query “An assert statement has a side-
effect.” where the model fails to predict the ground-truth span (highlighted
in Red).

###  6.2. Learned vs. Manual Analyses

As discussed in Section 2.1, as the external libraries evolve and as
developers adopt new ones, the existing program analyses tend to become stale.
It requires continuous manual efforts to keep the analyses up-to-date. The
learned analyses may offer generalization in such cases. Figure 11 is one such
example. CodeQL does not flag the assert statement at line 3 as having a side-
effect. The assert statement makes a call to process_list_param method of a
dictset which is a wrapper around the standard dict type. When pop is set to
True, it calls pop method of dict which removes the queried key from the
dictionary. In fact, note that the second assert (line 4) is actually
explicitly validating that the key has been removed. Our model correctly
identifies the first assert as having a side-effect, even though CodeQL has
incomplete coverage of this case and fails to flag this assert as having side-
effect.

⬇

1 def test_process_list_param_pop(self):

2 dset = dictset({’boo’: ’1,2’})

3 assert dset.process_list_param(’boo’, pop=True) == [’1’, ’2’]

4 assert ’boo’ not in dset result = testdir.runpytest("--tb=short")

Figure 11. Example showing generalization of learned analysis for an assert
with side-effect which is missed by CodeQL.

##  7\. Conclusions and Future Work

We presented the CodeQueries dataset which tests the ability of neural models
for code understanding on the proposed problem of answering semantic queries
over code. It poses a challenging setup in which the model is required to
perform single- or multi-hop reasoning, understand structure and semantics of
code, distinguish between positive and negative examples, and accurately
identify both answer and supporting-fact spans. Despite a diverse set of 52
queries requiring varied program analyses, the proposed models perform
reasonably well. The models are also robust to minor syntax errors in code and
to noisy labeling. If this line of work succeeds, it can make it possible for
regular developers, without the time or expertise to implement formal program
analyses, to formulate semantic queries through labeled examples and leverage
neural models to obtain answers and supporting-facts on previously unseen
code. As the models are robust to minor syntax errors in code, the developers
can use the neural models on incomplete or erroneous code that cannot be
analyzed by symbolic techniques.

Our systematic evaluation indicates that scalability to entire files and
learning from a limited number of examples have much room for improvement. The
two-step procedure, combining classification with span prediction, improves
the performance on files but still lags behind the file-level ideal setting.
In the future, we plan to explore models with the ability to handle larger
contexts (dai2019transformer) and to improve our relevance classifiers and
span predictors by pre-training using objectives conceptually closer to those
tasks (joshi2020spanbert; ram2021few). We are also considering extensions to
our dataset to include more semantic queries, and more programming languages.
Our evaluation shows that CodeQueries as-is is challenging for the best-in-
class pre-trained models under some pragmatic considerations. An orthogonal
extension to our work could be to devise techniques so that a model trained on
some queries can generalize to a new query without any additional supervision
specific to the new query.

##  Appendix A Illustrative Examples

In this section, we present examples to illustrate the complex and subtle
forms of reasoning required for answering semantic queries over code. We
discuss examples requiring single-hop and multi-hop reasoning in both positive
(examples containing answers) and negative (examples that do not contain
answers) scenarios.

⬇

1 def test_open_zipped(self):

2 zf = self._make_zip_file()

3 with ignore_warnings():

4 assert len(fits.open(self._make_zip_file())) == 5

5 with ignore_warnings():

6 assert len(fits.open(zipfile.ZipFile(zf))) == 5

7 assert len(fits.open(self._make_zip_file())) == 5

_Answer span 1_ _Answer span 2_

Figure 12. A positive example with multiple answer spans for the single-hop
query “An assert statement has a side-effect.”

⬇

1class Type(object):

2 def __eq__(self, other) :

3 return self.__class__ == other.__class__

4 return self.__class__ == other.__class__

5 def from_str(self, s):

6 raise NotImplementedError

7

8 class Choice(Type) :

9 def __init__(self, *options):

10 self._options = options

11

12

13 def from_str(self, s):

14 if s in self._options:

15 return s

16 raise ValueError("Unexpected value %s: must be one of %s" % (s, ",
".join(self._options)))

17

Block 1Block 2 _Answer span_ _Supporting fact 1_ _Supporting fact 2_ Block
3Block 4

Figure 13. A positive example with one answer span and two supporting-fact
spans for the multi-hop query “__eq__ not overridden when adding attributes”.

⬇

1 def test_tbstyle_short(testdir):

2 p = testdir.makepyfile("""

3 def pytest_funcarg__arg(request):

4 return 42

5 def test_opt(arg):

6 x = 0

7 assert x

8 """)

9 result = testdir.runpytest("--tb=short")

10 s = result.stdout.str()

11 assert ’arg = 42’ not in s

12 assert ’x = 0’ not in s

13 result.stdout.fnmatch_lines([

14 "*%s:5*" % p.basename,

15 " assert x",

16 "E assert*",

17 ])

18 result = testdir.runpytest()

19 s = result.stdout.str()

20 assert ’x = 0’ in s

21 assert ’assert x’ in s result = testdir.runpytest("--tb=short")

Figure 14. A negative example with multiple plausible, but no actual, answers
for the single-hop query “An assert statement has a side-effect.” It has four
assert statements but none of them has a side-effect.

⬇

1class TagNode(Node):

2 def __init__(self, name):

3 self.name = name

4

5 def __eq__(self, other):

6 return (self.__class__ == other.__class__ and

7 self.name == other.name)

8

9 def __repr__(self):

10 return ’<TagNode: %s>’ % self.name

11

12class RegexTagNode(TagNode):

13 def __init__(self, name, regex):

14 self.name = name

15 self.regex = regex

16

17 def __eq__(self, other):

18 return (self.__class__ == other.__class__ and

19 self.name == other.name and

20 self.regex == other.regex)

21

22 def __repr__(self):

23 return ’<RegexTagNode %s: %s>’ % (self.name, self.regex)

24

Figure 15. A negative example with a plausible, but not actual, answer for the
multi-hop query “__eq__ not overridden when adding attributes”. The subclass
RegexTagNode adds a new attribute regex but also overrides the __eq__ method
appropriately.

###  A.1. Positive Examples

Figure 12 shows a positive example777Part of
spacetelescope/PyFITS/pyfits/tests/test_core.py file in the ETH Py150 Open
dataset. for the single-hop query “An assert statement has a side-
effect”888https://github.com/github/codeql/blob/main/python/ql/src/Statements/SideEffectInAssert.ql.
This query aims to find assert statements that match syntactic patterns
consistent with code having side effects (e.g., calling a function whose name
contains the substring “open”, “close”, etc.). As this query can be answered
by analyzing a single method body, we refer to it as a _single-hop query_. In
Figure 12, both the assert statements call fits.open, which matches the side-
effect pattern of the query, so both statements are identified as the ground
truth answer spans.

Figure 13 shows a positive example999Part of
rllab/rllab/rllab/envs/box2d/parser/xml_attr_types.py file in the ETH Py150
Open dataset.  for the multi-hop query “__eq__ not overridden when adding
attributes”101010https://github.com/github/codeql/blob/main/python/ql/src/Classes/DefineEqualsWhenAddingAttributes.ql.
If a subclass adds attributes that are not present in its superclasses, it
should override the __eq__ method. Otherwise, the equality function might not
work properly. Since this query requires simultaneous reasoning about code
spread across the subclass and its superclasses, we refer to it as a _multi-
hop query_. Figure 13 shows the answer span and the supporting-fact spans. The
declaration of the Choice class is the answer span in this example. The
highlighted supporting facts from different code blocks indicate that this
class has added an attribute (_options) which is absent in its superclass
(class Type), and the superclass has defined the __eq__ method, which should
be overridden.

###  A.2. Negative Examples

In practice, a developer may want to validate the absence of a pattern (e.g.,
validate that there is no unused variable). We therefore handle unanswerable
scenarios also. We call examples that do not contain answer spans _negative
examples_. Naïvely, any code on which a query does not return an answer could
be viewed as a negative example; for instance, in the case of conflicting
attributes (Figure 1), it would be trivial to answer that there are no
conflicting attributes if the code does not even contain class definitions. In
natural-language question answering, DBLP:conf/emnlp/Yang0ZBCSM18 recommend
that unanswerable contexts should contain _plausible, but not actual, answers_
; otherwise, it is simple to distinguish between answerable and unanswerable
contexts (DBLP:conf/conll/WeissenbornWS17). Therefore, to obtain negative
examples _with plausible answers_ , we manually derive logical negations of
the CodeQL queries and evaluate them on the ETH Py150 Open dataset. We ensure
that a _negative query_ identifies code similar to the original (positive)
query but which does not satisfy the key properties required for producing an
answer for the original query.

As an example, we refer the reader to the query definition of the “An assert
statement has a side-effect” query, which is a single-hop query. The CodeQL
queries have syntax similar to SQL and use the select and where clauses. The
predicates are used in these queries to identify appropriate answer spans. In
this case, the where clause checks if an assert statement has an expression e
that has a probable side-effect probable_side_effect(e). As discussed earlier,
Figure 12 is a positive example for this query.

To obtain the corresponding negative query, we simply change the where clause
to not probable_side_effect(e) while keeping everything else the same. When
evaluated, it gives us plausible yet incorrect answers with respect to the
original, positive query since the answer does contain an assert but it does
_not_ match the pattern indicative of side effects defined by the query.
Figure 14 shows a negative example111111Part of pytest-
dev/pytest/testing/test_terminal.py file in the ETH Py150 Open dataset. we get
by evaluating the negative query. On this code (which lacks the pattern sought
by the query), we do not want the model to produce any answer or supporting-
fact spans; essentially validating that the code does not contain any assert
statement with side-effects.

Figure 15 shows a negative example121212Part of
codysoyland/surlex/src/surlex/grammar.py file in the ETH Py150 Open dataset.
for the multi-hop query “__eq__ not overridden when adding attributes”. Though
the subclass RegexTagNode adds a new attribute regex that is not present in
its base class TagNode, it also overrides the __eq__ method appropriately.
Therefore, on this code, we expect the model to not identify any answer spans.

Note that negative queries are produced manually by analyzing the
corresponding positive query: distinguishing the portions of the query
predicates that make the example plausible from the portion that evaluates
whether the plausible example is also a positive example; although this query
manipulation could be automated using statistical heuristics, in this work we
performed it manually.

##  Appendix B Query-wise Dataset Statistics

We report the query-wise statistics for multi-hop queries, aggregated across
all splits, in Table LABEL:tab:querywise-multihop. We report the statistics
for _All Examples_ , _Positive_ examples and _Negative_ examples. _Count_
gives the number of examples. We sort all the tables from here on by the
descending order of count of all examples. Under all examples, we give the
average length of the input sequences in terms of sub-tokens. Here, the sub-
tokenization is performed using the CuBERT vocabulary. For positive examples,
we report the average number of answer (abbreviated as _Ans._) spans and
supporting fact (abbreviated as _SF_) spans. The column _Avg. Spans after
Pruning_ is the average number of spans (both answer and supporting fact spans
combined) after the sub-tokenized sequence is pruned to the length of 1024 for
the CuBERT-1K model. Note that the number of answer or supporting fact spans
is zero for negative examples and are hence omitted. We highlight the minimum
and maximum values per column in bold face.

Table 11. Query-wise statistics for the multi-hop queries. |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
Index |  Query Name | All Examples | Positive |  Negative
|  |  Count |  Avg. Length |  Count |  Avg. Ans. Spans |  Avg. SF Spans |  Avg. Spans after Pruning |  Count
Q1 |  Unused import |  48,555 |  3220.68 |  19,178 |  2.10 |  0 |  2.04 |  29,377
Q2 |  Missing call to __init__ during object initialization |  1,540 |  287.18 |  770 |  1.00 |  2.02 |  2.98 |  770
Q3 |  Use of the return value of a procedure |  1,013 |  1078.05 |  432 |  1.44 |  1.04 |  1.98 |  581
Q4 |  __eq__ not overridden when adding attributes |  857 |  1837.72 |  778 |  1.00 |  3.87 |  3.29 |  79
Q5 |  Wrong number of arguments in a call |  794 |  872.75 |  355 |  1.28 |  1.04 |  2.08 |  439
Q6 |  Comparison using is when operands support __eq__ |  540 |  1063.45 |  230 |  1.52 |  0 |  1.21 |  310
Q7 |  Signature mismatch in overriding method |  531 |  3610.39 |  247 |  1.32 |  1.22 |  1.40 |  284
Q8 |  Non-callable called |  422 |  1867.71 |  162 |  1.65 |  1.44 |  2.36 |  260
Q9 |  __init__ method calls overridden method |  393 |  5457.02 |  193 |  1.29 |  4.23 |  2.81 |  200
Q10 |  Conflicting attributes in base classes |  364 |  2757.73 |  182 |  1.07 |  3.05 |  1.75 |  182
Q11 |  __iter__ method returns a non-iterator |  329 |  213.20 |  209 |  1.00 |  1.08 |  2.08 |  120
Q12 |  Flask app is run in debug mode |  243 |  343.55 |  123 |  1.00 |  0 |  0.98 |  120
Q13 |  Inconsistent equality and hashing |  241 |  1657.24 |  121 |  1.00 |  1.00 |  1.78 |  120
Q14 |  Wrong number of arguments in a class instantiation |  212 |  1317.06 |  99 |  1.21 |  0.91 |  1.84 |  113
Q15 |  Incomplete ordering |  174 |  2252.13 |  87 |  1.00 |  1.31 |  2.05 |  87
Aggregate |  56,208 |  2985.95 |  23,166 |  1.94 |  0.34 |  2.10 |  33,042

Table LABEL:tab:querywise-singlehop gives the query-wise statistics for
single-hop queries aggregated across all splits. The column headings have the
same meaning as those of Table LABEL:tab:querywise-multihop. We highlight the
minimum and maximum values per column in bold face.

Table 12. Query-wise statistics for the single-hop queries. |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
Index |  Query Name | All Examples | Positive |  Negative
|  |  Count |  Avg. Length |  Count |  Avg. Ans. Spans |  Avg. SF Spans |  Avg. Spans after Pruning |  Count
Q16 |  Unused local variable |  34,202 |  464.42 |  14,326 |  1.43 |  0 |  1.36 |  19,876
Q17 |  Except block handles BaseException |  22,596 |  561.92 |  10,377 |  1.27 |  0 |  1.16 |  12,219
Q18 |  Imprecise assert |  16,311 |  306.26 |  6,060 |  2.05 |  0 |  2.00 |  10,251
Q19 |  Variable defined multiple times |  10,300 |  758.36 |  3,694 |  1.82 |  3.49 |  3.51 |  6,606
Q20 |  Testing equality to None |  5,454 |  573.88 |  2,354 |  1.36 |  0 |  1.24 |  3,100
Q21 |  Unreachable code |  4,612 |  790.50 |  2,098 |  1.20 |  0 |  1.06 |  2,514
Q22 |  First parameter of a method is not named self |  4,088 |  177.55 |  2,044 |  1.00 |  0 |  1.00 |  2,044
Q23 |  Unnecessary pass |  2,517 |  388 |  1,114 |  1.26 |  0 |  1.14 |  1,403
Q24 |  Module is imported with import and import from |  1,959 |  605.6 |  953 |  1.06 |  0 |  1.03 |  1,006
Q25 |  Module is imported more than once |  1,051 |  899.96 |  489 |  1.17 |  1.20 |  1.98 |  562
Q26 |  Comparison of constants |  909 |  795.2 |  78 |  10.73 |  0 |  9.50 |  831
Q27 |  Implicit string concatenation in a list |  871 |  1819.25 |  319 |  1.75 |  0 |  1.43 |  552
Q28 |  Suspicious unused loop iteration variable |  823 |  713.69 |  387 |  1.12 |  0 |  1.02 |  436
Q29 |  Duplicate key in dict literal |  715 |  2588.83 |  150 |  3.99 |  3.82 |  2.09 |  565
Q30 |  Unnecessary else clause in loop |  676 |  576.45 |  338 |  1.02 |  0 |  1.01 |  338
Q31 |  First argument to super() is not enclosing class |  657 |  180.22 |  326 |  1.02 |  0 |  1.01 |  331
Q32 |  Redundant assignment |  597 |  1045.49 |  260 |  1.30 |  0 |  1.24 |  337
Q33 |  An assert statement has a side-effect |  537 |  458.88 |  206 |  1.65 |  0 |  1.59 |  331
Q34 |  Nested loops with same variable |  520 |  920.02 |  241 |  1.16 |  1.05 |  1.90 |  279
Q35 |  Import of deprecated module |  515 |  709.62 |  243 |  1.12 |  0 |  1.05 |  272
Q36 |  NotImplemented is not an Exception |  476 |  138.42 |  237 |  1.01 |  0 |  1.01 |  239
Q37 |  Redundant comparison |  463 |  908.08 |  188 |  1.47 |  1.30 |  2.52 |  275
Q38 |  Deprecated slice method |  439 |  117.18 |  215 |  1.04 |  0 |  1.04 |  224
Q39 |  Constant in conditional expression or statement |  423 |  610.60 |  164 |  1.58 |  0 |  1.34 |  259
Q40 |  Comparison of identical values |  415 |  707.73 |  164 |  1.53 |  0 |  1.47 |  251
Q41 |  import $\star$ may pollute namespace |  397 |  1536.95 |  197 |  1.02 |  0 |  0.99 |  200
Q42 |  Unnecessary delete statement in function |  382 |  582.99 |  199 |  1.00 |  1.00 |  1.92 |  183
Q43 |  Illegal raise |  374 |  553.21 |  173 |  1.16 |  0 |  1.1 |  201
Q44 |  Insecure temporary file |  287 |  440.16 |  139 |  1.09 |  0 |  1.07 |  148
Q45 |  Modification of parameter with default |  238 |  827.17 |  96 |  1.48 |  1.02 |  2.26 |  142
Q46 |  Should use a with statement |  228 |  368.79 |  113 |  1.02 |  0 |  1.00 |  115
Q47 |  Special method has incorrect signature |  222 |  101.90 |  111 |  1.00 |  0 |  1.00 |  111
Q48 |  Non-standard exception raised in special method |  205 |  174.25 |  102 |  1.01 |  0 |  1.01 |  103
Q49 |  Use of global at module level |  193 |  606.58 |  72 |  1.69 |  0 |  1.44 |  121
Q50 |  Modification of dictionary returned by locals() |  173 |  776.82 |  73 |  1.34 |  0 |  1.18 |  100
Q51 |  Incomplete URL substring sanitization |  168 |  826.67 |  70 |  1.43 |  0 |  1.31 |  98
Q52 |  Unguarded next in generator |  145 |  512.22 |  67 |  1.22 |  0 |  1.21 |  78
Aggregate |  115,138 |  538.71 |  48,437 |  1.46 |  0.31 |  1.53 |  66,701

##  Appendix C Statistics of Syntactic Patterns of Spans

In our dataset, the answer and supporting-fact spans cover various types of
programming language constructs. Hence, in LABEL:tab:span-type-stat, we
tabulate the number of spans in terms of syntactic patterns of Python
constructs in decreasing order of their frequency in the combined data of all
three splits. To find the pattern of a span, we have used tree-sitter
(treeSitter) to get the closest ancestor node which encloses the tokens
appearing in the span. Two special entries in the table are block and module.
A _block_ node can represent any block of code, i.e., a block of code, a
function, a class. Sometimes the closest ancestor node is the root node of the
source code, for those cases _module_ node is used as a representative node.

Table 13. Statistics of syntactic patterns of spans. Syntactic Pattern | Count |  Syntactic Pattern | Count |  Syntactic Pattern | Count
---|---|---|---|---|---
import statement | 43,013 |  raise statement | 375 |  module | 56
assignment | 32,422 |  function parameters | 373 |  dictionary keys | 47
call | 15,978 |  assert statement | 368 |  break statement | 43
except clause | 13,269 |  delete statement | 358 |  while statement | 43
function definition | 8,937 |  if statement | 243 |  argument list | 34
non-boolean binary operator | 5,319 |  sequence expressions | 192 |  with statement | 26
class attributes | 2,844 |  identifier | 186 |  parenthesized expression | 14
class definition | 2,882 |  decorator | 138 |  boolean operator | 13
block | 2,331 |  print statement | 126 |  elif clause | 12
pass statement | 1,451 |  global statement | 125 |  expression list | 12
string literal | 1,279 |  list comprehension | 101 |  lambda | 11
for statement | 1,164 |  subscript | 72 |  conditional expression | 8
concatenated string | 558 |  not operator | 71 |  yield | 5
return statement | 395 |  try statement | 65 |  continue statement | 3
|  Aggregate | 134,962