embeddings from both incoming and outgoing directions for the program graph
$g(V,\bm{A})$ extracted from the code snippet. During each hop $n$, for node
$v$, it applies an aggregation function to take a set of incoming (or
outgoing) neighboring node vectors as input and outputs a backward (or
forward) aggregation vector. The summation function is selected as the
aggregation function, where $N_{(v)}$ denotes the neighbors of node $v$ and
$\dashv$ / $\vdash$ is the backward or forward direction.

(25)
$\displaystyle\bm{h}_{\mathcal{N}\dashv(v)}^{n}=\mathrm{SUM}(\\{\bm{h}_{u}^{n-1},\forall
u\in\mathcal{N}_{\dashv(v)}\\})$ (26)
$\displaystyle\bm{h}_{\mathcal{N}\vdash(v)}^{n}=\mathrm{SUM}(\\{\bm{h}_{u}^{n-1},\forall
u\in\mathcal{N}_{\vdash(v)}\\})$

Then, the node embeddings for both directions are fused as follows:

(27)
$\displaystyle\bm{h}_{\mathcal{N}_{(v)}}^{n}=\mathrm{Fuse}(\bm{h}_{\mathcal{N}_{\lnot(v)}}^{n},\bm{h}_{\mathcal{N}_{\vdash(v)}}^{n})$

The fusion function is formulated as a gated sum of two inputs:

(28)
$\displaystyle\mathrm{Fuse}(\bm{a},\bm{b})=\bm{z}\odot\bm{a}+(1-\bm{z})\odot\bm{b}$
(29) $\displaystyle
z=\sigma(\bm{W}_{z}[\bm{a};\bm{b};\bm{a}\odot\bm{b};\bm{a}-\bm{b}]+\bm{b}_{z})$

where $\odot$ is the component-wise multiplication, $\sigma$ is a sigmoid
function and $z$ is gating vector. Then, GRU is used to update node
representations. At last, after $k$ hops of computation, the final node
representation $\bm{h}_{v}^{k}$ is obtained and max-pooling is applied over
all nodes $\\{\bm{h}_{v}^{k},\forall v\in V\\}$ to get a $d$-dim graph
representation $\bm{h}^{g}$:

(30)
$\displaystyle\bm{h}^{g}=\mathrm{maxpool}(\mathrm{FC}(\\{\bm{h}_{v}^{k},\forall
v\in V\\}))$

where $\mathrm{FC}$ is the fully-connected layer.

Summary for RQ2: Similar to the query end, existing code search techniques
mainly optimize the code end from two aspects: code feature mining and code
feature representation. Regarding code feature mining, The usage frequency of
the three code textual features (i.e., method name, API sequence, tokens) is
similar. Moreover, there are many works that simultaneously utilize multiple
types of code textual features. Among code structural features, AST is used
most frequently, followed by DFG, CFG, and PDG. Some recent works have
explored some new code structural features, e.g., VFG, program paragraph, and
program slice. Also, some works simultaneously utilize multiple code
structural features. It is worth noting that there are many code search
techniques that consider both code textual features and code structural
features. Regarding code feature representation, existing code search
techniques have designed distinct code representation methods for various code
features. Works in recent years have widely adopted DL-based methods to
generate numerical vector representations of code textual or structural
features.

##  6\. Answering RQ3: What are the match-end optimization methods in code
search studies?

In general, developers anticipate that a code search system will prioritize
the most relevant and possible code snippets as search results to facilitate
their development. However, identifying the code snippet that best matches a
query from thousands of snippets can be highly complex. Even with a decent
understanding of the semantics of the query and the code snippet, it may still
cost a large amount of time. In order to find the code snippet that matches
the query efficiently and correctly, numerous methods and algorithms have been
devised to optimize the match-end of the code search. In short, a better
match-end optimization method can improve the performance of code search. It
finds the most effective and efficient method to match the code snippet and
query after making comprehensive use of their syntax and semantics.

According to the feature representation form of the query and code, existing
match-end optimization techniques can be divided into three distinct groups:
text-based matching, vector-based matching, and classification-based matching,
as shown in Figure 2. The text-based matching techniques aim to utilize the
keywords from query and code snippet to make the matching, detailed in Section
6.1. The vector-based matching techniques are widely used in recent work. They
can be further divided into two categories: vector distance-based, and
embedding distance-based, detailed in Section 6.2. The classification-based
matching techniques aim to regard the matching tasks into classification
tasks, detailed in Section 6.3. Figure 10 showcases the evolution of the above
three types of match-end optimization techniques. In the following
subsections, we will discuss them in detail and summarize their development
trends.

for tree= grow=east, reversed=true, anchor=base west, parent anchor=east,
child anchor=west, base=left, font=, rectangle, draw=black, rounded corners,
align=left, minimum width=4em, edge+=darkgray, line width=1pt, s sep=3pt,
inner xsep=2pt, inner ysep=3pt, line width=1pt, ver/.style=rotate=90, child
anchor=north, parent anchor=south, anchor=center, , where level=1text
width=10em, font=,, where level=2text width=7em, font=,, where level=3text
width=4.5em, font=,, where level=4text width=4.5em, font=,, where level=5text
width=4.5em, font=,, where level=6text width=4.5em, font=,, where level=7text
width=4.5em, font=,, where level=8text width=4.5em, font=,, where level=9text
width=4.5em, font=,, [ Match
End [ Text-based Matching
(§6.1) [ (Lv et al., 2015), l=203.5pt [ (Nie et al., 2016), [ (Yang and Huang,
2017), [(Karnalim, 2018; Huang et al., 2018) [ (Liu et al., 2019) [(Hu et al.,
2020b; Fu et al., 2020) [(Liu et al., 2021a; Cai et al., 2021)
(Abid et al., 2021) [ [ (Mao et al., 2023) ] ]]]]]]]] ] [ Vector-based
Matching
(§6.2) [Vector
Distance-based
(§6.2.1) [ [ (Niu et al., 2016) [ [(Sirres et al., 2018; Zhang et al., 2018) [
[ (Chen et al., 2020) [ [ [] ]]]]]]]] ] [ Embedding
Distance-based
(§6.2.2) [ [ [ (Yang and Huang, 2017) [(Gu et al., 2018; Sachdev et al., 2018)
(Chen and Zhou, 2018) [(Wan et al., 2019; Yao et al., 2019) [(Haldar et al.,
2020; Gu et al., 2020)
(Shuai et al., 2020; de Rezende Martins and Gerosa, 2020)
(Ren et al., 2020; Hu et al., 2020a)
(Sun et al., 2020; Ling et al., 2020)
(Wang et al., 2020b; Feng et al., 2020) [(Xu et al., 2021; Gu et al., 2021)
(Liu et al., 2021b; Ling et al., 2021)
(Meng, 2021; Fang et al., 2021)
(Guo et al., 2021) [(Wu and Yan, 2022; Yu et al., 2022)
(Deng et al., 2022; Kong et al., 2022b)
(Zhao and Liu, 2022; Wang et al., 2022)
(Niu et al., 2022) [(Cai et al., 2023; Zeng et al., 2023)
(Tang et al., 2023) ] ]]]]]]]]] ] [ Classification-based
Matching
(§6.3) [, l=203.5pt [ [ [ [ (Jiang et al., 2019) [ (Zhu et al., 2020) [
(Gotmare et al., 2021) [ (Gu et al., 2022) [] ]]]]]]]]] ]

(year)201520162017201820192020202120222023

Figure 10. Evolution of techniques in match end

###  6.1. Text-based Matching

As a traditional matching method, text-based matching utilizes textual
features such as word frequency or keywords to match code snippets with
queries. These matching methods are simple and efficient and are usually
adopted by the early code search methods. As shown in Figure 10, we have
summarized a total of 12 articles that have used text-based matching methods
and provided detailed summaries and overviews of them.

A basic method of text-based matching is calculating the number of common
keywords shared in the text of code snippets and queries. CodeMatcher (Liu et
al., 2021a) is a good example of directly utilizing the keywords for matching.
CodeMatcher is an IR-based code search model that inherits the advantages of
DeepCS (i.e., the capability of understanding the sequential semantics in
important query words). CodeMatcher first collects metadata for query words to
identify irrelevant/noisy ones. According to this preprocessed step, the
collected keywords are used to launch an iterative fuzzy match on indexed
method names. Then, iteratively performs the fuzzy search with important query
keywords on the codebase to obtain the top $k$ candidate code snippets. To
refine the fuzzy search results, CodeMatcher designs a reranking step to
measure the matching degree between query and candidate code snippets. During
the reranking, the method name and body are regarded as two different
components for the search. This is because the method name is often defined in
natural language whose semantic representation is close to the query, but the
method body implements the goal of the method name in programming languages.
Therefore, when matching keywords for a candidate code snippet, CodeMatcher
calculates the characters matched keywords in the method name of code snippets
as $S_{name}$. It indicates a level of the ranked method with more overlapped
tokens between the query and code snippet. CodeMatcher reorders code snippets
based on $S_{name}$ in descending order to retrieve the final search results.
In the case of tied scores, a similar approach is employed to calculate the
keyword coverage scores on code methods as $S_{body}$. A higher $S_{body}$
implies better keyword matching between query and code snippets in the method
body. Thus, the candidate code snippet with a higher $S_{body}$ will rank
ahead.

Extended Boolean Model (EBM) is an information retrieval model that
incorporates features of both the traditional Boolean model and the VSM. It
combines the advantages of precisely matching from the Boolean model with the
semantic similarity from VSM. CodeHow (Lv et al., 2015) applies the EBM to
consider the impact of both text similarity and APIs on code search. According
to the CodeHow introduction in Section 4.1.2, it can calculate similar scores
to obtain the potentially relevant APIs that match the query. Thus, CodeHow
considers both the text and APIs in EBM to match the query with code snippets.
EBM conducts keyword retrieval on method name and method body of code snippets
through Boolean operators and obtains similar scores. As mentioned in
CodeMatcher, the method name and body are regarded differently in matching the
query and code snippets. The method name is more important than the method
body. Therefore, the similarity scores for method name and method body are
weighted differently in the calculation of the total score, with weights of
1.5 and 1.0, respectively. Besides, the final score also includes the API
score obtained during the API understanding phase, with a weighting
coefficient of 1.5. nIECS (Yang and Huang, 2017) also utilizes EBM to conduct
the similarity between the original query with code and the expanded query
with code, respectively. The sum results are used to match the code snippets
with the query as the final output.

BM25 is a well-known information retrieval technique that is designed to
improve the traditional TF-IDF model. BM25 is employed to assess the relevance
between textual and queries, so it is commonly applied to optimize the
matching end. Both query expansion code search models, NQE (Liu et al., 2019)
and QECK (Nie et al., 2016) leverage BM25 to match code snippets with queries.
Similar to the computation for TF-IDF, it uses the following formula to give
the score for the code snippets document d and a given query word $x$:

(31)
$\displaystyle\operatorname{BM}25(d,x)=\operatorname{IDF}(x)\cdot\frac{\mathrm{TF}(x,d)\cdot(k+1)}{\operatorname{TF}(x,d)+k\cdot\left(1-b+b\cdot\frac{|d|}{\operatorname{avgdl}}\right)},$

where $TF(\cdot)$ is a function that calculates the term frequency;
$IDF(\cdot)$ is a function that computes the inverse document frequency; and
${avgdl}$ is the average document length of all code snippets. $k$ and $b$ are
tunable parameters that adjust the impact of term frequency on the final
score, thereby enhancing the flexibility of the BM25 model. Based on the
scores calculated using BM25, NQE and QECK output the code snippets with the
highest score as the matching result for the given query. Furthermore, many
code search methods that do not involve deep neural networks also utilize BM25
as a retrieval model to match queries with code snippets (Karnalim, 2018;
Huang et al., 2018; Hu et al., 2020b). This indicates that BM25 is not only
effective but also easy to use.

Lucene is a conventional text search engine behind many existing code search
tools. It is a versatile tool that can be embedded in various code search
methods, providing efficient code retrieval capabilities. It incorporates text
similarity, FQN (full quality name) of entities, and code popularity to rank
the code snippets. The DCSE (Cai et al., 2021), FACER (Abid et al., 2021), and
SSQR (Mao et al., 2023) use Lucene as a retrieval tool to match the expanded
queries with code snippets, demonstrating the effectiveness of their query
expansion technique.

Fu et al. (Fu et al., 2020) regard each method as recommendation objects,
denoted as documents, and store the objects in the indexes in the Lucene
framework. Each document will be divided into multiple fields according to
different code characteristics. When matched with the given query, they get a
series of keyword tokens from the query and implement the keyword-based search
in Lucene to find the retrieved results.

###  6.2. Vector-based Matching

According to Section 4.2 and Section 5.2, it can be found that vector is the
most common form of feature representations of queries and code snippets.
Therefore, there are naturally more code search techniques that use vector-
based matching methods to find relevant code snippets. Vector-based matching
methods calculate distances using vectors that extract fine-grained semantic
and syntactic features from code snippets and queries. Therefore, compared to
text-based matching, these methods can retrieve code snippets that have
similar features to the query, and improve overall code search performance.
The second line of Figure 10 shows that most current code search papers adopt
vector-based matching methods. Generally, there are two categories of methods
for generating feature vector representations for code snippets and queries:
using traditional IR methods to generate feature vectors and using DL neural
networks to generate embeddings (vectors generated by DL-based methods
referred to collectively as embeddings). Thereby, in this section, we divide
vector-based matching into vector distance-based methods and embedding
distance-based methods. Then, we will discuss and analyze vector distance
matching and embedding distance matching as two separate parts.

####  6.2.1. Vector Distance-based Matching

As mentioned earlier, vector distance-based matching methods retrieve the
relevant code snippets by calculating the distance between the IR-based
feature representation of both query and code snippet. In intuition, the
closer the code snippet and the query are in these spatial distances, the more
likely they are compatible in the semantics. Therefore, finding the code
snippet in the vector space that is closest to the query is the core of
distance-based matching.

Before calculating the distance between code snippets and queries,
representing them into the vectors is the first step of vector distance-based
matching methods. Traditional IR methods are always used to establish the
vectors. These vectors represent the semantics information of queries and code
snippets. After obtaining those vectors, they are used to match the query with
code snippets in the distance and retrieve the ranking results. The most
general way to measure the distance is cosine distance or Euclidean distance
(Salton et al., 1983). In the following, we will introduce 4 representative
code search works that execute the vector distance-based matching.

As a traditional IR technique, researchers find that the Vector Space Model
(VSM) is suitable for basic code search matching tasks (Niu et al., 2016;
Sirres et al., 2018). VSM utilizes the common representation method TF-IDF to
convert queries and code into vectors and then calculates their similarity
using the cosine similarity formula:

(32)
$\displaystyle\cos({v_{c}},{v_{q}})=\frac{{v_{c}}^{T}{v_{q}}}{\|{v_{c}}\|\|{v_{q}}\|}$

where $v_{c}$ and $v_{q}$ are the vectors of the code snippet and query,
respectively. The higher the similarity, the more related the code is to the
query. Thus, we categorize the technique that employed the VSM as a part of
vector distance-based matching.

Zhang et al. (Zhang et al., 2018) also utilize VSM to calculate vector cosine
distances, but in addition, they consider other features to measure the
matching results between queries and code snippets. Therefore, their proposed
weighted-sum ranking schema incorporates a total of five feature components in
the calculation, including $fv$, which is the similarity score generated by
Lucene; $fs$, which is the cosine distance similarity; $fn$, which represents
keyword term frequency; $fp$, which is the number of parameters in the code
example; and $fa$, which denotes the score for recommended API class names in
query expansion. The scores of these components are weighted and summed to
produce the final matching result.

Chen et al. (Chen et al., 2020) propose a semantics-based search for Java
methods named Quebio. Compared with most methods in the matching end, Quebio
combines a customized keyword-based search with a distance-based search to
check the relevant code snippets with a given query quickly. Quebio designs
the matching method in two steps. In the first step, a keyword-based search is
employed for quick filtering, which calculates the query keyword frequencies
in the code text. Only code snippets with results exceeding the threshold
enter the second step. In the second step, the selected code snippets are
preprocessed and sorted using the TF-IDF method, with the top 5 most
frequently appearing words selected as the code snippet’s summary. Then, VSM
is used to construct vectors for these summaries, and the cosine distance
between the summary vector of the code snippet and the query vector is
calculated as the final retrieval result.

####  6.2.2. Embedding Distance-based Matching

As its name implies, embedding distance-based matching methods utilize the
embeddings of the code snippet and query to calculate the distance, and
optimize the ranking results. The embeddings are obtained by the deep neural
network model, which can capture the semantics and structure in fine-grained,
thus the ranking performance can be significantly improved.

Similar to the vector distance-based matching method mentioned above, it is
necessary to measure the degree of match between queries and code snippets by
computing the distance between their embeddings.

Cosine distance is the most widely used in the field of feature similarity
comparison (Wang and Jiang, 2017). Cosine distance evaluates the similarity of
embeddings by calculating the cosine of their angle. The computation is
similar to Formula 32 but replaces $v_{c}$ and $v_{q}$ with $\bm{v_{c}}$ and
$\bm{v_{q}}$, representing the embeddings of code and query. Specifically,
most works (Gu et al., 2018; Sachdev et al., 2018; Chen and Zhou, 2018; Wan et
al., 2019; Gu et al., 2020; Shuai et al., 2020; de Rezende Martins and Gerosa,
2020; Ren et al., 2020; Hu et al., 2020a; Sun et al., 2020; Ling et al., 2020;
Xu et al., 2021; Gu et al., 2021; Liu et al., 2021b; Ling et al., 2021; Meng,
2021; Fang et al., 2021; Wu and Yan, 2022; Yu et al., 2022; Niu et al., 2022;
Deng et al., 2022; Kong et al., 2022b; Zhao and Liu, 2022; Cai et al., 2023;
Zeng et al., 2023) match the code snippets with query on their cosine distance
in the embedding space.

Actually, cosine similarity can be widely used not only in Euclidean spaces
but also in non-Euclidean spaces (i.e., hyperbolic spaces). As we mentioned in
Section 4, HyCoQA (Tang et al., 2023) introduces the Hyperbolic space to
express connections between code snippets and their corresponding queries.
Unlike traditional Euclidean spaces, hyperbolic spaces excel in representing
hierarchical structures, which frequently underlie the connection between code
and its corresponding natural language description. Thus, after utilizing the
BERT embedding layer to represent code snippets and queries into embeddings,
HyCoQA utilizes a hyperbolic embedder to transform the initial embeddings into
hyperbolic space. Finally, using Formula 32 to calculate the cosine similarity
between code snippet embeddings and query embeddings in hyperbolic spaces, the
obtained rankings are used for the retrieved result.

Both CoaCor (Yao et al., 2019) and ${TranS}^{3}$ (Wang et al., 2020b) generate
comments for code snippets to assist in code search. Therefore, they combine
two cosine similarities as the final score:

(33)
$\displaystyle\operatorname{score}(Q,C)=\beta*\operatorname{cos}\left(e_{q},e_{ci}\right)+(1-\beta)*\operatorname{cos}\left(e_{q},e_{si}\right)$

where $e_{q}$, $e_{ci}$, and $e_{si}$ are the embeddings of query, comments,
and code snippets, respectively. $\beta$ is a weight parameter that ranges
from 0 to 1, and $cos(\cdot)$ is the cosine similarity function. The first
cosine similarity is calculated based on the query and code snippet, and the
second one is calculated based on the query and generated comments. While, it
applies a weighting parameter to each of the two cosine distances, and then
combines the results to obtain the final similarity score for ranking code
snippets for a given query.

Similar to Formula 33, QueCos (Wang et al., 2022) also employs a hybrid
ranking approach that combines the weighted sum of two cosine similarities as
the final similarity score. The difference lies in the first cosine similarity
calculates between enriched queries and code snippets, and the second cosine
similarity calculates between the original query and code snippets. QueCos can
enrich the original query semantically through reinforcement learning,
generating enriched queries. Therefore, the designed hybrid ranking approach
considers both the original query and enriched queries to return the ultimate
search results.

The inner product distance is also a widely used measure of the similarity
between embeddings. Both CodeBERT (Feng et al., 2020) and GraphCodeBERT (Guo
et al., 2021) leverage the inner product distance to match the code snippets
with the query. CodeBERT and GraphCodeBERT are the pre-trained models for the
programming language, which can be used in a series of downstream tasks,
including code search. In their experiments, six programming language datasets
are used to fine-tune the downstream code search tasks. All the tasks
calculate the inner product of code and query embeddings as relevant scores to
rank candidate codes.

Euclidean distance (i.e., L2 distance) calculates the straight-line distance
between two points in a multidimensional space. It is also a commonly used
measure of the similarity between embeddings. MP-CAT (Haldar et al., 2020)
computes the L2 distance in the similarity module:

(34)
$\displaystyle\operatorname{sim}\left(e_{c},e_{q}\right)=1-d\left(\frac{e_{c}}{\left\|e_{c}\right\|_{2}},\frac{e_{q}}{\left\|e_{q}\right\|_{2}}\right)$

where $d(\cdot)$ denotes the L2 distance calculation for the dimensional code
embeddings $e_{c}$ and query embeddings $e_{q}$. The similarity module selects
the code snippet with a close similarity to the given query. Finally, the
output of the similarity module is regarded as the final output of the
retrieved result.

###  6.3. Classification-based Matching

The classification-based matching methods are different from the distance-
based method. Instead of calculating the distance of vectors or embeddings
from the code snippets and queries or using the textual information to match
the code snippets with the query, classification-based matching methods
transform the matching task into a classification task. Such methods utilize
the classifier to predict the probability of semantics matching and use the
predicted probabilities to rank the candidate code snippets. In the following,
we will discuss them in detail.

To implement an effective and efficient code search system, Gotmare et al.
(Gotmare et al., 2021) propose a hybrid ranking framework called CASCODE,
which includes the fast encoder and slow classifier to improve the performance
of retrieving the search results. They prove that leveraging classification
tasks involving NL-PL sequence pairs for code retrieval can achieve an optimal
result. However, adopting this approach would be impractical due to the large
number of candidates to be considered for each query. Therefore, they divide
the retrieval process into two stages. In the first stage, the transformer
encoders jointly transform the natural language query and code snippet into
the embeddings and calculate the cosine distance to provide the top $k$
candidate code snippets. In general, this fast encoder stage ensures that the
framework can quickly retrieve the top few candidate code snippets that are
relatively close to the query from the code repository. In the slow classifier
stage, they utilize a transformer encoder-based classifier to predict the top
$k$ candidate code snippets from the first stage. The researchers’ experiments
demonstrate that the well-trained classifier retrieves code snippets by
returning the match probabilities between queries and code snippets in
semantics, resulting in excellent performance. However, using a classifier is
costly and slower. As for the limited number of candidate code snippets, a
transformer classifier jointly processes the query sequence with each of the
candidates to predict the probability of their semantics matching has become
feasible. The predicted score of each candidate is regarded as the metric to
rank the final retrieved results.

To address the problem of overlaps (e.g., “message” and “msg”), which has a
negative impact on retrieved results, Zhuet et al. (Zhu et al., 2020) propose
a novel neural architecture called OCoR. In the matching end, similar to the
slow classifier stage of CASCODE, OCoR also utilizes a transformer classifier
to predict the probability of two classes. The first class denotes that the
input natural language query and the input code are related, whereas the
second class denotes that the input natural language query and the input code
are unrelated. The predicted classification probability of the first class is
the relevance score between the input natural language query and the code
snippet. The relevance scores are used for ranking the relevant code snippets.

To provide a better code retrieved result, Jiang et al. (Jiang et al., 2019)
propose a novel method combining both information retrieval and code
classification, called ROSF. ROSF decomposes code search into two stages.
Specifically, in the first stage, ROSF utilizes the IR-based method (BM25) to
retrieve a candidate set that contains $N$ code snippets. This is a coarse-
grained search for identifying a few relevant candidate code snippets. In the
next fine-grained re-ranking stage, ROSF views the problem of ranking the
candidate code snippets as a multi-class classification task. For each
instance of the candidate set for a new query, ROSF employs the learned linear
predictor function to predict the probabilities of four possible relevance
scores. In other words, each instance has four probability values
corresponding to four relevance scores. Then, the relevance score with the
maximum probability value is selected as the predicted relevance score for the
instance. Among the candidate sets, ROSF finally sorts the subset containing
the code snippets with predicted $score4$ according to the predicted
probability values in descending order.

To accelerate the retrieval efficiency of deep learning-based code search
approaches, Gu et al. (Gu et al., 2022) propose a novel approach called CoSHC,
which adopts the recall and re-rank mechanism with the integration of code
classification and deep hashing to improve code search performance. CoSHC
first generates the code and description embeddings from deep learning
networks. Then, a deep hashing module is utilized to generate the
corresponding binary hash codes for the embeddings in binary hashing space.
Since the capacity of binary hashing space is very limited compared to
Euclidean space, they cluster the source code whose representation vectors are
close to each other into the same category. After obtaining the code
representation categories in hashing space, the classifier in the category
prediction module will calculate the probability distribution of categories
for the given query. The number of code candidates $R_{i}$ for each category
$i$ will be recalled according to this probability distribution, which can be
computed as:

(35) $\displaystyle R_{i}=\min\left(\left\lfloor
p_{i}\cdot(N-k)\right\rfloor,1\right),i\in 1,\ldots,k,$

where $p_{i}$ is the predicted probability for category $i$; $N$ is the total
recall number of source code and $k$ is the number of categories. In the final
re-ranking stage, the original representation vectors of these recalled code
candidates will be retrieved and utilized for the cosine similarity
calculation.

Summary for RQ3: There are three optimization methods utilized in the match
end by existing code search techniques, i.e., text matching-based methods,
vector matching-based methods, and classification-based methods. All three
types of methods have received widespread attention. Among them, with the
popularity of DL technology, vector matching-based methods (especially
embedding distance-based methods) are the most common. Although
classification-based methods are also based on embeddings, they require
training additional classification layers, which means they have higher
training costs. It is foreseeable that embedding distance-based methods will
remain mainstream for some time in the future.

##  7\. Challenges and Opportunities for Future Work

After analyzing the existing code search techniques in both advantages and
disadvantages, we can suggest that further research will also have practical
and research significance. From the practical significance of implementing
code search, these excellent techniques will certainly help us solve the
difficulties in the development of software. On the other hand, code search
still has many challenges to be solved. We pose some of the challenges and
opportunities that have emerged from the papers below.

###  7.1. Challenges

Challenges in Query Feature Mining. Query reduction aims to remove
noisy/redundant terms of the query and preserve important query features.
However, it may remove certain terms might result in a loss of semantic
context, making it challenging to capture the user’s intent accurately. In
addition, users might have different expectations regarding the reduction
process, and striking a balance between code search precision and recall is
challenging. Query expansion aims to add new terms to the original query to
enrich query features. However, the relevance and quality of the newly added
terms play a significant role. Using irrelevant or inaccurate terms can hinder
the search process. How to control the quality of extensions is challenging.
Query transformation aims to transform the original query feature into an
alternative form feature, e.g., code API. However, transforming a query into
an appropriate code-related feature while preserving its semantic meaning is
challenging. Ensuring accurate transformation is crucial for retrieving
relevant results. Additionally, some users search for code in domain-specific
programming languages that might not have direct mappings from natural
language queries. Adapting query transformation rules to handle domain-
specific code retrieval is a challenge.

Challenges in Query Feature Representation. IR-based query feature
representation methods represent queries as some form of index (such as
keyword terms and vectors). However, queries and code snippets might use
different vocabularies or terminologies, leading to a mismatch in the
representation. How to align the vocabularies effectively is crucial and
challenging for accurate retrieval. DL-based query feature representation
methods apply deep neural networks to encode the given query feature to
produce embeddings. However, queries often contain ambiguous terms or phrases
that can have multiple meanings in the context of code. It is challenging for
deep learning models to discern the intended meaning of ambiguous queries
accurately. In addition, queries can vary significantly in length, making it
challenging to process them uniformly. The applied models need to handle
variable-length inputs effectively.

Challenges in Code Feature Mining. Although existing code search techniques
have mined various code features to represent code semantics, how to fully
utilize each feature is still full of challenges. In addition, there is
information overlap between different code features. For example, Token and
the labels of AST leaf nodes overlap. Therefore, how to deal with this
redundant information is also a challenge when wanting to consider multiple
features simultaneously. Finally, whether structural features without Token
information (e.g., syntactic structure in AST, control flow in CFG, and data
flow in DFG) actually help facilitate code representation and code search
tasks still lacks systematic research. The diversity of feature parsing tools
and preprocessing methods makes such systematic research challenging.

Challenges in Code Feature Representation. Using deep learning techniques to
represent code features has become mainstream nowadays. Compared to
traditional IR-based code feature representation methods, the code
representations generated by DL-based cod feature representation methods can
better capture the semantics of code. However, DL-based methods require large,
high-quality labeled datasets for effective training. Obtaining such datasets
with accurately labeled code snippets can be challenging, especially for
specific programming languages or domains. In addition, code snippets also
vary significantly in length, from short expressions to lengthy functions.
Designing representations that handle variable-length inputs effectively is a
challenge. Moreover, as mentioned earlier, code contains multiple textual and
structural features. Integrating these diverse features into a cohesive
representation without losing essential information is challenging.

Challenges in Query-Code Matching. Since different developers have different
wording habits, this will cause the text-based matching method to fail to
retrieve the expected results. Therefore, mapping relationships between query
vocabulary and code vocabulary is challenging for methods of this kind. For
vector distance-based matching methods, it is necessary for the query vectors
and code vectors to be in a unified vector space, and the distance between
vectors should accurately reflect semantic relevance. However, whether
utilizing IR techniques or DL techniques, mapping queries and code snippets to
a unified vector space while preserving their respective semantics is a
challenging task. Instead of directly calculating distance using query and
code vectors, classification-based matching methods involve concatenating the
two vectors and utilizing a classifier for binary classification. However,
correctly concatenating query and code vectors is challenging because vectors
are abstract, and deep learning is low in interpretability. Last but not
least, an aspect that requires more attention is the efficiency challenge
faced by all query-code matching methods when dealing with extensive code
search corpus. Therefore, designing efficient matching methods is also a
challenging task.

###  7.2. Opportunities

Opportunities in Query End: As discussed in Section 4, the effectiveness of
query reduction, expansion, and transformation methods has been validated in
multiple code search techniques, demonstrating their ability to enhance
accurate code retrieval. Therefore, the simultaneous integration of these
three methods presents an opportunity to enhance query feature mining and
subsequently improve code search performance. Whether it is possible to
utilize open-source data to establish alignment between query terms and
commonly used tokens in code before inputting them into the model, thereby
enhancing the representation of query features, is also a good opportunity.

Opportunities in Code End: While existing research has indicated that
combining multiple code features can enhance code search performance, it
appears that each code feature has not yet exerted its maximum value.
Therefore, there are still ample opportunities for research in the integration
of multiple code features (especially, multimodal code feature data).
Additionally, a thorough analysis of the impact of various code features on
code representation and subsequent code search performance is a valuable
research opportunity. Lastly, the recent emergence of large language models
(e.g., ChatGPT 161616https://chat.openai.com), particularly large code models
(e.g., StarCoder 171717https://github.com/bigcode-project/starcoder), has
provided new opportunities for code feature exploration and representation.
For instance, the code feature extraction and representation capabilities of
large language models are worth investigating.

Opportunities in Match End: First, regarding the challenges mentioned in
Section 7.1, a research opportunity is investigating how to map queries and
code snippets to a unified vector space while preserving their respective
semantics. Second, the impact of the concatenation of query and code vectors
on classification-based query code matching needs to be explored, which also
poses research opportunities. Last, there are still numerous research
opportunities to enhance the efficiency of matching queries and code. For
example, one could explore first clustering a large number of code snippets
and then performing query code matching.

##  8\. Threats to validity

In this chapter, we explore the validity threats to our study. Validity
concerns the relationship between the research results and the actual
situation, as well as how the conclusions might be incorrect. We have
categorized the threats to the validity of our study into two types, including
internal validity and external validity. Our discussions on these identified
threats are as follows:

###  8.1. Threats to Internal Validity

The threats to internal validity refer to experiment errors and human biases
(Tian et al., 2014). In our study, we employ a cautious research strategy to
mitigate threats to internal validity. Firstly, we establish three specific
research questions covering the different essential dimensions of the code
search technique (detailed in Section 3.1). Subsequently, we utilize the PIO
approach to generate keywords from these research questions and collect papers
from six widely used electronic databases (including DBLP, Google Scholar,
IEEE Explore, etc.). These collected papers under multiple rounds of manual
filtering and screening based on our defined research scope and criteria
(detailed in Section 3.2, 3.3). From the remaining 1427 papers, we
meticulously review 68 of them, analyzing their technical aspects according to
the three research perspectives. Additionally, we explore and discuss
potential future developments based on existing work within each perspective.
As a result, the systematic exploration has instilled confidence in our work
and established a certain level of effectiveness in the research outcomes.

###  8.2. Threats to External Validity

External validity concerns the extent to which the results can be generalized
beyond the scope of the study, even when specific cause-and-effect
relationships have been established in the study. Threats to external validity
involve the generalizability of reported research findings (Tian et al.,
2014). For our study, we meticulously select 68 papers from six commonly used
electronic databases. Hence, these papers likely cover the core studies in the
code search field. Therefore, the conclusions and findings drawn in our study
can potentially apply to the techniques inadvertently missed in this paper or
those emerging as new techniques in the near future. Furthermore, within each
research question section, we have conducted theoretical qualitative analyses.
Hence, we believe that external validity threats are not particularly
challenging for this study.

##  9\. Conclusion

This article provides a 3-dimensional survey of the technological developments
in code search over the past thirty years. This survey focuses on the three
core components of code search technology, i.e., query understanding
component, code understanding component, and query-code matching component. We
classify and discuss the optimization techniques proposed for each component.
Specifically, we divide the techniques for optimizing the query and code ends
into two parts, namely feature mining and feature representation,
respectively. For the match end, we categorize existing optimization
techniques into three major classes, i.e., text-based matching, vector
distance-based matching, and classification-based matching. For each end (each
optimization category), we provide a detailed introduction to some
representative optimization technologies and summarize the technology
development trends. Based on the comprehensive observation of optimization
techniques proposed in existing code search papers, we summarize some
challenges that still need to be addressed and suggest research opportunities.

##  10\. Acknowledgments

This work is supported partially by the National Natural Science Foundation of
China (61932012, 62141215), and the Program B for Outstanding PhD Candidate of
Nanjing University (202201B054).

## References

  * (1)
  * Abid et al. (2021) Shamsa Abid, Shafay Shamail, Hamid Abdul Basit, and Sarah Nadi. 2021.  FACER: An API usage-based code-example recommender for opportunistic reuse.  _Empirical Software Engineering_ 26, 5 (2021), 110\.
  * Allamanis et al. (2018) Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. 2018\.  Learning to Represent Programs with Graphs. In _Proceedings of the 6th International Conference on Learning Representations_. OpenReview.net, Vancouver, BC, Canada, 1–17.
  * Alon et al. (2018) Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. 2018.  A general path-based representation for predicting program properties. In _Proceedings of the 39th Conference on Programming Language Design and Implementation_. ACM, Philadelphia, PA, USA, 404–419.
  * Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003\.  Latent dirichlet allocation.  _Journal of machine Learning research_ 3, January (2003), 993–1022.
  * Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomás Mikolov. 2017.  Enriching Word Vectors with Subword Information.  _Transactions of the Association for Computational Linguistics_ 5 (2017), 135–146.
  * Brandt et al. (2009) Joel Brandt, Philip J. Guo, Joel Lewenstein, Mira Dontcheva, and Scott R. Klemmer. 2009.  Two studies of opportunistic programming: interleaving web foraging, learning, and writing code. In _Proceedings of the 27th International Conference on Human Factors in Computing Systems_. ACM, Boston, MA, USA, 1589–1598.
  * Cai et al. (2023) Bo Cai, Yaoxiang Yu, and Yi Hu. 2023.  CSSAM: Code Search via Attention Matching of Code Semantics and Structures. In _Proceedings of the 30th International Conference on Software Analysis, Evolution and Reengineering_. IEEE, Taipa, Macao, 402–413.
  * Cai et al. (2021) Fuqi Cai, Changjing Wang, Qing Huang, Zhengkang Zuo, and Yunyan Liao. 2021.  Search for Compatible Source Code.  _International Journal of Software Engineering and Knowledge Engineering_ 31, 3 (2021), 477–502.
  * Cambronero et al. (2019) José Cambronero, Hongyu Li, Seohyun Kim, Koushik Sen, and Satish Chandra. 2019\.  When deep learning met code search. In _Proceedings of the 13th Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. ACM, Tallinn, Estonia, 964–974.
  * Cao et al. (2007) Guihong Cao, Jianfeng Gao, Jian-Yun Nie, and Jing Bai. 2007\.  Extending query translation to cross-language query expansion with markov chain models. In _Proceedings of the 16th conference on Conference on information and knowledge management_. ACM, Lisbon, Portugal, 351–360.
  * Chen and Zhou (2018) Qingying Chen and Minghui Zhou. 2018.  A neural framework for retrieval and summarization of source code. In _Proceedings of the 33rd International Conference on Automated Software Engineering_. ACM, Montpellier, France, 826–831.
  * Chen et al. (2018) Xinyun Chen, Chang Liu, and Dawn Song. 2018.  Tree-to-tree Neural Networks for Program Translation. In _Proceedings of the 6th International Conference on Learning Representations_. OpenReview.net, Vancouver, BC, Canada, 1–11.
  * Chen et al. (2020) Zhengzhao Chen, Renhe Jiang, Zejun Zhang, Yu Pei, Minxue Pan, Tian Zhang, and Xuandong Li. 2020.  Enhancing example-based code search with functional semantics.  _Journal of Systems and Software_ 165, 1 (2020), 110568\.
  * Cheng and Kuang (2022) Yi Cheng and Li Kuang. 2022\.  CSRS: Code Search with Relevance Matching and Semantic Matching.  _arXiv preprint arXiv:2203.07736_ (2022).
  * Chung et al. (2014) Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. 2014\.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling.  _CoRR_ abs/1412.3555 (2014).
  * de Rezende Martins and Gerosa (2020) Marcelo de Rezende Martins and Marco Aurélio Gerosa. 2020\.  CoNCRA: A Convolutional Neural Networks Code Retrieval Approach. In _Proceedings of the 34th Brazilian Symposium on Software Engineering_. ACM, Natal, Brazil, 526–531.
  * Deng et al. (2022) Zhongyang Deng, Ling Xu, Chao Liu, Meng Yan, Zhou Xu, and Yan Lei. 2022.  Fine-grained Co-Attentive Representation Learning for Semantic Code Search. In _Proceedings of the 29th International Conference on Software Analysis, Evolution and Reengineering_. IEEE, Honolulu, HI, USA, 396–407.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Minneapolis, MN, USA, 4171–4186.
  * Du et al. (2021) Lun Du, Xiaozhou Shi, Yanlin Wang, Ensheng Shi, Shi Han, and Dongmei Zhang. 2021\.  Is a Single Model Enough? MuCoS: A Multi-Model Ensemble Learning Approach for Semantic Code Search. In _Proceedings of the 30th International Conference on Information & Knowledge Management_. ACM, Queensland, Australia, 2994–2998.
  * Eisenstein (2019) Jacob Eisenstein. 2019\.  _Introduction to natural language processing_.  MIT press.
  * Fang et al. (2021) Sen Fang, Youshuai Tan, Tao Zhang, and Yepang Liu. 2021\.  Self-Attention Networks for Code Search.  _Information & Software Technology_ 134, 1 (2021), 106542\.
  * Feng et al. (2020) Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020.  CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In _Findings of the Association for Computational Linguistics: EMNLP_. Association for Computational Linguistics, Online Event, 1536–1547.
  * Ferrante et al. (1987) Jeanne Ferrante, Karl J. Ottenstein, and Joe D. Warren. 1987\.  The Program Dependence Graph and Its Use in Optimization.  _ACM Transactions on Programming Languages and Systems_ 9, 3 (1987), 319–349.
  * Floyd and Kinser (1990) Edwin T Floyd and J Tom Kinser. 1990.  Service Code Search High Performance Medical Records Indexing on a Budget. In _Proceedings of the Annual Symposium on Computer Application in Medical Care_. American Medical Informatics Association, USA, 408.
  * Frakes and Nejmeh (1987) William B. Frakes and Brian A. Nejmeh. 1987.  Software Reuse Through Information Retrieval.  _SIGIR Forum_ 21, 1-2 (1987), 30–36.
  * Fu et al. (2020) Shanqing Fu, Bing Li, Yi Cai, Zhuang Liu, and Junxia Guo. 2020.  Recommendation Based on Java Code Analysis and Search. In _Proceedings of the 6th Fuzzy Systems and Data Mining_ , Vol. 331. IOS Press, Virtual Event, 514–521.
  * GitHub (2008) Inc. GitHub. 2008\.  GitHub.  site: https://github.com.  Accessed: 2022.
  * Gotmare et al. (2021) Akhilesh Deepak Gotmare, Junnan Li, Shafiq Joty, and Steven C.H. Hoi. 2021.  Cascaded Fast and Slow Models for Efficient Semantic Code Search.  _CoRR_ abs/2110.07811, 1 (2021), 1–12.
  * Grazia and Pradel (2022) Luca Di Grazia and Michael Pradel. 2022.  Code Search: A Survey of Techniques for Finding Code.  _CoRR_ abs/2204.02765, 1 (2022), 1–30.
  * Gu et al. (2021) Jian Gu, Zimin Chen, and Martin Monperrus. 2021.  Multimodal Representation for Neural Code Search. In _Proceedings of the 37th International Conference on Software Maintenance and Evolution_. IEEE, Luxembourg, 483–494.
  * Gu et al. (2020) Wenchao Gu, Zongjie Li, Cuiyun Gao, Chaozheng Wang, Hongyu Zhang, Zenglin Xu, and Michael R. Lyu. 2020.  CRaDLe: Deep Code Retrieval Based on Semantic Dependency Learning.  _Neural Networks_ 141, 1 (2020), 385–394.
  * Gu et al. (2022) Wenchao Gu, Yanlin Wang, Lun Du, Hongyu Zhang, Shi Han, Dongmei Zhang, and Michael R. Lyu. 2022.  Accelerating Code Search with Deep Hashing and Code Classification. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Dublin, Ireland, 2534–2544.
  * Gu et al. (2018) Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. 2018\.  Deep code search. In _Proceedings of the 40th International Conference on Software Engineering_. ACM, Gothenburg, Sweden, 933–944.
  * Gu et al. (2016) Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, and Sunghun Kim. 2016.  Deep API learning. In _Proceedings of the 24th International Symposium on Foundations of Software Engineering_. ACM, Seattle, WA, USA, 631–642.
  * Guo et al. (2021) Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin B. Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021\.  GraphCodeBERT Pre-training Code Representations with Data Flow. In _Proceedings of the 9th International Conference on Learning Representations_. OpenReview.net, Virtual Event, Austria, 1–18.
  * Haiduc et al. (2013) Sonia Haiduc, Gabriele Bavota, Andrian Marcus, Rocco Oliveto, Andrea De Lucia, and Tim Menzies. 2013\.  Automatic query reformulations for text retrieval in software engineering. In _Proceedings of the 35th International Conference on Software Engineering_. IEEE Computer Society, San Francisco, CA, USA, 842–851.
  * Haldar et al. (2020) Rajarshi Haldar, Lingfei Wu, Jinjun Xiong, and Julia Hockenmaier. 2020.  A Multi-Perspective Architecture for Semantic Code Search. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,ACL_. Association for Computational Linguistics, online, 8563–8568.
  * Hinton et al. (2006) Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh. 2006\.  A Fast Learning Algorithm for Deep Belief Nets.  _Neural Comput._ 18, 7 (2006), 1527–1554.
  * Hu et al. (2020b) Gang Hu, Min Peng, Yihan Zhang, Qianqian Xie, Wang Gao, and Mengting Yuan. 2020b.  Unsupervised software repositories mining and its application to code search.  _Software - Practice and Experience_ 50, 3 (2020), 299–322.
  * Hu et al. (2020a) Gang Hu, Min Peng, Yihan Zhang, Qianqian Xie, and Mengting Yuan. 2020a.  Neural joint attention code search over structure embeddings for software Q&A sites.  _Journal of Systems and Software_ 170, 1 (2020), 110773\.
  * Hu et al. (2018) Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2018.  Deep code comment generation. In _Proceedings of the 26th Conference on Program Comprehension_. ACM, Gothenburg, Sweden, 200–210.
  * Huang et al. (2019) Qing Huang, Yang Yang, and Ming Cheng. 2019.  Deep learning the semantics of change sequences for query expansion.  _Software - Practice and Experience_ 49, 11 (2019), 1600–1617.
  * Huang et al. (2018) Qing Huang, Yangrui Yang, Xue Zhan, Hongyan Wan, and Guoqing Wu. 2018.  Query expansion based on statistical learning from code changes.  _Software: Practice and Experience_ 48, 7 (2018), 1333–1351.
  * Hussain et al. (2008) Ishrar Hussain, Leila Kosseim, and Olga Ormandjieva. 2008\.  Using Linguistic Knowledge to Classify Non-functional Requirements in SRS documents. In _Proceedings of the 13th International Conference on Applications of Natural Language to Information Systems_. Springer, London, UK, 287–298.
  * Inc; (2008) Stack Exchange Inc;. 2008\.  Stack Overflow.  site: https://stackoverflow.com/.  Accessed: 2022.
  * Jiang et al. (2019) He Jiang, Liming Nie, Zeyi Sun, Zhilei Ren, Weiqiang Kong, Tao Zhang, and Xiapu Luo. 2019.  ROSF: Leveraging Information Retrieval and Supervised Learning for Recommending Code Snippets.  _IEEE Transactions on Services Computing_ 12, 1 (2019), 34–46.
  * Karnalim (2018) Oscar Karnalim. 2018\.  Language-agnostic source code retrieval using keyword & identifier lexical pattern.  _International Journal of Software Engineering and Computer Systems_ 4, 1 (2018), 29–47.
  * Kim et al. (2021) Seohyun Kim, Jinman Zhao, Yuchi Tian, and Satish Chandra. 2021\.  Code prediction by Feeding Trees to Transformers. In _Proceedings of the 43rd International Conference on Software Engineering_. IEEE, Madrid, Spain, 150–162.
  * Kitchenham and Brereton (2013a) Barbara A. Kitchenham and Pearl Brereton. 2013a.  A systematic review of systematic review process research in software engineering.  _Information & Software Technology_ 55, 12 (2013), 2049–2075.
  * Kitchenham and Brereton (2013b) Barbara A. Kitchenham and Pearl Brereton. 2013b.  A systematic review of systematic review process research in software engineering.  _Information & Software Technology_ 55, 12 (2013), 2049–2075.
  * Kobayashi and Niimi (1989) Yutaka Kobayashi and Yasuhisa Niimi. 1989.  An Efficient VQ Code Search Algorithm Using Signal Continuity. In _Processings of the 1st European Conference on Speech Communication and Technology_. ISCA, Paris, France, 1446–1449.
  * Kong et al. (2022a) Xianglong Kong, Hongyu Chen, Ming Yu, and Lixiang Zhang. 2022a.  Boosting Code Search with Structural Code Annotation.  _Electronics_ 11, 19 (2022), 3053.
  * Kong et al. (2022b) Xianglong Kong, Supeng Kong, Ming Yu, and Chengjie Du. 2022b.  Joint Embedding of Semantic and Statistical Features for Effective Code Search.  _Applied Sciences_ 12, 19 (2022), 10002.
  * Kovalenko et al. (2019) Vladimir Kovalenko, Egor Bogomolov, Timofey Bryksin, and Alberto Bacchelli. 2019.  PathMiner: a library for mining of path-based representations of code. In _Proceedings of the 16th International Conference on Mining Software Repositories_. IEEE / ACM, Montreal, Canada, 13–17.
  * Leacock and Chodorow (1998) Chodorow Leacock and M Chodorow. 1998.  Wordnet: An electronic lexical database-combining local context and wordnet similarity for word sense identification, in wordnet: An electronic lexical database.
  * Ling et al. (2020) Chunyang Ling, Zeqi Lin, Yanzhen Zou, and Bing Xie. 2020\.  Adaptive Deep Code Search. In _Proceedings of the 28th International Conference on Program Comprehension_. ACM, Seoul, Republic of Korea, 48–59.
  * Ling et al. (2021) Xiang Ling, Lingfei Wu, Saizhuo Wang, Gaoning Pan, Tengfei Ma, Fangli Xu, Alex X. Liu, Chunming Wu, and Shouling Ji. 2021.  Deep Graph Matching and Searching for Semantic Code Retrieval.  _ACM Transactions on Knowledge Discovery from Data_ 15, 5 (2021), 88:1–88:21.
  * Linstead et al. (2009) Erik Linstead, Sushil Krishna Bajracharya, Trung Chi Ngo, Paul Rigor, Cristina Videira Lopes, and Pierre Baldi. 2009.  Sourcerer: mining and searching internet-scale software repositories.  _Data Mining and Knowledge Discovery_ 18, 2 (2009), 300–336.
  * Liu et al. (2022) Chao Liu, Xin Xia, David Lo, Cuiyun Gao, Xiaohu Yang, and John C. Grundy. 2022\.  Opportunities and Challenges in Code Search Tools.  _Comput. Surveys_ 54, 9 (2022), 196:1–196:40.
  * Liu et al. (2021a) Chao Liu, Xin Xia, David Lo, Zhiwe Liu, Ahmed E Hassan, and Shanping Li. 2021a.  CodeMatcher: Searching Code Based on Sequential Semantics of Important Query Words.  _ACM Transactions on Software Engineering and Methodology (TOSEM)_ 31, 1 (2021), 1–37.
  * Liu et al. (2019) Jason Liu, Seohyun Kim, Vijayaraghavan Murali, Swarat Chaudhuri, and Satish Chandra. 2019.  Neural query expansion for code search. In _Proceedings of the 3rd International Workshop on Machine Learning and Programming Languages_. ACM, Phoenix, AZ, USA, 29–37.
  * Liu et al. (2021b) Shangqing Liu, Xiaofei Xie, Lei Ma, Jing Kai Siow, and Yang Liu. 2021b.  GraphSearchNet: Enhancing GNNs via Capturing Global Dependency for Semantic Code Search.  _CoRR_ abs/2111.02671, 1 (2021), 1–14.
  * Liu et al. (2023) Shangqing Liu, Xiaofei Xie, Jing Kai Siow, Lei Ma, Guozhu Meng, and Yang Liu. 2023\.  GraphSearchNet: Enhancing GNNs via Capturing Global Dependencies for Semantic Code Search.  _IEEE Transactions on Software Engineering_ 49, 4 (2023), 2839–2855.
  * Liu et al. (2018) Xiaoyu Liu, LiGuo Huang, and Vincent Ng. 2018.  Effective API recommendation without historical software repositories. In _Proceedings of the 33rd International Conference on Automated Software Engineering_. ACM/IEEE, Montpellier, France, 282–292.
  * Lu et al. (2018) Jinting Lu, Ying Wei, Xiaobing Sun, Bin Li, Wanzhi Wen, and Cheng Zhou. 2018\.  Interactive Query Reformulation for Source-Code Search With Word Relations.  _IEEE Access_ 6, 1 (2018), 75660–75668.
  * Lu et al. (2015) Meili Lu, Xiaobing Sun, Shaowei Wang, David Lo, and Yucong Duan. 2015.  Query expansion via WordNet for effective code search. In _Proceedings of the 22nd International Conference on Software Analysis, Evolution, and Reengineering_. IEEE, Montreal, QC, Canada, 545–549.
  * Lv et al. (2015) Fei Lv, Hongyu Zhang, Jian-Guang Lou, Shaowei Wang, Dongmei Zhang, and Jianjun Zhao. 2015\.  CodeHow: Effective Code Search Based on API Understanding and Extended Boolean Model (E). In _Proceedings of the 30th International Conference on Automated Software Engineering_. IEEE/ACM, Lincoln, NE, USA, 260–270.
  * Mao et al. (2023) Yuetian Mao, Chengcheng Wan, Yuze Jiang, and Xiaodong Gu. 2023\.  Self-Supervised Query Reformulation for Code Search.
  * Mar et al. (2011) Lee Wei Mar, Ye-Chi Wu, and Hewijin Christine Jiau. 2011\.  Recommending Proper API Code Examples for Documentation Purpose. In _Proceedings of the 18th Asia Pacific Software Engineering Conference_. IEEE Computer Society, Ho Chi Minh, Vietnam, 331–338.
  * McMillan et al. (2011) Collin McMillan, Mark Grechanik, Denys Poshyvanyk, Qing Xie, and Chen Fu. 2011.  Portfolio: finding relevant functions and their usage. In _Proceedings of the 33rd International Conference on Software Engineering_. ACM, Waikiki, Honolulu ,HI, USA, 111–120.
  * Meng (2021) Yao Meng. 2021.  An Intelligent Code Search Approach Using Hybrid Encoders.  _Wirel. Commun. Mob. Comput._ 2021, 1 (2021), 9990988:1–9990988:16.
  * Mikolov et al. (2013a) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a.  Efficient Estimation of Word Representations in Vector Space. In _Proceedings of the 1st International Conference on Learning Representations_. OpenReview.net, Scottsdale, Arizona, USA, 1–12.
  * Mikolov et al. (2013b) Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013b.  Distributed Representations of Words and Phrases and their Compositionality. In _Proceedings of the 27th Annual Conference on Neural Information Processing Systems_. MIT Press, Lake Tahoe, Nevada, United States, 3111–3119.
  * Mitra and Craswell (2017) Bhaskar Mitra and Nick Craswell. 2017.  Neural Models for Information Retrieval.  _CoRR_ abs/1705.01509 (2017), 1–52.
  * Nie et al. (2016) Liming Nie, He Jiang, Zhilei Ren, Zeyi Sun, and Xiaochen Li. 2016.  Query Expansion Based on Crowd Knowledge for Code Search.  _IEEE Transactions on Services Computing_ 9, 5 (2016), 771–783.
  * Niu et al. (2022) Changan Niu, Chuanyi Li, Vincent Ng, Jidong Ge, Liguo Huang, and Bin Luo. 2022\.  SPT-Code: Sequence-to-Sequence Pre-Training for Learning the Representation of Source Code.  _arXiv_ (2022), 1–13.
  * Niu et al. (2016) Haoran Niu, Iman Keivanloo, and Ying Zou. 2016\.  Learning to rank code examples for code search engines.  _Empirical Software Engineering_ 22, 1 (2016), 259–291.
  * Pandita et al. (2016) Rahul Pandita, Kunal Taneja, Laurie A. Williams, and Teresa Tung. 2016.  ICON: Inferring Temporal Constraints from Natural Language API Descriptions. In _Proceedings of the 32nd International Conference on Software Maintenance and Evolution_. IEEE Computer Society, Raleigh, NC, USA, 378–388.
  * Petersen et al. (2015) Kai Petersen, Sairam Vakkalanka, and Ludwik Kuzniarz. 2015\.  Guidelines for conducting systematic mapping studies in software engineering: an update.  _Information & Software Technology_ 64, 1 (2015), 1–18.
  * Polydoros and Weber (1984) Andreas Polydoros and Charlesl Weber. 1984.  A unified approach to serial search spread-spectrum code acquisition-Part I: General theory.  _IEEE Transactions on communications_ 32, 5 (1984), 542–549.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ 21, 1 (2020), 5485–5551.
  * Raghothaman et al. (2016) Mukund Raghothaman, Yi Wei, and Youssef Hamadi. 2016\.  SWIM: synthesizing what i mean: code search and idiomatic snippet synthesis. In _Proceedings of the 38th International Conference on Software Engineering_. Association for computing machinery, Austin, TX, USA, 357–367.
  * Rahman and Roy (2016) Mohammad Masudur Rahman and Chanchal K. Roy. 2016.  QUICKAR: automatic query reformulation for concept location using crowdsourced knowledge. In _Proceedings of the 31st International Conference on Automated Software Engineering_. ACM, Singapore, 220–225.
  * Rahman and Roy (2021) Mohammad Masudur Rahman and Chanchal K. Roy. 2021.  A Systematic Literature Review of Automated Query Reformulations in Source Code Search.  _CoRR_ abs/2108.09646, 1 (2021), 1–68.
  * Rahman et al. (2016) Mohammad Masudur Rahman, Chanchal Kumar Roy, and David Lo. 2016\.  RACK: Automatic API Recommendation Using Crowdsourced Knowledge. In _Proceedings of the 23rd International Conference on Software Analysis, Evolution, and Reengineering_. IEEE Computer Society, Suita, Osaka, Japan, 349–359.
  * Rahman et al. (2019) Mohammad Masudur Rahman, Chanchal K. Roy, and David Lo. 2019\.  Automatic Query Reformulation For code search using crowdsourced knowledge.  _Empirical Software Engineering_ 24, 4 (2019), 1869–1924.
  * Ren et al. (2020) Leiming Ren, Shinmin Shan, Kai Wang, and Kun Xue. 2020\.  CSDA: A novel attention-based LSTM approach for code search. In _Journal of Physics: Conference Series_ , Vol. 1544. 012056.
  * Sachdev et al. (2018) Saksham Sachdev, Hongyu Li, Sifei Luan, Seohyun Kim, Koushik Sen, and Satish Chandra. 2018\.  Retrieval on source code: a neural code search. In _Proceedings of the 2nd International Workshop on Machine Learning and Programming Languages_. ACM, Philadelphia, PA, USA, 31–41.
  * Salton et al. (1983) Gerard Salton, Edward A. Fox, and Harry Wu. 1983.  Extended Boolean Information Retrieval.  _Commun. ACM_ 26, 11 (1983), 1022–1036.
  * Salton et al. (1975) Gerard Salton, Anita Wong, and Chung-Shu Yang. 1975\.  A Vector Space Model for Automatic Indexing.  _Commun. ACM_ 18, 11 (1975), 613–620.
  * Satter and Sakib (2016) Abdus Satter and Kazi Sakib. 2016.  A search log mining based query expansion technique to improve effectiveness in code search. In _Proceedings of the 19th International Conference on Computer and Information Technology_. IEEE, Dhaka, Bangladesh, 586–591.
  * Shuai et al. (2020) Jianhang Shuai, Ling Xu, Chao Liu, Meng Yan, Xin Xia, and Yan Lei. 2020.  Improving Code Search with Co-Attentive Representation Learning. In _Proceedings of the 28th International Conference on Program Comprehension_. Association for Computing Machinery, Seoul, Republic of Korea, 196–207.
  * Singhal (2001) Amit Singhal. 2001\.  Modern information retrieval: A brief overview.  _IEEE Data Eng. Bull._ 24, 4 (2001), 35–43.
  * Sirres et al. (2018) Raphael Sirres, Tegawendé F. Bissyandé, Dongsun Kim, David Lo, Jacques Klein, Kisub Kim, and Yves Le Traon. 2018.  Augmenting and structuring user queries to support efficient free-form code search.  _Empirical Software Engineering_ 23, 5 (2018), 2622–2654.
  * Staffs et al. (2007) Keele Staffs et al. 2007\.  _Guidelines for performing systematic literature reviews in software engineering_.  Technical Report. Technical report, ver. 2.3 ebse technical report. ebse.
  * Standish (1984) Thomas A. Standish. 1984\.  An Essay on Software Reuse.  _IEEE Transactions on Software Engineering_ 10, 5 (1984), 494–497.
  * Stolee et al. (2016) Kathryn T. Stolee, Sebastian G. Elbaum, and Matthew B. Dwyer. 2016.  Code search with input/output queries: Generalizing, ranking, and assessment.  _Journal of Systems and Software_ 116, 1 (2016), 35–48.
  * Sui and Xue (2016) Yulei Sui and Jingling Xue. 2016.  SVF: interprocedural static value-flow analysis in LLVM. In _Proceedings of the 25th International Conference on Compiler Construction_. ACM, Barcelona, Spain, 265–266.
  * Sun et al. (2022) Weisong Sun, Chunrong Fang, Yuchen Chen, Guanhong Tao, Tingxu Han, and Quanjun Zhang. 2022\.  Code Search based on Context-aware Code Translation. In _Proceedings of the 44th International Conference on Software Engineering_. ACM, Pittsburgh, PA, USA, 388–400.
  * Sun et al. (2020) Zhensu Sun, Yan Liu, Chen Yang, and Yu Qian. 2020\.  PSCS: A Path-based Neural Model for Semantic Code Search.  _CoRR_ abs/2008.03042, 1 (2020), 1–7.
  * Tang et al. (2023) Xunzhu Tang, Zhenghan Chen, Saad Ezzini, Haoye Tian, Yewei Song, Jacques Klein, and Tegawendé F. Bissyandé. 2023\.  Hyperbolic Code Retrieval: A Novel Approach for Efficient Code Search Using Hyperbolic Space Embeddings.  _CoRR_ abs/2308.15234 (2023).
  * Tian et al. (2014) Yuan Tian, David Lo, and Julia Lawall. 2014.  Automated construction of a software-specific word similarity database. In _Proceedings of the 2014 Software Evolution Week-IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering_. IEEE, 44–53.
  * Treude et al. (2015) Christoph Treude, Martin P. Robillard, and Barthélémy Dagenais. 2015\.  Extracting Development Tasks to Navigate Software Documentation.  _IEEE Transactions on Software Engineering_ 41, 6 (2015), 565–581.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is All you Need. In _Proceedings of the 31st Annual Conference on Neural Information Processing Systems_. MIT Press, December 4-9, 5998–6008.
  * Wan et al. (2019) Yao Wan, Jingdong Shu, Yulei Sui, Guandong Xu, Zhou Zhao, Jian Wu, and Philip S. Yu. 2019.  Multi-modal Attention Network Learning for Semantic Source Code Retrieval. In _Proceedings of the 34th International Conference on Automated Software Engineering_. IEEE, San Diego, CA, USA, 13–25.
  * Wang et al. (2022) Chaozheng Wang, Zhenhao Nong, Cuiyun Gao, Zongjie Li, Jichuan Zeng, Zhenchang Xing, and Yang Liu. 2022.  Enriching query semantics for code search with reinforcement learning.  _Neural Networks_ 145, 1 (2022), 22–32.
  * Wang et al. (2023) Chong Wang, Xin Peng, Zhenchang Xing, Yue Zhang, Mingwei Liu, Rong Luo, and Xiujie Meng. 2023.  XCoS: Explainable Code Search based on Query Scoping and Knowledge Graph.  _ACM Transactions on Software Engineering and Methodology_ (2023), 1–27.
  * Wang et al. (2020a) Hao Wang, Jia Zhang, Yingce Xia, Jiang Bian, Chao Zhang, and Tie-Yan Liu. 2020a.  COSEA: Convolutional Code Search with Layer-wise Attention.  _CoRR_ abs/2010.09520 (2020), 1–10.
  * Wang and Jiang (2017) Shuohang Wang and Jing Jiang. 2017.  A Compare-Aggregate Model for Matching Text Sequences. In _Proceedings of the 5th International Conference on Learning Representations_. OpenReview.net, Toulon, France, 1–11.
  * Wang et al. (2020b) Wenhua Wang, Yuqun Zhang, Zhengran Zeng, and Guandong Xu. 2020b.  TranS^3: A Transformer-based Framework for Unifying Code Summarization and Code Search.  _Computing Research Repository_ abs/2003.03238, 1 (2020), 1–12.
  * Wu and Yan (2022) Chen Wu and Ming Yan. 2022\.  Learning Deep Semantic Model for Code Search using CodeSearchNet Corpus.  _CoRR_ abs/2201.11313, 1 (2022), 1–6.
  * Xu et al. (2021) Ling Xu, Huanhuan Yang, Chao Liu, Jianhang Shuai, Meng Yan, Yan Lei, and Zhou Xu. 2021.  Two-Stage Attention-Based Model for Code Search with Textual and Structural Features. In _Proceedings of the 28th International Conference on Software Analysis, Evolution and Reengineering_. IEEE, Honolulu, HI, USA, 342–353.
  * Yang and Huang (2017) Yangrui Yang and Qing Huang. 2017.  IECS: Intent-enforced code search via extended Boolean model.  _Journal of Intelligent & Fuzzy Systems_ 33, 4 (2017), 2565–2576.
  * Yao et al. (2019) Ziyu Yao, Jayavardhan Reddy Peddamail, and Huan Sun. 2019.  CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning. In _Proceedings of the 28th The World Wide Web Conference_. ACM, San Francisco, CA, USA, 2203–2214.
  * Yu et al. (2016) Haibo Yu, Wenhao Song, and Tsunenori Mine. 2016.  APIBook: an effective approach for finding APIs. In _Proceedings of the 8th Asia-Pacific Symposium on Internetware_. ACM, Beijing, China, 45–53.
  * Yu et al. (2022) Hao Yu, Yin Zhang, Yuli Zhao, and Bin Zhang. 2022\.  Incorporating Code Structure and Quality in Deep Code Search.  _Applied Sciences_ 12, 4 (2022), 2051.
  * Zeng et al. (2023) Chen Zeng, Yue Yu, Shanshan Li, Xin Xia, Zhiming Wang, Mingyang Geng, Linxiao Bai, Wei Dong, and Xiangke Liao. 2023.  deGraphCS: Embedding Variable-based Flow Graph for Neural Code Search.  _ACM Transactions on Software Engineering and Methodology_ 32, 2 (2023), 34:1–34:27.
  * Zhang et al. (2018) Feng Zhang, Haoran Niu, Iman Keivanloo, and Ying Zou. 2018\.  Expanding Queries for Code Search Using Semantically Related API Class-names.  _IEEE Transactions on Software Engineering_ 44, 11 (2018), 1070–1082.
  * Zhao and Liu (2022) Wei Zhao and Yan Liu. 2022\.  Utilizing Edge Attention in Graph-Based Code Search. In _Proceedings of the 34th International Conference on Software Engineering and Knowledge Engineering_. KSI Research Inc., Virtual Conference Center, USA, 60–66.
  * Zhu et al. (2020) Qihao Zhu, Zeyu Sun, Xiran Liang, Yingfei Xiong, and Lu Zhang. 2020.  OCoR: An Overlapping-Aware Code Retriever. In _Proceedings of the 35th International Conference on Automated Software Engineering_. Institute of Electrical and Electronics Engineers, Melbourne, Australia, 883–894.