# Code Structure-Aware through Line-level Semantic Learning for Code
Vulnerability Detection

Ziliang Wang1, Ge Li1, Jia Li1, Yihong Dong1, Yingfei Xiong1, Zhi Jin1 {lijia,
<EMAIL_ADDRESS>Corresponding Author: Ge Li<EMAIL_ADDRESS>1Key Lab
of High Confidence Software Technology, MoE, School of Computer Science,
Peking University, Beijing, China
Email: {wangziliang, lige,xiongyf<EMAIL_ADDRESS>
###### Abstract

Different from the flow semantics of natural languages, programming languages
are inherently rigid in structure and grammar. Existing fine-tuning
methodologies for code vulnerability detection generally treat code as long
text sequences, stripping away structural elements such as newlines (’/n’) and
whitespace. However, this approach inadvertently results in the loss of
crucial structural information, diminishing the distinct characteristics of
code and impairing the accuracy of vulnerability detection. To address these
challenges, we propose a novel network architecture method based on pre-
trained code models, which incorporates structural information awareness. We
propose an enhanced code text processing workflow that retains structural
elements prior to modeling. This refinement allows the model to retain and
exploit line-level structural information and semantic information during the
modeling process. Furthermore, we introduce a new network architecture, the
Code Structure-Aware Network through Line-level Semantic Learning (CSLS),
which integrates three key components: global vulnerability awareness, line-
structural awareness, and sensitive-line awareness. We have conducted
comprehensive experiments using vulnerability detection datasets from real-
world projects. Extensive experiments were conducted on vulnerability
detection datasets derived from real-world projects. The results demonstrate
that our new code pre-processing flow significantly improves existing
baselines (e.g., a 3% accuracy improvement on the Devign dataset when applied
to popular models such as CoderBert and UniXcoder). The proposed network
architecture also demonstrates superior accuracy in detecting vulnerabilities,
surpassing newly established benchmarks. These findings underscore the
importance of structural information in enhancing the efficacy of code
vulnerability detection models.

###### Index Terms:

component, formatting, style, styling, insert

##  I Introduction

Software vulnerabilities refer to weaknesses in code that can be exploited,
leading to severe consequences such as unauthorized information disclosure and
cyber extortion [1, 2]. The growing magnitude of this issue is highlighted by
recent statistics: in the first quarter of 2022, the US National Vulnerability
Database (NVD) reported 8,051 vulnerabilities, a 25% increase from the
previous year [3]. Additionally, a study found that 81% of 2,409 analyzed
codebases contained at least one known open-source vulnerability. The
widespread nature and increasing number of these vulnerabilities underscore
the urgent need for robust automated vulnerability detection mechanisms.
Implementing such systems is crucial for enhancing software security and
preventing a range of potential threats [1, 2, 4, 5].

Existing literature on vulnerability detection models can be broadly
categorized into three main types: (1) traditional detection models [6, 7],
(2) deep learning (DL)-based models [8, 9, 10, 11] and (3) pre-trained models
based models [12].

Traditional detection models often require experts to manually develop
detection rules [13, 14]. This approach is labor-intensive and tends to
struggle with maintaining low rates of false positives and false negatives
[10, 15]. In contrast, deep learning (DL)-based detection methods learn
vulnerability patterns from training datasets [10, 11, 16], eliminating the
need for manual heuristics and enabling autonomous feature identification.
They avoid manual heuristic methods and autonomously learn and identify
vulnerability features. To further enhance vulnerability semantics
comprehension, recent advancements have introduced pre-trained code models
[12] and large language models (LLMs). These tools have surpassed the
performance of traditional static analysis methods [17, 18, 10].

Vulnerability detection tasks are notably more domain-specific compared to
other code tasks like code generation, as they require unique vulnerability
criteria and logic for each project. Additionally, acquiring large volumes of
high-quality vulnerability detection data is challenging, as it necessitates
the expertise of senior programmers familiar with the project. Consequently,
language models with extensive parameters, such as Llama2 and CodeLlama,
struggle to obtain sufficient data for fine-tuning. The pre-trained models
with smaller parameters (CodeBERT, CodeT5) show a wider range of applications
[12].

However, existing methods based on pre-trained code models usually treat the
code task as a natural language task for preprocessing and fine-tuning, even
if the code text is significantly different from the natural language text.
Unlike code generation, code completion and other code tasks, The
vulnerability detection task requires a more complete understanding of the
differences between codes. Current methods based on pre-trained models
generally ignore this point, and treat the code text as natural language text
processing, resulting in the loss of structure, damage to the differences
between codes, and reduce the performance of vulnerability detection tasks[19,
20]111https://github.com/microsoft/CodeXGLUE/blob/main/Code-Code/Defect-
detection/[21]222https://github.com/daiquocnguyen/GNN-ReGVD/[22]. The
structure of the code is typically preserved during pretraining data for the
code model [23, 20]. Model pre-training is not usually a classification task.
But pre-trained models often remain sensitive to the input structure. In
natural language tasks, the latest research also found that the period based
identifier information can increase the understanding ability of the model
[24].

Figure 1: Code processing flow of existing open source model vulnerability
detection methods.

_Status analysis._ Figure 1 illustrates the code preprocessing process of
existing methods before performing vulnerability task fine-tuning. earlier
studies on vulnerability detection using fine-tuned code models, it was common
to preprocess code data by removing all newline characters and spaces before
fine-tuning for downstream tasks. For example, a piece of code with an initial
token length of 150 is reduced to 130 tokens after preprocessing. CodeBERT,
which has an input length limit of 400 tokens, benefits from such
preprocessing as it reduces the input length. However, this preprocessing
approach fails to account for the importance of structural information in the
code, especially as the context length of the code model increases (e.g.,
UniXcoder supports 1024 tokens). For vulnerability detection tasks, the
removal of structural elements such as newlines and spaces can significantly
obscure the differences between samples. Structural elements such as newlines,
whitespace, and indentation in the code provide important contextual and
semantic information. This loss of structural information is detrimental to
classification tasks, as it undermines the ability to accurately distinguish
between vulnerable and non-vulnerable code. Therefore, it is crucial to
maintain the structural integrity of the code during preprocessing to preserve
essential contextual information that aids in vulnerability detection.

_Our approach._ To address the limitations of existing fine-tuning methods
based on pre-trained code models, we propose CSLS, a structure-aware
vulnerability detection method designed to capture the structural information
of the code. We enhance the code preprocessing process and introduce a new
model architecture that preserves and utilizes the structural information at
the line level, helping the model to learn both global semantics and
structural semantics of the code.

_Code Preprocessing Process._ To verify the impact of the aforementioned
issues on vulnerability detection, this paper proposes a structural
information enhancement strategy to strengthen the perception of code
structure in vulnerability detection tasks. (1) _Global Processes._ We first
replace the previous code model fine-tuning preprocessing with a simpler
method to obtain global word segmentation, as shown in fig. 1 CSLS preserves
line breaks and all whitespace before entering the tokenizer. (2) _Line
Processes._ Benefiting from the preservation of code structure information,
CSLS splits the code snippets into lines and obtains a row-level Token array
for each code snippets. This approach preserves the original structure of the
code as much as possible, while allowing the model to model the code text from
both global and line-level supervision.

_CSLS Model._ In order to further enhance the perception of code structure,
this paper proposed a new vulnerability detection network architecture, which
used two code models to learn code semantics and structure by accepting code
texts with different structures respectively. (1) _Global Semantic-Aware
Model._ This model takes global word segmentation as input and perceives
global information, including structural information. (2) _Line Semantic-Aware
Model._ This model takes line-level code as input and perceives structural and
semantic relationships between lines of code. Finally, the CSLS model analyzes
the risk of code vulnerabilities from three perspectives of global semantics,
line structure and line semantics, and realizes high-precision vulnerability
detection.

_Results._ Fine-tuning and evaluation were performed after preprocessing using
our proposed method. We report state-of-the-art results on the most common
code models available today: CodeBERT’s accuracy increased from 63% [19] to
65% on the Devign dataset [25], and UniXcoder-base’s accuracy rose from 64.8%
[12] to 68.8%. Other datasets, such as ReVeal [26], also show positive
effects. Our proposed CSLS further improves the accuracy of vulnerability
detection. We achieved state-of-the-art results with CSLS, reporting 70.57%
accuracy on Devign, 91.86% accuracy, and 49.59% F1-score on ReVeal. In
summary, the main contributions of this paper are as follows:

  * a)

We propose an enhanced code preprocessing process for vulnerability detection
tasks, highlighting the defects of existing preprocessing methods and
revealing the importance of preserving code structure information.

  * b)

We introduce a vulnerability detection network architecture that enhances code
structure awareness. It achieves line-level semantic learning by pre-training
the code model and combining it with global information to achieve high-
precision vulnerability detection.

  * c)

We evaluate the proposed preprocessing procedure on two real datasets. The
experimental results demonstrate that preserving code structure information
can effectively improve the performance of existing code models on
vulnerability detection tasks without increasing any cost.

  * d)

We conduct extensive experiments on two real-world datasets. The results show
that the proposed CSLS architecture can further improve the accuracy of code
vulnerability detection by perceiving code line-level structure information
333Our anonymous replication package (data and code)
:https://figshare.com/articles/dataset/CSLS/24831567.

##  II Related work

###  II-A Traditional Vulnerability Detection

Over the years, numerous methods for vulnerability detection have been
developed. Initially, research in this area predominantly focused on
identifying vulnerabilities through manually customized rules [13, 14]. Static
analysis tools rely on manual rules and precise specification of code
behavior, which are difficult to obtain automatically. While these heuristic
approaches offered solutions for vulnerability detection, they necessitated
extensive manual analysis and the formulation of defect patterns. Furthermore,
the repetition of syntactic elements across different code fragments, as
dictated by certain rules, has been shown to contribute to higher rates of
both false positives and false negatives [6, 7, 15].

###  II-B Deep Neural Network for Vulnerability Detection

To perceive code text nonlinear characteristics, recent research has turned to
the model based on neural network, hole features extracted from code snippets
[11, 9]. Existing deep learning-based vulnerability detection models are
predominantly divided into two categories: token-based and graph-based models.

Token-based models treat code as a linear sequence and use neural networks
(e.g., LSTM or Transformer) to learn vulnerability features from known cases,
aiming to identify vulnerability features [9, 10, 27]. For instance, Russell
et al.[9] leveraged recurrent neural networks (RNNs) and convolutional neural
networks (CNNs) to learn feature sets from code token sequences tailored for
vulnerability identification. Concurrently, Li et al.[10] employed BiLSTM [28]
to encode a segmented version of input code, known as ’code gadgets,’ centered
on key markers, especially library/API function calls. However, these token-
based models often overlook the complexity of the source code structure,
potentially leading to inaccurate detection.

In parallel, another research direction explores the potential of graph-based
methods for vulnerability detection [29, 26, 30, 12]. For example, DeepWukong
[27] uses GNNs for feature learning, focusing on compressing code fragments
into a dense, low-dimensional vector space to enhance the detection of various
vulnerability types. DeepTective [31] addresses vulnerabilities common to PHP
scripts, such as SQL injection, Cross-Site Scripting (XSS), and command
injection, by deploying a combination of Gated Recurrent Units (GRUs) and
Graph Convolutional Networks. Graph-based detection models learn code
structure through various graph representations, utilizing neural networks for
vulnerability detection [32, 17]. For instance, Zhou et al.[25] used a gated
graph recurrent network[33] to extract structural details from triadic graph
representations—AST, CFG, and DFG. Chakraborty et al.[26] introduced REVEAL,
an innovative approach that combines a gated graph neural network, re-sampling
techniques[34], and triplet loss [35]. Wu et al.[32] proposed an approach that
efficiently converts the source code of a function into an image while
preserving the program details.

###  II-C Pre-Trained Models for Vulnerability Detection

Inspired by the success of pre-trained models in natural language processing
(NLP), recent research has increasingly focused on leveraging these models to
enhance code vulnerability detection accuracy [19, 36, 37, 38, 39, 20].

The core concept behind these works is to use a model pre-trained on a large
corpus of source code data, followed by specialized fine-tuning for specific
tasks [36]. For instance, Feng et al.[19] proposed CodeBERT, designed
specifically for understanding and generating source code, combining the
processing capabilities of both natural and programming languages. Similarly,
CuBERT employs masked language modeling with sentence prediction for code
representation[36]. Additionally, some pre-trained models incorporate
structural information of code fragments during the initial training phase
[37, 38]. For example, Guo et al.’s GraphCodeBERT [40] leverages graph
structures to infer data flow in code fragments. DOBF [41] introduces a novel
pre-training objective aimed at enhancing the model’s ability to learn the
syntactic and structural complexity of source code, specifically addressing
the structural dimension of programming languages. The objective is
specifically tailored to address the structural dimension of programming
languages. In comparative evaluations, CodeBERT is positioned as a baseline
standard for various code-related tasks, including code clone detection and
code translation. UniXcoder, a unified cross-modal pre-trained programming
language model, also serves as a baseline method, trained on extensive code
and natural language data [20].

Given the superior performance of pre-trained models in various code-related
tasks, some studies have applied them to vulnerability detection [1, 2, 42].
Hanif et al. proposed VulBERTa, which pretrains a RoBERTa model with a custom
tokenization pipeline for real-world C/C++ projects [43]. Nguyen et al.
introduced ReGVD, combining graph structure and pre-trained models to address
source code vulnerability detection [12]. Zhang et al. decomposed the code
segment Control Flow Graph (CFG) into multiple execution paths and used the
pre-trained model CodeBERT for vulnerability detection [44]. Thapa et al. [2]
explored the performance of fine-tuned language models for multi-class
classification of similar types of vulnerabilities.

However, fine-tuning large language models and code models requires high-
quality vulnerability data due to the domain-specific nature of vulnerability
detection and the scarcity of data. Consequently, many vulnerability detection
methods based on smaller code models have been extensively studied. For
example, Wang et al. proposed a detection method where multiple code models
and large models cooperate to enhance the semantics of vulnerabilities [45].
However, these methods generally inherit the original way of code
preprocessing and face the challenge of structural information loss.

Figure 2: The CSLS framework implements vulnerability detection by sensing
three different semantics :1. Global vulnerability semantics (step 5) 2. Line-
level structural vulnerability semantics (step 3) and 3. Line vulnerability
semantics (step 4).

##  III Approach

In this section, we introduce the network structure of vulnerability detection
based on code structure awareness (CSTA). Figure 2 shows an overview of SFTF,
consisting of three main phases: (1) Data pre-processing, (2) code line-level
semantic awareness, and 3) code global semantic awareness

###  III-A Data Pre-processing

We denote the historical vulnerability dataset as $L=(P,Y)$, where $p_{i}$
represents a code snippet in a programming language, and $y_{i}$ represents
the corresponding vulnerability label for the $i$-th snippet, with $0<i\leq
M$. Here, $M$ is the total number of code snippets. The labels $y_{i}$ can
take values of either 0 or 1, where $y_{i}=0$ indicates that the code is free
of vulnerabilities, and $y_{i}=1$ indicates the presence of a vulnerability in
the code.

Global Pre-processing. Unlike existing methods that might employ extensive
pre-processing steps, we only truncate the code fragments to fit within the
input size constraints of the model used. The formalization of this step is as
follows:

$C_{i}=C_{i_{0}}C_{i_{1}}\ldots C_{i_{n}},\quad 0<i\leq M$ (1)

Here, $C_{i_{n}}$ represents the $n$-th token of the $i$-th code fragment,
with $n$ being the maximum input length allowed by the code model.

Code Line Pre-processing. The number of lines in a code fragment typically
exceeds the number of snippets. We define $L_{i_{n}}$ as the $n$-th line of
the $i$-th code fragment. To expedite the learning process for the code model,
we implement the following pre-processing steps for code batches:

Line-Token Alignment: Each line in the code snippet is tokenized individually.
Tokens are then padded to ensure uniform length. For snippets within the same
batch, we align the number of tokens per line. Based on statistical analysis,
after removing blank lines, the Devign dataset has an average of 12.18 tokens
per line. Thus, CSLS presets the maximum number of tokens per line to $p=20$.

$L_{i_{n}}=T_{i_{n0}}T_{i_{n1}}\ldots T_{i_{np}},\quad 0<i\leq M$ (2)

Here, $T_{i_{np}}$ represents the $p$-th token of the $n$-th line of the
$i$-th code fragment, with $p$ being the maximum number of tokens per line.

Line Alignment: For all snippets in the same batch, we align the number of
lines in each snippet. Statistical analysis shows that the Devign dataset has
an average of 50.05 lines per snippet after deleting blank lines, and 113.37
lines when blank lines are retained. Therefore, CSLS presets $MAX(k)=100$
lines per code fragment split. For all code fragments $n$ in the same batch,
we use $k=\max(\text{len}(n.\text{line}),100)$.

$L_{i}=L_{i_{0}}L_{i_{1}}\ldots L_{i_{k}},\quad 0<i\leq M$ (3)

The pre-processed sentences and global input IDs are batched and padded to
align with the maximum sentence length and number of lines in the batch. This
approach ensures that all snippets in a batch are standardized in terms of
line count and token count per line, facilitating efficient batch processing
and learning.

###  III-B Line-level semantic awareness

Line Semantic Learning. After the code pre-processing process, we obtain an
array of tokens based on line segmentation for all code fragments. And, in the
same batch, this number has a consistent shape $[b,k,p]$. Where $b$ denotes
the number of snippets in the batch, $k$is the number of lines in each
snippet, and $p$ is the maximum number of tokens per line.

Next, we flatten the pre-processed token array to form a two-dimensional array
$z$ with size $[b*k,p]$. This allows us to process an entire batch of data in
a single operation without iterating over each row or fragment. CSLS uses a
line semantic-aware model, this paper uses the UniXcoder-nine model by
default, and we process $z$ to quickly obtain deep semantic representation
through a single batch. The formalization process is as follows:

$CE=f_{l}(z,z_{mask}),$ (4)

where $CE$ has the shape $[b\times k,h]$, with $h=768$. Here, $b$ represents
the batch size, $l$ represents the sequence length, and $h$ represents the
dimension of the hidden layer. The $f_{l}$ stands for line semantic-aware
model. $z_{mask}$ is provided by the completion process during data pre-
processing. The output $CE$ of the line semantic-aware model contains multiple
hidden states, and we use the hidden state of the first token of the last
layer (usually the [CLS] token) to represent the semantics of each lines.

Finally, to restore the processed data to its original batch structure, we re-
shape $CE$ to $[b,k,h]$, where each element now contains an embedded
representation of the corresponding line of code. Such refactoring facilitates
subsequent steps such as further analysis or specific row-level tasks.

Line Structure Aware. After the above process, CSLS obtains the semantic array
CE for each line of the code fragment. In this module, CSLS uses the
Transformer layer to structurally model the code line semantics.

Figure 3: A transformer model is used to sense the structural relationships
between lines of code.

It is worth noting that the input line semantics CE maintains the line number
order, and the default position encoding of the Transformer layer is
consistent with the line number. This allows the Transformer to effectively
sense the structure of the code line. Finally, the output of the Transformer
layer is the vulnerability judgment of the row structure. The formalization is
as follows:

$S_{repr}=f_{t}(CE_{1}..CE_{k})$ (5)

Where $f_{t}$ is a standard transformer model with 8 layers and 8 attention
heads. In this process, we introduce a crucial step where we detach the
sentence embeddings before passing them through the Transformer encoder. This
is done to prevent the gradients from flowing back into the global encoder
during the optimization process of the Transformer encoder. The purpose of
this detachment is to enforce that the Transformer encoder focuses solely on
learning the structural relationships among code lines, rather than on
semantic information already captured by the global encoder. By doing so, we
ensure that the global encoder and Transformer encoder are specialized in
their respective tasks: the former in capturing semantic information and the
latter in understanding structural dependencies. This separation of concerns
improves the model’s ability to accurately assess the structural risks and
potential vulnerabilities within the code based on the line-wise semantic
representations provided.

Line Sensitive Aware. In the line-sensitive perception module, we obtain the
overall semantic representation of each line of code by calculating the
average of the 768-dimensional vectors in the semantic representation of each
line of code. For the code snippets in each batch, we identify the most
representative lines of code by computing the minimum of the semantic
representation of each line of code. The specific process is as follows:

For the semantic representation matrix CE for each sentence, we first compute
the average of each token over 768 dimensions:

$CE_{mean}=\frac{1}{h}\sum_{j=1}^{h}CE_{i,j}$ (6)

Then, we find the row index with the smallest mean for each batch:

$L_{min}=\arg\min(CE_{mean})$ (7)

Finally, we select the semantic representation of the row with the smallest
mean value from each batch to form the final row-sensitive semantic
representation:

$L_{repr}=CE[L_{min}]$ (8)

###  III-C Global semantic awareness

Global Semantic Learning. In the global semantic learning module, we use a
code model to capture the global semantic information of code fragments. We
pass the line-level semantic representation of each code snippet to the code
model and output the global semantic representation.

First, we feed the row-level semantic representation matrix CE into the
UniXcoder-nine model:

$G_{repr}=f_{g}(C_{i})$ (9)

Vulnerability Prediction. In the vulnerability prediction module, we
concatenate the global semantic representation $G_{repr}$, the line-sensitive
semantic representation $L_{repr}$, and the global CLS representation
$global\\_cls$ to form the final representation vector $H$:

$H=[S_{repr},L_{repr},G_{repr}]$ (10)

Then, we input the final representation vector into a Multi-Layer Perceptron
(MLP) classifier for vulnerability prediction:

$P=\sigma(f_{m}(H))$ (11)

where $\sigma$ is the Sigmoid activation function, and $P$ is the predicted
vulnerability probability. The $fm$is a classification head.

###  III-D Loss Function

The loss function adopted for the code models training is the cross-entropy
loss [25], commonly used in classification problems for its effectiveness in
penalizing the predicted labels and the actual labels:

$H(y,\hat{y})=-y\log(\hat{y})-(1-y)\log(1-\hat{y})$ (12)

##  IV Study Design

TABLE I: Comparison results for different models on Devign and Reveal datasets.  Dataset | Devign [25] | Reveal [26]
---|---|---
Models |  Acc |  Recall |  Prec |  F1 |  Acc |  Recall |  Prec |  F1
ChatGPT 3.5 COT |  49.83 |  32.24 |  33.00 |  30.61 |  63.72 |  26.34 |  30.54 |  27.70
ChatGPT 4o COT |  53.73 |  7.46 |  45.94 |  4.06 |  20.97 |  22.17 |  97.33 |  12.51
Devign |  56.89 |  52.50 |  64.67 |  57.59 |  87.49 |  31.55 |  36.65 |  33.91
ReGVD |  61.89 |  48.20 |  60.74 |  53.75 |  90.63 |  14.47 |  64.70 |  23.65
CodeBERT |  63.59 |  41.99 |  66.37 |  51.43 |  90.41 |  25.87 |  54.62 |  35.11
UniXcoder-base |  65.77 |  51.55 |  66.42 |  58.05 |  90.50 |  39.91 |  53.52 |  45.72
CodeT5+-base |  65.62 |  55.29 |  64.73 |  59.64 |  90.94 |  31.14 |  59.16 |  40.80
TRACED |  64.42 |  61.27 |  60.03 |  61.05 |  91.11 |  21.49 |  68.05 |  32.66
UniXcoder-nine |  66.98 |  56.33 |  66.63 |  61.05 |  90.72 |  33.77 |  56.20 |  42.19
CSLS |  70.57 |  59.36 |  71.70 |  64.95 |  91.86 |  39.91 |  65.46 |  49.59

###  IV-A Datesets

To evaluate the effectiveness of M2CVD, we employ two datasets from real
projects :(1) Devign [25], and (2) Reveal [26]. The Devign dataset, derived
from a graph-based code vulnerability detection study [25], stands as a
dataset of function-level C/C++ source code from the well-established open-
source projects QEMU and FFmpeg. Aligning with the methodology articulated by
Li et al. [25], the partitioning of the Devign dataset adheres to a
conventional 80:10:10 ratio, demarcating the bounds for training, validation,
and testing data, respectively. The dataset completes the labeling of
vulnerable code by a group of security researchers performing a rigorous two-
stage review. In the task of software vulnerability detection, the REVEAL
dataset is a representative dataset, as presented in [26]. It is a further
exploration of data redundancy and unrepresentative class distributions in
existing datasets. As a detection code dataset, REVEAL encompasses source code
extracted from two open-source forays: the Linux Debian kernel and Chromium.
Similar to the real-world situation, this dataset has an imbalanced label
distribution, with the number of normal code fragments much larger than the
number of vulnerable ones (10:1). Similarly, in the Reveal dataset, a split
ratio of 80:10:10 was set.

During the experiment, the proportion of positive and negative samples in the
training set, validation set and test set is consistent with the original
dataset.

###  IV-B Performance Metrics

In the process of evaluating the performance of the model, the proposed method
includes four metrics widely recognized in the field of software testing and
analysis[25]:

Precision: Denoted as the quotient of the sum of true positives and false
positives and is a measure of the accuracy of instances that are identified as
positive. Formally, it is defined as:

$Precision=\frac{TP}{TP+FP}$ (13)

where TP and FP represent the number of true positives and false positives,
respectively.

Recall: Recall evaluates the fraction of actual positives that are correctly
identified and is calculated as the fraction of true positives over the sum of
true positives and false negatives:

$Recall=\frac{TP}{TP+FN}$ (14)

where FN signifies the number of false negatives.

F1 Score: The F1 score provides an indicator of the accuracy of the test by
combining precision and recall into a single metric by taking their harmonic
mean:

$F1\ Score=2\times\frac{Precision\times Recall}{Precision+Recall}$ (15)

Accuracy: This metric reflects the proportion of true positives and true
negatives amongst all evaluated instances, thus offering an overall measure of
the model’s performance:

$Accuracy=\frac{TP+TN}{TP+TN+FN+FP}$ (16)

where TN representing the number of true negatives.

###  IV-C Baseline Methods

In our evaluation, we compare M2CVD with seven state-of-the-art methods.

(1) ChatGPT [46]: The ChatGPT models showcases the capabilities of DL in code
generation and processing, albeit not specifically tailored for the domain of
software vulnerability detection. We used COt-based cue words to test the test
set in full, and repeated three times to take the average.

(2) Devign [25]: Devign is a graph-based model that uses Gated Graph Recurrent
network (GGN) to represent the graph combining AST, CFG, DFG and code sequence
of the input code fragment for vulnerability detection.

(3) ReGVD [12]: ReGVD treats the problem as text classification by
transforming the source code into a graph structure, using token embedding
from GraphCodeBERT [40], and applying a mixture of graph-level sum and max-
pooling techniques.

(4) SySeVR [15]: SySeVR uses code statements, program dependencies, and
program slicing as capabilities, and utilizes bidirectional recurrent neural
networks to detect vulnerable code fragments.

(5) CodeBERT [19]: CodeBERT use a pre-trained structure that amalgamates
natural language and programming language, facilitating a broad spectrum of
coding tasks, including but not limited to code understanding and generation.

(6) CodeT5+ [47]: CodeT5+, an encoder-decoder llm family for code, where
component modules can be flexibly combined to accommodate a wide range of
downstream code tasks.

(7) UniXcoder [20]: UniXcoder extends the capabilities of models like CodeBERT
by incorporating a comprehensive understanding of code syntax and semantics,
thus improving model performance on coding tasks such as code summarization,
translation, and completion. UniXcoder-nine is its latest extension in 2023,
further training on UniXcoder-base to obtain powerful code models.

Experiment Environment: We implemented CSLS in Python using Pytorch 2.10. The
experiments were performed on a machine containing a NVIDIA GeForce GTX A6000
GPU and two Intel Xeon Gold 6226R 2.90 GHz CPU.

(a) Number of vulnerabilities discovered in dataset Devign

(b) Number of missed reports in dataset Devign

(c) Number of vulnerabilities discovered in dataset Reveal

(d) Number of missed reports in dataset Reveal

Figure 4: Comparison of vulnerability detection performance of different
models on two data sets.

##  V EXPERIMENTS

In this section, we conduct extensive experiments to justify our model’s
superiority and analyze the reasons for its effectiveness. Specifically, we
aim to answer the following research questions:

RQ1: How effective is CSLS compared with the state-of-the art baselines on
vulnerability detection?

RQ2: What are the effects of the code pre-processing process on vulnerability
detection methods based on fine-tuning of pre-trained models?

RQ3: What are the effects of different pre-trained code models for CSLS?

All experimental procedures are saved with the random seed used in the
existing literature: seeds=123456 [12]. All experiments are repeated 3 times
to ensure repeatability.

###  V-A Implementation

CSLS contains a pre-trained model fine-tuning process and is therefore
affected by the learning rate and Batch size. We provide our full setup and
code along with the data splits used in a public repository. Due to page
limitations, we do not discuss the impact of different Settings in this
category. In general, a Batch Size of 12 and a learning rate of 2e-5 is a
recommended choice.

Scope: Currently, CSLS reports if a function is vulnerable. Among other works,
deep learning explanation tools are commonly used for other applications
reporting row-level vulnerabilities, such as the work of Li et al. Benefiting
from the modeling of row-level semantics by our approach, we can directly
output the index of row-vulnerability semantics as dangerous rows. But
detailed verification requires a lot of expert experience or a canonical
verification method, and we leave this evaluation for future work.

###  V-B RQ1. Effectiveness of CSLS

To answer the first question, we compare M2CVD with the seven baseline methods
on the two datasets as shown in table 1. We can draw conclusions about the
performance of CSLS compared to the baselines across the evaluated datasets.

TABLE II: Comparison results for different models on Devign dataset. The best result for each metric is highlighted in bold. Data |  our |  Exist |  Ratio |  $>$512 |  $>$1024
---|---|---|---|---|---
Devign |  723.70 |  559.74 |  77% |  31.37% |  14.64%
Reveal |  488.40 |  457.34 |  94% |  20.48% |  8.95%
TABLE III: Comparison results for different models on Devign dataset. (S) stands for preserving structural information Dataset | Devign [25]
---|---
Models |  L |  Acc |  Recall |  Prec |  F1
CodeBERT |  400 |  63.59 |  41.99 |  66.37 |  51.43
CodeBERT(S) |  400 |  65.95 |  44.46 |  70.54 |  54.54
UniXcoder-base |  1024 |  65.77 |  51.55 |  66.42 |  58.05
UniXcoder-base(S) |  1024 |  68.85 |  58.64 |  68.91 |  63.36
CodeT5+-base |  512 |  65.62 |  55.29 |  64.73 |  59.64
CodeT5+-base(S) |  512 |  66.43 |  44.38 |  71.77 |  59.64
TRACED |  512 |  64.42 |  61.27 |  60.03 |  61.05
TRACED(S) |  512 |  67.45 |  54.26 |  68.37 |  60.50
UniXcoder-nine |  1024 |  66.98 |  56.33 |  66.63 |  61.05
UniXcoder-nine(S) |  1024 |  69.10 |  56.73 |  70.28 |  62.78
CSLS |  1024 |  70.57 |  59.36 |  71.70 |  64.95

Table 1 presents the performance of ChatGPT on the vulnerability detection
task. It is evident that large-scale language models employ aggressive
detection logic. Specifically, in ChatGPT 4o, nearly all code snippets were
identified as vulnerable, leading to considerably low F1 scores across both
datasets.

CSLS demonstrates a marked superiority on both datasets. In the Devign
dataset, M2CVD attains the highest Accuracy of 70.57%, the highest F1 score of
64.95% and the highest Precision outperforming all other models. As shown in
Fig.4(a), CSLS detected 110 different vulnerabilities, while TRACED detected
68 and UniXcoder-nine detected 25. This indicates that CSLS has a higher
detection capability, identifying more vulnerabilities than the other models.
In terms of missed reports, CSLS had the fewest false negatives (18), whereas
TRACED and UniXcoder nine missed 34 and 143 vulnerabilities, respectively.
This demonstrates CSLS’s superior accuracy in minimizing missed detection as
shwon in fig.4(b). This indicates that CSLS has the most balanced performance
in correctly identifying vulnerabilities without being skewed towards over-
predicting (which would increase recall but decrease precision) or under-
predicting (which would do the opposite).

On the Reveal dataset, since the proportion of negative samples in this
dataset is 90, the model with strong fitting performance generally exceeds 90%
on ACC. In this dataset, people are generally interested in the ability of the
model to find positive samples (vulnerability). For CSLS, Both Recall and F1
metrics maintain the level of optimal level. As show in Fig. 4(c), CSLS
detected 22 vulnerabilities, outperforming TRACED (7) and UniXcoder-nine (2).
This again shows CSLS’s higher effectiveness in detecting vulnerabilities. In
fig. 4(b), CSLS had 10 missed reports, which is significantly lower than
TRACED (27) and UniXcoder nine (5), indicating a balanced performance in both
detecting vulnerabilities and minimizing false negatives. These figures not
only show that CSLS maintains its high performance in different testing
conditions but also that it consistently understands and predicts code
vulnerabilities with high precision and recall.

Result Answer to RQ1: Devign performed much better than all baselines. In
particular, it shows great improvement on Devign data with more complex code
structure.

TABLE IV: Comparison results for different models on Devign dataset. (S) stands for preserving structural information. Dataset | Reveal [26]
---|---
Models |  Acc |  Recall |  Prec |  F1
CodeBERT |  90.10 |  28.50 |  51.18 |  36.61
CodeBERT(S) |  89.18 |  29.80 |  44.15 |  35.60
UniXcoder-base |  90.50 |  39.91 |  53.52 |  45.72
UniXcoder-base(S) |  90.50 |  42.98 |  53.26 |  47.57
CodeT5+-base |  90.94 |  31.14 |  59.16 |  40.80
CodeT5+-base(S) |  91.20 |  29.38 |  63.20 |  40.11
TRACED |  90.12 |  24.14 |  67.07 |  35.48
TRACED(S) |  91.95 |  24.12 |  84.61 |  37.54
UniXcoder-nine |  90.14 |  31.57 |  51.42 |  39.13
UniXcoder-nine(S) |  90.14 |  32.59 |  51.36 |  40.10
CSLS |  91.86 |  39.91 |  65.46 |  49.59

###  V-C RQ2: Effect of different code preprocessing procedures on detection
performance

To illustrate the impact of the code preprocessing process on the
vulnerability detection task, we conduct experiments on several models. Since
we propose to keep line characters and Spaces in code snippets, this increases
the number of tokens per sample. This can lead to bad results in cases where
the input length of the pre-trained model is limited. Therefore, we analyze
the token length after code-word segmentation.

In Table II, it can be seen that the two data sets after passing the word
segmentator PBE in different pre-processing ways. The data shows that the
Devign dataset is severely affected, with the number of tokens reduced by 23%
after processing by the currently commonly used pre-processing method (Exist).
The code snippet in Reveal is much less affected. There is possible reasons
for this. First, the complexity of the code function of the projects in Reveal
is lower than that of Devign, and the average number of tokens is only half of
that of Devign. At the same time, the proportion of code functions with more
than 1024 tokens in Dvign is more than 14.64, which is much higher than that
of Reveal data.

Table 3 shows the performance of vulnerability detection of different models
under different word segmentation on the dataset Devign. It can be seen that
all the models have a large degree of performance improvement compared to the
previously reported metrics. For example, CodeBERT, whose best known
performance reported in the literature was only 63%@ACC, went up to 65.95%@ACC
using the new code pre-processing pipeline without making any model
modifications. At the same time, it is clear that models with input sequence
lengths of 1024 get a bigger boost because they can see more structural
information while preserving the semantics of the code. Models with lower
input lengths often have to trim their code. The vulnerability detection
accuracy of both UniXcoder-base and UniXcoder-nine models is surprisingly
improved. CodeT5+ and TRACED models have a huge improvement in the accuracy of
detecting non-vulnerability datasets, but both of them have a decrease in
Recall metrics. We speculate that this may be due to the limited input length
of the model.

Table 4 shows the situation on Reveal, where there is a significant difference
between Reveal and the dataset Devign. Firstly, the two processing methods
have less impact on this dataset than Devign. Secondly, there are far more
non-vulnerability data than vulnerability data in this data, which increases
the difficulty of vulnerability detection. The performance of CodeBERT
decreases after our way of pre-processing. This is due to the limited input,
the model learns fewer tokens after introducing structural information, and
the unbalanced samples lead to a decrease in the modeling ability of the
model. As the input length of the model increases, the other models get
better, and the performance of different models on this dataset improves. This
may be due to the fact that this dataset has much less structural information
than Devign (only 6%).

Result Answer to RQ2: Pre-processing methods that preserve structural features
can effectively improve the accuracy of vulnerability prediction, especially
in complex code structures. Meanwhile, code models with longer input lengths
maintain an advantage on this task.

###  V-D RQ3. Effects of using different code models in CSLS

TABLE V: Comparison results for different models on Devign dataset.  Dataset | Devign [25]
---|---
Models |  Acc |  Recall |  Prec |  F1
CodeBERT(S) |  65.95 |  44.46 |  70.54 |  54.54
CB+CB |  65.84 |  44.39 |  70.32 |  54.42
TRACED |  64.42 |  61.27 |  60.03 |  61.05
T+T |  67.60 |  57.37 |  67.28 |  61.93
UniXcoder-base(S) |  68.85 |  58.64 |  68.91 |  63.36
Un-b+Un-b |  69.43 |  58.08 |  70.23 |  63.58
UniXcoder-nine(S) |  69.10 |  56.73 |  70.28 |  62.78
Un-N+Un-N |  70.57 |  59.36 |  71.70 |  64.95

In this section, we investigate the effects of using different code models in
the Code-Semantic Learning System (CSLS). The experiment involves two code
models, allowing for various combinations. The results, as summarized in Table
V, demonstrate that our method provides enhancements regardless of the base
model used.

Table V presents a comparison of different models on the Devign dataset. The
metrics considered include Accuracy (Acc), Recall, Precision (Prec), and F1
score. Here are the detailed observations from the results: Table V shows that
combining different code models leads to improved performance, with UniXcoder
models showing significant gains. The Un-N+Un-N combination achieves the
highest metrics, including an accuracy of 70.57%, recall of 59.36%, precision
of 71.70%, and an F1 score of 64.95%. These results indicate that leveraging
different model configurations can significantly enhance vulnerability
detection performance.

From these results, it is evident that using different combinations of code
models in CSLS leads to consistent improvements across all evaluated metrics.
Notably, the combinations involving UniXcoder exhibit significant
enhancements, with the Un-N+Un-N combination standing out as the most
effective. These experimental results verify that our method can be
effectively extended based on different code models for different application
scenarios.

Result Answer to RQ3: In different scenarios, using different code models to
construct CLSs vulnerability detection models can effectively improve the
accuracy of vulnerability detection. This indicates that the line-level
structural semantics is helpful for the model to perform the vulnerability
detection task.

###  V-E Hyperparameter Experiments

CSLS has two hyperparameters: the number of lines $k$ and the default number
of tokens per line $p$. Given the strong domain-specific nature of the
vulnerability detection task, it is advisable to choose the settings of these
hyperparameters based on the scenario. For the experiments in this paper, we
selected the hyperparameter settings according to the code style of the
projects included in the Devign dataset.

As shown in Table VI, we experimented with different combinations of $p$ and
$k$. Specifically, we tested $p$ values of 10 and 20, and $k$ values of 70,
100, and 120. The results indicate that the best performance in terms of
accuracy (70.57) and F1-score (64.95) was achieved with $p=20$ and $k=100$. We
did not explore other combinations of $p$ and $k$ as the average number of
lines in the code snippets of this dataset is 117, with an average of 15
tokens per line.

TABLE VI: Comparison results for different Hyperparameter.  | Hyper-parameter | Devign [25]
---|---|---
P |  K |  Acc |  Recall |  Prec |  F1
20 |  70 |  69.32 |  63.10 |  67.86 |  65.40
20 |  100 |  70.57 |  59.36 |  71.70 |  64.95
20 |  120 |  69.21 |  51.47 |  73.57 |  61.07
10 |  70 |  69.61 |  61.59 |  68.95 |  65.06
10 |  100 |  69.83 |  60.23 |  69.93 |  64.72
10 |  120 |  69.10 |  59.92 |  68.80 |  64.05

##  VI Discussion

###  VI-A Influence of structural information on positive and negative samples

This experiment is designed to evaluate the influence of different pre-
processing on the structural information of positive (vulnerable) and negative
(non-vulnerable) samples in the dataset. Specifically, the dataset is pre-
processed in two distinct ways, and then the UniXcoder model is fine-tuned for
four epochs. The global CLS semantic features are subsequently reduced to a
two-dimensional classification plane. In the figures provided, label 0
represents vulnerable samples, and label 1 represents non-vulnerable samples.
Fig. 5(a) and Fig. 5(b) illustrate the results of the two pre-processing
methods.

(a) Pre-processing that preserves the structure in dataset Devign

(b) Pre-processing that does not preserve the structure in dataset Devign

Figure 5: Comparison of vulnerability detection performance of different prep-
rocessing process.

Fig. 5(a), the pre-processing method preserves the structural information of
the dataset. This is evident from the distinct clustering of the positive
(blue) and negative (orange) samples. The separation between the two clusters
suggests that the preserved structural information facilitates better
differentiation between vulnerable and non-vulnerable samples. As a result,
the classifier can more easily identify and segregate these samples based on
their inherent characteristics.

Conversely, in Fig. 5(b), the pre-processing method does not preserve the
structural information. This results in a more scattered distribution of both
positive and negative samples across the classification plane. The overlap
between the blue and orange points indicates that the structural distinctions
between the samples have been diminished. Consequently, the classifier faces
increased difficulty in distinguishing between vulnerable and non-vulnerable
samples, likely leading to reduced detection performance.

As shown in Fig. 5(a), the non-vulnerable samples (located in the lower left)
exhibit clear clustering. This suggests that such a pre-processing method is
more beneficial for the model in classifying positive samples.

##  VII THREATS TO VALIDITY

Threats: We used multiple code models as well as a transformer model for the
vulnerability detection task, which may increase the model parameters
(297.74M). This can lead to higher training and deployment costs. However, due
to the scarcity of vulnerability data, researchers generally can only use
smaller code models (110M-770M) to complete fine-tuning, so we believe that
this overhead is in an acceptable range. Inside. At the same time, the batch
design with line-level semantic awareness does not show a multiple increase in
memory footprint.

Internal Validity. During validation on different datasets, we found that CSLS
is affected by code style. Code with a minimalist style is likely to be less
affected by the approach as they almost have similar code structure. The
vulnerability detection of code with complex structure is more beneficial than
the structure-based detection model.

##  VIII CONCLUSION

In this paper, we introduced the Code Structure-Aware Network through Line-
level Semantic Learning (CSLS) to enhance code vulnerability detection. The
primary motivation was to address the limitations of existing methodologies
that often disregard the structural elements of code, treating it as mere text
sequences. This oversight results in the loss of crucial structural
information, adversely affecting the accuracy of vulnerability detection.

Our approach involves a refined code text processing workflow that retains
structural elements, such as newlines and indentation, before modeling. This
allows our proposed CSLS architecture to effectively capture and utilize line-
level structural and semantic information. The CSLS architecture integrates
three key components: global vulnerability awareness, line-structural
awareness, and sensitive-line awareness, which together improve the model’s
ability to detect vulnerabilities. The findings underscore the importance of
preserving and exploiting structural information in code to enhance the
efficacy of vulnerability detection models. By integrating structural and
semantic learning at the line level, CSLS provides a robust framework for more
accurate and efficient vulnerability detection, paving the way for improved
software security.

Future work will focus on further refining the model architecture and
exploring its applicability to other programming languages and broader
categories of software vulnerabilities. Additionally, we aim to investigate
the impact of different pre-processing techniques and model configurations to
further enhance the detection capabilities of CSLS.

## References

  * [1] M. Fu and C. Tantithamthavorn, “Linevul: A transformer-based line-level vulnerability prediction,” in _Proceedings of the 19th International Conference on Mining Software Repositories_ , 2022, pp. 608–620.
  * [2] C. Thapa, S. I. Jang, M. E. Ahmed, S. Camtepe, J. Pieprzyk, and S. Nepal, “Transformer-based language models for software vulnerability detection,” in _Proceedings of the 38th Annual Computer Security Applications Conference_ , 2022, pp. 481–496.
  * [3] X. Cheng, G. Zhang, H. Wang, and Y. Sui, “Path-sensitive code embedding via contrastive learning for software vulnerability detection,” in _Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis_ , 2022, pp. 519–531.
  * [4] J. Jang-Jaccard and S. Nepal, “A survey of emerging threats in cybersecurity,” _Journal of computer and system sciences_ , vol. 80, no. 5, pp. 973–993, 2014.
  * [5] A. Johnson, K. Dempsey, R. Ross, S. Gupta, D. Bailey _et al._ , “Guide for security-focused configuration management of information systems,” _NIST special publication_ , vol. 800, no. 128, pp. 16–16, 2011.
  * [6] F. Yamaguchi, “Pattern-based vulnerability discovery.” Ph.D. dissertation, University of Göttingen, 2015.
  * [7] F. b. Yamagu chi, “Pattern-based methods for vulnerability discovery,” _it-Information Technology_ , vol. 59, no. 2, pp. 101–106, 2017.
  * [8] X. Duan, J. Wu, S. Ji, Z. Rui, T. Luo, M. Yang, and Y. Wu, “Vulsniper: Focus your attention to shoot fine-grained vulnerabilities.” in _IJCAI_ , 2019, pp. 4665–4671.
  * [9] R. Russell, L. Kim, L. Hamilton, T. Lazovich, J. Harer, O. Ozdemir, P. Ellingwood, and M. McConley, “Automated vulnerability detection in source code using deep representation learning,” in _2018 17th IEEE international conference on machine learning and applications (ICMLA)_. IEEE, 2018, pp. 757–762.
  * [10] Z. Li, D. Zou, S. Xu, X. Ou, H. Jin, S. Wang, Z. Deng, and Y. Zhong, “Vuldeepecker: A deep learning-based system for vulnerability detection,” _arXiv preprint arXiv:1801.01681_ , 2018.
  * [11] H. K. Dam, T. Tran, T. Pham, S. W. Ng, J. Grundy, and A. Ghose, “Automatic feature learning for vulnerability prediction,” _arXiv preprint arXiv:1708.02368_ , 2017.
  * [12] V.-A. Nguyen, D. Q. Nguyen, V. Nguyen, T. Le, Q. H. Tran, and D. Phung, “Regvd: Revisiting graph neural networks for vulnerability detection,” in _Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings_ , 2022, pp. 178–182.
  * [13] Checkmarx, “Online,” _Available: https://www.checkmarx.com/_ , 2022.
  * [14] Flawfinder, “Online,” _Available: http://www.dwheeler.com/ flawfinde/r_ , 2022.
  * [15] Z. Li, D. Zou, S. Xu, H. Jin, Y. Zhu, and Z. Chen, “Sysevr: A framework for using deep learning to detect software vulnerabilities,” _IEEE Transactions on Dependable and Secure Computing_ , vol. 19, no. 4, pp. 2244–2258, 2021.
  * [16] G. Lin, J. Zhang, W. Luo, L. Pan, and Y. Xiang, “Poster: Vulnerability discovery with function representation learning from unlabeled projects,” in _Proceedings of the 2017 ACM SIGSAC conference on computer and communications security_ , 2017, pp. 2539–2541.
  * [17] S. Cao, X. Sun, L. Bo, R. Wu, B. Li, and C. Tao, “Mvd: memory-related vulnerability detection based on flow-sensitive graph neural networks,” in _Proceedings of the 44th International Conference on Software Engineering_ , 2022, pp. 1456–1468.
  * [18] Y. Ding, S. Suneja, Y. Zheng, J. Laredo, A. Morari, G. Kaiser, and B. Ray, “Velvet: a novel ensemble learning approach to automatically locate vulnerable statements,” in _2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)_. IEEE, 2022, pp. 959–970.
  * [19] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang _et al._ , “Codebert: A pre-trained model for programming and natural languages,” in _Findings of the Association for Computational Linguistics: EMNLP 2020_ , 2020, pp. 1536–1547.
  * [20] D. Guo, S. Lu, N. Duan, Y. Wang, M. Zhou, and J. Yin, “Unixcoder: Unified cross-modal pre-training for code representation,” in _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2022, pp. 7212–7225.
  * [21] B. Steenhoek, H. Gao, and W. Le, “Dataflow analysis-inspired deep learning for efficient vulnerability detection,” in _Proceedings of the 46th IEEE/ACM International Conference on Software Engineering_ , 2024, pp. 1–13.
  * [22] S. Chakraborty, R. Krishna, Y. Ding, and B. Ray, “Deep learning based vulnerability detection: Are we there yet?” _arXiv preprint arXiv:2009.07235_ , 2020.
  * [23] S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang _et al._ , “Codexglue: A machine learning benchmark dataset for code understanding and generation,” _arXiv preprint arXiv:2102.04664_ , 2021.
  * [24] O. Golovneva, T. Wang, J. Weston, and S. Sukhbaatar, “Contextual position encoding: Learning to count what’s important,” _arXiv preprint arXiv:2405.18719_ , 2024.
  * [25] Y. Zhou, S. Liu, J. Siow, X. Du, and Y. Liu, “Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [26] S. Chakraborty, R. Krishna, Y. Ding, and B. Ray, “Deep learning based vulnerability detection: Are we there yet,” _IEEE Transactions on Software Engineering_ , 2021.
  * [27] X. Cheng, H. Wang, J. Hua, G. Xu, and Y. Sui, “Deepwukong: Statically detecting software vulnerabilities using deep graph neural network,” _ACM Transactions on Software Engineering and Methodology (TOSEM)_ , vol. 30, no. 3, pp. 1–33, 2021.
  * [28] M. Schuster and K. K. Paliwal, “Bidirectional recurrent neural networks,” _IEEE transactions on Signal Processing_ , vol. 45, no. 11, pp. 2673–2681, 1997.
  * [29] Y. Li, S. Wang, and T. N. Nguyen, “Vulnerability detection with fine-grained interpretations,” in _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2021, pp. 292–303.
  * [30] W. Zheng, Y. Jiang, and X. Su, “Vu1spg: Vulnerability detection based on slice property graph representation learning,” in _2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE)_. IEEE, 2021, pp. 457–467.
  * [31] R. Rabheru, H. Hanif, and S. Maffeis, “Deeptective: Detection of php vulnerabilities using hybrid graph neural networks,” in _Proceedings of the 36th annual ACM symposium on applied computing_ , 2021, pp. 1687–1690.
  * [32] Y. Wu, D. Zou, S. Dou, W. Yang, D. Xu, and H. Jin, “Vulcnn: An image-inspired scalable vulnerability detection system,” in _Proceedings of the 44th International Conference on Software Engineering_ , 2022, pp. 2365–2376.
  * [33] Y. Li, R. Zemel, M. Brockschmidt, and D. Tarlow, “Gated graph sequence neural networks,” in _Proceedings of ICLR’16_ , 2016.
  * [34] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “Smote: synthetic minority over-sampling technique,” _Journal of artificial intelligence research_ , vol. 16, pp. 321–357, 2002.
  * [35] C. Mao, Z. Zhong, J. Yang, C. Vondrick, and B. Ray, “Metric learning for adversarial robustness,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [36] A. Kanade, P. Maniatis, G. Balakrishnan, and K. Shi, “Learning and evaluating contextual embedding of source code,” in _International conference on machine learning_. PMLR, 2020, pp. 5110–5121.
  * [37] C. Niu, C. Li, V. Ng, J. Ge, L. Huang, and B. Luo, “Spt-code: Sequence-to-sequence pre-training for learning source code representations,” in _Proceedings of the 44th International Conference on Software Engineering_ , 2022, pp. 2006–2018.
  * [38] J. Lin, Y. Liu, Q. Zeng, M. Jiang, and J. Cleland-Huang, “Traceability transformed: Generating more accurate links with pre-trained bert models,” in _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. IEEE, 2021, pp. 324–335.
  * [39] J. Bai, Y. Wang, Y. Chen, Y. Yang, J. Bai, J. Yu, and Y. Tong, “Syntax-bert: Improving pre-trained transformers with syntax trees,” in _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , 2021, pp. 3011–3020.
  * [40] D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, L. Shujie, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu _et al._ , “Graphcodebert: Pre-training code representations with data flow,” in _International Conference on Learning Representations_ , 2020.
  * [41] M.-A. Lachaux, B. Roziere, M. Szafraniec, and G. Lample, “Dobf: A deobfuscation pre-training objective for programming languages,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 14 967–14 979, 2021.
  * [42] D. Hin, A. Kan, H. Chen, and M. A. Babar, “Linevd: Statement-level vulnerability detection using graph neural networks,” in _Proceedings of the 19th International Conference on Mining Software Repositories_ , 2022, pp. 596–607.
  * [43] H. Hanif and S. Maffeis, “Vulberta: Simplified source code pre-training for vulnerability detection,” in _2022 International joint conference on neural networks (IJCNN)_. IEEE, 2022, pp. 1–8.
  * [44] J. Zhang, Z. Liu, X. Hu, X. Xia, and S. Li, “Vulnerability detection by learning from syntax-based execution paths of code,” _IEEE Transactions on Software Engineering_ , 2023.
  * [45] Z. Wang, G. Li, J. Li, Y. Xiong, and Z. Jin, “M2cvd: Multi-model collaboration for code vulnerability detection,” _arXiv preprint arXiv:2406.05940_ , 2024.
  * [46] openAI, “Online,” _Available: https://www.chat.openai.com/_ , 2022.
  * [47] Y. Wang, H. Le, A. D. Gotmare, N. D. Bui, J. Li, and S. C. Hoi, “Codet5+: Open code large language models for code understanding and generation,” _arXiv preprint arXiv:2305.07922_ , 2023.