# A comparative study of neural network techniques for automatic software
vulnerability detection

Gaigai Tang, Lianxiao Meng,
Huiqiang Wang  School of computer science
and technology,
Harbin Engineering University,
Harbin, China  Shuangyin Ren, Qiang Wang, Lin Yang  National Key Laboratory of
Science and
Technology on Information System Security,
Institute of System Engineering,
Chinese Academy of Military Science,
Beijing, China  Weipeng Cao  College of Computer Science
and Software Engineering,
Shenzhen University,
Shenzhen, China

###### Abstract

Software vulnerabilities are usually caused by design flaws or implementation
errors, which could be exploited to cause damage to the security of the
system. At present, the most commonly used method for detecting software
vulnerabilities is static analysis. Most of the related technologies work
based on rules or code similarity (source code level) and rely on manually
defined vulnerability features. However, these rules and vulnerability
features are difficult to be defined and designed accurately, which makes
static analysis face many challenges in practical applications. To alleviate
this problem, some researchers have proposed to use neural networks that have
the ability of automatic feature extraction to improve the intelligence of
detection. However, there are many types of neural networks, and different
data preprocessing methods will have a significant impact on model
performance. It is a great challenge for engineers and researchers to choose a
proper neural network and data preprocessing method for a given problem. To
solve this problem, we have conducted extensive experiments to test the
performance of the two most typical neural networks (i.e., Bi-LSTM and RVFL)
with the two most classical data preprocessing methods (i.e., the vector
representation and the program symbolization methods) on software
vulnerability detection problems and obtained a series of interesting research
conclusions, which can provide valuable guidelines for researchers and
engineers. Specifically, we found that 1) the training speed of RVFL is always
faster than Bi-LSTM, but the prediction accuracy of Bi-LSTM model is higher
than RVFL; 2) using $doc2vec$ for vector representation can make the model
have faster training speed and generalization ability than using $word2vec$;
and 3) multi-level symbolization is helpful to improve the precision of neural
network models.

###### Keywords:

software vulnerability, neural network, machine learning, Bi-LSTM, RVFL

##  I Introduction

As the functions of modern software systems become more and more complicated,
software vulnerabilities caused by design flaws and implementation errors
become an inevitable problem in engineering. According to the statistics
released by the Common Vulnerabilities and Exposures (CVE) [1] and National
Vulnerability Database (NVD) [2], the number of software vulnerabilities has
increased from 1600 to nearly 100000 since 1999 [3]. These vulnerabilities
pose a serious threat to the security of existing software systems.

Most of the classical vulnerability detection techniques are driven by rules
[4],[5],[6],[7],[8],[9] and code similarity metrics [10],[11], which are
generated by experts experience. Although rule-based detection techniques can
be applied to vulnerability detection in specific scenarios, the performance
of these methods depends heavily on the experience of developers. Generally,
it is very difficult to describe the features of software vulnerabilities
accurately, which leads to the corresponding detection rules are also
difficult to be defined accurately and completely.

To alleviate the above problems, neural network-based automatic vulnerability
detection technology (source code level) has been proposed and shows great
potential in related issues [12],[13],[14],[15],[16]. Neural networks can
automatically extract complex features from input data, avoiding the problems
of high cost, instability, and incompleteness of manually constructing
features and empirically defining rules. A representative work is the
VulDeePecker system proposed in [15], which is the first effort to use neural
network techniques to solve software vulnerability detection problems.
Specifically, the VulDeePecker first extracts the $code\ gadget$, which is a
form of source code fragments related to the vulnerability, and then
symbolizes them to get the corresponding symbolic data. After that, it uses
$word2vec$ [17] to transform the symbolic data into the vector representation
and then applies the vectors to the model training of the Bidirectional Long
Short-term Memory network (Bi-LSTM) [18] to get the detection model. The
authors have used several experiments to prove that VulDeePecker is superior
to other classical vulnerability detection techniques in terms of the accuracy
of vulnerability detection.

The success of VulDeePecker shows that it is feasible to use neural networks
to detect software vulnerabilities, but there are still some shortcomings in
that work, as follows.

  * •

VulDeePecker only tried the Bi-LSTM with the iterative training mechanism to
train the detection model, but this type of neural network often requires a
long training time and will face many obstacles in practical applications.

  * •

VulDeePecker used $word2vec$ to vectorize the software codes and then used the
generated vector (variable length) as the input of the neural network. This
method sometimes requires us to perform additional work on the preprocessing
of the vector (e.g., padding zeros), which will cause the dimension of the
vector to be very high and affect the training efficiency of the model.
Moreover, this method may also lose important semantic information about the
source codes and affect the effectiveness of the vulnerability detection
model.

  * •

VulDeePecker used two types of symbolization to do symbolic representation,
but it is still unclear which one is the most suitable for neural networks.

The above unsolved problems inspired us to choose two completely different
neural networks and different data preprocessing methods to model software
vulnerability detection problems in this study. Specifically, we selected the
Bi-LSTM with the iterative training mechanism and the Random Vector Functional
Link network (RVFL) [19] with the non-iterative training mechanism to train on
the vulnerability data set to get corresponding detection models, and then
compared the precision and training efficiency of the models. We found that
the precision of the Bi-LSTM model is always higher than that of the RVFL
model, but the training efficiency of the RVFL is always higher than that of
the Bi-LSTM. Moreover, we have implemented a multi-level symbolization method
for symbolic representation and proposed the use of $doc2vec$ [20] for vector
representation. In details, we first perform three symbolizations to obtain
the symbolic representations of the source codes related to the vulnerability.
The advantage of using three levels of symbolization is that one can
significantly reduce the noise introduced by irrelevant information in
vulnerable codes. Then we use the $doc2vec$ to automatically transform the
symbolic representations of the source codes to their vector representations.
Our study found that $doc2vec$ is more suitable for modeling this problem than
$word2vec$ used in [15], because it can not only convert source codes with
arbitrary length into a fixed-length feature representation, but also better
grasp the semantic information. These advantages are helpful to improve the
precision and training efficiency of the model. These experimental results can
provide valuable guidance for researchers and engineers to choose appropriate
neural networks and data preprocessing methods for software vulnerability
detection problems.

The rest of this paper is organized as follows. Section II discusses the work
related to the automatic detection of software vulnerability. Section III
describes the details of the automatic software vulnerability detection system
using neural network techniques. In Section IV, we give details of our
experimental setting, results, and corresponding analysis. The conclusions and
future works are discussed in Section V.

##  II Related Work

Traditional machine learning techniques such as Decision Tree [21] and Support
Vector Machine [22], mainly extract the vulnerability features from the pre-
classified vulnerabilities. However, the vulnerability detection patterns
generated with this type of features are usually tailored for specific
vulnerabilities. In [23], both simple text features (e.g., character count,
character diversity, maximum nesting depth and complex text features (e.g.,
character n-grams, word n-grams and suffix trees) are extracted from the
source codes and analyzed by using the naive Bayes classifier. The
experimental results show that simple features performed unexpectedly better
compared to the complex features.

Neural network based techniques are able to learn vulnerability features
automatically. In [15], the authors present a vulnerability detection system
_VulDeePecker_ based on deep learning. _VulDeePecker_ obtains the samples by
first extracting code gadgets (i.e., snippets of codes that are semantically
related to the vulnerability) from the buggy programs and then transforming
into the vector representations. The learning algorithm is based on the Long
Short-Term Memory (LSTM). Experimental evaluations show that _VulDeePecker_
outperforms the state-of-the-art vulnerability detection systems in terms of
both accuracy and efficiency. [13] has implemented various machine learning
models for the detection of bugs that can lead to security vulnerabilities in
C/C++ code. They use features derived from the build process and the source
code. [14] has developed a vulnerability detection tool based on deep feature
representation learning that can directly interpret the parsed source codes.
The source codes are firstly transformed into tokens and then embedded as
vectors for both convolutional neural networks (CNNs) and recurrent neural
networks(RNNs). Experimental results show that the tool results in high
accuracy. [24] has proposed a systematic framework for using deep learning to
detect vulnerabilities that dubbed syntax-based, semantics-based and vector
representations (_SySeVR_). _SySeVR_ can accommodate syntax and semantic
information pertinent to vulnerabilities. The source codes are successively
represented by syntax-based, semantics-based and vector representations.
Experiments show that _SySeVR_ is able to detect several vulnerabilities that
are not reported in the National Vulnerability Database. [16] performs a
quantitative evaluation of the impacts of different factors (e.g., data
dependency and control dependency) on the effectiveness of neural network
based vulnerability detection techniques. In [12], the authors propose a tool
called _VuRLE_ for automatic detection and repair of vulnerabilities.
Experiments show that _VuRLE_ results in a large improvement on the accuracy
of detection and the number of repaired vulnerabilities.

There are also some vulnerability detection techniques that make use of
manually-defined features [4], [5], [6], [7], [8], [9], or code similarity
metrics [10],[11]. However, there are several drawbacks. First, defining
vulnerability features is error-prone and costs a lot of manual labor. Second,
the features can hardly be complete and usually contain only partial
information about the vulnerabilites, which may lead to a high rate of false-
positive and false-negative [15]. Moreover, code similarity based approaches
are limited to detecting vulnerabilities caused by code clone.

##  III The Methodology

We show in Figure. 1 an overview of the source code level automatic software
vulnerability detection system using neural network techniques. It first
creates a symbolic form of the $code\ gadget$ by using multilevel
symbolization. The $code\ gadgets$ are divided into two parts. One is for
training the detection model and the other one is for vulnerability detection.
Then it transforms the symbolic form into the vector representation with a low
dimension by using the tool $doc2vec$. Finally, it applies two neural network
techniques namely the Bi-LSTM and the RVFL to train the detection model. In
the subsequent sections, we present the details of the main components of the
system.

Figure 1:  Overview of the automatic software vulnerability detection system
using neural networks

###  III-A Symbolic representation

$Code\ gadget$ is a fragment of the codes that are closely relevant to the
vulnerability. By using symbolization, $code\ gadget$ can be further
transformed into a symbolic representation. The benefit of symbolic
representation is that it can effectively reduce the time cost of vector
representation precess since it can condense the size of the corpus, which is
used for training the vector representation tool. In symbolization,
vulnerability features in each $code\ gadget$ such as local variables or user-
defined functions are transformed into certain short and fixed symbolic
presentations, where the same features are mapped to the same symbolic
presentation. In this work, we deploy three symbolization types sorted by
priority.

  * •

Function calls symbolization (F): user-defined function names are symbolically
represented as FN. This symbolization type is set to the first priority
because vulnerability is usually caused by improper utilization of library/API
function calls. Symbolization on user-defined functions can improve the
Signal-Noise Ratio (SNR) of the library function in vulnerability information.

  * •

Variable symbolization (V): variable names including parameters and local
variables are symbolically represented as VN.

  * •

Data type symbolization (D): data types of variable and user-defined function
are symbolically represented as TN. It has the least priority since usually
many data types are not related to vulnerability information.

The symbol $N$ mentioned above in symbolization is a number representing the
sequence of the first occurrence of the feature. Besides, all the
symbolization types will reserve the keywords of C/C++ language.

Table I:  multilevel symbolization  Symbolization level | Symbolization group
---|---
Level 1 | F
Level 2 | F+V;F+D
Level 3 | F+V+D

We build a multilevel symbolization mechanism according to the priority of a
single symbolization shown in Table. I. Level 2 includes two symbolization
groups F+V and F+D. This is because symbolization V and D may have different
effects on improving the SNR of vulnerability information with different data-
sets.

We take Sample 0 below as an example and explain how the symbolization works.
We choose the symbolization group F+V in Level 2. From Sample 0 we can observe
that there are 2 user-defined functions, 5 variables and 2 data types.

  * •

In Level 1, the two user-defined functions are symbolically represented as F1
and F2.

  * •

In Level 2, the five variable names are symbolically represented as
$Vi,i\in[1,5]$.

  * •

In Level 3, the two data types are symbolically represented as T1 and T2.

As a result, with the three levels of symbolization, Sample 0 is gradually
simplified to a generalized symbolic representation, which may effectively
characterize different manifestations of the same vulnerability.

Sample 0

1:$static\ void\ goodG2B()$

2:$list<char*>\ dataList;$

3:$goodG2BSink(dataList);$

4:$void\ goodG2BSink(list<char*>\ List)$

5:$char*data=List.back();$

6:$if(sscanf(data,"\&d",\&n)==1))$

Level 1 F

1:$static\ void\ F1()$

2:$list<char*>\ dataList;$

3:$F2(dataList);$

4:$void\ F2(list<char*>\ List)$

5:$char*data=List.back();$

6:$if(sscanf(data,"\&d",\&n)==1))$

Level 2 F+V

1:$static\ void\ F1()$

2:$list<char*>\ V1;$

3:$F2(V1);$

4:$voidF2(list<char*>\ V2)$

5:$char*V3=V2.back();$

6:$if(sscanf(V3,V4,\&V5)==1))$

Level 3 F+V+D

1:$T1\ F1()$

2:$list<T2*>\ V1;$

3:$F2(V1);$

4:$void\ F2(list<T2*>\ V2)$

5:$T2*V3=V2.back();$

6:$if(sscanf(V3,V4,\&V5)==1))$

###  III-B Vector representation

Since the input type accepted by neural networks is a vector, it still needs
to be converted to the vector representation after symbolizing the source
codes. Currently, the most popular vectorization methods are $word2vec$ [25]
and $doc2vec$ [26].

$word2vec$ is a distributed representation method for words. Compared with the
one-hot representation, a high-dimensional and sparse representation method,
$word2vec$ can get low-dimensional and dense vector representation, which is
conducive to the improvement of training efficiency and accuracy of the model,
so it has been widely used in vulnerability detection scenarios recently [13],
[15], [16]. However, $word2vec$ is not suitable for vectorizing sentences or
documents because it ignores the influence of the word order on the
information of sentence or document.

$doc2vec$ was proposed in [26], where the authors proposed the unsupervised
algorithm $Paragraph\ Vector$ that can learn fixed-length feature
representations from variable-length pieces of texts, ranging from sentences
to documents. The $Paragraph\ Vector$ can memorize the missing content in the
current context or the topic of the paragraph, so it can extract global
features better than $word2vec$.

The input vector of neural networks is required to be fixed length. $word2vec$
converts words to vector representations in a one-to-one fashion, therefore
the length of the converted vector representation varies with the length of
the text. To satisfy the requirement of neural networks, the vector
representations generated by $word2vec$ need to be further processed to get
the corresponding fixed-length version. Different from $word2vec$, $\ doc2vec$
can directly get fixed-length vector representations from pieces of texts with
arbitrary length. As mentioned above, $\ doc2vec$ can also extract more
semantic information from the context of the text than $word2vec$. Therefore,
one can infer that $\ doc2vec$ is more suitable than $word2vec$ in the
scenarios with rich semantic information.

###  III-C Neural network model

Here we introduce two types of neural networks, that is, Recurrent Neural
Network (RNN) [27] using the iterative training mechanism and RVFL [28] using
the non-iterative training mechanism.

RNN is a typical neural network for processing sequence data [27][29]. One of
its biggest advantages is the ability to mine the context information of the
data. However, since the original RNN can only mine the text information in a
forward way, while the vulnerability information usually relates to both the
preceding and backward parts of the codes, it can be inferred that the
original RNN may not be suitable for dealing with the vulnerability detection
problems. For this reason, we choose the improved version of RNN, that is, Bi-
LSTM, which can not only express the long-term dependency information in the
input data but also can bi-directionally mine the context information of the
data (i.e., forward and backward). These advantages make Bi-LSTM very suitable
for processing software vulnerability detection data (source code level). Note
that all parameters of the Bi-LSTM are iteratively fine-tuned using methods
like gradient descent. A typical network structure of Bi-LSTM is shown in
Figure. 2.

Different from Bi-LSTM, RVFL is a special type of feed-forward neural networks
with the non-iterative training mechanism. RVFL was proposed by Pao YH et al.
in the 1990s [28], which randomly assigns values to some parameters according
to certain rules and keeps these parameters be frozen throughout the training
process, while other parameters are calculated by the least square method.
This training mechanism can bring RVFL much faster training speed than
traditional neural networks with the iterative training mechanism on some
tasks with a relatively small data scale. In recent years, RVFL and its
variants have been widely concerned and applied in many scenarios [30]. The
network structure of the RVFL is shown in Figure. 3. For the technical details
of RVFL, please refer to [31].

Figure 2:  A typical network structure of Bi-LSTM  Figure 3:  A typical RVFL
with a single hidden layer network structure

##  IV Experiment and Evaluation

Our aim is to quantitatively evaluate the performances of the neural network
based automatic software vulnerability detection techniques. To be specific,
we investigate the following questions in the experiments.

  * •

Question1 (Q1): How different neural network models perform on vulnerability
detection on the source code level?

  * •

Question2 (Q2): How different vector representation methods affect the
performances of neural networks? Specifically, does $doc2vec$ outperform
$word2vec$ for the task of vulnerability detection?

  * •

Question3 (Q3): What are the effects of different symbolization on the
performances of neural networks?

###  IV-A Experiment setting and implementation

####  IV-A1 Data-set

We include the following three data-sets from [15] in our experiments. Each
sample is a source code file with known vulnerabilities. In Table. II we show
the number of samples (i.e., source code files) in each data-set.

  * •

BE-ALL: samples with Buffer Error vulnerabilities (CWE-119) and ALL
library/API function calls.

  * •

RM-ALL: samples with Resource Management error vulnerabilities (CWE-399) and
ALL library/API function calls.

  * •

HY-ALL: sample with HYbrid buffer error vulnerabilities (CWE-119) and resource
management error vulnerabilities (CWE-399) and ALL library/API function calls.

Table II:  number of samples in each data-set  Data-set | Code Gadgets |  Vulnerable code gadgets |  Not vulnerable code gadgets
---|---|---|---
BE-ALL | 39753 |  10440 |  29313
RM-ALL | 21885 |  7285 |  14600
HY-ALL | 61638 |  17725 |  43913

Each data-set is partitioned into two parts with a proportion of 80% and 20%,
where the larger part is for training and the other one is for detection. All
samples in the data-set are in the form of code gadgets with ground truth
labels.

####  IV-A2 Parameters setting for neural networks

In our experiments, we used two types of neural networks for model training:
Bi-LSTM and RVFL. For both, there is only one hidden layer in the network
structure. In Bi-LSTM, hidden layer neurons perform two-time computation
operations (i.e., forward and backward), while in RVFL they only perform a
one-time computation operation. We considered a one-time computation operation
of the neurons as a unit of measurement. To make a fair comparison, we set the
number of hidden layer neurons of RVFL to twice that of Bi-LSTM, which
guarantees the structural complexity of the two neural networks is essentially
the same. We have implemented the CPU versions of Bi-LSTM and RVFL, and all
the models are trained in the PC environment with CPU. For Bi-LSTM, the batch
size, dropout rate, the number of epochs, and the number of the hidden layer
neurons were set to 64, 0.5, 2, and 60, respectively, and the RMSProp was
chosen as the optimizer. For RVFL, the number of the hidden layer neurons was
set to 120 and the Sigmoid function was chosen as the activation function. The
input weights and the hidden biases of RVFL were generated randomly from (-1,
1) and (0, 1) under a uniform distribution, respectively.

###  IV-B Results and evaluation

####  IV-B1 Evaluation metrics

In our experiment, we used the same indexes as [32] to evaluate the
effectiveness of the vulnerability detection model, that is, False Positive
Rate (FPR), False Negative Rate (FNR), Precision (P), and F1-measure (F1). The
value range of these four indicators is [0, 1]. For FPR and FNR, the closer
their values are to 0, the better the performance of the model; for other
indicators, the closer their values are to 1, the better the performance of
the model.

The quality of vector representation can be evaluated by $CosineDistance$
($cosine$) between vectors in the vector space, which can be calculated by the
following formula. The range of the $cosine$ value is [0,1]. The closer the
value is to 1, the more similar the two vectors are.

$cosine(A,B)=\frac{A\cdot B}{||A||_{2}||B||_{2}}$

where A and B refer to vectors.

In our experiments, we have the following three configurations:

  * •

$word2vec$ with Bi-LSTM (w+B), which was used by VulDeePecker;

  * •

$doc2vec$ with Bi-LSTM (d+B);

  * •

$doc2vec$ with RVFL (d+R).

####  IV-B2 Results for Q1

Regarding the impacts of different neural network models on the performances
of vulnerability detection. We evaluate the precision of the above three
configurations on all data-sets, while we evaluate the efficiency of
configuration d+B and d+R on the largest data-set HY-ALL. In the experiments,
all the three data-sets are preprocessed with the symbolization group F+V.

Table III:  effect of different vector representation tools on vulnerability detection precision Data-set | Pattern | FPR(%) | FNR(%) | P(%) | F1(%)
---|---|---|---|---|---
BE-ALL | w+B | 2.9 | 18.0 | 91.7 | 86.6
| d+B | 3.9 | 16.9 | 88.1 | 85.5
| d+R | 6.7 | 32.5 | 85.1 | 74.3
RM-ALL | w+B | 2.8 | 4.7 | 94.6 | 95.0
| d+B | 3.8 | 9.7 | 91.9 | 91.1
| d+R | 3.1 | 20.0 | 90.5 | 86.5
HY-ALL | w+B | 5.1 | 16.1 | 86.9 | 85.4
| d+B | 3.3 | 16.2 | 91.0 | 87.2
| d+R | 7.7 | 29.7 | 85.0 | 75.6

As illustrated in Table. III, the configurations d+B and w+B outperform d+R on
the metrics P and F1 for all the three data-sets. In terms of FPR and FNR, the
configurations w+B and d+B have a smaller sum of FPR and FNR than d+R. The
general result is that the configurations with Bi-LSTM perform better than the
ones with RFVL on the precision of vulnerability detection. This can be
explained by the fact that Bi-LSTM is able to express the long-term dependency
information in the input, while RVFL is based on forward neural network, which
is slightly inferior to Bi-LSTM in context processing.

Table IV: effect of different neural network models on vulnerability detection efficiency Pattern |  Training code gadgets |  Detection code gadgets |  Training Time(s) |  Detection Time(s)
---|---|---|---|---
w+B |  48744 |  12894 |  36372.2 |  156.2
d+B |  49310 |  12328 |  1123.5 |  3.2
d+R |  49310 |  12328 |  4.62 |  0.16

From Table. IV, we can observe that the configuration w+B takes the longest
time for training and detection on the data-set HY-ALL, while the
configuration d+B takes less than 1/30 of the time needed by w+B. It’s because
that configuration w+B in [15] outputs vectors with a long dimension of 2500,
which creates a higher computation complexity for the neural networks.
Moreover, compared with the configuration d+B, configuration d+R further
improves the efficiency of training and detection to the level of a few
seconds. This can be explained by the fact that the non-iterative training
mechanism of RVFL reduces the computation of parameters, however with a
sacrifice of precision.

We can conclude that the configuration with Bi-LSTM achieves a higher
precision, while the configuration with RVFL is more effective. Bi-LSTM
results in a high accuracy for detecting vulnerabilities because of its
advantage of context processing, while RVFL results in a high speed for
training model because of its non-iterative training mechanism. In the
practice, we usually need to make a trade-off between the prediction accuracy
and efficiency of the model. Specifically, if the speed of model training
takes precedence over its accuracy, then RVFL is recommended; otherwise, Bi-
LSTM is recommended.

####  IV-B3 Results for Q2

To answer the second question, we evaluate the accuracy of the two vector
representation methods $doc2vec$ and $word2vec$. We implement experiments with
the samples shown in Sample 1, Sample 2 , Sample 3 and Sample 4. Sample 2 and
Sample 4 are labeled as ’vulnerable’ while the Sample 1 and Sample 3 are not.
All samples are from the data-set BE-ALL and HY-ALL. We evaluate vector
representations by using the similarity measure $cosine$. Both samples are
preprocessed with symbolization group of F+V. The vector dimension of
$word2vec$ is set to be 2500, where the vector dimension of one word is set to
50 and the number of words to represent a paragraph is set to be 50. The
vector dimension of $doc2vec$ is set to be 250. The different dimension
settings of $word2vec$ and $doc2vec$ is because that if both dimensions are
set to be the same (e.g. 250), then for $word2vec$ the vector dimension of one
word will be 5, or the number of words to represent a paragraph will be 5,
which may have a great impact on the effectiveness of vector representation.
As a result, the comparison of the accuracy of $word2vec$ and $doc2vec$ is
carried out under the condition that they both use a proper dimension of
vector representation.

Sample 1

1:$data=(char*)malloc(100*sizeof(char));$

2:$goodG2BSource(data);$

3:$void\ goodG2BSource(char*\&data)$

4:$memset(data,^{\prime}A^{\prime},50-1);$

5:$data[50-1]=^{\prime}\backslash 0^{\prime};$

6:$char\ dest[50]="";$

7:$strcpy(dest,data);$

Sample 2

1:$char*\leavevmode\nobreak\ \leavevmode\nobreak\ data;$

2:$data=(char*)malloc(100*sizeof(char));$

3:$if(5==5)$

4:$memset(data,^{\prime}A^{\prime},100-1);$

5:$data[100-1]=^{\prime}\backslash 0^{\prime};$

6:$char\ dest[50]="";$

7:$strcpy(dest,data);$

Sample 3

1:$data=-1;$

2:$char\ inputBuffer[CHAR\\_ARRAY\\_SIZE]=^{\prime\prime\prime\prime};$

3:$if(fgets(inputBuffer,CHAR\\_ARRAY\\_SIZE,$

4:$stdin)!=NULL)$

Sample 4

1:$char\ inputBuffer[CHAR\\_ARRAY\\_SIZE]=^{\prime\prime\prime\prime};$

2:$if(fgets(inputBuffer,CHAR\\_ARRAY\\_SIZE,$

3:$stdin)!=NULL)$

4:$data=atoi(inputBuffer);$

Table V: effectiveness of word2vec compare with doc2vec Sample | Tool |  Vector dimension |  Cosine(BE-ALL) |  Cosine(HY-ALL)
---|---|---|---|---
1&2 | word2vec |  2500 |  0.832 |  0.828
doc2vec |  250 |  0.642 |  0.639
3&4 | word2vec |  2500 |  0.482 |  0.514
doc2vec |  250 |  0.586 |  0.578

From Table. V, we observe that for Sample1 and Sample2 $word2vec$ outputs
vectors with a higher $cosine$ than $doc2vec$ , while for Sample3 and Sample4
it outputs a opposite outcome. Obviously, the similarity between Sample1 and
Sample2 is not as high as the result of $word2vec$, but it is close to the
result of $doc2vec$. While the similarity between Sample3 and Sample4 is
closer to the result of $doc2vec$, proving that $doc2vec$ can generate a more
accurate vector representation with lower dimension than $word2vec$. It can be
explained by the fact that as noted in [15], in order to obtain a fixed length
of vector representation, vectors generated from $word2vec$ should be padded
with zeros, which may cause the loss of semantic information of the samples.
Moreover, it is understood that vectors with low dimensions can speed up the
training process of the neural network model. This is also justified by the
results in Table. III, where the configurations with $doc2vec$ show better
results than the ones with $word2vec$ due to the benefits gained from the
vector representation of $doc2vec$.

####  IV-B4 Results for Q3

Given that fact that the configuration d + B has the highest precision of
vulnerability detection, we take the configuration d + B as the baseline to
discuss whether symbolization can further improve the accuracy. We implement
experiments with all the three data-sets. For each data-set, we apply the
symbolization level from 1 to 3 for preprocessing.

Table VI:  Effectiveness of symbolization levels on precision Data-set |  Symbolization group | FPR(%) | FNR(%) | P(%) | F1(%)
---|---|---|---|---|---
BE-ALL |  F | 2.9 | 13.8 | 91.2 | 88.6
F+V | 3.9 | 16.9 | 88.1 | 85.5
F+D | 3.2 | 15.4 | 90.4 | 87.4
F+V+D | 3.5 | 15.5 | 89.4 | 86.9
RM-ALL |  F | 2.8 | 7.8 | 94.1 | 93.1
F+V | 3.8 | 9.7 | 91.9 | 91.1
F+D | 2.6 | 6.5 | 94.5 | 94.0
F+V+D | 3.4 | 9.1 | 92.7 | 91.8
HY-ALL |  F | 3.2 | 12.2 | 92.0 | 89.8
F+V | 3.3 | 16.2 | 91.0 | 87.2
F+D | 3.2 | 14.4 | 91.7 | 88.5
F+V+D | 3.3 | 14.0 | 91.1 | 88.5

Table. VI summarizes the results of how different levels of symbolization can
affect the precision.

From the perspective of data-set types, different symbolization levels have a
bigger impact on the precision of vulnerability detection with smaller data-
sets, which show the maximum precision deviation of 3.1% in BE-ALL and 2.6% in
RM-ALL. However, in large data-set HY-ALL, this phenomenon is much weakened,
which is shown as 0.9%. This may be because since the richness of data-sets
improves the generalization ability of the detection model itself, and the
impact of symbolization is gradually reduced. From the perspective of
symbolization levels, the configuration d + B with the symbolization level of
1 shows a better and more stable performance than other symbolization levels,
while the symbolization level of 2 results an unstable performance, and the
symbolization level of 3 outcomes the worst performance. The main reason is
that with a high level of symbolization we may lose some key vulnerability
information in the source codes. Moreover, it should be mentioned that
symbolization groups of F +D outperforms than symbolization groups of F + V
with all data-sets, it may due to there are many contents related to data type
in the source codes, after symbolizing them can better capture the
vulnerability information.

##  V Conclusions

In this paper, we conducted a comprehensive experiment to analyze the impact
of the types of neural networks and different data preprocessing methods on
solving software vulnerability detection problems. Our experimental results
show that the RVFL outperforms Bi-LSTM on the efficiency of vulnerability
detection, while Bi-LSTM performs better on the precision. Moreover, vector
representation using $doc2vec$ and an appropriate level of symbolization can
effectively improve the accuracy of vulnerability detection. These
experimental conclusions will provide researchers and engineers with
guidelines when choosing neural networks and data preprocessing methods for
vulnerability detection.

In the future, we will study the combination of Bi LSTM and RVFL to design a
new neural network and use it for source code level vulnerability detection.
It is expected that the new method will meet real-life engineering needs in
terms of efficiency and precision.

## Acknowledgment

This research is partially supported by the vulnerability analysis technology
for UAV communication protocol project of National Key Laboratory of Science
and Technology on Information System Security, the Natural Science Foundation
of China (No. 61872104), and the Opening Project of Shanghai Trusted
Industrial Control Platform (TICPSH202003008-ZC).

## References

  * [1] Common vulnerabilities and exposures.  In https://cve.mitre.org.
  * [2] National vulnerability database.  In https://nvd.nist.gov/.
  * [3] Tiantian Ji, Yue Wu, Chang Wang, and et al.  The coming era of alphahacking?: A survey of automatic software vulnerability detection, exploitation and patching techniques.  In Third IEEE International Conference on Data Science in Cyberspace, DSC 2018, Guangzhou, China, June 18-21, 2018, pages 53–60, 2018\.
  * [4] FlawFinder.  In http://www.dwheeler.com/flawfinder.
  * [5] Rough Audit Tool.  In https://code.google.com/archive/p/rough-auditing-tool-for-security/.
  * [6] John Viega, J. T. Bloch, Y. Kohno, and Gary McGraw.  ITS4: A static vulnerability scanner for C and C++ code.  In 16th Annual Computer Security Applications Conference (ACSAC 2000), 11-15 December 2000, New Orleans, Louisiana, USA, page 257, 2000.
  * [7] Checkmarx.  In https://www.checkmarx.com/.
  * [8] Coverity.  In https://scan.coverity.com/.
  * [9] HP Fortify.  In https://www.hpfod.com/.
  * [10] Seulbae Kim, Seunghoon Woo, Heejo Lee, and Hakjoo Oh.  VUDDY: A scalable approach for vulnerable code clone discovery.  In 2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA, USA, May 22-26, 2017, pages 595–614, 2017.
  * [11] Zhen Li, Deqing Zou, Shouhuai Xu, and et al.  Vulpecker: an automated vulnerability detection system based on code similarity analysis.  In Proceedings of the 32nd Annual Conference on Computer Security Applications, ACSAC 2016, Los Angeles, CA, USA, December 5-9, 2016, pages 201–213, 2016.
  * [12] Siqi Ma, Ferdian Thung, David Lo, and et al.  Vurle: Automatic vulnerability detection and repair by learning from examples.  In Computer Security - ESORICS 2017 - 22nd European Symposium on Research in Computer Security, Oslo, Norway, September 11-15, 2017, Proceedings, Part II, pages 229–246, 2017.
  * [13] Jacob A. Harer, Louis Y. Kim, Rebecca L. Russell, and et al.  Automated software vulnerability detection with machine learning.  CoRR, abs/1803.04497, 2018.
  * [14] Rebecca L. Russell, Louis Y. Kim, Lei H. Hamilton, and et al.  Automated vulnerability detection in source code using deep representation learning.  In 17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018, Orlando, FL, USA, December 17-20, 2018, pages 757–762, 2018.
  * [15] Zhen Li, Deqing Zou, Shouhuai Xu, and et al.  Vuldeepecker: A deep learning-based system for vulnerability detection.  In 25th Annual Network and Distributed System Security Symposium, NDSS 2018, San Diego, California, USA, February 18-21, 2018, 2018\.
  * [16] Zhen Li, Deqing Zou, Jing Tang, and et al.  A comparative study of deep learning-based vulnerability detection system.  IEEE Access, 7:103184–103197, 2019.
  * [17] Word2vec.  In https://nvd.nist.gov/.
  * [18] Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio.  On the properties of neural machine translation: Encoder-decoder approaches.  In Proceedings of SSST@EMNLP 2014, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, Doha, Qatar, 25 October 2014, pages 103–111, 2014.
  * [19] Boris Igelnik and Yoh-Han Pao.  Stochastic choice of basis functions in adaptive function approximation and the functional-link net.  IEEE Trans. Neural Networks, 6(6):1320–1329, 1995.
  * [20] Doc2vec.  In https://radimrehurek.com/gensim/models/doc2vec.html.
  * [21] J. Ross Quinlan.  Induction of decision trees.  Machine Learning, 1(1):81–106, 1986.
  * [22] Corinna Cortes and Vladimir Vapnik.  Support-vector networks.  Machine Learning, 20(3):273–297, 1995.
  * [23] Boris Chernis and Rakesh M. Verma.  Machine learning methods for software vulnerability detection.  In Proceedings of the Fourth ACM International Workshop on Security and Privacy Analytics, IWSPA@CODASPY 2018, Tempe, AZ, USA, March 19-21, 2018, pages 31–39, 2018.
  * [24] Zhen Li, Deqing Zou, Shouhuai Xu, and et al.  Sysevr: A framework for using deep learning to detect software vulnerabilities.  CoRR, abs/1807.06756, 2018.
  * [25] Lior Wolf, Yair Hanani, Kfir Bar, and Nachum Dershowitz.  Joint word2vec networks for bilingual semantic representations.  Int. J. Comput. Linguistics Appl., 5(1):27–42, 2014.
  * [26] Quoc V. Le and Tomas Mikolov.  Distributed representations of sentences and documents.  In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, pages 1188–1196, 2014\.
  * [27] Jinsong Su, Zhixing Tan, and Deyi Xiong et al.  Lattice-based recurrent neural network encoders for neural machine translation.  In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pages 3302–3308, 2017.
  * [28] Yoh-Han Pao and Yoshiyasu Takefuji.  Functional-link net computing: Theory, system architecture, and functionalities.  IEEE Computer, 25(5):76–79, 1992.
  * [29] Mike Schuster and Kuldip K. Paliwal.  Bidirectional recurrent neural networks.  IEEE Trans. Signal Processing, 45(11):2673–2681, 1997.
  * [30] Weipeng Cao, Xizhao Wang, Zhong Ming, and Jinzhu Gao.  A review on neural networks with random weights.  Neurocomputing, 275:278–287, 2018.
  * [31] Weipeng Cao, Jinzhu Gao, Zhong Ming, Shubin Cai, and Hua Zheng.  Impact of probability distribution selection on RVFL performance.  In Meikang Qiu, editor, Smart Computing and Communication - Second International Conference, SmartCom 2017, Shenzhen, China, December 10-12, 2017, Proceedings, volume 10699 of Lecture Notes in Computer Science, pages 114–124. Springer, 2017.
  * [32] Marcus Pendleton, Richard Garcia-Lebron, Jin-Hee Cho, and Shouhuai Xu.  A survey on systems security metrics.  ACM Comput. Surv., 49(4):62:1–62:35, 2017.


#  $b\rightarrow cl\overline{\nu}$ decays at LHCb

C. Chen
(on behalf of the LHCb Collaboration)  Aix Marseille Univ, CNRS/IN2P3, CPPM,
163, avenue de Luminy - Case 902, 13288 Marseille, France

###### Abstract

This report introduces two recent measurements of semileptonic $b$-hadron
decays at the LHCb experiment, including a test of Lepton Flavour Universality
(LFU) using ${{\kern 1.79993pt\overline{\kern-1.79993ptB}}{}^{0}}\rightarrow
D^{(*)+}l^{-}{\overline{\nu}}_{l}$ decays where $l\in\\{\mu,\tau\\}$, and a
study of the ${D}^{*+}$ longitudinal polarisation in the ${{\kern
1.79993pt\overline{\kern-1.79993ptB}}{}^{0}}\rightarrow{{D}^{*+}}{\tau^{-}}{{\overline{\nu}}_{\tau}}$
decay. With the inclusion of the new results of the LFU ratios, the world
average on $R(D)$ and $R({{D}^{*}})$, still shows a tension over three
standard deviations from the SM prediction, while the measured ${D}^{*+}$
longitudinal polarisation is compatible with the SM value.

##  1 Introduction

Semileptonic decays of $b$-hadrons in the Standard Model (SM) proceed via the
$b\rightarrow cl{\overline{\nu}}$ transition mediated by the $W$ boson at the
tree level. Such decays present valuable opportunities to search for New
Physics (NP) beyond the SM. Additional particles not included in the SM , e.g.
charged Higgs and leptoquarks [1], might also contribute to the $b\rightarrow
cl{\overline{\nu}}$ process alongside the $W$ boson, leading to deviations of
physics observables from SM expectations. Precise measurements of these
observables are crucial for testing the SM and exploring potential NP. Notable
examples of these observables are Lepton Flavour Universality (LFU) ratios,
defined as the branching-fraction ratio of two semileptonic decays involving
leptons in different generations, and angular coefficients that characterise
the differential decay width across the phase space.

There is a long-standing tension in the world averages of the LFU ratios,
$R(D^{(*)})\equiv\mathcal{B}({\kern
1.79993pt\overline{\kern-1.79993ptB}}\rightarrow
D^{(*)}{\tau^{-}}{{\overline{\nu}}_{\tau}})/\mathcal{B}({\kern
1.79993pt\overline{\kern-1.79993ptB}}\rightarrow
D^{(*)}\mu^{-}{\overline{\nu}}_{\mu})$, which deviate from the SM predictions
by over three standard deviations ($\sigma$) [2]. This discrepancy has
generated considerable interest within the particle physics community to
explore whether this tension is due to the NP. This report presents two recent
LHCb measurements that contribute to this effort: the simultaneous measurement
of $R(D^{(*)+})$ [3] and the analysis of the ${D}^{*+}$ longitudinal
polarisation in the ${{\kern
1.79993pt\overline{\kern-1.79993ptB}}{}^{0}}\rightarrow{{D}^{*+}}{\tau^{-}}{{\overline{\nu}}_{\tau}}$
decay [4]. These analyses benefit from the dedicated design and the excellent
performance of the LHCb detector for reconstructions of $b$-hadron decays.

##  2 Simultaneous measurement of $R({{D}^{*+}})$ and $R({{D}^{+}})$

Using proton-proton ($pp$) collision data corresponding to an integrated
luminosity of $2\text{\,fb}^{-1}$ collected by the LHCb experiment during the
periods 2015-2016, the LFU ratios $R({{D}^{*+}})$ and $R({{D}^{+}})$ are
measured as follows [3]:

$R(D^{(*)+})\equiv{{{\cal B}({{\kern
1.79993pt\overline{\kern-1.79993ptB}}{}^{0}}\rightarrow{D}^{(*)+}{\tau^{-}}{{\overline{\nu}}_{\tau}})}\over{{\cal
B}({{\kern
1.79993pt\overline{\kern-1.79993ptB}}{}^{0}}\rightarrow{D}^{(*)+}{\mu^{-}}{{\overline{\nu}}_{\mu}})}}\,.$
(1)

The tauoinc and muonic decays are denoted as signal and normalisation
channels, respectively. This is the first LHCb analysis measuring $R(D^{(*)})$
with the ${D}^{+}$ meson. The ${D}^{*+}$ meson decays into the ${D}^{+}$ meson
and an unreconstructed neutral pion or photon, followed by the
${{D}^{+}}\rightarrow{{K}^{-}}{{\pi}^{+}}{{\pi}^{+}}$ decay. The $\tau^{-}$
lepton decays into ${\mu^{-}}{{\overline{\nu}}_{\mu}}{{\nu}_{\tau}}$ where
neutrinos escape from detection. Consequently, the only reconstructed final
state in both signal and normalisation channels is ${{D}^{+}}{\mu^{-}}$,
enabling direct and simultaneous determination of $R(D^{(*)+})$ with the same
data sample.

Simulation samples are used to optimise selections, construct template
probability density functions (PDF) and calculate efficiencies. This analysis
marks the first use of the Tracker-Only simulation at LHCb, where only the
detector responses from the tracking system are simulated. The effects of the
particle identification (PID), calorimeter and muon systems are emulated
during the offline analysis. This Tracker-Only configuration speeds up the
simulation process by a factor of eight compared to the full simulation,
allowing for the production of large amounts of simulated events. This
capability is crucial for reducing relevant systematic uncertainties which are
typically one of the main constraints on the precision of $R(D^{(*)})$ results
at LHCb.

The selection of signal candidates starts with triggers that persist the muon
kinematic features to distinguish tauonic from muonic decays, followed by
kinematic, track-quality and PID requirements for final-state particles. The
${D}^{+}$ decay vertex must be significantly displaced from the $pp$ collision
vertex to reduce background directly from $pp$ collisions. Several Boosted
Decision Trees (BDT) [5] are employed to suppress backgrounds from fake
tracks, accidental track combinations forming fake ${D}^{+}$ candidates, and
partially reconstructed decays with missing neutral or charged objects in
addition to ${{D}^{+}}{\mu^{-}}$. The residual fake ${D}^{+}$ contribution is
statistically subtracted using the sPlot technique [6] with
$m({{K}^{-}}{{\pi}^{+}}{{\pi}^{+}})$ as the discriminating variable.

Signal and normalisation yields are determined via a three-dimensional binned-
likelihood fit to the squared four-momentum transferred to the lepton system
($q^{2}$), lepton energy in the ${\kern
1.79993pt\overline{\kern-1.79993ptB}}{}^{0}$ rest frame ($E_{l}^{*}$) and the
invariant-mass of unreconstructed particles ($m_{\rm miss}^{2}$). The PDFs of
the components are parameterised as histogram templates, which are derived
from simulation for $B$ semileptonic and double charm decays, and from data
for backgrounds due to $\mu$ misidentification and random ${{D}^{+}}{\mu^{-}}$
combinations. The form factors for the signal and normalisation decays are
parameterised using the BGL formalism [7], while for semileptonic $B$ decays
involving higher excited $D$ states, the BLR formalism is used [8]. The form-
factor parameters are allowed to vary in the fit. The fit is performed
simultaneously across four data samples, each enriched with specific decays.
The signal sample, comprising ${{D}^{+}}{\mu^{-}}$ candidates, primarily
contains signal and normalisation decays. Three control samples, which
additionally include either one or two charged pions, or a charged kaon,
predominantly consist of feature contributions from feed down of semileptonic
$B$ decays involving higher excited $D$ states and double charm decays with
two charm mesons. The three control samples provide crucial constraints on the
background fractions in the signal sample.

Distributions of the three fit variables in the signal sample are shown in
Fig. 1 with the fit results overlaid. There is a good agreement observed
between the fit results and data distributions. With the efficiencies of the
signal and normalisation channels determined from simulation, the LFU ratios
are measured to be

$R\left(D^{+}\right)=0.249\pm 0.043\pm 0.047,\quad
R\left(D^{*+}\right)=0.402\pm 0.081\pm 0.085,$ (2)

with a correlation factor of $\rho=-0.39$. Here, the first uncertainties are
statistical and the second systematic. The primary sources of systematic
uncertainties are the form-factor parameterisation and background modelling.
The limited size of simulation samples is also a main source contributing to
the uncertainty on $R\left(D^{*+}\right)$. The $R\left(D^{+}\right)$ and
$R\left(D^{*+}\right)$ results shown in Eq. 2 are compatible with the world
average of all relevant measurements and the SM predictions [2]. With these
results included, the updated world average still exhibits a tension with the
SM expectation at a $3\sigma$ level.

Figure 1: Distributions of the three fit variables in the signal sample with
the fit result overlaid.

##  3 Measurement of the ${D}^{*+}$ longitudinal polarisation

In the ${{\kern
1.79993pt\overline{\kern-1.79993ptB}}{}^{0}}\rightarrow{{D}^{*+}}{\tau^{-}}{{\overline{\nu}}_{\tau}}$
decay, the ${D}^{*+}$ longitudinal polarisation factor $F_{L}^{D^{*}}$ is
defined as

$F_{L}^{D^{*}}=\frac{a_{\theta_{D}}(q^{2})+c_{\theta_{D}}(q^{2})}{3a_{\theta_{D}}(q^{2})+c_{\theta_{D}}(q^{2})}.$
(3)

The parameters $a_{\theta_{D}}(q^{2})$ and $c_{\theta_{D}}(q^{2})$ are the
coefficients characterising the sizes of unpolarised and polarised components
in the differential decay width

$\frac{\mathrm{d}^{2}\Gamma}{\mathrm{d}q^{2}\mathrm{d}\cos\theta_{D}}=a_{\theta_{D}}(q^{2})+c_{\theta_{D}}(q^{2})\cos^{2}\theta_{D}.$
(4)

In the case that ${{D}^{*+}}$ decays into ${{D}^{0}}{{\pi}^{+}}$, $\theta_{D}$
is the angle between the ${D}^{0}$ momentum and the opposite of the ${\kern
1.79993pt\overline{\kern-1.79993ptB}}{}^{0}$ momentum in the ${D}^{*+}$ rest
frame. The polarisation factor $F_{L}^{D^{*}}$ is sensitive to NP
configurations and thus can be used to constrain NP parameters [9].

The LHCb experiment performed a measurement of $F_{L}^{D^{*}}$ using $pp$
collision data collected in the periods 2011-2012 and 2015-2016 [4]. The
${\tau^{-}}$ candidates are reconstructed using the hadronic modes
${\tau^{-}}\rightarrow{{\pi}^{-}}{{\pi}^{+}}{{\pi}^{-}}({{\pi}^{0}}){{\nu}_{\tau}}$
and the ${D}^{*+}$ are required to decay as
${{D}^{*+}}\rightarrow{{D}^{0}}{{\pi}^{+}}$ followed by
${{D}^{0}}\rightarrow{{K}^{-}}{{\pi}^{+}}$. The coefficients
$a_{\theta}(q^{2})$ and $c_{\theta}(q^{2})$ are extracted from a four-
dimensional binned template fit to $q^{2}$, $\cos\theta_{D}$, $\tau^{-}$
lifetime $t_{\tau}$ and anti-$D_{s}$ BDT output [5] that distinguishes the
signal decay from the double charm background. The fit projections for the
$\cos\theta_{D}$ distribution are shown in Fig. 2. The polarisation factor
$F_{L}^{{D}^{*}}$ is evaluated in two $q^{2}$ bins and in the full region:

$\displaystyle q^{2}<7\mathrm{GeV}^{2}/c^{4}:$ $\displaystyle 0.51\pm 0.07\pm
0.03,$ (5) $\displaystyle q^{2}>7\mathrm{GeV}^{2}/c^{4}:$ $\displaystyle
0.35\pm 0.08\pm 0.02,$ $\displaystyle q^{2}\text{ integrated}:$ $\displaystyle
0.43\pm 0.06\pm 0.03,$

where the first uncertainty is statistical and the second systematic. The
primary systematic uncertainties stem from the limited size of simulation
samples, parameterisation of the form factor and modelling of double charm
background. The results in Eq. 5 are the most precise to date and are
compatible with the Belle result [10] and the SM prediction [11].

Figure 2: The $\cos\theta_{D}$ distributions in two $q^{2}$ regions for the
(left) Run 1 and (right) Run 2
${{B}^{0}}\rightarrow{{D}^{*-}}{\tau^{+}}{{\nu}_{\tau}}$ data samples with the
fit results overlaid.

##  4 Conclusion and prospects

In summary, this report presents two recent LHCb measurements related to $B$
semileptonic decays. The results of $R({{D}^{+}})$ and $R({{D}^{*+}})$,
obtained from the first LHCb analysis using the
${{D}^{+}}\rightarrow{{K}^{-}}{{\pi}^{+}}{{\pi}^{+}}$ mode to measure
$R(D^{(*)})$, align with the previous world averages and the SM expectations
[2, 3]. The updated world averages with these new results included are
slightly closer to the SM predictions but the tension persists at a $3\sigma$
level. The first LHCb measurement of the ${D}^{*+}$ longitudinal polarisation
in the ${{\kern
1.79993pt\overline{\kern-1.79993ptB}}{}^{0}}\rightarrow{{D}^{*+}}{\tau^{-}}{{\overline{\nu}}_{\tau}}$
decay offers the most precise result to date, which is compatible with the SM
prediction [4, 11]. These measurements contribute to the continuous effort of
testing the SM and searching for NP in semileptonic decays of $b$-hadrons.

There are numerous physics topics related to $B$ semileptonic decays to
explore with ongoing data collections at LHCb. The $R(D)$-$R(D^{*})$ tension
is anticipated to be better understood through more precise measurements using
larger dataset, as uncertainties typically scale with luminosity. Full angular
analyses can be performed in some decays with high statistics, where different
angular coefficients are sensitive to different NP scenarios. Additionally, as
the accessible dataset grows, it becomes feasible to investigate decays with
low statistics, such as those involving excited charm hadrons. Measurements of
these decays can provide new insights for NP searches.

## References

## References

  * [1] A. Crivellin, J. Heeck and P. Stoffer, Phys. Rev. Lett. 116, 081801 (2016).
  * [2] Y. Amhis et al., Phys. Rev. D 107, 052008 (2023).
  * [3] LHCb Collaboration, LHCb-PAPER-2024-007, in preparation.
  * [4] LHCb Collaboration, arXiv:2311.05224.
  * [5] Y. Freund and R. E. Schapire, J. Comput. Syst. Sci. 55, 119 (1997).
  * [6] M. Pivk and F. R. Le Diberder, Nucl. Instrum. Methods A555, 356 (2005).
  * [7] D. Bigi and P. Gambino, Phys. Rev. D 94, 094008 (2016); Fermilab Lattice and MILC Collaborations, Eur. Phys. J. C 82, 1141 (2022).
  * [8] F. U. Bernlochner and Z. Ligeti, Phys. Rev. D 95, 014022 (2017).
  * [9] A. K. Alok, D. Kumar, S. Kumbhakar and S U. Sankar, Phys. Rev. D 95, 115038 (2017).
  * [10] Belle Collaboration, BELLE-CONF-1805, CKM 2018.
  * [11] Z. R Huang, Y. Li, C. D. Lu, M. A. Paracha and C. Wang, Phys. Rev. D 98, 095018 (2018); S. Bhattacharya, S. Nandi and S. K. Patra, Eur. Phys. J. C 79, 268 (2019); D. Bečirević, M. Fedele, I. Nišandžić and A. Tayduganov, arXiv:1907.02257; G. Martinelli, S. Simula and L. Vittorio, Eur. Phys. J. C 84, 400 (2024).


# A Feature-Based Model for Nested Named-Entity Recognition at VLSP-2018 NER
Evaluation Campaign

Pham Quang Nhat Minh
Alt Vietnam Co., Ltd
92 Trieu Viet Vuong, Hai Ba Trung, Hanoi
<EMAIL_ADDRESS>
###### Abstract.

In this report, we describe our participant named-entity recognition system at
VLSP 2018 evaluation campaign. We formalized the task as a sequence labeling
problem using BIO encoding scheme. We applied a feature-based model which
combines word, word-shape features, Brown-cluster-based features, and word-
embedding-based features. We compare several methods to deal with nested
entities in the dataset. We showed that combining tags of entities at all
levels for training a sequence labeling model (joint-tag model) improved the
accuracy of nested named-entity recognition.

##  1 Introduction

Word | Level-1 Tag | Level-2 Tag | Joint Tag
---|---|---|---
ông | O | O | O+O
Ngô_Văn_Quý | B-PER | O | B-PER+O
- | O | O | O+O
Phó | O | O | O+O
Chủ_tịch | O | O | O+O
UBND | O | B-ORG | O+B-ORG
TP | B-LOC | I-ORG | B-LOC+I-ORG
Hà_Nội | I-LOC | I-ORG | I-LOC+I-ORG
Table 1:  Generating joint-tags by combing entity tags at all levels of a
token

Named-entity recognition (NER) is an important task in information extraction.
The task is to identify in a text, spans that are entities and classify them
into pre-defined categories. There have been some conferences and shared tasks
for evaluating NER systems in English and other languages, such as MUC-6
Sundheim (1995), CoNLL 2002 Sang (2002) and CoNLL 2003 Sang and Meulder
(2003).

In Vietnamese language, VLSP 2016 NER evaluation Huyen and Luong (2016) is the
first evaluation campaign that aims to systematically compare NER systems for
Vietnamese language. Similar to CoNLL 2003 shared-task, in VLSP 2016, four
named-entity types were considered: person (PER), organization (ORG), location
(LOC), and miscellaneous entities (MISC). In VLSP 2016, organizers provided
the training/test with gold word segmentation, PoS and chunking tags. While
that setting can help participant teams to reduce effort of data processing
and solely focus on developing NER algorithms, it is not so realistic setting.
In VLSP 2018 NER evaluation, only raw texts with XML tags were provided.
Therefore, we need to choose appropriate Vietnamese NLP tools for
preprocessing steps such as word segmentation, PoS tagging, and chunking.

In the report, we describe our NER system at VLSP 2018 NER evaluation
campaign. We applied a feature-based model which combines word, word-shape
features, Brown-cluster-based features, and word-embedding-based features and
adopted Conditional Random Fields (CRF) Lafferty et al. (2001) for training
and testing.

In the VLSP 2018 NER task, similar as VLSP 2016, there are nested entities the
NER dataset. An entity may contain other entities inside them. We categorize
entities in VLSP 2018 NER dataset into three levels.

  * •

Level-1 entities are entities that do not contain other entities inside them.
For example: ¡ENAMEX TYPE=“LOC”¿Hà Nội¡/ENAMEX¿.

  * •

Level-2 entities are entities contain only level-1 entities inside them. For
example: ¡ENAMEX TYPE=“ORG”¿UBND thành phố ¡ENAMEX TYPE=“LOC”¿Hà
Nội¡/ENAMEX¿¡/ENAMEX¿.

  * •

Level-3 entities are entities that contain at least one level-2 entity and may
contain some level-1 entities. For example ¡ENAMEX TYPE=“ORG”¿Khoa Toán,
¡ENAMEX TYPE=“ORG”¿ĐHQG ¡ENAMEX TYPE=“LOC”¿Hà Nội¡/ENAMEX¿¡/ENAMEX¿¡/ENAMEX¿

In our data statistics, we see that the number of level-3 entities is too
small compared with the number of level-1 and level-2 entities, so we decided
to ignore them in building the model. We just consider level-1 and level-2
entities.

In order to deal with nested named-entities, we investigated two methods. The
first method trains separated models for each level of entities. The second
method trains a single model on the training data in which tags are generated
by combing entity tags of entities of all levels. Table 1 shows an example of
how we combined entity tags at all levels of a token to create join tags.

We showed that combining tags of entities at all levels for training a
sequence labeling model (joint-tag model) improved the accuracy of nested
named-entity recognition.

The rest of the paper is organized as follows. In section 2, we described our
participant NER system. In section 3, we present our evaluation results.
Finally, section 4 gives conclusions about the work.

##  2 System description

We formalize NER task as a sequence labeling problem by using the B-I-O
tagging scheme and we apply a popular sequence labeling model, Conditional
Random Fields to the problem. In this section, first we present how we
preprocess the data and then present features that we used in our model.

###  2.1 Preprocessing

In our NER system, we performed sentence and word segmentation on the data.
For sentence segmentation, we just used a simple regular expression to detect
sentence boundaries that match the pattern: period followed by a space and
upper-case character. Actually, to produce result submissions, we also try not
to perform sentence segmentation.

For word segmentation, we adopted RDRsegmenter Nguyen et al. (2018) which is
the state-of-the-art Vietnamese word segmentation tool. Both training and
development data are the converted into data files in CoNLL 2003 format with
two columns: words and their BIO tags. Due to errors of word segmentation
tool, there may be boundary-conflict problem between entity boundary and word
boundary. In such cases, we decided to tag words as “O” (outside entity).

###  2.2 Features

Basically, features in the proposed NER model are categorized into word, word-
shape features, features based on word representations including word clusters
and word embedding. Note that, we extract unigram and bigram features within
the context surrounding the current token with the window size of $5$. More
specifically, for a feature $F$ of the current word, unigram and bigram
features are as follows.

  * •

unigrams: $F$[-2], $F$[-1], $F$[0], $F$[1], $F$[2]

  * •

bigrams: $F$[-2]$F$[-1], $F$[-1]$F$[0], $F$[0]$F$[1], $F$[1]$F$[2]

####  2.2.1 Word Features

We extract word-identity unigrams and bigrams within the window of size 5. We
use both word surfaces and their lower-case forms. Beside words, we also
extract prefixes and suffixes of surfaces of words within the context of the
current word. In our model, we use prefixes and suffixes of lengths from 1 to
4 characters.

####  2.2.2 Word Shapes

In addition to word identities, we use word shapes to improve prediction
ability, especially for unknown or rare words and reduce data spareness
problem. We used the same word shapes as presented in Minh (2018).

####  2.2.3 Brown cluster-based features

Brown clustering algorithm is a hierarchical clustering algorithm for
assigning words to clusters Brown et al. (1992). Each cluster contains words
which are semantically similar. Output clusters are represented as bit-
strings. Brown-cluster-based features in our NER model include whole bit-
string representations of words and their prefixes of lengths 2, 4, 6, 8, 10,
12, 16, 20. Note that, we only extract unigrams for Brown-cluster-based
features.

In experiments, we used the Brown clustering implementation of Liang Liang
(2005) and applied the tool on the raw text data collected through a
Vietnamese news portal. We performed word clustering on the same preprocessed
text data which were used to generate word embeddings in Le-Hong et al.
(2017). The number of word clusters used in our experiments is 5120.

####  2.2.4 Word embeddings

Word-embedding features have been used for a CRF-based Vietnamese NER model in
Le-Hong et al. (2017). The basic idea is adding unigram features corresponding
to dimensions of word representation vectors.

In the paper, we apply the same word-embedding features as in Le-Hong et al.
(2017). We generated pre-trained word vectors by applying Glove Pennington et
al. (2014) on the same text data used to run Brown clustering. The dimension
of word vectors in 25.

##  3 Evaluation

###  3.1 Data sets

Type | Train | Dev | Test
---|---|---|---
| Level-1 | Level-2 | Level-3 | Level-1 | Level-2 | Level-3 | Level-1 | Level-2 | Level-3
LOC | 8831 | 7 | 0 | 3043 | 2 | 0 | 2525 | 2 | 0
ORG | 3471 | 1655 | 63 | 1203 | 690 | 14 | 1616 | 557 | 22
PER | 6427 | 0 | 0 | 2168 | 0 | 0 | 3518 | 1 | 0
MISC | 805 | 1 | 0 | 179 | 1 | 0 | 296 | 0 | 0
Total | 19534 | 1663 | 63 | 6593 | 694 | 14 | 7955 | 561 | 22
Table 2:  Number of entities of each type in each level in train/dev and test
set

Table 2 showed the data statistics on training set, development set, and
official test set. The number of organization entities (ORG) at level 3 is too
small, so we only consider level-1 and level-2 entities in training and
evaluation. Level-2 entities are almost of ORG types.

###  3.2 Evaluation Measures

We used Precision, Recall, F1 score as evaluation measures. Note that, due to
the fact that word segmentation may cause boundary conflict between entities
and words, we convert words in the data into syllables before we evaluate
Precision, Recall, F1 scores.

We consider four entity types: LOCATION, MISCELLANEOUS, ORGANIZATION, and
PERSON in evaluation, and use the evaluation script of CoNLL-2013 for
evaluation.

###  3.3 NER models

For evaluation on the development set, we train three NER models as follows on
the training data of VLSP 2018 NER task.

  * •

Level-1 model is trained by using level-1 entity tags.

  * •

Level-2 model is trained by using level-2 entity tags.

  * •

Joint model is trained using joint tags which combine level-1 and level-2 tags
of each word.

###  3.4 Results

Model | Precision | Recall | F1
---|---|---|---
Level-1 Model | 91.04 | 84.41 | 87.6
Joint Model | 90.42 | 84.72 | 87.47
Table 3:  Evaluation results on dev set of recognizing level-1 entities Method | Precision | Recall | F1
---|---|---|---
Level-2 | 85.81 | 72.44 | 78.56
Joint Model | 84.36 | 77.06 | 80.54
Table 4:  Evaluation results on dev set of recognizing level-2 entities

Table 3 and Table 4 shows the evaluation results on development set of
recognizing level-1 and level-2 entities, respectively. The level-1 model
obtained slightly better F1 score than joint model in recognizing level-1
entities while joint model outperformed level-2 model in recognizing level-2
entities. We also see that the level-2 model got higher precision than joint
model but much lower recall than joint model. A plausible explanation for that
phenomena is that information of level-1 tags helps to recognize more level-2
entities.

###  3.5 Result Submissions

We trained models on the data set obtained by combining provided training and
development data and used the trained models for recognizing entities on the
test set.

In order to produce submitted results, we use methods as follows.

  * •

Using level-1 and level-2 model for recognizing level-1 and level-2 entities,
respectively. We refer this method as Separated method.

  * •

We use joint model to recognize joint tags for each word of a sentence, then
split joint tags into level-1 and level-2 tags. We refer this method as Joint
method.

  * •

We use the joint model for recognizing level-2 entities and level-1 model for
recognizing level-1 entities. We refer this method as Hybrid method.

In recognition, there are some cases that predicted level-1 entities contains
level-2 entities inside them. In such cases, we omit predicted level-2
entities inside predicted level-1 entities. The reason is that accuracy of
level-1 entity recognition on dev set is much higher than the accuracy of
level-2 entity recognition.

We submitted six runs at VLSP 2018 NER evaluation campaign as showed in Table
5. We try two preprocessing approaches: with sentence segmentation and without
sentence segmentation. The reason why we try those preprocessing approaches is
that we would like to know the influence of sequence lengths on the accuracy
of our model.

Runs | Method | Sent Segmentation
---|---|---
Run-1 | Hybrid | YES
Run-2 | Hybrid | NO
Run-3 | Joint | YES
Run-4 | Joint | NO
Run-5 | Separated | YES
Run-6 | Separated | NO
Table 5:  Six submitted runs Run | Precision | Recall | F1
---|---|---|---
Run-1 | 76.08 | 70.68 | 73.28
Run-2 | 76.75 | 70.37 | 73.42
Run-3 | 76.32 | 70.25 | 73.16
Run-4 | 76.16 | 70.98 | 73.48
Run-5 | 75.70 | 70.28 | 72.89
Run-6 | 76.26 | 69.90 | 72.94
Table 6:  Official evaluation results on test set, which consider entities at all levels Category | Precision | Recall | F1
---|---|---|---
PER | 79.30 | 79.68 | 79.49
LOC | 79.21 | 79.69 | 79.45
ORG | 66.83 | 60.17 | 63.33
MISC | 51.40 | 25.00 | 33.64
All | 76.16 | 70.98 | 73.48
Table 7:  Evaluation results of Run-4 on test set for each entity category

Table 6 shows the official evaluation results for our six submitted runs. As
indicated in the table, run 4 which uses Joint model obtained the highest F1
score among six runs. Using Joint model or Hybrid model obtained better F1
scores than using Separated methods. We also see that the difference between a
system that performs sentence segmentation and a system that does not perform
sentence segmentation is very small.

Table 7 shows the Precision, Recall, F1 scores for each entity category of run
4.

Run | Precision | Recall | F1
---|---|---|---
Run-1 | 73.82 | 79.43 | 76.52
Run-2 | 73.45 | 80.04 | 76.60
Run-3 | 73.21 | 79.56 | 76.26
Run-4 | 73.95 | 79.33 | 76.55
Run-5 | 73.80 | 79.46 | 76.53
Run-6 | 73.46 | 80.08 | 76.63
Table 8:  Evaluation results on test set for level-1 entities Run | Precision | Recall | F1
---|---|---|---
Run-1 | 43.24 | 82.94 | 56.84
Run-2 | 43.06 | 82.59 | 56.61
Run-3 | 45.20 | 81.41 | 58.12
Run-4 | 44.48 | 82.51 | 57.80
Run-5 | 39.32 | 83.08 | 53.38
Run-6 | 36.83 | 84.15 | 51.24
Table 9:  Evaluation results on test set for level-2 entities

Table 8 and Table 9 showed the evaluation results on test set of six submitted
runs for level-1 and level-2 entities, respectively.

Run-6 (using level-1 and level-2 models separately without sentence
segmentation) obtained the best accuracy of recognizing level-1 entities among
submitted runs ($76.63$%) and Run-3 (Joint model, sentence segmentation)
obtained the best accuracy of recognizing level-2 entities ($58.12\%$).

Using joint model obtained better F1 scores of recognizing both levels of
entities than just those of the model trained on solely on level-1 and level-2
entity tags. That result is consistent with the result on the development set.

##  4 Conclusions

We haved presented a feature-based model for Vietnamese named-entity
recognition and evaluation results at VLSP 2018 NER evaluation campaign. We
compared several methods for recognizing nested entities. Experimental results
showed that combining tags of entities at all levels for training a sequence
labeling model improved the accuracy of nested named-entity recognition. As
the future work, we plan to investigate deep learning methods such as BiLSTM-
CNN-CRF Ma and Hovy (2016) for nested named entity recognition.

## References

  * Brown et al. (1992) Peter F. Brown, Peter V. deSouza, Robert L. Mercer, Vincent J. Della Pietra, and Jenifer C. Lai. 1992.  Class-based n-gram models of natural language.  _Comput. Linguist._ , 18(4):467–479.
  * Huyen and Luong (2016) Nguyen Thi Minh Huyen and Vu Xuan Luong. 2016.  Vlsp 2016 shared task: Named entity recognition.  In _Proceedings of Vietnamese Speech and Language Processing (VLSP)_.
  * Lafferty et al. (2001) John Lafferty, Andrew McCallum, and Fernando Pereira. 2001.  Conditional random fields: Probabilistic models for segmenting and labeling sequence data.  In _ICML_ , pages 282–289.
  * Le-Hong et al. (2017) Phuong Le-Hong, Quang Nhat Minh Pham, Thai-Hoang Pham, Tuan-Anh Tran, and Dang-Minh Nguyen. 2017.  An empirical study of discriminative sequence labeling models for vietnamese text processing.  In _Proceedings of the 9th International Conference on Knowledge and Systems Engineering (KSE 2017)_.
  * Liang (2005) Percy Liang. 2005.  _Semi-supervised learning for natural language_.  Ph.D. thesis, Massachusetts Institute of Technology.
  * Ma and Hovy (2016) Xuezhe Ma and Eduard Hovy. 2016.  End-to-end sequence labeling via bi-directional lstm-cnns-crf.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , volume 1, pages 1064–1074.
  * Minh (2018) Pham Quang Nhat Minh. 2018.  A feature-rich vietnamese named-entity recognition model.  _arXiv preprint arXiv:1803.04375_.
  * Nguyen et al. (2018) Dat Quoc Nguyen, Dai Quoc Nguyen, Thanh Vu, Mark Dras, and Mark Johnson. 2018.  A Fast and Accurate Vietnamese Word Segmenter.  In _Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018)_.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  Glove: Global vectors for word representation.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543.
  * Sang (2002) Erik F. Tjong Kim Sang. 2002.  Introduction to the conll-2002 shared task: Language-independent named entity recognition.  _CoRR_ , cs.CL/0209010.
  * Sang and Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the conll-2003 shared task: Language-independent named entity recognition.  In _CoNLL_.
  * Sundheim (1995) Beth Sundheim. 1995.  Overview of results of the muc-6 evaluation.  In _MUC_.