# Interpretable Multimodal Emotion Recognition using Facial Features and
Physiological Signals

Puneet Kumar and Xiaobai Li111Corresponding Author<EMAIL_ADDRESS>
CMVS, University of Oulu, Finland.
{puneet.kumar<EMAIL_ADDRESS>

###### Abstract

This paper aims to demonstrate the importance and feasibility of fusing
multimodal information for emotion recognition. It introduces a multimodal
framework for emotion understanding by fusing the information from visual
facial features and rPPG signals extracted from the input videos. An
interpretability technique based on permutation feature importance analysis
has also been implemented to compute the contributions of rPPG and visual
modalities toward classifying a given input video into a particular emotion
class. The experiments on IEMOCAP dataset demonstrate that the emotion
classification performance improves by combining the complementary information
from multiple modalities.

Keywords: Affective Computing, Interpretable & Deployable AI, Multimodal
Analysis, rPPG, Facial Features.

##  1 Introduction

Emotions, characterized by a rich and complex mix of physiological and
cognitive states, hold significant importance across multiple fields such as
psychology, human-computer interaction, affective computing, and even
extending to broader domains such as virtual reality, user experience design,
healthcare, and education [1]. Understanding and accurately interpreting
emotions is essential in human communication and social interactions [2]. With
the surge in the development and accessibility of multimodal sensing
technologies, researchers can explore multiple modalities to enhance the
accuracy and robustness of emotion recognition systems [3]. The current
research trend focuses on building Artificial Intelligence (AI) systems that
can be deployed for real-life applications [4].

Two such modalities, facial expressions and physiological signals, have
garnered significant attention due to the rich information they offer and
their non-invasive nature [5]. Facial expressions, direct and non-invasive
indicators of emotion, have been thoroughly investigated [6]. Various
techniques involving the extraction of facial landmarks, local descriptors, or
holistic representations have been proposed to capture nuanced variations in
facial muscle movements that reflect different emotional states [7].
Physiological signals, such as remote photoplethysmography (rPPG) signals,
provide another layer of emotional cues. These signals, obtained through non-
contact video-based techniques, offer insights into physiological changes
associated with emotional responses [5]. The interplay of these two modalities
offers a more holistic understanding of emotions, thus enhancing the
robustness of emotion recognition systems [8].

Emotion classification through audio-visual information is a well-established
research task [9, 10, 11]. However, recognizing emotion using the
physiological context along with the audio-visual information score for
further exploration [5]. Furthermore, despite the significant advancements,
many multimodal emotion recognition models do not provide meaningful
interpretations for their predictions [12, 13]. Most existing interpretability
techniques have been implemented for visual modality and have yet to be fully
explored for multimodal analysis [14, 15, 6].

This paper proposes an interpretable multimodal emotion recognition framework
that extracts rPPG signals and facial features from the input videos and uses
their combined context for emotion detection. The Haar cascades classifier
[16] has been implemented to extract the rPPG signals, whereas a pre-trained
ResNet-34-based network extracts the visual features. Further, early and late
fusion approaches that integrate the static facial expression features and
dynamic rPPG signals to capture both spatial and temporal aspects of emotions
have been incorporated.

An interpretability technique based on permutation feature importance (PFI)
[17] has also been incorporated that computes the contribution of rPPG and
visual modality towards classifying a given input video into a particular
emotion class. The experiments performed on Interactive Emotional Dyadic
Motion Capture (IEMOCAP) dataset [18] have resulted in an accuracy of 54.61%
while classifying the input videos into ten emotion classes (‘neutral,’
‘happy,’ ‘sad,’ ‘angry,’ ‘excited,’ ‘frustrated,’ ‘fearful,’ ‘surprised,’
‘distressed’ and ‘other’). The increased performance on using the multimodal
context than the individual accuracies on using rPPG or visual modality alone
advocates the importance of leveraging the multimodal context for emotion
understanding. The average contributions of rPPG and visual modalities towards
emotion recognition have been computed as 37.67% and 62.33%, respectively.

The contributions of this paper can be summarized as follows:

  * •

A multimodal emotion recognition framework has been proposed to classify a
given video into discrete emotion classes. It extracts the dynamic rPPG
signals from the input videos and combines them with static facial expressions
using early and late fusion approaches.

  * •

An interpretability technique has been incorporated that computes the
contribution of rPPG and visual modalities towards emotion classification
using the PFI algorithm.

  * •

Extensive experiments have been performed on the IEMOCAP dataset, and the
results have been presented in terms of accuracy, precision, recall, F1 score,
and modality-wise contributions toward emotion classification.

##  2 Proposed Method

The proposed framework has been diagrammatically depicted in Figure 1 and
described in the following sections.

Figure 1: Schematic illustration of the proposed framework.

###  2.1 Preprocessing and Feature Extraction

The video files are loaded and processed frame by frame using OpenCV (cv2)
library 222https://opencv.org/ and processed to extract rPPG signals and
facial features.

i) rPPG Signals Extraction: Face detection within each video frame during the
rPPG signal extraction process is accomplished using Haar cascades [16]. The
region of interest (ROI), predominantly the facial region, is isolated from
each frame, after which the mean intensity is computed to generate the rPPG
signal for each video. The calculation of the mean intensity within the ROI
($\bar{I}c$) is represented in Eq. 1.

$\bar{I}c=\frac{1}{N}\sum_{x=1}^{W}\sum_{y=1}^{H}I_{x,y,c}$ (1)

Where $I_{x,y,c}$ is the intensity of the pixel at location $(x,y)$ for color
channel $c$ in the ROI, and $N$ is the total number of pixels in the ROI,
whereas $W$ and $H$ represent the width and height of the ROI, respectively,
and $c\in{R,G,B}$.

ii) Facial Features Extraction: Facial feature extraction employs Dlib’s shape
predictor [19], which is a version of the ResNet-34 trained on Face Scrub
dataset[20] to identify the facial landmarks in a given image of a face. As
per Eq. 2, it identifies 68 facial landmarks for each detected face within
every frame, distinguishing unique facial characteristics.

$\begin{split}P&=D(F,\\{L_{i}\\})\\\ F&=[f_{1},f_{2},\ldots,f_{n}]\end{split}$
(2)

Where $F$ represents the face detected in a frame, $P$ represents the
predicted points on the face, $D(F,\\{L_{i}\\})$ is the function for
predicting points on the face, and $L_{i}$ is the set of landmark points for
the $i^{th}$ point. As signals from different videos might differ in length,
it becomes crucial to standardize the input for the neural network model. This
standardization is achieved by zero-padding $\bar{I}$ and $P$ to match the
maximum signal length.

###  2.2 Multimodal Feature Fusion

Early fusion and late fusion approaches are used to combine the rPPG signals
and facial features.

i) Early Fusion: In the early fusion approach, the rPPG signals and facial
features are concatenated before being fed into the model. The fused data are
then passed through a neural network comprising a flatten layer, followed by
CNN layers of dimensions 512 and 256, and the final layer of size equal to the
number of classes. The flatten layer transforms the 3D input tensor into a 1D
tensor, and the subsequent CNN layers functions perform the classification
task. The model structure is represented as per Eq. 3.

$\displaystyle I^{\prime}$ $\displaystyle=\text{concatenate}(\bar{I}c,P)$ (3)
$\displaystyle I^{\prime\prime}$ $\displaystyle=\text{flatten}(I^{\prime})$
$\displaystyle F_{early}$ $\displaystyle=\text{NNet}(I^{\prime\prime},C)$

Where $I$ is the input shape, $C$ denotes the number of classes, $\bar{I}c$ is
the mean intensity within the ROI from the rPPG signals, $P$ represents the
facial features, $NNet$ represents the early fusion network and $F_{early}$ is
the output of the early fusion.

ii) Late Fusion: In the late fusion approach, the rPPG and visual models are
trained separately, and their outputs are combined using a weighted average.
Eq. 4 represents a late fusion approach where the models are trained
separately, and their outputs are combined in the final output $F_{late}$.

$\begin{split}F_{late}&=w_{1}\cdot M_{\text{rPPG}}(\bar{I}c)+w_{2}\cdot
M_{\text{facial}}(P)\end{split}$ (4)

Where $M_{\text{rPPG}}(\bar{I}c)$ and $M_{\text{facial}}(P)$ represent the
outputs of the rPPG model and the visual model, respectively, and $w_{1}$ and
$w_{2}$ are the weights assigned to each model’s output in the final fusion.

###  2.3 Emotion Classification

This study employs three separate models for emotion classification. Two of
these models operate independently, utilizing rPPG signals and facial
features. The third model operates via ‘early fusion,’ exploiting the combined
context of data from the rPPG and visual models. The outputs of these
individual models are then collaboratively integrated through a ‘late fusion’
approach that uses a weighted addition technique. The individual models, based
on rPPG signals and facial features, are constructed as follows.

i) rPPG Model: This model utilizes a Deep Convolutional Neural Network (CNN)
with two hidden layers. It incorporates Rectified Linear Unit (ReLU)
activation functions for emotion classification derived from rPPG signals.

ii) Visual Model: This model, built on facial features, employs a ResNet-based
Deep CNN with two hidden layers and ReLU activation functions.

###  2.4 Interpretability

An explainability method based on permutation feature importance (PFI) [17] is
implemented, which is used to estimate the importance of features by permuting
the values of each feature and measuring the resulting impact on model
performance. The PFI of feature $j$ is the decrease in the model score when
values of feature $j$ are randomly permuted. PFI for a feature $j$ is the
difference in the model score when the values of feature $j$ are randomly
permuted. Eq. 5 mathematically represents the concept of permutation feature
importance.

$PFI(j)=E_{\pi}[f(X^{(i)})]-E_{\pi}[f(X^{(i)}_{\pi_{j}})]$ (5)

Where $PFI(j)$ is the permutation feature importance of feature $j$,
$E_{\pi}[f(X^{(i)})]$ is the expected value of the model score over all
samples in the dataset when the model is scored normally,
$E_{\pi}[f(X^{(i)}_{\pi_{j}})]$ is the expected value of the model score when
the values of feature $j$ are permuted according to some permutation $\pi$,
and $X^{(i)}_{\pi_{j}}$ denotes the dataset $X^{(i)}$ with the values of
feature $j$ permuted according to $\pi$.

##  3 Results and Discussion

###  3.1 Experimental Setup

The emotion classification experiments have been performed on the IEMOCAP
dataset [18] consisting of 10,039 videos labeled with ten discrete emotion
labels (‘neutral,’‘ happy,’ ‘sad,’‘ angry,’ ‘excited,’ ‘frustrated,’
‘fearful,’ ‘surprised,’ ‘distressed’ and‘other’). The model training has been
trained on NVIDIA RTX 4090 GPU for 50 epochs with a batch size of 32 and a
learning rate of 0.001. The performance has been evaluated using accuracy,
precision, recall, and F1 score metrics.

###  3.2 Results

Table 1 summarizes the accuracy of the individual and fusion models, whereas
the average contributions of rPPG and visual modalities towards emotion
recognition in the early fusion setup are presented in Table 2. The proposed
framework has demonstrated an emotion classification accuracy of 54.61%, and
the average contributions of rPPG and visual modalities towards emotion
recognition have been computed as 37.67% and 62.33%, respectively.

Table 1: Detailed performance of the individual and fusion models. Model | Accuracy | Precision | Recall | F1 Score
---|---|---|---|---
rPPG | 37.45% | 0.37 | 0.38 | 0.38
Facial Features | 46.42% | 0.49 | 0.49 | 0.49
Late Fusion | 41.17% | 0.43 | 0.42 | 0.42
Early Fusion | 54.61% | 0.56 | 0.58 | 0.57
Table 2: Average contribution of each modality towards emotion recognition. Modality | Contribution
---|---
rPPG | 37.67%
Visual | 62.33%

Table 1 shows that both the individual models performed reasonably well.
However, the fusion model outperformed the individual models, demonstrating
the advantage of combining rPPG signals and facial feature information for
emotion recognition.

###  3.3 Discussion

This paper presents a compelling case for including multimodal context in
emotion recognition. While the models trained on individual modalities show
moderate performance, their fusion significantly improves emotion recognition
accuracy. It emphasizes the complementarity of these modalities in capturing
emotional states. However, the late fusion of modalities underperforms
compared to the early fusion approach, indicating that integrating modalities
at an earlier stage allows for more effective learning of emotional states.

However, this study has a few limitations of the proposed work. The IEMOCAP
dataset, while widely used, may limit the generalizability of the findings.
Cross-dataset experiments on larger and more diverse datasets could further
strengthen the results. Moreover, more modalities such as audio, text, and
other physiological signals can also be incorporated for emotion recognition.
Finally, a more in-depth interpretability mechanism can be developed to
explain the role of individual features in emotion detection.

##  4 Conclusion

This work presents a multimodal emotion recognition framework using rPPG
signals and facial features. It paves the way for practical applications where
transparent and interpretable emotion understanding is important. The results
highlight the benefits of integrating multiple modalities for emotion
recognition, with an early fusion approach yielding the highest accuracy.
While there are limitations and potential improvements, our study provides a
promising direction for future research in emotion recognition, emphasizing
the importance of multimodal data and fusion techniques.

## References

  * [1] Soujanya Poria, Erik Cambria, Rajiv Bajpai, and Amir Hussain.  A Review of Affective Computing: From Unimodal Analysis to Multimodal Fusion.  Elsevier Information Fusion Journal, 37:98–125, 2017.
  * [2] Yucel Cimtay, Erhan Ekmekcioglu, and Seyma Caglar-Ozhan.  Cross Subject Multimodal Emotion Recognition Based on Hybrid Fusion.  IEEE Access, 8:168865–168878, 2020.
  * [3] Tadas Baltrušaitis, Chaitanya Ahuja, and Louis Morency.  Multimodal Machine Learning: A Survey and Taxonomy.  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 41(2):423–443, 2018.
  * [4] Andrei Paleyes, Raoul-Gabriel Urma, and Neil D Lawrence.  Challenges in Deploying Machine Learning: A Survey of Case Studies.  ACM Computing Surveys, 55(6):1–29, 2022.
  * [5] Zitong Yu, Xiaobai Li, and Guoying Zhao.  Facial Video-based Physiological Signal Measurement: Recent Advances and Affective Applications.  Signal Processing Magazine, 38(6):50–58, 2021.
  * [6] Sarthak Malik, Puneet Kumar, and Balasubramanian Raman.  Towards Interpretable Facial Emotion Recognition.  In The 12th Indian Conference on Computer Vision, Graphics and Image Processing, pages 1–9, 2021.
  * [7] Nannan Wang, Xinbo Gao, Dacheng Tao, Heng Yang, and Xuelong Li.  Facial Feature Point Detection: A Comprehensive Survey.  Neurocomputing, 275:50–65, 2018.
  * [8] Zhihong Zeng, Maja Pantic, Glenn I Roisman, and Thomas S Huang.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions.  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 31(1):39–58, 2009.
  * [9] Tianrong Rao, Xiaoxu Li, and Min Xu.  Learning Multi-level Deep Representations for Image Emotion Classification.  Neural Processing Letters, pp. 1–19, 2019.
  * [10] M Xu, F Zhang, and S Khan.  Improve Accuracy of Speech Emotion Recognition with Attention Head Fusion.  In IEEE Annual Computing and Communication Workshop and Conference (CCWC), pages 1058–1064, 2020.
  * [11] Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria.  DialogueRNN: An Attentive RNN for Emotion Detection in Conversations.  In The 31st AAAI Conference on Artificial Intelligence (AAAI), volume 33, pages 6818–6825, 2019.
  * [12] W James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu.  Definitions, Methods, and Applications in Interpretable Machine Learning.  Proceedings of the National Academy of Sciences, 116(44):22071–22080, 2019.
  * [13] Luca Longo et al.  Explainable Artificial Intelligence: Concepts, Applications, Research Challenges and Visions.  In The Springer International Cross-Domain Conference for Machine Learning and Knowledge Extraction (CD-MAKE), pages 1–16, 2020.
  * [14] Marco Tulio Ribeiro, S. Singh, and C. Guestrin.  Why Should I Trust You? Explaining Predictions of Any Classifier.  In International Conference on Knowledge Discovery & Data mining (KDD), pages 1135–1144, 2016.
  * [15] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization.  In The IEEE/CVF International Conference on Computer Vision (ICCV), pages 618–626, 2017.
  * [16] Sander Soo.  Object Detection using Haar Cascade Classifier.  Institute of Computer Science, University of Tartu, 2(3):1–12, 2014\.
  * [17] André Altmann, Laura Toloşi, Oliver Sander, and Thomas Lengauer.  Permutation Importance: A Corrected Feature Importance Measure.  Bioinformatics, 26(10):1340–1347, 2010.
  * [18] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan.  IEMOCAP: Interactive Emotional dyadic MOtion CAPture data.  Language Resources and Evaluation, 42(4), 2008.
  * [19] Davis E. King.  DLIB Models.  https://github.com/davisking/dlib-models, 2016.  Accessed on 21.05.2023.
  * [20] Hong-Wei Ng and Stefan Winkler.  A Data Driven Approach to Cleaning Large Face Datasets.  In IEEE International Conference on Image Processing (ICIP), pages 343–347. IEEE, 2014.


#  VulCurator: A Vulnerability-Fixing Commit Detector

Truong-Giang Nguyen  Singapore Management UniversitySingaporeSingapore
<EMAIL_ADDRESS>,  Thanh Le-Cong  Singapore Management
UniversitySingaporeSingapore<EMAIL_ADDRESS>,  Hong Jin Kang  Singapore
Management UniversitySingaporeSingapore<EMAIL_ADDRESS>,  Xuan-
Bach D. Le  University of MelbourneMelbourneAustralia<EMAIL_ADDRESS>and  David Lo  Singapore Management UniversitySingaporeSingapore
<EMAIL_ADDRESS>
(2022)

###### Abstract.

Open-source software (OSS) vulnerability management process is important
nowadays, as the number of discovered OSS vulnerabilities is increasing over
time. Monitoring vulnerability-fixing commits is a part of the standard
process to prevent vulnerability exploitation. Manually detecting
vulnerability-fixing commits is, however, time-consuming due to the possibly
large number of commits to review. Recently, many techniques have been
proposed to _automatically_ detect vulnerability-fixing commits using machine
learning. These solutions either: (1) did not use deep learning, or (2) use
deep learning on only limited sources of information. This paper proposes
VulCurator, a tool that leverages deep learning on richer sources of
information, including commit messages, code changes and issue reports for
vulnerability-fixing commit classification. Our experimental results show that
VulCurator outperforms the state-of-the-art baselines up to 16.1% in terms of
F1-score.

VulCurator tool is publicly available at
https://github.com/ntgiang71096/VFDetector and
https://zenodo.org/record/7034132#.Yw3MN-xBzDI, with a demo video at
https://youtu.be/uMlFmWSJYOE.

Vulnerability-Fixing Commits, Deep Learning, BERT

††copyright: acmlicensed††price: 15.00††doi:
10.1145/3540250.3558936††journalyear: 2022††submissionid:
fse22demo-p103-p††isbn: 978-1-4503-9413-0/22/11††conference: Proceedings of
the 30th ACM Joint European Software Engineering Conference and Symposium on
the Foundations of Software Engineering; November 14–18, 2022; Singapore,
Singapore††booktitle: Proceedings of the 30th ACM Joint European Software
Engineering Conference and Symposium on the Foundations of Software
Engineering (ESEC/FSE ’22), November 14–18, 2022, Singapore, Singapore††ccs:
Computing methodologies Supervised learning††ccs: Security and privacy
Vulnerability management††ccs: Computing methodologies Supervised learning by
classification

##  1\. Introduction

Open-source software (OSS) vulnerabilities can severely damage systems. An
infamous example is the Equifax Data
Breach111https://nvd.nist.gov/vuln/detail/cve-2017-5638, which led to millions
of cases of identity theft. Another example is
Log4Shell222https://nvd.nist.gov/vuln/detail/CVE-2021-44228 incident, which
led to many vulnerable cloud services and applications. For vulnerability
management, the information of vulnerabilities are collected in the Common
Vulnerabilities and Exposures (CVE) (Corporation, 1999) or National
Vulnerability Database (NVD) (of Standards and Technology, 1999). OSS users
can use vulnerability information such as vulnerable version(s) of a specific
third-party library or how the vulnerability is fixed to make informed
decisions, e.g., migrating the dependencies to invulnerable versions or
patching their own client code.

Unfortunately, in practice, there is often a delay between the time a
vulnerability is fixed and the time it is publicly disclosed (Sabetta and
Bezzi, 2018), leading to a risk that OSS users are unaware of vulnerabilities
in their applications. Therefore, OSS users would benefit from a tool that
automatically detect security-relevant changes, i.e., vulnerability-fixing
commits, that are not yet disclosed (Sabetta and Bezzi, 2018; Truong-Giang et
al., 2022).

Many existing techniques (Zhou and Sharma, 2017; Zhou et al., 2021b; Sabetta
and Bezzi, 2018; Chen et al., 2020; Truong-Giang et al., 2022; Le et al.,
2021; Sawadogo et al., 2020; Tian et al., 2012) have recently proposed
solutions for automatically identifying vulnerability-fixing commits. Several
approaches (Zhou et al., 2021c; Zhou et al., 2021b; Le et al., 2021; Sawadogo
et al., 2020) use deep learning, but only consider only commit messages and
code changes. Our recent work, HERMES (Truong-Giang et al., 2022), combines
information from commit messages, code changes, and issue reports, however,
uses Support Vector Machine (SVM). In this paper, we introduce VulCurator, a
tool using a deep learning to detect vulnerability-fixing commits based on
commit messages, code changes, and issue reports. Different from previous
works, VulCurator leverages BERT-based models to represent both text-based and
code-based information of a commit. Specifically, we use two RoBERTa (Liu et
al., 2019) models for commit messages and issue reports respectively, and a
CodeBERT (Feng et al., 2020) model for code changes. The output probabilities
from the aforementioned classifiers are aggregated using a stacking ensemble
to form the final output probability. Based on the output probability,
VulCurator provides a list of commits ranked by their likelihood of being
vulnerability-fixing commits.

To evaluate the performance of VulCurator, we conduct an empirical evaluation
on two benchmarks, including the SAP dataset proposed by Sabetta et al.
(Sabetta and Bezzi, 2018) and a newly collected dataset of TensorFlow
vulnerabilities. While the former contains 1,132 vulnerability-fixing and
5,995 non-vulnerability-fixing commits written in Java and Python, the latter
contains 290 vulnerability-fixing and 1,535 non-vulnerability-fixing commits
from TensorFlow (Abadi et al., 2016), a well-known deep learning framework. We
compare VulCurator with two recently proposed approaches, HERMES (Truong-Giang
et al., 2022), which uses Support Vector Machine classifiers using information
from commit messages, code changes and issue reports, and VulFixMiner (Zhou et
al., 2021b), a deep learning model classifying code changes from commits. Our
experiments show that VulCurator outperforms HERMES by 16.1% and 8.5% on the
SAP and TensorFlow dataset respectively, and VulCurator improves over
VulFixMiner by 3.9% and 4.7%.

##  2\. Background and Related Work

Vulnerability-fixing commit classification. Vulnerability-fixing commit
classification has been an active and challenging topic in software
engineering research. Zhou et al. (Zhou and Sharma, 2017) use word2vec
(Mikolov et al., 2013) to represent commit messages and forward it to a K-fold
stacking model for classification. Zhou et al. (Zhou et al., 2021b) fine-tuned
CodeBERT to transform code changes into embedding vectors and then use one-
layer neural network to classify commits. Sabetta et al. (Sabetta and Bezzi,
2018) and Zhou et al. (Zhou et al., 2021c) proposed to train message
classifier and code change classifier separately before combining them for
commit classification. The former approach uses Support Vector Machine, while
the latter uses LSTM and multi-layer CNN. Nguyen et al. recently proposed
HERMES (Truong-Giang et al., 2022), which uses issue reports as a third source
of information using an issue classifier and an issue linker. The issue linker
maps commits without explicitly linked issues to best-matching issues.

BERT-based models. RoBERTa (Liu et al., 2019) is a multi-layer bidirectional
Transformer model, which is trained on a large dataset of natural language.
CodeBERT (Feng et al., 2020), a variant of RoBERTa, is trained on large-scale
dataset consisting of bimodal data points which refer to natural language -
programming language pair, and unimodal data points which refer to only
programming language. Both RoBERTa and CodeBERT have shown to be effective in
various tasks, including vulnerability-fixing classification (Zhou et al.,
2021b; Zhou et al., 2021c), type inference (Kazerounian et al., 2021), program
repair (Mashhadi and Hemmati, 2021), program analysis (Le-Cong et al., 2022)
or defect prediction (Zhou et al., 2021a).

##  3\. VulCurator Architecture

Figure 1 provides an overview of VulCurator. Our tool takes as input a JSON
file ① containing a list of commits with their messages, code changes and
linked issues. Note that VulCurator allows commits without explicitly linked
issues. In these cases, VulCurator leverages an issue linker ②, which is built
based on an issue corpus ③ for mapping each commit to the most relevant issue
in the corpus. Then, VulCurator feeds each type of commit information to their
the corresponding classifiers, i.e. message classifier ④, patch classifier ⑤,
or issue classifier ⑥. Each classifier produces a probability indicating the
likelihood of a commit being a vulnerability-fixing commit. Then, the
predicted probabilities from three classifiers are combined using stacking
ensemble ⑦ to form the final probability.

Figure 1. Overview of VulCurator

Issue Linker. VulCurator first recovers commit-issue link for every commit
without any corresponding issues as only a fraction of commits are explicitly
linked to issue reports (Sun et al., 2017). Particularly, similar from HERMES
(Truong-Giang et al., 2022), VulCurator uses FRLink (Sun et al., 2017) to map
each commit without any corresponding issues to its most similar issue in the
input data based on a pre-defined similarity function. The similarity function
is calculated with respect to the Term Frequency-Inverse Document Frequency
(TF-IDF) of natural language terms and code terms in commit message, code
changes and issue content. The TF-IDF value of every word is calculated once
using TfidfVectorizer333https://scikit-
learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html
and stored locally using pickle444https://scikit-
learn.org/stable/model_persistence.html for the model inference phase. From
the findings of prior work (Truong-Giang et al., 2022), the accuracy of
commit-issue linking affects the classification performance. By limiting the
issue linker’s similarity threshold, only accurate links will be recovered.

Patch Classifier. We use the same approach as VulFixMiner (Zhou et al., 2021b)
for the patch classifier of VulCurator.
CodeBERT555https://huggingface.co/microsoft/codebert-base is used as the core
model. For code changes of each file, the added code and removed code version
of code changes are extracted separately. The codes are tokenized using
CodeBERT Tokenizer, and then formed as input for CodeBERT following the format
below:

(1) $[CLS]\langle\text{rem-code}\rangle[SEP]\langle\text{added-
code}\rangle[EOS]$

where $\mathit{rem-code}$ and $\mathit{add-code}$ are the sequence of tokens
of the removed code and added code, respectively; [CLS], [SEP], [EOS] are
special tokens given by CodeBERT, denoting the classification, separation and
end of sequence token, respectively. The input will be forwarded to the
CodeBERT to obtain an embedding vector, i.e. vector of numerical numbers,
representing the semantic of code changes of each file. Finally, the embedding
vectors are forwarded by an aggregator followed by a neural classifier to
output the final probability for each commit.

Message Classifier. The message classifier leverages the multi-layer
bidirectional Transformer model, RoBERTa (Liu et al., 2019). Specifically, a
commit message is tokenized into tokens using RobertaTokenizer and then
forwarded into the base version of the Roberta
model666https://huggingface.co/roberta-base and a softmax function to obtain
the output probability.

Issue Classifier. Similar to the message classifier, the issue classifier also
uses the base version of RoBERTa model. The model takes the commit issue’s
title and body as inputs, and outputs the predicted probability that the
commit corresponding to the issue is for vulnerability-fixing.

Stacking Ensemble and Output Prediction. Given the output probabilities from
the three aforementioned classifiers, VulCurator leverages a logistic
regression model which acts as a stacking ensemble classifier to produce the
final probability for each commit. Commits with a final probability larger
than a threshold will be deemed as vulnerability-fixing commits. By default,
the classification threshold is set as 0.5 but VulCurator allows users to
adjust the threshold (see details in Section 4).

##  4\. Usage

###  4.1. Installation

User can either clone our GitHub (Nguyen-Truong et al., 2022b) repository and
install required dependencies or use our Docker image to run VulCurator
(Nguyen-Truong et al., 2022a). For full customization of VulCurator, a user
can follow the following steps.

###  4.2. Preparation

VulCurator contains a built-in Issue Linker and pre-trained Classifiers, which
users can directly use. However, users can build their own Issue Linker and
Classifiers following instructions below.

Issue Linker. User can customize issue corpus by providing a folder that
contains files that store issue reports followed our pre-defined format (see
details in our GitHub repository (Nguyen-Truong et al., 2022b)), where each
issue contains issue title, issue body, and issue comments (optional). Given
the corpus, users can use their own Issue Linker by using the following
commands:

python linker_builder.py --corpus_path <corpus_path>

VulCurator models. Users can also train new classifiers for VulCurator with
their own dataset by using the following command:

python model_builder.py --data_path <path_to_data>

Note that the training dataset must follow a pre-defined format, which is
provided on our GitHub repository (Nguyen-Truong et al., 2022b).

###  4.3. Inference

VulCurator provides command line interface with two modes for end-users:
prediction and ranking.

Input format. To use VulCurator, users need to prepare data following our pre-
defined json format as below:

⬇

1[

2 {

3 {

4 "id": <commit_id>,

5 "message": <commit_message>,

6 "issue": {

7 "title": <issue_title>,

8 "body": <issue_body>,

9 "comments" : [<list_of_comments]

10 },

11 "patch": [list_of_code_change]

12 },

13 ...

Prediction mode. In prediction mode, given the input of a dataset of commits,
VulCurator returns a list of likely vulnerability fixing commits along with
the confidence scores. Although VulCurator sets the classification threshold
at 0.5 by default, VulCurator allows the threshold to be adjusted with the
option \--threshold. Users can use the following command to obtain the
results:

$\begin{array}[]{lll}\texttt{python
application.py}&\texttt{--mode}&\texttt{prediction}\\\
&\texttt{--input}&\texttt{<input\\_path>}\\\
&\texttt{--threshold}&\texttt{<threshold>}\\\
&\texttt{--output}&\texttt{<output\\_path>}\\\ \end{array}$

Ranking mode. In ranking mode, users can input data following our format and
VulCurator will output a list of commits sorted by the probability that the
commits is vulnerability-fixing. Users can use the following commands:

$\begin{array}[]{lll}\texttt{python application.py
}&\texttt{--mode}&\texttt{ranking}\\\
&\texttt{--input}&\texttt{<input\\_path>}\\\
&\texttt{--output}&\texttt{<output\\_path>}\\\ \end{array}$

##  5\. Performance Evaluation

In this section, we investigate the following research questions:

  * •

RQ1. How effective is VulCurator?

  * •

RQ2. How much does each classifier contribute?

###  5.1. Experimental Setting

####  5.1.1. Dataset

We empirically evaluate VulCurator using two datasets, the SAP dataset
proposed by Sabetta et al. (Sabetta and Bezzi, 2018) and a newly prepared
TensorFlow dataset. For each dataset, we use 80% data for training and the
remaining 20% for testing.

SAP dataset: We evaluate our tool on the SAP dataset, which is widely used
(Sabetta and Bezzi, 2018; Truong-Giang et al., 2022). The dataset contains
vulnerability-fixing commits of widely used open-source projects manually-
curated by SAP Security Research over a period of four years. Non-
vulnerability-fixing commits are randomly sampled with a ratio of five non-
vulnerability-fixing commits for one vulnerability-fixing commit from the same
project. In total, the dataset contains 1,132 vulnerability-fixing and 5,995
non-vulnerability-fixing commits, in which, 37% of the commits are explicitly
linked to issues.

TensorFlow dataset: We introduce a new dataset with commits from TensorFlow,
which is a well-known deep learning library. The purpose of the dataset is
two-fold. First, with the increase of vulnerabilities in deep learning
libraries in recent years, we would like to investigate whether VulCurator is
also applicable in this domain. Second, we wish to avoid overfitting our
experiments and tool design to the SAP dataset. To construct the dataset, we
collect all vulnerability-fixing commits of TensorFlow, which are listed on
National Vulnerability Database (NVD) (of Standards and Technology, 1999) up
until May 2022. We randomly sampled non-vulnerability-fixing commits from
TensorFlow’s repository using the same setting as Nguyen et al. (Truong-Giang
et al., 2022) and Sabetta et al. (Sabetta and Bezzi, 2018). As a result, our
dataset contains 290 vulnerability-fixing and 1,535 non-vulnerability-fixing
commits. In this dataset, no commit is explicitly linked to an issue.

####  5.1.2. Evaluation metrics

Similar to prior studies (Tian et al., 2012; Zhou et al., 2021c; Truong-Giang
et al., 2022; Chen et al., 2020), both precision and recall are important.
Therefore, we use F1-score, which is the harmonic mean of precision and
recall, to evaluate the effectiveness of VulCurator and HERMES.

In our task, a true positive (TP) is a vulnerability-fixing commit that is
correctly detected. A false positive (FP) is a non-vulnerability-fixing commit
that is incorrectly detected as vulnerability-fixing. A false negative (FN) is
a vulnerability-fixing commit that is not detected. Precision (P) and Recall
(R) are computed as follows:

$\text{P}=\frac{\text{ TP }}{\text{TP}+\text{FP}}$ $\text{R}=\frac{\text{ TP
}}{\text{TP}+\text{FN}}$

Then, the F1 score is calculated as follows:

$F1=\frac{2(P\times R)}{P+R}\\\ $

###  5.2. Experimental Result

Table 1. F1 score of VulCurator and HERMES on SAP dataset. The number with the asterisk(*) denotes the result of VulFixMiner Model | Message | Issue | Patch | Ensemble
---|---|---|---|---
HERMES | 0.67 | 0.51 | 0.60 | 0.68
VulCurator | 0.76 | 0.65 | 0.76* | 0.79
Table 2. F1 score of VulCurator and HERMES on TensorFlow dataset. The number with the asterisk(*) denotes the result of VulFixMiner Model | Message | Issue | Patch | Ensemble
---|---|---|---|---
HERMES | 0.87 | 0.75 | 0.69 | 0.82
VulCurator | 0.81 | 0.80 | 0.85* | 0.89

(a) SAP dataset

(b) TensorFlow dataset

Figure 2. Relationship between true positive cases predicted by three base
classifiers of VulCurator

####  5.2.1. RQ1: Effectiveness

To answer this question, we train and test both VulCurator and HERMES on the
two datasets. The experimental results are shown in Tables 1 and 2. On the SAP
dataset, all VulCurator’s base models and the whole model outperform HERMES’s.
Specifically, VulCurator’s message, issue, patch classifiers and the whole
model improve HERMES’s counterparts by 13.4%, 27.4%, 26.7%, and 16.1% in terms
of F1, respectively. On the TensorFlow dataset, while VulCurator’s message
classifier has a decrease of 6.9% in message classifier compared to HERMES,
VulCurator issue classifier and patch classifier improves over HERMES by 6.7%
and 23.2% respectively, leading to an overall 8.5% improvement over HERMES.
The experiment results suggest that VulCurator benefits from the use of pre-
trained deep learning models.

The patch classifier of VulCurator uses the same model as VulFixMiner (Zhou et
al., 2021b). The improvement in F1 of the ensemble model over the patch
classifier alone (from 0.76 to 0.79 on SAP dataset and 0.85 to 0.89 on
TensorFlow dataset) shows that combining multiple sources of information
allows VulCurator to outperform VulFixMiner (Zhou et al., 2021b). This result
also validates the finding of Nguyen et al. (Truong-Giang et al., 2022) that
using information from the issue tracker boosts classification performance.

####  5.2.2. RQ2: Ablation Study

We investigate if different sources of information capture different aspects
of a commit. On the SAP dataset (Figure. 2(a)), out of 221 discovered
vulnerability-fixing commits, there are 20, 15, and 16 commits that can only
be exposed by message classifier, issue classifier, patch classifier,
respectively. The similar finding is also found in TensorFlow (Figure. 2(b)).
The experimental results show that each classifier helps detect unique
vulnerability-fixing commits.

##  6\. Conclusion and Future Work

We present VulCurator, a tool for detecting vulnerability-fixing commits.
VulCurator combines multiple sources of information such as commit messages,
code changes, and issue reports in a deep learning model. In the future, to
better support security researchers in monitoring commits, we plan to apply
explainable AI techniques (Ribeiro et al., 2016; Pornprasit et al., 2021) to
provide explanations for each prediction.

## Acknowledgment

This project is supported by the National Research Foundation, Singapore and
National University of Singapore through its National Satellite of Excellence
in Trustworthy Software Systems (NSOE-TSS) office under the Trustworthy
Computing for Secure Smart Nation Grant (TCSSNG) award no. NSOE-TSS2020-02.
Any opinions, findings and conclusions or recommendations expressed in this
material are those of the author(s) and do not reflect the views of National
Research Foundation, Singapore and National University of Singapore (including
its National Satellite of Excellence in Trustworthy Software Systems (NSOE-
TSS) office).

Xuan-Bach D. Le is supported by the Australian Government through the
Australian Research Council’s Discovery Early Career Researcher Award, project
number DE220101057

## References

  * (1)
  * Abadi et al. (2016) Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016\.  TensorFlow: a system for Large-Scale machine learning. In _12th USENIX symposium on operating systems design and implementation (OSDI 16)_. 265–283.
  * Chen et al. (2020) Yang Chen, Andrew E Santosa, Ang Ming Yi, Abhishek Sharma, Asankhaya Sharma, and David Lo. 2020\.  A Machine Learning Approach for Vulnerability Curation. In _Proceedings of the 17th International Conference on Mining Software Repositories (MSR)_. 32–42.
  * Corporation (1999) The MITRE Corporation. 1999\.  Common Vulnerabilities and Exposures.  https://cve.mitre.org.
  * Feng et al. (2020) Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020\.  CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_. 1536–1547.
  * Kazerounian et al. (2021) Milod Kazerounian, Jeffrey S Foster, and Bonan Min. 2021\.  SimTyper: sound type inference for Ruby using type equality prediction.  _Proceedings of the ACM on Programming Languages_ 5, OOPSLA (2021), 1–27.
  * Le et al. (2021) Triet HM Le, David Hin, Roland Croft, and M Ali Babar. 2021\.  DeepCVA: Automated Commit-level Vulnerability Assessment with Deep Multi-task Learning.  _arXiv preprint arXiv:2108.08041_ (2021).
  * Le-Cong et al. (2022) Thanh Le-Cong, Kang Hong Jin, Truong Giang Nguyen, Stefanus Agus Haryono, David Lo, Xuan Bach Le Dinh, and Thang Huynh-Quyet. 2022.  AutoPruner: Tranformer-based Call Graph Pruning. In _2022 the 30th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE)_. ACM.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * Mashhadi and Hemmati (2021) Ehsan Mashhadi and Hadi Hemmati. 2021.  Applying codebert for automated program repair of java simple bugs. In _2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)_. IEEE, 505–509.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013\.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_ (2013).
  * Nguyen-Truong et al. (2022a) Giang Nguyen-Truong, Thanh Le-Cong, Hong Jin Kang, Xuan-Bach Dinh Le, and David Lo. 2022a.  VulCurator’s Docker Image.  https://hub.docker.com/r/nguyentruongggiang/vfdetector.
  * Nguyen-Truong et al. (2022b) Giang Nguyen-Truong, Thanh Le-Cong, Hong Jin Kang, Xuan-Bach Dinh Le, and David Lo. 2022b.  VulCurator’s Repository.  https://github.com/ntgiang71096/VFDetector.
  * of Standards and Technology (1999) U.S. National Institute of Standards and Technology. 1999.  National Vulnerability Database.  https://nvd.nist.gov.
  * Pornprasit et al. (2021) Chanathip Pornprasit, Chakkrit Tantithamthavorn, Jirayus Jiarpakdee, Michael Fu, and Patanamon Thongtanunam. 2021.  PyExplainer: Explaining the Predictions of Just-In-Time Defect Models. In _2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 407–418.
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016\.  ” Why should i trust you?” Explaining the predictions of any classifier. In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_. 1135–1144.
  * Sabetta and Bezzi (2018) Antonino Sabetta and Michele Bezzi. 2018.  A practical approach to the automatic classification of security-relevant commits. In _2018 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE, 579–582.
  * Sawadogo et al. (2020) Arthur D Sawadogo, Tegawendé F Bissyandé, Naouel Moha, Kevin Allix, Jacques Klein, Li Li, and Yves Le Traon. 2020\.  Learning to catch security patches.  _arXiv preprint arXiv:2001.09148_ (2020).
  * Sun et al. (2017) Yan Sun, Qing Wang, and Ye Yang. 2017.  Frlink: Improving the recovery of missing issue-commit links by revisiting file relevance.  _Information and Software Technology_ 84 (2017), 33–47.
  * Tian et al. (2012) Yuan Tian, Julia Lawall, and David Lo. 2012.  Identifying linux bug fixing patches. In _2012 34th International Conference on Software Engineering (ICSE)_. IEEE, 386–396.
  * Truong-Giang et al. (2022) Nguyen Truong-Giang, Kang Hong Jin, David Lo, Abhishek Sharma, Andrew Santosa, Asankhaya Sharma, and Ming Yi Ang. 2022.  HERMES: Using Commit-Issue Linking to Detect Vulnerability-Fixing Commits. In _The 2022 29th IEEE International Conference on Software Analysis, Evolution and Reengineering_. IEEE.
  * Zhou et al. (2021b) Jiayuan Zhou, Michael Pacheco, Zhiyuan Wan, Xin Xia, David Lo, Yuan Wang, and Ahmed E Hassan. 2021b.  Finding A Needle in a Haystack: Automated Mining of Silent Vulnerability Fixes.  (2021).
  * Zhou et al. (2021a) Xin Zhou, DongGyun Han, and David Lo. 2021a.  Assessing generalizability of CodeBERT. In _2021 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE, 425–436.
  * Zhou and Sharma (2017) Yaqin Zhou and Asankhaya Sharma. 2017.  Automated identification of security issues from commit messages and bug reports. In _Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (FSE)_. 914–919.
  * Zhou et al. (2021c) Yaqin Zhou, Jing Kai Siow, Chenyu Wang, ShangQing Liu, and Yang Liu. 2021c.  SPI: Automated Identification of Security Patches via Commits.  _ACM Transactions on Software Engineering and Methodology (TOSEM)_ (2021).

##  Appendix A Demonstration

This is a run-through demonstration for VulCurator using our Docker image. For
manual installation, please check our GitHub repository. We also provide a
demo video of VulCurator at the link https://youtu.be/uMlFmWSJYOE

Step 1: User installs VulCurator by pulling Docker image using the command:

docker pull nguyentruongggiang/vfdetector:v1

After a successful install, you should see a similar result to the screenshot
below:

Figure 3. Installation Success

Step 2: Open Docker container using the command:

docker run --name vfdetector -it --shm-size 16G --gpus all
nguyentruongggiang/vfdetector:v1

Step 3 : Move to VulCurator’s working folder

cd ../VFDetector

Step 4: Inferring an output

User needs to prepare a JSON input file follow our format. Below is an
example:

Figure 4. Input File Example

Next, run the command for either “prediction” mode or “ranking” mode:

python application.py -mode prediction -input sample_1.json -output
prediction_sample_1.json

Above is an example for ”prediction” mode, which takes sample_1.json as input
and return prediction_sample_1.json as output.

The following output should be seen:

Figure 5. Screenshot for Prediction Mode

The result of the prediction is written in prediction_sample_1.json:

Figure 6. Example Output for Prediction Mode

Similarly, when running VulCurator in ”ranking” mode, user will obtain a list
of sorted commit based on the computed confidence scores similar to below:

Figure 7. Example Output for Ranking Mode