# Investigating Reproducibility in Deep Learning-Based Software Fault
Prediction

Adil Mukhtar  Graz University of TechnologyGrazAustria<EMAIL_ADDRESS>,  Dietmar Jannach  University of KlagenfurtKlagenfurtAustria
<EMAIL_ADDRESS>and  Franz Wotawa  Graz University of
TechnologyGrazAustria<EMAIL_ADDRESS>
###### Abstract.

Over the past few years, deep learning methods have been applied for a wide
range of Software Engineering (SE) tasks, including in particular for the
important task of automatically predicting and localizing faults in software.
With the rapid adoption of increasingly complex machine learning models, it
however becomes more and more difficult for scholars to reproduce the results
that are reported in the literature. This is in particular the case when the
applied deep learning models and the evaluation methodology are not properly
documented and when code and data are not shared. Given some recent—and very
worrying—findings regarding reproducibility and progress in other areas of
applied machine learning, the goal of this work is to analyze to what extent
the field of software engineering, in particular in the area of software fault
prediction, is plagued by similar problems. We have therefore conducted a
systematic review of the current literature and examined the level of
reproducibility of 56 research articles that were published between 2019 and
2022 in top-tier software engineering conferences. Our analysis revealed that
scholars are apparently largely aware of the reproducibility problem, and
about two thirds of the papers provide code for their proposed deep learning
models. However, it turned out that in the vast majority of cases, crucial
elements for reproducibility are missing, such as the code of the compared
baselines, code for data pre-processing or code for hyperparameter tuning. In
these cases, it therefore remains challenging to exactly reproduce the results
in the current research literature. Overall, our meta-analysis therefore calls
for improved research practices to ensure the reproducibility of machine-
learning based research.

Reproducibility, Software Debugging, Fault Prediction, Fault Localization,
Defect Prediction, Bug Prediction, Deep Learning

††conference: ACM SIGSOFT International Symposium on Software Testing and
Analysis; 16-20 September, 2024; Vienna, Austria††ccs: Software and its
engineering Reusability

##  1\. Introduction

Deep learning techniques have become prevalent in Software Engineering (SE)
related tasks (Yang et al., 2022) and applied in tools related to code
summarization (Iyer et al., 2016; Li et al., 2023), fault prediction
(Pornprasit and Tantithamthavorn, 2022; Majd et al., 2020; Hoang et al.,
2019), vulnerability prediction (Li et al., 2021; Nguyen et al., 2022) and
more. This is mainly because of the adaptation and popularity of deep learning
in other domains such as computer vision (Voulodimos et al., 2018; O’Mahony et
al., 2020; Wu et al., 2017), natural language processing (Otter et al., 2020;
Young et al., 2018), and more, have attracted researchers from the software
engineering domain. Consequently, a large fraction of today’s research is
based on these learning approaches. Already back in 2018, Li et al. (Li et
al., 2018) reviewed 98 scientific articles from the software engineering
domain and found that 84.7% of the examined articles utilized some form of
deep learning approaches for a multitude of software engineering tasks. While
deep learning methods are popular and effective, their complex functioning
requires precise documentation and detailed description of guidelines (Abbess
et al., 2022; Hartley and Olsson, 2020) to reproduce the results (Haibe-Kains
et al., 2020). Additionally, the availability of relevant code packages,
including hyperparameter, evaluation, dataset, etc., is also a prerequisite
for reproducibility.

Reproducibility has indeed become an important consideration in general AI
research, and it is one of the main growing concerns in the scientific
community (Gundersen and Kjensmo, 2018; Liu et al., 2021). Generally,
reproducibility is about validating claims that are made in scientific papers
and thereby ensuring progress in producing the same results claimed by the
authors based on the provided documentation and information (Gundersen and
Kjensmo, 2018). Researchers are often encouraged to share their artifacts,
including datasets, executable files, and evaluation scripts, when publishing
their findings. However, recent studies (Ghanta et al., 2018; Isdahl and
Gundersen, 2019; Raff, 2019) concerning deep learning indicate that a
reproducibility crisis exists that undermines the reproducibility of the
proposed work. Another assessment of the reproducibility status of deep
learning models in the software engineering domain revealed that over 62% of
the published articles failed to provide quality code or comprehensive data,
thus affecting the reproducibility of the presented research (Liu et al.,
2021). Furthermore, Collberg and Proebsting inspected 601 source code
repositories that were shared alongside scientific articles (Collberg and
Proebsting, 2016). The results revealed a rather modest success rate, with
only 32.1% of the experiments proving to be executable with success.

In general, reproducibility is considered a desirable characteristic. In the
context of software engineering and artificial intelligence, it is expected
that researchers provide the required documentation and code packages to
reproduce the results because this helps other researchers validate the
experimental results and findings. However, much of the scientific work fails
in this regard. This situation extends beyond computer science and artificial
intelligence research, as highlighted by researchers in the field of
psychology, who have reported a success rate of just 36% in reproducing the
results reported in articles (Nosek et al., 2015).

Figure 1. Categorization of Software Fault Prediction Techniques, adapted from
(Wong et al., 2016)

.

Therefore, our main objective is to investigate the current state of
reproducibility in the domain of deep learning-based software engineering. We
narrow our focus on fault prediction111In this survey, we treat bug prediction
and defect prediction as forms of fault prediction. techniques. The motivation
for this investigation is based on several reasons. First, fault prediction is
an essential and crucial task in software engineering, and numerous deep
learning-based methods are proposed (Qiao et al., 2020; Mukhtar et al., 2023;
Li et al., 2021; Nguyen et al., 2022; White et al., 2019). Second, while
existing studies on reproducibility analysis have considered tools written in
a specific programming language, i.e., C++/C (Nong et al., 2023) or have
addressed software engineering tasks (Liu et al., 2021) and artificial
intelligence in general (Gundersen and Kjensmo, 2018), our research focuses on
the fault prediction task of software engineering. Lastly, reproducibility in
software engineering is often neglected or poorly documented (Liu and Zhang,
2018). With this research, we hope to contribute to the current findings on
the reproducibility status of deep learning-based software engineering. Hence,
we formulate our research question as follows:

_What is the current state of reproducibility of scientific research in the
domain of deep learning-based software fault prediction?_

To answer our research question, we conduct a systematic review in which we
follow the definition of reproducibility provided by the Association for
Computing Machinery
(ACM)222https://www.acm.org/publications/policies/artifact-review-and-badging-
current and Gundersen (Gundersen and Kjensmo, 2018), i.e., reproducibility
means that an independent group of researchers can obtain the same results
claimed by the authors using the same methods, i.e., deep learning in our
case, and measurement procedures provided by the authors. We selected research
articles related to fault prediction in software engineering from top
conferences published between 2019 and 2022. Then, we reviewed selected
articles to record various reproducibility aspects proposed in the literature
(Gundersen and Kjensmo, 2018; Abbess et al., 2022).

Our analysis revealed that approximately two-thirds of the research articles
provide source code repositories. However, upon further inspection, we
identified a significant gap in the availability of crucial details related to
evaluation, model construction, hyperparameter optimization, and baseline
methodologies. These critical aspects of information are absent in almost half
of the articles, and in some instances, they are missing in more than half of
the examined articles. Our investigation revealed that the scientific work
related to deep learning-based SE for fault prediction tasks is not properly
documented to a large extent to ensure reproducibility.

The paper is organized as follows: We discuss the background and related work
in Section 2 followed by our research methodology in Section 3. Our results
and findings are presented in Section 4. Finally, we discuss possible threats
to validity and summarize our research in Section 6 and Section 7
respectively.

##  2\. Background & Related Work

In this section, we provide a brief overview and discussion of the
applicability of deep learning models in the context of software engineering.
Subsequently, we present a summary of existing research that analyzes the
current state of reproducibility.

###  2.1. Deep Learning in Software Engineering

Over the years, the adoption of deep learning in software engineering has
gained significant traction (Liu et al., 2021; Harman, 2012; Alshammari et
al., 2022) and numerous deep learning methodologies have been introduced to
address and improve various SE tasks, e.g., requirements analysis (Navarro-
Almanza et al., 2017; Madala et al., 2017), clone detection (White et al.,
2016; Li et al., 2017; Lei et al., 2022; Fang et al., 2020), debugging (Huo et
al., 2016; Wong et al., 2011; Qiao et al., 2020; Wang and Zhang, 2018; Mukhtar
et al., 2023), and more.

In the context of requirements analysis, Navarro-Almanza et al. (Navarro-
Almanza et al., 2017) introduced a Convolutional Neural Network (CNN) designed
for the classification of functional and non-functional requirements.
Similarly, Madala and colleagues introduced an automated approach for
analyzing requirements, utilizing model-driven methods to autonomously
identify component state transitions (CST) (Madala et al., 2017). Their
approach incorporates Recurrent Neural Networks (RNN) with Long-Short Term
Memory (LSTM) deep learning models. To assess the effectiveness of their
model, they conducted a case study using a pacemaker requirements document.

Deep learning-based methods and tools for the software program clone detection
task are also proposed. For instance, Li et al. (Li et al., 2017) presented
CCLEARNER to detect clones in a software program. CCLEARNER operates by
leveraging method-level data associated with both clones and non-clones,
extracting tokens from this information to train a deep-learning classifier.
The tool’s performance was assessed using a substantial benchmark of clones
known as BigCloneBench. The approach underlying CCLEARNER involves feature
extraction techniques, such as token categorization and similarity scoring
among token vectors. The evaluation of the model and its comparison with
existing clone detection methods revealed that CCLEARNER can efficiently
detect clones while maintaining competitive performance levels. Likewise, Fang
et al. (Fang et al., 2020) proposed a novel technique that utilizes fusion
embedding techniques to detect patterns and semantics associated with features
of the source codes. Based on this information, a deep learning classifier is
trained to detect clones in a large dataset of C++ programs. The outcomes of
the experiments show that the proposed method significantly outperforms
existing clone detection techniques.

Predicting and localizing faults are crucial aspects of software debugging,
and numerous techniques are dedicated to fault prediction (Wong et al., 2016).
Typically, these methods are classified into different categories depending on
how they approach the task of fault prediction. This classification takes into
account factors such as the involved program components and the
characteristics of the used data, see Figure 1. These techniques also often
leverage deep learning models. Deep learning-based fault prediction techniques
usually rely on the structural features of the underlying program and extract
patterns that tend to correlate with the faultiness of the component. For
instance, Huo et al. (Huo et al., 2016) proposed a novel bug prediction
technique named NP-CNN. This approach utilizes program structural information,
bug reports, and source code to effectively identify buggy sections within the
code. Qiao et al. (Qiao et al., 2020) presented a fully connected neural
network aimed at predicting the number of defects in program modules. The
proposed technique leverages software metrics, such as the total number of
code lines, the total number of characters, and Halstead’s program length,
etc. The assumption underlying this approach is that these metrics contain
heuristic characteristics associated with the defective state. Similarly,
Mukhtar et al. (Mukhtar et al., 2023) presented a fault prediction technique
for spreadsheet programs. They proposed to combine spectrum-based information
with the characteristics of spreadsheet programs and utilize this information
in a learning approach. Numerous other techniques have been proposed in the
literature for fault prediction tasks (Lv et al., 2016; Manjula and Florence,
2019; Wang et al., 2020; Zheng et al., 2016; Batool and Khan, 2022; Alqasem
and Akour, 2019; Chen et al., 2020; Wong et al., 2011).

###  2.2. Reproducibility: A Growing Concern

In an extensively referenced _Nature_ news feature article titled _“1,500
scientists lift the lid on reproducibility”_ (Baker, 2016), Monay Baker
presented findings from a survey revealing that more than 70% of researchers
reported that they attempted but could not reproduce the scientific work put
forth by their peers. The survey showed that more than half of the researchers
from various scientific disciplines such as chemistry, biology, medicine,
computer science, and more, were unable to reproduce their own experimental
results. This analysis provides an overview of the current status of
reproducibility in scientific research as a whole, highlighting its emergence
as a growing concern.

Gundersen et al. (Gundersen and Kjensmo, 2018) targeted a specific domain,
i.e., artificial intelligence, and reviewed 400 published articles from two
highly-ranked conferences: International Joint Conference on Artificial
Intelligence (IJCAI) 333https://www.ijcai.org/ and the Association for the
Advancement of Artificial Intelligence Conference (AAAI) 444https://aaai.org/.
The analysis revealed shortcomings in the adequate documentation of research
published in these conferences. In particular, their findings indicated that
nearly 70% of the articles omitted clear articulation of research questions
and the used methodologies. Additionally, a majority of the research papers
exhibited non-reproducible outcomes. These were quantified through
reproducibility variables covering aspects such as _experiments_ , _data_ ,
and _method_. Remarkably, merely 25% of these aspects were documented to
facilitate reproducibility. In our present work, we follow the general
direction of the investigation done by Gundersen et al. (Gundersen and
Kjensmo, 2018), and we adapt various reproducibility variables used in their
work.

Within the software engineering domain, Nong and colleagues (Nong et al.,
2023) reviewed 55 deep learning-based vulnerability detection methods. Their
investigation unveiled that only 25% of the examined approaches offered
publicly available tools. Furthermore, even among those works that did offer
tools, there was a deficiency in documentation and implementation guidelines.
This analysis aligns with the findings put forth by Gundersen et al.
(Gundersen and Kjensmo, 2018), ultimately categorizing the state of the
published research as largely non-reproducible. While vulnerability detection
is a software engineering task that involves identifying vulnerable components
within systems, we treat vulnerability detection as a form of fault
prediction, as classified by Liu et al. (Liu et al., 2021), and we expand the
scope to include articles concerning vulnerability detection within the
context of fault prediction.

In the context of a comprehensive analysis of the reproducibility of deep
learning methods in software engineering, Liu et al. (Liu et al., 2021)
surveyed 147 scientific articles presented at deep learning and software
engineering conferences. The primary objective of this survey was to gain
insights and assess the challenges associated with reproducibility. The study
findings revealed an important statistic: only 30% of the examined articles
published in software engineering conferences provided high-quality
reproduction packages, whereas a notable 50% of the articles published in AI
venues provided such reproduction packages. They also found that only 10% of
the articles can address internal factors of models concerning reproducibility
such as model stability, sensitivity to test data, and optimization
convergence. In our review, however, we do not investigate these factors as we
aim to record information such as links to code packages, hyperparameters,
datasets, etc., provided by authors to ensure reproducibility. Another work on
the reproducibility of defect prediction studies by Mahmood et al. (Mahmood et
al., 2018) showed that only 6% of the reviewed studies could be reproduced by
other researchers.

Our investigation is partly similar to the state-of-the-art work presented by
Gundersen et al. (Gundersen and Kjensmo, 2018). Our review is however more
focused than previous works (Mahmood et al., 2018; Gundersen and Kjensmo,
2018; Nong et al., 2023; Liu et al., 2021), targeting the particular task of
fault prediction models. While our findings and implications may generally
align well with such earlier works, our work provides an up-to-date
perspective of the current status of reproducibility in an important subfield
of software engineering. Importantly, our study aims to provide an in-depth
characterization of what kind of documentation and software artifacts are
nowadays provided by scholars to support the reproducibility of their work.
Inherent factors of deep learning models such as randomness, test data
sensitivity, and convergence may certainly affect reproducibility as well.
However, the analysis of such challenges is out of the scope of our present
work, as we believe that explicit, clear, and correct documentation is a
primary factor to be considered first when studying the potential of
reproducibility of research work.

##  3\. Methodology

We now discuss the methodology of our study. To begin, Section 3.1 sketches
the overall design of our study. Section 3.2 describes the selection process
for scientific articles. Section 3.3, finally, explains how we measured the
various aspects of reproducibility.

###  3.1. Design

Our study is comprised of two main stages:  (1) article selection  (2)
recording of reproducibility aspects  . During the initial stage, we identify
keywords related to fault prediction and deep learning to formulate search
queries for the considered online research databases. We then apply our
inclusion criteria to identify a subset of articles that is relevant to our
research question. The detailed selection process is described in Section 3.2.

In the second stage, we review the selected articles under various
reproducibility aspects adapted from the existing literature and call these
aspects _reproducibility variables_ as proposed by Gundersen et al. (Gundersen
and Kjensmo, 2018). The variables are grouped into four categories: _Source
Code, Hyperparameter Tuning, Dataset, and Evaluation_. The main driving factor
of the review process is to look for explicit documentation of information
related to these variables in the articles and record measurements. The
details of recording the measurements of these variables are further described
in Section 3.3.

###  3.2. Selection Criteria

We apply a systematic retrieval process to ensure that the articles obtained
from the online research database, namely _IEEE Xplore_
555https://ieeexplore.ieee.org and _ACM Digital Library_
666https://dl.acm.org/, are not only relevant but also within the designated
date range, from 2019 to 2022. We selected these research databases because
these are commonly used as a primary corpus in related studies, particularly
in the field of deep learning-based software engineering applications (Watson
et al., 2022; Li et al., 2018; Del Carpio and Angarita, 2020). Moreover, these
databases provide access to millions of articles, such as journals,
conferences, letters, magazines, and more (Wilde, 2016).

#### Relevant Terms

We consider terms relevant that are strongly related to the term “fault
prediction”, such as “bug prediction” and “defect prediction”. We also
consider the term “fault localization” relevant to our study since both fault
prediction and fault localization techniques aim to locate faults (Sohn,
2020). These terms are combined with the relevant deep-learning related
keywords to expand our search retrieval. The keywords related to deep learning
are presented in Table 1.

deep learning | artificial neural network
---|---
neural network | neural
decoder | encoder
learning | reinforcement learning
reinforcement | transformers
Table 1. Deep-learning-related keywords used in the study

Conference Full Name | Abbreviation | Ranking
---|---|---
Automated Software Engineering | ASE | A*
European Software Engineering Conference and Symposium on the Foundations of Software Engineering | ESEC/FSE | A*
International Conference on Software Engineering | ICSE | A*
International Conference on Software Testing, Verification and Validation | ICST | A
International Conference on Software Analysis, Evolution and Reengineering | SANER | A
International Conference on Software Maintenance and Evolution | ICSME | A
International Symposium on Software Testing and Analysis | ISSTA | A
Mining Software Repositories | MSR | A

Table 2. Top Software Engineering conferences selected for the study

#### Search String

After identifying keywords that correspond to the domain of deep learning-
based software fault prediction methods, we formulate the search string as
follows: (“fault prediction” OR “bug prediction” OR “defect prediction” OR
“fault localization”) AND (“deep learning” OR “neural network” OR “neural” OR
“decoder” OR “encoder” OR “artificial neural network” OR “learning” OR
“reinforcement” OR “reinforcement learning” OR “transformers”)

#### Inclusion Criteria

As our initial criterion, we exclusively consider articles published in SE
conferences listed in Table 2 as these are ranked top according to the CORE
2023 ranking777https://www.core.edu.au/conference-portal. Moreover, these
conferences are also often selected for survey studies (Liu et al., 2021) in
software engineering. The conference article is included if:

  * •

it uses deep learning for a fault prediction task,

  * •

it is published between 2019 and 2022,

  * •

it is published in a conference listed in Table 2.

As mentioned earlier, we also consider tasks related to _vulnerability
prediction_ relevant to this study and consequently include those articles. As
part of the exclusion criteria, we exclude articles not published in English
and those that require payment to get access. However, in the end, we did not
have to exclude any papers because the selected conferences only accept
articles written in English, and because we had access to all proceedings
through the university.

#### Selection Process

Finally, we query the online research databases to retrieve articles and apply
our inclusion and exclusion criteria. Articles that are discovered in more
than one database are subjected to deduplication. The overall selection
process of the articles is shown in Figure 2. After applying the search query,
we initially identified 2,326 unique published articles associated with deep
learning-based fault prediction tasks. Subsequently, by applying the
publication year and venue-related criteria, we narrowed down the selection to
223 published articles. Finally, we conducted a manual review of each article
to confirm that the article addressed a fault prediction task and utilized
deep learning methods in its proposed methodology. This manual inspection
resulted in 56 unique articles, which we then reviewed regarding
reproducibility aspects. We share the list of selected articles along with the
recorded reproducibility assessments online888https://bitly.ws/XNiz.

Figure 2. Articles selection process

.

###  3.3. Reproducibility Variables

Table 3 presents the reproducibility variables used in our study.

| Variable | Description
---|---|---
Source Code | Repository Link | Is the link to an online source code repository provided and is the repository available?
Model Code | Is the source code for the proposed model(s) provided in the repository?
Baseline Code | Is the source code for the baseline(s) provided in the repository?
Hyperparameter-tuning Code | Is the source code for hyperparameter tuning provided in the repository?
Preprocessing Code | Is the source code for data preprocessing provided in the repository?
Supplementary Info. |  | Is any supplementary information related to the source code provided,
---
e.g., README, Dockerfile, requirements.txt?
Hyperparameter Tuning | Model Hyperparam. | Is hyperparameter optimization discussed for the proposed model(s)?
Baseline Hyperparam. | Is hyperparameter optimization discussed for the baseline(s)?
Optimization Procedure | Is the hyperparameter optimization procedure described?
Model Hyperparam. Ranges | Are the searched ranges for the hyperparameters reported for the proposed model?
Baseline Hyperparam. Ranges | Are the searched ranges for the hyperparameters reported for the baselines?
Model Best Hyperparam. | Are the best hyperparameters reported for the proposed model?
Baseline Best Hyperparam. | Are the best hyperparameters reported for the baselines?
Dataset | Public Dataset | Is a pre-existing public dataset used?
Downloadable Dataset | Are datasets available for download?
Preprocessed Dataset | Are preprocessed datasets available?
Preprocessing Steps | Are the preprocessing steps described?
Meta-data Info. Dataset | Is meta-data information available for the dataset?
Evaluation | Evaluation Procedure | Is the evaluation procedure described?
Evaluation Metrics | Are the metrics provided for the evaluation?
Significance Testing | Are the details of statistical significance tests provided?

Table 3. Reproducibility variables and their descriptions

As part of the review process, we record these variables and look for clear
mentions of information related to the variables. If the article provides a
link to an online source code repository, we also thoroughly inspect the
repository to verify whether further information related to the individual
aspect can be found there. During the review process, we use a checkmark (✓ )
to indicate the presence of appropriate information regarding a specific
variable, and we use the ✗ symbol to note its absence. We now explain the
variables within each category.

#### Source Code

Sharing of source code packages is considered an effective way to ensure
reproducibility (González-Barahona and Robles, 2012; Liu et al., 2021). In
this category, we mainly look for the link to a source code repository.
Ideally, the link should be provided within the article but in case it is not
available, researchers could request the source code from the authors, e.g.,
through email. However, this is considered unreliable as suggested by Liu et
al. (Liu et al., 2021), and we therefore only rely on the information and
links provided in the articles. This inspection allows us to report the
proportion of articles that share code packages.

This category contains six variables as presented in Table 3. During the
review of the variable Repository Link, we look for the link to an online
source code repository and check if the link is available and the repository
is available. We further inspect the repositories to locate the code scripts
for the creation of proposed model(s) and baseline(s) when reviewing the
variables Model Code and Baseline Code, respectively. Similarly, we check if
code is available for dataset preprocessing and fine-tuning of the model while
reviewing the variables Preprocessing Code and Hyperparameter-tuning Code.
Additionally, we record a ✓ if at least one of the files such as Dockerfile,
README, and requirement.txt is found; otherwise, we record a ✗ .

#### Hyperparameter Tuning

Deep learning models, like traditional machine learning models, usually have a
number of hyperparameters that can be chosen depending on the given situation,
such as embedding sizes, the learning rate, or weight factors for the loss
function. The optimal values for such hyperparameters for a given dataset and
optimization goal are commonly determined through a systematic procedure such
as grid search. This tuning process is essential because the choice of the
hyperparameters usually has a huge effect on the performance of a model
(Shehzad and Jannach, 2023). It is therefore highly important not only that
all models in a comparison are properly tuned, but also that the tuning
process is properly documented to ensure reproducibility, in particular in
terms of searched parameter spaces, the search method, and the final
parameters.

We identified seven variables that belong to the category of hyperparameter
tuning-related information. For these variables, we primarily rely on the
article content and pay particularly close attention to the sections where
this information is expected to be usually available. First, we check if the
discussion related to hyperparameter tuning is mentioned, i.e., fine-tuning
for the proposed and baseline models and register values for the Model
Hyperparam. and Baseline Hyperparam. variables. Then we record whether the
article describes the optimization procedure, such as random search or grid
search, during the review of the Optimization Procedure variable. Similarly,
we record if the information regarding parameter ranges and best parameters is
presented for both the proposed and the baseline models. This is done while
reviewing the Model Hyperparam. Ranges, Baseline Hyperparam. Ranges, Model
Best Hyperparam., and Baseline Best Hyperparam. variables. We also inspect the
code repository in case the information is not found in the manuscript.

#### Dataset

The availability of datasets is essential to reproduce reported results.
However, datasets are often preprocessed before feeding into the deep learning
models. For instance, log transformation and normalization are commonly
applied as preprocessing methods. The preparation of the data is an important
activity in developing effective deep-learning models. Therefore, the
availability of such information is also equally important for
reproducibility. Thus, dataset-related information is recorded through five
reproducibility variables as presented in Table 3. In many cases, methods are
assessed using benchmark datasets, which are frequently publicly accessible.
Sometimes, however, benchmark datasets are not used by authors and they create
their own datasets. To record such information, we look for links to dataset
sources and record the values for the Public Dataset and Downloadable Dataset
variables accordingly. The reason for including public datasets is that these
datasets, e.g., PROMISE, NASA, are usually accessible over the web. However,
these datasets are often available in some raw form and need to be processed
according to the proposed model. Therefore, we check if the preprocessed
dataset is accessible as well and record this in the Preprocessed Dataset
variable. To record the information related to preprocessing steps, we inspect
both the article content and supplementary information provided alongside the
repository, and we correspondingly register a value for the Preprocessing
Steps variable. Moreover, we record a value for the Meta-data Info. Dataset
variable to see if there is meta-information available pertaining to the
dataset, including feature descriptions, feature ranges, and meta-level
information. At times, authors reference previously published articles that
contain datasets used in their own work. We also record this information,
however, we only limit our investigation to the original article.

Figure 3. Frequencies of different deep learning architectures

#### Evaluation

Deep learning methods are typically evaluated in an offline environment based
on different evaluation protocols and based on various metrics. Commonly used
evaluation protocols include k-fold cross-validation or leave-one-out
evaluation; examples of evaluation metrics include precision and recall or the
root mean squared error. An accurate description of these evaluation aspects
is necessary for reproducibility. Therefore, we incorporate these aspects in
our investigation and record whether the evaluation procedures and metrics are
clearly explained. Additionally, we also take note if any statistical
significance tests were reported in the article.

First, we record information concerning the evaluation methodology for the
Evaluation Procedure variable. Specifically, we look for the mentions and
descriptions of the evaluation procedure. Then, we look for the evaluation
metrics that are used to record a value for the Evaluation Metrics variable.
We also record a value for the Significance Testing variable by checking if
the article conducts significance testing to compare the performance of the
proposed model with the baselines.

As an additional analysis, we also investigate the experimentation type used
during the evaluation, i.e., offline or online. If the experimentation is done
in an online setting then we record the following relevant information:  (1)
Is the supplementary information provided related to user study, e.g., user
manuals, documentation, etc.?  (2) What was the study size?

Generally, please recall that throughout our investigation, our focus is to
assess the availability of explicit documentation of information regarding
reproducibility. Importantly, our goal is _not_ to retrain and evaluate the
models using the information and materials provided by the authors. Instead,
we exclusively focus on recording reproducibility variables (Table 3) that
indicate what the authors have provided in order to help reproducing the
results they report in their papers.

We first present the results related to the analysis of the selected articles
in Section 3.4. In Section 4, we report and discuss the findings related to
our main research question.

###  3.4. Article Statistics

Table 4 presents the yearly distribution of the selected articles. We can
observe that the number of relevant articles increased sharply in 2022, while
it remained rather constant in previous years. Generally, our sample may be
too small—resulting from our specific focus—to derive reliable trends over
time. Nonetheless, the uptake in 2022 stands out and indicates that deep
learning models are becoming prevalent in top-tier software engineering
conferences, and we found at least one relevant paper in each conference.

Table 4. Number of articles published yearly Year Published | Number of Articles
---|---
2019 | 10
2020 | 9
2021 | 12
2022 | 25
Figure 4. Analysis of Source Code Reproducibility Variables

We furthermore recorded which deep learning architectures were used for fault
prediction in the considered papers. Figure 3 shows the distribution. Notably,
the basic multilayer perceptron (MLP) network is the most frequently employed
architecture. However, the much more recent _transformer_ architecture has
been used very frequently, both in its standard form or as part of a
specialized architecture like BERT (Bidirectional Encoder Representations from
Transformers). The remaining articles show that researchers explored a rich
range of popular architectures for the fault prediction task, including
Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM), Graph
Neural Networks (GNN), and Recurrent Neural Networks (RNN).

##  4\. Results and Findings

In this section, we provide the findings related to the reproducibility
variables described in Table 3, considering both offline experiments and user
studies.

###  4.1. Source Code

The overall findings related to the source code variables are presented in
Figure 4. We found that 41 articles, i.e., 73.2%, share links to a source code
repository. However, upon closer inspection, we noticed that 5 articles at the
time of our analysis provided links to non-existent (potentially private)
repositories or to empty repositories. Nonetheless, as there is at least some
code for a bit less than two-thirds (64.3%) of the cases, our findings suggest
that researchers are aware of the importance of ensuring reproducibility of
their work and put some effort into sharing their source code.

The existence of the five non-working repository links may point to certain
issues in the review and publication process. In these cases, reviewers were
either not checking the repositories, or the authors only promised to share
their code in case of acceptance, and did not share artifacts in the reviewing
period. To address such issues, expectations regarding reproducibility aspects
should be made explicit in the call for papers, and the existence of the
shared artifacts must be verified before publication.

Looking at the 36 articles with accessible repositories, we found that they
contained the source code for the _proposed_ model (Model Code variable) in
all but one case. In this particular case, we found that the code is provided
for only one of the intermediate tasks, and not for the learning model.

However, it turned out that the code for the models that were used as
_baselines_ (Baseline Code variable) in the empirical evaluations is missing
in the large majority (75%) of the cases. This is quite surprising because for
a reliable evaluation, researchers must have access to the original source
code of the baselines or have implemented them by themselves. In case the
authors implemented the baselines by themselves, there is a certain risk that
such implementations are not entirely correct and may lead to wrong
conclusions, see (Hidasi and Czapp, 2023). Thus, it is important that the
baseline code is shared as well. In case some original code is used, on the
other hand, it must be ensured that both the baselines and the proposed model
are benchmarked with some shared evaluation code. Otherwise, there is a risk
that the comparison is not exact, because of certain differences, e.g., how
datasets are split, how the metrics are computed, etc. In the literature,
there are also some indications that sometimes authors do not actually run the
baselines by themselves but simply copy the results from previous papers
(Shehzad and Jannach, 2023). Such an approach may however be very unreliable,
because of the commonly observed differences in how the evaluation is done.

Figure 5. Analysis of Hyperparameter Tuning Reproducibility Variables

Looking at the availability of code for the hyperparameter tuning process
(Hyperparameter-tuning Code variable), we find that this code is provided in
only 13.8% of the cases for which a repository with code was provided at all.
Upon further inspection, we found a number of instances in which authors
claimed to do hyperparameter tuning with a certain optimization method like
Grid Search. However, the code for such procedures was then not included in
the shared repository. Instead, we often found static hyperparameters in the
shared model code, which could simply be the last ones that the authors tried
before uploading their code.

In the context of hyperparameter tuning code, one might argue that this
process can be accomplished with a few library calls999E.g., when using the
scikit-learn Python library (https://scikit-learn.org). However, the details
of the process (e.g., regarding the explored parameter ranges, the number of
tries in a random search, etc.) can make a huge difference. Providing the
tuning code can therefore be very important to be able to reproduce the exact
results reported in a paper.

More positive observations were made regarding the availability of data
preprocessing code (Preprocessing Code variable), where more than two-thirds
of the 36 accessible code repositories contained the respective code.
Furthermore, all 36 repositories contained some at least minimal documentation
(Supplementary Info. variable), e.g., in the form of a README file. However,
only in four repositories we found a Docker101010https://www.docker.com file,
which is assumed to be particularly helpful for easy reproducibility
(Cremonesi and Jannach, 2021), as such containers relieve researchers from the
burden of setting up an execution environment with all needed software
dependencies (in the right version).

###  4.2. Hyperparameter Tuning

Figure 5 presents the statistics of the hyperparameter tuning reproducibility
variables. We found that 41 (73%) of the articles discuss hyperparameter
optimization for the _proposed_ model (Model Hyperparam. variable). Among
them, 31 articles explicitly provide the name of the optimization procedure
(Optimization Procedure variable). When it comes to the optimization of the
_baseline_ models (Baseline Hyperparam. variable), in 78% of the cases no
information is provided, which aligns with the recent findings in (Shehzad and
Jannach, 2023), where researchers found that generally little documentation is
provided about the tuning of baselines.

Looking at the documentation of explored hyperparameter ranges and the best
model parameters (Model Hyperparam. Ranges and Model Best Hyperparam.
variables), we find this information is very frequently missing: 14 articles
provide details for the hyperparameter ranges and only 4 articles report the
best hyperparameter. Here, it is notable that _none_ of the articles report
the best hyperparameter for the baselines, and a large number of articles,
i.e., 94%, do not provide the searched ranges for the baselines. Again, this
is surprising, because in case the authors had correctly tuned all baselines
per dataset as required, questions regarding suitable ranges must have come
up, and optimal hyperparameter values must have been determined at some stage.

Overall, we find that the _documentation_ of the details of the hyperparameter
tuning process—in particular for the baselines—is largely missing in the
analyzed literature. Clearly, we cannot know if the authors have actually
properly tuned the baselines. In any case, tuning many baselines on several
datasets can be computationally costly, in particular when it comes to modern
deep learning architectures, and such a tuning process may take days or weeks.
It is therefore a bit surprising that such efforts are not documented in a
research paper at all. An alternative explanation therefore is that authors
may sometimes spend extensive efforts to tune their proposed model, but do not
apply the same efforts to the baselines. Such problematic research practices
have been observed in other areas of applied machine learning earlier, e.g.,
in information retrieval, time series forecasting, or recommender systems
(Armstrong et al., 2009; Lin, 2019; Makridakis et al., 2018; Ferrari Dacrema
et al., 2019).

Figure 6. Analysis of Dataset Reproducibility Variables

###  4.3. Dataset

The findings for dataset-related reproducibility variables are presented in
Figure 6. In a large majority of cases, around 70%, researchers rely on pre-
existing publicly available datasets in their experiments (Public Dataset
variable). The remaining 17 works used proprietary (new) datasets. In only one
of these cases, the authors made their new dataset available by sharing a
public link. Clearly, research works that are entirely based on non-public
datasets cannot be validated or exactly reproduced by others.

In terms of the _accessibility_ of the used datasets (Downloadable Dataset
variable), explicit links to the dataset files are provided in almost 60% of
the cases, either in the paper, the source code, or in some additional
documentation.111111In two cases, links were provided where either the link
was broken or generically pointed to a conference website. Dataset
_preprocessing_ is very common in the reviewed papers. Almost two-thirds of
the papers (36 out of 56) describe how the used datasets were preprocessed.
The datasets that result from this preprocessing are however only shared in 24
of the papers. The reproducibility of the remaining 12 papers therefore
depends on the ability of a researcher to exactly reproduce the preprocessing
code.

Finally, in almost all papers, at least some meta-information about the
datasets was provided (Meta-data Info. Dataset variable), with the level of
detail varying strongly across articles. Overall, however, it shows that the
reproducibility of the reviewed works may frequently suffer from the use of
non-public datasets and insufficiently documented data preprocessing
procedures.

###  4.4. Evaluation

We finally turn our attention to the documentation of evaluation aspects.
Given their technical nature, all reviewed papers provide some empirical
results and correspondingly describe their evaluation methodology. Notably,
all except one article in our survey are exclusively based on _computational_
(offline) experiments; only one paper aimed to assess the proposed model’s
practical utility by involving developers in the fault prediction tasks.
Unfortunately, in this single case the documentation of certain aspects of the
study, e.g., participant demographics, was not very detailed.

Different procedural approaches (Evaluation Procedure variable) of different
complexity are used in the papers for offline experiments, see Figure 7. About
one-third of the paper relies on single train-test splits, another third also
includes a validation set for hyperparameter tuning, and about another third
of the papers use a cross-validation procedure. We recall here that relying on
one single train-test split may be considered risky or problematic for
different reasons. First, it may imply that the hyperparameters were tuned on
the test set—is not uncommon also in other areas of applied machine learning
(Sun et al., 2020)—but methodologically unsound. Second, a single (random)
train-test split may have been created in an “unlucky” way so that the
findings for this split do not generalize well. In this case, the code for
creating the data splits must be shared by the authors to ensure
reproducibility. Nevertheless, splitting techniques are the most common
procedures for the evaluation. However, sometimes models are tested with data
that is not part of the sample dataset, this is called out-of-sample
evaluation and considered less sensitive to bias and variance
(Tantithamthavorn et al., 2016). Looking at Figure 7, we see that it is
applied in only one article as this is not a commonly used evaluation
procedure.

A rich variety of metrics are used in the context of offline experiments, and
all papers describe the metrics they use and the values obtained in the
experiments (Evaluation Metrics variable). The most frequently used metrics
are general ones that are commonly used for classification or ranking tasks
such as precision, recall, area under the curve, and mean average precision.
Additionally, for the fault prediction task, various metrics are used such as
“number of bugs detected”, relative improvement (_RImp_) (Debroy et al., 2010)
or the EXAM score (Mao et al., 2014) and other commonly used metrics. Finally,
statistical significance tests are rather scarcely used in the reviewed papers
(Significance Testing variable), and only 8 papers provided outcomes of
applying statistical significance tests.

Figure 7. Evaluation methods used in the articles

##  5\. Implications

Overall, while our study reveals some major open gaps in terms of
reproducibility, we believe that the practices in the research community are
developing in the right direction. Compared, for example, to the analysis by
Gundersen et al.(Gundersen and Kjensmo, 2018) from 2018 in the broader field
of AI, the fraction of papers for which code is shared is much higher in our
survey than it was found by Gundersen et al. Also, looking even further back
in history, sharing any code and data, to our experience, was not very common
in many fields of computer science ten or fifteen years ago. As of today, many
conferences and journals at least mention reproducibility aspects in their
guidelines for authors and reviewers.

Nonetheless, the situation is far from being ideal today, as our survey shows.
For the majority of the published research, at least the code for the proposed
models is shared. This certainly helps to _reproduce the numerical results
reported for the proposed models_ , at least when other important information
like datasets, preprocessing code, and optimal hyperparameters are available
as well. A major problem here, however, is that in almost all cases the
artifacts are missing to _reproduce the claimed improvement over existing
models_ , as we, for example, cannot validate if the baselines were properly
tuned. If this is not the case, we may observe a lot of “phantom progress”
(Ferrari Dacrema et al., 2019) in the published literature and “improvements
that don’t add up” (Armstrong et al., 2009).

To increase the level of reproducibility, we must analyze the potential
reasons why scholars share or do not share artifacts, and we must subsequently
develop suitable mechanisms that increase the probability of making research
results more easily reproducible than today. Most intuitively, researchers
might share their code because they are aware of the importance of
reproducibility and they are seeking to report reliable and trustworthy
results. Also, given that reproducibility aspects are increasingly considered
explicitly in the academic reviewing process, sharing code may also increase
their chances of paper acceptance. Finally, researchers may expect that their
approaches are cited more often when others can easily use them, e.g., as
baselines in their own experiments.

On the other hand, creating all the materials that are needed for the
reproducibility of the entire experiment can be very time-consuming.121212In
our own experience, software development for machine learning models and for
research papers, in general, can be quite unstructured, leading to chaotic
code organization or dead fragments because of time pressure or because of
limited experience by the programmer. Therefore, if the chances to get a
research work published without sharing any materials or by sharing only one
central component, it may be tempting for a researcher to avoid this extra
work. Furthermore, some researchers may also fear that there are issues with
their code artifacts, which may be exposed if they share their code. Further
reasons for not sharing the code may exist as well.

We can think of different measures to further increase the level of
reproducibility of published research in machine-learning-based software
engineering. One measure is to make reproducibility more often one of the
central criteria for the evaluation of research papers. This could be
implemented through reproducibility guidelines published along with journal
submission guidelines and conference calls for papers.131313Various guidelines
and checklists for machine learning and general AI exist, e.g., (Pineau et
al., 2021) or https://aaai.org/conference/aaai/aaai-23/reproducibility-
checklist/ Furthermore, reproducibility criteria should be made part of review
forms so that the topic is explicitly checked and evaluated by reviewers. In
terms of positive incentives for reproducible research, more and more computer
science conferences nowadays have reproducibility tracks. Furthermore,
organizations like ACM offer special recognition (badges) for reproducible
research works141414https://www.acm.org/publications/artifacts.

Finally, and in our view most importantly, we believe that in the long run,
existing issues of reproducibility and, consequently, of somewhat limited
progress, can only be sustainably addressed through further increased
awareness and education, involving all relevant stakeholders from students,
teachers, scholars, or grant reviewers, see (Bauer et al., 2023). Along with
this, a shift in the academic incentivization and credit system seems
required, one that values reproducibility much more than it is the case today.

##  6\. Threats to Validity

We recall that the articles we reviewed were retrieved from two specific
research databases. One may thus argue that the selected sample of articles
may not be fully representative. However, the rationale behind our choice of
these particular databases is that these are commonly utilized in the academic
literature for conducting surveys related to deep learning in software
engineering (Li et al., 2018; Watson et al., 2022; Del Carpio and Angarita,
2020), and they offer access to a vast repository of articles (Wilde, 2016).
With the goal to comprehensively evaluate the latest research works, we
focused on articles published during the last four years. This selection
criteria resulted in 56 articles that were considered in our review. While the
number of research works published in the last decade is certainly higher, we
believe that our review is well representative of the current state of
research in terms of reproducibility.

We furthermore note that our research approach involves documenting
exclusively what is reported and provided by the authors. We do not engage in
the actual training and evaluation of the models. Furthermore, we do not take
into account other factors affecting reproducibility, such as randomness or
the non-deterministic characteristics of hardware (Chen et al., 2022). To
efficiently review hundreds of articles, our effort is primarily toward the
explicit documentation of reported information. To ensure the correctness of
the recorded variables, we performed two rounds of article reviewing.

##  7\. Conclusion

Deep learning techniques are gaining widespread popularity in the field of
Software Engineering. Their constantly increasing complexity can however make
it difficult to reproduce the results that are reported in research papers. To
ensure progress in this field, it is therefore of utmost importance that
scholars provide as much material as possible for others to validate and
improve on their findings. Our survey revealed that the current situation in
the subfield of software fault prediction is more than ideal. We as a
community must constantly strive to further improve our standards in this
important aspect of reproducibility, which is certainly a cornerstone of
scientific research.

## Acknowledgement

This work is funded by the Austrian Science Fund (FWF) under contract number P
32445.

## References

  * (1)
  * Abbess et al. (2022) Ali Ben Abbess, Leonardo Meneguzzi, Pedro Pizzigatti Corrêa, David Mouillot, Romain David, Gérard Subsol, Danton Ferreira Vellenich, Rodolphe Devillers, Shelley Stall, Nicolas Mouquet, et al. 2022\.  Checklist Strategies to Improve the Reproducibility of Deep Learning Experiments with an Illustration. In _RDA 19th Plenary Meeting, Part Of International Data Week_.
  * Alqasem and Akour (2019) Osama Alqasem and Mohammed Akour. 2019.  Software Fault Prediction Using Deep Learning Algorithms.  _International Journal of Open Source Software and Processes_ (2019).
  * Alshammari et al. (2022) Fahad H Alshammari et al. 2022\.  Trends in Intelligent and AI-Based Software Engineering Processes: A Deep Learning-Based Software Process Model Recommendation Method.  _Computational Intelligence and Neuroscience_ (2022).
  * Armstrong et al. (2009) Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel. 2009.  Improvements That Don’t Add Up: Ad-hoc Retrieval Results Since 1998. In _CIKM ’09_.
  * Baker (2016) Monya Baker. 2016\.  1,500 scientists lift the lid on reproducibility.  _Nature_ (2016).
  * Batool and Khan (2022) Iqra Batool and Tamim Ahmed Khan. 2022.  Software fault prediction using data mining, machine learning and deep learning techniques: A systematic literature review.  _Computers and Electrical Engineering_ (2022).
  * Bauer et al. (2023) Christine Bauer, Maik Fröbe, Dietmar Jannach, Udo Kruschwitz, Paolo Rosso, Damiano Spina, and Nava Tintarev. 2023.  Overcoming Methodological Challenges in Information Retrieval and Recommender Systems through Awareness and Education. In _Dagstuhl Seminar 23031: Frontiers of Information Access Experimentation for Research and Education_.
  * Chen et al. (2022) Boyuan Chen, Mingzhi Wen, Yong Shi, Dayi Lin, Gopi Krishnan Rajbahadur, and Zhen Ming (Jack) Jiang. 2022.  Towards Training Reproducible Deep Learning Models. Association for Computing Machinery.
  * Chen et al. (2020) Jinyin Chen, Keke Hu, Yitao Yang, Yi Liu, and Qi Xuan. 2020.  Collective transfer learning for defect prediction.  _Neurocomputing_ (2020).
  * Collberg and Proebsting (2016) Christian Collberg and Todd A. Proebsting. 2016.  Repeatability in Computer Systems Research.  _Commun. ACM_ (2016).
  * Cremonesi and Jannach (2021) Paolo Cremonesi and Dietmar Jannach. 2021.  Progress in recommender systems research: Crisis? What crisis?  _AI Magazine_ (2021).
  * Debroy et al. (2010) Vidroha Debroy, W Eric Wong, Xiaofeng Xu, and Byoungju Choi. 2010\.  A grouping-based strategy to improve the effectiveness of fault localization techniques. In _2010 10th Intl. Conference on Quality Software_. IEEE.
  * Del Carpio and Angarita (2020) Alvaro Fernández Del Carpio and Leonardo Bermón Angarita. 2020\.  Trends in software engineering processes using deep learning: a systematic literature review. In _2020 46th Euromicro Conference on Software Engineering and Advanced Applications_.
  * Fang et al. (2020) Chunrong Fang, Zixi Liu, Yangyang Shi, Jeff Huang, and Qingkai Shi. 2020.  Functional code clone detection with syntax and semantics fusion learning. In _Proceedings of the 29th ACM SIGSOFT international symposium on software testing and analysis_.
  * Ferrari Dacrema et al. (2019) Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. 2019\.  Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches. In _Proceedings of the 2019 ACM Conference on Recommender Systems (RecSys 2019)_.
  * Ghanta et al. (2018) Sindhu Ghanta, Lior Khermosh, Sriram Ganapathi Subramanian, Vinay Sridhar, Swaminathan Sundararaman, Dulcardo Arteaga, Qianmei Luo, Drew S. Roselli, Dhananjoy Das, and Nisha Talagala. 2018.  A Systems Perspective to Reproducibility in Production Machine Learning Domain.
  * González-Barahona and Robles (2012) Jesús M González-Barahona and Gregorio Robles. 2012.  On the reproducibility of empirical software engineering studies based on data retrieved from development repositories.  _Empirical Software Engineering_ (2012).
  * Gundersen and Kjensmo (2018) Odd Erik Gundersen and Sigbjørn Kjensmo. 2018.  State of the art: Reproducibility in artificial intelligence. In _Proceedings of the AAAI Conference on Artificial Intelligence_.
  * Haibe-Kains et al. (2020) Benjamin Haibe-Kains, George Alexandru Adam, Ahmed Hosny, Farnoosh Khodakarami, Massive Analysis Quality Control (MAQC) Society Board of Directors Shraddha Thakkar 35 Kusko Rebecca 36 Sansone Susanna-Assunta 37 Tong Weida 35 Wolfinger Russ D. 38 Mason Christopher E. 39 Jones Wendell 40 Dopazo Joaquin 41 Furlanello Cesare 42, Levi Waldron, Bo Wang, Chris McIntosh, Anna Goldenberg, Anshul Kundaje, et al. 2020\.  Transparency and reproducibility in artificial intelligence.  _Nature_ (2020).
  * Harman (2012) Mark Harman. 2012\.  The role of artificial intelligence in software engineering. In _2012 First International Workshop on Realizing AI Synergies in Software Engineering (RAISE)_. IEEE.
  * Hartley and Olsson (2020) Matthew Hartley and Tjelvar SG Olsson. 2020.  dtoolai: Reproducibility for deep learning.  _Patterns_ (2020).
  * Hidasi and Czapp (2023) Balázs Hidasi and Ádám Tibor Czapp. 2023\.  The Effect of Third Party Implementations on Reproducibility. In _Proceedings of the 17th ACM Conference on Recommender Systems, RecSys 2023_.
  * Hoang et al. (2019) Thong Hoang, Hoa Khanh Dam, Yasutaka Kamei, David Lo, and Naoyasu Ubayashi. 2019.  DeepJIT: an end-to-end deep learning framework for just-in-time defect prediction. IEEE.
  * Huo et al. (2016) Xuan Huo, Ming Li, Zhi-Hua Zhou, et al. 2016\.  Learning unified features from natural and programming languages for locating buggy source code.. In _IJCAI_.
  * Isdahl and Gundersen (2019) Richard Isdahl and Odd Erik Gundersen. 2019.  Out-of-the-Box Reproducibility: A Survey of Machine Learning Platforms. In _2019 15th International Conference on eScience (eScience)_. 86–95.  https://doi.org/10.1109/eScience.2019.00017
  * Iyer et al. (2016) Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016.  Summarizing Source Code using a Neural Attention Model. Association for Computational Linguistics.
  * Lei et al. (2022) Maggie Lei, Hao Li, Ji Li, Namrata Aundhkar, and Dae-Kyoo Kim. 2022.  Deep learning application on code clone detection: A review of current knowledge.  _Journal of Systems and Software_ (2022).
  * Li et al. (2017) Liuqing Li, He Feng, Wenjie Zhuang, Na Meng, and Barbara Ryder. 2017.  Cclearner: A deep learning-based clone detection approach. In _2017 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE.
  * Li et al. (2023) Mingchen Li, Huiqun Yu, Guisheng Fan, Ziyi Zhou, and Jiawen Huang. 2023.  ClassSum: a deep learning model for class-level code summarization.  _Neural Computing and Applications_ (2023).
  * Li et al. (2018) Xiaochen Li, He Jiang, Zhilei Ren, Ge Li, and Jingxuan Zhang. 2018.  Deep learning in software engineering.  _arXiv preprint arXiv:1805.04825_ (2018).
  * Li et al. (2021) Yi Li, Shaohua Wang, and Tien N. Nguyen. 2021.  Vulnerability Detection with Fine-Grained Interpretations. Association for Computing Machinery.
  * Lin (2019) Jimmy Lin. 2019\.  The Neural Hype and Comparisons Against Weak Baselines.  _SIGIR Forum_ (2019).
  * Liu et al. (2021) Chao Liu, Cuiyun Gao, Xin Xia, David Lo, John Grundy, and Xiaohu Yang. 2021\.  On the Reproducibility and Replicability of Deep Learning in Software Engineering.  _ACM Trans. Softw. Eng. Methodol._ (2021).
  * Liu and Zhang (2018) Yang Liu and Jiajun Zhang. 2018.  Deep learning in machine translation.  _Deep learning in natural language processing_ (2018).
  * Lv et al. (2016) Feiya Lv, Chenglin Wen, Zejing Bao, and Meiqin Liu. 2016\.  Fault diagnosis based on deep learning. In _2016 American control conference (ACC)_. IEEE.
  * Madala et al. (2017) Kaushik Madala, Danielle Gaither, Rodney Nielsen, and Hyunsook Do. 2017.  Automated identification of component state transition model elements from requirements. In _2017 IEEE 25th international requirements engineering conference workshops (REW)_. IEEE.
  * Mahmood et al. (2018) Zaheed Mahmood, David Bowes, Tracy Hall, Peter CR Lane, and Jean Petrić. 2018.  Reproducibility and replicability of software defect prediction studies.  _Information and Software Technology_ (2018).
  * Majd et al. (2020) Amirabbas Majd, Mojtaba Vahidi-Asl, Alireza Khalilian, Pooria Poorsarvi-Tehrani, and Hassan Haghighi. 2020\.  SLDeep: Statement-level software defect prediction using deep-learning model on static code features.  _Expert Systems with Applications_ (2020).
  * Makridakis et al. (2018) Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. 2018.  Statistical and Machine Learning forecasting methods: Concerns and ways forward.  _PloS one_ (2018).
  * Manjula and Florence (2019) C. Manjula and Lilly Florence. 2019.  Deep neural network based hybrid approach for software defect prediction using software metrics.  _Cluster Computing_ (2019).
  * Mao et al. (2014) Xiaoguang Mao, Yan Lei, Ziying Dai, Yuhua Qi, and Chengsong Wang. 2014.  Slice-based statistical fault localization.  _Journal of Systems and Software_ (2014).
  * Mukhtar et al. (2023) Adil Mukhtar, Birgit Hofer, Dietmar Jannach, Franz Wotawa, and Konstantin Schekotihin. 2023\.  Boosting Spectrum-Based Fault Localization for Spreadsheets with Product Metrics in a Learning Approach _(ASE ’22)_. Association for Computing Machinery.
  * Navarro-Almanza et al. (2017) Raul Navarro-Almanza, Reyes Juarez-Ramirez, and Guillermo Licea. 2017.  Towards supporting software engineering using deep learning: A case of software requirements classification. In _2017 5th International Conference in Software Engineering Research and Innovation (CONISOFT)_. IEEE.
  * Nguyen et al. (2022) Truong Giang Nguyen, Thanh Le-Cong, Hong Jin Kang, Xuan-Bach D. Le, and David Lo. 2022.  VulCurator: A Vulnerability-Fixing Commit Detector. Association for Computing Machinery.
  * Nong et al. (2023) Yu Nong, Rai Ramji Sharma, Abdelwahab Hamou-Lhadj, Xiapu Luo, and Haipeng Cai. 2023.  Open Science in Software Engineering: A Study on Deep Learning-Based Vulnerability Detection.  _IEEE Transactions on Software Engineering_ (2023).
  * Nosek et al. (2015) B. A. Nosek, G. Alter, G. C. Banks, D. Borsboom, S. D. Bowman, S. J. Breckler, S. Buck, C. D. Chambers, G. Chin, G. Christensen, M. Contestabile, A. Dafoe, E. Eich, J. Freese, R. Glennerster, D. Goroff, D. P. Green, B. Hesse, M. Humphreys, J. Ishiyama, D. Karlan, A. Kraut, A. Lupia, P. Mabry, T. Madon, N. Malhotra, E. Mayo-Wilson, M. McNutt, E. Miguel, E. Levy Paluck, U. Simonsohn, C. Soderberg, B. A. Spellman, J. Turitto, G. VandenBos, S. Vazire, E. J. Wagenmakers, R. Wilson, and T. Yarkoni. 2015.  Promoting an open research culture.  _Science_ (2015).
  * Otter et al. (2020) Daniel W Otter, Julian R Medina, and Jugal K Kalita. 2020\.  A survey of the usages of deep learning for natural language processing.  _IEEE transactions on neural networks and learning systems_ (2020).
  * O’Mahony et al. (2020) Niall O’Mahony, Sean Campbell, Anderson Carvalho, Suman Harapanahalli, Gustavo Velasco Hernandez, Lenka Krpalkova, Daniel Riordan, and Joseph Walsh. 2020.  Deep learning vs. traditional computer vision. Springer.
  * Pineau et al. (2021) Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Lariviere, Alina Beygelzimer, Florence d’Alche Buc, Emily Fox, and Hugo Larochelle. 2021.  Improving Reproducibility in Machine Learning Research.  _Journal of Machine Learning Research_ (2021).
  * Pornprasit and Tantithamthavorn (2022) Chanathip Pornprasit and Chakkrit Kla Tantithamthavorn. 2022\.  Deeplinedp: Towards a deep learning approach for line-level defect prediction.  _IEEE Transactions on Software Engineering_ (2022).
  * Qiao et al. (2020) Lei Qiao, Xuesong Li, Qasim Umer, and Ping Guo. 2020\.  Deep learning based software defect prediction.  _Neurocomputing_ (2020).
  * Raff (2019) Edward Raff. 2019\.  _A Step toward Quantifying Independently Reproducible Machine Learning Research_.  Curran Associates Inc.
  * Shehzad and Jannach (2023) Faisal Shehzad and Dietmar Jannach. 2023.  Everyone’s a Winner! On Hyperparameter Tuning of Recommendation Models. In _17th ACM Conference on Recommender Systems_.
  * Sohn (2020) Jeongju Sohn. 2020\.  Bridging fault localisation and defect prediction. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Companion Proceedings_.
  * Sun et al. (2020) Zhu Sun, Di Yu, Hui Fang, Jie Yang, Xinghua Qu, Jie Zhang, and Cong Geng. 2020.  Are we evaluating rigorously? benchmarking recommendation for reproducible evaluation and fair comparison. In _Fourteenth ACM Conference on Recommender Systems_.
  * Tantithamthavorn et al. (2016) Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E Hassan, and Kenichi Matsumoto. 2016.  An empirical comparison of model validation techniques for defect prediction models.  _IEEE Transactions on Software Engineering_ (2016).
  * Voulodimos et al. (2018) Athanasios Voulodimos, Nikolaos Doulamis, Anastasios Doulamis, Eftychios Protopapadakis, et al. 2018\.  Deep learning for computer vision: A brief review.  _Computational intelligence and neuroscience_ (2018).
  * Wang and Zhang (2018) Jinyong Wang and Ce Zhang. 2018.  Software reliability prediction using a deep learning model based on the RNN encoder–decoder.  _Reliability Engineering & System Safety_ (2018).
  * Wang et al. (2020) Song Wang, Taiyue Liu, Jaechang Nam, and Lin Tan. 2020\.  Deep Semantic Feature Learning for Software Defect Prediction.  _IEEE Transactions on Software Engineering_ (2020).
  * Watson et al. (2022) Cody Watson, Nathan Cooper, David Nader Palacio, Kevin Moran, and Denys Poshyvanyk. 2022\.  A systematic literature review on the use of deep learning in software engineering research.  _ACM Transactions on Software Engineering and Methodology_ (2022).
  * White et al. (2019) Martin White, Michele Tufano, Matías Martínez, Martin Monperrus, and Denys Poshyvanyk. 2019\.  Sorting and Transforming Program Repair Ingredients via Deep Learning Code Similarities. In _2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER)_.
  * White et al. (2016) Martin White, Michele Tufano, Christopher Vendome, and Denys Poshyvanyk. 2016.  Deep learning code fragments for code clone detection. In _Proceedings of the 31st IEEE/ACM international conference on automated software engineering_.
  * Wilde (2016) Michelle Wilde. 2016\.  Ieee xplore digital library.  _The Charleston Advisor_ (2016).
  * Wong et al. (2011) W Eric Wong, Vidroha Debroy, Richard Golden, Xiaofeng Xu, and Bhavani Thuraisingham. 2011\.  Effective software fault localization using an RBF neural network.  _IEEE Transactions on Reliability_ (2011).
  * Wong et al. (2016) W. Eric Wong, Ruizhi Gao, Yihao Li, Rui Abreu, and Franz Wotawa. 2016.  A Survey on Software Fault Localization.  _IEEE Transactions on Software Engineering_ (2016).
  * Wu et al. (2017) Qing Wu, Yungang Liu, Qiang Li, Shaoli Jin, and Fengzhong Li. 2017.  The application of deep learning in computer vision. IEEE.
  * Yang et al. (2022) Yanming Yang, Xin Xia, David Lo, and John Grundy. 2022\.  A Survey on Deep Learning for Software Engineering.  _ACM Comput. Surv._ , Article 206 (2022), 73 pages.
  * Young et al. (2018) Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria. 2018.  Recent trends in deep learning based natural language processing.  _IEEE Computational intelligence magazine_ (2018).
  * Zheng et al. (2016) Wei Zheng, Desheng Hu, and Jing Wang. 2016.  Fault Localization Analysis Based on Deep Neural Network.  _Mathematical Problems in Engineering_ (2016).


# Understanding of Emotion Perception from Art

Digbalay Bose  {dbose@,somandep@,souvikku@,rlahiri<EMAIL_ADDRESS>Krishna
Somandepalli  {dbose@,somandep@,souvikku@,rlahiri<EMAIL_ADDRESS>Souvik
Kundu  {dbose@,somandep@,souvikku@,rlahiri<EMAIL_ADDRESS>Rimita Lahiri
{dbose@,somandep@,souvikku@,rlahiri<EMAIL_ADDRESS>Jonathan Gratch
{dbose@,somandep@,souvikku@,rlahiri<EMAIL_ADDRESS>USC Institute of
Creative Technologies  Shrikanth Narayanan
{dbose@,somandep@,souvikku@,rlahiri<EMAIL_ADDRESS>
###### Abstract

Computational modeling of the emotions evoked by art in humans is a
challenging problem because of the subjective and nuanced nature of art and
affective signals. In this paper, we consider the above-mentioned problem of
understanding emotions evoked in viewers by artwork using both text and visual
modalities. Specifically, we analyze images and the accompanying text captions
from the viewers expressing emotions as a multimodal classification task. Our
results show that single-stream multimodal transformer-based models like MMBT
and VisualBERT perform better compared to both image-only models and dual-
stream multimodal models having separate pathways for text and image
modalities. We also observe improvements in performance for extreme positive
and negative emotion classes, when a single-stream model like MMBT is compared
with a text-only transformer model like BERT.

##  1 Introduction

Visual art is a rich medium for expressing human thoughts and emotions. This
can manifest in multiple diverse forms; for example, consider the well-known
works such as the dreamy starry night canvas created by Van Gogh, the mythical
figurines of Boticelli’s Primavera or the mystical characters like Da Vinci’s
Mona Lisa. A common unifying factor of an artwork is its ability to convey and
evoke emotional reaction in a viewer. In the affective computing realm, prior
works like [6], [11], have explored computational models for understanding
emotion elicited by art works. However, modeling emotional responses evoked by
art works is especially challenged by the uncertainty associated with the
subjectivity across individual viewers’ felt experiences. Based on familiarity
with art styles, formative experiences and their present mental state and
mood, different persons might interpret the same artwork from multiple diverse
perspectives. Since language is a powerful tool for explaining subjective
emotional responses [13], one approach is to have natural language
explanations of why people felt a certain emotion upon looking at a piece of
art.

Figure 1: Example of different captions and emotion labels associated with
Monalisa painting. Image taken from https://www.wikiart.org/ and the captions
with their emotion labels are taken from Artemis dataset [1].

In this work, we investigate the problem of modeling emotion perception from
artwork both through the lens of text-only classification from the associated
explanations and by considering multimodal interactions of image regions and
the associated textual explanations. The experiments of this work are based on
the recently released Artemis dataset [1] that has 80K artwork images from
WikiArt111https://www.wikiart.org/ along with 9 emotion class annotations.
Artemis was considered because it includes around 400K subjective explanations
regarding emotion labels associated with artwork. A sample artwork with
associated emotional labels and explanations is shown in Fig 1.

##  2 Related work

Image and video based emotion classification. In the domain of modeling
(categorical) affect evoked from art images, a system trained on the widely
used stimuli set from the international affective picture system (IAPS) was
evaluated on art masterpieces in [21]. 4000 artwork from the Wikiart
repository were leveraged in [11] and annotated for a range of emotions along
with title and other aspects.

Text based emotion classification. Linguistic descriptions can be used in
affective analysis such as reasoning about an expressed felt experience. A
variety of computational text analysis methods and tools are available in the
literature, building on a variety of word dictionary based resources such as
LIWC[19]. With the advent of transformer based methods [20] in various natural
language processing tasks, BERT [4] based models have been explored for
classifying emotion labels e.g., from Reddit comments [3].

Multimodal vision-language modeling. Recently transformer based models that
use vision-language pretraining on large-scale vision datasets have been fine
tuned for variety of downstream tasks. Examples of vision-language based
models employing transformers for individual streams followed by fusion
include LXMERT [18], VILBERT [10]. Examples of single stream models based on
concatenation of textual and visual inputs include MMBT [7], VisualBERT [8].

##  3 Multimodal adaptation for evoked emotion understanding

Textual and visual information pertaining to evoked emotions associated with
artwork contain complementary information: perceptually relevant affective
cues in images, direct information in text descriptions about felt emotion. A
trained text classifier model like BERT[4] predicts sadness if only the
caption it makes me think of how things used to be and how simple life once
was is used. When the visual artwork image (Fig 2) is shown, then the
associated caption along with the artwork evokes a feeling of contentment.
Hence we explore a series of multimodal model adaptations for evoked emotion
prediction.

Figure 2: The caption associated with Ivan Aivazovsky’s art work from Artemis
Dataset is it makes me think of how things used to be and how simple life once
was. Image regions used for feature extraction are shown as black boxes with
associated labels.

###  3.1 Dual-stream models

For the dual-stream model baseline, we pool the embeddings of text tokens
associated with the captions and the region wise embeddings (avg-pooled) from
image as shown in Fig. 3(A). The pooled embeddings are then concatenated
followed by FC layers for emotion classification.

Figure 3: (A) Dual-stream early fusion model (Early fusion avg pool: Average
of word token embeddings used as text embedding, Early fusion first token:
First word token embedding used as pooled text embedding). (B) Dual-stream
multi-layer encoder late fusion model (Weighted late fusion). Here Q: Query,
K: Key, V: Value. Txt: Text caption embeddings, Im: Region wise image
embeddings (C) Single-stream configurable MMBT model (MMBT)

We further explore modeling of explicit interactions between captions and
image region features by using image-guided and text-guided streams followed
by a weighted late fusion scheme of predicted class probabilities from
individual streams. As shown in Fig. 3(B),we use multi-layer self-attention
based encoder block. The output embeddings from the encoder blocks of the
respective streams are further average pooled and passed through fully
connected (FC) layers for classification followed by fusion using weights
$\mathbf{w_{1}}$ and $\mathbf{w_{2}}$.

###  3.2 Single-stream models

For single-stream design of multimodal models, we rely on the multimodal
bitransformer model (MMBT) [7] consisting of 12 encoder layers and include a
configurable image encoder to extract region wise features, as shown in Fig.
3(C). Since the image and text features are combined together as a single
input to the encoder model, separate segment IDs are provided to the image
(segment ID = 0) and text (segment ID = 1) modalities. For individual
modalities i.e., image and text, the input embeddings are obtained as the sum
of token, positional and segment embeddings. The representation associated
with [CLS] (classification) token is used for downstream emotion
classification.

##  4 Experiments and Results

We conduct our experiments on the Artemis dataset [1]. Artemis is built on top
of 81,446 art work images from the Wikiart dataset, spanning 27 art styles and
1119 artists from the 15th to 21st century. We perform the same train, test,
and validation set split as in [1].

###  4.1 Experimental Setup

We use categorical cross entropy as loss function with label smoothing [12] as
a regularization technique while training the multimodal models. We use the
AdamW optimizer [9] and linear scheduler with warm-up (warm-up steps varied
between 1000 to 10000). The learning rate is tuned in the range {1e-5, 6e-5}
in steps of 1e-5. For text-based baselines, we fine-tune BERT [4] to predict
emotion classes from the text captions. For the multi-layer encoder design in
dual stream weighted later fusion model, we use 5 encoder layers and 8 heads.
For the image modality, we extract bottom-up features [2] from the top-$k$
salient image regions (sorted by scores) by using a Visual-Genome based
pretrained Faster RCNN [14] model with a ResNet-101[5] backbone. We also look
at ResNeXt-152 C4 based object detection model [22] to extract enhanced
region-wise representations (VinVL). For the image based models, we fine-tune
VGG-16[17] and ResNet-50 [5] pretrained on ImageNet [15] by using a KL-
divergence loss between the network outputs and per-image distribution of
emotions based on responses of annotators.

###  4.2 Results

We compare performances in terms of 9 class accuracies and macro-F1 for image-
only, text-only and multimodal models. After tuning on the validation set, the
optimal weights for late fusion in the dual stream multi layer encoder model
are obtained as $\mathbf{w_{1}}=0.76$ and $\mathbf{w_{2}}=0.24$ for the image-
and text-guided streams respectively.

Model | Acc | F1 | Feat
---|---|---|---
[HTML]DAE8FCImage (N = 79327)
VGG-16 | 47.36 | 27.04 |
ResNet-50 | 44.98 | 21.31 |
[HTML]00D2CBText (N = 429431)
BERT | 66.2 | 61.42 |
[HTML]CBCEFBMultimodal (N = 429431)
Early fusion avg pool | 56.35 | 46.72 | BU+Bert
Early fusion first token | 56.98 | 48.34 | BU+Bert
Weighted late fusion | 65.14 | 60.27 | BU+Bert
MMBT | 66.33 | 62.24 | BU+Bert
VisualBERT | 66.03 | 61.47 | VinVL+Bert
Table 1: Performance comparison between image-only, text-only and multimodal
models. Under features column (Feat), BU+Bert indicates bottom up features
with 50 regions and token representations from BERT-base model. N indicates
number of samples used for the modalities.  Figure 4: Confusion matrix plots
for BERT (left) and MMBT (right). Red circles on the right confusion matrix
plot signify emotion classes where MMBT performs better as compared to BERT.

####  4.2.1 Performance variation with image features

We explore different configurations of bottom up and VinVL [22] features by
varying the number of detected boxes. From Table 2, it can be seen that there
is a slight improvement on usage of bottom up features ($k=50$), when compared
with VinVL [22]. However, the overall impact of varying the configuration of
region-wise features does not result in significant improvements in emotion
classification performance.

Img | Acc. | F1
---|---|---
BU-10 | 66.03 | 61.69
BU-30 | 66.07 | 61.39
BU-50 | 66.33 | 62.24
VinVL-10 | 66.26 | 61.79
VinVL-20 | 66.05 | 61.76
Table 2: Performance comparison(Accuracy and Macro-F1) for various input
feature configurations for MMBT model. The text token representations are
taken from pretrained BERT model. model-$k$: features from top-k bounding
boxes sorted by scores from BU(bottom-up) and VinVL.

###  4.3 Discussion

Figure 5: Visualization for image-only and multimodal models. Grad-cam
visualizations shown for image-only model (VGG-16) making correct predictions
for two cases involving (A) positive emotion (awe) and (B) negative emotion
(sadness). Important image regions marked for MMBT model in order of
increasing importance scores (1), (2), (3) for correct prediction.

As seen from Table 1, the MMBT model (initialized with pretrained BERT
weights) along with bottom up region features ($k$ = 50) performs on par with
the text-only finetuned BERT baseline. Further, as seen in Fig. 4, MMBT shows
an improvement in performance for extreme positive and negative emotion
classes with overt visual features: amusement, awe, anger and disgust. When
compared with dual stream models that rely on explicit modeling of
relationships between text tokens and region-wise features, single stream
models like MMBT, VisualBERT also perform better. For image-only models,
accuracies and F1-scores are reported based on the predicted dominant emotion
label (in terms of occurrences). Their inferior performance can be attributed
to the challenge involved in predicting a single emotion label from an art
work due to inherent subjectivity and multiple possible interpretations.

###  4.4 Visualizations

We provide visualizations for image-only model i.e. VGG16 using Grad-CAM [16].
As shown in Fig 5(A), for ground truth majority class “awe”, the VGG-16
network is focusing on the mountains and the sky region while predicting the
correct majority class. In case of “sadness” label shown in Fig 5(B), the
network focuses more on the expression shown on the man’s face, while
predicting the correct label “sadness”. For the MMBT model, since we rely on
region wise embeddings from images, we compute the importance of individual
image regions using gradient based attributions. We normalize the importance
of the image regions by dividing the individual importance scores by the
maximum important score and showcase the top-3 important regions in the image,
marked as (1), (2) and (3).

##  5 Conclusions

In this study, we explore multiple approaches for predicting evoked emotions
from art-works through the perspective of different modalities. Based on the
results, it can be seen that single-stream multimodal models like MMBT and
VisualBERT perform better when compared to dual-stream approaches. In terms of
visual information for multimodal analysis, including region-based features
results in improvements over the text-only baseline i.e., BERT. Future
directions include extraction of holistic image features based on art-styles,
color, lighting and modeling their interactions with the textual captions to
classify evoked emotions.

## References

  * [1] Panos Achlioptas, Maks Ovsjanikov, Kilichbek Haydarov, Mohamed Elhoseiny, and Leonidas J. Guibas.  Artemis: Affective language for visual art.  CoRR, abs/2101.07396, 2021.
  * [2] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang.  Bottom-up and top-down attention for image captioning and visual question answering.  In CVPR, 2018.
  * [3] Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi.  GoEmotions: A Dataset of Fine-Grained Emotions.  In 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020.
  * [4] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * [5] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  arXiv preprint arXiv:1512.03385, 2015.
  * [6] Jia Jia, Sen Wu, Xiaohui Wang, Peiyun Hu, Lianhong Cai, and Jie Tang.  Can we understand van gogh’s mood? learning to infer affects from images in social networks.  In Proceedings of the 20th ACM International Conference on Multimedia, MM ’12, page 857–860, New York, NY, USA, 2012. Association for Computing Machinery.
  * [7] Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, and Davide Testuggine.  Supervised multimodal bitransformers for classifying images and text.  arXiv preprint arXiv:1909.02950, 2019.
  * [8] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  Visualbert: A simple and performant baseline for vision and language, 2019\.
  * [9] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
  * [10] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  arXiv preprint arXiv:1908.02265, 2019.
  * [11] Saif M. Mohammad and Svetlana Kiritchenko.  An annotated dataset of emotions evoked by art.  In Proceedings of the 11th Edition of the Language Resources and Evaluation Conference (LREC-2018), Miyazaki, Japan, 2018.
  * [12] Rafael Müller, Simon Kornblith, and Geoffrey E Hinton.  When does label smoothing help?  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [13] Andrew Ortony, Gerald L Clore, and Allan Collins.  The cognitive structure of emotions.  Cambridge university press, 1990.
  * [14] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
  * [15] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  International Journal of Computer Vision (IJCV), 115(3):211--252, 2015.
  * [16] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
  * [17] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In International Conference on Learning Representations, 2015.
  * [18] Hao Tan and Mohit Bansal.  Lxmert: Learning cross-modality encoder representations from transformers.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, 2019.
  * [19] Yla R. Tausczik and James W. Pennebaker.  The psychological meaning of words: Liwc and computerized text analysis methods.  Journal of Language and Social Psychology, 29(1):24--54, 2010.
  * [20] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  arXiv preprint arXiv:1706.03762, 2017.
  * [21] Victoria Yanulevskaya, Jan C van Gemert, Katharina Roth, Ann-Katrin Herbold, Nicu Sebe, and Jan-Mark Geusebroek.  Emotional valence categorization using holistic image features.  In 2008 15th IEEE international conference on Image Processing, pages 101--104. IEEE, 2008.
  * [22] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao.  Vinvl: Revisiting visual representations in vision-language models.  In CVPR 2021, June 2021.