∎

11institutetext: Mehil Shah 22institutetext: Dalhousie University, Canada
22email<EMAIL_ADDRESS>33institutetext: Mohammad Masudur Rahman
44institutetext: Dalhousie University, Canada
44email<EMAIL_ADDRESS>55institutetext: Foutse Khomh 66institutetext:
Polytechnique Montreal, Canada
66email<EMAIL_ADDRESS>
# Towards Enhancing the Reproducibility of Deep Learning Bugs: An Empirical
Study

Mehil B. Shah  Mohammad Masudur Rahman  Foutse Khomh

(Received: date / Accepted: date)

###### Abstract

Context: Deep learning has achieved remarkable progress in various domains.
However, like any software system, deep learning systems contain bugs, some of
which can have severe impacts, as evidenced by crashes involving autonomous
vehicles. Despite substantial advancements in deep learning techniques, little
research has focused on reproducing deep learning bugs, which hinders
resolving them. Existing literature suggests that only 3% of deep learning
bugs are reproducible, underscoring the need for further research.

Objective: This paper examines the reproducibility of deep learning bugs. We
identify edit actions and useful information that could improve deep learning
bug reproducibility.

Method: First, we construct a dataset of 668 deep learning bugs from Stack
Overflow and Defects4ML across three frameworks and 22 architectures. Second,
out of the 668 bugs, we select 102 bugs using stratified sampling and try to
determine their reproducibility. While reproducing these bugs, we identify
edit actions and useful information necessary for their reproduction. Third,
we used the Apriori algorithm to identify useful information and edit actions
required to reproduce specific bug types. Finally, we conduct a user study
with 22 developers to assess the effectiveness of our findings in real-life
settings.

Results: We successfully reproduced 85 out of 102 bugs attempted. We
identified ten edit actions and five useful information categories that can
help us reproduce the deep learning bugs. Our findings improved bug
reproducibility by 22.92% and reduced reproduction time by 24.35% based on a
user study with 22 developers.

Conclusions: Our research addresses the critical issue of deep learning bug
reproducibility. Practitioners and researchers can leverage our findings to
improve deep learning bug reproducibility.

##  1 Introduction

Deep learning (hereby DL) has been widely used in many application domains,
including natural language processing, finance, cybersecurity med1 ; fin1 ;
cyb1 , autonomous vehicles pei2017 ; ma2018 , and healthcare systems
esteva2019guide .

Like any software system, deep learning systems are prone to bugs. Bugs in
deep learning systems could arise from various sources, including errors in
the dataset, incorrect hyperparameters, and incorrect structure of the deep
learning model BRAIEK2020110542 . These bugs could lead to program failures,
poor performance, or incorrect functionality, as reported by existing
literature islamfse19 . They can also lead to serious consequences, as shown
by the fatal crash involving Uber’s self-driving car selfdrivingcarcrash .
Thus, we must fix the bugs before deploying a deep learning model in
production. However, one must reproduce the bugs before fixing them to verify
their presence/absence in the system. Unfortunately, reproducing DL bugs is
challenging due to deep learning systems’ multifaceted nature and
dependencies, which encompass data, hardware, libraries, frameworks, and
client programs. Furthermore, deep learning systems are inherently non-
deterministic (i.e., random weight initialization), which leads to different
outcomes across multiple runs and thus makes the reproduction of deep learning
bugs challenging nagarajan2018impact . Moreover, they also suffer from a lack
of interpretability krishnan2020against , which makes the reproduction of deep
learning bugs challenging in comparison to traditional software bugs.
According to existing investigations defects4ml , only 3% of DL bugs are
reproducible, further demonstrating the challenges in reproducing DL bugs.

Existing literature investigates the challenges of reproducing programming
errors or bugs from various sources. Mondal et al. msr2019mondal  investigate
the challenges in reproducing programming issues reported on Stack Overflow
and suggest several edit actions to help reproduce them. Rahman et al.
icsme2020rahman  conduct a multi-modal study to understand the factors behind
the non-reproducibility of software bugs. They identify 11 significant factors
behind bug non-reproducibility, including missing information, bug
duplication, false positive bugs, and intermittency. Overall, these studies
highlight the challenges in reproducing traditional software bugs but do not
deal with any deep learning bugs, which warrants further investigation.

Recently, a few studies have attempted to tackle the challenges of deep
learning bugs. Liang et al. liang2022gdefects4dl  provide a dataset of 64 deep
learning bugs collected from GitHub issues. They classify these bugs into six
categories according to the taxonomy of Humbatova et al. taxonomyRealFaults .
Moravati et al. defects4ml  constructed a benchmark dataset containing 100
deep learning bugs collected from StackOverflow and GitHub; they reproduced
each of them. Although these studies offer benchmark datasets containing
reproducible bugs, their primary focus is dataset construction. They do not
report any detailed instructions (e.g., sequence of actions) essential to the
reproduction of the deep learning bugs. Our work attempts to fill this
important gap in the literature.

In this paper, we conduct an empirical study to better understand the
challenges in reproducing deep learning bugs. First, we collect a total of 568
DL bugs from Stack Overflow posts. Then, we extend them with 100 DL bugs from
the benchmark dataset of Moravati et al. defects4ml , which makes up our final
dataset of 668 DL bugs. Using the taxonomy of Humbatova et al. defects4ml , we
divide these bugs into five categories: 167 model, 213 tensor, 145 training,
113 GPU, and 30 API bugs. Second, by using stratified sampling, we select 102
of these bugs and determine the reproducibility status of these bugs by
attempting to reproduce them using the code snippet and complementary
information from Stack Overflow and the benchmark dataset. Third, the first
author manually analyzed the produced artefacts to determine the categories of
information that are useful for deep learning bug reproduction. Fourth, we use
the information gathered during the bug reproduction and create a dataset of
transactions that link the type of bug with edit actions and information
categories. Then, our study establishes connections among the information
categories, edit actions, and the type of bugs by employing the Apriori
algorithm apriori . Finally, to further validate our findings, we conducted a
developer study with 22 participants from industry and academia. Half of the
participants were asked to reproduce bugs using our identified information,
while the other half were asked to reproduce the same bugs without access to
that information. Results from the user study show that our recommended edit
actions and information categories can reduce the time to reproduce the bug by
24.35%. Thus, we answer three important research questions as follows:

  * •

RQ1: Which edit actions are useful for reproducing deep learning bugs?
Determining the key edit actions that are crucial for reproducing deep
learning bugs is important since almost none of the bugs can be reproduced
using the verbatim code snippet. By manually reproducing 85 deep learning
bugs, we identify ten key edit actions that could be useful to reproduce deep
learning bugs (e.g., input data generation, neural network construction,
hyperparameter initialization).

  * •

RQ2: What information categories and edit actions are useful for reproducing
specific types of deep learning bugs?
Different types of DL bugs need different types of information or edit
actions, which warrants further investigation. Using the Apriori algorithm, we
have determined the top 3 pieces of information and the top 5 edit actions
that can help one reproduce each type of bug. These insights can be used not
only to detect the missing information in a submitted bug report but also to
formulate the follow-up questions soliciting the missing information.

  * •

RQ3: Do the suggested edit actions and identified information categories
contribute to enhancing the reproducibility of deep learning bugs in practice?
To assess the effectiveness of our suggested edit actions and identified
information categories in improving bug reproducibility, we conducted a user
study involving 22 professional developers. In our developer study, the
participants assigned to the experimental group used our suggested edit
actions and information categories to reproduce deep learning bugs. In
contrast, the participants in the control group reproduced the bugs without
access to the suggested edit actions and information categories. Based on the
feedback from the developers, we found that our recommended edit actions and
highlighted information categories (a) helped them reproduce 22.92% more bugs
and (b) decreased their time to reproduce the deep learning bugs by 24.35%.

The remainder of the paper is organized as follows. Section 2 presents a
motivating example that highlights the challenges in reproducing deep learning
bugs using existing techniques. Section 3 then describes our study methodology
in detail, including data collection, dataset construction, environment setup,
qualitative analysis, and user study design. Next, Section 4 presents the
findings by answering our three research questions. It summarizes the key edit
actions, useful information categories, and their relationships with bug types
that enhance deep learning bug reproducibility. Section 5 and Section 6
provide a discussion of our results, including a proof-of-concept tool,
comparison with language models and threats to validity. Finally, Section 7
reviews related literature, and Section 8 concludes the paper.

##  2 Motivating Example

Bug reports or programming Q&A posts might not always provide sufficient
information to reproduce a deep learning bug. Let us consider the example
question shown in Fig. LABEL:fig:irreproducibleMotivatingExample
motivatingExample1 . Here, the user attempts to train a neural network using
Keras. The input data type is a $Sequence$ object, and the user aims to pass
the training dataset to the $fit\\_generator()$ method. However, s/he
discovers that their $Sequence$ object is not recognized by the Keras library.
The user also provides a code snippet to aid the reproducibility of the bug.
Unfortunately, the issue cannot be reproduced since the provided information
does not contain any dataset for training, network definition,
hyperparameters, or required dependencies. In Stack Overflow, this question
has failed to receive a precise response. Even though the above code (Fig.
LABEL:fig:irreproducibleMotivatingExample) could be made compilable or
executable using edit actions, the bug cannot be reproduced due to its complex
nature. Since the earlier study msr2019mondal  does not deal with any deep
learning bugs, their suggested edits might also not be effective.

Let us consider another example question shown in Fig.
LABEL:fig:motivatingExample motivatingExample2 . Here, the user wants to
obtain the confusion matrix from a multi-class classification model.
Unfortunately, s/he runs into a runtime error, as the $confusion$
$\\_matrix()$ method does not support multiple output labels. To reproduce the
bug, we first determine the type of this bug. By manually analyzing the Stack
Overflow post, code snippet, and the corresponding stack trace, we found that
the error was related to the incorrect shape of data being passed to the
$confusion\\_matrix()$ method. Since the bug was triggered by a wrong shape of
the training dataset, it falls into the ‘Tensor’ category according to the
taxonomy of Humbatova et al. taxonomyRealFaults . Then, we check the useful
information suggested by our study (see RQ2) for reproducing tensor bugs. The
top 3 items are Data, Logs, and Code Snippet, as highlighted in Table. 7.
Although the question in Fig. LABEL:fig:motivatingExample contains the
relevant code snippet and logs, it does not contain any information about the
dataset for training, which is the most useful information for reproducing
tensor bugs according to our findings (RQ2). Subsequently, we also check the
recommended edit actions to reproduce a tensor bug. From Table 8, we use the
top 3 edit actions – Input Data Generation, Import Addition, and
Hyperparameter Initialization. When these edit actions and information
categories were applied to the code, we were able to successfully reproduce
the bug, which indicates the effectiveness of our suggested actions and
information. Similarly, this issue was reproduced by other users of Stack
Overflow, and the question received a correct solution within two hours of its
submission, as shown in Fig. LABEL:fig:AcceptedAnswer.

##  3 Study Methodology

Fig. 2 shows the schematic diagram of our empirical study. We discuss
different steps of our study as follows.

###  3.1 Selection of Data Sources

We select Stack Overflow as a primary data source for our study. It is the
largest programming Q&A site for programming topics containing over 24 million
questions and 35 million answers SOUsageData . Thus, Stack Overflow could be a
potential source for deep learning (DL) bugs. Developers often submit their
encountered problems on Stack Overflow, when building their deep learning
applications. According to a recent work zhao2021state , Stack Overflow
contains at least 30 topics and 80K posts related to deep learning issues,
which makes it a potential source for our data. We also select the benchmark
dataset of Moravati et al. defects4ml  containing 100 DL bugs as our second
source of data.

Figure 2: Schematic diagram of our empirical study

To gather relevant posts from Stack Overflow, we use the Stack Exchange Data
Explorer platform111https://data.stackexchange.com/stackoverflow/query/new. We
employ multiple filters to capture relevant posts discussing deep learning
bugs. First, we select the recent posts (i.e., submitted between May 2020 and
May 2023) with the following tags: ‘tensorflow’, ‘keras’, and ‘pytorch’. These
tags were selected since they represent the most frequently used frameworks
for deep learning zhang2019empirical . This filtration resulted in 14,065
posts for PyTorch, 14,971 posts for Tensorflow, and 3,152 posts for Keras. We
then applied the following four filtration criteria to remove conceptual
questions from our collection:

  * •

Keyword Filtering: This filtration removes the how-to questions and the
questions requesting installation instructions. To filter these questions, we
use appropriate keywords (‘how’, ‘install’, and ‘build’), as recommended by
Humbatova et al. taxonomyRealFaults .

  * •

Code Snippet: Given our focus on the reproducibility of reported issues, the
questions must include code segments. Therefore, we consider such questions
that contain at least one line of code, as suggested by Mondal et al.
msr2019mondal .

  * •

Accepted Answer: We only select the questions with accepted answers, ensuring
that each reported issue (e.g., DL bug) was reproduced and fixed, as advised
by Humbatova et al. taxonomyRealFaults .

  * •

Negative Question Score: This filtration helps us discard questions that the
community has found to be of low quality, as proposed by Ponzanelli et al.
ponzanelli2014 .

We combine all four filtration criteria above, construct a SQL query, and then
execute the query against the Stack Exchange Data Explorer. The SQL query is
available in our replication package replicationPackage . After this
filtration step, we obtained a total of 279 posts for Keras, 1,433 posts for
Tensorflow, and 1,700 posts for PyTorch. To ensure the validity of our chosen
filtration criteria and that the posts are related to DL bugs, we conducted a
follow-up manual analysis on a representative sample of 75 bugs. We select the
sample for the manual analysis using a 90% confidence interval and a 10%
margin of error. We analyze the entire Stack Overflow discussion and code
changes in GitHub for each of these bugs, and we found that 96% of them
discuss deep learning bugs, confirming the validity of our filtration criteria
(Steps 1-2, Fig. 2).

###  3.2 Dataset Construction

The above filtration step resulted in 3,412 Stack Overflow posts. To construct
our final dataset, we apply another filter to these posts where we leverage
the tags from Humbatova et al. taxonomyRealFaults . Our idea was to select
representative samples from different types of deep learning bugs. To reduce
the risk of miscategorization, we thus only selected the tags in the
taxonomies and sub-taxonomies proposed by Humbatova et al. Table. 1 shows our
selected 3 tags for model bugs, 1 for tensor bugs, 11 for training bugs, 3 for
GPU bugs, and 7 for API bugs. We look for these tags in the above Stack
Overflow posts and collect 113 Model bugs, 193 Tensor Bugs, 95 Training Bugs,
101 GPU Bugs, and 24 API Bugs. We also discover 42 bugs belonging to multiple
categories (e.g., 5 Model & Tensor Bugs, 5 Tensor & GPU Bugs, 3 Training & API
Bugs). Thus, the final dataset contains a total of 668 bugs (568 from our
dataset + 100 from Defects4ML defects4ml ) and captures a balanced
representation from different types of DL bugs, as shown in Table 2 (Step 3,
Fig. 2).

Table 1: Tags used for filtering different types of bugs

Type of Bug | Tags
---|---
Model | layer, model, activation-function
Tensor | tensor
Training | loss-function, training-data, optimization, loss,
| data-augmentation, performance, learning-rate,
| hyperparameters, initialization, imbalanced-data, nan
GPU | gpu, nvidia, cuda
API | typeerror, valueerror, attributeerror, importerror,
| compilererrors, syntaxerror, modulenotfounderror

Table 2: Summary of the constructed dataset

Type of Bugs | Total Number of Bugs
---|---
Model | 113
Training | 95
GPU | 101
API | 24
Tensor and Input | 193
Mixed | 42
Total | 568

###  3.3 Environment Setup

For our experiments, we use the following environment setup:

  * •

Code Editors: We use Visual Studio Code
v1.79.0222https://code.visualstudio.com/ and PyCharm 2023.1.1
333https://www.jetbrains.com/pycharm/ to execute code snippets and reproduce
bugs from our dataset. Visual Studio Code and PyCharm are popular code editors
for building DL-based applications GeeksforGeeks_ide_dl .

  * •

Dependencies: To detect the API libraries adopted by the code snippets, we use
the pipreqs package444https://pypi.org/project/pipreqs/. We also install the
dependencies for each bug into a separate virtual environment using the
venv555https://docs.python.org/3/library/venv.html module.

  * •

Frameworks: Since our dataset contains bugs from Tensorflow, Keras, and
PyTorch, we used all three frameworks in our experiments.

  * •

Libraries: For generating the random inputs and visualizing the training
metrics, we leveraged several scientific computing libraries such as numpy,
pandas, and matplotlib.

  * •

Python Version: During our experimentation, we used Python v3.10 to reproduce
the deep learning bugs, the latest stable version. If the Stack Overflow issue
reported a Python version other than v3.10, we reproduce the bug using the
mentioned version instead.

  * •

Hardware Config: Our experiments were run on a desktop computer having a
64-bit Windows 11 Operating System with 16GB primary memory (i.e., RAM) and
8GB GPU Memory (Intel(R) Iris XE Graphics).

###  3.4 Qualitative Analysis

We follow a two-step approach to reproduce the deep learning issues reported
in the studied Stack Overflow posts. First, we gather complementary
information about the issue, such as the dataset, library versions, and the
framework used. Second, we attempt to reproduce the issue using the code
snippet and supporting data (e.g., dataset information, environment
configurations, hyperparameters, and training logs). Following the successful
reproduction of a bug, we capture various information related to each bug,
such as the deep learning architecture involved, edit actions used to
reproduce the issue, time taken to reproduce the issue, type of issue
reproduced, and the type of information present in the Stack Overflow post.
All these data gathered during issue reproduction will help us determine the
useful information and edit actions needed to reproduce specific types of deep
learning bugs (Step 4, Fig. 2).

Table 3: Edit Actions for Reproducing Deep Learning Bugs Edit Action |  Overview
---|---
A1: Input Data Generation (D) |  Generating input data which simulates the data used for training the model.
A2: Neural Network Construction (N) |  Reconstructing or modifying the neural network based on the information provided.
A3: Hyperparameter Initialization (H) |  Initializing the hyperparameters for training, such as batch size and number of epochs.
A4: Import Addition and Dependency Resolution (R) |  Determining the dependencies in the code snippet and adding the missing import statements.
A5: Logging (L) |  Adding appropriate logging statements to capture relevant information during reproduction.
A6: Obsolete Parameter Removal (O) |  Removing outdated parameters or functions to match the parameters of the latest library versions.
A7: Compiler Error Resolution (C) |  Debugging and resolving compiler errors that arise due to syntactic errors in the provided code snippet.
A8: Dataset Procurement (P) |  Acquiring the necessary datasets and using them to train the model.
A9: Downloading Models & Tokenizers (M) |  Fetching pre-trained models and tokenizers from external sources.
A10: Version Migration (V) |  Updating code to adapt to changes introduced in newer library or framework versions.

###  3.5 Identifying Type Specific Information and Edit Actions

####  3.5.1 Algorithm Selection

To establish a relationship among the bug types, information categories, and
the key editing actions necessary for their reproduction, we use the Apriori
apriori  algorithm. Apriori is a well-known algorithm for mining frequent item
sets from a list of transactions. It helps one identify common patterns and
associations between different elements. In our context, we apply the Apriori
algorithm to analyze the information gathered during bug reproduction. Our
goal was to determine the frequent combinations of bug types, information
categories, and edit actions during bug reproduction. We leveraged the Apriori
algorithm’s systematic pattern-mining capabilities to establish these
relationships.

####  3.5.2 Generating the Transactions

To create our datasets for the Apriori algorithm, we employed a character
encoding, where we encoded all labels into a unique character (e.g., ‘Training
Bug’ was encoded as ‘T’, ‘Model Bug’ was encoded as ‘M’, ‘Obsolete Parameter
Removal’ was encoded as ‘O’) and converted the data into transactions using
the following format.

$\text{{Bug Type}}\rightarrow\text{{Information Category}}$

Example: $T\rightarrow DH$
Description: The transaction above indicates that the reproduced bug is a
training bug (T). The corresponding bug report contains useful information
about the bug, such as the dataset used for training (D) and hyperparameters
used by the model (H).

$\text{{Bug Type}}\rightarrow\text{{Edit Action}}$

Example: $M\rightarrow OLN$
Description: The transaction above indicates that the reproduced bug is a
model bug (M). To reproduce the bug, we performed three edit actions, as
described below:

  * •

Obsolete Parameter Removal (O): We removed some of the parameters that were
absent in the recent library and framework version to ensure that the code
compiles.

  * •

Logging (L): We logged various intermediate program states to verify the bug’s
presence.

  * •

Neural Network Definition (N): We reconstructed the neural network based on
the information provided in the bug report.

Following the specified format, we created two datasets of transactions that
associate bug types with crucial information and edit actions, respectively.
After creating the transactions, we use the Apriori algorithm to compute the
support and confidence for our generated itemsets and association rules. We
talk about these metrics in detail below.

####  3.5.3 Metrics for Apriori Algorithm

Support is the proportion of transactions in the dataset that contain a
particular item set. Mathematically, the support of an itemset X is defined as
the ratio of the number of transactions containing X to the total number of
transactions. It is expressed as:

$\text{Support}(X)=\frac{\text{Transactions containing }X}{\text{Total number
of transactions}}$

For a rule $X\Rightarrow Y$, where $X$ and $Y$ are two itemsets, the support
is calculated for the combined itemset $X\cup Y$.

Confidence measures the likelihood that a rule $X\Rightarrow Y$ holds. It is
defined as the ratio of the support of the combined itemset $X\cup Y$ to the
support of the antecedent itemset $X$. Mathematically, confidence is expressed
as:

$\text{Confidence}(X\Rightarrow Y)=\frac{\text{Support}(X\cup
Y)}{\text{Support}(X)}$

Confidence values range from 0 to 1. A high confidence value indicates a
strong association between antecedents and consequent itemsets.

####  3.5.4 Association Rule Generation

We conducted two separate association rule mining operations in our study -
one focused on information categories while the other focused on edit actions
used to reproduce deep learning bugs. In particular, we extracted 27 itemsets
and 34 rules for information categories, highlighting the useful information
for reproducing deep learning bugs. Similarly, we extracted 126 item sets and
284 rules for edit actions, capturing the edit actions to reproduce bugs. We
generated association rules based on the entries in our dataset and did not
filter or remove any rules before determining confidence values. We then
calculate the confidence values for all generated rules to identify the most
influential ones for connecting bug types with edit actions and useful
information.

####  3.5.5 Computation of Confidence Values for the Generated Association
Rules

As discussed earlier, confidence indicates how frequently a rule occurs, while
support indicates the generality of the rule. To compute the confidence values
for each association rule, we performed the following steps:

  * •

Calculate Support for Antecedent ($X$): We calculate the support for the
antecedent, which is the bug type in our case (e.g., ‘T’ for Training Bug or
‘M’ for Model Bug). Support for $X$ is the proportion of all transactions that
contain the specific bug type.

  * •

Calculate Support for Combined Itemset ($X\cup Y$): We then calculate the
support for the combined itemset, $X\cup Y$, which includes both the bug type
and the information category or edit action (e.g., ‘T $\cup$ H’ for Training
Bug associated with Hyperparameter Information, or ‘M $\cup$ O’ for Model Bug
associated with Obsolete Parameter Removal).

  * •

Compute Confidence for the Rule ($X\Rightarrow Y$): The confidence of the rule
$X\Rightarrow Y$ is computed by dividing the support of the combined itemset
$X\cup Y$ by the support of the antecedent $X$. This step gives us the
confidence value, which indicates how often the information category or edit
action $Y$ is associated with the bug type $X$ in our transactions.

For example, consider the rule $T\Rightarrow D$. This rule indicates that if
the type of bug is a ‘Training Bug’ (T), it can be reproduced by the edit
action ‘Input Data Generation’ (D). To calculate the confidence of this rule,
we count the number of transactions in which the training bug is reproduced by
using the edit action ‘Input Data Generation’ and divide this count by the
total number of transactions involving training bugs in the dataset.

####  3.5.6 Identification of High Confidence Associations

We use high confidence and support values to detect the rules that reliably
capture the core factors necessary to reproduce specific types of deep
learning bugs. Based on these high-confidence rules, we identify the top 3
pieces of useful information and the top 5 edit actions used to reproduce each
bug type. The decision to select three useful pieces of information and five
edit actions was influenced by two key factors. First, we adhere to the
Parsimony Principle gori2023machine , which suggests that selecting the
simplest set of rules is preferable when multiple rules can predict or
describe the same phenomenon. We thus concentrate on the most significant
factors by selecting the top 3 and top 5 rules for edit actions and useful
information, respectively. Second, we filter the rules based on minimum
confidence values of 30%, as suggested by Liu et al. liu1999mining . With our
limited dataset, a 30% confidence threshold can indicate a substantial pattern
since finding associations in 30% of cases points to a meaningful correlation
given the data size. Furthermore, the 30% minimum confidence helps filter out
spurious correlations with small datasets that can occur by chance. Therefore,
for our dataset, the selected threshold strikes an effective balance - it is
high enough to identify meaningful associations in the data while eliminating
noise from false correlations. Overall, this filtration left us with 23 rules
for edit actions and 20 for useful information. Focusing on these high-
confidence, high-support rules can reveal the patterns that reproduce deep
learning bugs (Step 5a, 5b, Fig. 2).

###  3.6 User Study

To assess the benefits and implications of our findings in a real-life
setting, we conduct a user study involving 22 users (10 from academia + 12
from industry) (Step 6, Fig. 2). We discuss our study setup, including
instrument design and participation selection, as follows:

Instrument Design: We divided our questionnaire into three sections. We
discuss them in detail below:

  * •

Introduction: We first summarize our findings on the reproducibility of deep
learning bugs to provide the developers with the necessary context and
background information.

  * •

Demographic Information: After providing the contextual information about our
study, we collect demographic information from developers (e.g., experience
bug fixing in deep learning frameworks). We then ask the developers to
elaborate on the challenges that they face when reproducing deep learning bugs
in their daily lives.

  * •

Questionnaire Preparation: First, we select eight bugs (2 Tensor, 2 API, 2
Model, and 2 Training Bugs) from our dataset. Second, we prepare four sets of
2 bugs each by randomly selecting bugs from those eight bugs. Third, we
provide the users with the issue description and the code snippet from the
original Stack Overflow post. We also provide a Google Colaboratory notebook
containing the code for sample edit operations to aid the bug reproduction
process. Finally, we include a free-text box in the form to allow the users to
share any additional information about edit actions not covered in our study.

Study Session: During our study session, each participant completes the
following five tasks. First, the participant provides their demographic
information. Second, the participant explains their daily challenges when
reproducing deep learning bugs. Third, the participant reproduces two deep
learning bugs and reports the edit actions and information they used to
reproduce the bugs. Fourth, the participant also provides the rationale behind
their used edit actions and information categories to reproduce the bugs.
Finally, the participant provides information about any other edit actions
that they might have used to reproduce the bugs but are not covered in our
study.

Table 4: Distribution of participants in the control and experimental groups

DL Experience | Developers in Control Group | Developers in Experimental Group
---|---|---
<1 Year | 3 (27.27%) | 3 (27.27%)
1-5 Years | 6 (54.54%) | 5 (45.45%)
5-10 Years | 2 (18.18%) | 3 (27.27%)

Participant Selection: We first conducted a pilot study with two researchers
and two developers. Based on their feedback, we rephrased ambiguous questions
and added sample code to aid the manual bug reproduction by the users.
Incorporating this constructive feedback enabled us to refine and improve the
quality of our final questionnaire. Then, we invite professional developers
and researchers with relevant deep learning experience to our study. We send
our invitations to the potential participants using direct correspondences,
organization mailing lists (e.g., Mozilla Firefox), and public forums (e.g.,
LinkedIn and Twitter). A total of 22 participants responded to our
invitations. Out of them, 10 (45%) came from academia, and 12 (55%) came from
industry. In terms of bug-fixing experience with deep learning bugs, 14
participants (63.63%) had 1-5 years of experience, 4 (18.18%) had 5-10 years
of experience, and the remaining 4 (18.18%) had less than one year of
experience. In terms of deep learning frameworks, 19 participants (86.34%)
reported having working experience with Tensorflow, 21 (95.45%) reported
experience with PyTorch, and 20 (90.91%) reported experience with Keras. All
these statistics indicate a high level of cross-framework expertise within our
participants. We carefully divided the developers into control and experiment
groups, as shown in Table. 4. Our guiding principle was to ensure that both
groups had a similar distribution in their relevant experience. Specifically,
we surveyed all the developers about their experience and used a stratified
random sampling approach to assign developers to the two groups. This
randomization allowed us to minimize potential bias and confounding factors
across the groups. Using these two groups, we wanted to assess the benefit of
our recommended information in the context of bug reproduction. The control
group receives no hints about how to reproduce the bug. In contrast, the
experimental group receives hints (e.g., useful information, edit actions)
that could help them reproduce a bug.

Table 5: Summary of the Reproduced Bugs

Type of Bug | Bugs Reproduced | Model Architectures Covered
---|---|---
Training (T) | 26 | CNN, LSTM, AutoEncoder, MLP, RCNN,
|  | ResNet
Model (M) | 20 | BERT, CNN, GMM, LSTM, MLP
|  | VGG16, Transformers
API (A) | 11 | CNN, GAN, MLP, Transformers, VGG19,
|  | Variational RNN
GPU (G) | 3 | -
Tensor and Input (I) | 21 | CNN, GAN, Logistic Regression, MLP, ResNet
Mixed (X) | 4 | CNN, BERT, MLP

###  3.7 User Study Results Analysis

We perform an agreement analysis between our empirical findings and the
findings from the user study to assess their generalizability and strengthen
our understanding of the reproducibility of deep learning bugs. To perform the
agreement analysis, we compare the edit actions used in our manual bug
reproduction process with those used by participants in the user study. If
similar edit actions were used, it would indicate a strong agreement between
the empirical findings and user study results.

In our user study, the participants first reproduce their assigned bugs and
report their used information or actions to reproduce their bugs. To analyze
the effectiveness of our recommended information, we compare the control and
experimental groups in terms of their success rates in bug reproduction and
the time taken to reproduce the bugs. We select bug reproducibility rate and
time taken for bug reproduction as our key metrics to evaluate the
effectiveness of our findings. These quantitative metrics directly measure how
successful and efficient our recommended information is in assisting the
developers to reproduce deep learning bugs. Higher reproducibility rates and
reduced reproduction times in the experimental group compared to the control
group would indicate that our findings are effective for improving deep
learning bug reproducibility. We also analyze the qualitative feedback from
the participants to determine if provided hints were useful or not. We also
analyze them to uncover new insights into deep learning bug and their
reproduction.

By analyzing the user study results and the developers’ qualitative feedback,
we gain new and valuable insights into the reproducibility of deep learning
bugs. The user study results and feedback also provide information about the
effectiveness of our recommended actions for bug reproduction. This enables us
to deliver novel and actionable insights regarding the current state of deep
learning bug reproducibility based on empirical evidence (Step 7, Fig. 2).

##  4 Study Findings

In this section, we present the findings of our study by answering three
research questions as follows.

###  4.1 RQ1: Which edit actions are crucial for reproducing deep learning
bugs?

We reproduced 85 bugs of different types and architectures to answer the first
research question. Table. 5 briefly summarizes our reproduced bugs. Through
our comprehensive reproduction process, we identify ten edit actions that are
crucial for reproducing deep learning bugs. Table. 3 shows the identified
actions from our qualitative analysis. We explain these actions as follows.

Input Data Generation (A1) is one of the key edit actions for reproducing deep
learning bugs. This action involves generating input data that simulates the
data used for training. We generate the input data using the data shape and
distribution provided by the reporter in a bug report. Our manual bug
reproduction shows that $\approx$73% of our collected posts from Stack
Overflow have the relevant data characteristics. For example, the following
text from the Stack Overflow post (Issue #61781193) shows how the reporter
might submit the input data distribution.

R: My training data has input as a sequence of 80 numbers in which each
represent a word and target value is just a number between 1 and 3.

Using this information, we generated the random input data as follows and were
able to reproduce the corresponding bug.

⬇

train_data = torch.utils.data.TensorDataset(torch.randint(0, 200, (1000, 80)),
torch.randint(1, 3, (1000,)))

Neural Network Construction (A2) was one of the most used edit actions during
our bug reproduction. In this edit action, we construct a neural network based
on the architecture provided by the reporter. Similar to the data
characteristics, the information about the neural network is present in
$\approx$65% of the reproducible issue reports. Using the neural network
description from the issue reports, we were able to construct the models. For
example, the following text from the Stack Overflow post (Issue #63204176)
mentions the neural network but does not provide a code snippet for the
network.

R: I am trying to write a simple multinomial logistic regression using mnist
data.

Using this partial information, we constructed multiple multinomial logistic
regression models, and despite the lack of relevant code examples, we
successfully reproduced the corresponding bug.

Hyperparameter Initialization (A3) is one of the core edit actions for
reproducing deep learning bugs. As a part of this edit action, we initialize
various hyperparameters (e.g., number of epochs, batch size, optimizer) for
training the neural networks. Sometimes, the reporter did not provide these
configurations, and we had to initialize them using default parameters to
reproduce the bug. This is further evidenced by the fact that only
$\approx$53% of our collected posts from Stack Overflow have information about
the hyperparameters. For example, the following code snippet from the Stack
Overflow post (Issue #31880720) shows a case where the hyperparameters were
not initialized.

⬇

model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch,
show_accuracy=True, verbose=2, validation_data=(X_test, Y_test))

To reproduce the bug, we initialized the batch size with commonly-used values
$\\{32,64,128\\}$ and the number of epochs ranging from 1 to 10. Since we
initialized these hyperparameters, we ran the edited code snippet five times
to confirm the effectiveness of our edit operation. We were able to reproduce
the bug in all five iterations. We performed the steps mentioned above for all
37 bugs involving hyperparameter initialization.

Import Addition and Dependency Resolution (A4) are one of the most common edit
actions for all types of bugs. In this edit action, we analyze the code
manually and determine the dependencies required for the code snippet to run.
Then, we install the dependencies and manually import them to complete the
code snippet. This edit action was also used to reproduce traditional software
bugs, as reported by Mondal et al. msr2019mondal . In the Stack Overflow posts
we collected, $\approx$47% of the issue reports lacked the required dependency
information. This significant absence of the dependency information further
highlights the need for our suggested edit action. For instance, the code
snippet reported in the Stack Overflow post (Issue #50306988) was incomplete,
and the dependency details were missing. Hence, we resolved the dependencies
(keras, numpy and random) and added the required import statements to the code
snippet. With the resolved dependencies, we were able to reproduce the bug
successfully.

Logging (A${}_{5})$ plays a crucial role in the reproduction of different
types of bugs. This action involves adding log statements to the provided code
snippet to verify the reproduction of a bug. Let us consider the issue
reported in the Stack Overflow question (Issue #70546468). The reporter
provides a numpy array with shape (1, 3). The reporter then vertically
concatenates (stack depthwise) multiple copies of this array to create tensor
groups with shapes (2, 3) and (1, 3). After this vertical concatenation
process, the reporter expects the tensor’s shape to be (2, None, 3).
Unfortunately, the tensor’s shape was (2, None, None), according to the
reporter. To verify the claim made by the reporter, we introduce a log
statement within the code snippet. When we run the modified code snippet, we
observe the shape of the stacked tensor to be (2, None, None), thereby
confirming the reproduction and validity of the bug.

Obsolete Parameter Removal (A6) is a vital edit action to enhance the
reproducibility of bugs where the differences in the library and framework
versions may lead to problems. In this edit action, we remove all the obsolete
parameters not supported by the newer version of the library or framework. For
example, the code snippet in the Stack Overflow post (Issue #65992364) has the
following line of code.

⬇

script_model_vulkan = optimize_for_mobile(script_model, backend="Vulkan")

However, according to the API documentation of PyTorch 1.6.0, the `backend`
parameter was no longer supported. We removed the obsolete parameter and thus
were able to reproduce the bug.

Compiler Error Resolution (A7) is one of the extensively employed edit actions
in reproducing bugs. In this edit action, we resolve the compiler errors to
reproduce the bug. For example, in the Stack Overflow post (Issue #71514447),
the reporter reports that the training loss is significantly increasing after
every epoch. They suspect that the bug might be due to the incorrect
computation of loss values. Furthermore, to help the developers reproduce the
bug, they provide a detailed description of the bug and a complementary code
snippet. However, the code snippet provided by the user does not compile, as
the `criterion` function has not been defined. Hence, we replaced the function
parameter with the default value of cross-entropy loss, as shown below.

⬇

def train_model(model, optimizer, train_loader, num_epochs, criterion =
nn.CrossEntropyLoss()):

Using the edit action above, we resolved the compiler errors and thus were
able to reproduce the corresponding bug.

Dataset Procurement (A8) is a critical edit action to reproduce bugs that
require specific datasets. In this edit action, we analyze the issue report
and attempt to procure the dataset mentioned in the report. For instance, in
the Stack Overflow post (Issue #73966797), the reporter mentioned that they
used the CIFAR-10, a well-known dataset for object detection. To reproduce the
bug, we downloaded the dataset from its original
source666https://www.cs.toronto.edu/ kriz/cifar.html, and using the dataset
and the training code, we could reproduce the bug.

Downloading Models & Tokenizers (A9) is one of the edit actions that was
frequently used to reproduce bugs in Large Language Models and Transformer-
based architectures. In this edit action, we download the pre-trained models
and tokenizers for reproducing a bug. For example, in the Stack Overflow post
(Issue #69660201), the reporter has provided the following information.

R: In the preprocessing layer, I’m using a BERT preprocessor from TF-Hub.

Based on the above information, we added the relevant URLs and configured the
code to download the preprocessor and encoder. After downloading the
preprocessor and encoder, we successfully reproduced the corresponding bug.

⬇

tfhub_handle_preprocess =
"https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3"

tfhub_handle_encoder =
"https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-512_A-8/1"

Version Migration (A10) is one of the commonly used edit actions for all types
of bugs. In this edit action, we replace the outdated or deprecated versions
of a library or framework with the latest version to reproduce the bug. Mondal
et al. msr2019mondal  introduced a similar edit operation known as ‘Code
Migration’ in their work. This operation often confirms the validity of the
bug by reproducing it with the latest version. For example, the post on Stack
Overflow (Issue #45711636) describes an issue with the CNN architecture
constructed by the user. However, the code snippet provided in the issue uses
Tensorflow 1.3.0. To reproduce this issue in TensorFlow 2.14.0, we carefully
migrated the model built in TensorFlow 1.3.0 to the syntax of Tensorflow
version 2.14.0. Specifically, we updated the Sequential, Conv2D and
MaxPooling2D constructors to match the TensorFlow 2.14.0 API syntax. We also
changed padding and pool size parameter schemes and removed the obsolete input
shapes. Finally, we upgraded the model compilation and imported the required
Tensorflow 2.14.0 modules. After modifying the code snippet, we could
reproduce the bug successfully in our runtime environment.

###  4.2 RQ2: What information categories and edit actions are useful for
reproducing specific types of deep learning bugs?

####  4.2.1 Identifying useful Information in Bug Reports

While reproducing deep learning bugs, we kept track of information from bug
reports that helped us reproduce them. After reproducing 85 bugs successfully,
we have identified five useful pieces of information that can improve the
chance of reproducing a bug. We discuss these factors in detail below.

Data (F1): Data is one of the essential factors in ensuring the
reproducibility of deep learning bugs. Deep learning systems heavily rely on
the data breck2019data , and reproducing deep learning bugs becomes easier
with access to the original data. Data helps us reproduce the deep learning
bugs by providing the exact sample inputs that trigger the erroneous
behaviour. By understanding the training data distributions and ranges, we can
reconstruct the original training environment, which is crucial for
reproducibility. However, the issue reports often lack direct information
about the data. To address this problem, we collect information about the data
by extracting the shape of the data, data distribution, type of variables and
their corresponding ranges from the issue description. Leveraging this
information and our proposed edit actions, namely Input Data Generation (A1)
and Dataset Procurement (A8), we can generate or obtain the necessary data to
reproduce deep learning bugs. According to our investigation, $\approx$73% of
our reproduced bugs contained information about the data (see Table. 6) in
their issue reports. For example, in the Stack Overflow post (Issue
#43464835), the reporter has provided the dimensions and sample row of the
training dataset, as shown below.

R: I have a train dataset of the following shape: (300, 5, 720)

Sample Input: [[[ 6. 11. 389. …, 0. 0. 0.]]]]

Using this information and our proposed edit action Input Data Generation
(A1), we first generated a data frame of size (300, 5, 720) that contains
values in the range from 0 to 400. Then, by leveraging the generated data and
other useful information, we successfully reproduced the corresponding bug.

Model (F2): The model architecture describes the components of a deep learning
model and how they transform inputs into outputs. Understanding the model
architecture is crucial for reproducing deep learning bugs, as it provides
insights into the model’s components, connectivity, and the required neural
network architecture. According to our investigation, $\approx$65% of our
reproduced bugs contained information about the model architecture (see Table.
6) in their issue reports. However, the complete source code implementing the
model architecture might not always be available in the issue reports. To
overcome this limitation, we carefully gather information about a model’s
architecture, i.e., the number of layers, layer properties and activation
function from the issue description. Leveraging this information and our
proposed edit action Neural Network Construction (A2), we reconstruct the
model and reproduce the deep learning bugs. For example, let us consider the
following text from a Stack Overflow post (Issue #63204176) that mentions the
use of multinomial logistic regression on the MNIST dataset, as shown below.
Unfortunately, the reporter fails to provide the code snippet for the model.

R: I am trying to write a simple multinomial logistic regression using mnist
data.

Despite the absence of the code snippet, the issue description provides useful
hints about the model architecture and dataset. Using them and our edit action
– Neural Network Construction (A2), we successfully reproduced the bug.

Hyperparameters (F3): Hyperparameters play a crucial role in controlling the
learning process and model behaviour during training. They encompass
parameters such as learning rate, batch size, number of epochs, optimizer,
regularization techniques, and loss functions. The specific values chosen for
these hyperparameters significantly impact a model’s performance, training
process, and bug manifestation. Thus, reporting the complete set of
hyperparameters is essential to help reproduce deep learning bugs. According
to our investigation, 52% of our reproduced bugs contain information about the
hyperparameters used in their issue reports. For the 48% of bug reports that
did not include hyperparameter information, we used the Hyperparameter
Initialization edit action (A2) to reproduce the bugs by initializing the
hyperparameters with default values. In the example Stack Overflow post (Issue
#65993928), we can see how hyperparameters can play a crucial role in
reproducing the bug.

⬇

loss2 = (2 * (log_sigma_infer - log_sigma_prior)).exp() \ +((mu_infer -
mu_prior)/ log_sigma_prior.exp()) ** 2 \ - 2 * (log_sigma_infer -
log_sigma_prior) - 1

loss2 = 0.5 * loss2.sum(dim = 1).mean()

Since the reporter was using a custom loss function, it was vital for them to
share the value of constants and the formula used to calculate the loss. The
reporter’s custom loss function, with constants 2 and 0.5 and its formula, was
crucial for reproducing a bug in model training due to incorrect loss
calculation, emphasizing the need to provide a complete set of
hyperparameters.

Code Snippet (F4): Code snippets are critical for reproducing bugs, as
highlighted by the fact that 98% of professional developers consider them an
essential component of bug reproducibility soltani2020significance . They
include the data preprocessing, data splitting technique, code used for
training the model, and implementation of the evaluation metrics. However,
from our manual bug reproduction, we observe that even though code snippets
are present in $\approx$79% of the bug reports, only 9.41% of them can be used
verbatim for bug reproduction. To address this limitation, we use several of
our proposed edit actions, such as Import Addition and Dependency Resolution
(A4), Logging (A5), Obsolete Parameter Removal (A6), Compiler Error Resolution
(A7) and Version Migration (A10). These edit actions help us fix the errors in
the code and make the code compileable and runnable.

To demonstrate the importance of submitting a complete code snippet, let us
consider the Stack Overflow post with Issue #76186890. In this issue, a high-
quality code snippet helps us reproduce the bug related to a complicated
architecture (T5) without significant changes. The code snippet uses the T5
model from HuggingFace Transformers for text-to-text translation. It
preprocesses the input text data, configures the target IDs to predict a
specific answer, calculates loss and perplexity metrics, and trains the model.
Having a complete and runnable snippet was helpful in reproducing the bug. It
provides sufficient details like data preparation, loss calculation, model
training, and evaluation to improve the ease of reproducing deep learning
bugs, which were essential for bug reproduction.

Logs (F5): Logs provide a real-time record of the model’s behaviour during
training and inference. Traditional software logs consist of information such
as event logs and stack traces, whereas deep learning systems consist of
compiler error logs, training error logs, and evaluation logs Chen_Jiang_2022
. Sharing these logs is crucial for reproducing deep learning bugs as they
allow us to verify if we can reproduce the same erroneous behaviour reported
in the original issue. From our manual bug reproduction, we discover that
$\approx$86% of our reproduced bugs contain the necessary logs in their issue
reports. Such a high presence of logs in our dataset of reproduced bugs
highlights the importance of logging in deep learning bug reproduction. By
matching the logs from the original issue and the logs from reproduction on
our local machines, we were able to confirm the presence of several bugs in
the deep learning systems.

The Stack Overflow post (Issue #34311586) shows how logs can be used to
confirm the presence of deep learning bugs. In this particular issue, the
reporter shared the training logs and the code snippet. We modified the code
snippet with our proposed edit actions to make it compilable and runnable.
When executed, we observed the same anomalous behaviour in the training logs
as described by the original reporter. This demonstrates the importance of
sharing training and evaluation logs in the issue report. We also found them
to be one of the most useful pieces of information to reproduce deep learning
bugs.

Table 6: Prevalence of Useful Information in Reproducible Issue Reports Factor | Data | Model | Hyperparameters | Code Snippet | Logs
---|---|---|---|---|---
Prevalence | 72.9% | 64.7% | 52.7% | 78.8% | 85.8%

####  4.2.2 Relationship between Useful Information and Type of Bugs

During our manual analysis (Section 4.2.1), we identify useful information for
reproducing deep learning bugs. We used the Apriori algorithm to determine the
relationship between the type of bug and the information required to reproduce
the bug. The insights from this analysis could serve two purposes – detecting
missing information in submitted bug reports and formulating follow-up
questions to obtain any missing information needed for reproducibility. Table.
7 summarizes our findings, and we discuss them in detail below.

Data: The accurate reproduction of bugs in deep learning systems heavily
relies on the presence of data and its characteristics. They play a crucial
role in reproducing two key categories of bugs - training bugs and tensor
bugs.

For training bugs, the data has a confidence value of 80.77% according to our
generated rules (check Table. 7). This high confidence highlights the
significance of data in reproducing the numerous training issues that can
manifest during model development. The details about the data, such as the
number of samples, class distribution, feature distributions, data splitting
ratios, and preprocessing steps, are instrumental in reproducing training
bugs. For example, training a classification model on a highly skewed dataset
is prone to overfitting. With the knowledge of the data distribution, we can
recreate a representative dataset, which can help us reproduce the bugs, even
if the actual dataset is not available.

For tensor bugs, data characteristics have an even higher confidence of
95.24%, according to our generated rules (Table. 7). This suggests that tensor
attributes like shape, data type, sparsity, value ranges and origin are
pivotal for reliably reproducing many bugs stemming from invalid dimensions or
precision issues. For instance, bugs arising from tensor shape mismatches can
emerge if the shape of the input data does not match the expected input shape.
Furthermore, real-world data is often more vulnerable to human error and
biases compared to synthetic data talwar2020evaluating . Therefore,
comprehensive documentation and availability of these salient data
characteristics are imperative for the reliable reproduction of the numerous
training and tensor bugs in deep learning systems.

Model: A neural network model’s architecture and implementation details are
crucial to reproducing model and API bugs. The model architecture has 70% and
72.71% confidence values for model and API bugs, respectively, according to
our generated rules. These high values signify that access to model details is
vital for reliably reproducing issues stemming from model capacity,
connectivity, and API usage.

An access to the model architecture (e.g., layer types, layer connectivity,
weight initialization schemes, etc.) can help us systematically reproduce
model bugs. Insufficient learning capacity in specific layers and improper
weight initialization might lead to exploding/vanishing gradients
grosse2017lecture , whereas incorrect layer connectivity might lead to
representational bottlenecks Tishby_Zaslavsky_2015 . These issues usually
manifest as model bugs during training and inference. Thus, information about
the model architecture can help us localize and reproduce such bugs.

For reproducing the API bugs, model architecture can provide fundamental
context. Details like layer dimensions, bottlenecks, parallelization needs,
and memory requirements show how the model interacts with the API
yang2022comprehensive . Bottlenecks within a model, areas where data
processing slows down, and the need for parallel processing to handle large-
scale computations shape how APIs are utilized. Additionally, the varying
memory requirements of different architectures impact how the model leverages
the system’s resources via the API. This includes memory allocation for
storing weights and activations and managing the flow of data through the
network during operations like forward and backward propagation. Understanding
the model architecture is vital for reproducing bugs, as it indicates whether
issues stem from the model’s design or from its interaction with the API. For
example, a bug might arise due to the model’s inability to handle certain
operations efficiently, or it could be a result of the API not properly
supporting specific architectural features. Hence, model architecture helps us
reproduce the bugs triggered by incorrect usage of API. Therefore, the
presence of model architecture allows the reproduction of both model bugs and
API bugs in deep learning systems.

Code Snippet: Code snippets are invaluable for reproducing deep learning bugs
since they isolate and encapsulate the core logic that triggers them.
Developers can demonstrate and share the essence of buggy behaviour by
creating a minimal reproducible example. Code snippets have very high
confidence values of 92.3% for training bugs, 66.67% for tensor bugs, 70.0%
for model bugs, and 81.8% for API bugs according to our generated rules. These
high values across all bug types signify that code snippets are critical for
reliably reproducing deep learning bugs.

Each code snippet may contain the relevant model, data processing, training
and evaluation scripts. Developers can use such a snippet to recreate the bug-
inducing steps. For example, a snippet may compactly capture just a few lines,
mishandling tensor shapes or performing incorrect gradient calculations,
eliminating any confounding factors and enabling developers to reproduce these
bugs. Developers can systematically execute, analyze, and debug the code to
reproduce various bugs, including training, tensor, model, and API bugs.

Logs: Logs play a fundamental role in deep learning by facilitating the
reproduction and resolution of bugs. They comprehensively record every detail
during model training, capturing information about various conditions,
configurations, and events before the failures. Logs have high confidence
values of 84.77% for training bugs, 85.71% for tensor bugs, 80% for model
bugs, and 90.91% for API bugs according to our generated rules. Such high
confidence levels across all bug types highlight that comprehensive log
recording is important for the reliable reproduction of deep-learning bugs.

These logs include essential components such as hyperparameters, dataset
characteristics, hardware specifications, framework versions, and random
number generator seed values. The true power of logs lies in their ability to
recreate past training runs precisely. Developers can isolate and reproduce
the environment that led to the original bugs using the logged
hyperparameters, such as batch size, learning rate schedules, and gradient
clipping thresholds. Logs can also help developers reproduce specific deep-
learning bugs. For example, logged random seeds can help the developers
recreate a specific weight initialization or data batching order, which can
then be used to reproduce training bugs. Similarly, logs can be used to
reproduce Model, Tensor and API Bugs. Thus, logs are invaluable for accurately
recreating conditions that result in errors and enabling the reproduction of
deep learning bugs.

Table 7: Top 3 Useful Information Categories for Reproducing Specific Types of
Deep Learning Bugs

Training | Model
---|---
Code Snippet (0.9231) | Logs (0.80)
Logs (0.8477) | Model (0.70)
Data (0.8077) | Code Snippet (0.70)
Tensor | API
Data (0.9524) | Logs (0.9091)
Logs (0.8571) | Code Snippet (0.8182)
Code Snippet (0.6667) | Model (0.7271)

Table 8: Top 5 Edit Actions for Reproducing Specific Types of Deep Learning Bugs Training | Model
---|---
Input Data Generation (0.6154) | Neural Network Definition (0.75)
Import Addition (0.4231) | Import Addition (0.55)
Version Migration (0.3477) | Hyperparameter Initialization (0.50)
Hyperparameter Initialization (0.3077) | Logging (0.45)
Compiler Error Resolution (0.2692) | Dataset Procurement (0.40)
Tensor | API
Import Addition (0.4545) | Input Data Generation (0.5714)
Input Data Generation (0.3636) | Hyperparameter Initialization (0.5238)
Hyperparameter Initialization (0.3636) | Import Addition (0.4762)
Dataset Procurement (0.2727) | Logging (0.4286)
Logging (0.1818) | Obsolete Parameter Removal (0.3333)

####  4.2.3 Relationship between Edit Actions and Type of Bugs

After determining the association between the useful information needed and
the type of bug, we derived the relationship between the type of bug and the
edit actions. We use the Apriori algorithm to determine the relationship, as
done earlier. Table. 8 summarizes our findings, and we discuss them in detail
below.

Training Bug: Input data generation (61.54% confidence) is the most frequently
used edit action for reproducing training bugs. However, our analysis (Table.
6) shows that $\approx$73% of these reports contain information about the
data. Leveraging this information and our proposed edit action – Input Data
Generation (A1), we can reproduce the training bugs in deep learning systems.
Furthermore, import addition (42.3% confidence) and version migration (34.7%
confidence) are other edit actions which are often used to reproduce training
bugs. Although 79% of bug reports contain code snippets, they are often
incomplete. To reproduce the bug effectively, we thus need to complete these
snippets by adding imports and migrating them to the latest versions of
libraries and frameworks. Finally, hyperparameter initialization (30.77%
confidence) and compiler error resolution (26.92% confidence) may also help
reproduce training bugs by initializing the missing hyperparameters and
resolving compile-time errors from the code snippet.

Model Bug: Model bugs are often reproduced by modifying a neural network’s
architecture, as shown by the high confidence value (75%) for the edit action
– neural network construction. Since model bugs are primarily caused by errors
in the neural network architecture taxonomyRealFaults , reconstructing the
neural network might be the first step to reproduce them. Additionally, import
addition (55% confidence) and initializing hyperparameters (50% confidence)
are typical edit actions to reproduce the model bugs. Additionally, logging
can also help the reproduction of model bugs by recording important details
like layer outputs, gradients, and parameter values during training. These
recorded values can then be compared between buggy and bug-free models, which
can confirm the presence of a bug in the model. Thus, model bugs can be
reliably reproduced through several edits that construct the neural network
architecture, initialize hyperparameters, add logging, and procure datasets.

Tensor Bug: Data is the most important information for reproducing tensor
bugs, as shown by the strong correlation between Tensor Bug & Data (see Table.
7). This phenomenon can be attributed to the fact that tensor bugs primarily
relate to the data taxonomyRealFaults . Since tensor bugs are so data-
dependent, dataset procurement emerges as an important edit action to
reproduce them. Bug reports reference the exact dataset used (if well-known)
or describe the data type, shape and distribution. With this information,
input data generation and dataset procurement can generate or procure the data
required for the reproduction of the tensor bugs. Like other deep learning
bugs, import addition (45.45% confidence) and hyperparameter initialization
(36.36% confidence) also play a key role in completing the code snippet and
reproducing the tensor bugs. Thus, tensor bugs can be reliably reproduced
through several edits that procure the datasets, generate input data, add
logging, import the required dependencies, and initialize the hyperparameters.

API Bug: Since API Bugs stem from an improper usage of application programming
interfaces (APIs), they do not focus as heavily on the model training process.
As a result, bug reports might lack detailed information about the input data
or hyperparameters used. This encourages the use of common edit actions such
as input data generation (57.14% confidence) and hyperparameter initialization
(52.38% confidence) when reproducing API bugs. Additionally, code snippets in
bug reports are often incomplete, necessitating edit actions like import
addition (47.62% confidence) and logging (42.86% confidence) to trace the
intermediate states and outputs of the API. Thus, API bugs can be reliably
reproduced through several edit actions that generate input data, import the
required dependencies, add logging, initialize hyperparameters and remove
obsolete parameters.

###  4.3 RQ3: Do the suggested edit actions and information categories
contribute to enhancing the reproducibility of deep learning bugs in practice?

We conduct a developer study to determine if our suggested edit actions and
information help one reproduce deep learning bugs. We prepared four sets of
bugs, each consisting of two different types of bugs (see Section 3.6). We
randomly assigned one of the four bug sets to each participant and instructed
them to reproduce the bugs in their assigned set. We divided the participants
into the control and experimental groups using stratified random sampling (see
Section 3.6). The developers in both the control and experimental groups
reproduced the same bugs from our prepared set of bugs. The only difference
was that the experimental group received hints based on our findings to help
reproduce the bugs, while the control group did not receive these hints. Since
both groups received randomly selected sets from the same pool of four
identical bug sets, each bug was reproduced independently by at least five
developers across both control and experimental groups.

Table. 9 shows the bug reproducibility rate of the control and experimental
groups across different sets of bugs. We found that developers in the control
group could reproduce 72.91% bugs without hints. In contrast, the developers
in the experimental group could reproduce 95.83% of bugs with our hints - a
22.92% increase. This significant increase in reproducibility rate
demonstrates that our identified edit actions and information improved
developers’ ability to reproduce deep learning bugs.

Table 9: Percentage of bugs successfully reproduced by control and
experimental group across different sets.

|  % of bugs successfully reproduced by Control group |  % of bugs successfully reproduced by Experimental group |  % Increase
---|---|---|---
Set 1 |  75.00 |  100.00 |  25.00
Set 2 |  66.66 |  83.33 |  16.66
Set 3 |  83.33 |  100.00 |  16.66
Set 4 |  66.66 |  100.00 |  33.33
Average |  72.91 |  95.83 |  22.92

We also collect the qualitative feedback from the participants to better
understand their pain points in bug reproduction. Their feedback provided new
insights not covered in our original study. Notably, 31.78% of developers
highlighted the lack of standardized debugging tools as the primary challenge
in bug reproduction. Furthermore, the developers mentioned missing information
(data, logs, hyperparameters, and dependencies), and lack of unit testing or
version control in DL systems as the most challenging aspects of reproducing
deep learning bugs, as shown below.

Question: What are the challenges of bug reproduction in day-to-day
activities?

R1: lack of debugging support and missing information

R2: data quality issues and lack of standardized debugging procedures and
tools.

R3: no standardized debugging practices, and lack of clarity on the
information needed, also lot of dependencies (libraries, framework, data,
infra and so on)

R4: the flaky nature of deep learning models, and the unclear expectations of
how model is supposed to behave.

R5: memory issues, documentation issues (missing information in issues), weak
debugging support

R6: version control of deep learning is tricky, because of multiple snapshots
of models, model management and reproducibility is tricky. also, the lack of
standardized debugging practices makes it more tricky.

R7: distributed computing makes it difficult to find and reproduce the bugs.
test coverage is also a problem as we cannot find the bugs properly because of
lack of coverage and that is a problem with the reproduction of bugs.

We also analyze how our suggested actions and information help the
participants in bug reproduction. According to the qualitative responses,
40.91% of developers found our suggested edit actions to be helpful for
reproducing their assigned bugs. 54.55% of developers report that our
suggested hints about the useful information helped them narrow down where to
look in the code. Thus, the qualitative feedback below highlights the benefits
of our findings in real-life settings.

R1: I followed the guidance in the survey and used the hints to generate the
appropriate training dataset, which then allowed me to reliably reproduce the
problematic behavior during model training.

R2: had to add manual imports for torch and nn, and fixed the compiler errors
related to imports, as highlighted by the hint provided

R3: Following the hint, I systematically generated all of the necessary input
data that would be required in order to reliably reproduce the software bug
during testing.

R4: The hint mentions the Iris dataset, so I used the edit action called
“Dataset Procurement” and got the dataset, downloaded it and edited the code
snippet to reproduce the bug.

R5: with the given hint, I generated the input data required for the bug
reproduction.

R6: The imports were missing and as per the hint, data frame was generated for
specific columns, which helped the resolution of the bug.

R7: As hinted in the survey, explicitly specifying the columns when
constructing the data frame helped in the bug reproduction

We also calculate the time the control and experimental groups took to
reproduce their assigned bugs. This helps us assess our findings’ benefits in
reducing the time required for deep learning bug reproduction. Table. 10 shows
the time taken to reproduce bugs by the control and experimental groups. The
experimental group outpaced the control group in bug reproduction across all
sets, with the most notable difference in Set 4, where the experimental group
was 30.16% faster. On average, the control group lagged behind the
experimental group by 24.35% across all sets. These results demonstrate that
our recommended edit actions and information categories enabled the
experimental group to reproduce deep learning bugs much faster than the
control group that did not receive this information.

Table 10: Time taken for bug reproduction by control and experimental group

|  Average time taken by Control Group (seconds) |  Average time taken by Experimental Group (seconds) |  % Decrease
---|---|---|---
Set 1 |  1437 |  1088 |  24.28
Set 2 |  1803 |  1563 |  13.31
Set 3 |  2548 |  1792 |  29.67
Set 4 |  2165 |  1512 |  30.16

In short, the developer study demonstrates the real-world benefits of our
findings and provides evidence that our findings can enhance the
reproducibility of deep learning bugs.

##  5 Discussions

In this section, we first provide actionable insights about the
reproducibility of deep learning bugs and recommendations for further research
(see Section 5.1). We then describe a proof-of-concept tool, dlRepro, that we
developed to recommend edit actions and useful information for reproducing
deep learning bugs (see Section 5.2).

###  5.1 Reproducibility of Deep Learning Bugs

From our manual reproduction and user study, we observe that API, Model, and
Tensor bugs are relatively more straightforward to reproduce. This behaviour
can be explained by the fact that these bugs are more specific to the location
in the code where they originate. For example, faulty input data usually
triggers tensor bugs, whereas incorrect usage of a framework’s API causes API
bugs. In contrast, training bugs cover multiple issues related to deep
learning model training, and GPU bugs relate to GPU devices for deep learning
and manifest across GPU interactions.

This behaviour is further supported by the reproducibility rates and efforts
involved in reproducing different types of bugs. The reproducibility rates of
Training bugs and GPU bugs were 89.65% and 42.85%, respectively. On the other
hand, the reproducibility rates for API, Model and Tensor bugs were 81.81%,
88.46%, and 80.75%, respectively. Even though the training bugs could be
reproduced reliably, the efforts involved in reproducing the training bugs
were significantly more than those of other bugs. The average time to
reproduce the API, Model and Tensor bugs was 45.5, 43.9 and 43.8 minutes,
respectively. On the other hand, the average time to reproduce the Training
bugs was 52.5 and 57.33 minutes, respectively. These statistics highlight that
we need to put more effort into reproducing the Training and GPU bugs, which
calls for further research into the specific nature of Training and GPU bugs.

To assist future research in the reproducibility of deep learning bugs, we
provide directions for future research below:

Understanding Training Bug Reproducibility: Training bugs are the most common
type of bug in deep learning systems, accounting for 52.5% of all bugs
taxonomyRealFaults . Training bugs have high reproducibility rates but also
require significant effort to reproduce. This behaviour presents an
opportunity to understand better what factors make training bugs more
reproducible or harder to reproduce. By analyzing the training procedures,
model architectures, optimizers, hyperparameters, and other elements that
either aid or hinder reproducibility, we can uncover insights to guide the
diagnosis and repair of training bugs. There is also room to develop improved
tools and methodologies explicitly focused on efficiently reproducing the
nuanced nature of training bugs.

Analyzing the GPU Bug Reproducibility Gap: The lower reproducibility rate for
GPU bugs highlights a gap in understanding the interactions between deep
learning code and the underlying GPU hardware or drivers. By further studying
irreproducible GPU bugs and quantifying the aspects that impede
reproducibility, such as hardware differences, software dependencies, and
environmental factors, we can work towards solutions to increase
reproducibility. Opportunities exist to build infrastructure, leverage
containerization or virtualization, and create testing tools to better control
for sources of non-determinism that impact GPU bug reproduction.

Reproducible Testbeds for Deep Learning: Ultimately, the variability in
reproducing different categories of deep learning bugs motivates the need for
reproducible testbeds. Shared sets of reproducible deep learning bugs with
associated test cases, model architectures, training configurations,
dependencies, and environmental contexts will accelerate future research.
Constructing such testbeds requires systematic characterization of how these
factors influence reproducibility. Reproducible testbeds also support
developing specialized techniques for efficiently reproducing and debugging
deep learning bugs.

Figure 3: dlRepro, a proof-of-concept tool for deep learning bug
reproducibility

###  5.2 dlRepro: Proof-of-Concept Tool for Improving Bug Reproduction

From our user study (see RQ3), we observed that the lack of automated
debugging tools is one of the most significant challenges in deep learning bug
reproducibility. To aid the developers in reproducing deep-learning bugs, we
created a proof-of-concept tool, dlRepro 777https://dlrepro.streamlit.app/
(Fig. 3). This tool takes in either a Stack Overflow question ID or a bug
report as input. A bug report should include a bug description, code snippets
demonstrating the issue, and relevant tags categorizing the bug. Based on the
input provided, it recommends the useful information and edit actions needed
to reproduce the deep learning bug. We discuss the methodology of the tool
below.

  * •

Preparing the Bug Reports: Bug reports can be passed to the tool in two ways,
which are described below:

    * –

Manual Entry: The user can provide the bug description, code snippet, and tags
directly to the tool.

    * –

Stack Overflow: The user can input the Stack Overflow Question ID into the
tool. Using this question ID, our tool utilizes the Stack Overflow API to
retrieve the bug description, code snippet, and tags associated with that
question.

  * •

Tag Analysis: We match the tags provided in the Stack Overflow question to our
taxonomy of bug-related tags (Table 1). If there is a perfect match, we
determine the bug type directly. However, if there is no match, we use a
SentenceTransformer model called all-MiniLM-L6-v2 reimers2019sentence  to find
the semantic similarity between the tags in the post and our taxonomy tags for
different bug types. This allows us to determine the most likely bug type
based on the semantic similarity of the tags, even when there is no direct
match.

  * •

Recommendations: Once we determine the type of bug, we consult our findings
from the RQ2 (Table. 7 and Table. 8). From our analysis, we determine the
useful information and edit actions that can be used to reproduce the bug.

Use case of our POC tool: To demonstrate how our tool can be used to reproduce
deep learning bugs, let us consider the Stack Overflow Issue #54028667. This
issue 888https://stackoverflow.com/q/54028667 is caused by an incorrect usage
of the scikit-learn API. The tags for this issue are ‘python’, ‘scikit-learn’,
‘gridsearch’, ‘gridsearch-cv’. Since none of these tags exactly match the tags
from our analysis (Table. 1), the tool computes the semantic similarity to
determine the type of bug. Based on semantic similarity values, the tool
determines the type of bug to be API. After determining the type of bug, the
tool consults our findings from RQ2 (Table. 7 and Table. 8). It presents
useful information and key edit actions which can be used to reproduce the
bug. The outputs from the tool are shown in Fig. 4 and Fig. 5.

The bug report contained all the information required for reproduction. Hence,
by using the recommended edit actions and the complementary code snippets, we
could reproduce the bug in 23 minutes, which shows the effectiveness of our
findings and the tool. As the bug contained the required information, other
people could also reproduce the issue, and the question received an accepted
answer within 12 hours. Although our dataset consisted of posts containing
‘tensorflow’, ‘pytorch’, and ‘keras’, we could reproduce a bug related to
‘scikit-learn’. This highlights that our findings and tool are applicable
beyond the specific libraries used to build the original dataset. Reproducing
the ‘scikit-learn’ bug shows that our approach works even when applied to a
library not seen during initial development and testing, highlighting the
generalizability of our findings and tool.

Figure 4: Useful Information recommended by the tool Figure 5: Edit Actions
recommended by the tool

Extending and Enhancing Usability of dlRepro: While our study focused on
exploring and improving the state of deep learning bug reproducibility,
dlRepro was developed as a proof-of-concept tool to demonstrate how our
findings can serve as a step towards automated reproduction of deep learning
bugs. The tool’s flexible architecture enables continuous offline updates to
the underlying algorithm for tag similarities, information category mappings,
and edit recommendations without changes to the user-facing interface. It also
encourages broader deep-learning developer community contributions. Through
this collaborative access model, experts could voluntarily expand the tool’s
capabilities by sharing new insights uncovered. Such community participation
can lead to gradual, real-world refinements and expansion of both the dlRepro
tool and the taxonomy or tags that empower our tool. We have also open-sourced
dlRepro’s code to enable direct extensions by the community dlReproCitation ,
further showcasing the potential for collaborative improvement. While
demonstrating the applicability of our analysis, dlRepro establishes a
framework for ongoing communal efforts to enhance deep learning bug
reproducibility through collective intelligence.

###  5.3 Comparison with ChatGPT

To assess the effectiveness of our POC tool in deep learning bug
reproducibility, we conducted a comparison with ChatGPT, considering the
widespread usage of large language models. To do this, we provided ChatGPT
with a bug report, supplementary information, and a specific query, as shown
in Fig. LABEL:fig:chatGPTQuery. To evaluate ChatGPT’s capability to reproduce
deep learning bugs, we supplied it with the identical bug report, code
snippets, and other details utilized in our POC tool’s use case (Section 5.2).

The query included an instruction asking ChatGPT for help reproducing the bug,
while the query context included the detailed bug description and
corresponding code snippet. By providing this additional context, we aimed to
enhance ChatGPT’s understanding of the bug. We wanted the ChatGPT to reproduce
the bug.

Upon requesting ChatGPT to reproduce the bug, the model provided a potential
solution to the problem. Although this solution could potentially resolve the
bug, it might not be enough to reproduce the bug, which was our original goal.
Unlike ChatGPT’s response, our findings allowed us to reproduce the bug
successfully by combining edit actions and useful information, as shown in
Section 5.2.

This case study serves as a compelling demonstration of the practicality and
value of our findings, even in the era of language models. It highlights how
our approach can offer unique insights and solutions that are not readily
apparent through language models alone. By leveraging our findings and
language models’ capabilities, we can maximize the effectiveness of bug
reproducibility techniques and drive advancements in this field.

##  6 Threats to Validity

Threats to external validity relate to the generalizability of our findings.
We have reproduced multiple bugs from five different types to mitigate this
threat. Moreover, we have reproduced bugs from 14 different architectures,
ranging from Logistic Regression, CNN, Transformers and BERT-based
architectures. Finally, we have also reproduced the bugs from the past six
years (2017-2023), and our findings align with and extend the previous
findings of software bug reproducibility msr2019mondal , possibly indicating
the generalizability of our study.

Threats to internal validity pertain to experimental errors and confounding
variables during the reproduction of bugs. The possibility of introducing new
bugs into the code exists during bug reproduction. We have implemented several
strategies and precautions to mitigate these threats in our experimental
design. For instance, bug reproduction attempts are carried out in a
controlled and isolated environment. We refrain from using hardcoded or fixed
values in our code snippets to prevent potential errors during bug
reproduction. For instance, during the bug reproduction, we utilize random
hyperparameter initialization, construct multiple neural networks, and
generate randomized input data that aligns with the original distribution.
This methodology mitigates the influence of unforeseen variables on the
outcomes of our experiments. Following bug reproduction, we undertake rigorous
manual analysis to identify any discrepancies or anomalies in the outcomes.
Another threat to validity is the incomplete or incorrect information in the
bug report, which may lead to incorrect conclusions about the reproducibility
of deep learning bugs. To mitigate this threat, we apply extensive filtration
based on the criteria suggested by Moravati et al. defects4ml  and Humbatova
et al. taxonomyRealFaults , which leads to a clean and generalizable dataset.

Threats to construct validity relate to the use of appropriate metrics for
evaluating the results of this user study. We used the reproducibility rate
and time to reproduce the deep learning bugs as the metrics to evaluate the
benefits of our findings. These metrics directly measure how easily the bugs
can be reproduced, which aligns with our goal of making deep learning bugs
more reproducible. To ensure these metrics were accurately measured, we had
clear criteria for determining whether a bug was successfully reproduced. We
also recorded precise timestamps for how long it took participants to
reproduce bugs. This helped mitigate threats related to measurement errors or
subjective assessments of the metrics. Another threat to the validity might be
the accuracy of tags used to categorize the bugs. The incorrect categorization
of bugs might lead to incorrect conclusions for different bugs. To mitigate
this threat, we only select the tags that are present in the taxonomy and the
sub-taxonomies defined by Humbatova et al. taxonomyRealFaults . For example,
we specifically chose the “loss-function” as one of the tags to distinguish
the training bugs since the Loss Function falls within the sub-taxonomy of
Training bugs. Thus, the threats to construct validity were effectively
mitigated.

##  7 Related Work

There have been several studies that focus on the aspects of reproducibility
of software bugs msr2019mondal ; mondal2022reproducibility ;
mondalreproducibility , or focus on why some bugs cannot be reproduced
icsme2020rahman ; rahman2022works . Many studies attempt to learn the nature
of deep learning bugs tfProgramBugs ; islamfse19 ; taxonomyRealFaults ;
tarek2022icsme , and how they can be localized automatically nerdbug ;
yan2021exposing ; zhang2020detecting . A few studies also attempt to learn
about the state of reproducibility of deep learning in software engineering
trainingReproducibleDL ; reproducibilityDLTSE , and reproduce deep learning
bugs as a part of benchmark dataset creation defects4ml . Unfortunately, only
a little research has been done to understand the challenges in reproducing
deep learning bugs and how we can improve their reproducibility.

Mondal et al. msr2019mondal  extensively investigated the reproducibility of
programming issues reported on Stack Overflow. They identified several key
edit actions required to reproduce programming errors and traditional software
bugs. While their work is a source of inspiration, it does not provide the
edit actions for reproducing deep learning bugs. Since their dataset is
constructed in Java programming language with no questions related to deep
learning, their findings might not apply to deep learning bugs.

Chen et al. trainingReproducibleDL  proposed a unified framework to train
reproducible deep learning models. They also provide reproducibility
guidelines and mitigation strategies for conducting a reproducible training
process for DL models. However, the primary focus of their study was on the
reproducibility of deep learning models, not deep learning bugs. Furthermore,
the guidelines for deep learning models might not always extend to deep
learning bugs.

Moravati et al. defects4ml  constructed the faultload benchmark containing
deep learning bugs, defects4ML. As a part of their benchmark dataset, they
reproduced 100 bugs as a part of the reproducibility criterion for benchmark
datasets. While their study was the first to reproduce deep learning bugs
actively, they did not report the techniques and actions used to reproduce
bugs. Furthermore, they did not report the useful information from the issue
reports, which helped them reproduce the bugs.

Unlike many earlier studies above, we conduct an extensive empirical study to
understand the challenges of deep learning bug reproduction and how we can
improve the reproducibility of deep learning bugs. We construct a dataset of
668 bugs and manually reproduce 85 bugs, spanning three frameworks, five types
of bugs, and 14 architectures. We not only (1) define ten edit actions which
can be used to reproduce deep learning bugs but also (2) explore the
associations among the type of bugs, the information required to reproduce
them, and the specific edit actions that can be used to reproduce them, which
makes our work novel. Our findings are also generalizable due to the diversity
of our dataset. To ensure transparency and reproducibility, we have made our
dataset and replication package publicly
available999https://github.com/mehilshah/Bug_Reproducibility_DL_Bugs.

##  8 Conclusion & Future Work

The reproducibility of deep learning bugs is a significant challenge for
software practitioners, given that these bugs can lead to severe consequences.
In this paper, we conduct an empirical study by reproducing 85 deep learning
bugs and identifying ten key edit actions that could reproduce these bugs.
Furthermore, we investigate the relationships among bug types, information
categories, and edit actions. To validate our findings, we conducted a user
study with 22 participants. The developers equipped with our recommended edit
actions and information could reproduce 22.92% more bugs using 24.35% less
time compared to the developers without such hints. This demonstrates the
real-world value of our findings in improving the reproducibility of deep
learning bugs. Finally, we provide actionable insights about the state of deep
learning bugs that could improve the reproducibility of deep learning bugs. To
demonstrate the practical use of our findings, we built a proof-of-concept
tool, dlRepro, that takes bug reports and recommends the edit actions and
useful information for deep learning bugs. Moreover, we also conduct a
comparison with ChatGPT, which fails to reproduce the bug, while our
methodology succeeds. This demonstrates the real-world value of our findings
in improving the reproducibility of the deep learning bugs. For future work,
we plan to expand our study to additional deep learning frameworks and model
types. We will also improve dlRepro with more advanced natural language
processing capabilities to extract key information from bug reports. Finally,
we aim to integrate dlRepro into developer tools and workflows to provide
automated reproducibility support during deep learning development.
Furthermore, our findings open up the ability to create automated systems for
reproducing deep learning bugs. By training machine learning models on our
dataset of reproduction actions matched to bug reports, we can build AI agents
that take bug reports as input and automatically generate reproducibility
scripts, similar to the work by White et al. reproducibleBugReports . This
would significantly reduce the manual effort required to reproduce bugs and
enable large-scale debugging and testing of deep learning systems.

## Data Availability Statement (DAS)

All the data generated or analyzed during this study are available in the
GitHub repository to help reproduce our results replicationPackage .

## Conflict of Interest

The authors declare that they have no conflict of interest.

## References

  * (1) D. Shen, G. Wu, and H.-I. Suk, “Deep learning in medical image analysis,” _Annual review of biomedical engineering_ , vol. 19, pp. 221–248, 2017.
  * (2) P. M. Addo, D. Guegan, and B. Hassani, “Credit risk analysis using machine and deep learning models,” _Risks_ , vol. 6, no. 2, p. 38, 2018.
  * (3) D. S. Berman, A. L. Buczak, J. S. Chavis, and C. L. Corbett, “A survey of deep learning methods for cyber security,” _Information_ , vol. 10, no. 4, p. 122, 2019.
  * (4) K. Pei, Y. Cao, J. Yang, and S. Jana, “Deepxplore: Automated whitebox testing of deep learning systems,” _Commun. ACM_ , vol. 62, no. 11, p. 137–145, oct 2019. [Online]. Available: https://doi.org/10.1145/3361566
  * (5) L. Ma, F. Juefei-Xu, F. Zhang, J. Sun, M. Xue, B. Li, C. Chen, T. Su, L. Li, Y. Liu, J. Zhao, and Y. Wang, “Deepgauge: Multi-granularity testing criteria for deep learning systems,” in _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering_ , ser. ASE ’18. New York, NY, USA: Association for Computing Machinery, 2018, p. 120–131. [Online]. Available: https://doi.org/10.1145/3238147.3238202
  * (6) A. Esteva, A. Robicquet, B. Ramsundar, V. Kuleshov, M. DePristo, K. Chou, C. Cui, G. Corrado, S. Thrun, and J. Dean, “A guide to deep learning in healthcare,” _Nature medicine_ , vol. 25, no. 1, pp. 24–29, 2019.
  * (7) H. B. Braiek and F. Khomh, “On testing machine learning programs,” _Journal of Systems and Software_ , vol. 164, p. 110542, 2020. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0164121220300248
  * (8) M. J. Islam, G. Nguyen, R. Pan, and H. Rajan, “A comprehensive study on deep learning bug characteristics,” ser. ESEC/FSE 2019. New York, NY, USA: Association for Computing Machinery, 2019, p. 510–520. [Online]. Available: https://doi.org/10.1145/3338906.3338955
  * (9) D. Wakabayashi, “Self-driving uber car kills pedestrian in arizona, where robots roam,” Mar 2018, accessed on December 17, 2023. [Online]. Available: https://www.nytimes.com/2018/03/19/technology/uber-driverless-fatality.html
  * (10) P. Nagarajan, G. Warnell, and P. Stone, “The impact of nondeterminism on reproducibility in deep reinforcement learning,” 2018.
  * (11) M. Krishnan, “Against interpretability: a critical examination of the interpretability problem in machine learning,” _Philosophy & Technology_, vol. 33, no. 3, pp. 487–502, 2020.
  * (12) M. M. Morovati, A. Nikanjam, F. Khomh, and Z. M. J. Jiang, “Bugs in machine learning-based systems: A faultload benchmark,” _Empirical Softw. Engg._ , vol. 28, no. 3, apr 2023. [Online]. Available: https://doi.org/10.1007/s10664-023-10291-1
  * (13) S. Mondal, M. M. Rahman, and C. K. Roy, “Can issues reported at stack overflow questions be reproduced? an exploratory study,” in _2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)_ , 2019, pp. 479–489.
  * (14) M. M. Rahman, F. Khomh, and M. Castelluccio, “Why are some bugs non-reproducible? : –an empirical investigation using data fusion–,” in _2020 IEEE International Conference on Software Maintenance and Evolution (ICSME)_ , 2020, pp. 605–616.
  * (15) Y. Liang, Y. Lin, X. Song, J. Sun, Z. Feng, and J. S. Dong, “gdefects4dl: a dataset of general real-world deep learning program defects,” in _Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings_ , 2022, pp. 90–94.
  * (16) N. Humbatova, G. Jahangirova, G. Bavota, V. Riccio, A. Stocco, and P. Tonella, “Taxonomy of real faults in deep learning systems,” in _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_ , ser. ICSE ’20.
  * (17) R. Agrawal and R. Srikant, “Fast algorithms for mining association rules in large databases,” in _Proceedings of the 20th International Conference on Very Large Data Bases_ , ser. VLDB ’94. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 1994, p. 487–499.
  * (18) 2019, accessed on January 3, 2024. [Online]. Available: https://stackoverflow.com/q/58190114
  * (19) 2018, accessed on December 28, 2023. [Online]. Available: https://stackoverflow.com/q/50920908
  * (20) S. Exchange, “All sites - stack exchange,” accessed on December 12, 2023. [Online]. Available: https://stackexchange.com/sites?view=list
  * (21) H. Zhao, Y. Li, F. Liu, X. Xie, and L. Chen, “State and tendency: an empirical study of deep learning question&answer topics on stack overflow,” _Science China Information Sciences_ , vol. 64, pp. 1–23, 2021.
  * (22) T. Zhang, C. Gao, L. Ma, M. Lyu, and M. Kim, “An empirical study of common challenges in developing deep learning applications,” in _2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE)_. IEEE, 2019, pp. 104–115.
  * (23) L. Ponzanelli, A. Mocci, A. Bacchelli, and M. Lanza, “Understanding and classifying the quality of technical forum questions,” in _2014 14th International Conference on Quality Software_ , 2014, pp. 343–352.
  * (24) M. Shah, “mehilshah/bug_reproducibility_dl_bugs,” accessed on January 3, 2024. [Online]. Available: https://github.com/mehilshah/Bug_Reproducibility_DL_Bugs
  * (25) Jan 2023, accessed on December 25, 2023. [Online]. Available: https://www.geeksforgeeks.org/best-ides-for-machine-learning/
  * (26) M. Gori, A. Betti, and S. Melacci, _Machine Learning: A constraint-based approach_. Elsevier, 2023.
  * (27) B. Liu, W. Hsu, and Y. Ma, “Mining association rules with multiple minimum supports,” in _Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining_ , 1999, pp. 337–341.
  * (28) E. Breck, N. Polyzotis, S. Roy, S. Whang, and M. Zinkevich, “Data validation for machine learning.” in _MLSys_ , 2019.
  * (29) M. Soltani, F. Hermans, and T. Bäck, “The significance of bug report elements,” _Empirical Software Engineering_ , vol. 25, pp. 5255–5294, 2020.
  * (30) B. Chen and Z. M. J. Jiang, “A survey of software log instrumentation,” _ACM Computing Surveys_ , vol. 54, no. 4, p. 1–34, May 2022. [Online]. Available: https://dl.acm.org/doi/10.1145/3448976
  * (31) D. Talwar, S. Guruswamy, N. Ravipati, and M. Eirinaki, “Evaluating validity of synthetic data in perception tasks for autonomous vehicles,” in _2020 IEEE International Conference On Artificial Intelligence Testing (AITest)_. IEEE, 2020, pp. 73–80.
  * (32) R. Grosse, “Lecture 15: Exploding and vanishing gradients,” _University of Toronto Computer Science_ , 2017.
  * (33) N. Tishby and N. Zaslavsky, “Deep learning and the information bottleneck principle,” in _2015 IEEE Information Theory Workshop (ITW)_ , Apr. 2015, p. 1–5. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/7133169
  * (34) Y. Yang, T. He, Z. Xia, and Y. Feng, “A comprehensive empirical study on bug characteristics of deep learning frameworks,” _Information and Software Technology_ , vol. 151, p. 107004, 2022.
  * (35) N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” _arXiv preprint arXiv:1908.10084_ , 2019.
  * (36) M. Shah, “mehilshah/dlrepro: Tool for deep learning bug reproducibility,” accessed on January 3, 2024. [Online]. Available: https://github.com/mehilshah/dlRepro
  * (37) S. Mondal, M. M. Rahman, C. K. Roy, and K. Schneider, “The reproducibility of programming-related issues in stack overflow questions,” _Empirical Software Engineering_ , vol. 27, no. 3, p. 62, 2022.
  * (38) S. Mondal and B. Roy, “Reproducibility of issues reported in stack overflow questions: Challenges, impact & estimation,” _Impact & Estimation_.
  * (39) M. M. Rahman, F. Khomh, and M. Castelluccio, “Works for me! cannot reproduce–a large scale empirical study of non-reproducible bugs,” _Empirical Software Engineering_ , vol. 27, no. 5, p. 111, 2022.
  * (40) Y. Zhang, Y. Chen, S.-C. Cheung, Y. Xiong, and L. Zhang, “An empirical study on tensorflow program bugs,” in _Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis_ , ser. ISSTA 2018. New York, NY, USA: Association for Computing Machinery, 2018, p. 129–140. [Online]. Available: https://doi.org/10.1145/3213846.3213866
  * (41) T. Makkouk, D. J. Kim, and T.-H. P. Chen, “An empirical study on performance bugs in deep learning frameworks,” in _2022 IEEE International Conference on Software Maintenance and Evolution (ICSME)_ , 2022, pp. 35–46.
  * (42) F. Jafarinejad, K. Narasimhan, and M. Mezini, “Nerdbug: Automated bug detection in neural networks,” in _Proceedings of the 1st ACM International Workshop on AI and Software Testing/Analysis_ , ser. AISTA 2021. New York, NY, USA: Association for Computing Machinery, 2021, p. 13–16. [Online]. Available: https://doi.org/10.1145/3464968.3468409
  * (43) M. Yan, J. Chen, X. Zhang, L. Tan, G. Wang, and Z. Wang, “Exposing numerical bugs in deep learning via gradient back-propagation,” in _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2021, pp. 627–638.
  * (44) Y. Zhang, L. Ren, L. Chen, Y. Xiong, S.-C. Cheung, and T. Xie, “Detecting numerical bugs in neural network architectures,” in _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , 2020, pp. 826–837.
  * (45) B. Chen, M. Wen, Y. Shi, D. Lin, G. K. Rajbahadur, and Z. M. J. Jiang, “Towards training reproducible deep learning models,” in _Proceedings of the 44th International Conference on Software Engineering_ , ser. ICSE ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 2202–2214. [Online]. Available: https://doi.org/10.1145/3510003.3510163
  * (46) C. Liu, C. Gao, X. Xia, D. Lo, J. Grundy, and X. Yang, “On the reproducibility and replicability of deep learning in software engineering,” _ACM Trans. Softw. Eng. Methodol._ , vol. 31, no. 1, oct 2021. [Online]. Available: https://doi.org/10.1145/3477535
  * (47) M. White, M. Linares-Vásquez, P. Johnson, C. Bernal-Cárdenas, and D. Poshyvanyk, “Generating reproducible and replayable bug reports from android application crashes,” in _2015 IEEE 23rd International Conference on Program Comprehension_ , 2015, pp. 48–59.