# A Machine Learning Based Ensemble Method for Automatic Multiclass
Classification of Decisions

Liming Fu, Peng Liang∗, Xueying Li  School of Computer Science, Wuhan
UniversityWuhan, China{limingfu, liangp<EMAIL_ADDRESS>and  Chen Yang
IBO Technology (Shenzhen) Co., Ltd.Shenzhen<EMAIL_ADDRESS>
(2021)

###### Abstract.

Stakeholders make various types of decisions with respect to requirements,
design, management, and so on during the software development life cycle.
Nevertheless, these decisions are typically not well documented and classified
due to limited human resources, time, and budget. To this end, automatic
approaches provide a promising way. In this paper, we aimed at automatically
classifying decisions into five types to help stakeholders better document and
understand decisions. First, we collected a dataset from the Hibernate
developer mailing list. We then experimented and evaluated 270 configurations
regarding feature selection, feature extraction techniques, and machine
learning classifiers to seek the best configuration for classifying decisions.
Especially, we applied an ensemble learning method and constructed ensemble
classifiers to compare the performance between ensemble classifiers and base
classifiers. Our experiment results show that (1) feature selection can
decently improve the classification results; (2) ensemble classifiers can
outperform base classifiers provided that ensemble classifiers are well
constructed; (3) BoW + 50% features selected by feature selection with an
ensemble classifier that combines Naïve Bayes (NB), Logistic Regression (LR),
and Support Vector Machine (SVM) achieves the best classification result (with
a weighted precision of 0.750, a weighted recall of 0.739, and a weighted
F1-score of 0.727) among all the configurations. Our work can benefit various
types of stakeholders in software development through providing an automatic
approach for effectively classifying decisions into specific types that are
relevant to their interests.

Decision, Automatic Classification, Ensemble Classifier, Software Development,
Hibernate

††journalyear: 2021††copyright: acmcopyright††conference: Evaluation and
Assessment in Software Engineering; June 21–23, 2021; Trondheim,
Norway††booktitle: Evaluation and Assessment in Software Engineering (EASE
2021), June 21–23, 2021, Trondheim, Norway††price: 15.00††doi:
10.1145/3463274.3463325††isbn: 978-1-4503-9053-8/21/06

##  1\. Introduction

Stakeholders make numerous decisions with respect to software design,
requirements, management, and so on during the software development life
cycle. These decisions potentially affect the functional and non-functional
properties of software systems (Shahbazian et al., 2018) and thus have a
crucial impact on software development and maintenance. Considering the
importance of decisions, it is essential for stakeholders to document and
understand the past decisions. Documenting these decisions will help
stakeholders organize development knowledge and reduce its vaporization,
therefore controlling the development process and maintenance costs (van der
Ven et al., 2006). In addition, it also contributes to rationale management
through recording decisions. The availability of rationale not only increases
the developers’ understanding of the system, but also facilitates the training
of new members (Dutoit et al., 2006).

However, due to limited human resources, time, and budget, decisions are
rarely well documented in Open Source Software (OSS) (Ding et al., 2014) and
industrial projects (Ambler, 2002). The loss of this information can thus hurt
the development and maintenance of software systems (Shahbazian et al., 2018).
Even though decisions are not explicitly documented, they may be implicitly
captured in many tools, such as mailing lists, issue tracking systems, and
version control systems, which are used for stakeholders to communicate with
each other about the issues encountered in software development (Li et al.,
2019a). Stakeholders can learn diverse decision knowledge accumulated during
the process of project development, such as requirements, design, and testing,
from these discussions (Li et al., 2019b). It is thus desirable to identify
and classify decisions implicitly captured in these tools for documentation
purposes. However, there are massive textual artifacts in these tools, which
may also contain lots of meaningless information. Manually identifying and
classifying decisions is rather time-consuming, and therefore it may not be
applicable in practice. To this end, exploiting automatic methods to identify
and classify decisions in software development is a promising way.

While several studies have been conducted to use automatic methods to mine
decisions in software development, most of them focus on either identifying or
classifying specific type of decisions (e.g., design decisions in (Bhat et
al., 2017)) or identifying decisions without further classification (e.g., (Li
et al., 2019a)). To the best of our knowledge, limited work has been done on
classifying decisions into types that are beneficial for various types of
stakeholders in development. Stakeholders are engaged with different tasks in
software development, thus they are more concerned about the decisions related
to their interests. For example, developers may highlight decisions concerning
code issues. Considering this, we planned to automatically classify decisions
into five decision types (i.e., Design, Requirement, Management, Construction,
and Testing Decision) according to the types got in our recent work (Li et
al., 2019b).

In this work, we first collected our dataset from the Hibernate developer
mailing list. We then examined the performance of a feature selection
algorithm (i.e., CHI2), three feature extraction techniques, and four base
machine learning (ML) classifiers. Moreover, we used an ensemble method and
constructed five ensemble classifiers using the aforementioned four base ML
classifiers. Ensemble methods that train multiple classifiers and then combine
them are a kind of state-of-the-art learning approach. It is well known that
an ensemble classifier is usually significantly more accurate than a single
classifier (Zhou, 2012). Overall, 270 configurations regarding ten different
percentages of selected features in feature selection, three feature
extraction techniques, and nine classifiers were evaluated through weighted
precision, recall, and F1-score. The contribution of this work is threefold:
(1) the first empirical study focusing on automatic classification of
decisions into five decision types that are beneficial for various types of
stakeholders in development; (2) an effective automatic approach with ensemble
learning to classify decisions into five decision types with a weighted
precision of 0.750, a weighted recall of 0.739, and a weighted F1-score of
0.727; (3) empirical evidence about the effectiveness comparison between using
base supervised machine learning classifiers and ensemble classifiers to
automatically classify decisions in software development.

The rest of this paper is structured as follows: Section 2 introduces the
related work on decisions in software development and automatically mining
textual artifacts in software development. Section 3 describes the research
design and Section 4 presents the detailed procedures of the experiment. The
results and analysis of our experiment are provided in Section 5. Section 6
discusses the feature terms and the implications of our study. Section 7
clarifies the threats to the validity. Finally, we conclude this work with
future research directions in Section 8.

##  2\. Related Work

###  2.1. Decisions in Software Development

Decisions have a crucial effect on development and maintenance of software
systems. Many researches have focused on analyzing decisions and their making
in software development. For instance, Razavian et al. systematically analyzed
38 research papers on architecture decision-making with a special focus on
human aspects of architecture decision-making (Razavian et al., 2019). Their
study highlights that decisions as well as the decision-making process are
important knowledge in architecture design. Mendes et al. aimed at studying
how the personality of decision makers can influence their decision-making
(Mendes et al., 2019). They reported 75 relationships between 28 different
personality aspects and 30 different decision-making aspects, which can help
in improving how a decision is made in software engineering context.

There are also a few researches focusing on employing automatic approaches in
decision-related work. Bhat et al. proposed a two-phase supervised machine
learning based approach to automatically detect design decisions from issues
in issue tracking systems of two large OSS projects, and then classify the
detected design decisions into three categories (i.e., Behavioral decision,
Structural decision, and Ban decision) (Bhat et al., 2017). In their follow-up
work, they developed a tool named ADeX to automatically extract, enrich, and
generate specific views on architectural knowledge to support architects’
decisions-making process (Bhat et al., 2019). Shahbazian et al. developed a
technique called RecovAr to automatically recover design decisions from the
readily available history artifacts of projects (e.g., an issue tracker and
version control repository) (Shahbazian et al., 2018). Their work provides a
way that developers can follow to preserve design decision knowledge in their
projects. Li et al. conducted an experiment with 160 configurations to
automatically identify decisions from an OSS mailing list (Li et al., 2019a).
The evaluation results show that their approach can effectively identify
decisions from the mailing list.

Inspired by the work above (i.e., (Bhat et al., 2017, 2019; Shahbazian et al.,
2018; Li et al., 2019a)), we are motivated to develop an automatic approach to
help researchers and practitioners better understand specific types of
decisions in software development. The difference between our work and
existing work is that we considered classifying decisions into five decision
types that are beneficial to various types of stakeholders in software
development.

###  2.2. Automatically Mining Textual Artifacts in Software Development

Automatically mining textual artifacts is subject to widespread attention in
software development. Besides the studies that focus on decisions, many
studies have also focused on automatically mining artifacts that are highly
related to decisions. Li et al. used Word2Vec to extract features and trained
seven machine learning classifiers for automatically identifying assumptions
from an OSS developer mailing list (Li et al., 2019c). Their experiment
results show that the automatic approach can effectively identify assumptions
by using the SVM algorithm with a precision of 0.829, a recall of 0.812, and
an F1-score of 0.819. Rogers et al. compared text mining and text parsing
techniques to automatically mine rationale from bug reports (Rogers et al.,
2015), and they argued that the results obtained by their approach are
comparable in accuracy to those obtained by human annotators.

Automatic approaches have also been applied to mine other textual artifacts.
Shah et al. considered automatically classifying user reviews into feature
evaluation, feature requests, and bug reports (Shah et al., 2018). Their
results show that the simple BoW model is competitive compared with complex
models adopting rich linguistic features. Huang et al. proposed an automated
approach to detect self-admitted technical debt from source code comments of
eight OSS projects (Huang et al., 2018). Their approach trained an individual
sub-classifier for each source project, and then combined them as an ensemble
classifier to vote for the comments in the target project. The results
indicate that the ensemble classifier could improve F1-score.

The above automation related work (both on decisions and other textual
artifacts) motivates us to use an automatic approach to classify decisions
from OSS developer mailing lists. Their experiment setups lay the foundation
for our experiment.

##  3\. Research Design

In this section, we first formulate the goal with three Research Questions
(RQs) that we try to answer through this study. We then briefly describe the
five decision types. Finally, we introduce feature selection, feature
extraction techniques, machine learning classifiers, and ensemble classifiers
that are utilized for classifying decisions in the experiment.

###  3.1. Goal and Research Questions

The goal of this work is to analyze textual information in the Hibernate
developer mailing list for the purpose of automatic classification with
respect to various types of decisions in software development from the point
of view of OSS developers in the context of OSS development, which is
formulated through the Goal Question Metric approach (Basili et al., 1994).
This work comprises three RQs, which are presented in the following.

RQ1. Does feature selection improve the classification results?

RQ2. Do the ensemble classifiers outperform the base classifiers for
classifying decisions?

RQ3. What is the best configuration for classifying decisions into five
decision types?

###  3.2. Types of Decisions

In our previous study on decisions and their making in software development
(Li et al., 2019b), we analyzed 980 decisions from 9006 posts of the Hibernate
developer mailing list and identified five decision types (i.e., Design,
Requirement, Management, Construction, and Testing Decision) from those
decisions using grounded theory. We adapted and adopted the description of the
five decision types in this work based on our previous description in (Li et
al., 2019b):

Design Decisions are decisions made about the design of software. Design
decisions are related to various elements in software systems, such as
components, interfaces, layers, and subsystems. They indicate the
specification of the composition, properties, structures, behaviors, or
relationships of these elements, or reflect the rules, patterns, or
constraints that these elements should follow.

Requirement Decisions are related to functional requirements or non-functional
requirements of software systems. They aim to provide or improve functionality
for software, or meet a set of quality attributes that the software should
exhibit, such as reliability, portability, and performance.

Management Decisions are related to project management of software. They
indicate the maintenance and improvement in documentations of software,
version control of software, development plans of software, and other
management operations.

Construction Decisions are related to the specific implementation of software.
They focus on the source code of project such as source code annotation, the
configuration files of the project, and other specific operations for the
project.

Testing Decisions are related to the test and debugging of software. They
cover the test behavior, the creation or change of test cases, or the bug
report of software.

We found that the above five decision types can basically cover the decisions
made during the development and maintenance of Hibernate (Li et al., 2019b).
Therefore, we considered classifying decisions from the Hibernate developer
mailing list into these five decision types. The decision examples of these
types are provided in Table 1.

Table 1. Examples of Each Decision Type from the Hibernate Developer Mailing List Type |  | Examples |
---|---|---|---
Design Decision |  | “I have redefined what happens when a persistent object holds a reference to a transient object at flush() time.” |
Requirement Decision |  | “I am adding support for template-based functions as part of the EJBQL issue.” |
Management Decision |  | “We will release Hibernate 3.0alpha early next week. |
Construction Decision |  | “I added a new instance array to store the Types of each of the columns.” |
Testing Decision |  | “I will run the tests on all platforms before the next release just to make sure.” |

###  3.3. Feature Selection

Terms that exist in more types contain less type information, and consequently
they are less important for classification (Lu and Liang, 2017). Moreover,
research shows that feature selection can improve the performance of text
classification (Yang and Pedersen, 1997). Therefore, we apply feature
selection to identify a subset of features (i.e., terms) that provide expected
discrimination between the types of decisions and thus are useful for
classification.

There are several frequently used feature selection algorithms, such as Mutual
Information, Information Gain, and Chi Squared (i.e., CHI2) (Forman, 2003).
Among them, CHI2 is one of the most efficient methods used in many studies
related to text classification tasks (Guzman et al., 2015; Lu and Liang, 2017;
Rustam et al., 2020). The CHI2 value of term $T_{i}$ for a certain decision
type ($Type_{k}$) is calculated through Formula (1), where $N$ denotes the
total number of decisions in our dataset, variable $a$ denotes the number of
decisions that have term $T_{i}$ and belong to $Type_{k}$, variable $b$
denotes the number of decisions that have term $T_{i}$ and do not belong to
$Type_{k}$, variable $c$ denotes the number of decisions that do not have term
$T_{i}$ and belong to $Type_{k}$, and variable $d$ denotes the number of
decisions that do not have term $T_{i}$ and do not belong to $Type_{k}$. The
greater the value of $CHI^{2}(T_{i},Type_{k})$ is, the more type information
the Term $T_{i}$ contains.

(1) $CHI^{2}(T_{i},Type_{k})=\frac{N*(ad-bc)^{2}}{(a+c)(b+d)(a+b)(c+d)}$

In this work, we calculated the CHI2 value for each term in each decision type
(after decisions preprocessing). Then for each decision type, the terms were
ranked by the CHI2 value in a descending order and we chose the top $n$
percent of the terms as feature terms. The value of $n$ is a threshold that
depends on the specific classification task and the dataset, and it ranges
from 1% to 100%. The selection result of value of $n$ is described in Section
4.4.

###  3.4. Feature Extraction Techniques

In text classification tasks, texts are modeled as vectors that can represent
their features. Three well-performed techniques are widely used in text
classification tasks for extracting text features. Therefore, we also
experimented and compared these three techniques in this work, which are
briefly described below.

BoW: Bag of Words, builds a vocabulary composed of all unique terms in the
decisions of our dataset and then represents each decision by counting the
term frequency of every term in the vocabulary.

TF-IDF: Term Frequency - Inverse Document Frequency, uses the same textual
features (i.e., terms) as BoW does. But different from BoW, TF-IDF evaluates a
term’s weight by combining term frequency with the inverse document frequency,
which implies that the importance of a term increases with the term frequency
in a sentence, but decreases inversely with the frequency of its appearance in
the whole corpus.

Word2Vec: is prevalently used in the Nature Language Processing (NLP)
community. It can learn semantic knowledge from a large number of text corpus
and then transform words in the corpus into vectors with rich semantic
information (Rong, 2014). Continuous Bag-of-Words (CBoW) and Continuous Skip-
Gram (Skip-Gram) are two models of Word2Vec, which are both efficient methods
for various NLP tasks, such as text classification. CBoW can predict the
current word through the surrounding context while Skip-Gram can forecast the
context according to the current words. In this work, we use CBoW model to
generate word embeddings and represent each decision with a vector calculated
through Formula (2), where $d_{i}$ denotes the decision $i$, $V_{d_{i}}$
denotes the vector representation of $d_{i}$, $N$ denotes the total number of
words in $d_{i}$, $d(i,n)$ denotes the $n^{th}$ word in $d_{i}$, and
$W_{d}(i,n)$ denotes the word embeddings of $d(i,n)$ generated by Word2Vec.

(2) $V_{d_{i}}=\frac{1}{N}\sum_{n=1}^{N}W_{d(i,n)}$

###  3.5. Machine Learning Classifiers

A recent research shows that some supervised ML classifiers, such as SVM or
Random Forest (RF), can perform equally well or even better than deep learning
techniques in text classification tasks (Fu and Menzies, 2017). Also, deep
learning techniques usually are more complex, slower, and tend to over-fit the
models when a small dataset is used (Beyer et al., 2020). Considering this, we
decided to utilize supervised ML classifiers in our experiment. Many studies
show that different ML classifiers can lead to various performance in
different classification tasks and each ML classifier may have its own
suitable software artifacts for achieving the best classification results. For
instance, NB has been reported to outperform other machine learning algorithms
in the automatic classification of app reviews (Maalej et al., 2016). LR
achieves the best performance when identifying code-smell discussions from SO
(Shcherban et al., 2020). SVM gets the best classification result when
identifying assumptions and decisions from the Hibernate developer mailing
list (Li et al., 2019c, a). Therefore, four widely used machine learning
classifiers (i.e., NB, LR, SVM, and RF) are used as base classifiers in our
experiment to seek the best configuration to classify decisions.

###  3.6. Ensemble Classifiers

Ensemble learning is getting more and more attention in recent years and it
has been widely used in many areas to solve classification problems. In
contrast to base machine learning approaches, which construct one classifier
from the training data, an ensemble classifier constructs a set of base
classifiers and combines predictions from base classifiers into the final
prediction to solve the same problem (Zhou, 2012). Dietterich considered that
it is often possible to create excellent ensemble classifiers and attributed
the benefits of ensemble classifiers to three fundamental reasons:
Statistical, Computational, and Representational (Dietterich, 2000). Besides,
many research results also show that ensemble classifiers can obtain better
performance than base classifiers (Huang et al., 2018; Guzman et al., 2015; Da
Silva et al., 2014), which motivates us to apply ensemble methods in our
experiment and compare the performances between base classifiers and ensemble
classifiers when classifying decisions.

However, effective ensemble classifiers require that the base classifiers can
exhibit some level of diversity (Da Silva et al., 2014). Brown et al.
suggested a taxonomy of methods for creating diversity in ensemble
classifiers: (1) varying the start points in the hypothesis space; (2) varying
the set of accessible hypothesis (e.g., by manipulating the training set for
base classifiers); and (3) varying the way each base classifier traverses the
space of possible hypotheses (e.g., by using different base classifiers or
combination strategies) (Brown et al., 2005). In this work, we focus on the
third method and use different combinations of four base classifiers (see
Section 3.5) to construct five ensemble classifiers for classifying decisions.
Also, soft voting, one of the most popular and fundamental combination methods
(Zhou, 2012), is chosen as the combination method for the five ensemble
classifiers. It averages probabilities obtained by base classifiers for each
class, and then outputs the class with the maximum average probability as the
final prediction result. Table 2 presents the detailed information of these
ensemble classifiers, in which SVE denotes Soft Vote Ensemble.

Note that there is no guarantee that ensemble classifiers will always perform
better than base classifiers. Therefore, we used the five ensemble classifiers
to seek whether the ensemble classifiers can outperform the base classifiers
when classifying decisions.

Table 2. Combinations of Base Classifiers for Ensemble Classifiers Ensemble classifiers | Combination of base classifiers |
---|---|---
SVE1 | NB, LR, SVM, and RF |
SVE2 | NB, LR, and SVM |
SVE3 | LR, SVM, and RF |
SVE4 | NB, SVM, and RF |
SVE5 | NB, LR, and RF |

##  4\. Experiment

In this section, we present in detail how we designed our experiment to
automatically classify the decisions from the Hibernate developer mailing
list. We first specify the process of the decision classification experiment,
and then describe each phase in the process. Finally, we provide the
methodology to evaluate the performance of classification results.

###  4.1. Decision Classification Process

In this section, we show the execution process we followed and the techniques
we employed to conduct our experiment. Specially, the process consists of six
phases:

Phase 1: Collect Decisions: We collected decisions from the Hibernate
developer mailing list.

Phase 2: Preprocess Decisions: We preprocessed the collected decisions by
eliminating useless textual characters, carrying out lemmatization, and
removing stop words.

Phase 3: Select Features: We applied CHI2 feature selection algorithm to
identify a subset of features that are most useful in differentiating decision
types.

Phase 4: Extract Features: We applied three feature extraction techniques
(i.e., BoW, TF-IDF, and Word2Vec) to extract textual features and calculate
the vector value of each decision.

Phase 5: Train Classifiers: We employed four machine learning classification
algorithms (i.e., NB, LR, SVM, and RF) to separately train base classifiers
and then we trained five ensemble classifiers constructed by the
aforementioned four base classifiers. After that, decisions would be
classified by the four trained base classifiers and the five trained ensemble
classifiers.

Phase 6: Evaluate Trained Classifiers: We used weighted precision, weighted
recall, and weighted F1-score to evaluate the performance of the trained
classifiers in Phase 5.

###  4.2. Decisions Collection

In our recent study, we conducted a replication experiment for automatically
identifying decisions from the Hibernate developer mailing list (Fu et al.,
2021a). The dataset used in the replication experiment contains 844 decisions.
In this work, we reused the 844 decisions and manually labelled these
decisions into five decision types. To mitigate any personal bias in creating
the dataset, we first conducted a pilot labelling with 300 decisions by the
first and third authors independently. Conflicts were discussed and resolved
with the second author to make sure that they had a consistent understanding
of the criteria of data labelling. The first and third authors then labelled
the remaining decisions independently. Disagreements on labelled decisions
were still discussed and resolved with the second author. We measured the
inter-rater reliability and calculated Cohen’s Kappa coefficient (Cohen, 1960)
as a way to verify the consistency between the labelled decisions by the first
and third authors. The result is 0.856, which indicates that the two authors
reached a good agreement. The detailed statistics of our dataset is presented
in Table 3. In the meanwhile, the dataset has been provided online (Fu et al.,
2021b).

Table 3. Numbers and Proportions of Manually Labelled Decisions in the Dataset Type |  | Number |  | Proportion
---|---|---|---|---
Design Decision |  | 314 |  | 0.3720
Requirement Decision |  | 299 |  | 0.3543
Management Decision |  | 109 |  | 0.1291
Construction Decision |  | 99 |  | 0.1173
Testing Decision |  | 23 |  | 0.0273
Total |  | 844 |  | 1.0000

###  4.3. Decisions Preprocessing

The process of text preprocessing is composed of three steps:

1) Eliminating Useless Textual Characters: Due to the fact that Hibernate
developers may use natural language to communicate with other developers
through mails, there may be a lot of useless characters in the decision
sentences, such as “:)” and “*”. We only need to consider terms that can
provide valid information for classifying decisions and thus may improve
classification performance. Therefore, we eliminated useless characters and
numbers, and only retained valid terms.

2) Lemmatization: Lemmatization is one of the indispensable steps in the
natural language processing, which can transform a word into its root word. We
carried out lemmatization for decisions using NLTK, an open source Python
library for Natural Language Processing.

3) Removing Stop Words: Stop words refer to those common but convey little
meaning words, such as “and” and “that”. Removing stop words can help to
reduce the size of vocabulary of our dataset and reduce the dimension of
feature vectors of the decisions. In this work, we combined English stop words
list provided by NLTK and several self-defined stop words, such as “IMHO” that
means “in my humble opinion”, to remove stop words in decisions.

###  4.4. Feature Selection

In the feature selection step, we excluded terms that appear only once since
they may contain less valid information for classification. Additionally,
different percentages of selected features may also impact the performance of
the automatic approach, thus we considered the percentage of the selected
features as part of configurations and intended to seek the best configuration
for classifying decisions from the Hibernate developer mailing list.
Investigating the impact of all percentages (i.e., 1% to 100%) of the selected
features for each combination of feature extraction techniques and ML
classifiers is rather time-consuming. To simplify our experiment, the
percentage of selected features was set from 10% to 100% (with a step of 10%).
Note that feature selection will make no difference when all the (100%)
features are selected.

###  4.5. Feature Extraction and Classifier Training

We conducted a vectorization process for decisions in our dataset and
transformed them into vectors that could represent the features of decisions
through feature extraction techniques. As elaborated in Section 3.4, BoW, TF-
IDF, and Word2Vec were exploited to extract textual features. Then these
vectors were used to train and evaluate the four base classifiers and five
ensemble classifiers. Multinomial NB was chosen as the version of NB since it
is better than other NB algorithms (McCallum et al., 1998). Also, we
normalized the feature vectors obtained by Word2Vec in the range of [0,1]
since Multinomial NB fails when some of features are negative. The main
hyperparameters of the four base classifiers are listed in Table 4. We used
scikit-learn and gensim toolbox to implement the aforementioned feature
extraction techniques and classifiers because they provide solid
implementations of a bunch of state-of-the-art models.

Table 4. Main Hyperparameters of Four Base Classifiers Base classifiers | Main hyperparameters |
---|---|---
NB |  $alpha$: 1 |
LR |  $C$: 1.0; $max\\_iter$: 100; $solver$: lbfgs |
SVM |  $kernel$: linear; $C$: 1.0; |
RF |  $n\\_estimators$: 100; $criterion$: gini |

###  4.6. Evaluation

Precision, recall, and F1-score (see Formula (3), (4) and (5)) were adopted as
standard metrics to evaluate the performance of the aforementioned
classifiers. Furthermore, we introduced weighted average precision, recall,
and F1-score (see Formula (6)), which are commonly used to measure the
performance of classification results in imbalance datasets. To reduce
overfitting in the classification of decisions, a stratified 10-fold cross
validation was applied on the dataset, which is commonly used in machine
learning as it produces less biased accuracy estimations for datasets with
small sample sizes (Refaeilzadeh et al., 2009). In addition, we repeated the
10-fold cross validation ten times and calculated the average results.

(3) $Precision_{i}=\frac{TP_{i}}{TP_{i}+FP_{i}}$

(4) $Recall_{i}=\frac{TP_{i}}{TP_{i}+FN_{i}}$

(5) $F1_{i}-score=\frac{2\times Precision_{i}\times
Recall_{i}}{Precision_{i}+Recall_{i}}$

(6) $\displaystyle Weighted\ (Precision\backslash Recall\backslash F1)=$
$\displaystyle\frac{\sum\limits_{i\in type}(Precision_{i}\backslash
Recall_{i}\backslash F1_{i})\times Number_{i}}{\sum\limits_{i\in
type}Number_{i}}$

In Formula (3) and (4), $TP_{i}$ denotes the number of decisions that are
classified as type $i$ and actually belong to type $i$; $FP_{i}$ denotes the
number of decisions that are classified as type $i$ but actually belong to
another type $j$ ($i$ $\neq$ $j$); $FN_{i}$ denotes the number of decisions
that are of type $i$ but classified as another type $j$ ($i$ $\neq$ $j$). In
Formula (6), $Number_{i}$ denotes the number of decisions labelled as decision
type $i$ in the testing set.

##  5\. Results and Analysis

As described in Section 3, we performed our experiment with 270 configurations
(i.e, 10 different values of $n$ in features selection $\times$ 3 feature
extraction techniques $\times$ 9 ML classifiers). For each configuration, we
calculated the metrics (i.e., weighted precision, recall, and F1-score). In
this section, we report the experiment results and answer the RQs.

To answer RQ1: To seek whether feature selection improves the classification
results, we compared the classification results for all combinations of
feature extraction techniques and classifiers with and without feature
selection as shown in Table 5, in which FS denotes Feature Selection.
Especially, we listed the best F1-score and the corresponding value of $n$ for
each combination when the selected features vary from 10% to 90% (See Section
4.4). It can be found that the F1-scores of all combinations increase when
applying feature selection. The lowest improvement (1.9%) is achieved by BoW +
RF, whereas Word2Vec + RF achieves the highest improvement (40.6%). The
improvements also differ from different feature extraction techniques. For
BoW, the F1-scores are slightly increased with an average improvement of 5.3%.
For TF-IDF, the results are better compared with BoW. On average, the feature
selection improves F1-score by 6.6%. The F1-scores of Word2Vec are
significantly improved compared with BoW and TF-IDF. The average improvement
reaches 25.9%. Another important observation is that for the same feature
extraction technique, the percentages of selected features (i.e., the value of
$n$) are close when most classifiers achieve the best result. For example, 50%
for BoW, and 20%$\sim$40% for TF-IDF. Moreover, nine classifiers with Word2Vec
all achieve the best F1-score when 10% features are selected since the
F1-score will decrease when we increase the percentage of selected features.
Besides the improvement on classification results, feature selection can
provide extra performance since it can reduce the dimensions of data on which
we need to train. In summary, feature selection has an impact on the
classification results and can decently improve the classification results
when classifying decisions from the Hibernate developer mailing list.

Table 5. Comparison of F1-scores of All Combinations of Feature Extraction Techniques and Classifiers with and without Feature Selection Combinations | NoFS | FS ($n$) | Improve
---|---|---|---
BoW + NB | 0.653 | 0.709 (40%) | 8.6%
BoW + LR | 0.678 | 0.713 (50%) | 5.2%
BoW + SVM | 0.650 | 0.715 (50%) | 10.0%
BoW + RF | 0.645 | 0.657 (10%) | 1.9%
BoW + SVE1 | 0.695 | 0.722 (50%) | 3.9%
BoW + SVE2 | 0.689 |  0.727 (50%) | 5.5%
BoW + SVE3 | 0.680 | 0.712 (50%) | 4.7%
BoW + SVE4 | 0.693 | 0.722 (50%) | 4.2%
BoW + SVE5 | 0.695 | 0.719 (50%) | 3.5%
Avg | 0.675 | 0.711 | 5.3%
TF-IDF + NB | 0.584 | 0.684 (20%) | 17.1%
TF-IDF + LR | 0.640 | 0.703 (20%) | 9.8%
TF-IDF + SVM | 0.684 | 0.716 (50%) | 4.7%
TF-IDF + RF | 0.640 | 0.672 (40%) | 5.0%
TF-IDF + SVE1 | 0.690 | 0.720 (40%) | 4.3%
TF-IDF + SVE2 | 0.686 | 0.720 (30%) | 5.0%
TF-IDF + SVE3 | 0.693 |  0.724 (40%) | 4.5%
TF-IDF + SVE4 | 0.692 | 0.723 (40%) | 4.5%
TF-IDF + SVE5 | 0.656 | 0.700 (20%) | 6.7%
Avg | 0.663 | 0.707 | 6.6%
Word2Vec + NB | 0.377 | 0.500 (10%) | 32.6%
Word2Vec + LR | 0.568 |  0.683 (10%) | 20.2%
Word2Vec + SVM | 0.582 |  0.683 (10%) | 17.4%
Word2Vec + RF | 0.461 | 0.648 (10%) | 40.6%
Word2Vec + SVE1 | 0.533 | 0.673 (10%) | 26.3%
Word2Vec + SVE2 | 0.555 | 0.663 (10%) | 19.5%
Word2Vec + SVE3 | 0.563 | 0.677 (10%) | 20.2%
Word2Vec + SVE4 | 0.515 | 0.667 (10%) | 29.5%
Word2Vec + SVE5 | 0.501 | 0.665 (10%) | 32.7%
Avg | 0.517 | 0.651 | 25.9%

To answer RQ2: To seek whether ensemble classifiers outperform the base
classifiers, we analyzed the classification results of base classifiers and
ensemble classifiers in Table 5. For BoW and TF-IDF, most ensemble classifiers
perform better than base classifiers no matter whether feature selection is
applied. Moreover, the best ensemble classifier outperforms the best base
classifier (e.g., BoW + SVE2 + 50% features achieves higher F1-score than BoW
+ SVM + 50% features). However, not all ensemble classifiers are better than
base classifiers, e.g., the F1-score of TF-IDF + SVM + 50% features (0.716) is
better compared with the F1-score of TF-IDF + SVE5 + 20% (0.700). For
Word2Vec, we can learn that the results are different. The classification
results obtained by LR and SVM are better than the results of the five
ensemble classifiers, whereas the classification results of NB and RF are
worst. Since an ensemble classifier combines the predictions from base
classifiers into the final prediction, performance of the ensemble classifier
is thus influenced by the base classifiers. The base classifiers with good
performance could benefit to construct a well-performed ensemble classifier
while the base classifiers with bad performance would be harmful. For
instance, NB and RF perform significantly worse than SVM and LR when using
Word2Vec without feature selection, which results in the worse performance of
five ensemble classifiers. The results of NB, LR, and SVM are relatively good
when using BoW with feature selection, which results in that the ensemble
classifier that combines these three base classifiers achieve a better
classification result. In summary, the ensemble classifiers may outperform
base classifiers, but it depends on how these ensemble classifiers are
constructed when classifying decisions from the Hibernate developer mailing
list. It indicates that the construction of ensemble classifiers should also
be considered when using ensemble learning in automatic classification tasks.
In order to make the ensemble learning more valuable in practice, it is
necessary to seek suitable base classifiers, which is also our next step.

To answer RQ3: To determine the best configuration that achieves the best
performance to classify decisions from the Hibernate developer mailing list,
we compared and ranked all configurations based on their weighted F1-scores.
We listed the top 5 configurations in Table 6. It can be found that the
F1-scores of the top 5 configurations are all higher than 0.72. We can also
learn that the highest precision (i.e., 0.754) and recall (i.e., 0.739) are
achieved by TF-IDF + SVE4 + 40% features and BoW + SVE2 + 50% features,
respectively. Overall, the BoW + SVE2 + 50% features obtained the best
classification result with a weighted F1-score of 0.727. In addition, the top
5 configurations have some characteristics in common: (1) they all applied
feature selection; (2) they all employed ensemble classifier; and (3) they do
not utilize Word2Vec to extract features of decisions.

The results in Table 6 indicate that (1) feature selection has an advantage
when classifying decisions from the Hibernate developer mailing list, which is
consistent with our conclusion of RQ1; (2) ensemble classifiers could achieve
better classification results than base classifiers, which confirms our
conclusion of RQ2; (3) BoW and TF-IDF are more suitable feature extraction
techniques compared with Word2Vec in our cases. This finding is in line with
the result in (Li et al., 2019a), which also focused on decisions from the
Hibernate developer mailing list. One possible reasons could be that the size
of our dataset is limited. Thus, limited information is provided for Word2Vec
to learn enough semantic knowledge, which may lead to the poor performance of
Word2Vec. In summary, BoW + SVE2 + 50% features outperforms other
configurations. It can automatically classify decisions from the Hibernate
developer mailing list with a weighted precision of 0.750, a weighted recall
of 0.739, and a weighted F1-score of 0.727.

Table 6. Results of the Best 5 Configurations for Classifying Decisions Configurations | Precsion | Recall | F1
---|---|---|---
BoW + SVE2 + 50% features | 0.750 | 0.739 | 0.727
TF-IDF + SVE3 + 40% features | 0.753 | 0.737 | 0.724
TF-IDF + SVE4 + 40% features | 0.754 | 0.737 | 0.723
BoW + SVE1 + 50% features | 0.747 | 0.734 | 0.722
BoW + SVE4 + 50% features | 0.745 | 0.734 | 0.722

##  6\. Discussion

###  6.1. Feature Terms after Feature Selection

In our experiment, we use feature selection to select terms that may contain
valid type information for classification. We present the top 20 feature terms
in Table 7 after feature selection for each decision type. We focus on seeking
which terms may provide expected discrimination between types. However, since
the number of testing decisions in our dataset is limited (i.e., 23 of 844)
and we excluded terms that appear only once in feature selection (see Section
4.4), there are less than 20 terms in Testing Decision. Note that the CHI2
value indicates the amount of information carried by a term for predicting the
type of a decision, but a term with a high CHI2 value of one decision type
does not necessarily mean that the decision with this term belongs to this
type.

To better understand these feature terms, we also manually checked each term
in Table 7 and highlighted terms that are highly relevant to each decision
type by the first and second authors. Any conflicts were discussed and
resolved among all the authors. From the results, we can find that feature
selection can pick out the terms that are highly related to the definition of
each decision type (see Section 3.2). For Design Decision, to best of our
knowledge, terms like “move”, “refactor”, and “design” may be highly
correlated to design decisions (e.g., “He has started refactoring SessionImpl,
as the first step of redesigning Hibernate to be more event oriented, as we
all discussed a while ago”). However, there are also feature terms that are
not directly relevant to the design of software systems, such as “session”,
“exception”, “transaction”, and “bean”. Since Hibernate, an Object/Relation
Mapping (ORM) library for java projects, is used as the source of our dataset,
these terms are specific concepts in the Hibernate and would be involved when
Hibernate designers make decisions. For example, session is used to get a
physical connection with a database, and it plays a major role between the
database and Hibernate. Therefore, “session” is emphasized in the design
decisions, such as “Keep a single session in a ThreadLocal, or the DAOFactory,
to be shared by all DAO instances in that transaction”.

“support”, “feature”, “request”, and so on are feature terms of Requirement
Decision. Decisions with these terms are usually related to developers
improving the requirements on functionalities, such as “There should also be a
feature enquiry function to allow the API programmer to ask the underlying
Hibernate implementation to verify if a replication on identity table will
work”. Besides, “performance”, as one of the common non-functional
requirements, is also highly related to Requirement Decision. Hibernate
developers would make decisions to improve the quality of Hibernate products,
as well as “a performance gain could be achieved (on some databases) by
selectively updating columns”.

For Management Decision, “documentation”, “javadoc”, and so on are the terms
that indicate the maintenance and improvements in documentations of Hibernate,
such as “The new internationalized documentation will be in 2.1.3, so we have
to update the Wiki documentation page before release”, which meets the
definition of Management Decision. In the meanwhile, “release” is the term
with the highest CHI2 value for Management Decision. The correlative decisions
are made when a new version of Hibernate is completed and ready to release,
such as “We will release Hibernate 3.0alpha early next week”.

In addition, most feature terms of Construction Decision are related to the
source code and configuration files of the Hibernate, such as “override”,
“annotation”, and “buildxml”, since construction decisions focus on the
implementation details of the software systems. Hibernate developers would
discuss the code issues with each other in the mailing list, such as “I have
also overriden the hashCode() and equals() methods on StuffUnitFactor so that
they are equal when stuff && unit are equal”.

As for Testing Decision, “test” is the most frequent term and it is always
related to the testing activities during the development and maintenance of
software, such as “unit test” and “test cases” (e.g., “I will add a test case
for the new Transaction API to the FooBarTests as soon as I get a chance”).

We also find that different decision types may share some common terms, such
as “add” and “use”. Most of these terms are verbs and are frequently used in
the communication between stakeholders. For example, Hibernate developers
would use linguistic patterns like “add support” and “add ability” to express
the improvement on requirements.

Table 7. Top 20 Feature Terms after Feature Selection for Each Decision Type Design Decision | Requirement Decision | Management Decision | Construction Decision | Testing Decision |
---|---|---|---|---|---
add | support | release | override | test |
session | add | documentation | line | suite |
use | feature | page | parameter | unit |
support | implement | wiki | code | run |
join | performance | final | equal | testing |
move | write | readme | classpath | case |
application | hql | document | annotation | order |
exception | code | doc | lib | fix |
place | criterion | vbranch | hibernateproperties | functionality |
way | full | hibernateext | false | get |
transaction | request | reference | unsavedvalue | cvs |
remove | use | week | embrace | need |
refactor | generate | website | style | cache |
bean | task | section | figure | new |
design | eclipse | note | onetomany | file |
test | custom | branch | message | use |
put | syntax | guide | finally | make |
solution | test | distribution | cut | add |
return | select | jar | extra | change |
directly | provide | javadoc | bulidxml |  |

###  6.2. Implications for Practitioners

Our approach can benefit various types of stakeholders in software development
(e.g., project managers, requirements engineers, designers, and developers),
because it could effectively classify decisions into specific types that are
useful for them. Since there are approaches focusing on the automatic
identification of decisions (e.g., (Li et al., 2019a)) but lacking the
multiclass classification methods of decisions (not only design decisions),
our approach could help project managers better document decisions with
minimal human effort. Then other stakeholders could review, analyze, improve,
and reuse the decisions that they are concerned with. For instance, designers
could gain knowledge from past design decisions, such as ideas and insights
toward the design issues, which could help them make better decisions in a
similar design context. Requirement engineers could be aware of what decisions
have been made toward the functional and non-functional requirements, and then
determine new requirements to improve the software system. Developers can be
aware of the impact of software changes caused by related decisions, which
helps the developers to avoid making poor decisions that lead to bug proneness
and increased maintenance effort (Paixao et al., 2017). Moreover, our approach
could inform the newcomers of the decisions that are highly related to their
work so that they can have a better understanding of the software system and
get involved in the work more efficiently. Also, our approach is also scalable
in practice. Project managers could re-select feature terms and retrain our
approach while more decisions are classified.

###  6.3. Implications for Researchers

Decisions, as a type of inherent artifact in software development and
maintenance, get widespread attention from researchers in recent years (e.g.,
(Mendes et al., 2019)). Many researches have focused on analyzing decisions,
especially design decisions, and their making in software development. In this
work, we adopted the taxonomy from our previous research (Li et al., 2019b)
that classifies decisions in software development into five types, which can
basically cover the decisions in the Hibernate developer mailing list. We
further adapted the description of the five types of decisions and provided
the decision dataset online (Fu et al., 2021b). This dataset can be extended
with the decisions from other projects and sources, acting as a foundation for
other researchers to work on automatic classification of decisions. We also
propose an automatic approach (i.e., the BoW + 50% features selected by CHI2
algorithm with an ensemble classifier that combines NB, LR, and SVM) to
classify decisions. Although our approach is evaluated on one OSS project
(i.e., Hibernate), we believe that our approach could provide the basis for
designing and developing tools to automatically classify decisions, with the
knowledge and usage of feature selection and ensemble learning. In addition,
considering that the research in the area of automatic decision classification
is still in its infancy, we advocate that researchers can pay more attention
to the design of automatic and interactive tools for classifying decisions in
software development.

##  7\. Threats to Validity

In this section, we discuss several threats to the validity of this work
according to the guidelines proposed by Runeson et al. (Runeson and Höst,
2009), and how these threats were partially mitigated in our study. Internal
validity is not considered, since we do not address causal relationships
between variables and results.

Construct Validity reflects on the extent of consistency between the
operational measures of the study and the RQs (Runeson and Höst, 2009). A
potential threat in this study involves whether the decisions used for the
experiment were classified correctly by the researchers. To mitigate this
threat, we reviewed and labelled the decisions by two researchers. Any
disagreements were discussed and resolved with another researcher. Moreover,
we conducted a pilot labeling with 300 decisions to mitigate any personal bias
and achieve a consensus understanding on manually labelling. Another potential
threat in this study is whether the dataset for the experiment is sufficient
enough to obtain reasonable conclusions since our dataset only contains 844
decisions and is also quite imbalanced. To mitigate this threat and get
reasonable results, we repeated stratified 10-fold cross validation 10 times
with three weighted metrics (i.e., weighted precision, weighted recall, and
weighted F1-score) to evaluate our experiment results. In addition, Manning et
al. argued that the classification performance will be improved when
increasing the sample size of the dataset (Manning et al., 1999). Therefore,
we conjecture that we can obtain more accurate conclusions by increasing the
sample size of the dataset and making it balanced to mitigate the threat of
the insufficient dataset, which is also our next research goal.

External Validity refers to the degree to which our study results and findings
can be generalized in other cases (e.g., the developer mailing lists of other
OSS projects). We collected decisions from the developer mailing list of
Hibernate since it is one of the most popular OSS projects and is widely used
in industry as an ORM library for Java projects. However, we acknowledge that
our study may not be generalizable to other OSS projects. For instance, the
feature terms acquired for Hibernate may not be applicable to other projects.
Different projects can lead to different feature terms since these terms may
be related to a variety of aspects, such as programming language and domain of
the project. But we believe that our work can promote researches on
classifying decisions into types that are beneficial for various types of
stakeholders in software development. In the next step, we plan a study
through extending our dataset of decisions from more OSS projects with various
characteristics (e.g., other programming languages and domains), which will
alleviate this threat.

Reliability refers to whether the experiment yields the same results when it
is replicated by other researchers. To improve the reliability, we made a
research protocol and present all the details of the experiment in Section 4.
Furthermore, the dataset and the source code of our experiments are made
publicly available online in order to facilitate other researchers replicating
our experiment easily (Fu et al., 2021b). We believe that this measure can
partially alleviate this threat.

##  8\. Conclusions and Future Work

Decisions are a typical type of software artifacts and they have a crucial
impact on development and maintenance of software systems. Nevertheless,
decisions are not well documented due to considerable human resources, time,
and budget. To address this issue, we conduct an experiment to automatically
classify decisions from the Hibernate developer mailing list into five
decision types (i.e., Design, Requirement, Management, Construction, and
Testing Decision), which were got in our previous work using grounded theory
(Li et al., 2019b).

In this work, we first collected our dataset with 844 decisions of five
decision types. To obtain the best approach for classifying decisions, we
performed our experiment with 270 configurations. In feature selection step,
we varied the percentages of the selected features from 10% to 100% (with a
step of 10%). Three well-performed feature extraction techniques (i.e., BoW,
TF-IDF, and Word2Vec) were used to extract features of decisions. As for
classifiers, we utilized an ensemble learning approach and constructed five
ensemble classifiers using four base ML classifiers (i.e., NB, LR, SVM, and
RF). We also compared the performance between base classifiers and ensemble
classifiers. All the configurations were evaluated through weighted precision,
recall, and F1-score. The results show that (1) feature selection can decently
improve the classification results; (2) ensemble classifiers can outperform
base classifiers provided that ensemble classifiers are well constructed; (3)
BoW + 50% features selected by feature selection with an ensemble classifier
that combines NB, LR, and SVM achieves the best classification result (with a
weighted precision of 0.750, a weighted recall of 0.739, and a weighted
F1-score of 0.727) among all the configurations. Our approach can benefit
various types of stakeholders in software development (e.g., project managers,
requirements engineers, designers, and developers) by classifying decisions
into specific types that are relevant to their interests.

Given the importance of decisions in software development, we plan to: (1)
collect decisions from other OSS projects to get a larger and balanced
dataset, and then validate our approach on these projects; (2) evaluate other
state-of-art techniques (e.g., FastText and Text-CNN) to improve the
performance of our approach on classifying decisions; (3) use automatic
approaches to mine diverse information from decisions (e.g., rationale of
decisions) to enrich the documentation of decisions; and (4) use automatic
approaches to classify other types of software artifacts related to decisions,
such as assumptions, to promote the understanding and (re)use of decisions in
software development.

###### Acknowledgements.

This work has been partially supported by the National Key R&D Program of
China with Grant No. 2018YFB1402800.

## References

  * (1)
  * Ambler (2002) Scott Ambler. 2002\.  _Agile Modeling: Effective Practices for Extreme Programming and the Unified Process_.  John Wiley & Sons.
  * Basili et al. (1994) Victor R. Basili, Gianluigi Caldiera, and H. Dieter Rombach. 1994.  The Goal Question Metric Approach.  _Encyclopedia of Software Engineering_ 1 (1994), 528–532.
  * Beyer et al. (2020) Stefanie Beyer, Christian Macho, Massimiliano Di Penta, and Martin Pinzger. 2020.  What kind of questions do developers ask on Stack Overflow? A comparison of automated approaches to classify posts into question categories.  _Empirical Software Engineering_ 25, 3 (2020), 2258–2301.
  * Bhat et al. (2017) Manoj Bhat, Klym Shumaiev, Andreas Biesdorf, Uwe Hohenstein, and Florian Matthes. 2017\.  Automatic extraction of design decisions from issue management systems: A machine learning based approach. In _Proceedings of the 11th European Conference on Software Architecture (ECSA)_. Springer, 138–154.
  * Bhat et al. (2019) Manoj Bhat, Christof Tinnes, Klym Shumaiev, Andreas Biesdorf, Uwe Hohenstein, and Florian Matthes. 2019\.  Adex: A tool for automatic curation of design decision knowledge for architectural decision recommendations. In _Proceedings of the 2019 IEEE International Conference on Software Architecture Companion (ICSA-C)_. IEEE, 158–161.
  * Brown et al. (2005) Gavin Brown, Jeremy Wyatt, Rachel Harris, and Xin Yao. 2005\.  Diversity Creation Methods: A Survey and Categorisation.  _Information Fusion_ 6, 1 (2005), 5–20.
  * Cohen (1960) Jacob Cohen. 1960\.  A Coefficient of Agreement for Nominal Scales.  _Educational and Psychological Measurement_ 20, 1 (1960), 37–46.
  * Da Silva et al. (2014) Nadia F.F Da Silva, Eduardo R. Hruschka, and Estevam R. Hruschka Jr. 2014.  Tweet Sentiment Analysis with Classifier Ensembles.  _Decision Support Systems_ 66 (2014), 170–179.
  * Dietterich (2000) Thomas G Dietterich. 2000\.  Ensemble methods in machine learning. In _Proceedings of the 1st International Workshop on Multiple Classifier Systems (MCS)_. Springer, 1–15.
  * Ding et al. (2014) Wei Ding, Peng Liang, Antony Tang, Hans Van Vliet, and Mojtaba Shahin. 2014.  How do open source communities document software architecture: An exploratory survey. In _Proceedings of the 19th International Conference on Engineering of Complex Computer Systems (ICECCS)_. IEEE, 136–145.
  * Dutoit et al. (2006) Allen H Dutoit, Raymond McCall, Ivan Mistrík, and Barbara Paech. 2006.  Rationale Management in Software Engineering: Concepts and Techniques.  In _Rationale Management in Software Engineering_. Springer, 1–48.
  * Forman (2003) George Forman. 2003\.  An extensive empirical study of feature selection metrics for text classification.  _Journal of Machine Learning Research_ 3 (2003), 1289–1305.
  * Fu et al. (2021a) Liming Fu, Peng Liang, Xueying Li, and Yang Chen. 2021a.  Will data influence the experiment results?: A replication study of automatic identification of decisions. In _Proceedings of the 28th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)_. IEEE, 614–617.
  * Fu et al. (2021b) Liming Fu, Peng Liang, Xueying Li, and Chen Yang. 2021b.  Replication Package for the Paper: A Machine Learning Based Ensemble Method for Automatic Multiclass Classification of Decisions. https://github.com/DoneEI/Automatic-Classification-of-Decisions.
  * Fu and Menzies (2017) Wei Fu and Tim Menzies. 2017.  Easy over hard: A case study on deep learning. In _Proceedings of the 11th Joint Meeting on Foundations of Software Engineering (FSE)_. ACM, 49–60.
  * Guzman et al. (2015) Emitza Guzman, Muhammad El-Haliby, and Bernd Bruegge. 2015\.  Ensemble methods for app review classification: An approach for software evolution (N). In _Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 771–776.
  * Huang et al. (2018) Qiao Huang, Emad Shihab, Xin Xia, David Lo, and Shanping Li. 2018.  Identifying self-admitted technical debt in open source projects using text mining.  _Empirical Software Engineering_ 23, 1 (2018), 418–451.
  * Li et al. (2019c) Ruiyin Li, Peng Liang, Chen Yang, Georgios Digkas, Alexander Chatzigeorgiou, and Zhuang Xiong. 2019c.  Automatic identification of assumptions from the Hibernate developer mailing list. In _Proceedings of the 26th Asia-Pacific Software Engineering Conference (APSEC)_. IEEE, 394–401.
  * Li et al. (2019a) Xueying Li, Peng Liang, and Zengyang Li. 2019a.  Automatic Identification of Decisions from the Hibernate Developer Mailing List. In _Proceedings of the 24th International Conference on Evaluation and Assessment in Software Engineering (EASE)_. ACM, 51–60.
  * Li et al. (2019b) Xueying Li, Peng Liang, and Tianqing Liu. 2019b.  Decisions and Their Making in OSS Development: An Exploratory Study Using the Hibernate Developer Mailing List. In _Proceedings of the 26th Asia-Pacific Software Engineering Conference (APSEC)_. IEEE, 323–330.
  * Lu and Liang (2017) Mengmeng Lu and Peng Liang. 2017.  Automatic classification of non-functional requirements from augmented app user reviews. In _Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering (EASE)_. ACM, 344–353.
  * Maalej et al. (2016) Walid Maalej, Zijad Kurtanović, Hadeer Nabil, and Christoph Stanik. 2016.  On the automatic classification of app reviews.  _Requirements Engineering_ 21, 3 (2016), 311–331.
  * Manning et al. (1999) Christopher D Manning, Christopher D Manning, and Hinrich Schütze. 1999.  _Foundations of Statistical Natural Language Processing_.  MIT Press.
  * McCallum et al. (1998) Andrew McCallum, Kamal Nigam, et al. 1998\.  A comparison of event models for naive bayes text classification. In _Proceedings of the AAAI-98 Workshop on Learning for Text Categorization (TextCat)_. Citeseer, 41–48.
  * Mendes et al. (2019) Fabiana Freitas Mendes, Emilia Mendes, and Norsaremah Salleh. 2019\.  The relationship between personality and decision-making: A Systematic literature review.  _Information and Software Technology_ 111 (2019), 50–71.
  * Paixao et al. (2017) Matheus Paixao, Jens Krinke, DongGyun Han, Chaiyong Ragkhitwetsagul, and Mark Harman. 2017\.  Are developers aware of the architectural impact of their changes?. In _Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 95–105.
  * Razavian et al. (2019) Maryam Razavian, Barbara Paech, and Antony Tang. 2019\.  Empirical research for software architecture decision making: An analysis.  _Journal of Systems and Software_ 149 (2019), 360–381.
  * Refaeilzadeh et al. (2009) Payam Refaeilzadeh, Lei Tang, and Huan Liu. 2009.  Cross-Validation.  _Encyclopedia of Database Systems_ 5 (2009), 532–538.
  * Rogers et al. (2015) Benjamin Rogers, Yechen Qiao, James Gung, Tanmay Mathur, and Janet E Burge. 2015.  Using text mining techniques to extract rationale from existing documentation. In _Proceedings of the 6th International Conference on Design Computing and Cognition (DCC)_. Springer, 457–474.
  * Rong (2014) Xin Rong. 2014.  Word2vec parameter learning explained.  _arXiv_ abs/1411.2738 (2014).
  * Runeson and Höst (2009) Per Runeson and Martin Höst. 2009.  Guidelines for conducting and reporting case study research in software engineering.  _Empirical Software Engineering_ 14, 2 (2009), 131–164.
  * Rustam et al. (2020) Furqan Rustam, Arif Mehmood, Muhammad Ahmad, Saleem Ullah, Dost Muhammad Khan, and Gyu Sang Choi. 2020\.  Classification of shopify app user reviews using novel multi text features.  _IEEE Access_ 8 (2020), 30234–30244.
  * Shah et al. (2018) Faiz Ali Shah, Kairit Sirts, and Dietmar Pfahl. 2018\.  Simple app review classification with only lexical features. In _Proceedings of the 13th International Conference on Software Technologies (ICSOFT)_. SciTePress, 112–119.
  * Shahbazian et al. (2018) Arman Shahbazian, Youn Kyu Lee, Duc Le, Yuriy Brun, and Nenad Medvidovic. 2018.  Recovering architectural design decisions. In _Proceedings of the 2018 IEEE International Conference on Software Architecture (ICSA)_. IEEE, 95–104.
  * Shcherban et al. (2020) Sergei Shcherban, Peng Liang, Amjed Tahir, and Xueying Li. 2020\.  Automatic identification of code smell discussions on Stack Overflow: A preliminary investigation. In _Proceedings of the 14th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)_. ACM, 1–6.
  * van der Ven et al. (2006) Jan Salvador van der Ven, Anton GJ Jansen, Jos AG Nijhuis, and Jan Bosch. 2006.  Design Decisions: The Bridge between Rationale and Architecture.  In _Rationale Management in Software Engineering_. Springer, 329–348.
  * Yang and Pedersen (1997) Yiming Yang and Jan O Pedersen. 1997.  A comparative study on feature selection in text categorization. In _Proceedings of the 14th International Conference on Machine Learing (ICML)_ , Vol. 97. Morgan Kaufmann, 412–420.
  * Zhou (2012) Zhi-Hua Zhou. 2012\.  _Ensemble Methods: Foundations and Algorithms_.  CRC Press.