[style=theoremstyle]definitionDefinition

# OSS effort estimation using software features similarity and developer
activity-based metrics

Ritu Kapur  and  Balwinder Sodhi  Indian Institute of Technology
RoparDepartment of Computer Science and EngineeringP.O. Box
1212RupnagarPunjab140001India<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract.

Software development effort estimation (SDEE) generally involves leveraging
the information about the effort spent in developing similar software in the
past. Most organizations do not have access to sufficient and reliable forms
of such data from past projects. As such, the existing SDEE methods suffer
from low usage and accuracy.

We propose an efficient SDEE method for open source software, which provides
accurate and fast effort estimates. The significant contributions of our paper
are i) Novel _SDEE software metrics_ derived from developer activity
information of various software repositories, ii) _SDEE dataset_ comprising
the SDEE metrics’ values derived from $\approx 13,000$ GitHub repositories
from 150 different software categories, iii) an effort estimation tool based
on SDEE metrics and a _software description similarity model_. Our software
description similarity model is basically a machine learning model trained
using the Paragraph Vectors algorithm on the software product descriptions of
GitHub repositories. Given the software description of a newly-envisioned
software, our tool yields an effort estimate for developing it.

Our method achieves the highest Standard Accuracy score of 87.26% (with
cliff’s $\delta$=0.88 at 99.999% confidence level) and 42.7% with the
Automatic Transformed Linear Baseline model. Our software artifacts are
available at https://doi.org/10.5281/zenodo.5095723.

Effort estimation, software development effort, developer activity, software
maintenance, software planning

††journal: TOSEM††ccs: Software and its engineering Design patterns††ccs:
Software and its engineering Software development process management

##  1\. Introduction

Software Development Effort Estimation (SDEE) for a newly envisioned software
project is performed based on the experiential knowledge gained from software
developed in the past. The sources of such “past projects” are limited to the
development team and the organization procuring the new software. Also, the
existing SDEE methods are either dependent on experts or are inefficient and
slow. Therefore, it is desirable to have tools and techniques which help to
provide accurate software development effort estimates on time. The Open
Source Software (OSS) repository hosts, such as GitHub, are rich software
development sources for various software projects.

###### Definition 1.1 (Software project description).

By software project description, we refer to the main description document
available at the software’s GitHub repository. Such a file typically comprises
the following details:

  1. (1)

_Brief description of the software_ : A summary of its functionality and
characteristics. For instance, the software might be a library, plugin, or
framework.

  2. (2)

_Functionality_ : It describes _what_ the software does. The description of
all the primary functions performed by the software. For instance,
PMD111https://github.com/pmd/pmd – a source code analyzer tool, detects common
programming flaws such as unused variables, empty catch blocks, and
unnecessary object creation.

  3. (3)

_Supporting platform_ : It lists the supportable operating systems and
programming platforms.

  4. (4)

_Execution-specific details_ : These include the essential information about
the script and steps involved in running the software.

###### Definition 1.2 (Paragraph Vector Algorithm).

Paragraph Vector Algorithm (PVA) is an unsupervised machine learning (ML)
algorithm that learns fixed-length feature representations from variable-
length pieces of texts, such as sentences, paragraphs, and documents. The
algorithm represents each document by a dense vector trained to predict words
in the document (Le and Mikolov, 2014).

Stated broadly, the objective of the work presented in this paper is “ _Given
the software description of a newly-envisioned software, estimate the software
development effort required to develop it._ ” To estimate the effort, we
leverage the developer-activity information of the existing similar GitHub
repositories. To detect the similarity among software, we develop a software
similarity detection model by training PVA on the software product
descriptions taken from the OSS projects available at Version Control System
(VCS) repositories, such as GitHub222https://github.com/.

Figure 1. Essential idea of our work

###  1.1. Essential ideas behind our system

Fig. 1 depicts the essential ideas behind our system, listed as follows:

  1. (1)

SDEE for newly-envisioned software is performed by considering the past
software projects of similar nature (Pressman, 2005; Sommerville, 2011).

  2. (2)

VCSs, such as GitHub and GitLab333https://gitlab.com/explore, provide rich
details about the actual software artifacts, the programmers who developed
them, and the time spent in developing those artifacts. Some examples of high-
level data available on various VCS hosts are the details of contributors
working on a repository, the source code commits history, and the release
timeline.

  3. (3)

The software requirements and the project description are expressed in a
_natural language (NL)_ such as English. Today, ML techniques have matured to
enable accurate and efficient detection of text document similarity. It is
possible to identify existing software projects similar to those of a newly
envisioned one.

###### Definition 1.3 (Developer activity information).

By developer activity information of a software repository, we refer to the
following measures:

    1. (a)

_Source code contribution by a developer:_ The number of additions, deletions,
and modifications of SLOC performed by a developer on a software.

    2. (b)

_The timeline of various changes in a VCS:_ It acts as a valuable metric for
computing the time required to develop a software product.

    3. (c)

_The metadata information of the software repository:_ The information such as
the number of developers working on the software repository is essential for
computing the effort required to develop the software product.

A software developer is majorly involved in the following aspects of a
software development project:

    1. (a)

Developing source code, which is added to the repository in the form of
commits.

    2. (b)

_Addressing the issues or tickets:_ Changes to the source code are the most
likely and frequent outcome in this case, and is reflected, ultimately, in the
commits.

While performing the effort computation, we only include the effort spent by
the developers in _code development_.

Using developer activity information as SDEE metrics:

    1. (a)

A significant part (more than 25%) of the total software development effort is
spent in source code creation (Selamat et al., 2014).

    2. (b)

Developers are the key contributors towards the source code creation and,
thus, software development.

    3. (c)

A developer’s contribution often depends on several factors, such as his
productivity, participation, or activity (Amor et al., 2006; Gousios et al.,
2008).

Thus, leveraging the developer activity-based SDEE metrics as the first-class
parameter in our SDEE model is expected to produce estimates with better
accuracy.

  4. (4)

Developer activity information, such as the SLOC added, deleted or, modified
by a developer, represents the most _basic_ unit of the software development
effort contributed by a developer in developing a software project.

  5. (5)

Developer activity information present in VCS repositories can help provide a
valuable measure of the effort spent developing it. Developer activity
information can be derived from the release and commit statistics of various
software projects tracked in the VCSs. Thus, to estimate the effort required
to develop newly-envisioned software, the effort estimates derived using such
information from the existing software projects with similar functionality can
be leveraged.

###  1.2. Contributions

  1. (1)

We present a novel method to estimate the effort required to develop software.
Given the software requirements of a newly-envisioned software, our system
estimates the effort required to develop it. To derive the effort estimates,
we use the developer activity information of the past software.

  2. (2)

We propose new SDEE metrics based on the developer activity information. For
instance, the the developer count (devCount) and time spent in developing a
software project.

  3. (3)

By extracting the developer activity information for various software present
on GitHub, we develop a dataset that can be leveraged to perform SDEE.

  4. (4)

By utilizing the product descriptions of software considered in our dataset,
we develop a software similarity detection model $M$ using PVA. For a given
description of software requirements, $M$ predicts the functionally similar
software matches existing on GitHub.

  5. (5)

With the combination of the SDEE dataset and the trained software similarity
detection model, we develop our SDEE tool for predicting the effort required
to develop a newly-envisioned software, with a standardized accuracy (SA)
value of 59.89% over the random-guessing method.

  6. (6)

We present a comparison of our approach with the existing SDEE methods, and
the results show that our model outperforms the ATLM (Whigham et al., 2015),
as it achieves a SA of 42.7% when compared with it.

Paper organization: Next, we discuss some of the related works and their
shortcomings in Section 2. Section 3 highlights the critical design decisions
and the implementation details of our system. Section 4 gives the details of
the experiments conducted to verify the effectiveness of our system. Section 5
lists the threats to the validity of our approach. Finally, Section 6
summarizes our work and suggests the future work.

##  2\. Background

Figure 2. Categorization of the existing effort estimation methods

###  2.1. Existing approaches for SDEE

Fig. 2 shows a broad categorization of the existing effort estimation
techniques summarized by reviewing the existing literature (Molokken and
Jorgensen, 2003). Based on our literature review, we classify the existing
SDEE techniques broadly into three major categories: _Expert-based SDEE_
methods, _Formal SDEE_ methods, and _Hybrid SDEE_ methods. A brief description
of these methods is as follows:

  1. (1)

_Expert-based SDEE methods:_ These are based on the judgments provided by the
experts, such as the project managers, using the knowledge of software
projects developed in the past (Pressman, 2005). The expert-based SDEE methods
are further sub-categorized as _Bottom-up SDEE methods_ and _Group SDEE
methods_. Bottom-up methods perform the SDEE by using the following procedure:
a) Dividing the complete development process into sub-tasks, b) Estimating the
effort required to perform these sub-tasks, c) Finally, summing these effort
values to obtain an overall effort estimate. The estimates for performing the
sub-tasks are obtained from the specific people in charge of developing them.
On the contrary, Group SDEE methods comprise consulting a group of experts to
derive the effort estimate for developing a software project (Wu, 1997).

  2. (2)

_Formal SDEE methods:_ These are based on the software development effort
estimates derived using a mathematical quantification step (Jørgensen et al.,
2009), such as a formula derived from historical data from past projects.
These can be further classified as _Analogy-based SDEE methods_ , _Parametric
SDEE methods_ , _ML-based SDEE methods_ , and _Size-based SDEE methods_.
Analogy-based SDEE methods follow identifying one or more existing software
projects similar to the project being developed and then deriving the effort
estimate by analyzing similar software projects. Similarly, Parametric methods
are based on identifying the variables influencing the effort required to
develop a software project. The general mathematical form of such models is
represented as:

(1) $E=A+B(e_{\nu})^{C}$

In the above equation, $E$ represents the effort in person-months, $e_{\nu}$
represents the estimation variable (e.g., source lines of code (SLOC) or
function points (Albrecht, 1979; Dumke and Abran, 2016)), and $A,B$, and $C$
represent the empirically derived constants the past software data. Some of
the prominently used Parametric SDEE methods include COCOMO (Barry et al.,
1981), Putnam’s model (Putnam, 1978), and SEER-SEM (Galorath, 2007).

Size-based SDEE methods form a subset of parametric methods, as these use
size-based metrics, such as SLOC and function-points, to compute the effort
estimates for developing a software project. Some other examples of such
metrics are use-case points (UCPs) (Winters and Schneider, 1900), story points
(Cohn, 2005), and object points (Minkiewicz, 1997). Recently, COSMIC
Functional Size measurements (COSMIC FSM) are reported to overcome the
shortcomings of LOC-based metrics, and outperform the existing FP-based
methods (Di Martino et al., 2016; Abualkishik et al., 2017). COSMIC FSM is
based on the Functional User Requirements (FURs) of the software that are
independent from any requirements or constraints about their implementation.
They have been used for code size estimation of mobile applications (D’Avanzo
et al., 2015), UML models (De Vito et al., 2020), Web applications (Di Martino
et al., 2016; Abrahão et al., 2018), etc.

  3. (3)

_Hybrid SDEE methods:_ These methods use a combination of two or more existing
effort estimation methods. Some of the prominent hybrid SDEE methods derived
from the literature (Briand and Wieczorek, 2002) are _Mathematical Hybrid SDEE
methods_ and _Judgemental Hybrid SDEE methods_. Mathematical Hybrid SDEE
methods perform the SDEE using the analogy-based methods and the bottom-up
SDEE methods, whereas Judgemental Hybrid SDEE methods perform SDEE using the
expert-based SDEE methods and the parametric SDEE methods.

###  2.2. SDEE methods based on developer activity information and ML
techniques

Some existing studies (Kocaguneli et al., 2011a; Menzies et al., 2017) present
the variation of analogy-based SDEE methods where they use the k-Nearest
Neighbor (kNN) method with the euclidean distances to detect the similar
software matches from existing SDEE datasets, such as the PROMISE repository
(Menzies et al., 2012). A baseline of SDEE methods was established by (Whigham
et al., 2015) when they introduced their Automatically Transformed Linear
Baseline model, which is defined as a linear regression model. The ML-based
methods, such as Artificial Neural Networks (NeuralNet), are also employed to
train SDEE models (Minku and Yao, 2013).

ATLM (Whigham et al., 2015) and ABE (Kocaguneli et al., 2011a) use features
such as the analyst’s experience, programmer’s capability, and application
experience when trained using the existing datasets, such as the PROMISE
repository. Similarly, the LOC “straw man” estimator (Menzies et al., 2017)
uses the LOC of the software projects. Both ABE and the LOC “straw man”
estimator use the weighted euclidean distance measure to determine the top-k
similar matches. The weighted euclidean distance between two feature values x
and y can be expressed as:

(2) $dist(x,y)=\sqrt{\sum_{i=1}^{n}w_{i}(x_{i}-y_{i})^{2}}$

where $x_{i},y_{i}$ are values normalized to [0,1], and $w_{i}$ is a feature-
weighting factor (defaults to $w_{i}=1$).

In contrast, ATLM has a linear regression form expressed as:

(3)
$y_{i}=\beta_{0}+\beta_{1}x_{1i}+\beta_{2}x_{2i}+..\beta_{n}x_{ni}+\epsilon_{i}$

where $y_{i}$ represents the quantitative response variables, the $x_{i}$
represents the explanatory variables, and the $\beta_{i}$ is determined using
a least-squares estimator (Kutner et al., 2005). Similarly, the study in
(Minku and Yao, 2013) is based on training Artificial Neural Networks
(NeuralNet) using various datasets to predict the effort estimates.

An empirical approach for analyzing the developer activity traces present in
various VCSs to categorize the software developers as {occasional
contributors, full-time developers} is presented in (Robles et al., 2014).
They present the analysis for data collected by
Bitergia444https://www.bitergia.com/ for OpenStack
Foundation555https://https://www.openstack.org/foundation/. CVSAnalY is
deployed to collect the analytics from the source code management repository.
Developers of the project are then mailed a survey to inquire about the time
devoted to developing the project. The obtained survey data is used to train a
binary classification model by labeling the dataset entries as {occasional
contributors, full-time developers}. The approach is claimed to propose a
realistic estimation with a smaller weight (of the total effort) assigned to
the more error-prone estimations.

Similarly, a Change Impact Size Estimation (CISE) approach for software
development is introduced (Asl and Kama, 2013). The approach is designed by
integrating the static and dynamic approaches mentioned in (Kama, 2011).
Change Impact Analysis (CIA) is performed by tracing the affected requirements
by change requests to the impacted software artifacts and codes before they
are implemented. The essential requirements types are an addition, a
modification, and a deletion. The CISE works by first finding the classes
affected by the changes (by performing a CIA) and then performing estimation
for change in size. The study claims CISE to be 93.7% accurate.

A comprehensive study of the developer activity patterns in Free/Libre/OSS
(FLOSS) projects is presented by clustering developers around different time-
slots and days of a week (Capiluppi and Izquierdo-Cortázar, 2013). The study
compares the developer activity patterns in a company-driven environment with
those in a community-driven environment. The study is conducted by analyzing
the developer activity patterns based on the number of commits, lines of code
added, deleted, modified as recorded on several days of a week, and different
hours. In our work, we utilize these counts, viz., the number of commits,
lines of code added, deleted, and modified, extracted from various software
present on GitHub to estimate the effort expended to develop the software.

###  2.3. Computing the developer coding effort estimates for OSS

(Asl and Kama, 2013) introduce a Change Impact Size Estimation (CISE) approach
for software development. The approach is designed by integrating the static
and dynamic approaches mentioned in (Kama, 2011). Change Impact Analysis (CIA)
is performed by tracing the affected requirements by change requests to the
impacted software artifacts and codes before they are implemented. The major
requirements types are an addition, a modification and a deletion. The CISE
works by first finding the classes affected by the changes (by performing a
CIA), and then performing estimation for change in size with an accuracy of
93.7%. Similarly, there are existing works on developing defect prediction
solutions by measuring code changes in terms of SLOCs additions, deletions, or
modifications of software releases (Bell et al., 2011; Hassan, 2009).

(Capiluppi and Izquierdo-Cortázar, 2013) study the developer activity patterns
in Free/Libre/OSS (FLOSS) projects by clustering developers around different
time-slots and days of a week. The developer activity patterns in a company-
driven environment are compared with those in a community-driven environment
based on the number of commits, lines of code added, deleted, and modified as
recorded on various days of a week, and at various hours of a day. Similarly,
(Robles et al., 2014) propose an empirical approach for analyzing the
developer activity traces present in various VCSs to categorize the software
developer as {occasional contributors, full-time developers}. They present the
analysis for data collected by Bitergia666https://www.bitergia.com/ for
OpenStack Foundation777https://https://www.openstack.org/foundation/. CVSAnalY
is deployed to collect the analytics from the source code management
repository. Developers of the project are then mailed a survey to inquire
about the time devoted to developing the project. The obtained survey data is
used to train a binary classification model by labelling the dataset entries
as {occasional contributors, full-time developers}. The approach is claimed to
propose realistic estimation with a smaller weight (of the total effort)
assigned to the more error-prone estimations.

(Di Nucci et al., 2017) stress that developer’s personality characteristics
and the manner in which they perform the code changes play a major role in
software development. They study the developer personality characteristics by
measuring “how focused the developers are while making code changes?” and “how
scattered these changes are?” to devise a novel bug prediction model. The
model is validated using 26 Apache software projects and comparing with four
state-of-the-art prediction models based on size metrics and process metrics.
Similarly, (Catolino et al., 2018) leverage the developer-related factors to
enhance the existing change-prediction models by inspecting “how developers
perform modifications?” and “how complex is the development process?” The idea
is to use the complexity of the development process to identify the change-
prone classes, and then focus the developer’s attention to such classes. The
authors perform an empirical analysis of three developer-based change
prediction models considering the 408 releases, 193,274 commits, and 657
developers’ information of 20 GitHub projects, and propose a novel hybrid
change prediction model which exploits the developer-, process-, and product-
metrics to detect the change-proneness of source code classes.

Similar to the existing works (Capiluppi and Izquierdo-Cortázar, 2013; Asl and
Kama, 2013; Robles et al., 2014; Kocaguneli et al., 2011b), we utilize the
developer activity information such as the count of developers and time (in
days) required to estimate the effort required to develop an existing GitHub
software repository. We store these SDEE estimates as a dataset, which is then
used by our SDEE tool to provide the effort estimates for newly-envisioned
software.

###  2.4. Developing a software description similarity detection model for
detecting similar software matches

There exists a considerable amount of literature highlighting the use of
various NL processing techniques to develop software description similarity
detection models for various recommendation tasks, such as performing
recommendation of API methods (Thung et al., 2013), third-party libraries
(Chen and Xing, 2016), and research papers (Beel et al., 2013). Two prominent
techniques used in the existing works are _Term-frequency Inverse Document
Frequency (TF-IDF)_ (Salton and Buckley, 1988) model and _Word2vec model_
(Mikolov et al., 2013a, b).

TF-IDF methods compute document similarity based on the term-frequencies for
the terms or words present in a document, while the Word2vec model works by
learning the word representations for the words present in an input document,
to predict the co-occurring words in the same sentence.

(Trstenjak et al., 2014) present the use of K-Nearest Neighbour (KNN)
algorithm with TF-IDF method for text-classification of documents belonging to
four different categories, viz., sports, politics, finance, and daily news.
The results of the experiments show that the classification algorithm is
sensitive to type-of-documents. The classification algorithm yields the
highest accuracy of 92% in the case of documents belonging to the _sports_
category, while the lowest as 65% in the case of the _daily news_ category.

We use the Paragraph vectors algorithm (PVA) (Le and Mikolov, 2014) to develop
our software description similarity detection model. The PVA is inspired by
the Word2vec method and is also termed as a vector, as it generates fixed-
length feature representations (or vectors) corresponding to the text
documents provided as input.

Limitations of the TF-IDF method:

Since the TF-IDF method is based on the bag-of-words approach, some of its
limitations are (Ramos et al., 2003):

  1. (1)

It is unable to capture the semantic information of the text on which it is
trained.

  2. (2)

TF-IDF method does not consider the ordering of different words in a text
document or the co-occurrences of words in different documents.

  3. (3)

TF-IDF might take much time to train on a large vocabulary.

Due to the limitations specified in point 1 and 2 above, TF-IDF is not able to
capture the documents that differ in words having the same meaning i.e.
synonyms. For instance, happy, content, and joyful will be treated as
different words by TF-IDF method. Further, the method would also label two
documents as different, if they convey the same idea using different
descriptions. For instance, _it was a busy day_ , and _I was occupied with a
lot of work today_ both convey the same meaning, but will be labeled as
different. Similarly, the TF-IDF method would label two project descriptions
as different, if they use different words to convey the same meaning. For
instance, consider the following software requirements:

  1. (1)

_Require a fast, accurate, and anti-obfuscation library detection tool_ , and

  2. (2)

_Require a light-weight, effective, and obfuscation-resilient tool._

Both the examples convey the same software requirements, but with different
use of words, and hence are labeled as different. Thus, these limitations make
TF-IDF an inappropriate measure for detecting similarity in our case, i.e.,
when dealing with software requirements and software project descriptions.

Figure 3. High-level comparison at approach-level

###  2.5. High-level differences of our approach from the existing methods

Fig. 3 presents a broad idea of how our approach differs from the existing
SDEE methods. The existing effort estimation methods, such as the Analogy-
based effort estimation (ABE) (Kocaguneli et al., 2011a) and LOC “straw man”
(Menzies et al., 2017), generally follow the branch (a) approach shown in the
figure, where, to estimate the effort for a newly-envisioned software $z$, the
effort values of software projects developed in the past and similar to $z$
are used. On the contrary, the regression-based methods, such as the
Automatically Transformed Linear Baseline Model (ATLM) (Whigham et al., 2015)
and Artificial Neural Network (NeuralNet) (Minku and Yao, 2013) directly learn
and estimate the effort value of $z$ using the effort values of the past
software projects, represented by branch (b) of the figure. The existing SDEE
methods mostly train the models on the metadata values of the software
projects developed in the past, present in the existing effort estimation
datasets, such as the PROMISE repository (Boetticher, 2007).

In our approach, we hypothesize that “more relevant similar-software matches
can be detected if we use the software project descriptions as the input for
detecting similar software projects.” Also, the developer activity information
has been claimed to be an accurate representation of the actual development
effort on a software project (Capiluppi and Izquierdo-Cortázar, 2013; Robles
et al., 2014). Therefore, we use the software project descriptions of GitHub
repositories to determine the similar software projects (to $z$) and use the
developer activity information of the similar software matches to compute the
effort estimate for $z$.

Table 1. Table of Notations $R$ | $\triangleq$ |  The set of OSS repositories present on GitHub considered by us.
---|---|---
$e^{r}$ | $\triangleq$ |  Effort required to develop $r\in R$, computed using equation 5
$D^{r}$ | $\triangleq$ |  The set of developers involded in developing $r\in R$.
$t^{r}$ | $\triangleq$ |  The total amount of time spent in developing $r\in R$.
$C$ | $\triangleq$ |  The set of different software categories specified on MavenCentral888https://mvnrepository.com/.
$H$ | $\triangleq$ |  The set of software project descriptions of $R$.
$P$ | $\triangleq$ |  The set of PVA input parameter variation scenarios for training the software similarity detection models.
$M$ | $\triangleq$ |  The set of software similarity detection models trained using PVA on $H$.
$Y$ | $\triangleq$ |  The test-bed developed to test the performance of software similarity detection models.
$\phi$ | $\triangleq$ |  The vector representation of a project description.
$\alpha$ | $\triangleq$ |  The cosine similarity score between two PVA vectors.
$\hat{\alpha}$ | $\triangleq$ |  The threshold of cosine similarity between two PVA vectors to categorize them as similar.
$\psi$ | $\triangleq$ |  Number of training samples used for training a software similarity detection model.
$\gamma$ | $\triangleq$ |  It refers to the dimensionality of the feature vectors used to develop the dataset, referred to as the PVA vector size.
$\beta$ | $\triangleq$ |  Number of training iterations or epochs used for training a software similarity detection model.

To the best of our knowledge, such use of software project descriptions and
the developer activity information has not been done in performing the SDEE
task. However, since we have chosen the OSS (GitHub) repositories to develop
our dataset, our SDEE model provides estimates for only the software developed
in an open-source environment. This is because the software development for
open-source software is generally performed by source code contributors who do
not have fixed working hours or work patterns, whereas the proprietary
software is developed by full-time working employees (Capiluppi and Izquierdo-
Cortázar, 2013) having fixed working patterns. Thus, the developer activity
might differ due to the different working patterns.

Figure 4. Details of the proposed system - a system-context diagram

##  3\. Proposed system

The main objective of our approach is:

_Given the requirements description of a software project, estimate the effort
required to develop it._

To achieve this objective, we develop three major software artifacts (shown in
Fig. 4):

  1. (1)

A novel _SDEE dataset_ based on the developer activity information of software
present on GitHub,

  2. (2)

A _software similarity detection model_ (developed using PVA) trained on the
project description documents of software considered to develop the above
dataset,

  3. (3)

An _SDEE tool_ that utilizes the SDEE dataset and the software similarity
detection model to provide effort estimates.

Table 1 shows the notation used for various terms in this paper.

###  3.1. Developer activity-based SDEE metrics

An OSS development broadly comprises developers contributing source code in an
OSS repository. To measure the software development activity quantitatively,
we define novel SDEE metrics for an OSS repository $r$, as follows:

  1. (1)

$|D^{r}|$: The total number of developers involved in developing $r$. It is
measured in _persons_.

  2. (2)

$T^{r}$: The total amount of time spent in developing $r$, mathematically
represented as:

(4) $t^{r}\mbox{=}\ \sum_{i=1}^{j}(t_{s}^{i}-t_{e}^{i})/j$

where $t_{s}^{i}$ represents the start-time of developing the ith release of
repository r, and $t_{e}^{i}$ represents the end-time of the ith release. It
is expressed in days, months, or years.

  3. (3)

$e^{r}$: The effort expended to develop $r$, mathematically represented as:

(5) $e^{r}\mbox{=}\ |D^{r}|*t^{r}$

It is measured in person-months, or person-years, or person-days, depending
upon the units of $|D^{r}|$ and $t^{r}$.

###  3.2. Steps of our approach

The pivotal steps in our approach with the relevant design decisions addressed
at each step are listed below:

  1. (1)

Processing the GitHub software repositories and developing the SDEE dataset

    1. (a)

Download the software categorization listed on MavenCentral.

_Design Decision: Why choose the software categorization from MavenCentral?_

    2. (b)

Download a collection of GitHub software repositories belonging to some of the
significant software categories. We used the ones specified by MavenCentral.
Also, extract the developer activity information and download the software
project descriptions of the respective GitHub repositories.

    3. (c)

Compute the effort estimates for developing the considered software
repositories by utilizing their developer activity information. The critical
design decisions considered in this step were:

      1. (i)

_How do we ensure our dataset’s homogeneity in terms of the breadth of
software types or domains?_

      2. (ii)

_Why is only GitHub chosen to download software repositories? How do we select
a software repository?_

      3. (iii)

_What developer activity information should we collect from the selected
repositories? How can we determine whether the collected information helps
estimate the software development effort?_

      4. (iv)

_How do we utilize the developer activity information to compute the software
development effort estimates?_

  2. (2)

Developing the software similarity detection model and obtaining the PVA
vectors

    1. (a)

Train a software similarity detection model using the PVA on the software
project descriptions downloaded in Step 1b. The critical design decisions
considered in this step were:

      1. (i)

_Why use PVA and how to choose its tuning parameters?_

      2. (ii)

_Why do we use vector representations of software product descriptions?_

    2. (b)

Obtain the PVA vectors for each of the product descriptions downloaded in Step
1b.

_Design Decision: What is the minimum threshold value of $\alpha$ between two
PVA vectors that indicate a significant similarity between the corresponding
software project description samples?_

  3. (3)

Developing the SDEE tool to perform the SDEE for a newly-envisioned software
$z$

    1. (a)

Design a suitable GUI for the interaction with the end-user. The salient
design decisions considered at this step were:

      1. (i)

_What is the input format for specifying the requirements description of
software for which the development effort is predicted?_

      2. (ii)

_What type of details are required to be specified in the requirements
description provided as input to the tool?_

    2. (b)

Compute the effort estimate for developing $z$ by leveraging the SDEE dataset
and the PVA vectors.

      1. (i)

Obtain a PVA vector $\phi^{\prime}$ for the requirements description submitted
by the end-user.

      2. (ii)

Fetch all the PVA vectors $\phi\in V$, such that the similarity $\alpha$
between $\phi$ and $\phi^{\prime}$ is above $\hat{\alpha}$.

      3. (iii)

Derive an effort estimate for $z$ by utilizing the similar software matches’
effort values fetched in Step 3(b)ii.

_Design Decision: How do we compute the final effort estimate by utilizing the
similar software matches’ effort values?_

In the following sections, we describe developing our software artifacts and
the rationale for design decisions addressed in this process.

###  3.3. Design considerations in our approach

In this section, we discuss the rationale behind design decisions taken by us
while implementing our approach.

  1. (1)

Obtain the software categorization: To ensure our dataset’s homogeneity in
terms of the software types, it is necessary first to identify what types of
software constitute the major portion of professional software. MavenCentral
is one such source of software categorization taxonomy. Thus, we first fetch
the software categorization listed by Maven (MVN)
Repository999https://mvnrepository.com/, then select the software projects
under each of those categories to develop the dataset. The MVN repository
portal contains around 150 software category web pages, with each category-
specific web page containing the description and web links of the categorized
software. These categories are utilized as topics to fetch the GitHub links of
software using the GitHub REST API101010https://developer.github.com/v3/.

  2. (2)

The rationale for choosing GitHub: Though several OSS hosts such as
SourceForge.net111111https://sourceforge.net/, GitHub, GitLab, and
BitBucket121212https://bitbucket.org/dashboard/overview, only a few offer
support for extracting the developer activity information. We selected GitHub
over the other OSS hosts for the following reasons:

    1. (a)

SourceForge.net does not provide the details of source code development for
the software repositories hosted by it. Hence, no developer activity
information statistics were available.

    2. (b)

GitLab and BitBucket, the newer ones, have very few OSS repositories that
match our selection criteria. For instance, GitLab shows only 12 OSS
repositories with $>$500
stars131313https://gitlab.com/explore/projects/starred.

  3. (3)

Repository selection rationale: We select the top 100 repositories provided by
the GitHub RESTful API for each input category. The additional constraints
(Coelho et al., 2020; Han et al., 2019) applied to filter the OSS repositories
under each of the selected software categories (selected in Step 1 above) are:

    1. (a)

_Size Constraint:_ We select the repositories with a size of more than $5$ MB.
The 5 MBs threshold was taken since we observed that the most non-trivial OSS
repositories were larger than 5 MBs in their source code content. This means
that the repositories with a size less than 5 MBs had minimal SLOC count.

    2. (b)

_Date Constraint:_ To ensure a repository is not very old or inactive, we add
a date constraint to our search. We select the repositories which have been
updated at least once in the last three years.

    3. (c)

_Stars Constraint:_ We applied this constraint to ensure that the repositories
selected are well-liked to some extent and hence being used by a certain
number of developers. We selected the repositories with more than $500$ stars.
We sorted our search results based on the stars earned by them and selected
the top $100$ results under each category.

    4. (d)

_Keyword constraint:_ To filter the software repositories belonging to a
specific category, we perform a GitHub REST API query search with the category
names as the input keywords.

  4. (4)

Utilizing developer activity information to compute effort estimates: The
usefulness of developer activity information (Definition 1.3) in performing
the SDEE tasks has been established by (Robles et al., 2014; Capiluppi and
Izquierdo-Cortázar, 2013; Gousios et al., 2008; Amor et al., 2006; Rahman and
Devanbu, 2013; Herbsleb, 2014). To obtain the effort estimates for the
considered software repositories, we define the SDEE metrics based on the
repository’s developer activity information. We obtain the software
repositories’ effort values by inserting the SDEE metrics’ values in equation
5. Here, we consider the developer activity performed within the time-period
$\langle$ start-date of release, date of the last commit performed on the
release$\rangle$ to compute the release-wise effort estimates and store them
in the table _release_effort_estimate_. These release-wise estimates are then
averaged to estimate the effort values of the software project, which are
later utilized while performing the SDEE.

  5. (5)

The rationale for choosing PVA, its tuning parameters, and the similarity
threshold $\hat{\alpha}$: The following are the main reasons for PVA’s choice:
i) It allows us to compute the same length vectors for representing the
software project descriptions. Keeping the length of such vectors the same for
every source code sample is critical for implementing an efficient and fast
system. ii) PVA outperforms the existing text representation methods and thus
serves as an effective method of detecting text similarity (Le and Mikolov,
2014). Thus, we chose the PVA to develop our _software similarity detection
model_ and compute the vector representations of our approach’s software
product descriptions. The complete details of the parameter tuning of PVA are
discussed in detail in Appendix-A.1.

  6. (6)

Computing PVA vectors for software product descriptions: To expedite the
software similarity detection process, we perform the following steps:

    1. (a)

Obtain the PVA vector representations of the software product descriptions
using a suitable software similarity detection model (developed using
Algorithm 2) and store them in the database.

    2. (b)

Compute the cosine similarity ($cos\\_sim$) of the PVA vectors for a reference
vector of the same dimensions and store it in the database. The reference
vector is randomly chosen and kept fixed for the dataset.

(a) Details submission page

(b) SDEE estimates for the input software details

Figure 5. SDEE tool interface

  7. (7)

Structure of inputs provided to the tool: Fig. 5 shows the input and output
pages of our tool’s GUI. The software similarity detection model is likely to
detect similar software with better accuracy if provided the input using the
familiar format and structure as the training data. For instance, the software
descriptions provided as input should describe the software’s functionality to
be developed, the features required to be developed in it, and a detailed
description of all such functional requirements and software constraints.
Therefore, the format of the input expected by our tool is such that it
captures the software’s key functionalities and characteristics whose effort
is to be estimated. For the newly-envisioned software, the end-user needs to
enter the following inputs into the tool’s GUI (shown in 5(a)):

    1. (a)

_Software product title:_ A suitable title for the software product.

    2. (b)

_Software Description:_ A brief description of the software product comprising
its intended use, key functionalities, and characteristics.

    3. (c)

_Preferred programming language for software development:_ The programming
language(s) used for software development. In the case of multiple programming
languages, the inputs can be entered in a comma-supported format.

    4. (d)

_Type of software:_ The end-user must select a plausible software category and
a sub-category from the respective drop-down lists. We extract 11 _abstract_
software categories from 150 software categories specified by MavenCentral,
viz., _i) Software library, ii) Software utilities & plugin, iii) Software
tool, iv) Software metrics, v) Software driving engine, vi) A software
framework, vii) Software middleware, viii) Software client, ix) Software
server, x) Software driver, and xi) Software file system._

    5. (e)

_Supported Operating system:_ In this input, the user specifies the operating
system(s) the software product must support.

    6. (f)

_Product features and feature description fields:_ The end-user specifies the
major software features and their description using these input fields.

The rationale for deriving the abstract software categories: To determine
these abstract categories, we club the category names under their generic
category types. For instance, SSH libraries, DNS libraries, JWT libraries can
all be clubbed under the _Software library_ category. On selecting one of the
software categories, the end-user is displayed the specific sub-categories
fetched from MavenCentral.

Discussion on the similar software projects filtered by PVA: As shown in Fig.
5(b), for the given description of a compression library which is efficient
and quick (in Fig. 5(a)), we get the top matching result as “Google Draco”
library141414https://github.com/google/draco. The next close match as
projected by our tool is of “Centaurean Density”
library151515https://github.com/k0dai/density. Both the libraries are intended
to perform high-speed compression in an efficient manner as described in their
GitHub project descriptions. This is because the tool identifies the best
matching software by matching the input requirement description (shown in Fig
5(a)) with the GitHub software project descriptions present in our database.

Figure 6. Detecting similar matches to a functionality

  8. (8)

Computing the effort estimate for developing a newly-envisioned software: Most
of the GitHub hosted software repositories are used as software modules in
developing more complex software systems. For instance, GitHub hosts a
variety161616For example, see this collection of projects
https://github.com/collections of software modules, such as third-party
libraries, application development frameworks, and domain-specific
applications. Since our dataset contains the effort values of such software,
which act as modules in a more extensive software system, our tool can perform
the SDEE for such software modules. Fig. 6 shows a decision tree for detecting
existing software that matches the input. The detailed procedure for
estimating the software development effort of a newly-envisioned software $z$
is provided in Algorithm 1. To obtain the effort estimate for $z$, the effort
values of the top-k similar software matches are combined using the
Walkerden’s triangle function (Walkerden and Jeffery, 1999), defined as
follows for k=3:

If the effort estimates of the first, second, and third closest neighbors are
a, b, and c, respectively, then the effort required to develop $z$ is
expressed as:

(6) $Effort(z)\mbox{=}\ (3a+2b+1c)/6$

We use Walkerden’s triangle function to obtain the effort estimates as it
provides a weighted average result with the weights assigned based on the
closeness of the top-k neighbors and results in smaller residual values than
the non-weighted average method, i.e. when all top-k neighbors are assigned
equal weights. We achieved the best performance results for our SDEE method
for k=2 (shown in Section 4) and used it to compute the estimates.

Algorithm 1 Procedure to perform SDEE for a newly-envisioned software $z$

0: $\hat{M}:=$ The best-performing software similarity detection model
developed in Algorithm 2.$SDEE\\_DB:=$ Dataset containing the effort estimates
computed using equation 5 and the PVA vectors.$\hat{\alpha}:=$ The threshold
cosine similarity score for PVA vectors.$p_{z}:=$ Project description of $z$.

0: $e_{z}:=$ Effort estimate for the $z$.

1: $\phi_{d}:=InferVector(p_{z},\hat{M})$

2: $\phi_{ref}:=FetchReferenceVector(SDEE\\_{DB})$

3: $\tau_{z}:=ComputeCosineSimilarity(\phi_{z},\phi_{ref})$

4: $K_{z}:=FetchTopKSimilarProjects(SDEE\\_{DB},\phi_{z},\hat{\alpha})$

5: $E:=FetchEffortValues(K_{z},SDEE\\_DB)$

6: $e_{z}:=$ Return the averaged effort value using the Walkerden’s triangle
function equation 6 and the effort values $e\in E$.

###  3.4. Implementation details

We developed our software artifacts using the Python programming language due
to the available expertise. The two significant artifacts driving our approach
are:

Figure 7. Relational schema of the SDEE dataset

  1. (1)

SDEE dataset: Fig. 7 shows the relational schema of our SDEE dataset. The
dataset mainly comprises the following:

    1. (a)

Information about the releases and commits in various VCS repositories.

    2. (b)

Software development effort estimates computed by inserting the developer
activity-based SDEE metrics values in equation 5.

    3. (c)

Metadata information, such as the developers involved in developing various
software repositories.

    4. (d)

PVA vector representations of the _software product descriptions_ extracted
from VCS repositories.

The dataset contains the above-listed information extracted from $\approx$
13,000 GitHub repositories belonging to 150 different categories of software.
The SDEE metrics values’ extracted from various VCS repositories and the
associated software estimates are retained as effort estimates in the SDEE
dataset. The dataset is used by the SDEE tool to compute the effort estimates
for developing the newly-envisioned software. It is easy to add data from
additional sources. The details of the relational tables are described in
Appendix A.2.

Our SDEE dataset and tool are available at https://doi.org/10.21227/d6qp-2n13
and https://doi.org/10.5281/zenodo.5095723, respectively.

Figure 8. Estimating the effort for developing a newly-envisioned software

  2. (2)

SDEE tool: Fig. 8 highlights the major components of the tool. For a given
newly-envisioned software $z$, our tool performs the following steps:

    1. (a)

Using the _vectorizer_ module and the best-performing software similarity
detection model $\hat{M}$, it converts the software project description inputs
into a PVA vector $v$.

    2. (b)

Using $v$ as the input to the _similar-records fetcher_ , it fetches the top-k
most similar software matches ($K$) to $v$ from the SDEE dataset. We use the
cosine similarity171717http://bit.ly/2ODWoEy to measure the similarity between
the PVA vectors.

    3. (c)

To compute the effort estimate for developing $z$, it combines the effort
estimates of $K$ by using the Walkerden’s triangle function represented in
equation 6.

##  4\. Performance Evaluation and Comparison

Table 2. Developer activity information used in developing the SDEE dataset

Attribute name | Description
---|---
_ Developer count (devCount or $\mid D^{r}\mid$) _ |  The total number of developers working on an OSS repository $r$. It is measured in _persons_.
_$SLOC_{m}^{r}$_ |  The total number of SLOC modifications made in developing the OSS repository $r$. It is measured in _lines_ , with the whitelines excluded.
_ Development time $t^{r}$ _ |  The total amount of time spent in developing the OSS repository $r$. It is measured in _months_.
_ Development effort $e^{r}$ _ |  The total amount of effort spent in developing the OSS repository $r$. It is measured in _person-months_ and computed using equation 5.

A vital aspect of an SDEE method is detecting similar software matches to a
newly-envisioned software whose effort is estimated. Most of the existing SDEE
methods, such as ATLM(Whigham et al., 2015), ABE (Kocaguneli et al., 2011a),
LOC “straw man” (Menzies et al., 2017), Artificial Neural Networks (NeuralNet)
(Minku and Yao, 2013), use different methods to detect similar software
matches using the metadata of past software projects available as existing
datasets, such as PROMISE repository. In contrast, we use the software project
descriptions to develop our _software similarity detection model_ to perform
this task. One of our key hypotheses is to validate that our _software
similarity detection model_ trained using software product descriptions helps
detect more relevant software matches than the models trained using project
metadata.

We perform several experiments using the _randomised trials method_ (discussed
in Section 4.3) and the _k-fold cross-validation_ method (discussed in Section
4.4) to validate our hypothesis by comparing the performance of our method
with the above methods. We choose these methods for comparison as these are
the most related works to ours and have been used as baselines in the last ten
years by several prominent studies. The experiments’ details to obtain an
optimally tuned _software similarity detection model_ are provided in
Appendix-A.1. Also, as ABE and LOC “straw man” use the k-nearest neighbors
(kNN) method for detecting similar software matches, we compared the
performance of these methods by experimenting with k=1, 2, 3, 4, and 5 for kNN
in both the randomized trials and the cross-validation experiments. We used
the scikit-learn python library (Pedregosa et al., 2011) to implementing ATLM,
NeuralNet, and kNN. To validate the significance of our results, we performed
various statistical significance tests, such as t-test with bootstrapping and
cliff’s $\delta$ test (Cliff, 1993; Semenick, 1990) (discussed in Section 4.5
and Section 4.6). To validate the correlation between the SDEE metrics (used
to compute effort) and the software development effort, we test using
Pearson’s correlation coefficient (Benesty et al., 2009) (discussed in Section
4.7).

Since both the randomized and cross-validation are examples of internal
validation, and it has been considered essential to perform external
validation before adopting the prediction models in practice (Bleeker et al.,
2003), we perform Experiment # 6 (discussed in Section 4.8). To perform these
experiments, we invited professional programmers and asked them to evaluate
our tool’s behavior on various quality parameters, such as the response time,
accuracy, and the ease-of-use of our tool. The programmers were asked to use
the existing software project estimates available as existing datasets such as
COCOMO81181818https://bit.ly/3eMeFcN or by fetching the data from VCSs such as
GitHub. For performing this experiment, we provided access to our tool by
sharing it at https://doi.org/10.5281/zenodo.5095723. Table 3 summarizes the
significant findings of our experiments.

Table 3. SDEE: Experiments summary

Experiment | Objectives | Major findings
---|---|---
Experiment #1 (Section 4.3) |  Performance Comparison with the existing SDEE methods using the randomized training and test sets |  DevSDEE achieves the highest Standard Accuracy (SA)- of 59.89% over the random guessing method, 42.7% with- ATLM, and 84.22% with the LOC straw man estimator.
Experiment #2 (Section 4.4) |  Performance Comparison with the existing SDEE methods using the k-fold cross-validation method |  DevSDEE achieves the highest SA of 57.05% over- the random guessing method, 35.13% with ATLM, and 87.26% with the LOC straw man estimator.
Experiment #3 (Section 4.5) |  To validate the negligible difference between the DevSDEE estimates and the true effort values. |  t-significant test, Cliff’s delta, and the parametric- effect tests show the minimal differences in the DevSDEE estimates and the true effort values.
Experiment #4 (Section 4.6) |  To validate that DevSDEE provides better effort estimates than the existing methods using t-significance and effect tests |  Both t-tests and effect tests validate that DevSDEE results in more accurate (less error)- effort estimates than the existing methods.
Experiment #5 (Section 4.7) |  Effect of SDEE metrics on the overall development effort values |  Correlation value of +0.799 for developer- count and +0.644 for development time compared with the software effort values.
Experiment #6 (Section 4.8) |  Performance evaluation from a programmer’s perspective on various quality parameters, such as ease of use, response time, and accuracy. |  SDEE achieves ratings $>8/10$ for most of the considered quality parameters.

###  4.1. Test-bed Setup

To perform the comparison, we extracted the developer activity information
from $19,096$ releases of $1,587$ GitHub OSS repositories and created a
dataset of projects’ metadata attributes. We also extracted the respective
project descriptions. Thus, the input source for both our method and the
existing methods is the same, though the respective inputs are different,
viz., the metadata information and the project descriptions. After removing
the records for repositories having time (in months) or developer-count $<$ 1,
our dataset reduces to $1,184$ GitHub repositories’ records. Table 2 provides
the details of the project attributes used to develop the dataset. We want to
emphasize that we use the same dataset to compute the effort values across
different SDEE techniques. Therefore, the input remains the same for all the
methods.

###  4.2. Evaluation metrics

Most of the existing methods (Kocaguneli et al., 2011a; Minku and Yao, 2013;
Menzies et al., 2017; Whigham et al., 2015), with which we present our
comparison, use the evaluation metrics such as the magnitude of the relative
error (MRE), the mean magnitude of the relative error (MMRE), the median
magnitude of the relative error (MdMRE), and prediction level (PRED) for
performance evaluation (Conte et al., 1986). However, we also include some
additional experimental evaluations based on newer software metrics, such as
mean absolute error (MAR) (Langdon et al., 2016), logarithmic standard
deviation (LSD), RE*, standard accuracy (SA), and effect size (Shepperd and
MacDonell, 2012), as these are recommended by others (Kitchenham et al., 2002;
Miller, 2000).

For the evaluation metrics defined below, let $i$ be a software project, $e$
be the actual effort value to develop $i$, and $e^{\prime}$ be the estimated
effort value obtained by an SDEE model $M$. Also, let $n$ be the total number
of considered software projects.

  1. (1)

_Magnitude of Relative Error, MRE:_ It measures the relative size of the
difference between the actual and estimated effort value:

(7) $MRE=\frac{(\mid e^{\prime}-e\mid)}{e}$

The lower the MRE values for a model, the lesser is the error in its
estimates, and hence, better is its performance.

  2. (2)

_Mean MRE (MMRE):_ : It measures the percentage of the MREs averaged over the
n projects present in the test-bed:

(8) $MMRE=\frac{100}{n}\sum_{i=1}^{n}{MRE}_{i}$

The lower the MMRE value for a model, the smaller is the averaged error in its
estimates, and hence, better is its performance.

  3. (3)

_Median MRE (MdMRE):_ :: It measures the median of MRE values obtained by the
model using the software projects in the test-bed:

(9) $MdMRE=Median(\forall_{i=1}^{n}{MRE}_{i})$

The lower the MMRE value for a model, the smaller is the averaged error in its
estimates, and hence, better is its performance.

  4. (4)

_Mean absolute residual, MAR:_ It measures the mean of the absolute value of
the residuals, i.e., the error between the predicted and the true value of an
estimate, and is mathematically represented as:

(10) $MAR=\frac{\sum_{i=1}^{n}(\mid e^{\prime}_{i}-e_{i}\mid)}{n}$

The lower the MAR values for a model, the lesser is the error in its
estimates, and hence, better is its performance.

  5. (5)

_Logarithmic standard deviation, LSD:_ It measures standard deviation of the
log transformed prediction values, and is mathematically represented as:

(11) $LSD=\sqrt{\sum_{i=1}^{n}(ln\ e^{\prime}_{i}-ln\ \overline{e_{i}})^{2}}$

The lesser the LSD values, the lesser is the error, and the better is the
performance.

  6. (6)

_RE*:_ It is used as a baseline error measurement metric, and is
mathematically represented as:

(12) $RE*=\frac{var(residuals)}{var(measured)}$

where var represents the variance, residuals represents the error value, i.e.,
$e^{\prime}-e$ , and the measured values represent the actual effort values,
i.e., $e$ values. The lesser the $RE^{*}$ values for a model, the better is
the performance.

  7. (7)

_Standard Accuracy, SA:_ It provides a relative assessment of the performance
of the model by comparing it with the random guessing method, and is
mathematically represented as:

(13) $SA_{P}=(1-\frac{MAR_{P}}{\overline{MAR_{RG}}})*100$

where P represents the SDEE method whose performance is being evaluated, RG
represents the random guessing method (Shepperd and MacDonell, 2012). Over
large number of runs, $MAR_{RG}$ converges to sample mean. SA represents how
much better an effort estimation technique is than random guessing. The larger
the value of SA, the better is the model performance.

  8. (8)

_Effect size:_ It provides the magnitude of the relationship between treatment
variables and outcome variables and is computed based on the sample data to
make inferences about a population (analogously to the concept of hypothesis
testing) (Kampenes et al., 2007). It is further categorized into parametric
and non-parametric based on the nature of the data. The parametric method
assumes the data to have a normal distribution, whereas the non-parametric
method is free from any such assumption. The parametric versions of effect
size include cohen’s $d$, hedges’ $g$, and glass $\Delta$ (Cohen, 2013; Hedges
and Olkin, 2014; Hedges, 1981), while the non-parametric versions include
cliff’s $\delta$ effect size and bootstrap method (Cliff, 1993). The
mathematical expressions of these variations are listed below:

(14) $Cohen^{\prime}s\
d=(\overline{X_{1}}-\overline{X_{2}}).\sqrt{\frac{(n_{1}-1).var_{1}+(n_{2}-1).var_{2}}{n_{1}+n_{2}-2}}=t.\sqrt{\frac{1}{n_{1}}+\frac{1}{n_{2}}}$

where $n_{1},n_{2}$ represent the sample size of the two groups in
consideration ($n$ in the present case), $\overline{X_{1}},\overline{X_{2}}$
represent the mean of the considered groups ($\overline{e^{\prime}}$ and
$\overline{e^{\prime}}$ in the present case), and $var_{1},var_{2}$ represent
the variance of the values in the respective groups. The equivalent expression
with t (from the t-test) was given by Rosenthal (Rosenthal et al., 1994). It
was observed that for a small sample size ($<20$), cohen’s effect size values
are biased on the sample means of the population. To rectify this limitation,
Hedges proposed a correction expressed as:

(15) $Hedge^{\prime}s\ g=d.({1-\frac{3}{4.(n_{1}+n_{2})-9}})$

where d is the cohen’s effect size. Both d and g are considered as small
effect sizes if the values are $<=0.2$, medium effect sizes if the values are
$<=0.5$, and large effect sizes if values are $>=0.8$.

Similarly, Glass’s $\Delta$ is recommended for use when the standard
deviations of the considered groups differ substantially:

(16) $Glass^{\prime}s\
\Delta=\frac{\overline{X_{1}}-\overline{X_{2}}}{s_{control}}$

where $s_{control}$ represents the standard deviation of the control group,
and

(17) $Cliff^{\prime}s\
\delta=\frac{\\#(X_{1}>X_{2})-\\#(X_{2}>X_{1})}{n_{1}.n_{2}}$

where $\\#$ represents the count or cardinality. For instance,
$\\#(X_{1}>X_{2})$ implies the count of data values in group 1, which are
numerically larger (in value) than that of the corresponding data values in
group 2. The $\delta$ value lies in the range [-1,1], with the effect size of
-1 or 1 ,implying an absence of overlap of data values, whereas an effect size
of 0 representing a complete overlap (Macbeth et al., 2011).

###  4.3. Experiment #1: Randomized trials

Objective: To evaluate the performance of SDEE models and compare them with
the existing methods using the randomized trial method.

Procedure: When evaluating models’ performance using randomized trials with a
dataset containing n rows (or feature vectors), x rows are selected without
replacement for testing and the rest of the (n-x) rows for training. This
process is repeated for r number of trials, with the rows being shuffled in
each trial. x=10 is a suitable value (Menzies et al., 2006) and has been used
in the literature (Minku and Yao, 2013; Kocaguneli et al., 2011a). Some of the
existing studies take r = 30 (Minku and Yao, 2013), whereas some validate
their models using r = 20 (Kocaguneli et al., 2011a). With a dataset of 1184
rows, we selected x = 55 and r = 20 to have a considerable sample size for
each iteration. Thus, 55 rows were chosen in each iteration without
replacement for testing, and the remaining 1129 (=1184-55) rows for training.

Table 4. Performance comparison based on the Randomized trials

SDEE method | Evaluation metrics (mean $\pm$ std.dev) |
---|---|---
$LSD$ | $RE^{*}$ | $MAR$ | $MMRE$ | $MdMRE$ | $SA$
ATLM | 2.08 $\pm$ 0.21 | 0.46 $\pm$ 0.33 | 34.82 $\pm$ 6.31 | 192.40 $\pm$ 55.69 | 40.07 $\pm$ 17.05 | 53.57 $\pm$ 8.54
NeuralNet | 1.40 $\pm$ 0.62 | 4.31 $\pm$ 12.17 | 36.04 $\pm$ 26.96 | 222.78 $\pm$ 285.64 | 59.07 $\pm$ 16.65 | 9.51 $\pm$ 79.47
ABE | neigh =1 | 1.45 $\pm$ 0.63 | 1.69 $\pm$ 3.90 | 37.51 $\pm$ 47.21 | 557.74 $\pm$ 1968.9 | 557.74 $\pm$ 1968.9 | 9.69 $\pm$ 103.68
neigh =2 | 1.08 $\pm$ 0.43 | 1.24 $\pm$ 2.57 | 37.19 $\pm$ 41.16 | 632.46 $\pm$ 1615.35 | 632.46 $\pm$ 1615.35 | 9.71 $\pm$ 90.20
neigh =3 | 0.83 $\pm$ 0.46 | 1.21 $\pm$ 2.82 | 32.89 $\pm$ 42.28 | 596.31 $\pm$ 1429.21 | 596.31 $\pm$ 1429.21 | 20.41 $\pm$ 89.89
| neigh =4 | 0.83 $\pm$ 0.47 | 1.29 $\pm$ 3.09 | 35.20 $\pm$ 43.31 | 588.48 $\pm$ 1332.76 | 588.48 $\pm$ 1332.76 | 12.44 $\pm$ 90.94
| neigh =5 | 0.74 $\pm$ 0.41 | 1.40 $\pm$ 3.13 | 35.15 $\pm$ 43.01 | 582.21 $\pm$ 1218.80 | 582.21 $\pm$ 1218.80 | 14.27 $\pm$ 88.21
LOC straw man | neigh =1 | 1.38 $\pm$ 0.71 | 3.33 $\pm$ 7.36 | 76.37 $\pm$ 105.65 | 899.37 $\pm$ 1983.16 | 899.37 $\pm$ 1983.16 | -90.39 $\pm$ 278.33
neigh =2 | 1.13 $\pm$ 0.47 | 5.50 $\pm$ 9.96 | 65.58 $\pm$ 75.53 | 941.74 $\pm$ 1628.10 | 941.74 $\pm$ 1628.10 | -62.59 $\pm$ 179.23
neigh =3 | 0.93 $\pm$ 0.48 | 3.15 $\pm$ 5.98 | 58.50 $\pm$ 65.56 | 882.80 $\pm$ 1491.72 | 882.80 $\pm$ 1491.72 | -44.7 $\pm$ 148.32
| neigh =4 | 0.82 $\pm$ 0.45 | 3.28 $\pm$ 5.98 | 54.28 $\pm$ 60.62 | 850.19 $\pm$ 1434.05 | 850.19 $\pm$ 1434.05 | -34.4 $\pm$ 132.68
| neigh =5 | 0.81 $\pm$ 0.44 | 2.70 $\pm$ 5.47 | 53.64 $\pm$ 60.94 | 820.76 $\pm$ 1387.48 | 820.76 $\pm$ 1387.48 | -32.83 $\pm$ 133.53
DevSDEE | neigh =1 | 0.95 $\pm$ 0.19 | 0.52 $\pm$ 0.97 | 9.71 $\pm$ 8.69 | 130.61 $\pm$ 25.91 | 119.77 $\pm$ 42.0 | 57.54 $\pm$ 10.52
neigh =2 | 0.14 $\pm$ 0.34 | 0.34 $\pm$ 0.41 | 9.82 $\pm$ 7.92 | 166.01 $\pm$ 40.78 | 153.40 $\pm$ 53.81 | 59.89 $\pm$ 6.88
neigh =3 | 0.18 $\pm$ 0.19 | 0.41 $\pm$ 0.28 | 16.01 $\pm$ 4.24 | 540.24 $\pm$ 238.88 | 524 .19 $\pm$ 264.18 | 47.79 $\pm$ 7.75
| neigh =4 | 0.25 $\pm$ 0.10 | 0.47 $\pm$ 0.20 | 16.62 $\pm$ 3.65 | 662.20 $\pm$ 294.57 | 636.18 $\pm$ 335.09 | 39.97 $\pm$ 9.51
| neigh =5 | 0.29 $\pm$ 0.06 | 0.57 $\pm$ 0.13 | 17.7 $\pm$ 2.98 | 768.11 $\pm$ 334.27 | 729.34 $\pm$ 393.88 | 33.76 $\pm$ 11.95

Observations: Table 4 shows the results obtained from these experiments. The
averaged evaluation metrics values and their standard deviations (std. dev.)
for 20 iterations are listed. The grey-colored cells in the table represent
the SDEE models performing better than the rest. As observed from the table
values, DevSDEE outperforms the existing methods for the considered evaluation
metrics. We obtain the best performing DevSDEE values at neigh = 2 with the
highest SA of 59.89% (error=$\pm$6.88). However, the $SA$, $MAR$, $MMRE$, and
$MdMRE$ values have large standard deviation (std. dev. or error) values and
thus are not very reliable. Nevertheless, the $LSD$ and $RE^{*}$ values
present a clear picture of DevSDEE performing better than the compared SDEE
methods.

Critical Analysis: As it is evident from the results, the $LSD$ and $RE^{*}$
values of ABE, and LOC Straw man estimator decrease (or improve) with the
increase in the neighbor (neigh) count, whereas DevSDEE gives the best results
for neigh = 2 after which the values increase. The difference in DevSDEE’s
behavior from the existing methods can be explained due to the difference in
detecting similar software matches. To estimate the similar software matches
for a newly-envisioned software $z$, DevSDEE detects the similarity in the
software project descriptions of existing software and $z$. It is improbable
to have multiple software with very similar software project descriptions.
Hence, it is evident that only the effort values of few most-similar matches
would be close to the effort value of $z$, and would contribute towards the
SDEE process. However, for the existing methods working with project metadata
values to detect similarity, it is possible to have a considerable number of
software projects with similar project metadata attribute values. Thus, more
are such similar software matches, better would be the final estimate value,
and lesser would be the $LSD$ and $RE^{*}$ values.

Negative SA values: The negative SA values for the LOC straw man estimator
signify that the random guessing method performs better than it. However,
since the values have large std. dev. (or error) values associated with them,
we cannot reach any conclusion with confidence.

###  4.4. Experiment #2: k-fold cross-validation

Objective: To evaluate the performance of SDEE models and compare with the
existing methods using the k-fold cross-validation method.

Procedure: In k-fold cross-validation, the complete set of features is divided
into k parts, with one part used for testing and the rest of the (k-1) parts
for training. Thus, the complete training-testing phenomenon is performed for
k number of times with different training and testing sets. Three-fold, five-
fold, and ten-fold cross-validation experiments were performed to compare the
considered SDEE methods’ performance. Since the best results for most of the
methods were obtained corresponding to the ten-fold cross-validation, we
restrict our discussion to it. However, the complete set of results and models
are shared at https://bit.ly/3cuhRtQ.

Table 5. Performance comparison based on the Ten-Fold cross-validation

SDEE method | Evaluation metrics (mean $\pm$ std.dev) |
---|---|---
$LSD$ | $RE^{*}$ | $MAR$ | $MMRE$ | $MdMRE$ | $SA$
ATLM | 2.08 $\pm$ 0.21 | 0.48 $\pm$ 0.09 | 15 $\pm$ 2.82 | 192.4 $\pm$ 55.69 | 40.07 $\pm$ 17.05 | 53.57 $\pm$ 8.55
NeuralNet | 1.8 $\pm$ 0.88 | 22.29 $\pm$ 65.26 | 61 $\pm$ 76.3 | 328.93 $\pm$ 461.79 | 75.56 $\pm$ 42.38 | -33.99 $\pm$ 139.52
ABE | neigh =1 | 1.1 $\pm$ 0.37 | 0.65 $\pm$ 0.86 | 28.96 $\pm$ 24.14 | 75.07 $\pm$ 68.04 | 75.07 $\pm$ 68.04 | 33.028 $\pm$ 53.78
neigh =2 | 1.02 $\pm$ 0.49 | 0.51 $\pm$ 0.66 | 26.47 $\pm$ 22.56 | 51.52 $\pm$ 29.14 | 51.52 $\pm$ 29.14 | 38.73 $\pm$ 49.26
neigh =3 | 0.99 $\pm$ 0.28 | 0.53 $\pm$ 0.79 | 30.29 $\pm$ 24.83 | 76.01 $\pm$ 58.33 | 76.01 $\pm$ 58.33 | 29.93 $\pm$ 51.52
| neigh =4 | 0.91 $\pm$ 0.24 | 0.70 $\pm$ 0.98 | 33.20 $\pm$ 26.99 | 82.04 $\pm$ 72.82 | 82.04 $\pm$ 72.82 | 22.61 $\pm$ 58.31
| neigh =5 | 0.74 $\pm$ 0.35 | 0.84 $\pm$ 1.23 | 33.98 $\pm$ 31.34 | 107.29 $\pm$ 120.88 | 107.29 $\pm$ 120.88 | 20.94 $\pm$ 70.27
LOC straw man | neigh =1 | 0.97 $\pm$ 0.63 | 3.69 $\pm$ 7.93 | 64.21 $\pm$ 76.08 | 290.10 $\pm$ 505.84 | 290.10 $\pm$ 505.84 | -54.8 $\pm$ 189.05
neigh =2 | 0.9 $\pm$ 0.31 | 2.17 $\pm$ 3.63 | 53.98 $\pm$ 56.13 | 188.48 $\pm$ 237.89 | 188.48 $\pm$ 237.89 | -28.66 $\pm$ 134.81
neigh =3 | 0.71 $\pm$ 0.4 | 1.95 $\pm$ 3.25 | 50.77 $\pm$ 48.6 | 242.04 $\pm$ 418.77 | 242.04 $\pm$ 418.77 | -21.9 $\pm$ 119.10
| neigh =4 | 0.67 $\pm$ 0.32 | 1.44 $\pm$ 2.0 | 47.44 $\pm$ 40.02 | 213.73 $\pm$ 357.44 | 213.73 $\pm$ 357.44 | -12.99 $\pm$ 95.01
| neigh =5 | 0.6 $\pm$ 0.34 | 1.46 $\pm$ 1.94 | 46.16 $\pm$ 38.27 | 224.12 $\pm$ 366.30 | 224.12 $\pm$ 366.30 | -9.74 $\pm$ 90.32
DevSDEE | neigh =1 | 0.69 $\pm$ 0.51 | 0.19 $\pm$ 0.01 | 9.47 $\pm$ 1.12 | 94.31 $\pm$ 25.29 | 54.58 $\pm$ 7.27 | 56.79 $\pm$ 2.41
neigh =2 | 0.29 $\pm$ 0.22 | 0.41 $\pm$ 0.27 | 8.46 $\pm$ 0.08 | 47.88 $\pm$ 23.2 | 42.4 $\pm$ 23.2 | 57.05 $\pm$ 8.16
neigh =3 | 0.44 $\pm$ 0.20 | 0.53 $\pm$ 0.06 | 13.03 $\pm$ 0.67 | 48.72 $\pm$ 27.45 | 48.72 $\pm$ 27.45 | 50 $\pm$ 16.92
| neigh =4 | 0.43 $\pm$ 0.31 | 0.57 $\pm$ 0.80 | 17.22 $\pm$ 2.86 | 60.98 $\pm$ 15.24 | 60.98 $\pm$ 15.24 | 51.52 $\pm$ 21.79
| neigh =5 | 0.34 $\pm$ 0.21 | 0.70 $\pm$ 0.46 | 17.21 $\pm$ 0.44 | 49.17 $\pm$ 18.56 | 49.17 $\pm$ 18.56 | 50.37 $\pm$ 17.83

Observations: Table 5 shows the results obtained for ten-fold cross-validation
experiments. DevSDEE achieves the lowest mean and standard deviation values
for $LSD$, $RE^{*}$, $MAR$, $MMRE$, $MdMRE$, and $SA$ values and has the most
negligible errors in its SDEE estimates. The grey-colored cells in the table
represent the SDEE models performing attaining the best metrics’ values. As
observed from the table values, DevSDEE outperforms the existing methods and
achieves the highest $SA$ of 57.05$\pm$8.16 and least $RE^{*}$ of
0.19$\pm$0.01 with neigh=2 . Since the $MMRE$, $MdMRE$, and $SA$ values have
large std. dev. values (or errors), we decide using the $MAR$, $LSD$ and
$RE^{*}$ values of DevSDEE. The reason for the large std. dev. values could be
their dependency on the central measures (mean or median) of the data values
or the assumption of normal data distribution. Nevertheless, the $LSD$ and
$RE^{*}$ values present a clear picture of DevSDEE performing better than the
compared SDEE methods. However, we perform the additional non-parametric
statistical test, viz., t-test with the bootstrap procedure and the cliff’s
delta tests, that we discuss in Section 4.5 and Section 4.6.

Critical Analysis and Negative SA values: The large std. dev. values in the
case of $MMRE$, $MdMRE$, and $SA$ values signify the software effort estimates
have a considerable variation. This could be due to the difference in the
nature of software types and the kind of developer activity performed in
developing them. The effort required to develop a specific software library
might differ depending on their inherent functionalities or characteristics.
For instance, the Google Draco191919https://github.com/google/draco library is
used to compress and decompress 3D geometric meshes and point clouds, while
the Google Tink202020https://github.com/google/tink library is used to perform
encryption to provide secure APIs. Both the libraries differ in their
functionalities and hence would differ in the effort required to develop them.
However, the effort values could also differ for the same software library
types. For instance, Google snappy212121https://github.com/google/snappy is
another compression library, but it aims for very high speeds and reasonable
compression instead of maximum compression. Hence, the effort required to
develop Draco would differ from snappy as well but is expected to be closer to
snappy than Tink. The actual effort values essentially depend on the amount of
developer activity (in the form of $SLOC_{m}$) performed on a software
project, and we only aim to achieve an estimate using this approach. The
conclusions for critical analysis on DevSDEE behavior observed in Section 4.3
holds here as well, as DevSDEE performs best with neigh=1 or neigh=2.
Similarly, the inference for negative values of LOC straw man’s SA values
remains the same.

###  4.5. Experiment #3: Significance tests

Objective: To validate that there exists a negligible difference between the
actual effort values and the DevSDEE estimates

Need for statistical tests with the bootstrap procedure: Both the Randomized
trials and k-fold cross-validation methods have been considered to be
essential to assess the model prediction error and avoid misleading estimates
due to dataset instability (Turhan, 2012). However, these methods have been
remarked to produce a large variance for the datasets with outliers and are
considered unsuitable for SDEE datasets. This might be a reason behind the
large error values in the case of k-fold cross-validation results as compared
to the randomized trials’ results (as shown in Table 4 and Table 5), as the
number of iterations is larger in the case of randomized trials as compared to
the k-fold cross-validation method.

An alternative solution to this problem as suggested by the existing studies
(Whigham et al., 2015), is to test using many iterations to obtain a fair
estimate of performance and allow a meaningful statistical comparison. This is
particularly what is done by statistical tests with bootstrap procedures, and
therefore, we perform additional experiments using t-test, cliff-delta test,
and other parametric effect tests with bootstrapping to validate our findings.

Description: For DevSDEE to perform efficiently, it should yield effort
estimates close to the true effort values. Therefore, we perform this
experiment to validate that DevSDEE estimates are close to true effort values
or negligible difference between them. The objective of this experiment is
very similar to the null hypothesis $H_{0}$ of the effect tests, which states
that there lies no difference between the test group and control group data
values. All these effect size variations (defined in Section A.1.2) are
studied with $H_{0}$ and a significance value ($p$).

Procedure: We used the
researchpy222222https://researchpy.readthedocs.io/en/latest/ttest_documentation.html
and the DABEST python libraries232323https://github.com/ACCLAB/DABEST-python
to compute the values of t-test, cohen’s $d$, hedges’ $g$, glass’s $\Delta$,
and cliff’s $\delta$ (Ho et al., 2019). Both the libraries compute the
estimates with a 95% confidence interval and by taking 5000 bootstrap samples.
To compute the estimate values, we provide the DevSDEE effort estimates and
the true effort values obtained from the Random Trials and the k-fold cross-
validation experiments as discussed in Section 4.3 and Section 4.4.

Table 6. Significance Testing to evaluate the DevSDEE results

| t-significance test | Cliff’s delta effect test | Parametric effect tests
---|---|---|---
$t-val$ | $p-val$ | $cliff^{\prime}s\ \delta$ | $p-val$ | $cohen^{\prime}s\ d$ | $hedges^{\prime}\ g$ | $glass^{\prime}s\ \Delta$ | $p-val$
Random Trials | neigh=1 | 0.23 $\pm$ 0.29 | 0.77 $\pm$ 0.11 | 0 $\pm$ 0.17 | 0.63 $\pm$ 0.15 | 0.22 $\pm$ 0.22 | 0.13 $\pm$ 0.18 | 0.32 $\pm$ 0.25 | 0.67 $\pm$ 0.1
neigh=2 | 0.18 $\pm$ 0.27 | 0.80 $\pm$ 0.11 | 0.02 $\pm$ 0.09 | 0.65 $\pm$ 0.06 | 0.18 $\pm$ 0.21 | 0.1 $\pm$ 0.17 | 0.23 $\pm$ 0.27 | 0.66 $\pm$ 0.07
neigh=3 | 0.11 $\pm$ 0.35 | 0.83 $\pm$ 0.19 | 0.03 $\pm$ 0.16 | 0.6 $\pm$ 0.11 | 0.10 $\pm$ 0.25 | 0.06 $\pm$ 0.22 | 0.12 $\pm$ 0.36 | 0.62 $\pm$ 0.11
neigh-4 | 0.14 $\pm$ 0.4 | 0.81 $\pm$ 0.21 | 0.03 $\pm$ 0.16 | 0.60 $\pm$ 0.11 | 0.12 $\pm$ 0.28 | 0.08 $\pm$ 0.25 | 0.17 $\pm$ 0.43 | 0.6 $\pm$ 0.11
neigh=5 | 0.15 $\pm$ 0.38 | 0.8 $\pm$ 0.20 | 0.03 $\pm$ 0.16 | 0.6 $\pm$ 0.11 | 0.13 $\pm$ 0.27 | 0.09 $\pm$ 0.23 | 0.25 $\pm$ 0.48 | 0.61 $\pm$ 0.09
k-Fold | neigh=1 | 0.06 $\pm$ 0.61 | 0.7 $\pm$ 0.24 | 0 $\pm$ 0.37 | 0.52 $\pm$ 0.31 | 0.05 $\pm$ 0.43 | 0.04 $\pm$ 0.37 | 0.07 $\pm$ 0.43 | 0.68 $\pm$ 0.23
neigh=2 | 0.04 $\pm$ 0.58 | 0.70 $\pm$ 0.2 | 0.08 $\pm$ 0.19 | 0.62 $\pm$ 0.12 | 0.03 $\pm$ 0.41 | 0.02 $\pm$ 0.35 | 0.02 $\pm$ 0.48 | 0.67 $\pm$ 0.16
neigh=3 | 0.13 $\pm$ 0.79 | 0.56 $\pm$ 0.16 | 0.12 $\pm$ 0.33 | 0.44 $\pm$ 0.08 | 0.09 $\pm$ 0.56 | 0.08 $\pm$ 0.49 | 0.02 $\pm$ 0.79 | 0.51 $\pm$ 0.18
neigh=4 | 0.24 $\pm$ 0.88 | 0.48 $\pm$ 0.05 | 0.12 $\pm$ 0.33 | 0.44 $\pm$ 0.08 | 0.17 $\pm$ 0.62 | 0.15 $\pm$ 0.54 | 0.07 $\pm$ 0.96 | 0.43 $\pm$ 0.06
neigh-5 | 0.24 $\pm$ 0.85 | 0.5 $\pm$ 0.06 | 0.12 $\pm$ 0.33 | 0.44 $\pm$ 0.08 | 0.17 $\pm$ 0.6 | 0.15 $\pm$ 0.52 | 0.09 $\pm$ 1.04 | 0.47 $\pm$ 0.06

Observations and Analysis: Both the t-test and the parametric effect tests
measure the difference between the mean values of these groups, and the $p$
value represents the significance (or probability) of these estimates. A large
effort value associated with a $p<0.05$ (marked as 95% confidence level) leads
to the rejection of $H_{0}$, concluding that there is a considerable
difference between the values of the considered groups. Similarly, if the
effort values are small and are associated with large $p$ values, it is
concluded that there is considerable overlap in the data values of the
considered group, or the differences between the values are small (Cohen,
1988). However, $H_{0}$ can either be rejected or stated that it is impossible
to reject the null hypothesis (if $p$ values are large or $>0.1$). As observed
from Table 6 values, all the t-test values have t values $<0.24\pm 0.88$
associated with high $p$ values lie in $[0.48,0.83]$. A similar trend is
observed for all the parametric effect tests, viz., $cohen^{\prime}s\ d$,
$hedges^{\prime}\ g$, and $glass^{\prime}s\ \Delta$, where all the effect
values in the range $[0.02,0.32]$ and the p-values in $[0.43,0.68]$, which
imply a small effect (as discussed in Section A.1.2 point 8). Also, the
$cliff^{\prime}s\delta$ values range lies in $[0\pm 0.17,0.12\pm 0.33]$ with
large p values $[0.44,0.65]$, which again depicts a large overlap between the
DevSDEE estimates and true effort values. Therefore, with such small t-values
and effect values and large p-values, we can conclude that the differences
between the estimates are minimal, and thus, we fail to reject the null
hypothesis $H_{0}$. However, to validate our objective with better confidence,
we perform additional experiments as discussed in Section 4.6.

###  4.6. Experiment #4

Objective: To validate that DevSDEE provides better effort estimates than the
existing methods

Description: To validate our hypothesis, we consider the $MAR$ of the
estimates as inputs to the t-test and the effect experiments. While comparing
DevSDEE with an existing approach $Z$, a large positive difference (effect or
t values) between the $MAR$ values of the $Z$ estimates and the DevSDEE
estimates would imply an increased error in $Z$ estimates as compared to
DevSDEE’s estimates. Similarly, we use the $SA$ measure for comparing the two
estimates where we replace the $MAR$ of random guessing approach with the
$MAR$ values of $Z$ in equation 13. The null hypothesis $H_{0}$ for this
experiment would be defined as _there lies negligible difference between the
error ( $MAR$) values of the estimates made by $Z$ approach and DevSDEE_, with
$Z$ = {ATLM, NeuralNet, ABE, LOC straw man}.

Procedure: We perform the t-test, cliff’s $\delta$ test, and the parametric
effect tests, viz., cohen’s $d$, hedges’ $g$, glass’s $\Delta$ using the $MAR$
values obtained in the experiments #1 and #1 ( discussed in Section 4.3 and
Section 4.4). Additionally, we also compute the $SA$ measure values using the
$MAR$ values as input. We use the
researchpy242424https://researchpy.readthedocs.io/en/latest/ttest_documentation.html
and the DABEST python libraries252525https://github.com/ACCLAB/DABEST-python
to perform these tests. All the tests are performed with a 95% confidence
interval and by taking 5000 bootstrap samples.

Table 7. Significance Testing for comparison with the existing methods

| SA | t-significance test | Cliff’s delta effect test | Parametric effect tests
---|---|---|---|---
Method | $t-val$ | $p-val$ | $cliff^{\prime}s\ \delta$ | $p-val$ | $cohen^{\prime}s\ d$ | $hedges^{\prime}\ g$ | $glass^{\prime}s\ \Delta$ | $p-val$
Random Trials | ATLM | 42.7 | 2.49 | 0.02 | 0.74 | $<0.00001$ | 1.06 | 1.02 | 1.74 | 0.02
NeuralNet | 73 | 4.13 | 0.0002 | 0.86 | $<0.00001$ | 1.31 | 1.28 | 0.98 | $<0.00001$
ABE | 72.66 | 2.58 | 0.01 | 0.57 | 0.0006 | 0.82 | 0.8 | 0.59 | 0.001
LOC straw man | 84.22 | 2.81 | 0.008 | 0.88 | $<0.00001$ | 0.89 | 0.87 | 0.63 | $<0.00001$
k-Fold | ATLM | 35.13 | 2.46 | 0.02 | 0.74 | $<0.00001$ | 0.78 | 0.76 | 1.87 | 0.01
NeuralNet | 56.35 | 1.38 | 0.05 | 0.52 | 0.04 | 0.62 | 0.59 | 0.45 | 0.05
ABE | 74.06 | 2.58 | 0.01 | 0.57 | 0.0006 | 0.82 | 0.8 | 0.59 | 0.001
LOC straw man | 87.26 | 2.81 | 0.008 | 0.88 | $<0.00001$ | 0.89 | 0.87 | 0.63 | $<0.00001$

Observations and Analysis: The results of this experiment are listed in Table
7. As observed from the t-significance test results, the t-values lie in
$[1.38,2.81]$ with p-values $<=0.05$. Therefore, we can state that we can
reject $H_{0}$ with a 95% confidence level (and more in some cases where
$p<0.05$). Hence, this also implies that DevSDEE estimates are considerably
closer to the true effort values than the existing methods ($Z$). A similar
trend is observed for parametric effect tests with effect values ranging in
$[0.45,1.74]$ and p-values $<=0.05$, which represents the values ranging from
medium to a effect (as stated in Section A.1.2 point 8). Thus, the parametric
tests also reject $H_{0}$ with a 95% confidence level. When analyzing the
cliff’s $\delta$ effect test results, the effect values range in $[0.52,0.88]$
with p-values $<=0.04$ (mostly $<0.001$). Thus, these effect values provide a
96% level of confidence to reject $H_{0}$.

###  4.7. Experiment #5

Objective: To validate the correlation between Software Development Effort
Estimation metrics and software development effort

Description: We performed various experiments to validate our hypothesis that:
The developer activity information is useful in estimating the software
development effort, or more specifically, to validate that SDEE metrics
capture the developer activity information and show correlation with the
overall development effort.

Procedure: To perform these experiments, we compute the correlation between
various SDEE metrics values present in our dataset. We use the Pearson’s
correlation coefficient (Benesty et al., 2009), which is used to measure the
linear relationship between two variables. Note that Pearson’s correlation
coefficient has a range of values as [-1, 1].

To detect the correlation between different SDEE metrics, we computed
Pearson’s correlation coefficient values for the following SDEE metrics:

  1. A)

The total number of SLOC modifications ($SLOC_{m}$) performed while developing
a software repository vs. the effort expended in developing it.

  2. B)

The total number of developers (devCount) working on a software repository vs.
the effort expended in developing it.

  3. C)

The total time spent developing a software repository vs. the effort expended
in developing it.

Figure 9. Correlation between various SDEE metrics

Results and observations: Fig. 9 presents the correlation values for each of
the comparison types listed above. As observed from the figure, development
time has the highest correlation value of 0.799, which is explainable as more
is the time spent developing a software project, more will be the effort (in
person-months). Similarly, devCount shows a correlation value of 0.644
compared with the development effort and is explainable using the same reason
for development time. However, $SLOC_{m}$ shows a small positive correlation
value of $0.065$, which signifies that through the effort increases with the
increase in $SLOC$ modifications, the increase is much smaller than the
increase in devCount and development time. Thus, we can infer that devCount
and development time are useful SDEE metrics and influence the software
development effort more than the $SLOC_{m}$, which is the commonly used SDEE
metric.

###  4.8. Experiment #6

Objective: To validate the performance of our SDEE tool by professional
programmers and test if it works as intended to work for the user audience.

The research question addressed: _How does the SDEE tool perform from a
programmer’s perspective when judged on various quality parameters?_

Description: To evaluate our tool’s performance, we consider the following
quality parameters: the _ease-of-use_ , _response-time_ , and _accuracy_. We
invited professional programmers to perform this experiment and asked them to
use our tool and validate its performance using the available information of
existing GitHub repositories. One hundred and eleven programmers responded. We
had a mixture of candidates from final year computer science undergraduates
(53), postgraduates (47), and IT industry professionals (11) with experience
between 0 and 6 years. Each of the participants had prior knowledge of the
software engineering fundamentals. The experiment was performed in a
controlled industrial environment.

Procedure: The programmers were requested to perform the following steps:

  1. (1)

Enter the details of a known or an available software product using the tool’s
input interface. The participants could use the available information of
existing GitHub repositories to perform this validation and are not required
to compute the effort values themselves.

  2. (2)

Check the estimates provided as the response by the tool for the entered
software requirements description by comparing it with the original effort
values of the product. Note: We do not expect the participants to compute the
effort values on their own. The participants were asked to use the effort
values available online and validate that given a similar software description
if the tool outputs a similar effort value. For instance, the participants
could use the effort information of the existing GitHub repositories.

  3. (3)

Rate the tool’s ease of use, response time, and accuracy, as observed from the
experiment(s). The programmers were asked to provide their ratings on a scale
of $1$–$10$. Here, $1$ represents the worst performance, and $10$ represents
the best.

Interpretation of Table 8: Table 8 gives the details of the ratings received
by the tool on various quality parameters as evaluated by the participants.
The scores represented in the table are on a scale of $1$–$10$, where $1$
represents the worst performance, and $10$ represents the best performance.
Similarly, the votes are represented in the form of percentages. The sum of
votes across every column equals $100$ (or %). The first row of the table
shows that a score of $10$ was awarded for ease-of-use by $60\%$ voters, for
response time by $80\%$ voters, and for accuracy, it was awarded by $60\%$
voters. Similarly, the third row says that the score of $8/10$ was awarded for
all the quality parameters by only $10\%$ voters.

Table 8. Ratings recorded in Experiment #6

Score $Y$ | Votes for the score $Y$ (in %)
---|---
(scale 1-10) | _ease-of-use_ | _response time_ | _accuracy_
$10$ | $60$ | $80$ | $60$
$9$ | $30$ | $10$ | $30$
$8$ | $10$ | $10$ | $10$
$7$ | $0$ | $0$ | $0$
$6$ | $0$ | $0$ | $0$

Results and observations: Some of the significant observations from Table 8
are:

  1. (1)

Our tool received a rating of $>=8/10$ from all the participants for ease of
use. Thus, most participants found it easy to use (or learn how to use it).

  2. (2)

Our tool achieves impressive (or minimal) response time as it received a
rating of $10/10$ by 80% of the participants.

  3. (3)

All the participants found our tool’s estimates to be considered accurate as
they rated it $>=8/10$ for accuracy.

##  5\. Threats to validity

###  5.1. Threats to internal validity

  1. (1)

_Handling the Corner cases:_ A “refactoring” stage may occur in a software’s
life-cycle, producing a sudden spike in the modification and commits count. We
treat such cases as the “corner” cases for our work.

  2. (2)

_Removing the outliers:_ While developing the SDEE dataset, we performed the
following to avoid the outliers:

    1. (a)

Selecting OSS repositories having size $>=5$ MBs: We observed that the
repositories with size $<5$ MBs were either at the very initial stage of
software development or had very few source code files present in them.

    2. (b)

Selecting the repositories that have been updated at least once in the last
three years: To avoid the very old or inactive repositories.

    3. (c)

Selected the repositories with more than $500$ stars: To fetch repositories
with considerable developer involvement.

    4. (d)

Filtering the repositories belonging to specific categories: To ensure the
homogeneity of the dataset, we selected software repositories belonging to
different types.

  3. (3)

_Missed repositories:_ Due to this constrained repository selection, we might
have missed some relevant unlabeled (by categories) repositories or some
relevant repositories with smaller star counts. However our objective of
developing a homogeneous dataset (for software types) free from outliers is
fulfilled. Nevertheless, as is the case with any experiment, it is impossible
to cover all possibilities for the detection and avoidance of the swamping
effect (Chiang et al., 2007).

  4. (4)

_Validity of the OSS project descriptions:_ For developing our software effort
estimation method, we determine similar software matches using a PVA model
trained on the software product descriptions. Thus, we assume that the project
descriptions present in VCS repositories are valid and do not change very
often. The OSS software products considered while developing our tool are
present in the GitHub OSS repositories and usually serve as software modules
in developing more extensive software systems. Therefore, our system can only
provide effort estimates to develop such modules of the software. The effort
estimates provided corresponding to such modules of software modules can be
summed up to obtain the aggregate effort estimate required to develop the more
sophisticated software.

###  5.2. Threats to external validity

Our work is based on the developer activity information extracted from various
GitHub OSS repositories. Therefore, our experiments’ results may differ when
tested on proprietary software because industries have full-time working
employees (Capiluppi and Izquierdo-Cortázar, 2013). Thus the developer
activity while developing a proprietary software might differ due to the
different working patterns. However, our approach remains valid regardless of
whether we used the data from OSS or non-OSS projects. For example, if the
data used is taken from non-volunteer-driven projects such as proprietary
software development, the approach would remain valid. This is because the
similarity in software projects is detected based on the software descriptions
and not the developer activity.

###  5.3. Threats to statistical conclusion validity

To validate the efficacy of our approach over the existing methods, we perform
several statistical experiments as discussed in Section 4.5 and Section 4.6.
We report our findings at a 95% confidence level with a 0.05 significance
level. However, for many cases, we also obtained the results with a 99.999%
confidence level ($<0.00001$ significance).

##  6\. Conclusion and Future Directions

SDEE for a software project is performed by analyzing the information of past
software projects of similar nature. We propose a tool that estimates the
effort required to develop a newly-envisioned software project $z$ by
analyzing the developer activity information (tracked in VCSs) of software
projects with similar functionality as $z$. SDEE metrics that we proposed are
used to extract the developer activity information present in various VCS
repositories. To determine the functionally similar software projects to $z$,
we utilize the project descriptions associated with them.

Our experimental results show that the developer activity metrics serve as a
better input to an SDEE method than the existing software metrics, such as
SLOC. Further, when trained on software product descriptions, a PVA-based
document comparison allows for discovering more relevant software projects
from the past whose effort estimates allow for more accurate and reliable
SDEE. For instance, when estimating effort for a compression library ($z$)
with “high compression” and “efficiency” as characteristics (used as text
keywords by PVA for similarity detection), the top-2 similar matches obtained
are Google Draco and Centaurean Density libraries. Both these libraries have
the same characteristics highlighted in their project descriptions, and match
the software requirements of $z$. Our system achieves the highest standardized
accuracy of 87.26% (cliff’s $\delta$=0.88) when compared with the LOC straw
man estimator and 42.7% (cliff’s $\delta$=0.74) with the ATLM method at 99%
confidence level ($p<0.00001$). Based on our experimental findings, we
conclude that the use of developer activity information of past software helps
develop better software estimation methods and tools. We plan to extend our
work for providing incremental effort estimates (after one month, two months,
and so on) in the future. Since developer acts as the primary source of
source-code contributor, we plan to study novel developer-related factors,
such as developer’s characteristics (Stray et al., 2017; Yang et al., 2020; Li
et al., 2012), developer’s geographical location (Rastogi et al., 2018),
social interactions of developers (Iden and Bygstad, 2018; Zhang et al.,
2017), and test their effect on software development.

## References

  * Abrahão et al. (2018) Silvia Abrahão, Lucia De Marco, Filomena Ferrucci, Jaime Gomez, Carmine Gravino, and Federica Sarro.  Definition and evaluation of a cosmic measurement procedure for sizing web applications in a model-driven development environment.  _Information and Software technology_ , 104:144–161, 2018\.
  * Abualkishik et al. (2017) Abedallah Zaid Abualkishik, Filomena Ferrucci, Carmine Gravino, Luigi Lavazza, Geng Liu, Roberto Meli, and Gabriela Robiolo.  A study on the statistical convertibility of ifpug function point, cosmic function point and simple function point.  _Information and Software Technology_ , 86:1–19, 2017.
  * Albrecht (1979) Allan J Albrecht.  Measuring application development productivity.  In _Proc. Joint Share, Guide, and IBM Application Development Symposium, 1979_ , 1979.
  * Amor et al. (2006) Juan Jose Amor, Gregorio Robles, and Jesus M Gonzalez-Barahona.  Effort estimation by characterizing developer activity.  In _Proceedings of the 2006 international workshop on Economics driven software engineering research_ , pages 3–6. ACM, 2006.
  * Asl and Kama (2013) Mehran Halimi Asl and Nazri Kama.  A change impact size estimation approach during the software development.  In _Software Engineering Conference (ASWEC), 2013 22nd Australian_ , pages 68–77. IEEE, 2013.
  * Barry et al. (1981) Boehm Barry et al.  Software engineering economics.  _New York_ , 197, 1981.
  * Beel et al. (2013) Joeran Beel, Stefan Langer, Marcel Genzmehr, and Andreas Nürnberger.  Introducing docear’s research paper recommender system.  In _JCDL_ , pages 459–460, 2013.
  * Bell et al. (2011) Robert M Bell, Thomas J Ostrand, and Elaine J Weyuker.  Does measuring code change improve fault prediction?  In _Proceedings of the 7th international conference on predictive models in software engineering_ , pages 1–8, 2011.
  * Benesty et al. (2009) Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen.  Pearson correlation coefficient.  In _Noise reduction in speech processing_ , pages 1–4. Springer, 2009\.
  * Bleeker et al. (2003) SE Bleeker, HA Moll, EW Steyerberg, ART Donders, Gerarda Derksen-Lubsen, DE Grobbee, and KGM Moons.  External validation is necessary in prediction research:: A clinical example.  _Journal of clinical epidemiology_ , 56(9):826–832, 2003.
  * Boetticher (2007) G Boetticher.  The promise repository of empirical software engineering data.  _http://promisedata. org/repository_ , 2007.
  * Briand and Wieczorek (2002) Lionel C Briand and Isabella Wieczorek.  Resource estimation in software engineering.  _Encyclopedia of software engineering_ , 2002.
  * Capiluppi and Izquierdo-Cortázar (2013) Andrea Capiluppi and Daniel Izquierdo-Cortázar.  Effort estimation of floss projects: a study of the linux kernel.  _Empirical Software Engineering_ , 18(1):60–88, 2013.
  * Catolino et al. (2018) Gemma Catolino, Fabio Palomba, Andrea De Lucia, Filomena Ferrucci, and Andy Zaidman.  Enhancing change prediction models using developer-related factors.  _Journal of Systems and Software_ , 143:14–28, 2018.
  * Chen and Xing (2016) Chunyang Chen and Zhenchang Xing.  Similartech: automatically recommend analogical libraries across different programming languages.  In _2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE)_ , pages 834–839. IEEE, 2016.
  * Chiang et al. (2007) Jung-Tsung Chiang et al.  The masking and swamping effects using the planted mean-shift outliers models.  _Int. J. Contemp. Math. Sciences_ , 2(7):297–307, 2007.
  * Cliff (1993) Norman Cliff.  Dominance statistics: Ordinal analyses to answer ordinal questions.  _Psychological bulletin_ , 114(3):494, 1993.
  * Coelho et al. (2020) Jailton Coelho, Marco Tulio Valente, Luciano Milen, and Luciana L Silva.  Is this github project maintained? measuring the level of maintenance activity of open-source projects.  _Information and Software Technology_ , 122:106274, 2020\.
  * Cohen (1988) J Cohen.  Statistical power analysis for the behavioral science. lawrence erlbaum associates.  _New York_ , 1988.
  * Cohen (2013) Jacob Cohen.  _Statistical power analysis for the behavioral sciences_.  Academic press, 2013.
  * Cohn (2005) Mike Cohn.  _Agile estimating and planning_.  Pearson Education, 2005.
  * Conte et al. (1986) Samuel Daniel Conte, Hubert E Dunsmore, and YE Shen.  _Software engineering metrics and models_.  Benjamin-Cummings Publishing Co., Inc., 1986.
  * D’Avanzo et al. (2015) Loris D’Avanzo, Filomena Ferrucci, Carmine Gravino, and Pasquale Salza.  Cosmic functional measurement of mobile applications and code size estimation.  In _Proceedings of the 30th Annual ACM Symposium on Applied Computing_ , pages 1631–1636, 2015.
  * De Vito et al. (2020) Gabriele De Vito, Filomena Ferrucci, and Carmine Gravino.  Design and automation of a cosmic measurement procedure based on uml models.  _Software and Systems Modeling_ , 19(1):171–198, 2020.
  * Di Martino et al. (2016) Sergio Di Martino, Filomena Ferrucci, Carmine Gravino, and Federica Sarro.  Web effort estimation: function point analysis vs. cosmic.  _Information and Software Technology_ , 72:90–109, 2016\.
  * Di Nucci et al. (2017) Dario Di Nucci, Fabio Palomba, Giuseppe De Rosa, Gabriele Bavota, Rocco Oliveto, and Andrea De Lucia.  A developer centered bug prediction model.  _IEEE Transactions on Software Engineering_ , 44(1):5–24, 2017.
  * Dumke and Abran (2016) Reiner Dumke and Alain Abran.  _COSMIC Function Points: Theory and Advanced Practices_.  CRC Press, 2016.
  * Galorath (2007) Dan Galorath.  Software reuse and commercial off-the-shelf software.  _Galorath Incorporation, El Segundo, CA_ , 2007.
  * Gousios et al. (2008) Georgios Gousios, Eirini Kalliamvakou, and Diomidis Spinellis.  Measuring developer contribution from software repository data.  In _Proceedings of the 2008 international working conference on Mining software repositories_ , pages 129–132. ACM, 2008.
  * Han et al. (2019) Junxiao Han, Shuiguang Deng, Xin Xia, Dongjing Wang, and Jianwei Yin.  Characterization and prediction of popular projects on github.  In _2019 IEEE 43rd annual computer software and applications conference (COMPSAC)_ , volume 1, pages 21–26. IEEE, 2019.
  * Hassan (2009) Ahmed E Hassan.  Predicting faults using the complexity of code changes.  In _2009 IEEE 31st international conference on software engineering_ , pages 78–88. IEEE, 2009.
  * Hedges (1981) Larry V Hedges.  Distribution theory for glass’s estimator of effect size and related estimators.  _journal of Educational Statistics_ , 6(2):107–128, 1981.
  * Hedges and Olkin (2014) Larry V Hedges and Ingram Olkin.  _Statistical methods for meta-analysis_.  Academic press, 2014.
  * Herbsleb (2014) James D. Herbsleb.  Socio-technical coordination, Jun 2014.  URL https://www.youtube.com/watch?v=v0CSnYvd0C4.
  * Ho et al. (2019) Joses Ho, Tayfun Tumkaya, Sameer Aryal, Hyungwon Choi, and Adam Claridge-Chang.  Moving beyond p values: data analysis with estimation graphics.  _Nature methods_ , 16(7):565–566, 2019.
  * Iden and Bygstad (2018) Jon Iden and Bendik Bygstad.  The social interaction of developers and it operations staff in software development projects.  _International Journal of Project Management_ , 36(3):485–497, 2018.
  * Jørgensen et al. (2009) Magne Jørgensen, Barry Boehm, and Stan Rifkin.  Software development effort estimation: Formal models or expert judgment?  _IEEE software_ , 26(2):14–19, 2009.
  * Kama (2011) Nazri Mohd Kama.  _A change impact analysis framework for the software development phase_.  University of Western Australia, 2011.
  * Kampenes et al. (2007) Vigdis By Kampenes, Tore Dybå, Jo E Hannay, and Dag IK Sjøberg.  A systematic review of effect size in software engineering experiments.  _Information and Software Technology_ , 49(11-12):1073–1086, 2007.
  * Kitchenham et al. (2002) Barbara A Kitchenham, Shari Lawrence Pfleeger, Lesley M Pickard, Peter W Jones, David C. Hoaglin, Khaled El Emam, and Jarrett Rosenberg.  Preliminary guidelines for empirical research in software engineering.  _IEEE Transactions on software engineering_ , 28(8):721–734, 2002.
  * Kocaguneli et al. (2011a) Ekrem Kocaguneli, Tim Menzies, Ayse Bener, and Jacky W Keung.  Exploiting the essential assumptions of analogy-based effort estimation.  _IEEE transactions on software engineering_ , 38(2):425–438, 2011a.
  * Kocaguneli et al. (2011b) Ekrem Kocaguneli, Ayse T Misirli, Bora Caglayan, and Ayse Bener.  Experiences on developer participation and effort estimation.  In _2011 37th EUROMICRO Conference on Software Engineering and Advanced Applications_ , pages 419–422. IEEE, 2011b.
  * Kutner et al. (2005) Michael H Kutner, Christopher J Nachtsheim, John Neter, William Li, et al.  _Applied linear statistical models_ , volume 5.  McGraw-Hill Irwin Boston, 2005.
  * Langdon et al. (2016) William B Langdon, Javier Dolado, Federica Sarro, and Mark Harman.  Exact mean absolute error of baseline predictor, marp0.  _Information and Software Technology_ , 73:16–18, 2016\.
  * Le and Mikolov (2014) Quoc Le and Tomas Mikolov.  Distributed representations of sentences and documents.  In _International conference on machine learning_ , pages 1188–1196, 2014.
  * Li et al. (2012) Yan Li, Chuan-Hoo Tan, and Hock-Hai Teo.  Leadership characteristics and developers’ motivation in open source software development.  _Information & Management_, 49(5):257–267, 2012\.
  * Macbeth et al. (2011) Guillermo Macbeth, Eugenia Razumiejczyk, and Rubén Daniel Ledesma.  Cliff’s delta calculator: A non-parametric effect size program for two groups of observations.  _Universitas Psychologica_ , 10(2):545–555, 2011\.
  * Menzies et al. (2006) Tim Menzies, Zhihao Chen, Jairus Hihn, and Karen Lum.  Selecting best practices for effort estimation.  _IEEE Transactions on Software Engineering_ , 32(11):883–895, 2006.
  * Menzies et al. (2012) Tim Menzies, Bora Caglayan, Ekrem Kocaguneli, Joe Krall, Fayola Peters, and Burak Turhan.  The promise repository of empirical software engineering data, 2012.
  * Menzies et al. (2017) Tim Menzies, Ye Yang, George Mathew, Barry Boehm, and Jairus Hihn.  Negative results for software effort estimation.  _Empirical Software Engineering_ , 22(5):2658–2683, 2017.
  * Mikolov et al. (2013a) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  In _Advances in neural information processing systems_ , pages 3111–3119, 2013a.
  * Mikolov et al. (2013b) Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig.  Linguistic regularities in continuous space word representations.  In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 746–751, 2013b.
  * Miller (2000) James Miller.  Applying meta-analytical procedures to software engineering experiments.  _Journal of Systems and Software_ , 54(1):29–39, 2000.
  * Minkiewicz (1997) Arlene Minkiewicz.  Measuring object oriented software with predictive object points.  _PRICE Systems, LLC_ , pages 609–866, 1997.
  * Minku and Yao (2013) Leandro L Minku and Xin Yao.  Software effort estimation as a multiobjective learning problem.  _ACM Transactions on Software Engineering and Methodology (TOSEM)_ , 22(4):1–32, 2013.
  * Molokken and Jorgensen (2003) Kjetil Molokken and Magne Jorgensen.  A review of software surveys on software effort estimation.  In _2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings._ , pages 223–230. IEEE, 2003.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830, 2011\.
  * Pressman (2005) Roger S Pressman.  _Software engineering: a practitioner’s approach_.  Palgrave Macmillan, 2005.
  * Putnam (1978) Lawrence H. Putnam.  A general empirical solution to the macro software sizing and estimating problem.  _IEEE transactions on Software Engineering_ , 4:345–361, 1978.
  * Rahman and Devanbu (2013) Foyzur Rahman and Premkumar Devanbu.  How, and why, process metrics are better.  In _2013 35th International Conference on Software Engineering (ICSE)_ , pages 432–441. IEEE, 2013.
  * Ramos et al. (2003) Juan Ramos et al.  Using tf-idf to determine word relevance in document queries.  In _Proceedings of the first instructional conference on machine learning_ , volume 242, pages 133–142. Piscataway, NJ, 2003.
  * Rastogi et al. (2018) Ayushi Rastogi, Nachiappan Nagappan, Georgios Gousios, and André van der Hoek.  Relationship between geographical location and evaluation of developer contributions in github.  In _Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement_ , pages 1–8, 2018.
  * Robles et al. (2014) Gregorio Robles, Jesús M González-Barahona, Carlos Cervigón, Andrea Capiluppi, and Daniel Izquierdo-Cortázar.  Estimating development effort in free/open source software projects by mining software repositories: a case study of openstack.  In _Proceedings of the 11th Working Conference on Mining Software Repositories_ , pages 222–231. ACM, 2014.
  * Rosenthal et al. (1994) Robert Rosenthal, Harris Cooper, and L Hedges.  Parametric measures of effect size.  _The handbook of research synthesis_ , 621(2):231–244, 1994.
  * Salton and Buckley (1988) Gerard Salton and Christopher Buckley.  Term-weighting approaches in automatic text retrieval.  _Information processing & management_, 24(5):513–523, 1988.
  * Selamat et al. (2014) A Selamat, H Fujita, and Habibollah Haron.  _New Trends in Software Methodologies, Tools and Techniques: Proceedings of the Thirteenth SoMeT_14_ , volume 265.  Ios Press, 2014.
  * Semenick (1990) Doug Semenick.  Tests and measurements: The t-test.  _Strength & Conditioning Journal_, 12(1):36–37, 1990.
  * Shepperd and MacDonell (2012) Martin Shepperd and Steve MacDonell.  Evaluating prediction systems in software project estimation.  _Information and Software Technology_ , 54(8):820–827, 2012.
  * Sommerville (2011) Ian Sommerville.  _Software engineering_.  Addison-Wesley/Pearson, 2011.
  * Stray et al. (2017) Viktoria Stray, Nils Brede Moe, and Gunnar R Bergersen.  Are daily stand-up meetings valuable? a survey of developers in software teams.  In _International Conference on Agile Software Development_ , pages 274–281. Springer, 2017.
  * Thung et al. (2013) Ferdian Thung, Shaowei Wang, David Lo, and Julia Lawall.  Automatic recommendation of api methods from feature requests.  In _Proceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering_ , pages 290–300. IEEE Press, 2013.
  * Trstenjak et al. (2014) Bruno Trstenjak, Sasa Mikac, and Dzenana Donko.  Knn with tf-idf based framework for text categorization.  _Procedia Engineering_ , 69:1356–1364, 2014.
  * Turhan (2012) Burak Turhan.  On the dataset shift problem in software engineering prediction models.  _Empirical Software Engineering_ , 17(1):62–74, 2012.
  * Walkerden and Jeffery (1999) Fiona Walkerden and Ross Jeffery.  An empirical study of analogy-based software effort estimation.  _Empirical software engineering_ , 4(2):135–158, 1999.
  * Whigham et al. (2015) Peter A Whigham, Caitlin A Owen, and Stephen G Macdonell.  A baseline model for software effort estimation.  _ACM Transactions on Software Engineering and Methodology (TOSEM)_ , 24(3):1–11, 2015.
  * Winters and Schneider (1900) Jason P Winters and Geri Schneider.  _Applying use cases_.  Pearson Education Incorporated, 1900.
  * Wu (1997) Liming Wu.  The comparison of the software cost estimating methods.  _University of Calgary_ , pages 2–12, 1997.
  * Yang et al. (2020) Wenhua Yang, Minxue Pan, Yu Zhou, and Zhiqiu Huang.  Developer portraying: A quick approach to understanding developers on oss platforms.  _Information and Software Technology_ , 125:106336, 2020\.
  * Zhang et al. (2017) Yang Zhang, Huaimin Wang, Gang Yin, Tao Wang, and Yue Yu.  Social media in github: the role of@-mention in assisting software development.  _Science China Information Sciences_ , 60(3):1–18, 2017.

##  Appendix A Appendix

###  A.1. PVA parameter tuning experiments

The performance of PVA depends on its input parameters, viz., $\beta,\gamma,$
and $\psi$ (see Table 1). Further, one of the significant challenges is
determining the threshold similarity value ($\hat{\alpha}$) between two PVA
vectors, which helps us classify them as similar or dissimilar. We performed
several experiments to determine the optimal values of $\beta,\gamma,\psi$,
and $\hat{\alpha}$. Table 9 lists the experimental scenarios, while
Algorithm-2 describes the specific steps performed for training the software
similarity detection models.

Table 9. Scenarios for training the software similarity detection models

Parameters varied |
---|---
Epochs, $\beta$ | Vector size, $\gamma$ | Training samples, $\psi$ | Models
Fixed at 10 | Fixed at 10 |  Vary 0 to CorpusSize in steps of 30 |  CorpusSize $\div$ 30
Vary 5 to 50 in steps of 5 | Fixed at 10 | Fixed at CorpusSize | 10
Fixed at 10 | Vary 5 to 50 in steps of 5 | Fixed at CorpusSize | 10

Algorithm 2 Procedure to train PVA models

0: $H:=$ Collection of software project descriptions.$P:=$ Set of PVA
parameter variation scenarios shown in Table 9.

0: $M:=$ The set of software similarity detection models trained using $H$ and
$P$.

1: for all $\pi\in P$ do

2: $M(\pi):=trainPVAModel(H,\pi)$ {Uses gensim OSS library that implements
PVA}

3: Save $M(\pi)$ to disk

4: end for

####  A.1.1. Test-bed set up

To perform these experiments, we set up a test-bed $Y$ using the software
product descriptions collected from GitHub repositories. To perform the
validation testing of our software description similarity detection model, we
partition the software project descriptions data into $\langle train\
data,test\ data\rangle$ in ratio 2:1. $Y$ comprises of $\langle
same,different\rangle$ software product description pairs in equal proportion,
viz., 50:50.

####  A.1.2. Evaluation metrics

To evaluate the performance of our software description similarity detection
model, we compute the F1 score262626https://bit.ly/3mZ70Ns and ROC
area272727https://bit.ly/2Ev8BZP metrics defined as follows:

  1. (1)

F1 Score: It is defined as the harmonic mean of precision and recall.

(18) $\displaystyle F1\ score=\frac{2\times Precision\times
Recall}{Precision+Recall}$

where

(19) $\displaystyle Precision=\frac{TP}{TP+FP}$ (20) $\displaystyle
Recall=\frac{TP}{TP+FN}$

TP=True positive, FP = False positive, and FN = False negative.

  2. (2)

ROC area under the curve (AUC): It measures the quality of the output. ROC
curve is a plot that features the true positive rate (marked as Y-axis) vs.
the false positive rate (marked as X-axis) for an experiment.

Since the highest _ROC curve area_ value and the _F1 score_ value may differ
across the models, we take the average of the two as the final accuracy
measure of a model.

####  A.1.3. Objective:

The prime objective of this experiment is to address the research question
(RQ):

_What is the effect of PVA parameters in performing the software similarity
detection task?_

Some of the sub-RQs addressed under this are:

  1. (A)

_What is the highest accuracy achieved when detecting software similarity
using our models?_

  2. (B)

_What is the threshold similarity score between two PVA vectors for reliably
detecting two software products as similar?_

  3. (C)

_What is the effect of the parameters $\beta,\gamma,\psi$ on the performance
of the software description similarity detection model?_

  4. (D)

_What are the optimal values of $\beta,\gamma,\psi$ for training the software
description similarity detection model?_

Algorithm 3 Procedure to test the software description similarity detection
models trained using PVA

0: $Y:=$ Experimental test dataset containing $\langle same,different\rangle$
software product description pairs developed using the test-partition of the
project descriptions.$M:=$ The set of software description similarity models
trained using Algorithm 2.

0: $S_{same},S_{different}:=$ Similarity score collections for $\langle
same,different\rangle$ software product description pairs in $Y$.

1: $S_{same}=S_{different}=\phi$

2: for all project description pairs $\langle p_{i},p_{j}\rangle\in Y$ do

3: for all $\pi\in P$ do

4: $\phi_{i}=inferVector(p_{i},M(\pi))$ {Infer vector vectors using $M(\pi)$}

5: $\phi_{j}=inferVector(p_{j},M(\pi))$

6: $\alpha=computeCosineSimilarity({\phi}_{i},{\phi}_{j})$

7: if $p_{i}==p_{j}$ then

8: $S_{same}=S_{same}\cup\langle\alpha\rangle$

9: else

10: $S_{different}=S_{different}\cup\langle\alpha\rangle$

11: end if

12: end for

13: end for

####  A.1.4. Procedure

  1. (1)

Obtain the set of similarity scores ($S$) for the collection of test pairs
present in $Y$ using Algorithm 3.

  2. (2)

The $avg(S)$ value is stored as the threshold similarity score value
($\hat{\alpha}$). Note: the $\hat{\alpha}$ is computed separately for the
_same_ and _different_ software product description pairs. Table 10 lists the
threshold similarity score for the _same_ software product description pairs.

  3. (3)

For each parameter combination $\pi\in P$:

    1. (a)

Obtain the software description similarity detection model $\hat{M^{\pi}}$
such that, $\hat{M^{\pi}}$ achieves the highest evaluation scores among all
$M^{\pi}\in M$.

    2. (b)

Record the values of Accuracy scores and F1 scores obtained with
$\hat{M^{\pi}}$ when tested on $Y$, and the tuning parameter combination
($\pi$) used in developing $\hat{M^{\pi}}$.

Figure 10. Performance of software description similarity detection models with different parameter inputs Table 10. Similarity score stats for same software product description pairs Measure | $\alpha_{min}$ | $\alpha_{avg}$ | $\alpha_{stddev}$ | $\alpha_{max}$ | $\hat{\alpha}$
---|---|---|---|---|---
Value | 0.9999 | 0.9999 | 9.5305e-08 | 1 | 0.9999
Table 11. Best performance matrix (Models trained using 10 epochs, vector size 50, training samples 18260) Measure | $Accuracy$ | $Precision$ | $Recall$ | $F1\ score$ | $ROC\ area$
---|---|---|---|---|---
Value | 99.3% | 99.24% | 98.62% | 99.3% | 98.62%

####  A.1.5. Results and Observations

Fig. 10 shows the performance of software description similarity models (in
terms of F1 score and ROC area) with different input parameter combinations.
Some of the key observations are as follows:

  1. (1)

The evaluation metrics values remain almost constant on varying the epochs and
training dataset size but increase uniformly with the increase in vector size.

  2. (2)

The highest evaluation metrics values (viz., accuracy, F1 score, and ROC area)
are recorded corresponding to the $\gamma=50$. Thus, we select the fine-tuned
input parameters as $\gamma=50$ and $\beta=10$ for our further experiments.

  3. (3)

The highest accuracy, F1 score, and ROC area values achieved by our software
description similarity models are 99.3%, 99.3%, and 98.62%, respectively.

Table 11 lists the highest evaluation metrics’ values obtained by testing the
software description similarity model trained using optimally tuned parameters
$\beta=10$ and $\gamma=50$.

###  A.2. Details of the SDEE relational schema

A brief description of the tables of this schema is as follows:

  1. (1)

_release_info_ : This table stores the information about the releases of
various OSS repositories. The attributes _repo_ and _owner_ represent the
repository names and the respective repository owner names. The attributes
_release_no_ , _release_date_ and _size_ represent the release number,
deployment date and size of different releases of the repositories
respectively.

  2. (2)

_commit_stats_ : In this table, we store the developer-activity information
extracted corresponding to various commits of a repository. The attributes
_commit_id_ and _dev_id_ represents the unique IDentifiers (IDs) assigned to
various commits and developers, respectively.

For each of the commits stored in the table, we also store the development
_effort_ and time (_dev_time_) spent in generating the commit. To compute the
commit-wise effort and dev_time, we employ the use of _basic_ and _derived_
SDEE metrics, such as SLOC modified by the developer ($SLOC_{modifications}$),
developer productivity($P_{d_{i}}$), skill factor ($P_{d_{i}}$), etc.,
discussed in §3.1.

  3. (3)

_release_effort_estimate_ : This table stores release-level effort estimates
computed by utilizing the SDEE metrics’ values present in the commit_stats
table. The attributes _min_release_ids_ and _max_release_ids_ represent the
IDs of the considered releases corresponding to various {owner, repo} pairs of
repositories. Similarly, the attributes _start_release_date_ and
_end_release_date_ represent the considered release period of effort
computation, which corresponds to the release dates of the involved releases,
and _days_ represents the considered time-period interval expressed in the
form of days. The attribute _dev_count_ represents the total count of
developers contributing to the specific repository in the specified period.

  4. (4)

_soft_desc_pva_vec_ : This table stores the software project descriptions’ PVA
vectors and the respective cosine similarities with the reference vectors. The
“category” attribute represents the software category as specified on
MavenCentral.com.