∎

11institutetext: *Corresponding Author: Márcia Lima 22institutetext: Federal
University of Amazonas (UFAM) & Amazonas State University (UEA)
Tel.: +55 92 3305-1181
_Present address: Av. Rodrigo Otavio - 6200. Manaus-AM-Brazil_
22email: marcia.lima<EMAIL_ADDRESS>33institutetext:
Igor Steinmacher 44institutetext: Federal Technological University of Paraná
(UTFPR)
44email<EMAIL_ADDRESS>55institutetext: Denae Ford 66institutetext:
Microsoft Research
66email<EMAIL_ADDRESS>77institutetext: Evangeline Liu & Grace Vorreuter
88institutetext: GitHub
88email<EMAIL_ADDRESS>99institutetext: Tayana Conte & Bruno
Gadelha 1010institutetext: Federal University of Amazonas (UFAM)
1010email<EMAIL_ADDRESS>
# Looking for related discussions on GitHub Discussions

Márcia Lima Igor Steinmacher  Denae Ford  Evangeline Liu  Grace Vorreuter
Tayana Conte  Bruno Gadelha

(Received: date / Accepted: date)

###### Abstract

Software teams are increasingly adopting different tools and communication
channels to aid the software collaborative development model and coordinate
tasks. Among such resources, Programming Community-based Question Answering
(PCQA) forums have become widely used by developers. Such environments enable
developers to get and share technical information. Interested in supporting
the development and management of Open Source Software (OSS) projects, GitHub
announced _GitHub Discussions_ — a native forum to facilitate collaborative
discussions between users and members of communities hosted on the platform.
As GitHub Discussions resembles PCQA forums, it faces challenges similar to
those faced by such environments, which include the occurrence of related
discussions (duplicates or near-duplicated posts). While duplicate posts have
the same content—and may be exact copies—near-duplicates share similar topics
and information. Both can introduce noise to the platform and compromise
project knowledge sharing. In this paper, we address the problem of detecting
related posts in GitHub Discussions. To do so, we propose an approach based on
a Sentence-BERT pre-trained model: the RD-Detector. We evaluated RD-Detector
using data from different OSS communities. OSS maintainers and Software
Engineering (SE) researchers manually evaluated the RD-Detector results, which
achieved 75% to 100% in terms of precision. In addition, maintainers pointed
out practical applications of the approach, such as merging the discussions’
threads and making discussions as comments on one another. OSS maintainers can
benefit from RD-Detector to address the labor-intensive task of manually
detecting related discussions and answering the same question multiple times.

###### Keywords:

Communication tool GitHub Discussions forum Knowledge sharing OSS communities
Related discussions Sentence-BERT Software teams interaction

##  1 Introduction

Software engineering teams actively adopt social resources to support the
collaborative software development and coordinate team members’ tasks (Storey
et al.,, 2014, 2016; Tantisuwankul et al.,, 2019). Teams use such resources to
communicate, learn, answer questions, obtain and give feedback, show results,
manage, and coordinate activities (Storey et al.,, 2016). E-mail, chats, and
forums are examples of collaborative social media that support software teams’
communications (Storey et al.,, 2014, 2016; Pérez-Soler et al.,, 2018). In
recent years, Programming Community-based Question Answering (PCQA) has
increasingly attracted different users’ attention and has become widely used
by software developers (Wang et al.,, 2020; Pei et al.,, 2021; Ford et al.,,
2018). Developers rely on PCQA to quickly find answers for technical questions
(Mamykina et al.,, 2011; Zhang et al.,, 2017), impacting the software
development process (Yazdaninia et al.,, 2021).

Concerned with helping the development and management of OSS projects hosted
on GitHub, in 2020, the company announced GitHub Discussions (Liu,, 2022; Hata
et al.,, 2022). In order to be “A new way for software communities to
collaborate outside the codebase” (Niyogi,, 2020), the GitHub Discussions
forum provides opportunities for OSS communities to interact and discuss
project-related issues collaboratively. The Discussions forum is a place where
OSS communities can talk about work, ask questions, plan new releases, request
code reviews, make announcements, disclose information, recruit contributors,
get insights into the project, feature important information, or simply chat
(Hata et al.,, 2022; Liu,, 2022).

However, the PCQA type forums, like any other question and answering forums,
should be concerned about the quality decay of their contents. Silva et al.,
(2018) point out that repeated questions put at risk that quality. Prior
research handles this problem by focusing on automating the duplicate question
detection in PCQA forums (Zhang et al.,, 2015; Ahasanuzzaman et al.,, 2016;
Zhang et al.,, 2017; Silva et al.,, 2018; Wang et al.,, 2020; Pei et al.,,
2021). Research that aims to detect duplicate questions in Stack Overflow
highlights that duplicates can (1) pollute the platform with already-answered
questions (Silva et al.,, 2018); (2) consume the time of experts, as they must
manually analyze and look for duplicates (Zhang et al.,, 2017; Wang et al.,,
2020); and (3) make users wait unnecessarily for answers to questions that had
been already asked and answered (Ahasanuzzaman et al.,, 2016).

Although both GitHub Discussions and Stack Overflow resemble a PCQA forum,
they differ regarding the communicative intentions behind the posts. Posts on
GitHub Discussions cover a software project ecosystem (Hata et al.,, 2022) (on
GitHub, each project has its own forum). Stack Overflow questions cover a
broader context; they aim to answer developers’ technical questions regardless
of a specific software project. However, text evidence collected from GitHub
Discussions threads (discussion first posts and comments) shows that the
forum, like Stack Overflow, contains duplicate and related
posts111https://github.com/homebrew/discussions/discussions/1531222https://github.com/homebrew/discussions/discussions/707333https://github.com/vercel/next.js/discussions/22211.
Currently, maintainers manually manage duplicates on the GitHub Discussions
forum. Based on their previous knowledge, maintainers identify duplicates as
new discussions arise. However, manual strategies are less effective and are
subject to human subjectivity and imprecision (Ahasanuzzaman et al.,, 2016;
Zhang et al.,, 2017; Wang et al.,, 2020).

Different approaches have already been proposed and evaluated to solve the
problem of duplicates in PCQA forums. Such methods often use pre-labeled
datasets to train or optimize the duplicates detection process. However, the
posts on GitHub Discussions are project-related, and they are not previously
categorized using pre-defined topics. In addition, different project contexts
emerge every time an OSS community starts to use the Discussions forum.
Therefore, creating a manual labeled sample is costly and can always be
considered inefficient due to human imprecision, and small, given the
diversity of projects hosted on GitHub. Thence, we did not train or optimize
domain-specific machine learning models. The dynamic growth of the GitHub
Discussions and the diversity of projects hosted on the platform create
opportunities and challenges to develop an approach to detect related
discussions on GitHub Discussions.

Therefore, motivated by the context mentioned, this work aims to propose the
RD-Detector, an automated approach to detect related discussions in the GitHub
Discussions forum. More specifically, related discussions are those duplicated
or near-duplicated ones. Duplicate posts have the same content and could be
exact copies. Meanwhile, near-duplicates have similar topics and share similar
information. In this work, we consider duplicates a subtype of related
discussions for practical purposes. From now on, we use the term ’related
discussions’ to refer to duplicate and near-duplicate discussion posts. More
specifically, our research question (RQ) is:

RQ: Are general-purpose deep machine learning models to Natural Language
Processing (NLP) problems effective in detecting related discussions in OSS
communities?

To do so, we propose the RD-Detector approach. RD-Detector uses a Sentence-
BERT (SBERT) pre-trained general-purpose model to create sentence embedding
representations of discussion posts and compute the semantic similarity
between pairs of discussions. The approach outputs a set of related discussion
candidates. The RD-Detector differs from the previous approaches as (1) it
does not rely on pre-labeled datasets; (2) it does not rely on specific models
to understand OSS communities’ discussions context; (3) it can assess
different software contexts hosted on GitHub; and (4) it aims for precision in
detecting related discussion posts. OSS maintainers and Software Engineering
(SE) researchers evaluated the RD-Detector results. They both classified the
pairs of related discussion candidates as related or not and pointed to
evidence on the challenges and practical applications in detecting related
discussion posts in the GitHub Discussions.

Our results show that we can use a general-purpose machine learning model for
detecting related discussions in OSS communities. We assessed RD-Detector over
different datasets, achieving 75% to 100% in terms of precision. We identified
the imprecision of the term ‘related discussions’ as a limitation during the
evaluation. The general-purpose machine learning model brought flexibility to
the approach. As an advantage, we can point out that the results does not
restrict to a specific software project context. OSS maintainers can benefit
from the results to minimize the work overhead in manually detecting related
threads and the rework in answering the same question multiple times. Besides,
the results support OSS maintainers to tackle the platform pollution and to
address the project knowledge sharing degradation, as it occurs in different
discussions threads.

The main contributions of this research include:

  * •

The RD-Detector, an approach based on deep machine learning models to detect
related posts in GitHub Discussions.

  * •

Empirical evidence on using a general-purpose machine learning model to detect
related discussions held by OSS communities.

  * •

Empirical evidence regarding the RD-Detector practical applications in OSS
communities, from OSS maintainers’ perspective.

##  2 Background

OSS projects are highly distributed environments usually composed of self-
directed development teams (Chen et al.,, 2013; Tantisuwankul et al.,, 2019).
Therefore, knowledge sharing is a critical factor for the success of OSS teams
(Chen et al.,, 2013; Tantisuwankul et al.,, 2019). Chen et al., (2013) point
out that ‘knowledge sharing is an interactive cuing process in which knowledge
provided by one team member becomes the cue for other members to retrieve
relevant but different knowledge stored in their own memory.’ Given that the
GitHub Discussions is a collaborative communication channel where OSS
communities make questions, debate, and announce project-related issues, we
conjecture that the Discussions is an online environment that promotes
knowledge sharing in OSS communities. Tantisuwankul et al., (2019) point out
that the key to software projects hosted on collaborative platforms such as
GitHub success are the communities’ interaction and the project knowledge
sharing.

This section presents the main concepts used in this research and previous
works on detecting duplicates in PCQA forums. Section 2.1 describes the GitHub
Discussions forum (our object of study). Section 2.2 presents the concept of
duplicates from the researchers’ perspective and how they tackle duplicates or
related posts in programming Q&A forums.

###  2.1 GitHub Discussions

GitHub Discussions is a feature to any public or private repository on GitHub
(Liu,, 2022). It facilitates collaborative discussions among maintainers and
the community for a project on GitHub (Liu,, 2022). GitHub company suggests
using the Discussions forum to ask and answer questions, share information,
make announcements, and lead or participate in project-related conversations
(GitHub, 2021a, ). The Discussions is a collaborative communication forum for
OSS maintainers, code and non-code contributors, newcomers, and users to
discuss projects’ use, development, and updates in a single place without
third-party tools (GitHub,, 2022; GitHub, 2021a, ). In addition, the
Discussions stands out for being a place to distinguish day-to-day
conversations and conversations aimed at engineering teams (Issues or Pull
Request) (Hata et al.,, 2022).

Users, maintainers, contributors, and newcomers can join in a conversation by
creating, commenting, reacting, or reading a discussion post (GitHub, 2021a,
). The Discussions users can also search selected topics in discussion posts
(GitHub, 2021c, ). To do so, they must specify keywords in the GitHub search
engine. Users can restrict the search results to the discussion title, body
text, or comments by applying correct qualifiers.

Authors must specify the discussion title, body text, and category to create a
new discussion (Figure 1). The category is a mandatory attribute of a
discussion post. It helps organize conversations into predefined classes,
allowing community members to chat in the right place and find discussions
with similar characteristics (GitHub, 2021b, ). Authorized members can define,
create, or delete categories in a repository according to the project needs.
By default, the GitHub Discussions forum provides five types of categories:
Announcements, Q&A, Ideas, Show and tell, and General (GitHub, 2021b, ).
Maintainers can create Announcements discussions to share project updates and
news. Users can create Q&A discussions to ask questions, suggest answers, and
vote on the most appropriate feedback. Ideas discussions can report or share
ideas regarding the project improvements. Show and tell discussions discuss
relevant creations, experiments, or tests. Finally, General discussions
address any issue relevant to the project (GitHub, 2021b, ).

Maintainers report a positive acceptance of GitHub Discussions in the OSS
communities (Liu,, 2021). They highlight that the forum enables the growth of
the communities in the same place they use to collaborate, improve, and
increase the OSS community members’ engagement, and separate the issues
trackers from questions, feature requests, and general chatting (Liu,, 2021).
In addition, maintainers point out they can use the discussions’ threads to
access historical data (Liu,, 2021). They can keep track of the questions
already asked, their proposed solutions, and the suggestions made. With this
threading support, they can individually address demands without losing them
in broader discussions.

Liu, (2022) highlights the use of GitHub Discussions by five selected OSS
communities: Dogecoin,444https://github.com/dogecoin/dogecoin/discussions
NASA,555https://github.com/nasa/cFS/discussions
Next.js,666https://github.com/vercel/next.js/discussions
Pixar,777https://github.com/PixarAnimationStudios/OpenTimelineIO/discussions
and React.888https://github.com/reactwg/react-18/discussions The Dogecoin
community uses GitHub Discussions to centralize its developers’ conversations
in one place, NASA uses it to collaborate with the OSS community on its core
flight systems. The Next.js community uses Discussions to plan features,
promote community interaction, and exchange ideas about the project. The Pixar
project uses the forum to engage the OSS community responsible for creating
its OpenTimelineIO API. Finally, React started using discussions to introduce
a new release and give collaborators a place to ask questions about it.

This work uses public discussion posts collected from three OSS communities to
evaluate our approach. Our dataset comprises the discussion posts of Gatsby,
Homebrew, and Next.js communities (Table 1).

Figure 1: An example of a discussion post.

###  2.2 Related Work

Several studies aim to detect duplicates on PCQA forums by proposing different
approaches for duplicate question detection (Zhang et al.,, 2015;
Ahasanuzzaman et al.,, 2016; Zhang et al.,, 2017; Silva et al.,, 2018; Wang et
al.,, 2020; Pei et al.,, 2021). Generally, researchers model the duplicate
detection task as a ranking problem, a binary classification problem, or a
combination of both. Ranking problems combine features to sort the top-k most
similar documents (discussions, texts, questions) (Liu,, 2011) and predict
duplicates. Classification problems classify pairs of documents into
predefined categories (e.g., duplicates or non-duplicates) (Ahasanuzzaman et
al.,, 2016; Zhang et al.,, 2017).

The literature characterize duplicate posts on PCQA forums as: (1) questions
that address the same topic, but are not necessarily identical copies (Silva
et al.,, 2018; Wang et al.,, 2020); (2) questions conceptually equivalent to
other questions previously posted (Abric et al.,, 2019); (3) questions that
were already asked and answered before (Zhang et al.,, 2017; Wang et al.,,
2020); (4) questions asked to solve the same problem (Ahasanuzzaman et al.,,
2016); and (5) questions that ‘express the same point’ (Zhang et al.,, 2015).
We can note the duplicates conceptualization is not a rigid definition; it can
depend on human subjective criteria. Ahasanuzzaman et al., (2016) point out
that exact duplicates are significantly rarer, and many users duplicate
questions by asking the same thing in different ways.

Zhang et al., (2015) proposed DupPredictor, an approach to detect potential
duplicate questions in Stack Overflow. The DupPredictor combines the
similarity scores of different features. The authors evaluated the approach’s
effectiveness using a pre-labeled Stack Overflow dataset, which achieved 63.8%
of recall rate. Ahasanuzzaman et al., (2016) and Zhang et al., (2017)
addressed duplicate detection as a supervised classification problem. The
authors used a pre-labeled dataset to train and validate a classifier to
detect duplicates in Stack Overflow and measured the proposed approachs’
effectiveness using the recall rate. Silva et al., (2018) implemented the
DupPredictorRep and DupeRep approaches. The authors adopted an official Stack
Overflow dataset dump to train, optimize, and evaluate the approaches that
predict and classify questions as duplicate or not, respectively. Wang et al.,
(2020) used deep learning techniques to detect duplicate questions in Stack
Overflow. The authors proposed three different approaches based on
Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Long
Short-Term Memory (LSTM) to identify duplicates. The authors also used a pre-
labeled Stack Overflow dataset to train, evaluate, and test the classifiers
and measured the approaches’ effectiveness using the recall rate. Finally, Pei
et al., (2021) proposed an Attention-based Sentence pair Interaction Model
(ASIM) to predict the relationship between Stack Overflow questions. The
authors used a Stack Overflow dump to train the software engineering specif
domain. The proposed model achieved 82.10% precision and 82.28% recall rates,
outperforming the baseline used.

Conversely, Abric et al., (2019) analyzed how helpful duplicate questions are
to the software development community. The authors argue that duplicates on
Stack Overflow help the developer community by providing different
formulations of the same problem or solution. Also, the additional answers may
be more understandable for some users.

Previous work has also highlighted the problem of duplicates in issue reports
and pull requests (PRs) on GitHub (Li et al.,, 2017; Yu et al.,, 2018; Li et
al.,, 2018; Ren et al.,, 2019; Li et al.,, 2020; Zhang et al.,, 2020). Li et
al., (2018) analyzed explicit links in both issues reports and PRs. They
reported the importance of such links in identifying duplicates. The authors
analyzed 70,686 links that represented duplication relationships, from which
59.03% identified duplicate issues and 40.97% identified duplicate pull
requests. Li et al., (2020) presented empirical evidence on the impact of
duplicate PRs on the software development effort. They observed that the
’inappropriateness of OSS contributors’ work patterns and the drawbacks of
their collaboration environment would’ result in duplicates. In addition,
researchers have been proposing different approaches to address duplicates in
GitHub (Li et al.,, 2017; Yu et al.,, 2018; Ren et al.,, 2019; Zhang et al.,,
2020). Li et al., (2017) and Ren et al., (2019) proposed automatic approaches
based on traditional Information Retrieval (IR) and NLP techniques to detect
duplicates in pull requests. Yu et al., (2018) constructed a dataset of
historical duplicate PRs extracted from projects on GitHub - the DupPR dataset
- by using a semi-automatic approach. Zhang et al., (2020) proposed the
iLinker, an approach to detect related issues in GitHub. The authors trained
iLinker to learn the embedding corpus and models from the project issue text.

The mentioned approaches to detect duplicates on PCQA forums have two common
points: the use pre-labeled datasets to learn or better understand the domain
application, and their effectiveness is measured using the Recall rate.
However, this research faces some challenges that make it different from the
previously mentioned papers: (1) there is not a pre-labeled dataset of
duplicated or related discussions, extracted from GitHub Discussions, through
which we can train or optimize machine learning models; (2) creating a pre-
labeled base of related discussions is a time-consuming task and can always be
considered inefficient and small; and (3) there is a vast diversity of
repositories hosted on GitHub, which provide different software project
contexts (in January 2020 the platform already hosted more than 200 million
repositories999https://github.com). Different from the mentioned strategies to
detect duplicates in issue reports and pull requests, our approach is based on
general-purpose deep learning models to address the topic duplication problem
in Discussions. Therefore, in order to detect related discussion candidates
(duplicate or near-duplicate) in the GitHub Discussions forum, our proposed
approach (1) does not rely on a pre-labeled dataset; (2) does not rely on
domain-specific models; (3) applies to different OSS project domains; (4) is
based on a Sentence-BERT pretrained model, and (4) aims at improving precision
rates rather than the recall rates. In Section 3, we propose a method to
detect related discussions in GitHub Discussions.

##  3  RD-Detector: Related Discussions Detector

In this section, we present the RD-Detector approach. The RD-Detector aims to
detect pairs of related discussion candidates in collaborative discussion
forums of OSS communities. To this end, the RD-Detector scores the Semantic
Textual Similarity (STS) between discussion posts to find discussions with
similar meanings. Using the STS value, we can detect exact copies (duplicates)
and discussions that share the same topic (near-duplicates). The greater the
similarity values, the greater the chances of duplicates. We consider any
duplicated or near-duplicated discussions as ‘related discussions.’ The RD-
Detector’s output is a set of related discussion candidates, which, from now
on, we denote $R$.

Figure 2 shows the overall process conducted to detect the sets of related
discussion candidates. The process comprises three phases: Data collection
(Figure 2 \- A), Preprocessing (Figure 2 \- B), and Relatedness Checker
(Figure 2 \- C). We describe the phases of the proposed approach as follows.

Figure 2: The RD-Detector approach

###  3.1 Data collection

In the “Data collection” phase, we focus on gathering data from OSS
communities. RD-Detector was assessed using data collected from public
discussions threads held in three GitHub Discussions forums (Table 1).

Our data collection comprises three steps (Figure 2 \- A). In the first step,
we select the OSS repositories of interest. Next, we collect the discussions
threads of the selected repositories. Finally, we extract some preselected
attributes from each discussion thread. The RD-Detector considers the title
and the body of the discussion posts to calculate the semantic text similarity
values between pairs of discussion posts. We do not consider comments because
they are feedback or design reasoning about the discussions’ main topics.
During the data collection phase, the RD-Detector extracts the author, date,
and category type of the discussion posts. The RD-Detector uses the attributes
to select the discussion posts and create sub-datasets according to predefined
filters (Section 4.2).

###  3.2 Preprocessing

Applying text similarity algorithms in discussion content requires data
preprocessing to optimize the algorithm execution (Zhou et al.,, 2017). We
executed a preprocessing pipeline to remove noisy information and format the
dataset collection (Figure 2 \- B). The preprocessing phase comprises five
steps. In the “Data selection” step, the RD-Detector selects the title and the
body text of the discussion’ posts. In addition, in “Data selection” we can
filter discussion posts according to the values of their attributes. We can
select discussion posts that fulfill certain conditions, such as category type
(using the category attribute).

The following two preprocessing steps, “Code snippets and URL removal” and
“Punctuation, emoji, numbers, and stopwords removal,” aim to remove noise data
from the content of the discussions. We remove the code snippets since the
machine learning model used is a general-purpose model trained to handle NLP
problems. Furthermore, the structure and vocabulary of natural language texts
differ from machine codes (Sirres et al.,, 2018). We use HTML tags, e.g.
<code>, to remove code snippets and URLs.

The “Punctuation, emoji, numbers, and stopwords removal” step removes
punctuations, emojis, numbers, and stopwords from discussions content. To this
end, we use the Python Natural Language Tool Kit (NLTK) library
101010https://www.nltk.org/. We do not remove the following punctuation
symbols ‘.’ and ‘_’ because they concatenate words, e.g., ‘Next.js’ and
‘version_1.3’. In the next preprocessing step, we convert the remaining text
to lowercase and apply lemmatization (Zhang et al.,, 2017). Finally, we
eliminate zero-length discussion posts in the “Empty posts removal” step.

###  3.3 Relatedness Checker

The “Relatedness Checker” phase is the core of the RD-Detector (Figure 2 \-
C). In this phase, the approach computes the similarity score of discussions’
pairs and detects the related discussion candidates. The preprocessed data is
the input of this phase (Algorithm 1, lines 0-3). The set R, containing pairs
of related discussion candidates, is the output of the “Relatedness Checker”
phase (Algorithm 1, line 30). This phase comprises two steps: (1) the
similarity measurement (Similarity checker) and (2) the Selection of related
discussion candidates. The steps are as follows.

####  3.3.1 Similarity checker

We use a semantic text similarity (STS) checker (Agirre et al.,, 2015) to
score the similarity between pairs of discussion posts. The similarity checker
measures the similarity score using a general-purpose Sentence-BERT pre-
trained model (Reimers and Gurevych,, 2019).

The RD-Detector uses the public all-mpnet-
base-v2111111https://huggingface.co/sentence-transformers/all-mpnet-base-v2
model to compute semantically significance sentence embeddings of each
discussion post. The model maps sentences and paragraphs to a 768-dimension
dense vector space. At the time of this work execution, the all-mpnet-base-v2
model provided the best quality in sentences embeddings computing and semantic
searching performance (Reimers,, 2021).

We compare the sentences embeddings using the cosine-similarity score. We used
the score to set the similarity value between pairs of discussions and to rank
sentences with similar meaning ($similarity(master,target)$). The cosine
similarity value outcome is bounded in [0,1] (Abric et al.,, 2019). Cosine
similarity value 1 refers to identical discussion contents, value 0 refers to
dissimilar discussions (Abric et al.,, 2019). There is no hierarchy or
priority relationship between pairs of masters and targets discussions.
Masters discussions are the oldest post, and targets ones are the most recent.
Therefore, the similarity value for ($master_{i}$,$target_{j}$) is equal to
the similarity value for ($target_{j}$,$master_{i}$).

Researchers train and optimize custom-built machine learning models using
labeled dataset samples. We propose an approach based on a general-purpose
machine learning model to address the lack of a pre-labeled dataset and
generalize the outcomes to different OSS communities. One can benefit from
general-purpose models to detect related discussions on the GitHub Discussions
forum because of the lack of an official dump of related posts to train or
optimize automatic approaches and the significant number of OSS projects
ecosystems hosted on the platform. General-purpose models application is not
restricted to a single context, as they are trained and optimized in different
datasets. Specifically, the all-smpnet-base-v2’s base model is the
microsoft/mpnet-base model. Experts optimized the all-smpnet-base-v2 model
over 1 billion sentence pairs collected from various datasets (Reddit comments
(2015-2018), WikiAnswers Duplicate question pairs, Stack Exchange Duplicate
questions (titles+bodies), etc.) (Face,, 2021). Context-specific models are
trained and optimized according to pre-labeled samples that summarize context
specifics, not guaranteeing effectiveness when used outside the training
context. In addition, general-purpose models usage provide advantages such as
(1) no need for local complex computational structures for training,
validation, and testing of models, (2) no need for model parametrization, and
(3) no need for model retraining, revalidation, and retest whenever the
context change (Polyzotis et al.,, 2017; Zhou et al.,, 2017; Schelter et al.,,
2018; Lee and Shin,, 2020). The dynamism with which OSS communities grow
justifies general-purpose machine learning model usage. Indeed, public machine
learning models bring flexibility to the RD-Detector approach. As experts
release new exchangeable machine learning models, one can update the RD-
Detector.

Therefore, for each pair of a $(master,target)$ discussion, the RD-Detector
computes the semantic similarity value that captures the relatedness of the
posts, Algorithm 1 \- lines 4-13. The similarity values are the input to the
“Selection of related discussion candidates” step.

1

Input: Set of discussion posts $D$

Input: Set of discussion attributes $\Delta$

Input: K value

Input: Sentence-BERT model

2

3foreach  _discussion $d_{i}\in$ D’_ do

4 $D^{\prime}\leftarrow D^{\prime}+preprocessing(d_{i})$;

5

6 end foreach

7

8/* Similarity Checker */

9 foreach  _discussion $d_{i}\in D^{\prime}$_ do

10 foreach  _discussion $d_{j},(d_{j}\neq d_{i})\in D^{\prime}$_ do

11 $master_{i}\leftarrow d_{i}$;

12 $target_{j}\leftarrow d_{j}$;

13 $similarity\\_value_{i\\_j}\leftarrow similarity(master_{i},target_{j})$ /*
Using model */ ;

14 $tuple\leftarrow(master_{i},target_{j},similarity\\_value_{i\\_j})$;

15 $save(tuple,similarity\\_values\\_file)$;

16

17 end foreach

18

19 end foreach

20/* Threshold definition */

21 $S\leftarrow\\{\\}$ ;

22 foreach  _discussion $d_{i}\in D^{\prime}$_ do

23 $master_{i}\leftarrow d_{i}$;

24 $S=S+topK\\_sim\\_values(master_{i},K)$;

25

26 end foreach

27 $Q1\leftarrow 25th\\_percentile(S)$;

28 $Q2\leftarrow 50th\\_percentile(S)$;

29 $Q3\leftarrow 75th\\_percentile(S)$;

30 $IQR\leftarrow Q3-Q1$;

31 $T_{related}=Q3+(1.5*IQR)$;

32 /* Selection of related discussion candidates */

33 $R\leftarrow\\{\\}$ ;

34 foreach  _$master_{i},target_{j},similarity\\_value_{i\\_j}\in
similarity\\_values\\_file$_ do

35

36 if  _$(similarity\\_value_{i\\_j}\geq T_{related})$_ then $R\leftarrow
R+(master_{i},target_{j})$;

37

38 end foreach

39return $R$ ;

Algorithm 1 Identifying Related Discussions

####  3.3.2 Selection of related discussion candidates

We rank the related discussion candidates according to their similarity
values. We do not use a pre-determined threshold, $x$ to filter the related
discussions. Instead, the RD-Detector computes what we call ‘local threshold,’
denoted by $T_{related}$ based on each project dataset. The $T_{related}$ is
defined using descriptive statistics that describe the characteristics of a
specific similarity value set. The local threshold use allows the adaptation
of the proposed approach to different OSS communities’ contexts. We set the
$T_{related}$ value following four steps:

  1. 1.

Defining the $K$ value: The $K$ value delimits the search bounds for related
discussion candidates. The RD-Detector uses the $K$ value to select the
similarity values of the top-K most similar discussions to each post in the
input dataset.The greater the value of $K$, the greater is the number of
similarity values selected and the search bounds. Setting $K=5$, the approach
uses the similarity values of the top-5 most similar discussions to every post
in the dataset. Setting $K=10$, the RD-Detector selects the similarity values
of top-10 most similar discussion posts. One can specify different $K$ values.
$K$ is an input value (Algorithm 1).

  2. 2.

Creating the distribution $S$: The $S$ distribution is a collection of
similarity values. $S$ contains the similarity values of the $K$ most similar
target discussions for each discussion post in the dataset (Algorithm 1, lines
14- 19).

Let $n$ be the number of discussion posts in the dataset, $K$ the number of
the most similar target discussions to every discussion in the dataset, and
$value\\_ij$ the similarity value of a given $master_{i}$ and $target_{j}$
discussion pair, the distribution $S$ is:

$S=<value_{1\\_1},value_{1\\_2},...,value_{1\\_K},value_{2\\_1},$

$value_{2\\_2},...,value_{2\\_K},...,value_{n\\_1},value_{n\\_2},...,value_{n\\_K}>$

  3. 3.

Determining the descriptive statistics of $S$: We use descriptive statistics
variability measures to understand how dispersed the distribution is. To this
end, we calculate the interquartile range ($IQR$), along with the 25th
percentile ($Q1$), the 50th percentile ($Q2$), and the 75th percentile ($Q3$),
Algorithm 1 \- lines 20 to 23. Next, we find the Upper Inner Fence value
(Equation 1) that identifies the outliers in $S$ (Tukey et al.,, 1977).

$\textit{Upper inner fence}=Q3+(1.5*IQR)$ (1)

  4. 4.

Setting the local threshold ($T_{related}$): Because we assume that the
greater the semantic similarity value of a pair of discussion posts, the
greater the chances they are related discussion, we set the local threshold to
the upper inner fence value (Algorithm 1 \- line 24). Therefore, we have that:

$T_{related}=\textit{Upper inner fence}$ (2)

The $K$ value defines the $S$ distribution size. Consequently, it changes the
coefficients $Q1$, $Q2$, $Q3$, and $IQR$ values that summarize $S$. As a
result, it also causes changes in the local threshold value, $T_{related}$.
Since $T_{related}$ is directly influenced by $S$, we call $T_{related}$ as
‘local threshold’.

After setting the local threshold, the RD-Detector detects the pairs of
related discussion candidates. Related discussion candidates are those
discussion pairs that similarity values are equal to or greater than the local
threshold. RD-Detector creates $R$, the set of related discussion candidates
(Algorithm 1, lines 26-30). We consider related discussions those identified
as outliers in the $S$ distribution. Calefato et al., (2021) also use
descriptive statistics to identify core OSS developers’ inactivity periods.

We measured the precision of the RD-Detector based on the assessment of OSS
maintainers and SE researchers who manually evaluated the $R$ set. Since we do
not know the number of related posts or the truly related discussion posts in
the GitHub Discussions forum, we focus on increasing the RD-Detector precision
rather than its recall. Higher precision values ensure greater assertiveness
in detecting true positives.

The precision rate refers to the number of true positives divided by the
approaches’ total number of positive predictions. In this work, evaluators
judged the RD-Detector predictions and highlighted the true positives ones. We
present the overall process of the RD-Detector evaluation in Section 4.4.

Let $N$ be the set of true-positive related discussions, and $R$ be the set of
predicted related discussions made by RD-Detector. The precision rate
measurement is as follows (Kim et al.,, 2005):

$Precision=\frac{|N\cap R|}{|R|}$ (3)

The precision value ranges from 0% to 100% (Kim et al.,, 2005). A precision of
100% means that all identified related discussion candidates are indeed
related discussion posts.

The RD-Detector emerges as a tool that OSS maintainers can use to minimize the
work overhead in manually detecting related threads and reduce the rework in
answering the same question multiple times. The periodic execution of RD-
Detector supports maintainers to deal with related discussions propagation in
OSS communities.

##  4 Assessing RD-Detector over the GitHub Discussions forum

To assess RD-Detector, we collected and used pairs of discussion posts of
three selected OSS projects hosted on the GitHub platform (Table 1). RD-
Detector generated sets of related discussion candidates for each project.
Maintainers of the selected OSS projects and SE researchers evaluated the RD-
Detector outcomes.

###  4.1 Data collection - GitHub Discussions

To perform the data collection, we wrote a customized web crawler that
searches for discussion posts on the GitHub platform and collects all public
discussion posts of a given OSS community. To do so, we used the Python
BeautifulSoup library 121212https://pypi.org/project/beautifulsoup4/. We
stored the collected data in a local dataset. The web crawler input is the
project URL, and its outputs comprise of (1) a list with the discussions’ IDs,
categories, authors, dates, and title; and (2) the HTML files of the
discussion threads (discussion first post and comments).

We collected data from Gatsby ($D_{p=Gatsby}$), Homebrew ($D_{p=Homebrew}$),
and Next.js ($D_{p=Next.js}$) repositories (Table 1). We selected these three
repositories based on (1) the usage of the Discussions forum, (2) the GitHub
professional team members’ demands (coauthors in this research), and (3) OSS
maintainers’ availability. We executed multiples data collection runs between
October 2021 and November 2021. In total, we collected 11,309 discussions.

Table 1: Repositories used in the dataset Repository | #Discussions | Download date
---|---|---
Gatsby 131313https://github.com/gatsbyjs/gatsby/discussions/ | 1,276 | 11-10-2021
Homebrew141414https://github.com/homebrew/discussions/discussions/ | 1,464 | 10-09-2021
Next.js151515https://github.com/vercel/Next.js/discussions/ | 8,569 | 11-18-2021
sum | 11,309 |

####  4.1.1 Dataset characterization

According to the Gatsby community documentation (Community,, 2022), ‘Gatsby is
a free and open source framework based on React that helps developers build
blazing fast websites and apps.’. We call $D_{Gatsby}$ the set of discussion
posts collected from Gatsby’s repository. On the date we performed the data
collection, we collected 1,276 discussion posts from Gatsby,
$|D_{p=Gatsby}|=1,276$ (Table 1). Figure 3-(a) shows the frequency
distribution of discussions created in the Gatsby community over 21 months
(01-2020 to 09-2021). According to this time window, 01-2020 to 09-2021, the
average growth rate of forum usage by the Gatsby community was 22%
(considering the frequency of new discussions).Gatsby makes available the
following categories types community, help, ideas-feature-requests, RFC, and
umbrella-discussions. Help discussions are most common, totaling 73.35%,
followed by ideas-fearute-requests, umbrella-discussions, community, and RFC,
totaling 21.23%, 2.50%, 1.33%, 1.56% of discussion posts, respectively (Figure
3-(b)).

Homebrew is an OSS project that makes it easy to ‘install the UNIX tools Apple
didn’t include with macOS. It can also install software not packaged for your
Linux distribution to your home directory without requiring sudo.’(Project,,
2022). We call $D_{Homebrew}$ the set of discussion posts collected from
Homebrew’s repository. We collected 1,464 discussion posts from Homebrew,
$|D_{p=Homebrew}|=1,464$.

Figure 3-(c) shows the frequency distribution of discussion posts created in
Homebrew Discussions forum over 14 months (09-2020 to 10-2021). The oldest
discussion post collected from Homebrew dates from September 2020. According
to data in $D_{Homebrew}$, the average growth rate of Discussions forum usage
in Homebrew between 09-2020 and 10-2021 was 25.12% (considering the frequency
of new discussions). The months with the highest frequency of new discussions
were December/20, January, and February 2021, totaling 148, 151, and 195 new
discussions, respectively. Such values indicate the peak in the frequency
distribution graph (Figure 3 (c)). Unlike Gatsby and Next.js projects,
Homebrew organizes its discussion posts according to the problem types and not
by questions type. Homebrew provides the following categories: casks, getting-
started, tap-maintenance-and-brew-development, everyday-usage, linux, and
writing-
formulae-casks. The everyday-usage discussions are the most common in the
$D_{Homebrew}$ dataset, totaling 54.87%. Followed by getting-started, casks,
tap- maintenance-and-brew-development, linux, and writing-formulae-casks,
totaling 17.27%, 10.85%, 8.84%, 5.80% and 2.34% of discussions, respectively
(Figure 3 (d)).

Next.js is an OSS project that ‘provides a solution to build a complete web
application with React’(Vercel,, 2022). The Next.js community has stood apart
in supporting the launch of GitHub Discussions since the start (Liu,, 2022).
In January 2022, the project forum already had almost 400 pages of discussion
threads (Liu,, 2022). Compared to Gatsby and Homebrew, Next.js is the project
with the highest number of discussions analyzed, characterizing an active OSS
community. We collected 8,569 public discussion posts from Next.js. Figure 3
\- (e) shows the frequency distribution of new discussions over 22 months
(01-2020 to 10-2021). In this period, the average growth rate of the forum in
the Next.js community was 44.62%(considering the number of new discussions
created). The Next.js project has the highest average growth rate among the
three analyzed projects. The peaks of the distribution indicate the months
with the highest number of discussions created, which were May, June, and
October 2020 with 589, 544, and 562, respectively (Figure 3 \- (e)). When we
collected the dataset, the discussion categories were ideas, help, react-
server-components, and show-and-tell. Help discussions are the most frequent
in the Next.js dataset, totaling 87.01% of the discussion posts. Followed by
ideas, show-and-tell, and react-server-components, totaling 11.84%, 1.03% and
0.10% of discussions, respectively (Figure 3-(f)).

Finally, all discussion posts collected composes the dataset $D$ used in this
work. Therefore, $D=D_{p=Gatsby}\cup D_{p=Homebrew}\cup D_{p=Next.js}$.

Figure 3: Dataset characterization - Gatsby, Homebrew, and Next.js

###  4.2 Preprocessing phase applied to Discussions Dataset

We preprocessed the dataset $D$. First, we split the dataset into subsets
using predefined filters. We considered two filter types: project and category
filters. We used the project filter, $p$, to select discussions from a
specific OSS repository by defining the project’s name. We used the category
filter, $c$, to select discussion posts that matched predefined categories’
types.

We set both filter values according to the dataset characterization. One can
also combine filters to create different sub-datasets, e.g., we can select
discussions of type $\beta$ belonging to project $\alpha$. The category filter
values depend on the types of categories provided by the project. We express
the use of both filters by $p=\alpha|c=\beta$, where $p$ identifies to the
project name (ex.: Gatsby, Homebrew, and Next.js ) and $c$ identifies the
discussion category type.

We standardized the category labels to report our results, making them easier
to describe and follow. We use the following labels: Q&A, Ideas, and ALL. The
Q&A label refers to ‘Question and Answers’ (help) discussions types. The Ideas
category to refer to ideas or ideas-feature-requests categories. In addition,
we defined a special label, ALL, used to refer to all discussion posts,
regardless of their original type. As described in Section 4.1.1, the Homebrew
project discussion organization differs from the other projects. Due to this
fact, we chose not to filter Homebrew discussions according to the category
type. Therefore, we report the results of the Homebrew project by setting
$c=ALL$.

Besides, on the this step, RD-detector selected the discussions’ title and
body text. We derived seven sub-datasets by applying different filter
configurations to the original dataset $D$ (Table 2, column 3). We use
$D_{p=Gatsby|c=Q\&A}$ to designate the subset of Q&A discussion collected from
the Gatsby project, $D_{p=Gatsby|c=Ideas}$ to denote the subset of ideas-
feature-requests discussions collected from the Gatsby project, and so on. We
assessed the proposed approach (RD-Detector) over the seven sub-datasets.

Next, the sub-datasets went through cleaning, denoising, and formatting steps.
The last preprocessing step ignores zero-length discussions. In total, the
approach discarded 3, 17, and 127 discussions from Gatsby, Homebrew, and
Next.js projects, respectively.

###  4.3 Relatedness Checker applied to GitHub Discussions

After the preprocessing phase, the RD-Detector calculated the semantic
similarity values for each pair of discussions (as presented on Section
3.3.1), using the seven sub-datasets presented in Table 2, column 3. As
discussed in Section 3.3.2, the value of $K$ impacts the set of related
discussion candidates $R$. The RD-Detector uses the $K$ value to select the
similarity values of the top-K most similar discussions to each post in the
dataset. In order to evaluate the RD-Detector effectiveness, we set the $K$
value to 5 and 10. For each $D_{p=\alpha|c=\beta}$ sub-dataset presented in
Table 2, we configured the approach to run over $K=5$ and $K=10$. This way, we
evaluate the approach’s effectiveness by considering the five and the ten most
similar discussions to every discussion in the dataset. Values greater than 10
expand the search bounds for related discussions. We did not consider $K$
values greater than ten because the proposed approach aims at increasing the
precision rates rather than recall rates.

In total, we assessed the approach over 14 different configuration groups
(Table 2, column 4). For each configuration group, RD-Detector computed the
local threshold value, $T_{related}$, and detected the set of related
discussion candidates, $R$. We denote the set of related discussion candidates
detected considering the configuration group $p=Gatsby$, $c=Q\&A$, and $K=5$
by $R_{p=Gatsby|c=Q\&A|K=5}$. $R_{p=Gatsby|c=Q\&A|K=10}$ denotes the set of
related discussion candidates detected considering the configurations
$p=Gatsby|c=Q\&A|K=10$. One can use the same reasoning for the other 12
related discussion sets presented in Table 2, column 4. We used the
configuration groups to evaluate the RD-Detector effectiveness and report our
results.

Table 2: Configuration groups: projects, categories, and K-values | Project
---
| Category
---
Dataset |  | Set of related discussion
---
candidates
Gatsby | Q&A | $D_{p=Gatsby|c=Q\&A}$ |  | $R_{p=Gatsby|c=Q\&A|K=5}$
---
$R_{p=Gatsby|c=Q\&A|K=10}$
Idea | $D_{p=Gatsby|c=idea}$ |  | $R_{p=Gatsby|c=idea|K=5}$
---
$R_{p=Gatsby|c=idea|K=10}$
ALL | $D_{p=Gatsby|c=ALL}$ |  | $R_{p=Gatsby|c=ALL|K=5}$
---
$R_{p=Gatsby|c=ALL|K=10}$
Homebrew | ALL | $D_{p=Homebrew|c=ALL}$ |  | $R_{p=Homebrew|c=ALL|K=5}$
---
$R_{p=Homebrew|c=ALL|K=10}$
Next.js | Q&A | $D_{p=Next.js|c=Q\&A}$ |  | $R_{p=Next.js|c=Q\&A|K=5}$
---
$R_{p=Next.js|c=Q\&A|K=10}$
Idea | $D_{p=Next.js|c=idea}$ |  | $R_{p=Next.js|c=idea|K=5}$
---
$R_{p=Next.js|c=idea|K=10}$
ALL | $D_{p=Next.js|c=ALL}$ |  | $R_{p=Next.js|c=ALL|K=5}$
---
$R_{p=Next.js|c=ALL|K=10}$

###  4.4 The RD-Detector evaluation

We recruited maintainers of the three OSS projects analyzed and SE researchers
to evaluate the RD-Detector outcomes. Both OSS maintainers and SE researchers
manually classified pairs of related discussion candidates as duplicates,
related or not related. OSS maintainers judged pairs of related discussion
candidates according to their work project. The SE researchers judged pairs of
related discussion candidates detected on the three projects. The maintainers,
M_Gatsby, M_Homebrew, and M_Next.js, were contacted via the GitHub team of
professionals. The three SE researchers, SE_R1, SE_R2, and SE_R3, contacted
have different expertise in software development. SE_R1 is an industry
practitioner and a Software Engineering researcher, SE_R2 is an active OSS
contributor and a Software Engineering researcher, and SE_R3 is a Software
Engineering researcher.

Evaluators received online documents containing instructions to evaluate the
sets of the related discussion candidates. The documents (1) described the
concept of duplicates and related discussions, and (2) listed pairs of related
discussion candidates containing. We characterized each pair of related
discussions by describing the ID and title of the master and target
discussions. The ID was a link from which evaluators could access the original
discussion posts. We instructed the evaluators to add the label ‘D’ to
duplicates, ‘R’ to related ones, and ‘N’ for unrelated discussions. We also
asked the evaluators to add comments to justify their judgments.

We measured the RD-Detector precision rate compared to the OSS maintainers’
and SE researchers’ judgment. Their judgments identified the true-positive RD-
Detector predictions and defined the set $N$ used in Equation 3. Section 5
reports the precision rate reached assessing RD-Detector over the 14
configurations groups presented in Table 2.

##  5 Results

In this section, we first present the experimental results of RD-Detector
evaluation. Next, we describe the OSS maintainers’ feedback about the detected
related discussion candidates and the RD-Detector practical applications.

To evaluate the effectiveness of our approaches, we ran RD-Detector using
different configurations (Table 2) and measured its precision rate compared to
the OSS maintainers’ and SE researchers’ judgment (Section 4.4). First, to
better analyze and report our results, we filtered the dataset according to
the OSS repositories names and the discussion category. Next, we parameterized
the approach to use different values of $K$. In total, combining different
configurations, the RD-Detector outputs 14 related discussion candidates sets
(Table 2), over which we assessed the RD-Detector approach. The selected
evaluators judged all 14 sets of related discussion candidates.

As discussed in Section 3.3.2, as one changes the $K$ value, the $T_{related}$
threshold also changes and, consequently, the number of detected related
discussion candidate pairs. Tables 3, 4, and 5 present the local threshold
values, $T_{related}$, and the number of related discussion candidates pairs
detected, $|R|$, for Gatsby, Homebrew, and Next.js projects, respectively.
Tables 3, 4, and 5 also show, for each configuration group (project name,
category type, and $K$ value), the distribution size ($size(S)$) and the
descriptive statistics coefficients that summarize $S$ ($IQR$, $Q1$, $Q2$, and
$Q3$). We used the values of the coefficients to calculate the local threshold
$T_{related}$ (Equation 2).

We can note that as we increase the value of $K$, the local threshold values,
$T_{related}$, decrease (Tables 3, 4, and 5). Since the local threshold value
decreases, the RD-Detector detects new pairs of related discussion candidates.
The new pairs are those outliers considered by changing the Upper Inner Fence
value, according to Equations 1 and 2.

Table 3: Descriptive Statistics - Gatsby | Gatsby \- Similarity values
---|---
| c = Q&A | c = Ideas | c = ALL
| $K=5$ | $K=10$ | $K=5$ | $K=10$ | $K=5$ | $K=10$
Size(S) | 3924 | 7806 | 1063 | 2080 | 5277 | 10517
$IQR$ | 0.145 | 0.145 | 0.132 | 0.135 | 0.137 | 0.139
$Q1$ | 0.566 | 0.542 | 0.499 | 0.461 | 0.571 | 0.545
$Q2$ | 0.648 | 0.622 | 0.568 | 0.540 | 0.649 | 0.623
$Q3$ | 0.711 | 0.687 | 0.630 | 0.596 | 0.708 | 0.684
$T_{related}$ | 0.9278 | 0.9040 | 0.8278 | 0.7992 | 0.9127 | 0.8925
$|R|$ | 4 | 6 | 6 | 8 | 7 | 12

Table 3 presents the results for the Gatsby project. Considering $K=5$ and
$c=Q\&A$, RD-Detector calculated the local threshold value based on the
similarity values of 3,924 unique discussion pairs, $size(S)$, setting
$T_{related}=0.9278$ and detecting four related discussion candidates, $|R|$.
Considering the configuration group $|p=Gatsby|c=Q\&A|K=10$, the local
threshold value, $T_{related}=0.9040$, was calculated based on 7,806
similarity values of unique pairs of discussions, detecting six pairs of
related discussion candidates. Besides, considering $c=Ideas$ and $K=5$, the
local threshold $T_{related}$ calculated was 0.8278. This value was based on
the analysis of 1,063 similarity values and detected six pairs of related
discussion candidates. In addition, for the configuration group $p={\tt
Gatsby}|c=Ideas|K=10$, the RD-Detector detected eight pairs of related
discussion candidates, considering $T_{related}=0.7992$. Finally, one can
apply the same standard interpretation to Gatsby’s results by fixing $c=ALL$
and varying $K=5$ and $K=10$.

Table 4: Descriptive Statistics - Homebrew Homebrew \- Similarity values
---
| c = ALL
| $K=5$ | $K=10$
Size(S) | 6009 | 11948
$IQR$ | 0.130 | 0.130
$Q1$ | 0.535 | 0.508
$Q2$ | 0.602 | 0.578
$Q3$ | 0.665 | 0.638
$T_{related}$ | 0.8592 | 0.8336
$|R|$ | 15 | 34

Table 4 presents the RD-Detector outcomes for the Homebrew project.
Considering the configuration group $p={\tt Homebrew}|c=ALL|K=5$, the RD-
Detector calculated the local threshold value based on the similarity values
of 6,009 unique discussion pairs, $size(S)$, achieving $T_{related}=0.8592$.
As we can see in Table 4, the approach identified 15 pairs of related
discussion candidates considering $c=ALL$ and $K=5$. However, considering
$c=ALL$ and $K=10$, the approach calculated the local threshold value,
$T_{related}=0.8336$, based on 11,948 similarity values. Using this last
configuration group, the RD-Detector detected 34 related discussion candidate
pairs.

Table 5: Descriptive Statistics - Next.js | Next.js \- Similarity values
---|---
| c = Q&A | c = Ideas | c = ALL
| $K=5$ | $K=10$ | $K=5$ | $K=10$ | $K=5$ | $K=10$
Size(S) | 30665 | 60704 | 4006 | 7834 | 35171 | 69616
$IQR$ | 0.111 | 0.113 | 0.119 | 0.114 | 0.109 | 0.111
$Q1$ | 0.581 | 0.558 | 0.543 | 0.514 | 0.588 | 0.565
$Q2$ | 0.639 | 0.617 | 0.607 | 0.574 | 0.646 | 0.623
$Q3$ | 0.693 | 0.672 | 0.662 | 0.629 | 0.697 | 0.676
$T_{related}$ | 0.8605 | 0.8427 | 0.8402 | 0.8014 | 0.8608 | 0.8441
$|R|$ | 95 | 122 | 10 | 35 | 106 | 137

Table 5 presents the distribution size, $Size(S)$, and the number of detected
related discussion pairs, $|R|$, for the Next.js project. These numbers
endorse the scenario described by Ahasanuzzaman et al., (2016). The authors
highlighted that as PCQA forums become popular, the number of posts increases.
Consequently, it may also increase the number of duplicates or related posts.
Since the Next.js project is the largest considering the number of discussion
posts (Table 1), the chances of related discussion occurring can increase.
Using the configuration group $|p={\tt Next.js}|c=ALL|K=10$, the RD-Detector
detected 137 pairs of related discussion candidates - the most extensive
related discussion set detected. For this last configuration group, the local
threshold value, $T_{related}=0.8441$, was calculated based on the similarity
value of 69,616 unique pairs of discussions (Table 5).

For each of the 14 configuration groups presented in Table 2 column 4, the RD-
Detector calculated different values of local thresholds and detected
different amounts of related discussion candidate pairs. We highlight that all
sets of related discussion candidates detected considering $K=5$ are subsets
of the related discussion candidate sets setting $K=10$. Table 6 shows the
precision rate reached by the RD-Detector considering the 14 configuration
groups.

Table 6: Evaluation Results Category |  $K$ = 5 |  $K$ = 10
---|---|---
| Gatsby
Q&A | 100% | 100%
Ideas | 83.33% | 75%
ALL | 100% | 91.66%
| Homebrew
ALL | 100% | 91.17%
| Next.js
Q&A | 95.78% | 90.98%
Ideas | 90% | 88.57%
ALL | 97.16% | 93.43%

The smallest set size of related discussion candidates was detected using the
configuration group $p={\tt Gatsby}|c=Q\&A|K=5$ (Tables 3, 4, and 5).
According to this configuration group, the approach detected four related
discussion candidate pairs, $|R_{p={\tt Gatsby}|c=Q\&A|K=5}|=4$. Evaluators
emphasized that all four pairs are truly related discussions, which means that
all of them contain discussion posts related to one other. On the other hand,
the RD-Detector approach detected the biggest set of related discussion
candidates using the configuration $|p={\tt Next.js}|c=ALL|K=10$. In total,
the approach detected 137 pairs of related discussion candidates, $|R_{p={\tt
Next.js}|c=ALL|K=10}|=137$, of which 128 were classified as related by the
evaluators, achieving a precision rate of 93.43% (Table 6).

The OSS maintainer, M_Gatsby, and two SE researchers evaluated Gatsby
project’s related discussion candidates. In order to save M_Gatsby’s time and
effort in judging cases of duplicates that are exact copies, the GitHub
professional team requested researchers to judge pairs of related discussion
with a high similarity value. Two SE researchers judged pairs of related
discussion candidates with similarity values greater than or equal to 0.9415.
The researchers agreed that all judged pairs contained related discussions.
The maintainer M_Gatsby judged related discussion candidate pairs that
required prior technical knowledge about the project. According to Table 6,
RD-Detector reached the maximum precision value (100%) in detecting related
discussion pairs fixing $c=Q\&A$ and varying the $K$ value ($K=5$ and $K=10$)
for the Gatsby project. Considering the Gatsby project discussions classified
as Ideas ($c=Ideas$), the approach achieved an 83.33% precision rate for $K=5$
and 75% setting $K=10$. Still processing Gatsby’s discussions and setting
$c=ALL$, the RD-Detector achieved a precision rate of 100% and 91.66% for
$K=5$ and $K=10$, respectively.

The maintainer M_Homebrew and two SE researchers evaluated the Homebrew
project’s set of related discussion candidates. As we already mentioned, to
save the maintainer time and effort, the researchers judged pairs of related
discussion candidates with the highest similarity value (greater than or equal
to 0.9558). Considering the set $R_{p={\tt Homebrew},c=ALL,K=5}$, the RD-
Detector achieved the highest precision rate in detecting related discussion
pairs. Evaluators judged all of the 15 pairs of related discussion candidates
presented in $R_{p={\tt Homebrew},c=ALL,K=5}$ as related. However, setting the
configuration group $|p={\tt Homebrew}|c=ALL|K=10$, the approach achieved
91.17% of the precision rate. From evaluators’ perspective, 31 out of 34 are
truly related pairs of discussion posts. The maintainer M_Homebrew judged the
discussions of three pairs as unrelated.

As discussed in Section 4.1.1, compared to Gatsby and Next.js projects, the
Homebrew project organized the discussion posts differently. It does not
provide the same default categories provided by Gatsby and Next.js. Homebrew
discussions categorization is according to the problem type. That is why we
did not split the Homebrew dataset by category and present the achieved
precision rate setting the category filter to $ALL$ ($c=ALL$).

Finally, the number of detected related discussion candidates for the Next.js
project required greater participation of SE researchers in the evaluation
phase. Because the related discussion candidate sets of the Next.js project
are the largest, the project maintainer judgment became unfeasible. To ensure
the reliability of the reported results, we measured the researchers’ inter-
rater agreement using the Cohens Kappa Coefficient (Cohen,, 1960), which was
0.85. This value indicates almost perfect agreement according to the
interpretation proposed by Landis and Koch, (1977). Setting the configuration
group $|p=Next.js|c=Q\&A|K=5$, the approach reached precision values of
95.78%. The RD-Detector reached a precision of 90.98%, setting the $K$ value
to 10. Fixing $p={\tt Next.js}$ and $c=Ideas$, the RD-Detector reached
precision rates of 90% and 88.57% for $K=5$ and $K=10$, respectively. Besides,
when we fixed $p={\tt Next.js}$ and $c=ALL$, the RD-Detector reached precision
rates of 97.16% and 93.43% for $K=5$ and $K=10$, respectively.

Table 6 shows that precision rates degrade as we increase the value $K$. The
RD-Detector identified 18, 34, and 166 unique pairs of related discussion
candidates for the Gatsby, Homebrew, and Next.js projects, respectively. The
different numbers of detected related discussion pairs highlight the
uniqueness of the OSS communities regarding the use of the GitHub Discussions
forum. It also endorses the need for approaches that meet local thresholds to
detect related discussion pairs and adapt to different contexts of OSS
communities.

Table 7 lists all discussion pairs judged unrelated, grouped by project. The
table presents the master and target discussions identifiers (IDs) and titles
and their respective similarity values. Regarding Gatsby project results, two
pairs of related discussion candidates were classified as unrelated by the
M_Gatsby maintainer. Thus, 16 out of 18 detected pairs were indeed related
(precision=88.88%). From the perspective of the maintainer M_Homebrew, out of
the 34 related discussion candidate pairs detected, three are false positives.
Therefore, 91.17% of the detected pairs are truly related discussions.
Finally, 90.96% of discussion pairs detected for the Next.js project are truly
related. Out of 166 detected pairs, 15 are false positive. We discuss the
false positives in Section 6.

Table 7: False-positive related discussion candidates.

# |  | ID
---
Master
| Title Master
---
| ID
---
Target
| Title Target
---
| Similarity
---
value
|  Gatsby |
1 | 29766 |  | I want to lint ts file with Gatsby’s native
---
support.
32122 |  | On creating a better method to extend the
---
default ESLint configuration
0,897
2 | 26984 |  | More flexible default Typescript transpiler
---
handling
31662 |  | Need better CSS modules +
---
Typescript support
0,8172
|  Homebrew |
3 | 1906 |  | Questions on Homebrew’s third-party
---
mirroring policy
1917 | Setting up mirrors for Homebrew bottles | 0,8528
4 | 814 |  | Homebrew not installing on Big Sur v 11.2.1
---
\- Failed during: git fetch –force origin
1218 | Can’t get homebrew installed | 0,8396
5 | 72 | Installing Homebrew on a Windows OS | 657 | Unable to install HomeBrew | 0,8353
|  Next.js |
6 | 13368 |  | Can I replace next-routes with
---
the new versions?
20361 | nextjs with next-routes upgrade | 0,9281
7 | 15780 |  | ModuleNotFoundError: Module not found:
---
Error: Can’t resolve ’fs’
19154 | import sub module of other module is fail | 0,8887
8 | 13617 |  | NextJS not showing TypeScript
---
errors on Runtime
24996 |  | I am unable to setup “typescript-is”
---
with nextjs.
0,8646
9 | 23195 |  | Tailwind CSS not being bundled in
---
static export
25845 |  | NextJS with Tailwind does not work when
---
importing from Global.css
0,8624
10 | 23871 |  | Typescript - Can you change where Next.js
---
outputs the ”next-env.d.ts” file?
24996 |  | I am unable to setup “typescript-is”
---
with nextjs.
0,8593
11 | 14416 | deploy to Vercel error(Build error occurred) | 21107 |  | Deploy Vercel Error ’Error: Command
---
”yarn run build” exited with 1’
0,8522
12 | 19089 |  | Image Component with AWS Throws 400
---
Errors in Production
19568 |  | Image component not work when next is
---
deployed in sub path
0,8456
13 | 24778 | Cannot process DELETE request via CORS | 29589 |  | same-origin policy on API routes not
---
working
0,8455
14 | 15613 |  | next export - missing stylesheets - sass
---
files not being linked in the static html export
23945 |  | [Sass] Errors in basic use of Sass with
---
3rd party node modules
0,8449
15 | 14508 | Props are undefined from getServerSideProps | 20529 |  | Run getServerSideProps again on back
---
broswer
0,8446
16 | 22688 |  | Changes to .env.local are not loaded even
---
after restarting
22918 | Problem with less loader in<EMAIL_ADDRESS>| 0,843
17 | 17320 | [RFC] ESLint out of the box | 24900 | ESLint in Next.js and Create Next App | 0,8479
18 | 17320 | [RFC] ESLint out of the box | 28571 | Thoughts on Conformance in Next.js | 0,8099
19 | 24850 | Support Promises in next.config.js | 24851 |  | (re)Expose ‘nextConfig‘ on top-level
---
NextServer for custom servers
0,8088
20 | 24850 | Support Promises in next.config.js | 31205 | runtime env vars in next.config.js | 0,8037

Answering our RQ: The general purpose Sentence-BERT model applicable to NLP
problems effectively detects related discussion posts held in the GitHub
Discussions forum. The results presented in Table 6 show the effectiveness of
using the all-mpnet-base-v2 model to compute the sentence embeddings of the
discussion pots and detect related discussions.

###  5.1 OSS maintainers’ perspective regarding the judgment of related
discussion candidates

We asked OSS maintainers to comment on their judgment regarding their decision
about the related discussion candidates. Based on their comments, we could
identify (1) challenges deciding what are duplicate and related discussions,
(2) reasons why users create related discussions, and (3) practical
applications of RD-Detector.

####  5.1.1 Challenges deciding what are duplicate and related discussions

We noted that the concept of ‘related discussions’ — and even ‘duplicate
discussions’ — depends on the evaluators’ perspective. Therefore, we see this
conceptual imprecision as a limitation that may introduce some bias on the RD-
Detector evaluation.

Based on maintainers’ feedback regarding the sample of related discussion
candidates analyzed, we found four ways to conceptualize duplicates:

  1. 1.

Duplicates can be exact copies \- ‘…(the discussions) are identical
duplicates…’ (M_Next.js).

  2. 2.

Duplicates address the same issues \- ‘The initial problem is essentially the
same and they have the same primary solution…’ (M_Homebrew), ‘The two
discussions are about the same issue and the solution is functionally the
same, although the specific commands given are slightly different…’
(M_Homebrew), ‘The problem the user in the target discussion is experiencing
appears to be identical (as noted by them in their description)…’
(M_Homebrew), ‘These topics are identical …’ (M_Homebrew), ‘Different context
but same problem and solution.’(M_Next.js).

  3. 3.

Duplicates share common information (topic overlapping) \- ‘…the target
discussion also discusses a secondary problem that was raised during the
initial fix.’ (M_Homebrew), ‘Same thing exposed using an external library for
data fetching’ (M_Next.js).

  4. 4.

Sometimes, the same user can create duplicate discussions \- ‘…and were opened
by the same user’ (M_Homebrew), ‘Posted by the same author also.’ (M_Next.js).

We found that the conceptualization of duplicates tends to vary according to
objective and subjective criteria. The first criterion - exact copies - is an
objective one. To judge duplicates that are exact copies does not require
evaluators to do a deep semantic analysis of the discussions’ content. Exactly
copies tend to have high similarity values (the similarity value of the
discussion pair judged by M_Next.js as duplicates is 0.9793). However, the
second and the third criteria require evaluators to analyze discussions
semantically in order to identify the overlapping topic occurrence. Although
the fourth criterion is objective, evaluators can not consider it separately
to judge duplicates.

Compared to duplicates, maintainers classified related discussions as those
posts that are not precisely copies but have similar topics and share similar
information. They justify their judgment by highlighting that related
discussions:

  1. 1.

Are not precisely copies \- ‘…similar problems but not an exact duplicate’
(M_Homebrew), ‘Since different users are on different networks they will never
be 100% duplicate.’ (M_Homebrew), ‘These are all related to images so I’d say
they’re ’related’, they’re not exactly the same though’ (M_Next.js).

  2. 2.

Have the same solution and similar problems \- ‘Same solution and similar
problems…’ (M_Homebrew), ‘These (discussions) are both discussing the same
thing…’ (M_Homebrew), ‘Both users are experiencing the same initial problem…’
(M_Homebrew), ‘…These are all related to images…’ (M_Next.js).

  3. 3.

Express complimentary topics \- ‘The target discussion is actually a
continuation of the main discussion…’ (M_Homebrew), ‘This one is the initial
RFC of a feature implementation, the other discussion is a post release
issue.’ (M_Next.js), ‘This discussion caused the 14890 RFC to be created’
(M_Next.js).

  4. 4.

Cover different aspects of the same topic \- ‘Very similar issue on the
surface that relates to the same initial problem, but the actualy underlying
problem in the two was totally different’ (M_Homebrew), ‘…One user opened the
discussion for help because the suggested solution didn’t work. The other user
is asking why the problem occured in the first place.’ (M_Homebrew), ‘A
feature request for the image optimization feature, related but talking about
different things.’ (M_Next.js), ‘So the overall topic (guide) is the same, the
guide which it should be about is different.’ (M_Gatsby).

  5. 5.

May address the same problem in different contexts \- ‘…is the same exact bug…
but different context.’ (M_Next.js).

Given the subjective conceptualization of both duplicates and related
discussions concepts, and based on OSS maintainers’ feedback, we note they
also faced challenges in classifying the discussion pairs as duplicates or
related - ‘This blurs the lines a bit between R and D…’ (M_Homebrew), ‘There’s
definitely some similarity between the two, but it’s not really clear whether
the issue is the same…So, it’s hard to tell, but I don’t think they’re the
same’ (M_Homebrew), ‘…I would consider them to be duplicates without the
additional context of what was needed to fix the issue in the target
discussion’ (M_Homebrew), ‘I’m struggling with this one bit. They’re
technically different questions, so I understand why the user opened both
(they both have the same author)…’ (M_Homebrew), ‘Looks very similar, almost
duplicated … but he has a solution using a different…’ (M_Next.js). This
challenge endorses our decision to consider duplicates as a subtype of related
discussions.

We also note that the way users describe their questions, problems, or
requests may impact the RD-Detector results and the evaluator’s judgment.
Sometimes, it can be challenging for users to state the issue clearly and with
enough contextual detail: ‘Both people had a similar problem but described so
vaguely that it could be anything’ (M_Homebrew).

####  5.1.2 Reasons to create related discussions

Based on the feedback of Homebrew’s maintainer, we note the three reasons
Discussions users create related discussion posts. Although we likely do not
list all causes, the following three items enable OSS maintainers to
understand the reasons behind the occurrence of related discussions and plan
actions to deal with them. Users of GitHub Discussions create related
discussions because:

  1. 1.

They want to emphasize their need for help \- ‘… After the maintainers stopped
responding, the user opened a new discussion in an attempt to continue.’
(M_Homebrew).

  2. 2.

They want to add new information to better detail the discussion’s main topic
\- ‘The user never got a response, so they opened a new issue with some more
information a few weeks later.’ (M_Homebrew).

  3. 3.

They want to ask for different solutions to the same issue \- ‘…but they are
unhappy with the solution presented in the master discussion and would like an
alternative’ (M_Homebrew).

Specifically, for the Homebrew project results, the maintainer identified
multiple pairs of related discussions related. The maintainer states that the
discussion posts refer to a peculiar situation, ‘These are a little tricky… .’
Although they are related discussions, ‘so, I’d call the issues duplicates
because they all are the same problem…’, the solutions of the problems depend
on the users’ hardware and software configuration, ‘..each situation is
different it totally depends on the user’s specific machine and
configuration.’ The maintainer further states that the solutions are different
and that there is no single way to help users, ‘The actual solutions may be
different, but there’s no way for us to help with that so…’ Consequently, they
provide the same cues to users participating in the multi-related discussions
‘…from our perspective, the advice we can give is the same’ (M_Homebrew).
Based on this feedback, we envision the RD-Detector can also provide results
to maintainers prioritize project issues solving or intervene in recurring
situations demanded by the OSS communities.

####  5.1.3 Practical applications of RD-Detector

Finally, maintainers suggested some practical applications for RD-Detector
results, as follows :

  1. 1.

to combine the discussions’ content merging the related discussion threads \-
‘These (discussions) could have been combined into one discussion and it would
have made sense…’ (M_Homebrew), ‘… But, if it were up to me, they should have
gone together in the same discussion.’ (M_Homebrew).

  2. 2.

to move a discussion content to another location reorganizing the discussions
threads as comments to each other \- ‘…the new issue should probably have been
posted as a comment in the master discussion’ (M_Homebrew), ‘…(they) would’ve
received better traction as a comment on one another.’ (M_Next.js), ‘This
discussion could’ve sufficed as a comment on 21633.’ (M_Next.js).

  3. 3.

to recruit collaborators for specific tasks \- ‘…could be useful though for
people looking for other guides to contribute to (in this instance).’
(M_Gatsby).

The variety of OSS projects on the GitHub platform provides opportunities to
develop innovative approaches to detect related discussion candidates on
GitHub Discussions. However, the significant number of OSS communities, the
projects’ singularity, and the mentioned limitations challenge developing and
validating such systems. Those reasons endorse the RD-Detector design decision
to use a general-purpose machine learning model, calculate local threshold
values, and achieve better precision rates.

##  6 Discussions

As we reported in Section 5, increasing the $K$ values impacts on the local
threshold values and increases the number of detected related discussion
candidates. In this section, we discuss the effects of changing the $K$ value,
the false positives presented in Table 7, and the implications of this
research from the perspective of the OSS communities and software engineering
researchers.

###  6.1 The impacts of changing the $K$ value

As discussed in Section 3.3.2, the $K$ value delimits the search bounds for
related discussion candidates. As one changes the value of $K$, the threshold
$T_{related}$ also varies and, consequently, the number of detected related
discussion candidates. By configuring the RD-Detector to run over $K=5$ and
$K=10$, we note that the sets of related discussion candidates created when we
set $K=5$ are subsets of $K=10$. Consequently, there is a risk of false-
positive predictions propagation through the related discussion candidates’
sets. We identified this propagation problem by analyzing the sets of related
discussion candidates detected considering the configuration groups $p=Gatsby$
and $c=Ideas$. The same unrelated discussion pair $(master,target)$ occurs in
$R_{p={\tt Gatsby}|c=Ideas|K=5}$ and $R_{p={\tt Gatsby}|c=Ideas|K=10}$.
Although the precision rate tends to decrease, the approach detects new pairs
of related discussions when we vary the $K$ values from 5 to 10.

Gatsby project: Analyzing the sets of candidates $R_{p={\tt
Gatsby}|c=Q\&A|K=5}$ and $R_{p={\tt Gatsby}|c=Q\&A|K=10}$, RD-Detector
detected two new pairs of related discussions when changing the $K$ value from
5 to 10. The maintainer of the Gatsby project, M_Gatsby, judged discussions
from both new pairs as related. Since all related discussion candidates from
both sets were indeed related, the RD-Detector achieved the best precision
rate (100%). Considering $p={\tt Gatsby}|c=Ideas|K=5$ and $p={\tt
Gatsby}|c=Ideas|K=10$, the RD-Detector also detected two new pairs of related
discussion candidates by increasing the $K$ value. However, one out of the two
new pairs was judged unrelated by M_Gatsby. In this case, the precision rate
decreased from 83.33% to 75%.

As discussed in Section 4.2, when we set the category filter to ALL ($c=ALL$),
the RD-Detector calculates the similarity values between all discussion pairs
in the dataset. The approach detected seven related discussion candidates
using the configuration $p=Gatsby|c=ALL|K=5$ ($R_{p={\tt
Gatsby}|c=ALL|K=5}=7$), of which five elements are also present in set
$R_{p={\tt Gatsby}|c=Q\&A|K=10}$. Evaluators judged all five elements in
$R_{p={\tt Gatsby}|c=Q\&A|K=10}$ set as related. M_Gatsby judged that the
discussions of the two new pairs were related. We analyzed the discussions’
content and found that users create related discussions in different
categories. The discussions of these two specific pairs were from Idea and Q&A
categories, respectively.

Regarding the set $R_{p={\tt Gatsby}|c=ALL|K=10}$, six of the 12 pairs are
also present in the set $R_{p={\tt Gatsby}|c=Q\&A|K=10}$, two pairs are in
$R_{p={\tt Gatsby}|c=Idea|K=5}$, and two pairs also belongs to $R_{p={\tt
Gatsby}|c=ALL|K=5}$. All those ten related discussion pairs contain indeed
related discussion posts. The maintainer endorsed the relatedness between the
discussion posts of one new pair and refuted the second one. By increasing the
value of $K$ ($K=5$ to $K=10$) and setting $p={\tt Gatsby}|c=ALL$, the RD-
Detector detected five new pairs of related discussion candidates, of which
one was judged as unrelated by M_Gatsby.

Homebrew project: Analyzing the sets of related discussion candidates created
considering the configuration groups $p={\tt Homebrew}|c=ALL|K=5$ and $p={\tt
Homebrew}|c=ALL|K=10$, the RD-Detector detected 19 new related discussion
candidates when changing the $K$ value from 5 to 10. Out of the new 19
candidates, M_Homebrew judged three pairs as not related (Table 7). The
precision rate decreased from 100%, for $p={\tt Homebrew}|c=ALL|K=5$, to
91.17% for $p={\tt Homebrew}|c=ALL|K=10$.

Next.js project: The RD-Detector detected 27 new related discussion candidates
when we varied the $K$ value and fixed $p=Next.js|c=Q\&A$. Out of the 27 new
candidates, SE researchers judged seven pairs as not related. This means that
the approach detected 20 new pairs of related discussions by increasing the
value of $K$. However, the false positives decreased the RD-Detector precision
rate from almost 96% to nearly 91%. This scenario repeats to
$p=Next.js|c=Ideas$ and $p=Next.js|c=ALL$. When we changed the $K$ value from
5 to 10, the approach detected 25 and 31 new related discussion candidates for
$c=Ideas$ and $c=ALL$, respectively. In total, three and six new candidates
for $c=Ideas$ and $c=ALL$ were judged unrelated by SE researchers,
respectively. In both cases, the precision rate decreased.

Because of the intersection relationship between sets of related discussion
candidates, SE researchers had already judged 101 of the 106 pairs of related
discussion candidates in set $R_{p={\tt Next.js}|c=ALL|K=5}$. Researchers
evaluated the new five pairs as related. The five pairs have two common
characteristics: (1) they had discussions created by the same user, and (2)
they had one of the discussions created as Q&A and the other as Ideas. This
finding corroborates the Gatsby project findings. Users create related
discussion posts in different categories. Regarding $R_{p={\tt
Next.js}|c=ALL|K=10}$, researchers had already judged 133 out of the 137
related discussion pairs in the set. They also judged the four new pairs as
being related.

Maintainers can set the value of $K$ according to their respective interests.
Decreasing the $K$ value increases the RD-Detector precision rate. Higher
precision values ensure greater assertiveness in detecting true positives.
Conversely, increasing the $K$ value may reduce the precision rate. However,
increasing the $K$ value also increases the number of detected related
discussion candidates.

###  6.2 False-positive RD-Detector predictions

Four authors of this paper manually analyzed the false positives presented in
Table 7. Based on evidence extracted from the discussion posts, we identified
some limitations on using the proposed approach. We describe the false
positive as follows.

Gatsby project: The two false-positive cases are related to discussions
classified as $Idea$. The researchers identified that the RD-Detector did not
capture the project-related specifics on both pairs. Although the two posts of
pair #1 (Table 7) address the same topic (‘JavaScript linting utility ESLint’)
and have keywords intersection, they address different problems. The
discussion posts of the #2 pair (Table 7) address the topic ‘Typescript’ and
have project keywords intersection. However, the specificity of the issues
described in the discussion posts differs.

In both cases, we observed that the general-purpose model used to calculate
the similarity values of the discussions identified similar topical
conversations. However, it did not identify the project issue specificity.
Based on these findings, we observed that the RD-Detector could fail to treat
particular contexts of software projects. We will call this limitation the
‘project-specific limitation.’

Homebrew project: The analysis of the first Homebrew’s unrelated pair, #3
(Table 7), shows that the target discussion contains a link to the master
discussion, ‘… for Homebrew mirror configurations. #1906’. Link references can
endorse or refute relationships between discussion posts. In this case, the
text fragment that contains the link reference does not refute that the
discussions are related; however, it also does not clearly emphasize that they
are related. We analyzed the content of discussion pair #3. We identified a
limitation regarding the concept of ‘related discussions’ that can directly
influence the evaluation of the approach. We concluded that the interpretation
of the ‘related discussions’ concept depends on the evaluators’ perspective.
We call this limitation ‘concept imprecision.’

However, M_Homebrew’s feedback pointed out that pair #3 is unrelated due to
the ‘project-specific limitation.’ The maintainer claims that both discussions
address the same project feature, but they differ on the specificity of the
issue addressed. According to M_Homebrew maintainer, one discussion ‘…is
asking what the policy is’ and the other one ‘…is announcing support for a new
feature. ’ We also identified that the ‘project-specific limitation’ justifies
the other two false positives detected for the Homebrew project. The
maintainer justifies the unrelatedness of the pair #4 (Table 7) arguing that
the strategy captured a high level of abstraction from the two discussions, ‘…
problems with installing Homebrew.’ However, the RD-Detector did not capture
the specificity of the problem, ‘…the problem that’s encountered is different…
.’ Therefore, the relationship between the two is not confirmed because,
according to M_Homebrew ‘…each needs a different solution.’

We also analyzed the discussions’ content of the third unrelated pair #5
(Table 7). Like the pair #4, the discussions in #5 address issues related to
‘Homebrew installation’ but in different computing environments. Again, the
RD-Detector did not capture the specificity of the problem; however, both
discussion posts address the same issue and have the project-related keywords.

Next.js project: Analyzing the Next.js false-positives, we note that the
discussion creators (1) used screenshots to detail or describe the issues and
(2) used error logs descriptions to show the stack trace of where the error
took place. We also note the predominance of (3) template keywords and (4)
project keywords in the false-positive discussion posts.

Users can add screenshots in the discussion body to help explain their
problems. However, the RD-Detector measures the semantic textual similarity
between discussions content. The approach does not use images as a source of
evidence. The discussion pairs #7 and #15 (Table 7) exemplify this scenario.
In addition, the master and target discussions use the same description
template. After preprocessing, the templates’ keywords may stand out against
the real discussion content. We also identified the predominance of the
template keywords in the false-positive pair #16.

Discussions creators also use error logs or descriptions to describe the
system’s discrepancies or nonconformities. During the preprocessing phase, we
remove error descriptions embedded in HTML tags. However, when users use error
log content to express their questions, they usually intend to ask for help in
solving a specific problem highlighted in the error log content. Removing the
log also eliminates the problem specificity. The false-positive discussion
pair #11 (Table 7) exemplifies the use of error log descriptions in discussion
posts. However, project keywords in discussions of pair #11 show similarities
in the discussions’ contents.

The analysis of the pairs #13 and #14 shows that (again) both discussions use
the same description templates and have the same project keywords. Finally,
the false-positive pairs #6, #8, #9, #10, #12, #17, #18, #19, #20 (Table 7)
present the same set of keywords, which are limited due to the scope of the
project. Keywords match lead to the ‘project-specific limitation.’

Based on maintainers’ feedback, we can propose improvements to the proposed
approach. For example, we can use the maintainers’ judgments to optimize the
classifier by providing samples of related and not related discussions.
Furthermore, we can design strategies to minimize the project-specific
limitation by treating the predominance of projects’ keywords and templates’
keywords.

###  6.3 The implications of this research

We envision this research enables opportunities for OSS communities and
software engineering researchers as follows.

Long-term sustainability of OSS communities: the proposed approach increases
the effectiveness of related discussions detection in GitHub Discussions.
Their detection will no longer depend on the knowledge and availability of OSS
maintainers. Maintainers can benefit from RD-Detector to address the labor-
intensive task of manually detecting related discussions and the time-
intensive task of answering the same question multiple times. Such benefits
enable maintainers to focus efforts on performing other activities to support
the OSS community build and growth and the project sustainability, as pointed
out by Dias et al., (2021). The precision values presented in Table 6 show
that we can use a general-purpose deep learning model to detect related
discussions in GitHub Discussions. Consequently, we aim to build an automated
agent (Bot) to run the RD-Detector algorithm and report occurrences of related
discussion posts in the GitHub repositories. In addition, we envision a
practical application of our results in supporting the commenters’ tasks on
GitHub. The commenters are contributors that enrich discussions in the OSS
project by adding comments on collaborative conversation threads (Cánovas
Izquierdo and Cabot,, 2022).

Knowledge sharing in OSS communities: due to the importance of knowledge
sharing to OSS teams (Chen et al.,, 2013; Tantisuwankul et al.,, 2019), RD-
Detector can help maintainers reorganize project-related issues repeatedly
discussed in OSS communities. According to the OSS maintainers’ perspective,
the approach helps in identifying discussion threads that could be merged or
made as comments on one another (Section 5.1), allowing users to find the
correct answers in one place and avoid project knowledge decentralization
(Ahasanuzzaman et al.,, 2016; Silva et al.,, 2018). In addition, managing
related discussions can make finding the information community members are
looking for easier since it enables reducing noise occurrences and helps
valuable discussions posts become ‘more visible’ (Mamykina et al.,, 2011).

OSS communities’ growth: the remarkable acceptance of GitHub Discussions in
OSS projects (Liu,, 2021) creates expectations about the forum growth. As
GitHub Discussions grows and becomes popular, it faces the same limitations
and problems that other similar forums, such as related discussions (duplicate
or near-duplicate) occurrence (Zhang et al.,, 2015; Ahasanuzzaman et al.,,
2016; Zhang et al.,, 2017; Silva et al.,, 2018; Wang et al.,, 2020; Pei et
al.,, 2021). Identifying related threads can help users find previous posts
already asked and answered, reducing the waste of time while waiting for
feedback (Ahasanuzzaman et al.,, 2016) and the fear of submitting duplicates
(Ford et al.,, 2016). It also helps to suggest related posts before creating a
new thread (Zhang et al.,, 2017), assisting users to find posts with similar
issues (Ahasanuzzaman et al.,, 2016; Zhang et al.,, 2017), helping new users
asking better questions (Ford et al.,, 2018), and reducing the manual
responsibility of maintainers to sustain a healthy community (Guizani et al.,,
2022). Besides, every online community depends on volunteer activity, so it is
essential to attract and retain new members to guarantee community
sustainability (Cho and Wash,, 2021; Guizani et al.,, 2022). However, it is
well-known that newcomers face barriers to onboard to OSS communities
(Steinmacher et al.,, 2015). Given that the GitHub Discussions is a rich
project source of information, newcomers and inexperienced users also may
benefit from identifying related posts in the forum. Such users can analyze
the related discussion threads to get different formulations of the same
problem and better understand a project-related issue (Abric et al.,, 2019).
Through a cognitive process called ‘social or collaborative reflection,’
newcomers and inexperienced users can learn from the reflection that others
registered in the related discussion threads (Prilla et al.,, 2020). In this
way, our approach can support the long-term growth and maintenance of the
GitHub Discussions forum.

OSS Maintainers: previous research report that OSS maintainers need to manage
multiple aspects of the project to ensure that the project vision endures
(Guizani et al.,, 2022) and the projects’ long-term sustainability (Dias et
al.,, 2021). To do so, maintainers’ perform different activities encompassing
code and non-coding tasks (Dias et al.,, 2021; Trinkenreich et al.,, 2021).
Such responsibilities intensify the workload of OSS maintainers. Tan and Zhou,
(2020) highlight proposing tools as a best practice to decentralize
maintainers’ responsibilities. In addition, Dias et al., (2021) report that
‘to sustain a long term vision of the project, maintainers should delegate
tasks.’ In this way, the RD-Detector emerges as a tool to alleviate the
maintainers’ labor-intensive task of detecting related discussion occurrences
and managing duplicates or near-duplicate discussions.

Research Opportunities: to address the diversity and uniqueness of OSS
communities hosted on GitHub, the RD-Detector is based on a Sentence-BERT
general-purpose machine learning model. Besides, the approach uses descriptive
statistics to calculate the local threshold used to detect related discussion
pairs. Given that software projects are unique development ecosystems, we can
not use a single and universal number to detect related discussion pairs. Our
results (Table 6) endorse the effectiveness of the RD-Detector. However, we do
not know in advance how many or which discussions are genuinely related in the
GitHub Discussions forum, which prevents us from measuring the approach recall
rate. This gap brings opportunities for future work.

##  7 Limitations

Although we proposed a parameterizable approach based on general-purpose
machine learning models and descriptive statistics, this research may likely
present limitations.

The research’s main limitation relates to the generalization of the results.
We are aware of the diversity and uniqueness of the OSS communities hosted on
GitHub. Since we assessed RD-Detector over three OSS communities, we cannot
guarantee whether our findings generalize to all projects. GitHub’s
professionals (coauthors in this research) singled out the selected OSS
communities to minimize this limitation. In addition, the dataset refers to a
specific time window that does not reflect the current moment of GitHub
Discussions. However, the Next.js project stands out as it has a high rate of
use of the forum. More experiments are needed to assess the results in
different OSS communities.

Moreover, judging the relatedness between discussions is a subjective task and
could introduce biases in reporting the RD-Detector effectiveness. The RD-
Detector evaluation presents some challenges: (1) people who judge the
relatedness of the discussions need to semantically analyze the content of the
posts; (2) judging the technical aspects of the discussion posts requires
prior knowledge about the project; and (3) the judgment involves human
(in)precision regarding the concept of relatedness, although we defined the
concept of related and duplicate discussions, the interpretation depends on
the evaluators’ perspective. To minimize this threat, we introduced the
‘related discussions’ meaning to OSS maintainers and SE researchers before
classifying related discussion candidates. We also contacted selected OSS
maintainers to judge the related discussion candidates.

The local threshold calculation can also be a limitation. We consider related
discussion candidates those pairs identified as outliers in a distribution
$S$. $S$ contains the similarity values of the $K$ most similar target
discussions for each discussion post in the dataset. As we increase the value
of $K$, the median of the distribution decreases, and so does the local
threshold value. However, we focused on improving the RD-Detector precision.
Higher precision values ensure greater assertiveness in detecting related
discussions. So, we set the $K$ value to 5 and 10.

Finally, it was not possible to measure the RD-Detector’s recall rate. To the
best of our knowledge, this is the first work detecting related discussions in
the GitHub Discussions forum. However, achieved precision values ( Table 6 )
show the effectiveness of the RD-Detector in detecting related discussions.
Maintainers benefit from the RD-Detector quality by decreasing the time spent
reviewing related discussion candidates.

##  8 Conclusion

In this work, we presented the RD-Detector, an approach to detect related
discussion candidates in the GitHub Discussions forum. We assessed RD-Detector
over public discussions collected from three OSS communities. In total, the
approach evaluated the semantic similarity of 11,162 discussion posts. OSS
maintainers and SE researchers judged the detected related discussion
candidates. They classified pairs of related discussion candidates as
duplicates, related or not related. We measured the RD-Detector precision rate
using OSS maintainers’ and SE researchers’ judgment. Our results show that we
can use a general-purpose deep machine learning model applicable to NLP
problems to detect related discussions in the GitHub Discussions forum. The
RD-Detector achieved an average precision rate of 90.11%, considering the
top-10 most similar discussion pairs for each discussion post in the dataset
$D$ ($K=10$).

The RD-Detector uses a Sentence-BERT (SBERT) pre-trained general-purpose model
to compute semantically significant sentence embeddings of discussion posts
and detect related discussion candidates. Using publicly available machine
learning models brings flexibility to the approach. As researchers release new
exchangeable models, one can update the RD-Detector. Besides, the RD-Detector
calculates local threshold values to detect related discussion candidates in
different OSS communities. We use descriptive statistics to calculate the
upper inner fence value of a distribution containing the similarity values
data of the $K$ most similar target discussions to each discussion under
processing. We consider related discussions those identified as outliers in
the distribution.

The approach’s outputs were unique (Tables 3, 4, and 5). Our results highlight
the need for strategies designed to quickly adapt to the dynamism with which
OSS communities grow and change. We found that OSS communities create related
discussions to emphasize their need for help, add new information on previous
discussions threads, and ask for alternative solutions. From the maintainers’
perspective, we also found that related discussions have the same resolution
and similar problems, address additional issues, and address different
specifics within the same topic.

Maintainers can benefit from RD-Detector to address the labor-intensive task
of manually detecting related discussions and the time-intensive task of
answering the same question multiple times. In addition, OSS maintainers can
benefit from the approach to prioritize the development or update project-
related issues frequently discussed, understand why users duplicate questions,
control the propagation of duplicates, and support the project knowledge
sharing. According to the maintainers’ feedback, one can merge related
discussion threads or make related discussions as comments on one another.

We reported and discussed our results with the professional GitHub team (co-
authors of this work). Our findings showed a real need to plan and tackle
related discussions on GitHub Discussions. Consequently, the GitHub
engineering team is testing some changes to the Discussions interface. In
addition to providing the discussion title, body text, and category, users
would have to confirm they have searched for similar threads before creating
new posts (using a checkbox).

As the next step, we intend to implement RD-Detector as a bot to run over the
GitHub Discussions data and fetch, analyze, detect, and report related
discussions occurrences. We believe that this research also brings
opportunities to enable project knowledge acquisition and transfer by
providing users with project-related issues and by making the projects’
knowledge easy to find. In addition, our results can enable project knowledge
reuse as users can access related discussions that have already been asked and
answered and the project knowledge categorization by identifying similar
documents.

###### Acknowledgements.

We would like to thank GitHub and the OSS maintainers for supporting this
research. We also would like to thank the financial support granted by CNPq
through processes number 314174/ 2020-6 and 313067/2020-1. CAPES financial
code 001. FAPESP under grant 2020/05191-2. FAPEAM through process number
062.00150/2020. This research was also carried out within the scope of the
Samsung-UFAM Project for Education and Research (SUPER), according to Article
48 of Decree number 6.008/2006(SUFRAMA).

## Declaration of competing interest

The fourth and the fifth authors of this manuscript participate in the GitHub
Discussions engineering team. The other authors have no competing interests to
declare that are relevant to the content of this article.

## References

  * Abric et al., (2019) Abric, D., Clark, O. E., Caminiti, M., Gallaba, K., and McIntosh, S. (2019).  Can duplicate questions on stack overflow benefit the software development community?  In 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR), pages 230–234. IEEE.
  * Agirre et al., (2015) Agirre, E., Banea, C., Cardie, C., Cer, D., Diab, M., Gonzalez-Agirre, A., Guo, W., Lopez-Gazpio, I., Maritxalar, M., Mihalcea, R., Rigau, G., Uria, L., and Wiebe, J. (2015).  SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretability.  In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 252–263, Denver, Colorado. Association for Computational Linguistics.
  * Ahasanuzzaman et al., (2016) Ahasanuzzaman, M., Asaduzzaman, M., Roy, C. K., and Schneider, K. A. (2016).  Mining duplicate questions of stack overflow.  In 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR), pages 402–412. IEEE.
  * Calefato et al., (2021) Calefato, F., Gerosa, M. A., Iaffaldano, G., Lanubile, F., and Steinmacher, I. (2021).  Will you come back to contribute? investigating the inactivity of oss core developers in github.  arXiv preprint arXiv:2103.04656.
  * Cánovas Izquierdo and Cabot, (2022) Cánovas Izquierdo, J. L. and Cabot, J. (2022).  On the analysis of non-coding roles in open source development.  Empirical Software Engineering, 27(1):1–32.
  * Chen et al., (2013) Chen, X., Li, X., Clark, J. G., and Dietrich, G. B. (2013).  Knowledge sharing in open source software project teams: A transactive memory system perspective.  International Journal of Information Management, 33(3):553–563.
  * Cho and Wash, (2021) Cho, J. and Wash, R. (2021).  How potential new members approach an online community.  Computer Supported Cooperative Work (CSCW), 30(1):35–77.
  * Cohen, (1960) Cohen, J. (1960).  A coefficient of agreement for nominal scales.  Educational and psychological measurement, 20(1):37–46.
  * Community, (2022) Community, G. (2022).  Gatsby v4. url: https://github.com/gatsbyjs/gatsby/ blob/master/readme.md .accessed 23 january 2022.
  * Dias et al., (2021) Dias, E., Meirelles, P., Castor, F., Steinmacher, I., Wiese, I., and Pinto, G. (2021).  What makes a great maintainer of open source projects?  In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pages 982–994. IEEE.
  * Face, (2021) Face, H. (2021).  sentence-transformers/all-mpnet-base-v2. url:https://huggingface.co/sentence-transformers/all-mpnet-base-v2 . accessed 13 january 2022.
  * Ford et al., (2018) Ford, D., Lustig, K., Banks, J., and Parnin, C. (2018).  ”we don’t do that here”: How collaborative editing with mentors improves engagement in social q&a communities.  In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, page 1–12, New York, NY, USA. Association for Computing Machinery.
  * Ford et al., (2016) Ford, D., Smith, J., Guo, P. J., and Parnin, C. (2016).  Paradise unplugged: Identifying barriers for female participation on stack overflow.  In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016, page 846–857, New York, NY, USA. Association for Computing Machinery.
  * (14) GitHub, I. (2021a).  Github discussions. url: https://docs.github.com/en/ discussions. accessed 23 january 2022.
  * (15) GitHub, I. (2021b).  Managing categories for discussions in your repository. url: https://docs.github.com/en/discussions/managing-discussions-for-your-community/managing-categories-for-discussions-in-your-repository. accessed 23 january 2022.
  * (16) GitHub, I. (2021c).  Searching discussions. url: https://docs.github.com/en/github/searching-for-information-on-github/searching-discussions. accessed 23 january 2022.
  * GitHub, (2022) GitHub, I. (2022).  What is github discussions? a complete guide. url: https://resources.github.com/devops/ process/planning/discussions/. accessed 21 april 2022.
  * Guizani et al., (2022) Guizani, M., Zimmermann, T., Sarma, A., and Ford, D. (2022).  Attracting and retaining oss contributors with a maintainer dashboard.  arXiv preprint arXiv:2202.07740.
  * Hata et al., (2022) Hata, H., Novielli, N., Baltes, S., Kula, R. G., and Treude, C. (2022).  Github discussions: An exploratory study of early adoption.  Empirical Software Engineering, 27(1):1–32.
  * Kim et al., (2005) Kim, Y., Lee, S., Dollmann, M., and Geierhos, M. (2005).  Improving classifiers for semantic annotation of software requirements with elaborate syntatic structure.  International Journal of Advanced Science and Technology, ISSN, 4238:123–136.
  * Landis and Koch, (1977) Landis, J. R. and Koch, G. G. (1977).  The measurement of observer agreement for categorical data.  biometrics, pages 159–174.
  * Lee and Shin, (2020) Lee, I. and Shin, Y. J. (2020).  Machine learning for enterprises: Applications, algorithm selection, and challenges.  Business Horizons, 63(2):157–170.
  * Li et al., (2018) Li, L., Ren, Z., Li, X., Zou, W., and Jiang, H. (2018).  How are issue units linked? empirical study on the linking behavior in github.  In 2018 25th Asia-Pacific Software Engineering Conference (APSEC), pages 386–395. IEEE.
  * Li et al., (2017) Li, Z., Yin, G., Yu, Y., Wang, T., and Wang, H. (2017).  Detecting duplicate pull-requests in github.  In Proceedings of the 9th Asia-Pacific Symposium on Internetware, pages 1–6.
  * Li et al., (2020) Li, Z., Yu, Y., Zhou, M., Wang, T., Yin, G., Lan, L., and Wang, H. (2020).  Redundancy, context, and preference: An empirical study of duplicate pull requests in oss projects.  IEEE Transactions on Software Engineering.
  * Liu, (2021) Liu, E. (2021).  Github discussions is out of beta. url: https://github.blog/2021-08-17-github-discussions-out-of-beta/. accessed 15 january 2022.
  * Liu, (2022) Liu, E. (2022).  How five open source communities are using github discussions. url:https://github.blog/2022-01-13-how-five-open-source- communities-are-using-github-discussions/.
  * Liu, (2011) Liu, T.-Y. (2011).  Learning to rank for information retrieval.
  * Mamykina et al., (2011) Mamykina, L., Manoim, B., Mittal, M., Hripcsak, G., and Hartmann, B. (2011).  Design lessons from the fastest q&a site in the west.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’11, page 2857–2866, New York, NY, USA. Association for Computing Machinery.
  * Niyogi, (2020) Niyogi, S. (2020).  New from satellite 2020: Github discussions, codespaces, securing code in private repositories, and more. url: https://github.blog/2020-05-06-new-from-satellite-2020-github-codespaces-github- discussions-securing-code-in-private-repositories-and-more/. accessed 21 january 2022.
  * Pei et al., (2021) Pei, J., Wu, Y., Qin, Z., Cong, Y., and Guan, J. (2021).  Attention-based model for predicting question relatedness on stack overflow.  In 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), pages 97–107. IEEE.
  * Pérez-Soler et al., (2018) Pérez-Soler, S., Guerra, E., and de Lara, J. (2018).  Collaborative modeling and group decision making using chatbots in social networks.  IEEE Software, 35(6):48–54.
  * Polyzotis et al., (2017) Polyzotis, N., Roy, S., Whang, S. E., and Zinkevich, M. (2017).  Data management challenges in production machine learning.  In Proceedings of the 2017 ACM International Conference on Management of Data, pages 1723–1726.
  * Prilla et al., (2020) Prilla, M., Blunk, O., and Chounta, I.-A. (2020).  How does collaborative reflection unfold in online communities? an analysis of two data sets.  Computer Supported Cooperative Work (CSCW), 29(6):697–741.
  * Project, (2022) Project, H. (2022).  Homebrew documentation. url:https://docs.brew.sh/. accessed 23 january 2022.
  * Reimers, (2021) Reimers, N. (2021).  Sentencetransformers documentation. url: https://www.sbert.net/. accessed 15 january 2022.
  * Reimers and Gurevych, (2019) Reimers, N. and Gurevych, I. (2019).  Sentence-bert: Sentence embeddings using siamese bert-networks.  arXiv preprint arXiv:1908.10084.
  * Ren et al., (2019) Ren, L., Zhou, S., Kastner, C., and Wasowski, A. (2019).  Identifying redundancies in fork-based development.  In 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 230–241. IEEE.
  * Schelter et al., (2018) Schelter, S., Biessmann, F., Januschowski, T., Salinas, D., Seufert, S., and Szarvas, G. (2018).  On challenges in machine learning model management.
  * Silva et al., (2018) Silva, R. F., Paixão, K., and de Almeida Maia, M. (2018).  Duplicate question detection in stack overflow: A reproducibility study.  In 2018 IEEE 25th international conference on software analysis, evolution and reengineering (SANER), pages 572–581. IEEE.
  * Sirres et al., (2018) Sirres, R., Bissyandé, T. F., Kim, D., Lo, D., Klein, J., Kim, K., and Traon, Y. L. (2018).  Augmenting and structuring user queries to support efficient free-form code search.  Empirical Software Engineering, 23(5):2622–2654.
  * Steinmacher et al., (2015) Steinmacher, I., Conte, T., Gerosa, M. A., and Redmiles, D. (2015).  Social barriers faced by newcomers placing their first contribution in open source software projects.  In Proceedings of the 18th ACM conference on Computer supported cooperative work & social computing, pages 1379–1392.
  * Storey et al., (2014) Storey, M.-A., Singer, L., Cleary, B., Figueira Filho, F., and Zagalsky, A. (2014).  The (r) evolution of social media in software engineering.  In Future of Software Engineering Proceedings, pages 100–116.
  * Storey et al., (2016) Storey, M.-A., Zagalsky, A., Figueira Filho, F., Singer, L., and German, D. M. (2016).  How social and communication channels shape and challenge a participatory culture in software development.  IEEE Transactions on Software Engineering, 43(2):185–204.
  * Tan and Zhou, (2020) Tan, X. and Zhou, M. (2020).  Scaling open source software communities: Challenges and practices of decentralization.  IEEE Software, 39(1):70–75.
  * Tantisuwankul et al., (2019) Tantisuwankul, J., Nugroho, Y. S., Kula, R. G., Hata, H., Rungsawang, A., Leelaprute, P., and Matsumoto, K. (2019).  A topological analysis of communication channels for knowledge sharing in contemporary github projects.  Journal of Systems and Software, 158:110416.
  * Trinkenreich et al., (2021) Trinkenreich, B., Guizani, M., Wiese, I. S., Conte, T., Gerosa, M., Sarma, A., and Steinmacher, I. (2021).  Pots of gold at the end of the rainbow: What is success for open source contributors.  IEEE Transactions on Software Engineering.
  * Tukey et al., (1977) Tukey, J. W. et al. (1977).  Exploratory data analysis, volume 2.  Reading, Mass.
  * Vercel, (2022) Vercel, I. (2022).  Create a next.js app. url:https://nextjs.org/learn/basics/create-nextjs-app.accessed 23 january 2022\.
  * Wang et al., (2020) Wang, L., Zhang, L., and Jiang, J. (2020).  Duplicate question detection with deep learning in stack overflow.  IEEE Access, 8:25964–25975.
  * Yazdaninia et al., (2021) Yazdaninia, M., Lo, D., and Sami, A. (2021).  Characterization and prediction of questions without accepted answers on stack overflow.  In 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC), pages 59–70. IEEE.
  * Yu et al., (2018) Yu, Y., Li, Z., Yin, G., Wang, T., and Wang, H. (2018).  A dataset of duplicate pull-requests in github.  In Proceedings of the 15th International Conference on Mining Software Repositories, pages 22–25.
  * Zhang et al., (2017) Zhang, W. E., Sheng, Q. Z., Lau, J. H., and Abebe, E. (2017).  Detecting duplicate posts in programming qa communities via latent semantics and association rules.  In Proceedings of the 26th International Conference on World Wide Web, pages 1221–1229.
  * Zhang et al., (2015) Zhang, Y., Lo, D., Xia, X., and Sun, J.-L. (2015).  Multi-factor duplicate question detection in stack overflow.  Journal of Computer Science and Technology, 30(5):981–997.
  * Zhang et al., (2020) Zhang, Y., Wu, Y., Wang, T., and Wang, H. (2020).  ilinker: a novel approach for issue knowledge acquisition in github projects.  World Wide Web, 23(3):1589–1619.
  * Zhou et al., (2017) Zhou, L., Pan, S., Wang, J., and Vasilakos, A. V. (2017).  Machine learning on big data: Opportunities and challenges.  Neurocomputing, 237:350–361.