#  This is the author’s version of the paper that has been accepted for
publication in
the 38th IEEE/ACM International Conference on Automated Software Engineering
(ASE 2023)
Personalized First Issue Recommender for Newcomers in Open Source Projects

Wenxin Xiao1, Jingyue Li2, Hao He1, Ruiqiao Qiu1, Minghui Zhou13  1School of
Computer Science, Peking University, China
Key Laboratory of High Confidence Software Technologies, Ministry of
Education, China
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>2Norwegian University of Science and Technology, Trondheim,
Norway
<EMAIL_ADDRESS>
###### Abstract

Many open source projects provide good first issues (GFIs) to attract and
retain newcomers. Although several automated GFI recommenders have been
proposed, existing recommenders are limited to recommending generic GFIs
without considering differences between individual newcomers. However, we
observe mismatches between generic GFIs and the diverse background of
newcomers, resulting in failed attempts, discouraged onboarding, and delayed
issue resolution. To address this problem, we assume that personalized first
issues (PFIs) for newcomers could help reduce the mismatches. To justify the
assumption, we empirically analyze 37 newcomers and their first issues
resolved across multiple projects. We find that the first issues resolved by
the same newcomer share similarities in task type, programming language, and
project domain. These findings underscore the need for a PFI recommender to
improve over state-of-the-art approaches. For that purpose, we identify
features that influence newcomers’ personalized selection of first issues by
analyzing the relationship between possible features of the newcomers and the
characteristics of the newcomers’ chosen first issues. We find that the
expertise preference, OSS experience, activeness, and sentiment of newcomers
drive their personalized choice of the first issues. Based on these findings,
we propose a Personalized First Issue Recommender (PFIRec), which employs
LamdaMART to rank candidate issues for a given newcomer by leveraging the
identified influential features. We evaluate PFIRec using a dataset of 68,858
issues from 100 GitHub projects. The evaluation results show that PFIRec
outperforms existing first issue recommenders, potentially doubling the
probability that the top recommended issue is suitable for a specific newcomer
and reducing one-third of a newcomer’s unsuccessful attempts to identify
suitable first issues, in the median. We provide a replication package at
https://zenodo.org/record/7915841.

33footnotetext: Minghui Zhou is the corresponding author.

##  I Introduction

Open source software (OSS) project developers face challenges when it comes to
selecting suitable issues to contribute to, particularly when they are new to
a project [1]. If a newcomer’s initial attempts to resolve their chosen issues
fail, it can waste their and the maintainers’ time and effort, delaying issue
resolution and blocking opportunities for other newcomers. To address this
problem, GitHub encourages OSS projects to label good first issues (GFIs) to
highlight suitable tasks for project newcomers.

To reduce projects’ effort to label GFIs, several automated GFI recommending
approaches and tools are proposed and have achieved reliable performance [2,
3, 4, 5]. However, they treat newcomers as an undifferentiated mass without
considering the individual differences among contributors, and such generic
GFIs may not match newcomers’ diverse backgrounds, leading to unsuccessful
resolution attempts. For example, in our dataset (see Section IV-D for more
details), we identified 199 GFIs wherein 238 newcomers actively engaged by
providing comments on the issues or participating in issue events [6]. These
interested newcomers were potential GFI resolvers. However, 224 (94.1%) of the
newcomers did not resolve the GFIs they were involved in, and these GFIs were
resolved by other newcomers who did not engage in commenting or participating
in issue events. Merely 14 (5.9%) of the 238 newcomers resolved the GFIs they
participated in.

Furthermore, we encountered instances that vividly portray the divergence
between specific newcomers and GFIs:

  * •

In Issue #151196 of VS Code [7], a PR made by a newcomer was closed 26 days
after he claimed the issue due to his lack of experience with the TextMate
grammar necessary to resolve the issue; only four days later, the issue was
successfully resolved by another newcomer.

  * •

In Issue #50207 of the pandas project [8], a newcomer stated, 36 days after
claiming to take the issue, that he did not have enough time to resolve the
issue; then, a second newcomer took five days to resolve the issue.

The two examples show that GFIs are not generally suitable for arbitrary
newcomers and that newcomers may be unable to resolve some GFIs if they lack
the necessary knowledge or skills or sufficient time. Some projects have
specific requirements for newcomers’ experience, which makes it even more
necessary to match the background of newcomers to the requirements of the
projects and GFIs. For instance, the Bitcoin Core project states that a good
first issue isn’t targeted toward software development beginners. At the very
least, you’ll need basic Git proficiency and ideally C++ and/or Python
proficiency, too, given that the Bitcoin Core codebase is written in these
languages [9].

Therefore, we assume that personalized first issue (PFI) recommendations,
based on the historical behavior of each newcomer, may significantly improve
the effectiveness of GFI recommendation tools and therefore, reduce the failed
attempts of project newcomers. To justify the assumption, we posit the first
research question (RQ) to check whether each newcomer exhibits distinct first-
issue preferences across projects and provide a theoretical underpinning for
the implementation of PFI recommenders:

  * •

RQ1: Does each OSS newcomer exhibit personalized preferences when choosing the
first issue in a new project?

To answer RQ1, we identify 37 newcomers contributing to more than two projects
from the GFI-Bot dataset [5] and analyze the 123 first issues111We will
abbreviate the first issue that newcomers resolve in a project as FI
hereafter. they resolved. We then compare the FI characteristics (i.e., task
types, programming languages, and project domains) for each newcomer across
projects as well as across newcomers. The results show that the FI
characteristics are similar across projects for each individual but different
among newcomers. This indicates that newcomers’ choice of the FIs in new
projects is mostly personalized, and it will be beneficial to propose PFIs to
project newcomers.

To propose a PFI recommender, we need to know which features of newcomers
influence their personalized choice of FIs. Thus, we pose the second research
question:

  * •

RQ2: Which features of newcomers are related to their choices of the first
issues in projects?

To answer RQ2, we analyze the relationship between possible influential
features and the characteristics of the FIs selected by the 37 newcomers. We
find that their resolved FIs are more semantically akin to their historically
resolved ones than those resolved by other newcomers, and the newcomers’
experience, activeness, and sentiment features impact their FI choices.

Based on the identified features from RQ2, we propose PFIRec, a personalized
issue recommendation approach that ranks issues for specific newcomers using
the LambdaMART model [10]. To train the model and evaluate the effectiveness
of PFIRec, we construct a dataset by collecting resolver (i.e., newcomer)
features and issue features of 11,615 FIs and their 57,243 contemporary issues
from 100 GitHub projects aiming to attract newcomers. The experimental results
show that PFIRec outperforms state-of-the-art GFI recommenders. Compared with
the best state-of-the-art GFI recommender, PFIRec can double the probability
that the top recommended issue is suitable for a specific newcomer and reduce
one-third of the number of a newcomer’s unsuccessful attempts to find
appropriate issues, in the median.

This paper makes the following contributions:

  * •

We give empirical evidence that newcomers’ choices of the first issues in new
projects are mostly personalized. To the best of our knowledge, this is the
first study to explore personalized recommendations for the first issue.

  * •

We provide insights that historical contributions, general OSS experience,
activeness, and sentiment of newcomers can influence their personalized choice
of first issues.

  * •

We propose and evaluate PFIRec, a personalized issue recommender that ranks
issues for specific newcomers, outperforming state-of-the-art GFI
recommenders.

The rest of the paper is organized as follows. Section II introduces
background and related work. The methodology and results of the empirical
study (answering RQ1 and RQ2) are presented in Section III. PFIRec’s design,
implementation, and evaluation are in Section IV. Section V discusses
comparison with related work and threats to validity. Section VI concludes the
paper.

##  II Background and Related Work

###  II-A Newcomer Onboarding

OSS projects depend on many volunteers and their contributions to evolve and
meet various requirements [11]. However, the sustainability of OSS projects is
at risk due to the loose organizational structure, which can lead to the
potential loss of contributors [12]. For example, the disclosure of HeartBleed
and Log4Shell vulnerabilities brought attention to the fact that the
maintenance of OpenSSL and Log4j relied on contributions from only two and
four developers, respectively [13, 14], causing considerable damage to
numerous dependent projects [15, 16]. To ensure the sustainability of OSS
projects and overcome the unmaintained status, consistently attracting and
retaining newcomers is crucial [17, 18].

Despite motivations, such as learning software development skills and pursuing
future career prospects, which motivate newcomers to participate in OSS
projects [19, 20], they often face barriers, such as lack of project support
and the necessary skills[21, 1], making successful onboarding challenging.
Multiple failed attempts may cause newcomers to abandon their contributions,
which wastes not only their time and effort but also those of project
maintainers.

Numerous studies explore the barriers newcomers face when contributing to new
projects, along with strategies to overcome these challenges. Steinmacher et
al. [22] provide onboarding guidelines, such as creating a newcomer portal.
Jensen et al. [23] emphasize the importance of timely responses to newcomers.
Santos et al. [24] indicate that newcomers want more onboarding support, Tan
et al. [25] find expert involvement helpful, and He et al. [26] show that
courses can be an effective intervention. Zhou and Mockus [27] recommend
mentoring potential long-term contributors, and Panichella [28] proposes a
tool to match suitable mentors with newcomers.

Despite the above strategies, one critical challenge remains, i.e.,
identifying suitable tasks for newcomers who may lack the requisite skills and
knowledge [29, 30]. Wang and Sarma [31] present an interactive tool to
visualize bug descriptions for newcomers. However, it is still difficult for
newcomers to find suitable tasks among a massive number of tasks. To address
these challenges, good first issue labels are applied to highlight
opportunities for newcomers to contribute [32].

###  II-B Newcomer Task Recommendation

Several studies investigate the current status and limitations of the GFI
mechanism. Horiguchi et al. [33] show that GFIs are resolved by less
experienced developers compared to regular issues, while Alderliesten and
Zaidman [34] reveal that only a small percentage of GFIs are addressed by
newcomers. Xiao et al. [35] discover that the GFI mechanism has limited
effectiveness in the early stages of OSS projects. Tan et al. [17] find that
manually labeled GFIs are often highly insufficient, and cognitive mismatch
between newcomers and veterans can lead to inappropriate GFIs labeled by
veterans. To overcome the limitations of the GFI mechanism due to sparse and
subjective labeling, automated approaches are proposed to recommend issues to
newcomers [3, 4, 5, 2]. Among them, the most efficient automated approach is
GFI-Bot [5], which is developed as a GitHub bot. These automated approaches
extract various features to identify issues that may be suitable for any
newcomer. However, these approaches do not take into account the diverse
backgrounds of newcomers. When using these tools, newcomers may be recommended
issues that do not match their preferences or that they find difficult or
uninteresting to resolve.

Personalized recommendation have been explored in other software engineering
tasks. For example, Anvik et al. [36], Ashok et al. [37], and MacDonald et al.
[38] recommend experts to a specific task based on their familiarity with
task-related artifacts or source code. Wang et al. [39] recommend crowdworkers
with relevant experience to test tasks on crowdsourcing platforms using text
term matching. Ye et al. [40] and Zhou et al. [41] propose approaches to
personally recommend collaborative teammates and GitHub repositories for
developers based on their skill proficiency and collaboration history, and
topic similarity. In general, previous research on matching developers and
tasks focuses on recommending experts who are familiar with the task rather
than newcomers who do not have experience in the project. Other personalized
recommendation studies recommend collaborators or projects instead of tasks to
developers. These approaches cannot be used directly to resolve our problem.

##  III Empirical study

As previously stated, generic GFIs may not align with newcomers’ individual
abilities and interests, and personalized recommendations have been applied in
software engineering to improve the efficacy of recommendations. Therefore, we
are motivated to answer RQ1 and RQ2 to empirically investigate whether it is
rational to recommend personalized FIs and what factors influence newcomers’
personalized choices.

###  III-A RQ1 Methodology

According to previous studies (e.g., [42, 43, 17, 4]), certain types of tasks,
such as document changes, are suitable for newcomers. Additionally, project
background, including programming language and project domain, may serve as an
indicator of whether newcomers can resolve the issues [4, 5]. Hence, we aim to
compare the issues resolved by newcomers based on task type, programming
language, and project domain to investigate whether there are variations among
issues resolved by different newcomers and whether the FIs resolved by the
same newcomer in different projects are similar.

####  III-A1 Data Collection

To answer RQ1, we need a dataset containing issue- and resolver-related data
collected from projects that are attractive to and have already onboarded many
newcomers. The GFI-Bot dataset [5], which is currently the largest and most
up-to-date dataset for recommending issues to newcomers, has collected 159,919
issues until June 2022 from the 100 GitHub projects with the highest number of
good first issues, including prestigious projects, such as pandas and VS Code
[4]. These projects are willing to attract newcomers and are highly desirable
for newcomers to join. The dataset contains issues, reporter and project owner
data. We identified 12,755 newcomers and the 13,266 FIs they resolved from the
dataset to conduct this study.

Investigating FIs resolved by the same newcomer across multiple projects can
help us understand the newcomers’ tendency to select issues in new projects.
However, insufficient records of a newcomer’s FIs make it challenging to
explore their inclination toward new project issues. Among the 12,755
newcomers, 12,293 newcomers addressed FIs in one project, while 425, 26, 10,
and one newcomer resolved FIs in two, three, four, and five projects
(considering only the 100 projects in the dataset), respectively. We manually
label and analyze the 123 (=26$\times$3+10$\times$4+1$\times$5) FIs resolved
by the 37 newcomers who participated in more than two projects to answer RQ1.

####  III-A2 Data Analysis

We first manually categorize the 123 FIs based on task type, which encompasses
two kinds of categories: 1) code/document tasks; and 2)
corrective/adaptive/perfective tasks, following the widely adopted software
change classification proposed by Mockus and Votta [44]. We then record the
primary programming languages of the projects, which involve JavaScript,
Python, Java, TypeScript, C#, PHP, and C++. Finally, we label the project
domains of issues with six categories, consistent with the classification of
GitHub project domains proposed by Borges et al. [45]: application software,
system software, web libraries and frameworks, non-web libraries and
frameworks, software tools, and documentation. The classification of GitHub
repositories is widely recognized and used [46, 47]. Two authors independently
mark these issues. The inter-rater reliability between the two authors is
assessed through Cohen’s Kappa ($\kappa$), which yielded values of 0.98, 0.86,
1, and 0.91 for task type (code/document), task type
(corrective/adaptive/perfective), programming language, and project domain,
respectively. To resolve any conflicts that arise during the labeling process,
an arbitrator with top-tier software engineering conference paper publications
and four years of OSS development experience is involved until all labels are
agreed upon within the labeling group. We analyze the proportion of different
issue types to investigate the tendency of each of the 37 newcomers toward the
FIs.

###  III-B RQ2 Methodology

The reinforcement theory [48] states, behavior is driven by its consequences
and positive reinforcements (rewards) serve as facilitators that increase the
likelihood of the desired behavior’s recurrence. Based on this theory, we
suspect that newcomers tend to select FIs that are similar to their historical
contributions to replicate their successes. Moreover, newcomers with different
OSS experiences may have varying preferences in terms of issue types and
difficulty levels. Thus, we investigated features that may influence
newcomers’ choices of FIs. Table I shows the feature groups and the rationale
for choosing them.

TABLE I: The feature groups we choose and the justifications for choosing them Feature Groups | Why can the feature group possibly influence the newcomers’ choice of FIs?
---|---
Expertise Preference
Content Preference | Features of this dimension can reflect similarities between newcomers’ historical contributions and candidate issues regarding their task types and programming languages.
Domain Preference | Features of this dimension can gauge newcomers’ preference for the relevant project domain.
General OSS Experience | Newcomers with more OSS experience may exhibit a preference for tackling more challenging tasks.
Activeness | The resolution of specific types of issues and tasks may require a prolonged period of time. Activeness measures how active the newcomers have been on GitHub recently. According to Wang et al. [39], activeness, i.e., recent activity, can be used to measure the availability of newcomers.
Sentiment | Studies [4, 25] reveal that interactions with experienced developers can be advantageous for newcomers in resolving issues. We posit that newcomers who demonstrate a more positive communication style are more likely to receive aid from experienced developers when grappling with complex issues.

####  III-B1 Data Collection

We use the same labeled data from the 123 FIs resolved by the 37 newcomers in
our analysis of RQ1. We collect the following features of newcomers regarding
their expertise preference, general OSS experience, activeness, and sentiment.

Expertise preference features. As explained in Table I, we use content
preference and domain preference to measure the FI preference of newcomers in
terms of task type, programming language, and project domain.

Textual data collection and preprocessing to calculate content and domain
preference. We calculate content preference by comparing the title and
description of candidate issues with reported issues, resolved issues, PRs,
and commit messages made by newcomers in history. Similarly, we calculate
domain preference by comparing READMEs, topics, descriptions of candidate
issues’ projects and the projects that the newcomer has contributed to.

For issues and PRs, we first concatenate their title and description and then
remove URLs, code snippets, and images to extract plain text. Next, we reduce
noise by applying textual document processing steps, including word
segmentation, lemmatization, and stop word filtering, which are commonly used
in software engineering studies [39, 49]. Thus, we can calculate Jaccard
similarity between the texts.

We also calculate cosine similarity between texts using pre-trained language
models like bidirectional encoder representations from transformers (BERT)
[50]. However, the embeddings created by BERT are highly anisotropic and
semantically non-smoothing, making it difficult to apply directly to
similarity metrics [51, 52]. Simple contrastive learning of sentence
embeddings (SimCSE) [51] is proposed to address this problem, which
regularizes the pre-trained embeddings in an anisotropic space by optimizing a
contrastive learning objective. This encourages similar embeddings to be
closer in the embedding space and dissimilar embeddings to be farther apart,
making the embeddings more uniform. We adopt SimCSE to extract features from
texts using the uncased SimCSE model [53] to tokenize and pad the texts and
generate embeddings of the texts with pooler output.

Calculating content preference. Following textual data preprocessing, we
utilize Jaccard and cosine similarity to measure the similarity of the used
terms and the semantic similarity of texts, respectively. For example, the
Jaccard and cosine similarities between a newcomer’s cumulative PRs and the
candidate issue are calculated as follows.

Cumulative PR cosine similarity calculates and accumulates cosine similarity
between the embedding vectors of each historical PR made by newcomer $d$ and
candidate issue $s$:

$\footnotesize
PRCos(d,s)=\sum\limits_{r=1}^{R}\frac{\sum\limits_{n=1}^{N}PRemb_{r,n}(d)ISSemb_{n}(s)}{\sqrt{\sum\limits_{n=1}^{N}PRemb^{2}_{r,n}(d)}\sqrt{\sum\limits_{n=1}^{N}ISSemb^{2}_{n}(s)}}$
(1)

where $R$ is the number of PRs, $N$ is number of dimensions of embedding
vector, $PRemb_{r,n}(d)$ is the $n$th dimensional value of embedding vector of
the $r$th PR made by newcomer $d$, and $ISSemb_{n}(s)$ is the $n$th
dimensional value of embedding vector of candidate issue $s$.

Cumulative PR Jaccard similarity calculates the sum of Jaccard similarity
between historical PRs from newcomer $d$ and candidate issue $s$:

$\footnotesize PRJac(d,s)=\sum\limits_{r=1}^{R}\frac{A_{r}(d)\cap
B(s)}{A_{r}(d)\cup B(s)},$ (2)

where $A_{r}(d)$ is the term set of preprocessed text of the $r$th PR made by
newcomer $d$, $B(s)$ is the term set of preprocessed text of issue $s$, $\cap$
represents intersection and $\cup$ denotes union.

Besides calculating the cosine and Jaccard similarities, we also calculate the
cumulative number of relevant PRs.

Cumulative number of relevant PRs count the number of PRs from newcomer $d$
that have common labels with the candidate issue $s$:

$\footnotesize PRLab(d,s)=\sum\limits_{r=1}^{R}int((PRlabels_{r}(d)\cap
Isslabels(s))\neq\emptyset),$ (3)

where $PRlabels_{r}(d)$ denotes the label set of the $r$th PR made by newcomer
$d$, and $Isslabels(s)$ is the label set of candidate issue $s$.

Then we calculate the average values of these measurements by dividing the
cumulative values by the number of PRs:
$PRJac/Cos/Lab\\_ave(d,s)=\frac{PRJac/Cos/Lab(d,s)}{R}$. The cumulative
measurements imply whether the newcomer has acquired sufficient experience in
the relevant content, while the average similarity measurements showcase the
overall similarity between the newcomer’s historical contributions and the
candidate issue.

We apply the same similarity measurements as for PRs to analyze the similarity
of historical commit messages, resolved issues, reported issues, commented
issues, and reviewed PRs to candidate issues, except for commit messages
lacking label-related data.

We assess the newcomer’s familiarity with the programming language of the
candidate issues by counting the number of commits they have made in prior
projects that share the same primary programming language as the candidate
issue.

Calculating domain preference. We collect the descriptions and READMEs of the
projects corresponding to the candidate issues and the historical commits of
the newcomers. Then we calculate the aforementioned similarity measurements
between the descriptions and the READMEs, except for the label-related
measures.

To measure the newcomers’ accumulated experience on the topics of the
candidate issues, we record the number and ratio of commits made by them in
projects that share at least one common topic with the project to which the
candidate issue belongs.

General OSS experience features. According to Argote and Miron-Spektor [54],
developer experience can be measured by the cumulative number of tasks
performed. We measure the general cumulative OSS experience of newcomers
through the quantification of their contributions, including commits, PRs, PR
reviews, repositories contributed to, and issues reported. Additionally, we
consider the number of issues they have reported in the new project as an
indication of their interest.

Activeness features. To gauge newcomer activeness, we count the number of
their recent contributions within the past one, two, and three months,
including the number of commits, PRs, and reported issues on GitHub.

Sentiment features. We calculate sentiment measures for newcomers with the
Python TextBlob package [55] using historical PRs’ and issues’ titles and
descriptions. The measurements include the mean and median textual sentiment
polarity values of their historically contributed PRs and reported issues.

####  III-B2 Data Analysis

We first make pairwise comparisons between each FI and the issues resolved by
the same newcomer before resolving the FI, as well as between each FI and the
FIs resolved by the other 36 newcomers. The comparisons are conducted by
calculating the cosine similarity and Jaccard similarity of the issues’ texts
and corresponding project descriptions. By doing so, we measure both the
content and domain similarity of the issues to explore whether the newcomers
tend to choose FIs that are similar to their historical contributions. We then
collect features of the general OSS experience, activeness, and sentiment
groups for the 37 newcomers. We compare the distribution of the features
corresponding to FIs of different task types and domains to explore the
relationship between these features and the FIs.

###  III-C RQ1 Results: Newcomers’ Different FI Preferences

Figure 1: Task type distribution of FIs Figure 2: Programming language
distribution of FIs

Figure 1 to 3 show the percentage of different types of FIs resolved by each
newcomer, which offer insights into their FI preferences. Each of the 37 bars
depicted in each figure corresponds to one of the newcomers.

Task type. Manual labeling of the 123 FIs shows that 69 FIs are code related,
whereas only 54 FIs are document related. Figure 1 reveals the task type
distribution of the issues resolved by the 37 newcomers, demonstrating that 11
newcomers exclusively resolved code-related issues, while seven newcomers
dedicated their efforts solely to document-related issues in new projects.
This finding suggests that simply recommending document-related issues to
newcomers may not be an optimal strategy. Figure 1 also shows that 20
newcomers focused on a single type of corrective, adaptive, or perfective
issue, indicating their consistent preference for task type in new projects.

Programming language. As depicted in Figure 2, among the 37 newcomers, ten
exclusively addressed Python FIs, four exclusively resolved JavaScript FIs,
and ten only resolved JavaScript and TypeScript FIs. Notably, no newcomer
submitted initial contributions in more than three programming languages,
indicating a preference for leveraging their expertise in specific programming
languages when contributing.

Project domain. Figure 3 shows that most newcomers concentrated their efforts
on particular domains, with six newcomers focusing on addressing FIs from
projects in a single domain and 24 newcomers dedicating their efforts to
resolving issues within two domains. Only seven newcomers resolved FIs across
three domains, and none distributed their efforts across more than three
domains. Note that none of the 123 FIs belong to documentation projects.

Figure 3: Project domain distribution of FIs

Summary for RQ1: Regarding task type, programming language, and project
domain, newcomers tend to exhibit consistent preferences in their first issues
of new projects.

###  III-D RQ2 Results: Newcomer Features Influencing FI choices

(a) Distribution of similarity measurements (issues’ cosine/Jaccard
similarity, project descriptions’ cosine/Jaccard similarity)

(b) Distribution of newcomer features across different task types

(c) Distribution of newcomer features across different project domains

Figure 4: Example of features that are related to newcomer FI preferences.

Newcomers’ content and domain preference influence their FI selection. Figure
4(a) demonstrates that the FIs chosen by the newcomers have greater textual
similarity (content preference) and project description similarity (domain
preference) with their own historically resolved issues compared to FIs of
other newcomers, measured using the median cosine and Jaccard similarity.
These findings indicate that FIs are more similar to newcomers’ historically
resolved issues than to others’ FIs. Thus, recommending issues to newcomers
that are similar to their previous contributions may be more efficient and
appropriate than suggesting identical issues to all newcomers.

Newcomers’ general OSS experience, activeness, and sentiment influence their
FI selection. The results indicate that 19 features associated with the 37
newcomers’ general OSS experience, activeness, and sentiment are related to
the newcomers’ FI selection. In this section, we use the number of commits,
the number of commits within the past month, and the mean of the textual
sentiment polarity values for reported issues as example features for the
three feature groups to show their relationship with newcomers’ FI choices.
More results are included in the replication package.

Figure 4(b) reveals that newcomers who have made more commits and are more
active recently than their peers tend to choose document rather than code
issues. Figure 4(b) also shows that newcomers with less coding experience and
more negative sentiment characteristics than their peers prefer adaptive
rather than corrective or perfective issues. Moreover, Figure 4(c) reveals
that newcomers with more coding experience and greater activeness tend to
resolve FIs of system software and software tool projects. Newcomers who
contributed to FIs of web library and framework projects tend to exhibit more
positive sentiment characteristics.

Summary for RQ2: Newcomers’ expertise preference, and their general OSS
experiences, activeness, and sentiment characteristics are related to their
choices of FIs.

##  IV The PFIRec implementation and evaluation

Building on the insights of RQ1 and the identified influential factors of RQ2,
we propose a personalized ranking model named Personalized First Issue
Recommender (PFIRec), which aims to provide a specific newcomer in new
projects with a tailored ranking of candidate issues that are most suitable
for the newcomer. In this section, we formulate the problem, explain how the
model is constructed, trained, and evaluated, and present the evaluation
results.

###  IV-A Problem Formulation

The primary goal of this research is to develop an automated recommendation
system that suggests appropriate FIs in a new project for a given newcomer
$d_{i}\in D=\\{d_{1},\ldots,d_{I}\\}$ from a set of candidate issues
$S=\\{s_{1},\ldots,s_{N}\\}$ available in the project. Our aim is to rank each
issue in order of the probability that it is suitable for the respective
newcomer $d_{i}$. For each issue $s_{n}$, relevant factors can be extracted
for the issue-newcomer pair $(s_{n},d_{i})$ to construct an input vector
$\bm{x}_{n,i}$. We formulate this as an automated new project first issue
recommendation problem: $y_{n,i}=f(\bm{x}_{n,i})\in[0,1]$, where $y_{n,i}$
represents the probability that the issue $s_{n}$ can be resolved by the
newcomer $d_{i}$, and $\bm{x}_{n,i}\in\mathbb{R}^{c}$ denotes the input vector
comprising $c$ features. These features include both characteristics
influencing a newcomer’s choice of their FI and characteristics of candidate
issues. The objective is to learn the model $f(\cdot)$ using training pairs
$(\bm{x}_{n,i},y_{n,i})$ constructed from historical issue-newcomer pairs
$(s_{n},d_{i})\in S\times D$, then utilize the trained model to predict the
probabilities that candidate issues are suitable for a given newcomer.
Finally, the candidate issues are recommended in descending order of
probability for the newcomer.

###  IV-B Model Training Data Specification

To train the model $f(\cdot)$, we need to construct an input vector
$\bm{x}_{n,i}$ for each issue-newcomer pair $(s_{n},d_{i})$ by pairing each
newcomer with each candidate issue and extracting relevant factors. Regarding
$\bm{x}_{n,i}\in\mathbb{R}^{c}$, we follow the findings of RQ2 and collect all
newcomer-related features presented in Section III-B1.

In addition, to build an effective model for issue recommendation, it is also
important to extract relevant features from candidate issues. Previous studies
[4, 5], have identified features that may indicate the difficulty, importance,
and urgency of an issue based on its content and background. Therefore, we
collect the following issue-related features:

Content of candidate issues. The content-related features consist of sentiment
polarity, readability metrics (including Coleman-Liau formula, Kincaid Grade
Level, Flesch Reading Ease, and Automated Readability Index), the length of
the issue title, description, number of code snippets, URLs, images, the total
number of labels, and the numbers of labels belonging to the 12 label
categories according to [4]: Bug, Documentation, Test, Build, Enhancement,
Coding, New Feature, Newcomer-friendly (e.g., good first issue, easy labels),
Medium Difficulty, Difficult, Triaged, Untriaged. These features of issues may
indicate their type and difficulty.

Background of candidate issues. These features include:

  1. 1.

Information of the project to which the issue belongs: number and ratio of
open issues, number and ratio of GFIs, number of commits, PRs, closed issues,
stars, and commit contributors, all of which can indicate whether the tasks
provided by the project are sufficient for newcomers and whether the project
is likely to attract many newcomers to compete for a candidate issue;

  2. 2.

Profile of the issue reporter: number of commits, PRs, reviewed PRs, reported
issues, the ratio of GFIs among all reported issues in the project, maximum
stars of contributed projects, and whether the reporter has commits in the
project, all of which measures their experience and may be related to the
difficulty of candidate issues [4];

  3. 3.

Profile of the project owner, using the identical metrics as those for the
issue reporter.

###  IV-C Ranking Model

To rank the candidate issues based on the newcomers’ individual features, we
use LambdaMART [10], a state-of-the-art learning-to-rank model, as our ML
model for training and testing. LambdaMART is a well-established machine
learning technique that has demonstrated its effectiveness in software
engineering sorting tasks [56, 57]. It is the MART (Multiple Additive
Regression Tree)-based boosted tree [58] version of LambdaRank [59], which is
a learning-to-rank algorithm derived from RankNet [60]. LambdaRank redefines
the gradient of the objective function and considers ranking evaluators to
improve the performance of RankNet. The use of MART in the LambdaMART
algorithm enables it to perform gradient descent in the function space. The
prediction value is a linear combination of the outputs of the regression
trees [10]. Our experiment results demonstrate the superior effectiveness of
LambdaMART in sorting issues for newcomers.

###  IV-D Model Training

Data collection and labeling. To prepare the training data, we create lists
that contain the features (inputs) and ground truth labels (outputs) of
candidate issue-newcomer pairs. This is achieved by extracting all features
($\bm{x}_{n,i}$) and assigning the corresponding ground truth label $y_{n,i}$
for issue $s_{n}$ and newcomer $d_{i}$ via the process illustrated in Figure
5.

Figure 5: Feature extraction and ground truth setting.

We consider FIs resolved in our chosen projects along with other issues that
remain open until the FIs are resolved as candidate issues for the resolver of
the FIs. Thus, for each of the 13,266 FIs identified in Section III-A1, we
constructed a candidate issue list. For each list, we pair each candidate
issue with the resolver of the corresponding FI. To obtain the aforementioned
features for each pair, we extract newcomer data using the GitHub GraphQL API
[61] and issue data from the GFI-Bot dataset. The pair corresponding to the FI
is labeled one, while the remaining pairs of candidate issues are labeled
zero.

As the automatic issue recommendation approach has limited practical value for
projects with only a few candidate issues, we exclude the lists from projects
containing fewer than ten candidate issues. This exclusion accounts for
approximately one-eighth of the total list count in our dataset, resulting in
11,615 lists covering a total of 68,858 issues (with some issues appearing in
multiple lists). The median number of candidate issues per list is 32.

Process. The practical application scenario envisioned for PFIRec involves
utilizing training data collected based on historically resolved issues to
train a model for ranking opening candidate issues. After some opening
candidate issues are resolved regularly, PFIRec can update the training data
to include newly resolved issues and retrain the model. To simulate such
scenarios, we adopt a longitudinal data configuration that has been widely
adopted in related studies [62, 39, 5]. In our study, we sort the 11,615 lists
based on their corresponding FIs’ closing time in descending order and divide
them into 20 equally sized folds (with additional lists included in the last
fold). As illustrated in Figure 6, we designate the first $T-2$ fold(s) as the
training set, the $(T-1)$th fold as the validation set, and the $T$th fold as
the test set. Then, we slide the window one fold and repeat the process for
$T$ values ranging from 3 to 20 to train, validate, and test 18 models. For
each model, we use its corresponding test set to evaluate the model’s
performance.

Using the same feature extraction process as for the training and validation
sets, we collect features for each newcomer in the test set and predict the
likelihood of the newcomer resolving each candidate issue with the trained
model. The resulting probabilities are used to rank the candidate issues in
descending order. The ranked issues are presented by PFIRec and are evaluated.

Figure 6: Building the datasets with the chronological FIs.

###  IV-E Evaluation

In the evaluation, we aim to answer the following three evaluation questions
(EQs).

  * •

EQ1: How does the performance of PFIRec compare with its variants that employ
different language and ranking models? This is to justify our choice of
language and ranking model to build PFIRec.

  * •

EQ2: How does the effectiveness of PFIRec in recommending issues to newcomers
compare to the baselines? The purpose is to evaluate the performance
improvement of PFIRec compared to state-of-the-art approaches.

  * •

EQ3: How sensitive is PFIRec to different feature groups and cross-project
scenarios? The purpose is to gain a deeper understanding of the impact of
various feature groups on prediction and evaluate PFIRec’s generalizability in
cross-project scenarios.

Evaluation Metrics. The top-$k$ recall ($R@k$) and FirstHit ($FH$) are popular
ranking metrics to evaluate recommendation performance [63, 64, 39]. We employ
these metrics to measure how early the models can recommend the “correct” FIs.
Specifically, the top-$k$ recall calculates the ratio of successfully
predicted “correct” FIs to the total number of “correct” FIs in the test set:
$R@k=\frac{\sum_{i=1}^{m}n_{i}@k}{\sum_{i=1}^{m}n_{i}}$, where $n_{i}@k$
denotes the number of “correct” FIs in the top-$k$ of the recommendation list
for the $i$th newcomer and $n_{i}$ represents the number of “correct” FIs for
the $i$th newcomer. Regarding our problem, only one issue in a candidate issue
list is the newcomer’s “correct” FI, so the top-$k$ recall simplifies to
$R@k=1$ if the matching FI is ranked in the top $k$ of the recommendation
list, and $R@k=0$ otherwise. PFIRec aims to assist newcomers in identifying
suitable issues at an early stage, thereby minimizing their erroneous
attempts. We use $FH$ (FirstHit) as a metric to determine how early newcomers
can identify suitable issues using PFIRec. $FH$ records the median rank of the
“correct” FI in each recommendation list, and $FH-1$ represents the number of
possibly unsuccessful attempts by the newcomer before finding a suitable
issue.

###  IV-F EQ1 Results

To demonstrate the competitiveness of SimCSE and LambdaMART for our problem
compared to other BERT-based language models and non-dedicated learning-to-
rank models, we compare PFIRec’s performance with various variants that use
different language and ML models.

As introduced in Section III-B1, we employ a BERT-based pre-trained language
model for text embedding extraction. We compare PFIRec’s ranking performance
using SimCSE and the following three other BERT-based pre-trained models as
feature extractors in Figure 7.

  * •

RoBERTa [65] is one of the most popular BERT-based language models pre-trained
with larger batch size and learning rates than the original BERT.

  * •

CodeBERT [66] is pre-trained based on the checkpoint of RoBERTa with the
CodeSearchNet dataset [67] and can be used for both natural and programming
languages.

  * •

BERTOverflow [68] is pre-trained with sentences and tokens from Stack Overflow
and has demonstrated effectiveness in the software engineering domain.

It can be seen that all four models perform similarly and that the SimCSE-
based model achieves the best performance on $FH$, likely due to its
optimization for similarity measurement problems.

Figure 7: Performance of different language models. Figure 8: Performance of
different ML models. Figure 9: Performance of PFIRec and baselines.

As explained in Section IV-C, we employ LambdaMART as the machine learning
model to rank candidate issues. An alternate method is to utilize a non-
dedicated learning-to-rank model to estimate the probability of issues being
suitable for specific newcomers and rank candidate issues based on the
probability. In Figure 8, we compare LambdaMART’s predictive performance with
that of three other ML models, including the eXtreme Gradient Boosting
(XGBoost) classifier [69], random forest model, and logistic regression model
from the Python scikit-learn package [70]. The XGBoost model has the best
performance among the three models, but is still slightly inferior to
LambdaMART. The median $FH$ of LambdaMART is 7, whereas for XGBoost it is 8.
Thus, under our problem setting, the learn-to-rank model LambdaMART is the
preferred choice over the other ML models.

Summary for EQ1: PFIRec, which utilizes the SimCSE language model and the
LambdaMART ML model, shows superior performance in ranking candidate issues
compared to its variants employing other language models and ML models.

###  IV-G EQ2 Results

To evaluate the effectiveness of PFIRec in addressing the challenge of finding
appropriate issues for specific newcomers, we compare it with the following
four baselines.

  * •

Stanik et al. [2]: This approach leverages a random forest-based model that
considers issue title and description length, sentiment score, and term
frequency to identify suitable issues for newcomers.

  * •

GFI-Bot [5]: This is an under-development GitHub bot for automatically
recommending GFIs based on RecGFI [4], which is a state-of-the-art automated
approach. It predicts the probability that an issue suits newcomers and
provides a ranked list of issues based on the probability. We do not choose
RecGFI as a baseline due to its alignment with GFI-Bot’s deployed model.

  * •

Random: This corresponds to randomly ranking candidate issues, which is
equivalent to newcomers’ practice of randomly checking open issues to identify
appropriate ones.

  * •

GFIRandom: This corresponds to the practice of randomly checking open issues
with GFI labels first, and then randomly checking the remaining ones.

As we do not have access to the modified files and code to resolve the issues
in our scenario, we do not compare PFIRec with the automatic GFI
recommendation approach proposed by Huang et al. [3], which incorporates
features of the modification information from the files and code.

Figure 9 shows that the GFIRandom model only marginally outperforms the Random
model, suggesting that manually-labeled GFIs may have limited effectiveness in
assisting newcomers in issue selection. This aligns with the findings of Tan
et al. [17], which suggest that the problem of insufficient GFIs limits the
GFI mechanism’s effectiveness. Among the four baselines, GFI-Bot demonstrates
the best performance. Compared to GFI-Bot, PFIRec reduces the median $FH$ by
3, indicating that it can reduce one-third (6 vs. 9) of newcomers’
unsuccessful attempts to find a FI in the median. Besides, compared with GFI-
Bot, PFIRec improves the median $R@1,3,5,10$ by 114.3% (0.15 vs. 0.07), 55.0%
(0.31 vs. 0.20), 35.5% (0.42 vs. 0.31), and 17.3% (0.61 vs. 0.52),
respectively. Compared to the best practice without utilizing automated
recommendation tools (i.e., GFIRandom), PFIRec can reduce the median of FH by
6, and increase $R@1,3,5,10$ by 150.0% (0.15 vs. 0.06), 121.4% (0.31 vs.
0.14), 90.9% (0.42 vs. 0.22), and 45.2% (0.61 vs. 0.42), respectively.

Summary for EQ2: Compared to state-of-the-art approaches, PFIRec doubles the
probability of the top-recommended issue being suitable for a specific
newcomer, while reducing one-third of a newcomer’s unsuccessful attempts to
identify suitable first issues in the median.

###  IV-H EQ3 Results

To evaluate the impact of each feature group on the model’s predictive
performance, we construct seven variants of the model by removing the Content
preference  (noCont), Domain preference (noDom), General OSS experience
(noGener), Activeness (noAct), Sentiment (noSenti), Issue content (noIssCont),
and Issue background (noIssBack) groups. Since the issues are collected from
the 100 projects introduced in Section III-A, we can conduct cross-project
experiments to assess the model’s generalizability to new projects, i.e., the
prediction performance in projects outside the training set. To achieve this,
we collect training and validation data from 90 projects and test data from
the remaining ten projects. We perform a ten-fold cross-validation on the 100
projects and record its mean as the result. The other experimental steps are
the same as described in Section IV-D.

Table II illustrates that removing any group leads to a retrograde alteration
in the placement of the “correct” FI in the recommendation list (resulting in
an elevated $FH$) and a decrease in the values of $R@1,3,5,10$. Removing Issue
content, Issue background, and General OSS experience has the most significant
impact, resulting in an increase in the median $FH$ by 4, 1, and 1,
respectively. When PFIRec is applied to the cross-project prediction scenario,
a marginal shift is observed in the five evaluation metrics, indicating the
generalizability of PFIRec to new projects.

TABLE II: Performance (mean/median) of variants of PFIRec and PFIRec for the cross-project scenario. Models |  R@1 |  R@3 |  R@5 |  R@10 |  FH
---|---|---|---|---|---
noCont |  0.14/0.14 |  0.30/0.29 |  0.41/0.41 |  0.61/0.61 |  7.6/7.5
noDom |  0.14/0.15 |  0.31/0.31 |  0.42/0.42 |  0.61/0.62 |  7.3/7
noGener |  0.13/0.12 |  0.28/0.28 |  0.40/0.39 |  0.60/0.60 |  7.7/8
noAct |  0.14/0.14 |  0.31/0.31 |  0.41/0.41 |  0.61/0.60 |  7.4/7
noSenti |  0.15/0.15 |  0.31/0.31 |  0.43/0.43 |  0.62/0.61 |  7.3/7.5
noIssCont |  0.14/0.15 |  0.30/0.30 |  0.40/0.40 |  0.60/0.59 |  7.8/8
noIssBack |  0.07/0.08 |  0.19/0.19 |  0.28/0.28 |  0.49/0.48 |  10.9/11
CrossPro |  0.15/0.16 |  0.32/0.32 |  0.44/0.44 |  0.63/0.63 |  7.5/7.5
PFIRec |  0.15/0.15 |  0.31/0.31 |  0.42/0.42 |  0.62/0.61 |  7.2/7

Summary for EQ3: The removal of Issue content, Issue background, and General
OSS experience result in the most significant decrease in model performance.
PFIRec maintains its generalizability in cross-project prediction scenario.

##  V Discussion

###  V-A Comparison with Related Work

Previous studies [36, 37, 38, 39] on matching specific tasks with individual
developers focus on recommending experts for tasks. On the contrary, our study
aims to provide personalized task recommendations to individual project
newcomers, who have not contributed code to the project, to assist in their
onboarding process. To our knowledge, this is the first study to explore
newcomers’ personalized preferences in selecting the first issues to resolve
in new projects. Adopting the PFIRec approach can provide two main benefits
for projects. Firstly, it can help interested newcomers in finding suitable
issues earlier and increase their chances of successfully joining the project.
Secondly, this approach can reduce the likelihood of unsuitable newcomers
attempting GFIs, allowing the issues to attract and retain matched newcomers.
Reducing the number of attempts made by unsuitable newcomers can decrease the
likelihood of rejected PRs and unfavorable communication, which in turn helps
maintain their motivation to participate in the project. Furthermore, our
findings emphasize the importance of experience matching in recommending tasks
for newcomers. Issue reporters and project maintainers can consider practices
such as providing task-type labels and listing required skills for GFIs to
alleviate the difficulty faced by newcomers in finding suitable issues.

Our study can offer insight into future task-developer matching research: 1)
Recommendation context modeling. Previous studies [40, 41] use only skill
proficiency, collaboration history, and contributed projects’ topics to model
developers’ contexts and do not use a learn-to-rank model. Our work confirms
that general OSS experience accumulated by developers and the task’s attribute
features are useful for the recommendation model in addition to the previously
used features; 2) Language model. Our results demonstrate the capability of
the SimCSE language model based on BERT optimized for similarity metrics in
computing text similarity; and 3) Ranking model. Although Wang et al. [39]
also use LambdaMART to rank the recommendation list, our work shows that
LambdaMART outperforms other non-learn-to-rank models in the recommendation
scenario.

Although our personalized recommendation objectives deviate from those for
personalized recommendations of collaborators and repositories in software
engineering [40, 41], our evaluation of distinct language models and ranking
models can provide valuable insights for future research endeavors in their
research direction.

Compared to the state-of-the-art automatic GFI recommendation approach,
searching for issues based on the recommendation lists provided by PFIRec can
significantly reduce newcomers’ unsuitable attempts by up to one-third in the
median before finding a suitable issue. In fact, PFIRec doubles the success
rate of newcomers who only check the first recommended issue.

###  V-B Threats to Validity

Internal Validity. Firstly, some issues on GitHub may not be task-related
[71]. PFIRec cannot distinguish whether an issue is task-associated or not. We
suggest filtering out non-task issues by manual or automated methods to
mitigate this threat. Filtering out non-task issues based on labels can be an
effective strategy for projects that employ well-organized issue labeling.
Secondly, following previous work on automatically recommending GFIs [4], we
only consider the issues resolved by newcomers to be suitable for them.
However, some issues may be suitable for particular newcomers but not seen by
them or resolved by someone else first, resulting in incomplete positive
labeling. However, we ensure that existing positive instances are correctly
labeled. This approach is more appropriate than labeling data based solely on
GFI tags, as it addresses problems regarding sparse GFI tags and potential
mislabeling. Thirdly, newcomers without GitHub experience may have different
preferences for FIs, but PFIRec may recommend the same issues to them.
Collecting newcomer experience data from sources beyond GitHub can mitigate
this risk. Fourthly, the dataset may miss issues resolved by newcomers if they
are not associated with commits or PRs. A tool that automatically associates
commits, PRs, and the corresponding issues can be used to reduce this risk.
Finally, our study treats developers who have not made any contribution to the
project as newcomers. However, developers who have made several contributions
may still want to search for suitable newcomer issues [17]. This can be
addressed by adjusting the ground truth of the dataset according to the
definition of newcomers (e.g., developers with less than a threshold number of
commits in the project are considered newcomers).

External Validity. The results of RQ1 and RQ2 are based on a sample of 37
newcomers with multiple FIs, and the conclusions may not be generalized to
other newcomers. The effectiveness of the identified features in predicting
suitable FIs in a larger number of newcomers in Section IV may partially
mitigate this threat. Future work can investigate the selection of
personalized first issues from more newcomers to verify our findings.
Furthermore, our experiments are carried out on 100 GitHub projects, and the
performance of PFIRec on other projects is unknown. To reduce this risk, we
performed cross-project experiments, demonstrating the high generalization of
PFIRec.

##  VI Conclusion

Newcomers to OSS projects often rely on GFI labels to identify issues that may
suit them. Although some studies propose approaches to tag GFIs automatically,
they do not consider newcomers’ diverse backgrounds, which may result in a
mismatch between their preferences and the identified issues. In this study,
we analyzed the contributions of newcomers who have successfully resolved
issues in multiple new projects and found that the issues resolved by them in
new projects are more similar to their historical contributions than those
resolved by other newcomers. This highlights the need for a personalized
approach to issue recommendations to newcomers. By leveraging newcomer-related
features from four feature groups (expertise preference, general OSS
experience, activeness, and sentiment) along with variables encompassing the
content and background of candidate issues, we propose PFIRec. This novel
approach uses a LambdaMART ranking model to deliver personalized issue
recommendations specifically tailored for newcomers. The effectiveness of
PFIRec is experimentally evaluated and compared with existing approaches that
recommend issues to newcomers, demonstrating its superior performance.

Future research can explore the trajectories of newcomers on GitHub and their
resolved first issues, to further uncover patterns in their first issues
selection. It is also essential to investigate how diverse patterns of first
issue selection impact the subsequent participation of newcomers from
different backgrounds in projects. These studies have the potential to improve
personalized first issue recommenders and provide best practice guidelines for
newcomers when choosing first issues.

## Acknowledgment

This work is supported by the National Natural Science Foundation of China
Grant 61825201 and 62332001.

## References

  * [1] B. Shibuya and T. Tamai, “Understanding the process of participating in open source communities,” in _2009 ICSE Workshop on Emerging Trends in Free/Libre/Open Source Software Research and Development_. IEEE, 2009, pp. 1–6.
  * [2] C. Stanik, L. Montgomery, D. Martens, D. Fucci, and W. Maalej, “A simple nlp-based approach to support onboarding and retention in open source communities,” in _2018 IEEE International Conference on Software Maintenance and Evolution, ICSME 2018, Madrid, Spain, September 23-29, 2018_. IEEE Computer Society, 2018, pp. 172–182.
  * [3] Y. Huang, J. Wang, S. Wang, Z. Liu, D. Wang, and Q. Wang, “Characterizing and predicting good first issues,” in _ESEM ’21: ACM / IEEE International Symposium on Empirical Software Engineering and Measurement, Bari, Italy, October 11-15, 2021_ , F. Lanubile, M. Kalinowski, and M. T. Baldassarre, Eds. ACM, 2021, pp. 13:1–13:12.
  * [4] W. Xiao, H. He, W. Xu, X. Tan, J. Dong, and M. Zhou, “Recommending good first issues in github OSS projects,” in _44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022_. IEEE, 2022, pp. 1830–1842.
  * [5] H. He, H. Su, W. Xiao, R. He, and M. Zhou, “Gfi-bot: automated good first issue recommendation on github,” in _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November 14-18, 2022_ , A. Roychoudhury, C. Cadar, and M. Kim, Eds. ACM, 2022, pp. 1751–1755.
  * [6] Issue event types - github docs. [Online]. Available: https://docs.github.com/en/webhooks-and-events/events/issue-event-types
  * [7] hediet. (2022) Support <…> bracket matching in typescript · issue #151196 · microsoft/vscode. [Online]. Available: https://github.com/microsoft/vscode/issues/151196
  * [8] MarcoGorelli. (2022) Ci check each minimum dependency is enforced in *.yaml and environment.yml files · issue #50207 · pandas-dev/pandas. March 17, 2023. [Online]. Available: https://github.com/pandas-dev/pandas/issues/50207
  * [9] (2022) Op ed: Want to learn how to contribute to bitcoin? try a ‘good first issue’ - bitcoin magazine. [Online]. Available: https://bitcoinmagazine.com/technical/op-ed-want-to-learn-how-to-contribute-to-bitcoin-try-a-good-first-issue
  * [10] C. J. Burges, “From ranknet to lambdarank to lambdamart: An overview.”
  * [11] A. Mockus, R. T. Fielding, and J. D. Herbsleb, “Two case studies of open source software development: Apache and mozilla,” _ACM Transactions on Software Engineering and Methodology (TOSEM)_ , vol. 11, no. 3, pp. 309–346, 2002\.
  * [12] M. Zhou and A. Mockus, “What make long term contributors: Willingness and opportunity in OSS community,” in _34th International Conference on Software Engineering, ICSE 2012, June 2-9, 2012, Zurich, Switzerland_. IEEE Computer Society, 2012, pp. 518–528.
  * [13] (2022) Volkan Yazıcı on Twitter. [Online]. Available: https://twitter.com/yazicivo/status/1469349956880408583?lang=en
  * [14] (2014) The Internet is being protected by two guys named steve. [Online]. Available: https://www.buzzfeed.com/chrisstokelwalker/the-internet-is-being-protected-by-two-guys-named-st
  * [15] (2022) Google online security blog: Understanding the impact of Apache Log4j vulnerability. [Online]. Available: https://security.googleblog.com/2021/12/understanding-impact-of-apache-log4j.html
  * [16] (2020) Heartbleed bug. [Online]. Available: https://heartbleed.com/
  * [17] X. Tan, M. Zhou, and Z. Sun, “A first look at good first issues on github,” in _ESEC/FSE ’20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020_. ACM, 2020, pp. 398–409.
  * [18] J. Coelho and M. T. Valente, “Why modern open source projects fail,” in _Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Paderborn, Germany, September 4-8, 2017_ , E. Bodden, W. Schäfer, A. van Deursen, and A. Zisman, Eds. ACM, 2017, pp. 186–196.
  * [19] Y. Ye and K. Kishida, “Toward an understanding of the motivation open source software developers,” in _Proceedings of the 25th International Conference on Software Engineering_ , ser. ICSE ’03. USA: IEEE Computer Society, 2003, p. 419–429.
  * [20] G. Krogh, S. Haefliger, S. Spaeth, and M. Wallin, “Carrots and rainbows: Motivation and social practice in open source software development,” _MIS Quarterly_ , vol. forthcoming, 06 2012.
  * [21] I. Steinmacher, M. A. G. Silva, and M. A. Gerosa, “Barriers faced by newcomers to open source projects: a systematic review,” in _Open Source Software: Mobile Open Source Technologies: 10th IFIP WG 2.13 International Conference on Open Source Systems, OSS 2014, San José, Costa Rica, May 6-9, 2014. Proceedings 10_. Springer, 2014, pp. 153–163.
  * [22] I. Steinmacher, C. Treude, and M. A. Gerosa, “Let me in: Guidelines for the successful onboarding of newcomers to open source projects,” _IEEE Softw._ , vol. 36, no. 4, pp. 41–49, 2019.
  * [23] C. Jensen, S. King, and V. Kuechler, “Joining free/open source software communities: An analysis of newbies’ first interactions on project mailing lists,” in _44th Hawaii International International Conference on Systems Science (HICSS-44 2011), Proceedings, 4-7 January 2011, Koloa, Kauai, HI, USA_. IEEE Computer Society, 2011, pp. 1–10.
  * [24] F. Santos, B. Trinkenreich, J. F. Pimentel, I. Wiese, I. Steinmacher, A. Sarma, and M. A. Gerosa, “How to choose a task? mismatches in perspectives of newcomers and existing contributors,” in _ESEM ’22: ACM / IEEE International Symposium on Empirical Software Engineering and Measurement, Helsinki Finland, September 19 - 23, 2022_ , F. Madeiral, C. Lassenius, T. Conte, and T. Männistö, Eds. ACM, 2022, pp. 114–124.
  * [25] X. Tan, Y. Chen, H. Wu, M. Zhou, and L. Zhang, “Is it enough to recommend tasks to newcomers? understanding mentoring on good first issues,” in _45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023_. IEEE, 2023, pp. 653–664.
  * [26] H. He, M. Zhou, Q. Wang, and J. Li, “Open source software onboarding as a university course: An experience report,” in _45th IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training, SEET@ICSE 2023, Melbourne, Australia, May 14-20, 2023_. IEEE, 2023, pp. 324–336.
  * [27] M. Zhou and A. Mockus, “Who will stay in the FLOSS community? Modeling participant’s initial behavior,” _IEEE Trans. Software Eng._ , vol. 41, no. 1, pp. 82–99, 2015.
  * [28] S. Panichella, “Supporting newcomers in software development projects,” in _2015 IEEE International Conference on Software Maintenance and Evolution, ICSME 2015, Bremen, Germany, September 29 - October 1, 2015_ , R. Koschke, J. Krinke, and M. P. Robillard, Eds. IEEE Computer Society, 2015, pp. 586–589.
  * [29] I. Steinmacher, T. U. Conte, and M. A. Gerosa, “Understanding and supporting the choice of an appropriate task to start with in open source software communities,” in _48th Hawaii International Conference on System Sciences, HICSS 2015, Kauai, Hawaii, USA, January 5-8, 2015_ , T. X. Bui and R. H. S. Jr., Eds. IEEE Computer Society, 2015, pp. 5299–5308.
  * [30] Y. Park and C. Jensen, “Beyond pretty pictures: Examining the benefits of code visualization for open source newcomers,” in _Proceedings of the 5th IEEE International Workshop on Visualizing Software for Understanding and Analysis, VISSOFT 2009, Edmonton, Alberta, Canada, September 25, 2009_ , H. A. Müller, M. Lanza, and M. D. Storey, Eds. IEEE Computer Society, 2009, pp. 3–10.
  * [31] J. Wang and A. Sarma, “Which bug should I fix: helping new developers onboard a new project,” in _Proceedings of the 4th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 2011, Waikiki, Honolulu, HI, USA, May 21, 2011_ , M. Cataldo, C. R. B. de Souza, Y. Dittrich, R. Hoda, and H. Sharp, Eds. ACM, 2011, pp. 76–79.
  * [32] GitHub, Inc. (2023) Encouraging helpful contributions to your project with labels. [Online]. Available: https://docs.github.com/en/communities/setting-up-your-project-for-healthy-contributions/encouraging-helpful-contributions-to-your-project-with-labels
  * [33] H. Horiguchi, I. Omori, and M. Ohira, “Onboarding to open source projects with good first issues: A preliminary analysis,” in _28th IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2021, Honolulu, HI, USA, March 9-12, 2021_. IEEE, 2021, pp. 501–505.
  * [34] J. W. D. Alderliesten and A. Zaidman, “An initial exploration of the ”good first issue” label for newcomer developers,” in _14th IEEE/ACM International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE@ICSE 2021, Madrid, Spain, May 20-21, 2021_. IEEE, 2021, pp. 117–118.
  * [35] W. Xiao, H. He, W. Xu, Y. Zhang, and M. Zhou, “How early participation determines long-term sustained activity in github projects?” _arXiv preprint arXiv:2308.06005_ , 2023.
  * [36] J. Anvik and G. C. Murphy, “Reducing the effort of bug report triage: Recommenders for development-oriented decisions,” _ACM Trans. Softw. Eng. Methodol._ , vol. 20, no. 3, pp. 10:1–10:35, 2011.
  * [37] B. Ashok, J. M. Joy, H. Liang, S. K. Rajamani, G. Srinivasa, and V. Vangala, “Debugadvisor: a recommender system for debugging,” in _Proceedings of the 7th joint meeting of the European Software Engineering Conference and the ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2009, Amsterdam, The Netherlands, August 24-28, 2009_ , H. van Vliet and V. Issarny, Eds. ACM, 2009, pp. 373–382.
  * [38] C. Macdonald and I. Ounis, “Voting for candidates: adapting data fusion techniques for an expert search task,” in _Proceedings of the 2006 ACM CIKM International Conference on Information and Knowledge Management, Arlington, Virginia, USA, November 6-11, 2006_ , P. S. Yu, V. J. Tsotras, E. A. Fox, and B. Liu, Eds. ACM, 2006, pp. 387–396.
  * [39] J. Wang, Y. Yang, S. Wang, Y. Hu, D. Wang, and Q. Wang, “Context-aware in-process crowdworker recommendation,” in _ICSE ’20: 42nd International Conference on Software Engineering, Seoul, South Korea, 27 June \- 19 July, 2020_ , G. Rothermel and D. Bae, Eds. ACM, 2020, pp. 1535–1546.
  * [40] L. Ye, H. Sun, X. Wang, and J. Wang, “Personalized teammate recommendation for crowdsourced software developers,” in _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, September 3-7, 2018_ , M. Huchard, C. Kästner, and G. Fraser, Eds. ACM, 2018, pp. 808–813.
  * [41] Y. Zhou, J. Wu, and Y. Sun, “Ghtrec: A personalized service to recommend github trending repositories for developers,” in _2021 IEEE International Conference on Web Services, ICWS 2021, Chicago, IL, USA, September 5-10, 2021_ , C. K. Chang, E. Daminai, J. Fan, P. Ghodous, M. Maximilien, Z. Wang, R. Ward, and J. Zhang, Eds. IEEE, 2021, pp. 314–323.
  * [42] (2022) What makes a ”good first issue”? - dev community. [Online]. Available: https://dev.to/cerchie/what-makes-a-good-first-issue-4fn0
  * [43] (2022) Observations on finding good first issues. [Online]. Available: https://blog.scottlogic.com/2022/08/16/observations-on-finding-good-first-issues.html
  * [44] A. Mockus and L. G. Votta, “Identifying reasons for software changes using historic databases,” in _2000 International Conference on Software Maintenance, ICSM 2000, San Jose, California, USA, October 11-14, 2000_. IEEE Computer Society, 2000, pp. 120–130.
  * [45] H. Borges, A. C. Hora, and M. T. Valente, “Understanding the factors that impact the popularity of github repositories,” in _2016 IEEE International Conference on Software Maintenance and Evolution, ICSME 2016, Raleigh, NC, USA, October 2-7, 2016_. IEEE Computer Society, 2016, pp. 334–344.
  * [46] F. Zanartu, C. Treude, B. Cartaxo, H. S. Borges, P. Moura, M. Wagner, and G. Pinto, “Automatically categorising github repositories by application domain,” _CoRR_ , vol. abs/2208.00269, 2022.
  * [47] H. Borges and M. T. Valente, “What’s in a github star? understanding repository starring practices in a social coding platform,” _J. Syst. Softw._ , vol. 146, pp. 112–129, 2018.
  * [48] C. B. Ferster and B. F. Skinner, “Schedules of reinforcement.” 1957.
  * [49] G. Huang, Y. Li, Q. Wang, J. Ren, Y. Cheng, and X. Zhao, “Automatic classification method for software vulnerability based on deep neural network,” _IEEE Access_ , vol. 7, pp. 28 291–28 298, 2019.
  * [50] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [51] T. Gao, X. Yao, and D. Chen, “Simcse: Simple contrastive learning of sentence embeddings,” in _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , M. Moens, X. Huang, L. Specia, and S. W. Yih, Eds. Association for Computational Linguistics, 2021, pp. 6894–6910.
  * [52] B. Li, H. Zhou, J. He, M. Wang, Y. Yang, and L. Li, “On the sentence embeddings from pre-trained language models,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , B. Webber, T. Cohn, Y. He, and Y. Liu, Eds. Association for Computational Linguistics, 2020, pp. 9119–9130.
  * [53] (2021) princeton-nlp/sup-simcse-bert-base-uncased · hugging face. [Online]. Available: https://huggingface.co/princeton-nlp/sup-simcse-bert-base-uncased
  * [54] L. Argote and E. Miron-Spektor, “Organizational learning: From experience to knowledge,” _Organ. Sci._ , vol. 22, no. 5, pp. 1123–1137, 2011.
  * [55] (2022) Textblob: Simplified text processing. [Online]. Available: https://textblob.readthedocs.io/en/dev/index.html
  * [56] Z. Cao, Y. Tian, T. B. Le, and D. Lo, “Rule-based specification mining leveraging learning to rank,” _Autom. Softw. Eng._ , vol. 25, no. 3, pp. 501–530, 2018.
  * [57] Y. Zhou, X. Yang, T. Chen, Z. Huang, X. Ma, and H. C. Gall, “Boosting API recommendation with implicit feedback,” _IEEE Trans. Software Eng._ , vol. 48, no. 6, pp. 2157–2172, 2022.
  * [58] J. H. Friedman, “Greedy function approximation: a gradient boosting machine,” _Annals of statistics_ , pp. 1189–1232, 2001.
  * [59] C. J. C. Burges, R. Ragno, and Q. V. Le, “Learning to rank with nonsmooth cost functions,” in _Advances in Neural Information Processing Systems 19, Proceedings of the Twentieth Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 4-7, 2006_ , B. Schölkopf, J. C. Platt, and T. Hofmann, Eds. MIT Press, 2006, pp. 193–200.
  * [60] C. J. C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. N. Hullender, “Learning to rank using gradient descent,” in _Machine Learning, Proceedings of the Twenty-Second International Conference (ICML 2005), Bonn, Germany, August 7-11, 2005_ , ser. ACM International Conference Proceeding Series, L. D. Raedt and S. Wrobel, Eds., vol. 119. ACM, 2005, pp. 89–96.
  * [61] (2022) Graphql api. [Online]. Available: https://docs.github.com/en/graphql
  * [62] A. Tamrawi, T. T. Nguyen, J. M. Al-Kofahi, and T. N. Nguyen, “Fuzzy set and cache-based approach for bug triaging,” in _SIGSOFT/FSE’11 19th ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE-19) and ESEC’11: 13th European Software Engineering Conference (ESEC-13), Szeged, Hungary, September 5-9, 2011_ , T. Gyimóthy and A. Zeller, Eds. ACM, 2011, pp. 365–375.
  * [63] J. K. Siow, C. Gao, L. Fan, S. Chen, and Y. Liu, “CORE: automating review recommendation for code changes,” in _27th IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2020, London, ON, Canada, February 18-21, 2020_ , K. Kontogiannis, F. Khomh, A. Chatzigeorgiou, M. Fokaefs, and M. Zhou, Eds. IEEE, 2020, pp. 284–295.
  * [64] X. Yang, H. Steck, Y. Guo, and Y. Liu, “On top-k recommendation using social networks,” in _Sixth ACM Conference on Recommender Systems, RecSys ’12, Dublin, Ireland, September 9-13, 2012_ , P. Cunningham, N. J. Hurley, I. Guy, and S. S. Anand, Eds. ACM, 2012, pp. 67–74.
  * [65] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized BERT pretraining approach,” _CoRR_ , vol. abs/1907.11692, 2019.
  * [66] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang _et al._ , “Codebert: A pre-trained model for programming and natural languages,” _arXiv preprint arXiv:2002.08155_ , 2020.
  * [67] H. Husain, H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “Codesearchnet challenge: Evaluating the state of semantic code search,” _CoRR_ , vol. abs/1909.09436, 2019.
  * [68] J. Tabassum, M. Maddela, W. Xu, and A. Ritter, “Code and named entity recognition in stackoverflow,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , D. Jurafsky, J. Chai, N. Schluter, and J. R. Tetreault, Eds. Association for Computational Linguistics, 2020, pp. 4913–4926.
  * [69] T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,” in _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016_ , B. Krishnapuram, M. Shah, A. J. Smola, C. C. Aggarwal, D. Shen, and R. Rastogi, Eds. ACM, 2016, pp. 785–794.
  * [70] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. VanderPlas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in python,” _J. Mach. Learn. Res._ , vol. 12, pp. 2825–2830, 2011.
  * [71] R. Kallis, A. D. Sorbo, G. Canfora, and S. Panichella, “Ticket tagger: Machine learning driven issue classification,” in _2019 IEEE International Conference on Software Maintenance and Evolution, ICSME 2019, Cleveland, OH, USA, September 29 - October 4, 2019_. IEEE, 2019, pp. 406–409.