# Does Documentation Matter? An Empirical Study of Practitioners’ Perspective
on Open-Source Software Adoption

Aaron Imani<EMAIL_ADDRESS>0000-0001-7183-5468 University of California,
IrvineIrvineCaliforniaUSA ,  Shiva Radmanesh  University of
CalgaryCalgaryCanada<EMAIL_ADDRESS>,  Iftekhar Ahmed
University of California, IrvineIrvineCaliforniaUSA<EMAIL_ADDRESS>and
Mohammad Moshirpour  University of California, IrvineIrvineCaliforniaUSA
<EMAIL_ADDRESS>
###### Abstract.

In recent years, open-source software (OSS) has become increasingly prevalent
in developing software products. While OSS documentation is the primary source
of information provided by the developers’ community about a product, its role
in the industry’s adoption process has yet to be examined. We conducted semi-
structured interviews and an online survey to provide insight into this area.

Based on interviews and survey insights, we developed a topic model to collect
relevant information from OSS documentation automatically. Additionally,
according to our survey responses regarding challenges associated with OSS
documentation, we propose a novel information augmentation approach,
DocMentor, by combining OSS documentation corpus TF-IDF scores and ChatGPT.
Through explaining technical terms and providing examples and references, our
approach enhances the documentation context and improves practitioners’
understanding. Our tool’s effectiveness is assessed by surveying
practitioners.

open source, documentation, natural language processing, industry, interview

††ccs: Information systems Open source software††ccs: Computing methodologies
Natural language processing††ccs: Software and its engineering Documentation

##  1\. Introduction

Open-source software (OSS) constitutes up to 95% of codebases across various
industrial sectors (Synopsis, 2023). Projections indicate that the OSS market
value will surge from $21.7 billion in 2021 to over $50 billion in 2026,
marking a remarkable 130% increase (Rathee and Chobe, 2022). These statistics
underscore a growing trend in the industry’s adoption of OSS. However, OSS
adoption carries significant risks when decisions are uninformed. For example,
a design flaw in Log4j, a widely used OSS from Apache, led to a vulnerability
allowing Remote Code Execution (RCE) attacks (Gupta et al., 2022). This had
repercussions on major software companies like Microsoft and Apple,
emphasizing the critical role of reading and analyzing OSS documentation—the
primary source of information for adoption and assessing OSS quality
(Blumenthal, 2022).

Practitioners base their adoption decisions on various information sources. In
a recent survey conducted by Li et al. involving 23 participants from four
different companies, five key sources of information in adoption decisions
were identified (Li et al., 2022). These sources include version control
systems, issue tracking systems, question and answer portals (Q&A), forums and
blogs, and security-related platforms. Notably, neither the survey
participants nor the literature explicitly (Li et al., 2022) mentioned OSS
documentation as an information source during the Practitioners’ Adoption
Procedure (referred to as PAP). The absence of evidence regarding OSS
documentation as a reference point and the absence of prior work addressing
the identification of relevant OSS documentation types within PAP prompted the
formulation of our initial research question.

RQ1: Does OSS documentation play a role in practitioners’ adoption procedure?
What OSS documentation types are relevant to the adoption task?

Similar to any decision-making scenario, choosing one OSS product (A) over
another (B) relies on specific criteria. While software quality standards and
OSS-specific quality models provide valuable evaluation criteria, it’s
important to note that prior research has revealed a limited industry
inclination to utilize these readily available criteria. For instance, Yilmaz
et al. conducted a systematic literature review covering 36 studies from 2003
to 2020 to identify quality evaluation models or frameworks for open-source
software (Yılmaz and Kolukısa Tarhan, 2022). They found only one study that
provided evidence, in the grey literature, of practitioners using quality
evaluation models or frameworks to assess the quality of OSS within their
development teams. However, achieving a comprehensive understanding of the
role of OSS documentation in the PAP requires identifying the specific
criteria practitioners seek in this resource throughout the adoption process.
Thus, we outline our next research question.

RQ2: What adoption criteria do practitioners look for in OSS documentation?
Which criteria are most critical to them? Given the extensive categories
within OSS documentation, we posit that not all information is crucial during
the PAP. Practitioners should focus on identifying and extracting information
that aligns with their adoption criteria from OSS documentation to make
informed decisions. However, this task is not straightforward, as research
suggests that developers’ prior knowledge and cognitive biases can lead to
inefficient and ineffective knowledge retrieval (de Graaf et al., 2014).
Additionally, the information may not be easily discoverable, as documentation
commonly suffers from usability issues related to information findability,
where the intended information exists but users struggle to locate it
(Aghajani et al., 2019).

Consequently, the introduction of an automated mechanism aimed at extracting
PAP-related information from OSS documentation can help mitigate the impact of
practitioners’ prior knowledge (de Graaf et al., 2014) , which may render them
susceptible to cognitive biases during PAP. This mechanism can accomplish this
objective by discouraging practitioners from exclusively depending on the
sections of OSS documentation they typically consult during PAP. Instead,
providing them with a comprehensive catalog of pertinent information to
review. Our next research question is prompted by this need.

RQ3: How accurately can we identify the documentation topics related to the
industry’s selected criteria?

Practitioners should fully comprehend relevant information from OSS
documentation to make well-informed adoption decisions. Aghajani et al.
mentioned the technicality level of the documentation as one of the
readability issues of documentation (Aghajani et al., 2019). A search query of
(”jargon” or ”too technical”) and documentation on GitHub returns hundreds of
issues reported in OSS repositories (See (GitHub User, 2019a, b, 2020, 2022)
for sample issues). OSS practitioners do not all possess the same domain
knowledge in different OSS domains. Therefore, the provision of equal reading
experiences for practitioners is critical to the success of PAP. This
objective is the focus of our next research question.

RQ4: How can we tailor OSS documentation for different levels of domain
knowledge and reading goals?

To answer RQs 1 and 2, we conducted semi-structured interviews with 10
practitioners with OSS adoption experience. To triangulate our interview
results, we surveyed 42 practitioners. Following this, we created a corpus of
OSS documentation to serve as the basis for addressing RQs 3 and 4. Since
supervised deep learning relies heavily on label quality, we used the corpus
to fine-tune a topic model called BERTopic (Grootendorst, 2022) to address
RQ3. Then, we developed our novel information augmentation approach,
DocMentor, by combining OSS documentation corpus TF-IDF scores and ChatGPT
(OpenAI, 2023), a state-of-the-art large language model whose explanation
capability has been successfully deployed in similar tasks (Chen et al.,
2023), for answering RQ4. Lastly, we evaluated the usefulness and
effectiveness of our solution by conducting follow-up survey with the
individuals we interviewed initially.

The following are the key contributions of our study:

  * •

Providing the first OSS documentation corpus to enable further natural
language processing research on OSS documentation.

  * •

Identifying the role of OSS documentation in practitioners’ adoption
procedure.

  * •

Collecting practitioners’ adoption–relevant information from OSS
documentation.

  * •

Proposing a novel information augmentor that can not only enhance
practitioners’ understanding with any level of domain knowledge found in OSS
documentation, but can also serve as a framework for constructing information
augmentor applications across a variety of domains.

The remainder of this paper is arranged as follows: In Section 2, we review
major research bodies related to our work. In Section 3, we outline our survey
and interview methodology and discuss the obtained insights that help us
answer RQs 1 and 2. Then, we present our steps to fine-tune a BERTopic topic
model to answer RQ 3. In response to RQ 4, we introduce DocMentor in the final
part of the section. The results of our topic model in Section 4 are presented
in terms of its accuracy in extracting PAP-related information. Our evaluation
survey to assess DocMentor’s effectiveness and impact is discussed in Section
5. In Section 6, we analyze the threats to our methodologies’ validity and the
measures we followed to minimize them. In Section 7, our work is concluded,
and future directions are outlined.

##  2\. Related Work

Few studies have examined approaches to adopting OSS as components. Butler et
al., (Butler et al., 2022) interviewed 13 individuals from six Swedish
software companies to examine contemporary approaches to adopting OSS as
components, and the challenges encountered during this process. Based on
participants’ responses, the authors have categorized the challenges into four
categories: technical aspects, license-related matters, OSS attributes, and
risks. While Butler et al. investigated OSS quality factors, and adoption
challenges, our study is the pioneering effort to specifically examine OSS
adoption factors derived from OSS documentation, all within the context of
industry practitioners’ insights

In a separate line of research, prior work has looked into automated ways of
measuring the documentation quality. Tang et al. (Tang and Nadi, 2023)
introduced an automated documentation quality evaluation tool, which involved
the utilization of literature-derived metrics to assess documentation quality.
Treude et al. have proposed a ten criterion framework to evaluate software
documentation from different perspectives, such as structure, content, and
style (Treude et al., 2020). In particular, their framework identifies
quality, appeal, readability, understandability, structure, cohesion,
conciseness, effectiveness, consistency, and clarity as different aspects of
software documentation quality. Carvalho et al. have proposed a tree-based
toolkit called DMOSS to systematically assess the quality of non-source code
text available in software packages (Carvalho et al., 2014). Utilizing the
built-in plugins provided with DMOSS, they generate a report containing
information about features such as the validity of the links available in the
non-source code text, the validity of the license, spelling checks, the number
of comments, source code coverage for documentation, and analysis of changes.

In order to investigate the importance of software documentation quality on
software quality attributes and to identify the top important aspects of
software documentation quality, Plösch et al. surveyed 88 participants working
in the field of software engineering (Plösch et al., 2014). According to their
findings, the most influential quality attributes of software documentation
are accuracy, clarity, readability, structuredness, and understandability.
Moreover, the survey revealed that poor software documentation quality
significantly affects analyzability. The aforementioned studies primarily
center around software documentation quality. In contrast, our objective is to
pinpoint the challenges faced by practitioners when utilizing OSS
documentation throughout the PAP, with a specific focus on addressing the most
commonly reported issues.

In recent years, there has been a growing body of research leveraging NLP
techniques for various software engineering tasks. Ahmed et al. developed a
framework for extracting and prioritizing software quality attributes in
Agile-driven software development using Natural Language Processing (NLP)
techniques (Ahmed et al., 2023).

To facilitate early architecture decision-making in agile software
development, Gilson et al. used machine learning to extract software quality
attributes defined in ISO/IEC 25010 quality models from user stories (Gilson
et al., 2019). As a result of their analysis, they found that different
precision and recall scores were obtained for classifying user stories based
on each software quality attribute. Despite mentioning spaCy and pre-trained
models, the authors did not go into detail about the machine learning model
and manual features used in training the model.

Notably, our work is the first study to utilize an unsupervised deep-learning
approach to extract PAP-relevant information from OSS documentation.

##  3\. Methodology

In this section, we elaborate on our methodology to answer our RQs in three
stages. In each stage, steps are taken to reach an answer for the
corresponding stage’s RQs. In the first stage, we gathered practitioners’
perspectives through semi-structured interviews and an online survey to
address RQ1 and RQ2. Following the RQ1 and RQ2 answers, we designed a solution
to gather PAP-relevant information from OSS documentation to address RQ3 in
the second stage. In the final stage, our proposed information augmentation
approach, DocMentor, is presented. Our overall methodology is illustrated in
Figure 1, along with the steps involved in each stage.

Table 1. Interview participants’ demographics ID |  Experience in software industry (years) |  Experience in OSS (years) |  Company size | Role in company
---|---|---|---|---
P1 |  21 |  21 |  85 | Senior software developer
P2 |  17 |  17 |  200 | Technology principle
P3 |  20 |  20 |  750 | Senior stack developer
P4 |  6.5 |  6.5 |  750 | Senior software developer
P5 |  32 |  20 |  50 | Chief engineer
P6 |  34 |  34 |  1000 | Senior team lead
P7 |  30 |  20 |  100 | Principal staff engineer
P8 |  4 |  3 |  50,000 | Senior software engineer
P9 |  22 |  22 |  20 | Chief technical officer
P10 |  20 |  20 |  1,500 | Software engineering manager

###  3.1. Interviews

We conducted semi-structured interviews with ten individuals from the software
industry. These participants collectively possessed an average of 20 years of
experience in integrating OSS into their company’s products. The companies
involved in the study represented a diverse range of sectors, including
software development, software consulting services, software solutions for the
oil and gas industry, cybersecurity, finance technology, legal services, AI-
powered predictive tools for material and chemical companies, and engineering
services. The interviews were conducted remotely, with an average duration of
30 minutes per session. We recorded the interviews with participants’ consent,
following the university-approved Institutional Review Board (IRB) protocol,
and then transcripted them for analysis. Table 1 presents demographic
information about our participants.

Figure 1. Overall methodology overview

During the interview, we asked specific questions related to the participants’
utilization of OSS documentation during their decision-making process to adopt
OSS projects into their products. These questions encompassed various aspects,
including whether they rely on OSS documentation, the significance they
attribute to documentation in their decision-making, the specific ways in
which they utilize documentation during this process, the adoption criteria
they prioritize while evaluating OSS documentation, their level of confidence
in finding relevant information within OSS documentation that aligns with
these criteria, challenges encountered when using OSS documentation, other
textual resources they review when making decisions, how they utilize these
resources, and the metrics they seek within them. See supplementary (Imani et
al., 2024) for interview questions.

To enhance the depth of our interview data analysis, we categorized the
responses provided by the interviewees. This was done for two main purposes.
First, to facilitate the classification of various aspects of documentation
into categories that have greater significance for the industry. Second, to
employ these categories in the subsequent survey’s multiple-choice questions.
This approach allows for a more effective and consistent comparison between
the outcomes of the interviews and the survey.

To accomplish this task, two authors independently reviewed the transcriptions
and categorized participants’ responses of the following questions:  (1) How
do participants use OSS documentation while deciding to adopt an open-source
project for integration into their products?  (2) What specific adoption
criteria do participants seek while scrutinizing OSS documentation?  (3) What
challenges do participants encounter when using OSS documentation?  (4) Apart
from OSS documentation, what other textual resources do participants review
when making decisions?  (5) How do they utilize these resources?  (6) What
metrics do participants seek within these textual resources?  Following
individual categorization, the two authors engaged in negotiated agreement
(Forman and Damschroder, 2007; Garrison et al., 2006) on the categories for
each question to reach a consensus. This resulted in the identification of
nine distinct categories that participants consider when evaluating OSS
documentation for adoption purposes. These categories are as follows:

  1. 1.

Project Maintenance

  2. 2.

Compatibility

  3. 3.

Functionality

  4. 4.

License Compatibility

  5. 5.

Community Adoption

  6. 6.

Existence of Examples of Use Cases

  7. 7.

Ease of Use

  8. 8.

Project’s Versioning

  9. 9.

Performance

###  3.2. Online Survey

To triangulate our findings and further validate the insights gathered from
the interviews, we conducted a survey among practitioners within the software
industry. Specifically, we targeted individuals who have first-hand experience
in the process of adopting OSS for integration into their products.

####  3.2.1. Survey design

Researchers conducted a pilot study and iteratively tuned the survey questions
to ensure they were understandable and consistent with the survey objectives.
The survey comprised a total of 24 questions, which were designed to gather
insights from participants with diverse perspectives and experiences. The
questions were a mix of multiple-choice, Likert scale, and open-ended
questions, allowing us to capture both quantitative and qualitative data (See
supplementary (Imani et al., 2024) for survey questions). The structure of the
survey is as follows: Q3-Q6: Demographic questions. Q8-Q9: Questions assessing
the participants’ perception of the importance of OSS documentation in the OSS
adoption decision-making process. Q10: A question about participants’ approach
to selecting an OSS for adoption. Q11-Q13, Q15: Questions exploring the
factors influencing their decision to adopt an OSS. Q14: A question addressing
the challenges encountered when using OSS documentation for making adoption
decisions. Q15-Q20: Questions investigating the additional resources that
participants refer to while making OSS adoption decisions and how they utilize
these resources. Q21: A question focusing on the factors contributing to the
ease of understanding one OSS documentation compared to another.

####  3.2.2. Survey participants

We utilized Qualtrics as our survey distribution platform. We targeted
practitioners with prior experience in integrating OSS into their projects. We
initially distributed the survey to our industry connections and followed up
with periodic reminders. Additionally, we used the snowball sampling method
(Wohlin, 2014) by encouraging them to share the survey with other eligible
employees within their respective companies. And eventually, we shared the
survey on LinkedIn. The survey remained accessible for a duration of one
month, during which we gathered a total of 29 responses.

####  3.2.3. Participants’ demographics

Most of the respondents were affiliated with companies in the field of web
development. Participants were also from open-source software, manufacturing,
IoT, and finance companies. In terms of company size, 17.6% had less than 50
employees, 25.71% had 50-100 employees, 20% had 100-500 employees, 8% had
500-1000 employees, and 28.57% had more than 1000 employees. On average, the
participants had 11-15 years of experience in the software industry and 6-10
years of experience in integrating OSS into their projects.

###  3.3. PAP-relevant Information Gathering

In light of the findings from the survey and interviews, we systematically
outline our methodology to address RQ3 in a step-by-step manner.

####  3.3.1. Information Resource

Given the diversity of open-source software (OSS) documentation, encompassing
various platforms such as GitHub pages, Sphinx, MkDocs, and customized
websites, our approach needed to narrow down to a specific documentation
source. The selection criteria prioritized a unified structure both at the
page level and throughout the documentation. Sphinx, which employs the
reStructuredText (reST) markup language to create documentation in multiple
formats, including HTML, provides a uniform format for OSS documentation
(Sphinx developers, [n. d.]). Its hierarchical structure and standardized
format facilitate effective data extraction, contributing to the creation of
our OSS documentation corpus and dataset. Notably, Read the Docs hosts
documentation for over 100,000 OSS projects (Read the Docs Inc & contributors,
[n. d.]), offering a substantial and valuable dataset for OSS documentation
research. Furthermore, since it supports reStructuredText, it serves as a
hosting platform for OSS projects employing Sphinx for documentation. As a
result, to streamline the management of our OSS documentation collection, we
focused on Sphinx-based documentation hosted by Read the Docs.

####  3.3.2. Information Granularity

Determining the suitable level of granularity for extracting PAP-relevant
information from OSS documentation represents the initial step in this
process. One potential approach involves inferring relevance from the web page
titles of OSS documentation. This approach assumes that there is consistency
in content topics within a webpage. However, OSS documentation webpages often
contain multiple topics, especially in cases where poorly structured
documentation combines incoherent information into lengthy web pages.
Therefore, it becomes essential to pinpoint relevant information at a finer-
grained level. Since our study is centered on Sphinx-based documentation , we
can establish the appropriate granularity by utilizing the information
structure of reST. ReST organizes documents into sections, which subsequently
translate into header tags in HTML output. Each section can encompass various
types of content, including other sections. Consequently, the innermost
sections (hereafter referred to as ”sections”) constitute the most cohesive
units of information in Sphinx-based documentation. While one might contend
that paragraphs offer an appropriate level of granularity, it is important to
consider that paragraphs within a section often address specific subjects or
topics as whole. Consequently, selecting only a subset of paragraphs from a
section could result in incomplete coverage of the information presented
within that section.

####  3.3.3. Information Gathering Approach

One approach for identifying PAP-relevant sections in OSS documentation is
through the utilization of keywords and regular expressions. However, due to
the wide range of OSS domains, including areas like deep learning and web
development, domain-specific terminology can affect the completeness of our
keyword coverage. Additionally, this approach overlooks content semantics,
potentially resulting in the omission of sections that have indirect relevance
to an adoption criterion. Therefore, it becomes imperative to develop an
automated method that incorporates semantics and minimizes the need for human
judgment, ultimately reducing subjectivity in the process.

To address these challenges, we turned to Topic modeling, a well-established
text analysis approach that has been in use for over two decades (Zhao et al.,
2021). In recent years, the fusion of topic modeling with deep learning
techniques has given rise to innovative topic models that surpass traditional
Bayesian probabilistic models like Latent Dirichlet Allocation (LDA) (Blei et
al., 2003). Some noteworthy examples of state-of-the-art topic models that
rely on pre-trained embedding models include Top2vec (Angelov, 2020), CTM
(Bianchi et al., 2021), and the latest addition, BERTopic (Grootendorst,
2022).

Similar to Top2vec, BERTopic performs topic modeling by clustering document
embeddings. However, unlike Top2vec, which represents topics based on cluster
centroids, BERTopic uses a class-based version of TF-IDF to derive the topic
representation from each topic. In comparative evaluations with LDA, Top2vec,
and CTM, BERTopic has demonstrated state-of-the-art results on topic modeling
benchmark datasets in terms of topic coherence and frequency (Grootendorst,
2022; Egger and Yu, 2022). Consequently, we have chosen to employ BERTopic for
extracting PAP-relevant information for two primary reasons: 1) It is an
unsupervised technique that requires minimal human intervention, and 2) It
captures the semantics of sections, which distinguishes it from keyword-based
methods and traditional topic modeling.

####  3.3.4. OSS Documentation Dataset

Topic modeling with BERTopic necessitates a corpus of documents, which, in our
case, needed to consist of OSS documentation. As there was no readily
available corpus, we took the initiative to create one. Our rationale for this
approach was to avoid potential bias towards terminology specific to a
particular OSS domain. Consequently, we took measures to ensure diversity in
the OSS domains represented within our corpus.

In a recent study by Sas et al., they introduced GitRanking, a taxonomy of
software domains constructed from the ground up (Sas et al., 2022). To create
this ranking, they collected 121,000 topics from GitHub and selected the top
60% most frequently occurring topics. They further narrowed this selection
down to 3,000 topics based on annotations from three annotators, ultimately
extracting 368 software domains. For our study, we chose the top 52 domains,
collectively covering 50% of the repositories, to ensure representation across
a substantial portion of OSS domains.

Next, we performed GitHub searches using the GitHub API where for each OSS
domain, we used the query ”topic:¡OSS Domain¿ readthedocs.io in:wiki or
readthedocs.io in:readme sort:stars” to locate top-starred repositories that
have Sphinx-based documentation hosted on Read the Docs. Afterward, we
searched Read the Docs for projects with corresponding repository URLs and
names to extract their documentation URLs. This process resulted in the
collection of 1,387 OSS documentation URLs. To ensure convenient access and
minimize reliance on online references, we utilized HTTrack to create local
mirrors of the documentation for these projects. We were able to mirror 1,383
of the 1,387 documentation websites.

In the following phase, we encountered the challenge of deciding the optimal
content granularity to extract from each OSS documentation. Given that
BERTopic utilizes Sentence Transformers and acknowledging the significant
variability in section and paragraph lengths due to diverse contexts and
writing styles, we chose to extract sentences. This decision served two
primary purposes: 1) Extracting sentences helped mitigate the problem of
padding that could potentially lead to reaching the maximum token limit
imposed by the embedding model used in BERTopic. 2) By extracting sentences,
we ensured a consistent range of document lengths, thus creating a more
manageable and uniform dataset. Our final dataset encompassed 744,608 unique
sentences.

####  3.3.5. Topic Modeling

Different variations of topic modeling are supported by BERTopic, such as
dynamic, multimodal, hierarchical, online, and semisupervised approaches.
Using vanilla BERTopic, even though topics are relevant to one of our PAP-
selected criteria, their representations may mislead the topic model towards
OSS domain-specific terms. To address this issue and guide the topic model
towards creating topics that align more closely with our areas of interest, we
can introduce a weighting mechanism for specific words within the dataset.
This approach ensures that the resulting topic model contains topics whose
representations are more in line with our criteria. BERTopic introduces a
semisupervised variation of topic modeling known as Guided Topic Modeling,
which facilitates this process by allowing us to guide the model’s topic
creation using weighted terms called seed, thereby producing topics that
better match our intended criteria. Seed is a nested list of terms, where each
inner list consists of phrases relevant to one of our desired topics. We
curated our seed by inspecting OSS documentation pages and extracting common
terminology for each criterion, for instance, for the License criterion, we
created a list in our seed with the terms: ”license”, ”MIT”, ”bsd”, ”GPL”,
”Apache”, ”Redistribution”, ”copyright notice”, ”disclaimer”, ”LIABLE”, ”AS
IS”, and ”MERCHANTABILITY”. We tested our seed in three iterations and
improved it in each iteration. The seed we used to train the topic model is
available at supplementary (Imani et al., 2024).

BERTopic is designed to be modular, meaning each algorithm phase may be
customized. As such, ensuring the appropriate configuration at each step is of
the utmost importance. Topic modeling phases and their corresponding
hyperparameters are summarized in Figure 2.

The first phase in the BERTopic pipeline is calculating embedding vectors for
the input documents. Various embedding models are supported in this step.
Thus, our first variable that can significantly affect the final topics is the
embedding model employed. We focused on the top four SBERT (Reimers and
Gurevych, 2019) pre-trained embedding models known for their robust
performance in encoding sentences across a spectrum of 14 distinct tasks.
However, due to efficiency considerations, we opted to exclude all-
distilroberta-v1 from consideration, as it incurred an 87.5% increase in
processing time compared to all-MiniLM-L12-v2 with only a negligible 0.03
average performance difference on a magnitude scale of 100.

In the second phase, the dimensionality of the embedding vectors is reduced to
counteract the curse of the dimensionality problem in the clustering phase. To
handle large amounts of documents, cuML’s UMAP implementation can be used to
enable GPU utilization in this phase. Although other dimensionality reduction
algorithms could be used, we adopted UMAP as its superiority is discussed in
previous work (Grootendorst, 2022; McInnes et al., 2018). CuML’s UMAP offers a
range of tunable parameters, but among them, two variables highlighted by
BERTopic have a substantial impact on the algorithm: the number of neighbors
and the number of components. The number of neighbors, which can be set
between 2 and 100, plays a crucial role in determining the preservation of
local data. Smaller values favor the retention of more localized data.
Additionally, the number of components represents the dimensionality of the
reduced vectors. Increasing the value of this variable reduces the impact of
dimensionality reduction on clustering. In addition to the aforementioned
hyperparameters, there is another crucial parameter to consider in cuML UMAP:
the minimum distance between the reduced vectors. Ranging from 0 to 1, this
parameter plays a pivotal role in the balance between global and local
structure preservation. When set to smaller values, it leads to a more tightly
clustered embedding space, where vectors in close proximity on the manifold
are drawn even closer together. We initiated with a conservative minimum value
of 5 for both hyperparameters. Subsequently, we incremented these values in
steps of 5, gradually increasing them to 20. This enabled the exploration of
larger values in cases where we observed an upward trend in the topic model’s
performance.

In the third phase, reduced vectors are clustered. Similar to previous phases,
BERTopic supports customized clustering algorithms. Since not all the
information within OSS documentation aligns with practitioners’ adoption
criteria, it is prudent to employ a clustering algorithm that accommodates
outlier data points. One such algorithm, used as the default in BERTopic, is
HDBSCAN (McInnes et al., 2017). HDBSCAN simplifies the task by preventing the
need to select the number of clusters manually, as well as excelling at
automatically clustering dense areas of varying densities. To harness GPU
acceleration benefits, we opted for cuML’s HDBSCAN implementation. While this
algorithm offers flexibility through various customizable parameters, such as
the minimum sample size and the metric used for distance calculations, a
crucial parameter to consider is the minimum cluster size. This parameter
determines the coherence of the resulting topics. If the minimum cluster size
is too high relative to the number of documents, it can lead to the clustering
of irrelevant documents, adversely impacting the quality of topic
representations. Considering the substantial volume of sentences, numbering in
the hundreds of thousands, to ensure coherent topics, we chose a minimum value
of 50. It could be argued that this selection might be reduced further, but
our experiments indicated that lower values, such as 20, resulted in a
significant proliferation of topics, each containing only a minimal number of
sentences. Conversely, experimenting with larger values could have led to
topic merging, incompatible with our final topic prediction pipeline.

Topic representations are terms extracted from a topic’s documents that
effectively represent the topic documents. They are pivotal in predicting
topics for unseen documents. There are two modules that work together to
calculate them: Scikit-learn CountVectorizer and BERTopic’s c-TF-IDF. In
addition to the CountVectorizer parameters which control the creation of the
term-topic matrix used in c-TF-IDF, there are several other customizable
aspects to consider. One such aspect is the option to set stop words, enabling
the exclusion of common stop words from the topic representations.
Furthermore, the length of N-grams to be considered during the c-TF-IDF
calculation can be tailored within CountVectorizer. We conducted experiments
with both unigrams (N-grams of length 1) and bigrams (N-grams of length 2).
Through our observations, we discerned that increasing the N-gram length
introduced variations of the same word into topic representations. For
instance, a topic comprising sentences relevant to compatibility might be
represented by both ”install pip” and ”install conda”. However, reducing the
N-gram length to 1 results in ”install” appearing only once in the topic
representation. This, in turn, allows other N-grams to represent the topic,
fostering wider diversity in topic representations.

To address the issue of frequent terms that may not be classified as stop
words, it is possible to configure the c-TF-IDF algorithm to calculate the
square root of term frequency after normalizing the frequency matrix, thereby
refining topic representation quality. We enabled the stop word exclusion
option in CountVectorizer as well as the frequency reduction option in c-TF-
IDF so as to minimize the occurrence of repeating terms in topic
representations.

After training the model, the topic representations can be fine-tuned.
BERTopic supports various representation models, such as Maximal Marginal
Reference, KeyBERT, etc., that can be chained. Experimenting with different
topic representation models is crucial since topic models depend heavily on
topic representations for valid predictions. We conducted experiments using
four distinct approaches: Zero-shot classification, KeyBertInspired (KBI),
MaximalMarginalRelevance (MMR) (Carbonell and Goldstein, 1998), and a
combination of the latter two methods. Notably, zero-shot classification
limits the number of terms representing each topic to one. Consequently, it
becomes challenging to recognize the underlying themes of sentences within
each topic using this approach.

Drawing inspiration from KeyBERT (Grootendorst, 2020), the KBI representation
model selects topic representations based on semantic similarity to topic
embeddings. These embeddings are computed using representative documents per
topic, identified by their c-TF-IDF scores. To enhance the diversity of
keywords in topic representations and mitigate redundancy (e.g., the co-
occurrence of ”install” and ”installation” in a single topic representation),
we incorporated the use of MMR. By chaining the MMR and KBI approaches, we
aimed to balance preserving semantic relationships and ensuring diversity
within the topic representations simultaneously.

Figure 2. Topic modeling pipeline

For each hyperparameter, H, we froze the values of other hyperparameters and
experimented with potential values for H. We randomly sampled 447 sections
from our dataset, ensuring a sampling confidence of 95% with a margin of error
of 5%. Subsequently, the first and second authors independently labeled the
samples, and any instances with conflicting labels were designated as
outliers. In total, our labeling process yielded an agreement rate of 81% on
the labels. We refer to this validation sample as Groundtruth.

Figure 3. Topic prediction pipeline

###  3.4. Topics of Interest

In the process of predicting topics for unseen sentences, BERTopic relies on
the calculation of their embedding vectors and measures their cosine
similarity to topic embeddings. These topic embeddings are derived by
averaging the embeddings of sentences within each topic. To ensure
satisfactory performance, it is crucial to maintain minimal similarity between
the topics of interest (TOIs). To achieve this, we chose to merge conceptually
similar practitioners’ criteria, an essential step in mitigating potential
mispredictions. For example, topics like ”project’s versioning” and ”project’s
maintenance” share overlapping terminology, making it challenging to
distinguish sentences relevant to these two distinct topics. To achieve this
objective, thorough discussions were conducted between the first and second
authors, resulting in complete consensus. The aim of these discussions was to
minimize the merging of criteria while still encompassing practitioners’
preferences effectively. Ultimately, the merged criteria were closely aligned
with recent developments in OSS quality factors (Madaehoh and Senivongse,
2022).

  * •

License: License Compatibility

  * •

Functional Suitability: Ease of use, Functionality & Features, The existence
of examples of use cases, Project’s Performance

  * •

Compatibility: Compatibility

  * •

Project’s Maintenance: Project’s Maintenance, Project’s usage trends,
Project’s versioning, Community Adoption

Finally, we considered a topic called Outlier to include sentences irrelevant
to the aforementioned TOIs.

###  3.5. Topic Prediction

Due to the substantial size of our sentence dataset, which comprises 744,608
sentences from diverse OSS domains, BERTopic naturally generates a large
number of topics. However, our focus is on identifying five specific topics:
License, Functional Suitability, Compatibility, Project Maintenance, and an
outlier topic. To achieve this, we employ a strategy to identify topics
similar to each of our TOIs and subsequently merge them. It might be argued
that combining these topics is unnecessary, as they could simply be marked as
one of the TOIs. However, based on our experiments, not merging the topics
leads to an increased number of false positive predictions for the outlier
topic. By merging similar topics, we create larger composite topics, and their
embeddings are calculated through a weighted average of the embeddings of the
initially merged topics. This strategy enhances the likelihood of correctly
assigning an unseen sentence to the appropriate topic. It considers both the
original topics’ sizes and their embeddings, contributing to improved topic
prediction performance.

Project’s Maintenance Functional Suitability License Compatibility Outlier
Overall $0$$20$$40$$60$$80$$100$Median Weighted F1-Scoreall-MiniLM-L12-v2all-
MiniLM-L6-v2all-mpnet-base-v2 Figure 4. Embedding models prediction
performance comparison. The Y values present the median of the embedding
model’s weighted average F1-score within all hyperparameters.

Our topic prediction pipeline is depicted in Figure 3. It commences with the
identification of topics similar to our seed topics by concatenating the
phrases (search strings) within each TOI and computing the cosine similarity
between the search string and topic embeddings. If the similarity surpassed or
equaled the Topics Similarity Threshold, we added the topic and its similarity
score to the TOI’s merge list. Since certain topics appeared in the merge
lists of multiple TOIs, we resolved conflicts over mutual topics by removing
the topic from the TOI with the lower cosine similarity to it. Subsequently,
we merged the topics and labeled them with their corresponding TOI names.
Among the resulting merged topics, the outlier topic was the largest, which
increased the risk of false positive predictions for this topic. To address
this, we reduced the outliers by assigning their sentences to one of the TOIs
based on the Reduction Minimum Similarity criterion. The topic representations
of the merged topics were updated through a combination of MMR and KBI. For
each final topic, the Topic Representation Size parameter determined the
number of terms included in the topic representations.

Our final topic predictor operates by receiving a section, predicting the
topics for the sentences within it, and subsequently using the total
similarity values between each sentence and the topics to make a prediction
regarding the section’s topic. While alternative methods, such as calculating
the median or mean of sentence similarities to the topics, could be employed,
our experimental findings have shown that summing the similarities
consistently leads to more accurate predictions at the section level.

Accordingly, our topic prediction pipeline introduces three thresholds: Topics
Similarity Threshold, Reduction Minimum Similarity, and Topic Representation
Size. For the first two thresholds, we conducted an extensive search, ranging
from 0 to 1 with increments of 0.1. Additionally, we explored values within
the range of 10 to 50 for Topic Representation Size, with increments of 10.

We discuss optimal hyperparameters performance regarding the weighted average
F1 score on all our TOIs. We chose the F1 score to take into account precision
and recall simultaneously. The reason behind selecting a weighted average is
to address the class imbalance in Groundtruth. This is because both of these
metrics are critical to our topic predictor. Regarding the embedding model,
all-MiniLM-L6-v2, the default model suggested by BERTopic, outperformed other
embedding models in predicting three out of four TOIs. However, all-mpnet-
base-v2 outperformed the others in predicting license-related and outlier
sections. Considering overall performance, all-MiniLM-L6-v2 outperformed the
other embedding models with a slight superiority of 1%. The median weighted
average F1-score across all the TOIs achieved by all-MiniLM-L6-v2 is 67.4%.
Figure 4 illustrates embedding models’ performance across TOIs.

Setting the embedding model to all-MiniLM-L6-v2, we experimented with the rest
of the hyperparameters, the number of components, neighbors, and the minimum
distance between reduced embedding vectors. Changes in none of the
hyperparameters impacted the median weighted average F1-score of all-
MiniLM-L6-v2 to the magnitude of 0.001s. However the optimal values for these
hyperparameters that maximized the weighted average F1-score were 20, 20, and
0.1. Figure 5 presents the maximum weighted average F1 score achieved by each
threshold value. Topic Representation Size’s performance peaked at 20, the
Minimum Reduction Similarity at 0.2, and the Topics Similarity Threshold at
0.3.

Figure 5. Topic prediction thresholds performance comparison. Note that the Y
ranges have been limited to a range that eases observing the changes.

###  3.6. DocMentor

In order to enable practitioners with different OSS domain knowledge levels,
we propose DocMentor which combines TF-IDF and ChatGPT to identify technical
terms and provide explanations, references, and examples. Following, we
analyze TF-IDF’s application for technical terms detection.

Considering the broad spectrum of OSS domains, technical terms specific to an
OSS domain, $D_{i}$, may have a low frequency of occurrence in documents of
other OSS domains. When calculating TF-IDF, these terms will have a low term
frequency (TF) in non-relevant OSS domains, contributing to a higher TF-IDF
score when the term is present in $D_{i}$. Conversely, OSS domain-specific
terms often exhibit high document frequency (DF) within their respective
domains. For example, terms like ”f1-score” may appear frequently in numerous
OSS documentation sections within the Machine Learning domain. Consequently,
this elevated DF translates to a high Inverse Document Frequency (IDF),
thereby inflating the TF-IDF scores for technical terms within their specific
OSS domains. Accordingly, in the context of TF-IDF, a technical term is
identified by three characteristics: 1) High intradomain DF 2) High
intradomain TF 3) Low interdomain DF.

While our TF-IDF-based approach effectively captures known technical terms
within our corpus, it encounters limitations when identifying unseen technical
terms. To optimize the efficacy of our method, complementary approaches become
essential. In this regard, we leveraged OpenAI ChatGPT, a state-of-the-art
large language model. ChatGPT was harnessed not only to enhance our technical
term identification performance but also to provide explanations for the
identified technical terms, references, and examples. It is worth noting that
ChatGPT has demonstrated its utility in similar tasks, including code
explanation (Chen et al., 2023), making it a valuable asset in DocMentor.

Accordingly, DocMentor’s workflow commences with inputting a text segment,
such as a paragraph. In this initial stage, technical terms are detected
utilizing TF-IDF. Subsequently, leveraging the capabilities of OpenAI’s
ChatGPT, the list of technical terms is further expanded. Finally, ChatGPT
provides brief explanations, and depending on the context, examples, and
references for the identified technical terms. To arrive at a suitable prompt
for each interaction with ChatGPT, the first author engaged in an iterative
process, guided by our predefined evaluation criteria. Once the final prompts
were established, all authors collectively reviewed ChatGPT’s responses in a
pilot study before subjecting them to evaluation by practitioners. The
supplementary (Imani et al., 2024) provides an example of DocMentor’s input,
our ChatGPT prompts and the generated output.

In order to evaluate DocMentor’s performance, we initially explored the
availability of existing benchmarks that could facilitate a systematic
evaluation. However, to the best of our knowledge at the time of this study,
no such benchmark existed for assessing augmented information within the
domain of OSS. Hence, we devised an evaluation survey. To ensure an unbiased
representation of various criteria, we incorporated one paragraph from each
practitioner’s criteria. This approach was chosen to maintain manageability
for practitioners, as evaluating multiple paragraphs for each criterion would
have been impractical. To achieve this, we employed our trained topic model to
predict the topics of sections within our OSS documentation corpus.
Subsequently, we randomly sampled medium-length paragraphs from each category.
If, by chance, the initially selected paragraph was misclassified, we replaced
it with another randomly selected paragraph. The resulting set of selected
paragraphs was then processed through DocMentor, with the outputs from
DocMentor being collected for evaluation through the survey. In order to
ensure that our survey questions effectively align with the goal of evaluating
DocMentor’s responses from multiple perspectives, researchers conducted a
pilot study and iteratively refined the questions. This iterative process
aimed to enhance the clarity and precision of the survey questions, Our
evaluation survey structure is outlined below.

In the first three questions, we sought participants’ demographic information.
Following that, for each of our TOI-relevant paragraphs, we asked participants
to assign a 5-point scale Linkert score to the augmented information on
various aspects: 1) its informativeness, 2) The degree to which it clarified
technical terms within the paragraph, 3) The relevance of the provided
examples or references, 4) The extent to which it would aid in facilitating
adoption decision-making, and 5) The extent to which it improved the
participant’s comprehension of the paragraph.

In order to distribute the survey, we contacted the initial survey and
interview participants who opted for participating a potential follow-up
survey. To ensure an adequate number of participants, two individuals were
invited separately by the researchers, as not all potential survey
participants were able to take part in the evaluation survey.

##  4\. Results

Version-Documentation Inconsistency Missing Information Reading Challenges
Inaccurate Dependency Documentation Finding relevant information
$0$$20$$40$$60$$80$$100$Percentage of choiceInterviewSurvey Figure 6.
Challenges with using OSS documentation

###  4.1. Interview and Survey Insights

Our findings indicate that within the industry, OSS documentation is regarded
as a significant resource for making decisions concerning the adoption of
open-source software. All ten participants confirmed that they rely on OSS
documentation when making decisions regarding the adoption of open-source
projects into their products. A significant majority of our survey
participants, specifically 91.18%, indicated that they rely on open-source
software documentation when making decisions about integrating them into their
products.

Table 2. Participants’ Confidence Level in Finding Relevant Criteria in OSS Documentation Adoption Category |  Median of Confidence Level (Interview) |  Median of Confidence Level (Survey)
---|---|---
Project Maintenance |  4 |  3
Compatibility |  4 |  3
Functionality |  3 |  4
License Compatibility |  3 |  5
Community Adoption |  4 |  3
Examples of Use Cases |  3 |  4
Ease of use |  4 |  3
Project’s Versioning |  2.75 |  3.5
Performance |  4 |  2.5

When asked to rate the importance of OSS documentation in their decision on a
scale of 1 to 5, with 5 being the highest, the median of interview responses
was 5. Specifically, one participant rated it as 2, another as 3, two
participants as 4, and six participants as the highest rating, 5. The median
of the survey participants’ responses to this question was 4. More
specifically, 3.7% rated it as 1, 22.22% as 2, 37.04% as 4, and another 37.04%
as 5. This indicates a strong consensus among the participants regarding the
significant role of OSS documentation in their decision-making process. Table
3 displays the survey participants’ responses regarding the reasons for
utilizing open-source software documentation when making decisions about
adopting OSS.

In Figure 6, we present the challenges reported by the participants when using
OSS documentation. Each challenge is accompanied by the corresponding
percentage, indicating the proportion of participants who mentioned that issue
when using OSS documentation.

To assess the importance of each criterion, participants were asked to rank
them based on their importance. Figure 7 illustrates the relative importance
of each category as perceived by the participants. Table 2 displays the
participants’ confidence levels for each adoption criterion when seeking
relevant information in OSS documentation. They were asked to rate their
confidence level, ranging from 1 to 5, with 5 indicating the highest level of
confidence.

MaintenanceCompatibilityFunctionalityLicenseCommunity AdoptionExamples of Use CasesEase of useProject’s VersioningPerformance$0$$20$$40$$60$$80$$100$S: SurveyI: InterviewIIIIIIIIPercentage of choice$1^{\text{st}}$ Priority$2^{\text{nd}}$ Priority$3^{\text{rd}}$ Priority$+4^{\text{th}}$ PriorityNot a PrioritySSSSSS Figure 7. Importance of Adoption Metrics in OSS Documentation Table 3. Participants’ Reasons for using open-source documentation during decision-making for OSS adoption  Reason for Using OSS Documentation | Choice Percentage
---|---
Implement a proof of concept | 10.60%
Understand how the open-source software works | 11.92%
Realize the open-source software functionalities and features | 16.56%
Analyze how the open-source software can be used | 11.26%
Get introduced to the open-source software | 7.28%
Project’s Versioning | 2.75%
Analyze the open-source software community and contributors | 5.30%
Monitor the open-source software maintenance | 3.31%
Study examples of using the open-source software | 12.58%
Verify compatibility (Including license compatibility) | 8.61%
Analyze the open-source software maturity | 8.61%
Understand the contribution procedure | 3.31%

###  4.2. PAP-relevant Information Gathering Performance

Answering RQ3, using threshold values and the optimal hyperparameters
(Explained in Section 3), resulted in a weighted average of 73% precision, 68%
recall, and 69% F1-score.

The performance of BERTopic hinges on the cosine similarity between the
embeddings of unseen sentences and the topic embeddings. However, this
approach introduces the potential for false positives, where sentences may be
misclassified into topics they do not belong to. For example, sentences
explaining the functional suitability of an OSS library in the context of web
servers might include terms like ”permission” and ”license,” which also have
high c-TF-IDF values for License-related sentences. This overlap in
terminology can result in false predictions for both Topics of Interest
(TOIs), namely License and Functional Suitability, underscoring the challenge
of disambiguating topics with shared terminology. This prompts the need for
future research to alleviate this challenge.

###  4.3. DocMentor Evaluation Results

Our evaluation survey was completed by 10 practitioners, whose demographic
information is available at our supplementary (Imani et al., 2024). Following,
we discuss responses to each evaluation aspect as we discussed in Section 3.

This question focused on evaluating ChatGPT’s explanations of the technical
terms identified by DocMentor. It received a median score of 3.75 when
assessed across all paragraphs. Notably, the compatibility paragraph achieved
the highest median score of 4, signifying that practitioners found the output
particularly informative in this context. Conversely, the output for the
license paragraph was rated as less informative, with a median score of 3.50.
Several factors may have contributed to this observation. It is possible that
the practitioners possessed a high level of domain knowledge regarding
licensing terms, which could have influenced their perception of DocMentor’s
output’s informativeness. Alternatively, there might have been issues with
accurately identifying technical terms within the paragraph by DocMentor, or
the explanations provided may not have sufficient information. Further
investigation may be warranted to pinpoint the specific reasons behind these
informativeness scores differences.

In terms of the outputs’ ability to clarify technical terms, the survey
revealed a median score of 3.75 when considering all paragraphs collectively.
Interestingly, when evaluating individual paragraphs, the project maintenance-
related-related paragraph received the highest score in this regard, with a
median of 4. In contrast to the previous question, the compatibility-related
paragraph obtained the lowest median score of 3.5 in clarifying technical
terms. This intriguing divergence implies that while the output for the
compatibility paragraph may have been informative, it fell short in providing
the necessary clarification for technical terms. This discrepancy could be
attributed to the nature of the technical terms themselves, which may not have
aligned with practitioners’ perceptions or expectations.

Among the outputs evaluated, the functional suitability paragraph was the sole
instance where examples were provided. Participants assessed the relevance of
these examples, resulting in a median score of 4. This observation underscores
the potential value of leveraging ChatGPT to generate quality OSS-specific
examples.

Based on the feedback provided by participants on the potential DocMentor’s
helpfuless in PAP, which yielded a median score of 3, it is evident that there
is room for further tailoring DocMentor to better align with the specific
needs of the adoption procedure. Notably, practitioners scored the minimum
score for the project maintenance paragraph. The participants’ evaluations
have raised important questions and considerations, prompting the need for a
more in-depth investigation. This further exploration aims to gain a deeper
understanding of how DocMentor’s responses can be optimized to effectively
facilitate PAP and cater to the unique requirements and expectations of
practitioners.

Practitioners’ understanding of the paragraphs was improved by achieving a
median score of 3.75 across all paragraphs. Consistent with previous question
findings, practitioners’ understanding of the project’s maintenance paragraph
was minimally impacted by DocMentor’s output. This outcome was anticipated, as
project maintenance terminology (such as version control-relevant terms) tends
to be familiar to practitioners with even minimal domain knowledge. In
contrast, DocMentor’s impact on practitioners’ comprehension of the license-
related paragraph was more pronounced, reaching a median score of 4. This
suggests that DocMentor played a notably helpful role in empowering
practitioners to better understand license-related information during the PAP.

The evaluation survey has unveiled the intrinsic capabilities of DocMentor’s
augmented information, providing valuable insights into its potential
benefits. However, as the inaugural endeavor in augmenting information to
facilitate PAP, it is evident that DocMentor should undergo further
enhancements to make it more applicable and effective in real-world scenarios.
Addressing RQ4, the combination of TF-IDF scores calculated from an OSS
documentation corpus with the capabilities of large language models has the
potential to enhance practitioners’ comprehension of OSS documentation. This
approach enables practitioners with diverse levels of domain knowledge to
better grasp the information contained within this resource.

##  5\. Threats to Validity

We have adopted measures to mitigate potential threats to the validity of our
work, which we outline in this section.

###  5.1. Construct validity

To mitigate the potential threat to construct validity stemming from
participant misunderstandings of our survey questions, we took proactive
measures by conducting pilot studies. Subsequently, we used the feedback from
these pilot studies to refine and update our surveys. This iterative process
allowed us to enhance the clarity and comprehensibility of our survey
questions.

###  5.2. Internal validity

In this study, we utilized a snowball sampling approach for participant
recruitment in the interview phase. Although snowball sampling is inherently
non-random and can introduce selection bias, we implemented measures to
address this potential bias. Initially, we intentionally identified a diverse
group of key informants with relevant insights into the research topic. To
mitigate bias related to the initial informants, we encouraged them to suggest
additional participants from their networks who had experience with software
development and OSS adoption experiments. Additionally, we ensured sample
diversity by including participants from various industrial sectors to capture
a range of perspectives on OSS adoption.

###  5.3. External validity

To create our OSS documentation dataset, we specifically concentrated on
Sphinx-based documentation hosted on Readthedocs. We made this decision
because such documentation follows a standardized structure and consistently
uses common tags, making data extraction more manageable. Nevertheless, we
were aware of the potential limitation of this approach in terms of source
diversity. To mitigate this concern, we strategically diversified the OSS
domains in our dataset. By including a broad range of OSS domains, we
effectively increased the inclusivity and comprehensiveness of our dataset.

##  6\. Conclusion and Future Work

In this study, we investigated the essential role of OSS documentation in
practitioners’ adoption processes through interviews and surveys. Our findings
not only highlight the significance of OSS documentation in adoption but also
offer insights into the criteria practitioners use for their adoption
decisions. From the data we collected, we introduced a novel automated method
utilizing BERTopic to extract PAP-relevant information. This method lays the
foundation for creating a web service that can take an OSS library name as a
query and provide well-categorized OSS documentation, customized for the
adoption process. As ours was the initial step toward automating the retrieval
of OSS adoption-related information, future work can aim to enhance the
performance further. Another natural progression of this work would be
introducing a personalized OSS adoption assistant based on Natural Language
Inference. This assistant could analyze adoption criteria and produce
compliance reports, reducing human judgment and the likelihood of errors in
the adoption process.

##  7\. Data Availability

Our study’s data, including interview scripts, the initial and evaluation
survey results, all the Python code for different sections, our trained topic
model, and the DocMentor’s code along with its ChatGPT prompts, are available
at supplementary (Imani et al., 2024).

## References

  * (1)
  * Aghajani et al. (2019) Emad Aghajani, Csaba Nagy, Olga Lucero Vega-Márquez, Mario Linares-Vásquez, Laura Moreno, Gabriele Bavota, and Michele Lanza. 2019.  Software Documentation Issues Unveiled. In _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_. 1199–1210.  https://doi.org/10.1109/ICSE.2019.00122
  * Ahmed et al. (2023) Mohsin Ahmed, Saif Ur Rehman Khan, and Khubaib Amjad Alam. 2023\.  An NLP-based quality attributes extraction and prioritization framework in Agile-driven software development.  _Automated Software Engineering_ 30, 1 (2023), 7\.  https://doi.org/10.1007/s10515-022-00371-9
  * Angelov (2020) Dimo Angelov. 2020\.  Top2vec: Distributed representations of topics.  _arXiv preprint arXiv:2008.09470_ (2020).
  * Bianchi et al. (2021) Federico Bianchi, Silvia Terragni, Dirk Hovy, Debora Nozza, and Elisabetta Fersini. 2021.  Cross-lingual contextualized topic models with zero-shot learning.  _EACL 2021 - 16th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference_ (4 2021), 1676–1683.  https://doi.org/10.18653/V1/2021.EACL-MAIN.143
  * Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003\.  Latent dirichlet allocation.  _Journal of machine Learning research_ 3, Jan (2003), 993–1022.
  * Blumenthal (2022) Johanna Blumenthal. 2022\.  Thinking like a Lawyer: Why You or Your IT Team Needs to Keep Your Software Systems up-to-Date.  _SIGCAS Comput. Soc._ 50, 3 (8 2022), 10\.  https://doi.org/10.1145/3557900.3557905
  * Butler et al. (2022) Simon Butler, Jonas Gamalielsson, Björn Lundell, Christoffer Brax, Anders Mattsson, Tomas Gustavsson, Jonas Feist, Bengt Kvarnström, and Erik Lönroth. 2022.  Considerations and challenges for the adoption of open source components in software-intensive businesses.  _Journal of Systems and Software_ 186 (4 2022), 111152\.  https://doi.org/10.1016/J.JSS.2021.111152
  * Carbonell and Goldstein (1998) Jaime Carbonell and Jade Goldstein. 1998.  The use of MMR, diversity-based reranking for reordering documents and producing summaries. In _Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval_. 335–336.
  * Carvalho et al. (2014) Nuno Ramos Carvalho, Alberto Simoes, and José Joao Almeida. 2014.  DMOSS: Open source software documentation assessment.  _Computer Science and Information Systems_ 11, 4 (2014), 1197–1207.
  * Chen et al. (2023) Eason Chen, Ray Huang, Han-Shin Chen, Yuen-Hsien Tseng, and Liang-Yi Li. 2023.  GPTutor: a ChatGPT-powered programming tool for code explanation.  _arXiv preprint arXiv:2305.01863_ (2023).
  * de Graaf et al. (2014) Klaas Andries de Graaf, Peng Liang, Antony Tang, and Hans van Vliet. 2014.  The Impact of Prior Knowledge on Searching in Software Documentation. In _Proceedings of the 2014 ACM Symposium on Document Engineering_ _(DocEng ’14)_. Association for Computing Machinery, New York, NY, USA, 189–198.  https://doi.org/10.1145/2644866.2644878
  * Egger and Yu (2022) Roman Egger and Joanne Yu. 2022.  A topic modeling comparison between lda, nmf, top2vec, and bertopic to demystify twitter posts.  _Frontiers in sociology_ 7 (2022), 886498.
  * Forman and Damschroder (2007) Jane Forman and Laura Damschroder. 2007.  Qualitative content analysis.  In _Empirical methods for bioethics: A primer_. Emerald Group Publishing Limited, 39–62.
  * Garrison et al. (2006) D Randy Garrison, Martha Cleveland-Innes, Marguerite Koole, and James Kappelman. 2006.  Revisiting methodological issues in transcript analysis: Negotiated coding and reliability.  _The internet and higher education_ 9, 1 (2006), 1–8.
  * Gilson et al. (2019) Fabian Gilson, Matthias Galster, and François Georis. 2019\.  Extracting Quality Attributes from User Stories for Early Architecture Decision Making. In _2019 IEEE International Conference on Software Architecture Companion (ICSA-C)_. 129–136.  https://doi.org/10.1109/ICSA-C.2019.00031
  * GitHub User (2019a) GitHub User. 2019a.  Improve documentation.  https://github.com/SatoshiPortal/cyphernode/issues/62
  * GitHub User (2019b) GitHub User. 2019b.  This page is too technical.  https://github.com/google/WebFundamentals/issues/7282
  * GitHub User (2020) GitHub User. 2020\.  Term too technical.  https://github.com/nus-cs2113-AY2021S1/pe-dev-response/issues/1723
  * GitHub User (2022) GitHub User. 2022\.  Normie-fy the documentation.  https://github.com/abdulqshabbir/star-wars/issues/5
  * Grootendorst (2020) Maarten Grootendorst. 2020\.  KeyBERT: Minimal keyword extraction with BERT.  _Zenodo_ (2020).
  * Grootendorst (2022) Maarten Grootendorst. 2022\.  BERTopic: Neural topic modeling with a class-based TF-IDF procedure.  (3 2022).
  * Gupta et al. (2022) Hritik Gupta, Alka Chaudhary, and Anil Kumar. 2022\.  Identification and Analysis of Log4j Vulnerability. In _2022 11th International Conference on System Modeling & Advancement in Research Trends (SMART)_. 1580–1583.  https://doi.org/10.1109/SMART55829.2022.10047372
  * Imani et al. (2024) Aaron Imani, Iftekhar Ahmed, and Mohammad Moshirpour. 2024\.  Does Documentation Matter? An Empirical Study of Practitioners’ Perspective on Open-Source Software Adoption.  (3 2024).  https://doi.org/10.6084/m9.figshare.24210366.v1
  * Li et al. (2022) Xiaozhou Li, Sergio Moreschini, Zheying Zhang, and Davide Taibi. 2022.  Exploring factors and metrics to select open source software components for integration: An empirical study.  _Journal of Systems and Software_ 188 (2022), 111255.  https://doi.org/10.1016/j.jss.2022.111255
  * Madaehoh and Senivongse (2022) Arkom Madaehoh and Twittie Senivongse. 2022.  OSS-AQM: An Open-Source Software Quality Model for Automated Quality Measurement. In _2022 International Conference on Data and Software Engineering (ICoDSE)_. 126–131.  https://doi.org/10.1109/ICoDSE56892.2022.9972135
  * McInnes et al. (2017) Leland McInnes, John Healy, and Steve Astels. 2017\.  hdbscan: Hierarchical density based clustering.  _J. Open Source Softw._ 2, 11 (2017), 205.
  * McInnes et al. (2018) Leland McInnes, John Healy, and James Melville. 2018\.  UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction.  (2 2018).  https://arxiv.org/abs/1802.03426v3
  * OpenAI (2023) OpenAI. 2023.  ChatGPT.
  * Plösch et al. (2014) Reinhold Plösch, Andreas Dautovic, and Matthias Saft. 2014\.  The Value of Software Documentation Quality. In _2014 14th International Conference on Quality Software_. 333–342.  https://doi.org/10.1109/QSIC.2014.22
  * Rathee and Chobe (2022) Sachin Rathee and Amol Chobe. 2022.  Open Source Growth and Trends.  In _Getting Started with Open Source Technologies_. Apress, Berkeley, CA, 149–169.  https://doi.org/10.1007/978-1-4842-8127-7{_}8
  * Read the Docs Inc & contributors ([n. d.]) Read the Docs Inc & contributors. [n. d.].  Read the Docs.  https://docs.readthedocs.io/en/stable
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.  (8 2019).
  * Sas et al. (2022) Cezar Sas, Andrea Capiluppi, Claudio Di Sipio, Juri Di Rocco, and Davide Di Ruscio. 2022\.  Gitranking: A Ranking of Github Topics for Software Classification Using Active Sampling.  _SSRN Electronic Journal_ (2022).  https://doi.org/10.2139/ssrn.4182105
  * Sphinx developers ([n. d.]) Sphinx developers. [n. d.].  Sphinx Documentation.  https://www.sphinx-doc.org/en/master/index.html
  * Synopsis (2023) Synopsis. 2023\.  _Open Source Security and Risk Analysis Report_.  Technical Report.
  * Tang and Nadi (2023) Henry Tang and Sarah Nadi. 2023.  Evaluating Software Documentation Quality. In _2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR)_. 67–78.  https://doi.org/10.1109/MSR59073.2023.00023
  * Treude et al. (2020) Christoph Treude, Justin Middleton, and Thushari Atapattu. 2020\.  Beyond Accuracy: Assessing Software Documentation Quality. In _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ _(ESEC/FSE 2020)_. Association for Computing Machinery, New York, NY, USA, 1509–1512.  https://doi.org/10.1145/3368089.3417045
  * Wohlin (2014) Claes Wohlin. 2014\.  Guidelines for snowballing in systematic literature studies and a replication in software engineering. In _Proceedings of the 18th international conference on evaluation and assessment in software engineering_. 1–10.
  * Yılmaz and Kolukısa Tarhan (2022) Nebi Yılmaz and Ayça Kolukısa Tarhan. 2022.  Quality evaluation models or frameworks for open source software: A systematic literature review.  _Journal of Software: Evolution and Process_ 34, 6 (2022), e2458.  https://doi.org/10.1002/smr.2458
  * Zhao et al. (2021) He Zhao, Dinh Phung, Viet Huynh, Yuan Jin, Lan Du, and Wray Buntine. 2021\.  Topic modelling meets deep neural networks: A survey.  _arXiv preprint arXiv:2103.00498_ (2021).