# Open Source Software Development Tool Installation: Challenges and
Strategies For Novice Developers

Larissa Salerno<EMAIL_ADDRESS>0009-0004-0010-5109 The
University of MelbourneMelbourneAustralia ,  Christoph Treude
<EMAIL_ADDRESS>0000-0002-6919-2149 Singapore Management
UniversitySingaporeSingapore and  Patanamon Thongtatunam
<EMAIL_ADDRESS>0000-0001-6328-8839 The University of
MelbourneMelbourneAustralia

(2018)

###### Abstract.

As the world of technology advances, so do the tools that software developers
use to create new programs. In recent years, software development tools have
become more popular, allowing developers to work more efficiently and produce
higher-quality software. Still, installing such tools can be challenging for
novice developers at the early stage of their careers, as they may face
challenges, such as compatibility issues (e.g., operating systems). Therefore,
this work aims to investigate the challenges novice developers face in
software development when installing software development tools. To
investigate these, we conducted an analysis of 24 live software installation
sessions to observe challenges and comprehend their actions, the strategies
they apply, and the type of source of information they consult when
encountering challenges. Our findings show that unclear documentation, such as
installation instructions, and inadequate feedback during the installation
process are common challenges faced by novice developers. Moreover,
reformulating search queries and relying on non-official documentation were
some of the strategies employed to overcome challenges. Based on our findings,
we provide practical recommendations for tool vendors, tool users, and
researchers.

software development tool, tool installation, challenges, strategies

††copyright: acmcopyright††journalyear: 2018††doi: XXXXXXX.XXXXXXX††ccs:
Software and its engineering Documentation††ccs: Software and its engineering
Software usability††ccs: Software and its engineering Empirical software
validation††ccs: Software and its engineering Open source model

##  1\. Introduction

Software development tools play an essential role in software development
processes. They help software teams in the development of features, as well as
in other activities such as monitoring the progress of software tasks
(Ramírez-Mora et al., 2020). These tools support developers in many different
ways; for instance, Astah UML helps software developers with architecture
design, and Git supports versioning of source code. Additionally, each tool
has its own features and sometimes different purposes, which may be better
suited for specific contexts or projects. Installing software development
tools can be considered an intricate process (Hassan and Wang, 2017), given
that they may have specific system requirements (e.g., compatible machine
environments), various forms of installers (e.g., a binary file, source code),
and numerous steps of installation. Therefore, developers may encounter issues
during the installation process.

Recent studies have shed light on how installing software development tools is
a challenge for novice developers. For example, research conducted by
Steinmacher et al. (Steinmacher et al., 2015a) pointed out that one of the
barriers newcomers face when first contributing to open-source projects was
the hurdles of configuring the development environment, which also involves
the installation of tools. Other studies (Valez et al., 2020; Jenkins et al.,
2012) also highlight the hurdles that junior developers face when installing
tools. In both papers, participants complained that they sometimes spent hours
or even days installing and configuring software development tools.

Although studies have identified that installing software development tools is
often challenging for newcomers to software development, no studies have
investigated what kind of challenges they actually face and what strategies
they apply to overcome such challenges, as well as the type of information
novice developers seek during the installation of software development tools.
Identifying the challenges novice software developers face when installing
tools could help software organisations better understand how the installation
process of software development tools can be enhanced, making the installation
process less difficult for both newcomers and experienced software developers.
The findings can also help us in creating potential solutions to these
problems, such as establishing a tool or a guideline, therefore speeding up
the software development process and increasing tool accessibility and
quality.

In this work, we conduct qualitative research to investigate the challenges
novice developers face when installing software development tools and
comprehend the decisions and actions they make when encountering such
problems. More specifically, we conduct think-aloud sessions to observe
participants while they install the tools. We analysed the data using
qualitative analysis and extracted from the transcripts the challenges and
strategies from each session.

Based on 24 think-aloud sessions with 18 participants to install eight
software development tools, we found that the main challenges participants
face are related to the unavailability of simpler ways to install the tools,
and also knowing if the installation was successful or not. In terms of
strategies, rephrasing search queries and seeking help from alternative
sources were the most recurring strategies. Regarding the sources the
participants consulted, seeking help from online community forums proved to
help the participants overcome some of the challenges. To address challenges
and strategies, we make a list of recommendations for tool vendors, tool
users, and researchers.

##  2\. Related Work

The installation of software development tools can be a challenging task for
developers, especially for newcomers in the field, as they may face obstacles
along the process (e.g., complex tool configuration and documentation). To the
best of our knowledge, none of the related work has investigated the actual
challenges software developers face when they install software development
tools, as well as the strategies they apply to overcome challenges. In this
section, we present the findings of the related work that motivates our study.
We review research related to Newcomers and Open-Source Contributions since
installation challenges have often been considered as one of the many problems
newcomers encounter when trying to join Open Source, although none of the
related work has focused on the actual installation challenges. Work related
to Software Packages is relevant in particular in the context of Web IDEs
which might make installation processes obsolete, and Documentation is one of
the key sources that novice developers consider when installing software
development tools.

###  2.1. Newcomers and Open Source Contributions

Recent studies have shed light on how installing software development tools is
challenging for developers. Researchers conducted a literature review on the
social and technical barriers that newcomers face when they contribute to
open-source projects (Steinmacher et al., 2015a). In total, they identified 20
studies that provide empirical evidence of barriers newcomers often face when
making a contribution. Barriers such as the set-up of the local environment,
particularly the complexities associated with installing additional software
development tools have shown to be a significant barrier for newcomers seeking
to contribute effectively to open source projects. In a follow-up study
(Steinmacher et al., 2016), the researchers developed and explored the
effectiveness of a portal called FLOSSCoach, a tool that overcomes hurdles
identified in the previous study. The study showed that the portal could
successfully overcome some of the social challenges, especially regarding
communication issues. Despite that, the portal was not successful at
overcoming technical barriers such as workspace setup issues and
documentation. The portal could not diminish these aspects because it does not
provide any tool or mechanism to support newcomers when it comes to technical
issues. According to participants, these aspects bring a sentiment of
“frustration, irritation or demotivation” when facing these challenges.

From the results of the two studies mentioned previously, the researchers
developed a guideline (Steinmacher et al., 2018) for communities seeking to
support newcomers and also for newcomers who want to start contributing to OSS
projects. Regarding the hurdles of the local environment setup, the
researchers recommend OSS communities to make it easy for newcomers to build
the system locally. They emphasise that setting up the local workspace was the
most reported challenge in their first study, and this barrier reinforced how
newcomers feel frustrated and demotivated, even after using the FLOSSCoach
portal. Researchers suggest creating detailed step-by-step tutorials that
include links to information about common problems and potential solutions,
such as a FAQ section.

Another study collected students’ perceptions regarding the use of open source
projects in the teaching process of Software Engineering courses (Pinto et
al., 2019). The researchers conducted semi-structured interviews with 21
students (undergraduate and graduate level) from five different universities.
Despite the students’ positive impressions, the findings revealed that setting
up the development environment remained a significant challenge for many
participants. One participant described it as ”not uncommon to have a headache
to set up the project”. This shows that the process of setting up the
environment can demotivate students to contribute to OSS projects.

Researchers also allege that students hardly gain the industry-focused, hands-
on experience required to develop software products and services for employers
after they graduate (Kal, 2006). To resolve this issue, the researchers
propose a pedagogy to enrich the practical aspects of Software Engineering
(SE) education, identifying what educators can do to make good use of OSS
tools and components. One of his findings validates that OSS projects in most
cases do not provide guidance and resources for students to employ open source
projects.

Previous works have identified that installing software development tools is a
hurdle from the perspective of newcomers. Studies (Steinmacher et al., 2015b;
Stol et al., 2010; Pinto et al., 2019; Steinmacher et al., 2015a; Kal, 2006)
provided evidence that challenges range from lack of documentation and
instructions to the need to install dependencies. However, none of the studies
have observed developers to document the actual challenges they face when
installing software development tools, as well as the strategies they apply to
overcome such challenges, which is the main motivation for our study.
Additionally, even though researchers have developed effective tool support
(Steinmacher et al., 2016, 2018) to help them overcome social issues, little
is known about effective strategies that help developers overcome technical
issues in the process of installing software development tools.

###  2.2. Software Packages

Although the previous studies mentioned above are strictly about OSS projects,
a recent study (Valez et al., 2020) presents KODETHON, a Web Integrated
Development Environment (web IDE) application to support students who are
learning programming to focus on the primary learning objectives. According to
the researchers, this type of application is ideal in teaching environments
because it can reduce the time students waste installing and configuring
tools. The application has been used by more than 3,000 students from 15
different courses. According to the findings, 61% of participants reported
that one of the valuable features of the application is not having to install
and configure anything on their machine. In a similar study, researchers
designed an application called JavaWIDE (Java Wiki Integrated Development
Environment) (Jenkins et al., 2012). The application was created to support
the teaching of programming using Java. According to students and teachers,
they highlighted how having a tool that does not require installation or
configuration eliminated a considerable number of technical problems that they
could have faced.

Along the same line, a researcher developed Online Python Tutor (Guo, 2013), a
web-based tool that does not require any software or plugin installation. The
tool has gained popularity, with over 200,000 people, including instructors
and students, using it for teaching and learning purposes. The fact that the
tool does not require installation has been highlighted by many students as a
positive aspect. This eliminates the need for time-consuming installations and
configurations, allowing students to focus on learning and programming tasks
instead.

Researchers (Helminen et al., 2013) investigated the challenges students face
during the process of learning programming, and according to student feedback,
software packages were indeed helpful. Although Web IDEs can be a solution to
potential challenges during tool installation, these previous studies do not
conduct an investigation of the actual challenges faced by novice developers.
In addition, it is important to note that Web IDEs are not a remedy for all
software development tool installation scenarios. Not all software packages
and development tools can be seamlessly integrated or utilized within a Web
IDE framework due to limitations in accessing hardware resources, complexities
in simulating specific development environments, or restrictions in
customising the development setup to match real-world conditions.

###  2.3. Documentation

Researchers have been investigating the issues that practitioners in Software
Engineering encounter in software documentation (Uddin and Robillard, 2015;
Chen and Huang, 2009). In a first study (Aghajani et al., 2019), the
researchers mined the problems related to documentation in four sources:
Issues, mailing lists, pull requests, and Stack Overflow. After manually
labelling the data, they identified 162 software document issues. In order to
validate the results from the study, the researchers conducted a second study
(Aghajani et al., 2020) where they validated the 162 taxonomies with 146
professional software practitioners. One of the most significant results was
regarding installation instructions, where 65% of the participants reported
that faulty tutorial is a problem they usually find in software documentation.
A second aspect is the inappropriate installation instructions, which was the
most frequent issue according to the practitioners. This shows that software
documentation is a problem for developers when they are installing software
development tools.

Another study (Gao et al., 2023) has focused on simplifying README files to
make them more comprehensive for software developers. Despite the availability
of guidelines and automated tools to extract information from software (Liu et
al., 2022; Hassan and Wang, 2017; Robillard et al., 2017), software developers
often do not create clear and reliable documentation. One of the ultimate aims
of our research is to lay the groundwork for a technique that can
automatically extract reliable instructions and also present solutions to the
most common errors that users encounter when installing software development
tools.

In order to understand to what extent software tutorials are executable,
researchers conducted an empirical study (Mirhosseini and Parnin, 2020) to
understand the issues software tutorials have and what can be done to avoid
them. Through the lens of executability, the researchers measured to what
extent the step-by-step tutorials could be followed until their final working
set-up. After analysis of over 600 tutorials, the results showed that none of
the tutorials executed all the steps successfully until the end. Some of the
issues identified were the resources’ inaccessibility, missing instructions,
the inconsistency in handling file content, and the bad quality of
documentation. A study on Application Programming Interfaces (APIs) highlights
the dispersion of documentation across the internet (Treude and Aniche, 2018),
which can be confusing for developers who are seeking software documentation.
To address this issue, the researchers replicated a previous work on the topic
(Parnin and Treude, 2011) and complemented the work by selecting ten popular
APIs with the aim of covering a variety of programming languages such as Java,
Javascript, PHP, C++, and Python. The study found that, in addition to
official documentation, GitHub and Stack Overflow are the sources that appear
most frequently on the first page of Google search results for the selected
APIs. This indicates that developers often rely on these platforms for finding
information related to APIs. However, the dispersed documentation across
multiple sources can also lead to duplicate information, which can further
confuse developers.

Researchers have also found that creating and maintaining step-by-step
tutorials is difficult (Lafreniere et al., 2013; Krosnick, 2015; Pavel et al.,
2014), and in an attempt to overcome some of the challenges tools have been
developed (Mysore and Guo, 2017; Lafreniere et al., 2013). The studies
mentioned in this section highlight several documentation issues and also
introduce tools designed to overcome some of these problems. To better
understand how novice developers search for information in software
development tool installation, we analyse and map the types of sources they
consulted during think-aloud sessions, connecting them with the challenges
they encounter and the strategies they apply.

##  3\. Methodology

We used qualitative research methods (i.e., surveys and observation) to
conduct the study, as well as a number of tools to support data collection and
analysis. To guide our research, we addressed the following research
questions:

  * •

RQ1. What are the challenges during the installation process of software
development tools? So far, no studies have focused on explicitly identifying
the hurdles novice developers face when installing such tools. We aim to
identify the challenges via software installation sessions, where participants
will be asked to install a software development tool.

  * •

RQ2. What strategies do participants apply to overcome the challenges?
Understanding the strategies they employ, we can identify effective or non-
effective strategies that can support other software developers and tool
vendors.

  * •

RQ3. What sources do participants consult when they face such challenges? Our
objective is to identify sources on which participants rely when implementing
strategies to overcome the challenges. As part of this, we study the
relationship between challenges, strategies, sources, and tools.

###  3.1. Participants

After obtaining approval from our institution’s ethical review board, the
participants for our study were university students undertaking a computer-
related degree. We have decided to choose university students close to
graduation, as previous work, such as the study carried out by Salman et al.
(Salman et al., 2015), evince that students can perform the role of
professional developers in Software Engineering experiments. The challenges
encountered by students close to graduation are a good proxy for the
challenges encountered by professionals at the beginning of their career. The
recruitment process occurred through posting announcements about the research
and verbally advertising the research in computing and/or software engineering
classes. To express interest in participation, students will submit a
demographic survey, and qualified students (e.g., undertaking a computer-
related degree) will be invited to participate. To incentivise participation,
students were offered compensation in the form of a gift card to attend the
think-aloud sessions to install software tools.

Research conducted by Falessi et al. (Falessi et al., 2018) provides empirical
support that students are a suitable sample for conducting experiments in
Software Engineering. In this study, 64 experts in conducting experiments in
SE were invited to participate in focus group sessions, where they were asked
to report and discuss their past experiences running experiments with both
professionals and students. The results revealed a misconception that students
are not representatives of professionals. Another study conducted by Sjøberg
et al. (Sjoberg et al., 2002) reinforces this argument by pointing out that in
2 of their previous experiments, 40% of the students were working part-time or
had previously worked in full-time jobs.

In total, 18 students participated in the study, of which 13 were Master’s
students and the other five were undergraduate students who were majoring in a
computer-related discipline. We consider this sample size to be satisfactory
given the time and effort required to conduct and analyse the think-aloud
sessions. With this group of participants, we were able to run the study with
participants with different levels of knowledge and experience (see Table 1).
The number of participants is also similar to previous qualitative studies
(Thiselton and Treude, 2019) (Robillard and Treude, 2020) that conducted
think-aloud sessions with 16 and 18 participants. Towards the end of the
analysis process, we realised that there was a saturation of codes, as no new
major codes were identified with the last few participants.

###  3.2. Data Collection

Each participant individually responded to a short questionnaire and then
participated in live software installation sessions in a think-aloud format.
As shown in Table 1, the short questionnaire included questions concerning
their degree, their previous experience installing software development tools
using different operating systems (e.g., Windows or Linux), and the tools they
would like to install during the think-aloud session. A list of software
development tools was provided in the questionnaire to facilitate and ensure
that all the tools on the list were installed at least once. We chose the
tools based on the OSSRank website (Walker, 2022), which is a website that
ranks the most popular open-source software development tools among
developers.

Table 1. Demographics of participants.

Survey Questions |  Type | Distribution of answers
---|---|---
What degree are you currently studying? |  Multiple choice | Master of Information Technology: 33.33%
Master of Software Engineering: 33.33% |  |
Bachelor of Science: 27.77% |  |
Master of Computer Science: 5.55% |  |
What operating system do you prefer for using |  |
software development tools? |  Multiple choice | Windows: 44.44%
MacOS: 44.44% |  |
Linux: 11.11% |  |
Tell us how frequently you use the command prompt |  |
for installing software development tools? |  Likert scale | Never: 5.55%
Rarely: 5.55% |  |
Sometimes: 22.22% |  |
Often: 50% |  |
Always: 16.66% |  |
How confident are you in installing software |  |
development tools using the operating systems below? |  |
Where: 0 - Not confident at all 4 - Very confident |  Likert scale | Windows: 2.72
MacOS: 2.28 |  |
Linux: 2 |  |

Inspired by the study by Teague and Lister (Teague and Lister, 2014), live
software installation sessions were held in a think-aloud format. We consider
think-aloud sessions a suitable method for this study, since we can observe
the actual behaviours of the participants along with their thoughts. Each
participant was observed for their progress by a researcher throughout the
session; the observer took notes about their actions and thoughts during the
installation process, and at the end of the session, the observer verbally
asked what challenges they faced during the installation process. During the
sessions, our primary focus was to capture real-time occurrences rather than
solely focusing on the participants’ thoughts. For this reason, the observer
frequently asked questions to the participants, such as ”What is happening
now?” or ”Have you ever faced this (challenge) before?”. In the think-aloud
sessions, participants were asked to install the tools they had selected in
the questionnaire, and if they would like to, they had permission to install
additional tools from the list. At the beginning of the sessions, we explained
to them the concept of a think-aloud session and that during the installation
process they would have to share their thoughts and whatever was going through
their minds.

In order to support data collection, several tools were used to record actions
and interviews during the observation sessions. The actions related to
software installations were observed and captured via screen recording
software to track their actions during the installation tasks. We also made
use of audio recordings to keep track of participants’ think-aloud thoughts.
Screen and voice recordings were captured using Zoom. As a backup plan, we
video-recorded the sessions to help researchers during the analysis process of
the think-aloud sessions (e.g., if the participant was pointing at the screen
to show a determined aspect). The camera was placed in a way that did not
record the participant’s face.

###  3.3. Data Analysis

We used qualitative analysis to analyse the data collected during the think-
aloud sessions. In total, we conducted 24 live software installation sessions,
in which we collected 534 minutes of video recording and 187 pages of
transcripts. The analysis process was divided into 3 steps, as shown in Figure
1. In step 1, Author 1 transcribed the think-aloud sessions. Then Author 1
identified and highlighted the challenges and strategies reported by the
participants in all transcripts. To confirm the results from Author 1, in step
2, Author 2 and Author 3 individually identified and highlighted the
challenges and strategies they could identify in the same two transcripts. The
three authors compared and discussed the results multiple times and revisited
the transcripts for consistency until they reached general agreement. In step
3, Author 1 started extracting codes from the challenges and strategies
highlighted in the transcripts. Then, all authors discuss and reformulate the
codes iteratively. In total, we extracted 23 codes, where 10 are related to
the challenges developers face when installing software development tools, and
13 are the strategies they employ when faced with these challenges.

Figure 1. Data analysis process.

Figure 2 illustrates an example of our code extraction process for an excerpt
taken from one of the transcripts. The text highlighted in dark grey is
identified as a challenge, the text highlighted in light grey is identified as
a strategy, and the text in bold represents the participant’s action. After
the codes emerged from Step 3, we identified the challenge ”I didn’t find any
useful links here” is part of the code Under-specified Query, and the strategy
”so I go back and maybe choose the first one for Windows goes back to Google
and adds ”for windows” on Google search” is related to the code Reformulating
the Query.

Figure 2. Example of code extraction.

###  3.4. Validation Survey

To validate our findings from the think-aloud sessions, we conducted a survey
with a larger group of students to substantiate the findings of our study. The
questionnaire was distributed to 148 third-year undergraduate university
students who took a Software Engineering course with a focus on software
processes. In this course, students were introduced to OSS development,
including the history of OSS development, processes, practices, licenses, and
tools. These topics were addressed through lectures and practical exercises,
covering aspects such as source code management, code review, and continuous
integration using GitHub.

In the questionnaire, we listed the challenges and strategies that we
identified from the think-aloud questions and asked them to indicate the
challenges they had previously encountered and the strategies they had applied
to address the challenges. We phrased the survey questions as ”Please select
below challenges you have faced when installing software development tools in
the past (Choose all that apply)” and ”Please select below the strategies you
usually apply when you face challenges installing software development tools
(Choose all that apply)” As part of the questionnaire, we offered participants
”Other(s)” and ”None of the above” options, just in case they wished to
include additional challenges or strategies. In total, we received 144
responses, from which we analysed and counted how many times a challenge and
strategy were selected. The results of the survey can be found in Section 4
(see Table 4 and Table 5). The results are displayed in the tables listing the
Challenges and Strategies as a column labelled ”Survey”. In both tables, a few
of the challenges and strategies are marked with ”-”, indicating their absence
in the survey as they emerged from the think-aloud sessions with the
additional participants after we sent out the survey.

##  4\. Results

In our study, 18 Master’s and undergraduate students participated in software
installation sessions in a think-aloud format. In total, seven software
development tools (listed in Table 3) were installed. Four participants
installed three tools and three participants installed four tools, resulting
in 24 sessions in total. We based our tool selection criteria on the ranking
of the top ten tools within the ten most popular categories of tools, which
included: Python libraries, Frameworks, and Database.

Table 3. List of tools selected for our study.

Tool ID |  | Tools |  | Type |  | No. of Sessions |  | Participants
---|---|---|---|---|---|---|---|---
T |  | TensorFlow |  | Python Library |  | 4 |  | P1, P6, P7, P11
F |  | Flutter |  | Framework |  | 4 |  | P2, P3, P6, P17
M |  | MongoDB |  | Database |  | 4 |  | P2, P8, P14, P17
P |  | PyTorch |  | Python Library |  | 3 |  | P5, P7, P10
J |  | Jupyter Notebook |  | Python Library |  | 3 |  | P4, P13, P16
N |  | Node.js |  | Framework |  | 3 |  | P12, P17, P18
S |  | Spring Boot |  | Framework |  | 3 |  | P9, P15, P16

We extracted ten challenges and 13 strategies from the think-aloud sessions
listed in Tables 4 and 5. In order to simplify the understanding of the
results, we organised the challenges and strategies into four categories:
Information Quality/Value, Information Seeking, Installation Process, and Tool
Characteristics. The category ”Information Quality/Value” represents
challenges and strategies that are related to the installation instructions
provided to the user (e.g., instructions from a website). The category
”Information Seeking” represents information other than official installation
instructions, but is still related to the installation that the participants
were looking for (e.g., tutorial). The ”Installation Process” category
pertains to the events that occurred when the participants were in the process
of or about to initiate the tool installation. Lastly, the ”Tool
Characteristics” category comprises the challenges or strategies that occurred
due to a particular attribute of the tool.

###  4.1. Challenges (RQ1)

Table 4 lists the ten challenges grouped into four main categories that we
identified from our think-aloud sessions.

Table 4. List of challenges the participants encountered during the think-
aloud sessions.

Challenges | No. of Sessions | Survey
---|---|---
Information Quality/Value |  |
Unclear Instructions: installation instructions provided are not clear | 11 | 88
Dysfunctional Instructions: instructions do not work | 8 | 62
Information Seeking |  |
Under-specified Query: participant searches for information using a non-specific query | 4 | 33
Information Overload: web page or terminal shows the user too much information | 4 | 51
Poorly-presented Information: source is not user-friendly | 5 | 67
Installation Process |  |
Complicated Installation Process: source does not provide a straightforward way to install the tool | 20 | 52
Lack of Installation Progress Feedback: no feedback is provided on the status of the installation | 12 | 29
Error Message: participant gets stuck with error messages | 8 | -
Tool Characteristics |  |
Account Required: tool required the user to create an account on their platform | 1 | 41
Version Incompatibility: CPU or library version is not compatible with the tool | 4 | 30

####  4.1.1. Information Quality/Value

We identified two challenges that fall under the Information Quality/Value
category, i.e., Unclear Instructions and Dysfunctional Instructions.
Participants faced the challenge of unclear instructions in nine sessions and
the challenge of dysfunctional instruction in seven sessions.

Unclear Instructions refers to the challenges when participants read
instructions to install the tool, but could not follow or understand due to
the somewhat unclear or complicated instructions. This challenge occurred
during the installation of MongoDB, PyTorch, Node.js, and Flutter. In two of
the sessions on MongoDB installation, participants searched for instructions
to download and install MongoDB using Google search engine. Both participants
found that the official website had two different pages containing different
instructions on how to install the tool, i.e., one page instructs one to
download and install the tool from a zip file and another page instructs to
use a package manager (Homebrew) to install the tool. Such inconsistent and
unclear instructions confused the participants: ”Now there is another type of
instructions on another page and it says that to install it I need to use
Homebrew, and this is annoying” (P2). In another session of installing
MongoDB, participants accessed the official website and found the page with
the installation instructions. The participants got confused because the page
showed two different instances of MongoDB (see Figure 3), and the participants
did not know which one they were supposed to download $-$”I don’t know what
the difference is […]” (P8). Three participants faced this uncertainty about
the right version when installing MongoDB. Besides, 88 respondents in our
validation survey reported having encountered unclear instructions when
installing a software development tool.

Figure 3. Screenshot of MongoDB official website displaying a list of
different versions.

Dysfunctional Instructions refers to the challenges when the participants
followed the installation instructions, but the installation outcomes were not
as described in the step-by-step guide. This challenge occurred when the
participants installed TensorFlow, MongoDB, Spring Boot, and Node.js. When
installing TensorFlow, one of the participants followed the installation
instructions of the official website as the main source of information. While
the participant followed the steps, the participant was required to run a
command on the terminal. When the participant ran the command, they got an
error message saying that the terminal did not recognise the command. In the
MongoDB installation session, the participant followed the instructions on the
official website. When the participant copied the command from the instruction
page and pasted it into the terminal, they received an error message stating
”Error: Your Command Line Tools are too outdated”. The participant was
confused and could not proceed further or solve the problem: ”What? ”Your
Command Line Tools are too outdated?” That does not show you any updates on
how to do this. Will this take a long time? I do not know” (P14). In our
validation survey, 62 respondents revealed to have faced this challenge in the
past.

####  4.1.2. Information Seeking

We identified three challenges that are related to information, i.e., Under-
specified query, Information Overload, and Poorly-structured Information.

Under-specified Query refers to the challenges the participants faced due to
unspecified queries when searching for how to install the tool. Participants
faced this challenge in four sessions when installing TensorFlow, MongoDB,
Flutter, and Spring Boot. In all of the sessions, the participants tried to
search for a web page of installation by querying search engines (e.g.,
Google). However, on first attempt, they could not find a web page with
instructions because they were only searching for the tool name without
specific context or purpose, e.g., adding the word ”install” or ”tutorial”.
For instance, when installing TensorFlow, the participant first searched on
Google for ”TensorFlow”, and the results did not show any page with
instructions. The participant realised that they would need to reformulate the
query in order to find the page with instructions $-$”[..] I’m gonna search
”TensorFlow”. Hum… I’m looking for a Mac version and I want to have a GPU
acceleration for it, so I’m gonna search for ”mac GPU TensorFlow.” (P1). The
same problem occurred when installing Spring Boot with a different search
engine. The participant searched for ”Spring Boot download” on YouTube, but
the search results only showed tutorials on how to install Spring Tool that
are irrelevant to Spring Boot: Wait… what is it? “Spring Tool”? This is
confusing…” (P9). This challenge was also reported by 33 respondents in our
validation survey.

Information Overload refers to challenges when the web page or a terminal
shows excessive information that participants could not follow. This challenge
occurred in four sessions, where both participants were installing TensorFlow.
In one of the sessions, the participant was in the process of installing the
tool using the terminal. By running one of the commands according to the
instructions, the participant was surprised with the amount of information
from the terminal, and the participant could not understand if the information
was related to an error or not (see Figure 4): ”The terminal shows a big
amount of information, but it doesn’t say if it’s an error or not” (P11). In
another session, while the participant was following the installation
instructions, the terminal showed the License Terms information. The
participant felt bothered by the amount of information and said $-$ ”A lot of
text to read…” (P1). Information overload was also perceived by 51 respondents
in our validation survey.

Figure 4. Terminal output displaying overloaded information during the
installation of TensorFlow.

Poorly-presented Information refers to the challenges when participants
struggled with installation instructions due to presentation of the
documentation. This challenge occurred in five sessions when participants were
installing TensorFlow, MongoDB, Spring Boot, Jupyter Notebook, and Node.js.
When installing MongoDB, one of the participants downloaded the package
containing executable files. Within the folder, a README file was included
(see Figure 5), which was supposed to provide instructions on how to proceed
with the installation. However, the structure of the document made the
participant express confusion, stating: ”So… this is quite odd… the README
doesn’t give much instruction about it” (P2). In another session, when
installing TensorFlow, the participant found the page with the installation
instructions on the official website, but the information was confusing:”This
page is not great…” (P6). Among the respondents in our validation survey, 67
reported having encountered this challenge when installing software
development tools in the past.

Figure 5. Instructions on the README file when installing MongoDB.

####  4.1.3. Installation Process

We identified three challenges that are related to the installation process,
i.e., Complicated Installation Process, Lack of Installation Progress
Feedback, and Error Message.

Complicated Installation Process refers to the challenges when the
participants found that there was no ”straightforward” way to install the
tools or that it required too much time.

This challenge commonly occurred in most of the sessions during the
installation of MongoDB, Spring Boot, TensorFlow, Flutter, and PyTorch. The
participant who installed MongoDB was annoyed when there was no direct link to
download the tool: ”Now something that I always dislike is there is no direct
link to the download part” (P2). The other two participants who installed
MongoDB had issues with the download of additional tools. Specifically, they
had to install a package manager called Homebrew. Participants found that
installing Homebrew and setting up the environment was more complicated and
took longer than downloading the tool itself. It took more than four minutes
to process each of the commands to install Homebrew: ”I do not know why it is
taking so long to update it, should not take that long to update things!”
(P14).

During the installation of PyTorch, the participant wanted to install the tool
without downloading any additional tools. The PyTorch website recommends
installing the tool using a Python package manager called Anaconda, but the
participant wished to find a different installation approach. Thus, the
participant searched for ”install PyTorch mac m1 without anaconda”, and when
they checked the search results, they realised that they all required Anaconda
$-$ ”Everything had Conda on it… Do I need to download Anaconda? Does it come
natively?” (P7). The Complicated Installation Process challenge was selected
by 52 respondents in our validation survey.

Lack of Installation Progress Feedback refers to the challenges when
participants did not receive progress feedback when installing the tool, which
makes participants confused if the installation was successful. This challenge
occurred in 12 sessions when installing TensorFlow, MongoDB, Flutter, Jupyter
Notebook, PyTorch, and Node.js. In the session of installing Flutter, when the
participant ran one of the installation commands on the terminal, the
participant did not receive any feedback or response after waiting for two
minutes. A similar problem occurred when installing Jupyter Notebook. This
participant followed the instructions from the official website, and when
running one of the commands, the terminal stopped showing data processing and
nothing happened for three minutes. Participants also expressed confusion when
there was no confirmation that the installation process was done. For
instance, when the participant executed all the installation instructions of
Node.js, they did not receive a confirmation message that the tool was
successfully installed: ”I don’t know if it’s finished or not, how can I
check?” (P12). This challenge was also mentioned by 29 respondents in our
validation survey.

Error Message was encountered when participants faced error messages while
attempting to install certain tools. This challenge appeared specifically
during the installation process of TensorFlow, MongoDB, and PyTorch. For
example, during the installation of PyTorch, one participant faced an error
message after following instructions from a website. This participant was
stuck, grappling with the same error for a long period, lasting more than ten
minutes.

####  4.1.4. Tool Characteristics

This category represents the challenges that participants faced when they had
to deal with the characteristics of the tool they were installing. We
identified two challenges that fall under the Tool Characteristics category,
i.e., Account Required and Version Incompatibility.

Account Required refers to the challenges when the tool requires participants
to create an account before installing it. This happened in one session during
the MongoDB installation process. When installing MongoDB, one participant was
looking for installation instructions on MongoDB’s official website. When
looking for instructions, MongoDB required the user to create an account
before having access to the installation instructions. According to our
validation survey results, 41 respondents have faced issues related to the
Account Required challenge in the past.

Version Incompatibility refers to challenges when the configuration of the
participant’s machine is incompatible with the tool they were installing. This
challenge occurred in four sessions where participants were installing MongoDB
and PyTorch. In all cases, they had incompatibility issues because their
machine had the Macbook M1 chip. Besides, this challenge was mentioned by 30
respondents in our validation survey.

As a result of the ”Other(s)” category from the survey, respondents pointed
out two additional challenges: ”The environment config [sic] is difficult
after installation” and ”The tool required a subscription to be used and the
subscription price is expensive”.

###  4.2. Strategies (RQ2)

Table 5 lists 13 strategies that participants applied to overcome the
challenges. In this subsection, we organise the strategies into the
corresponding challenge categories. See Table 7 for a more detailed analysis
of the relationship between challenges, strategies, and tools.

Table 5. Strategies participants applied to overcome the challenges.

Strategies | No. of Sessions | Survey
---|---|---
Information Quality/Value |  |
Searching for Different Documentation: participant searches for information on a different source | 22 | 95
Reading Help or Feedback in Terminal: participant uses the terminal instructions to find information | 7 | 82
Reading Instructions Carefully: participant overcomes the challenge by reading the instructions carefully | 7 | 83
Information Seeking |  |
Reformulating the Query: participant rephrases the query to get more specific results | 13 | 40
Web Searching: participant searches information (e.g., error messages) on the web | 7 | 101
Focusing on One Source: participant decides to follow instructions from a single source | 4 | 28
Installation Process |  |
Giving Up: participant gives up on installing the tool | 5 | 16
Resetting the Environment: participant opens a new tab on the terminal | 5 | 29
Reinstalling: participant decides to restart the installation process | 4 | 77
Waiting: participant decides to wait for something to happen | 3 | -
Random Actions: participant types random keys/commands | 3 | -
Ignoring: participant ignores error messages or information during the installation process | 8 | -
Tool Characteristics |  |
Versioning: participant changes or installs a new version of a tool | 2 | 48

####  4.2.1. Information Quality/Value

In this category, we identified three strategies, i.e., Searching for
Different Documentation, Reading Help or Feedback in Terminal, and Reading
Instructions Carefully.

Searching for Different Documentation was used in 22 sessions when installing
Flutter, PyTorch, MongoDB, TensorFlow, Spring Boot, Jupyter Notebook,
TensorFlow, and Node.js. Stack Overflow was one of the preferred alternative
sources that the participants used: ”For most things, yes. If it’s a Python
package I just look for the PIP or the keyword for PIP, but other than that I
would go to Stack Overflow” (P6). Similarly, in the session of installing
PyTorch, the participants attempted to follow the instructions on the tool’s
official website. However, the participants perceived that the instructions
there were not helpful. Then, they decided to follow the installation
instructions in the non-official documentation, as the participants perceived
it was easier to understand than the official documentation: ”Yes. I prefer
this because it has more explanation and it is more user-friendly” (P10). Out
of the nine sessions, looking for alternative sources helped the participants
overcome the challenges in three sessions. In eight out of nine sessions, the
participants preferred non-official documentation over the tool’s official
documentation. Similarly, participants often preferred to watch video
tutorials over reading extensive documents: ”I usually just look for them in
the software documentation, but sometimes it’s clearer to just see [sic] a
video, like maybe a five-minute walkthrough. If I get stuck I usually go watch
a video” (P8).

While some participants preferred non-official documentation, other
participants found that such documentation can be outdated. In one of the
sessions of installing MongoDB, the participant first followed the
installation instructions from a non-official website. When faced with
challenges, they searched for information in the official documentation as
they believed that it was a more reliable source:”The official website is more
recent, and if there is some sort of update, some other discussion forum would
be outdated.” (P14). Nevertheless, when participants faced an installation
error, reading the official documentation was not helpful to the participants
as it did not provide a solution for such an error. This strategy was applied
to the challenges: Unclear Instructions, Dysfunctional Instructions, Under-
specified query, Error Message, Complicated Installation Process, Lack of
Installation Progress Feedback, Account Required, and Version Incompatibility.
Additionally, 95 respondents in our validation survey mentioned having applied
this strategy in the past when installing software development tools.

Reading Help or Feedback in Terminal was used when participants tried to
understand the installation instructions. This strategy was used in seven
sessions when installing MongoDB, TensorFlow, Flutter, and Jupyter Notebook.
During the session of installing MongoDB, participants struggled to understand
how to run the installation command as per the instructions from the official
documentation. Then, one of the participants was looking for help on the
shell’s documentation using the ”–help” command. Similarly, in the TensorFlow
installation session, the participant had issues with an error message when
executing a command. One of their strategies was to carefully read all the
information shown on the terminal. This strategy was applied to the
challenges: Unclear Instructions, Dysfunctional Instructions and Information
Overload. In our validation survey, 82 respondents highlighted the use of this
strategy to overcome challenges.

Reading instructions carefully was used when the participants could not
understand or follow the installation instructions. This strategy was used in
seven sessions where participants were installing MongoDB, TensorFlow, and
Spring Boot, this strategy successfully helped the participants overcome the
challenge once. This strategy was applied to the challenges: Unclear
Instructions, Dysfunctional Instructions, Poorly-structured Information,
Information Overload, Complicated Installation Process, Lack of Installation
Progress Feedback, and Version Incompatibility. In total, 83 respondents
reported applying this strategy previously.

####  4.2.2. Information Seeking

In this category, we identified three strategies (i.e., Reformulating the
Query, Web Searching, and Focusing on One Source).

Reformulating the Query was used in 13 sessions when installing MongoDB,
TensorFlow, PyTorch, Flutter, Jupyter Notebook, and Spring Boot. As pointed
out in the challenge of Under-specified Query, the initial search keyword was
not specific (e.g, using only the tool name as a keyword search). Thus, the
participants attempted to overcome this challenge by reformulating the query
and adding more specifications or context, such as the operating system they
were using. For instance, when installing MongoDB, one of the participants
used only ”MongoDB” as a keyword to search on Google and did not find any
useful links. The participant then realised that additional keywords should be
used to retrieve more useful results: ”I couldn’t find the download directly
from that page, so I had to go back and search for ”download MongoDB”. Then, I
downloaded something that did not work, and then, by making a full query of
like ”installing MongoDB on MacOS”, then I got the instructions to use, like
the command line with Homebrew, and then it worked” (P2). This strategy was
applied to the challenges: Unclear Instructions, Dysfunctional Instructions,
Under-specified Query, Poorly-structured Information, Error Message, and
Complicated Installation Process. In our validation survey, 40 respondents
mentioned using this strategy as a means of overcoming challenges.

Focusing on One Source was used when the participant used installation
instructions on multiple sources at the same time, which turned out to be
problematic due to inconsistent instructions. To overcome that challenge, the
participant decided to follow the instructions of one source only, and that
solved the problem. This strategy was applied in two sessions during the
TensorFlow and PyTorch installation process. This strategy was applied to the
challenges: Dysfunctional Instructions, Under-specified Query, Poorly-
structured Information, Error Message, Complicated Installation Process, Lack
of Installation Progress Feedback, and Account Required. This strategy was
reported by 28 respondents in our validation survey.

####  4.2.3. Installation Process

In this category, we identified six strategies (i.e., Giving Up, Resetting the
Environment, Reinstalling, Waiting, Random Actions, and Ignoring).

Giving Up was applied when participants felt exhausted after spending a
considerable amount of time attempting to install the tool. One of the
participants mentioned: ”I’m gonna give up, the unzipping and extracting files
process is taking too long” (P6). In addition, some participants gave up on
installing the tool because they wanted to install it without having to
install additional tools. This strategy occurred in five sessions when
installing TensorFlow, MongoDB, and PyTorch. This strategy was applied to the
challenges: Dysfunctional Instructions, Poorly-structured Information, Error
Message, Complicated Installation Process, and Version Incompatibility. In
total, 16 respondents in our validation survey mentioned using this strategy
to address challenges they encountered previously.

Resetting the Environment was applied when participants faced errors and could
not understand the problem due to a lack of feedback on installation progress
when installing TensorFlow and PyTorch. In the five sessions, the participants
attempted to resolve the problem by resetting the environment (i.e., opening a
new terminal tab) and found that this strategy was helpful. This strategy was
applied to the challenge: Lack of Installation Progress Feedback. According to
our validation survey results, this strategy was selected by 29 respondents.

Reinstalling was another strategy used to overcome challenges when installing
MongoDB and TensorFlow. Participants in four sessions applied this strategy
because they thought it could be helpful; however, this strategy worked once
only. The student was unable to complete the installation of the tool after
multiple attempts and decided that reinstalling the tool from scratch could
work. This strategy was applied to the challenges: Dysfunctional Instructions
and Version Incompatibility. In our validation survey, 77 respondents revealed
having applied this strategy previously.

Waiting was an option for the participants in three sessions when installing
MongoDB and Jupyter Notebook. In one of the sessions, the participant
installing MongoDB experienced a waiting period of more than 10 minutes before
receiving any feedback on the progress of the installation. This strategy was
applied to the challenge Lack of Installation Progress Feedback.

Random Actions were taken during the installation process of TensorFlow and
MongoDB. When installing TensorFlow, the participant decided to click on
random keys on the keyboard. Similarly to the previous strategy, this choice
was motivated by the absence of feedback regarding the installation progress.
This strategy was applied to the challenges: Lack of Installation Progress
Feedback and Unclear Instructions.

Ignoring was employed in eight sessions when installing MongoDB and Rust. In
one of the sessions, the participant was following the installation
instructions from the official Spring Boot website. The official website was
perceived as less user-friendly, making the installation instructions somewhat
difficult to follow. Consequently, this participant opted to ignore certain
installation steps described on the website. This strategy was applied to the
challenges: Unclear Instructions, Poorly-structured Information, Account
Required, and Error Message.

####  4.2.4. Tool Characteristics

Versioning is the only strategy of the Tool Characteristics category. It was
applied in two sessions when participants were installing TensorFlow and
PyTorch. In both cases, the participants were facing issues installing the
tools and were trying to change the Python version to see if the issue could
be solved. This strategy was successful when the two participants were
installing TensorFlow, but it was not successful with the participant
installing Python. This strategy was applied to the challenges: Error Message
and Version Incompatibility. This strategy was selected by 48 respondents in
our validation survey.

Regarding the ”Other(s)” option in our validation survey, we only had one
single addition for the strategies - ”call my excellent [sic] friend for
help”.

###  4.3. Sources (RQ3)

Previously, we explained each challenge and strategy by providing examples
extracted from the think-aloud sessions in situations where these challenges
and strategies were perceived and applied. In this subsection, we present the
sources that were consulted when applying each strategy and the tools in which
those sources were applied during the 24 think-aloud sessions. Moreover, we
present a detailed analysis of the relationship between challenges and
strategies, introducing the type of sources (see Table 6) consulted by the
participants when applying the strategies. For each challenge (see Table 7),
we document the specific strategies applied, the sources consulted during that
period, and the tools for which those sources were accessed. As in the
previous section, the challenges are displayed in four categories.

Table 6. Sources type we used for the detailed analysis.

Source Symbol |  | Source Name | Occ. |  | Source Symbol |  | Source Name | Occ. |
---|---|---|---|---|---|---|---|---|---
|  | Community Knowledge | 18 |  |  |  | Sign Up Page | 2 |
|  | How-To Tutorial | 20 |  |  |  | Google Results | 3 |
|  | Video Tutorials | 7 |  |  |  | Download Page | 6 |
|  | Installation Guide | 55 |  |  |  | Terminal | 18 |
|  | Getting Started | 38 |  |  |  | README | 1 |
|  | Dashboard | 6 |  |  |  | No Source | 14 |

We mapped and categorised the sources consulted during the think-aloud
sessions. To facilitate the categorisation process, we partially adopted a
terminology consistent with previous research (Aghajani et al., 2020).
Regarding the sources that did not align with any pre-existing categories
established in the prior work, we formulated new categories. In Table 6, each
source category is represented by a specific symbol. The Community Knowledge
source encompasses online community forums (e.g., Stack Overflow), How-
To/Tutorial refers to general guides, typically in the form of blogs. Video
tutorials comprise instructional videos illustrating tool installation
procedures. Installation Guide denotes comprehensive, step-by-step documents
or web pages providing installation instructions, while Getting Started refers
to materials offering an introductory overview of the tool and its
installation process. The Dashboard category refers to the homepage of the
tool’s official website, while Sign Up Page means the registration page on the
official tool website. The category Google Results  represents findings from a
Google search, and Download Page designates a webpage facilitating the
installation process (e.g., a .exe file). Information originating from the
terminal prompt is categorised as Terminal . README represents documentation
provided in README files, and No Source is used when no external source was
consulted.

Table 7. Detailed analysis of challenges and strategies.

Challenges — Strategies — Sources — Tools
---
Information Quality/Value
Unclear Instructions | Sources | Dysfunctional Instructions | Sources
Searching for Different Documentation |  ${}_{\textit{M}}$ ${}_{\textit{F}}$ ${}_{\textit{F, N}}$ ${}_{\textit{N, S}}$ ${}_{\textit{S}}$ | Searching for Different Documentation |  ${}_{\textit{M}}$ ${}_{\textit{S}}$ ${}_{\textit{J, N}}$ ${}_{\textit{S, N}}$
Reading Help or Feedback in Terminal |  ${}_{\textit{M, F}}$ | Reading Help or Feedback in Terminal |  ${}_{\textit{J}}$ ${}_{\textit{F}}$
Reading Instructions Carefully |  ${}_{\textit{M}}$ ${}_{\textit{N}}$ ${}_{\textit{J, S}}$ | Reading Instructions Carefully |  ${}_{\textit{M}}$ ${}_{\textit{F, J, S}}$ ${}_{\textit{F, S}}$
Focusing on One Source |  ${}_{\textit{J}}$ ${}_{\textit{S}}$ ${}_{\textit{S}}$ | Focusing on One Source |  ${}_{\textit{M}}$ ${}_{\textit{S}}$ ${}_{\textit{S}}$
Reformulating the Query |  ${}_{\textit{T}}$ ${}_{\textit{S}}$ | Reformulating the Query |  ${}_{\textit{S}}$ ${}_{\textit{M}}$
Web Searching |  ${}_{\textit{M}}$ ${}_{\textit{M, N}}$ | Web Searching |  ${}_{\textit{T}}$ ${}_{\textit{M}}$
Random Actions |  ${}_{\textit{M}}$ ${}_{\textit{S}}$ | Giving Up |  ${}_{\textit{S}}$
Reinstalling |  ${}_{\textit{T, M}}$ ${}_{\textit{S}}$ | Ignoring |  ${}_{\textit{J}}$
Ignoring |  ${}_{\textit{S}}$ ${}_{\textit{S}}$ ${}_{\textit{N}}$ |  |
Information Seeking
Under-specified Query | Sources | Poorly-structured Information | Sources
Searching for Different Documentation |  ${}_{\textit{P}}$ | Reading Instructions Carefully |  ${}_{\textit{J}}$ ${}_{\textit{N}}$
Focusing on One Source |  ${}_{\textit{N}}$ | Focusing on One Source |  ${}_{\textit{J, M}}$ ${}_{\textit{M}}$ ${}_{\textit{S}}$
Reformulating the Query |  |  ${}_{\textit{P}}$ ${}_{\textit{M, P}}$ ${}_{\textit{T, P}}$ ${}_{\textit{M}}$ ${}_{\textit{J}}$
---
Reformulating the Query |  ${}_{\textit{M}}$
Web Searching |  ${}_{\textit{P}}$ | Random Actions |  ${}_{\textit{S}}$
|  | Ignoring |  ${}_{\textit{S}}$
Information Overload | Sources | Error Message | Sources
Reading Instructions Carefully |  ${}_{\textit{T}}$ | Searching for Different Documentation |  ${}_{\textit{P}}$ ${}_{\textit{J}}$ ${}_{\textit{S}}$
Reading Help or Feedback in Terminal |  ${}_{\textit{T}}$ | Reading Help or Feedback in Terminal |  ${}_{\textit{T}}$
Web Searching |  ${}_{\textit{T}}$ | Reading Instructions Carefully |  ${}_{\textit{S, J}}$
Ignoring |  ${}_{\textit{N, M}}$ | Web Searching |  ${}_{\textit{P, N, T}}$
|  | Focusing on One Source |  ${}_{\textit{P}}$ ${}_{\textit{M}}$
|  | Reformulating the Query |  ${}_{\textit{M}}$
|  | Giving Up |  ${}_{\textit{P}}$
|  | Versioning |  ${}_{\textit{P}}$
|  | Reinstalling |  ${}_{\textit{S}}$
|  | Ignoring |  ${}_{\textit{J}}$ ${}_{\textit{M}}$
Installation Process
Complicated Installation Process | Sources | Lack of Installation Progress Feedback | Sources
Searching for Different Documentation |  ${}_{\textit{T, F, S}}$ ${}_{\textit{P}}$ ${}_{\textit{T, P, N}}$ |  |
|  ${}_{\textit{P, N, S}}$ ${}_{\textit{N}}$ ${}_{\textit{S}}$ ${}_{\textit{M}}$ | Searching for Different Documentation |  ${}_{\textit{F}}$ ${}_{\textit{M}}$
Reading Instructions Carefully |  ${}_{\textit{F, S, M}}$ ${}_{\textit{F, S}}$ ${}_{\textit{M}}$ ${}_{\textit{F}}$ ${}_{\textit{M}}$ | Reading Instructions Carefully |  ${}_{\textit{M}}$
Focusing on One Source |  ${}_{\textit{T}}$ ${}_{\textit{T, M}}$ ${}_{\textit{T, M}}$ ${}_{\textit{M}}$ ${}_{\textit{S}}$ | Focusing on One Source |  ${}_{\textit{M}}$ ${}_{\textit{J}}$
Reformulating the Query |  ${}_{\textit{F}}$ ${}_{\textit{P, J}}$ ${}_{\textit{M}}$ | Web Searching |  ${}_{\textit{M}}$ ${}_{\textit{P}}$
Web Searching |  ${}_{\textit{F}}$ ${}_{\textit{P}}$ ${}_{\textit{S}}$ | Resetting the Environment |  ${}_{\textit{M}}$
Giving Up |  ${}_{\textit{F}}$ | Random Actions |  ${}_{\textit{M}}$ ${}_{\textit{N}}$
Ignoring |  ${}_{\textit{S, F}}$ | Waiting |  ${}_{\textit{M, J}}$ ${}_{\textit{N}}$
|  | Reinstalling |  ${}_{\textit{M}}$
|  | Ignoring |  ${}_{\textit{M}}$
Tool Characteristics
Account Required | Sources | Version Incompatibility | Sources
Searching for Different Documentation |  ${}_{\textit{M}}$ | Searching for Different Documentation |  ${}_{\textit{M}}$ ${}_{\textit{P}}$
Focusing on One Source |  ${}_{\textit{M}}$ | Reading Instructions Carefully |  ${}_{\textit{T}}$
|  | Giving Up |  ${}_{\textit{M}}$
|  | Reinstalling |  ${}_{\textit{T}}$
|  | Versioning |  ${}_{\textit{T, P}}$
|  | Ignoring |  ${}_{\textit{S}}$
T = TensorFlow — F = Flutter — M = MongoDB — P = PyTorch — J = Jupyter
Notebook — N = Node.js — S = Spring Boot

Community Knowledge emerged as a frequently consulted source, particularly for
the challenges Unclear Instructions,Dysfunctional Instructions, Error Message,
Complicated Installation Process, and Version Incomatibility. This source was
mainly consulted when employing the strategies Searching for Different
Documentation and Web Searching. The Community Knowledge source was applied
when installing TensorFlow, MongoDB and Flutter. These findings show that
online forums play an important role in supporting newcomers to overcome
challenges during the software development tool installation process.

How-To/Tutorial sources were used in six challenges, mainly addressing the
challenges of Unclear Instructions, Under-specified Query, and Poorly-
structured Information. Similar to the Community Knowledge source, these were
accessed primarily while implementing the strategies of Searching for
Different Documentation, Web Searching, and Focusing on One Source. This
source was frequently accessed when installing PyTorch, MongoDB, and
TensorFlow.

Video Tutorials were consulted for five challenges, including Dysfunctional
Instructions, Error Message, and Lack of Installation Progress Feedback. This
source was used in conjunction with the strategies Searching for Different
Documentation, Web Searching, and Focusing on One Source, particularly during
the installation process of MongoDB, Flutter, and Spring Boot.

Installation Guide proved to be the most frequently used source, being
employed for eight challenges in total, but mostly for Unclear Instructions,
Dysfunctional Instructions, Under-specified Query, Poorly-structured
Information, Error Message, Complicated Installation Process, and Version
Incompatibility. Participants consulted this source when applying six
strategies, for the most part, Searching for Different Documentation, Reading
Instructions Carefully and Web Searching. This demonstrates the utility of
step-by-step installation guides for newcomers who navigate the tool
installation process. The Installation Guide source was mostly consulted when
installing PyTorch, TensorFlow, Flutter, Node.js, and MongoDB.

Getting Started was applied to eight challenges, including Under-specified
Query and Version Incompatibility. It was associated with a total of seven
strategies, with Searching for Different Documentation, Focusing on One
Source, and Reading Instructions Carefully being the most common. This source
was accessed during the installation process of Flutter, Node.js, MongoDB,
PyTorch, TensorFlow, and Jupyter Notebook.

Dashboard was comparatively less utilised, featuring only three challenges,
specifically Unclear Instructions, Complicated Installation Process, and
Account Required. This source was accessed primarily during the installation
of MongoDB and Spring Boot and was correlated with strategies such as
Searching for Different Documentation and Focusing on One Source.

Subsequently, there are three sources with fewer occurrences. Sign Up Page
source, for instance, was applied in Under-specified Query and Complicated
Installation Process, with the strategy Reading Instructions Carefully being
employed. This source was consulted during the MongoDB installation. Google
Results was accessed for Under-specified Query and Information Overload
challenges, with strategies such as Focusing on One Source, Reformulating the
Query, and Web Searching. These instances were associated with the
installations of Node.js, Jupyter Notebook, and TensorFlow. Download Page
source was consulted in three challenges, Unclear Instructions, Error Message,
Complicated Installation Process, and was accessed primarily during the
strategy  Searching for Different Documentation during the installation
process of Node.js.

Terminal source was used in six challenges, mainly addressing issues such as
Unclear Instructions, Dysfunctional Instructions, and Lack of Installation
Progress Feedback. At the same time, this source was consulted in connection
with five strategies, with a focus on Reading Help or Feedback in Terminal and
Reading Instructions Carefully. These strategies were predominantly applied
during the installation processes of Flutter, Node.js, MongoDB, PyTorch, and
TensorFlow, indicating that the participants found valuable information in the
terminal output.

README source made a single appearance in the challenge Unclear Instructions,
primarily employed in the Reading Instructions Carefully strategy, and was
consulted with the installation process of MongoDB.

No Source Despite the helpful role of sources in supporting participants to
overcome challenges, some participants chose not to use external sources for
specific challenges and strategies. This was particularly perceived when
dealing with the Lack of Installation Progress Feedback challenge. Regarding
strategies, this approach of not consulting sources was observed across six
strategies, including Reformulating the Query, Giving Up, Resetting the
Environment, Random Actions, Ignoring, and Waiting. These strategies were
predominantly employed during the installation processes of Flutter, MongoDB,
and Jupyter Notebook.

##  5\. Discussion

Looking at the challenges (see Table 7), it is noticeable that some of the
challenges had more strategies applied than others. For instance, the
challenges Lack of Installation Progress Feedback, Unclear Instructions and
Dysfunctional Instructions were the challenges with the highest number of
strategies applied. Regarding the strategies, we notice that the strategy
Searching for Different Documentation stands out, being applied in nine out of
the eleven challenges. Implying that, at some point, all participants opted to
consult a non-official source in order to find installation instructions,
suggesting that official sources are often unreliable. In this section, we
outline practical recommendations for tool vendors, tool users, and
researchers derived from our findings. Additionally, we address potential
threats to the validity of our study.

###  5.1. Tool vendors

Progress Feedback: Participants in our study reported difficulties due to
insufficient progress feedback. These concerns included not knowing the number
of completed and remaining steps, as well as uncertainty about the success of
the installation after following all available instructions. The latter issue
led to confusion among several participants. Consequently, we advise tool
vendors to incorporate progress feedback throughout the installation process,
with particular emphasis on confirming the success of the installation at its
conclusion. This can be achieved, for example, by providing a test case for
users to execute.

Alternative Sources: Participants primarily sought ”official” documentation,
but often resorted to consulting sources, such as crowd-sourced platforms like
Stack Overflow. We recommend that tool vendors monitor the information on
these platforms to the best of their ability. If incorrect information is
present, many users may be misled. In some instances, it may be feasible to
rectify the issue at its source, for example, by submitting a new response to
a Stack Overflow question.

Download Link: Interestingly, participants experienced difficulties locating
download links. Although tool vendors may have intentionally designed it this
way to encourage potential users to review the documentation before
downloading tools, we observed that participants tended to be impatient and
anticipated finding download links in more prominent locations.

Dependency Awareness: Participants expressed frustration when they perceived
that the installation of one tool led to the installation of multiple tools.
Although prerequisites and dependencies are often unavoidable, we recommend
that tool vendors clearly communicate to potential users from the outset of
the installation process that multiple tools will be involved. Additionally,
vendors should consider bundling dependencies whenever possible to streamline
the installation experience.

Video Tutorials: Accurately documenting a process that requires frequent
transitions between tools and environments, such as multiple command-line
environments, desktop applications, and websites, can be challenging in a
textual format. Often, a video walkthrough can effectively demonstrate
concepts that would require extensive explanation in text. We recommend that
tool vendors create video tutorials for at least the most common scenarios,
such as popular operating systems.

Prerequisites: To prevent situations where users only discover partway through
the installation that their setup does not meet the requirements for a tool,
such as due to hardware constraints, prerequisites should be clearly
documented. To help users determine whether their setup meets the
requirements, tool vendors could provide automated checks as the initial step
in the installation process.

###  5.2. Tool users

Rephrasing Queries: In the majority of sessions where the participants used
rephrasing, this strategy proved successful. As a result, we advise users
experiencing challenges during tool installation not to abandon their efforts
if their initial query fails to yield desired outcomes. In many cases, adding
more terms about their specific scenario or context led to success.

Alternative Sources: Official documentation frequently lacks guidance on
recovering from error scenarios. However, other sources like Stack Overflow
are well suited for providing this type of information. Our study revealed
that most of the participants who relied exclusively on official sources were
unsuccessful. Therefore, we recommend that potential tool users explore
additional sources while remaining aware that they may be less authoritative.

###  5.3. Researchers

Mapping Installation Processes: While this study identified challenges and
strategies associated with the installation of software development tools,
along with the connections between them, future research should focus on
obtaining a more canonical understanding of the steps required to install
tools, e.g., through additional empirical studies. A higher-level
understanding, such as one codified in an overarching process, could
facilitate more systematic research and tool building aimed at addressing the
inherent challenges in this field, as well as help organise and consolidate
knowledge within the domain.

Automated Techniques for Tool Vendors: Our research presents numerous
opportunities for future work on the development of automated methods to
address the identified challenges. From the tool vendors’ perspective, these
automated techniques could focus on tasks such as eliminating duplicated
instructions, automatically providing progress feedback, automatically testing
instructions, augmenting official documentation with relevant information from
other sources when necessary, and reducing information overload, among other
examples.

Automated Techniques for Tool Users: Likewise, there are numerous
opportunities to create automated techniques to assist potential tool users,
such as query reformulation methods tailored to the context of tool
installation, and tools that evaluate a user’s environment to determine
whether it meets the requirements of a particular tool and its dependencies.

###  5.4. Threats to Validity

Our findings are based on data collected from 18 participants, which we
believe to be an adequate sample size. This is because during the think-aloud
sessions, we were able to gather more than 500 minutes of video recordings and
187 pages of transcripts. Although we consider this sample size sufficient,
involving a larger number of participants could present opportunities to
identify new challenges and strategies, as well as validate those we have
already discovered. Considering that we distributed the validation survey
before conducting the additional think-aloud sessions, new challenges and
strategies emerged. Furthermore, we took the opportunity to refine and clarify
the names and descriptions of both the challenges and the strategies.

The majority of our study’s participants were Master’s students who, as
mentioned in Subsection 3.1, have been demonstrated to be an adequate proxy
for professionals, such as junior developers, in empirical studies.
Additionally, our findings are constrained to early-career stage software
developers, who may not be familiar with many programming languages,
frameworks, and tools. This study could also be replicated with different
populations (e.g., experienced developers) to potentially provide further
empirical evidence and compare the challenges of installing software
development tools from the perspective of senior developers. In our study, we
chose to use seven software development tools that cover a wide range of
functionalities and purposes, including data management tools (e.g., MongoDB)
and machine learning tools (e.g., TensorFlow). Although we believe seven tools
to be a sufficient number, we acknowledge that we cannot generalise the
results to all software development tools in the market. Increasing the number
of tools could potentially help us identify new challenges and strategies.

##  6\. Conclusion

This research aimed to explore the challenges novice developers face during
the software development tool installation process and to discuss the
strategies they employ to overcome these obstacles, along with the type of
sources they consult when applying these strategies. By analysing data from 18
participants and 24 think-aloud sessions, we identified that novice developers
often encounter unclear documentation (e.g., installation instructions) and
insufficient installation progress feedback. Our study also revealed that
rephrasing search queries and consulting non-official documentation were
strategies employed to address some of the challenges (e.g., unclear
documentation).

Future research can expand upon our findings by identifying additional
challenges and understanding the various strategies software developers
implement when installing software development tools in specific domains or
industries. Further research could also validate our findings using a larger
sample size and examine the characteristics of the tools, connecting them to
the challenges and strategies observed.

##  7\. Data Availability

The transcripts from the sessions are available at
https://github.com/larisalerno/TOSEMChallengesandStrategies.

## References

  * (1)
  * Aghajani et al. (2020) Emad Aghajani, Csaba Nagy, Mario Linares-Vásquez, Laura Moreno, Gabriele Bavota, Michele Lanza, and David C Shepherd. 2020.  Software documentation: the practitioners’ perspective. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_ (Online) _(ICSE ’20)_. ACM, New York, NY, USA, 590–601.  https://doi.org/10.1145/3377811.3380405
  * Aghajani et al. (2019) Emad Aghajani, Csaba Nagy, Olga Lucero Vega-Márquez, Mario Linares-Vásquez, Laura Moreno, Gabriele Bavota, and Michele Lanza. 2019.  Software documentation issues unveiled. In _2019 IEEE/ACM 41st International Conference on Software Engineering_ (Montreal, CA) _(ICSE ’19)_. IEEE, Piscataway, New Jersey, 1199–1210.  https://doi.org/10.1109/ICSE.2019.00122
  * Chen and Huang (2009) Jie-Cherng Chen and Sun-Jen Huang. 2009.  An empirical analysis of the impact of software development problem factors on software maintainability.  _Journal of Systems and Software_ 82, 6 (2009), 981–992.  https://doi.org/10.1016/j.jss.2008.12.036
  * Falessi et al. (2018) Davide Falessi, Natalia Juristo, Claes Wohlin, Burak Turhan, Jürgen Münch, Andreas Jedlitschka, and Markku Oivo. 2018.  Empirical software engineering experts on the use of students and professionals in experiments.  _Empirical Software Engineering_ 23 (Feb 2018), 452–489.  https://doi.org/10.1007/s10664-017-9523-3
  * Gao et al. (2023) Haoyu Gao, Christoph Treude, and Mansooreh Zahedi. 2023.  Evaluating Transfer Learning for Simplifying GitHub READMEs. In _Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (, San Francisco, CA, USA,) _(ESEC/FSE 2023)_. Association for Computing Machinery, New York, NY, USA, 1548–1560.  https://doi.org/10.1145/3611643.3616291
  * Guo (2013) Philip J. Guo. 2013.  Online python tutor: embeddable web-based program visualization for cs education. In _Proceeding of the 44th ACM technical symposium on Computer science education_ (Denver, CO, USA). ACM, New York, NY, USA, 579–584.  https://doi.org/10.1145/2445196.2445368
  * Hassan and Wang (2017) Foyzul Hassan and Xiaoyin Wang. 2017.  Mining readme files to support automatic building of java projects in software repositories. In _IEEE/ACM 39th International Conference on Software Engineering Companion_ (Buenos Aires, ARG) _(ICSE ’17)_. IEEE, Piscataway, New Jersey, 277–279.  https://doi.org/10.1109/ICSE-C.2017.114
  * Helminen et al. (2013) Juha Helminen, Petri Ihantola, and Ville Karavirta. 2013.  Recording and analyzing in-browser programming sessions. In _Proceedings of the 13th Koli Calling International Conference on Computing Education Research_ (Koli, FI). ACM, New York, NY, USA, 13–22.  https://doi.org/10.1145/2526968.2526970
  * Jenkins et al. (2012) Jam Jenkins, Evelyn Brannock, Thomas Cooper, Sonal Dekhane, Mark Hall, and Michael Nguyen. 2012.  Perspectives on active learning and collaboration: JavaWIDE in the classroom. In _Proceedings of the 43rd ACM Technical Symposium on Computer Science Education_ (Raleigh, NC, USA). ACM, New York, NY, USA, 185–190.  https://doi.org/10.1145/2157136.2157194
  * Kal (2006) Toth Kal. 2006.  Experiences with open source software engineering tools.  _IEEE Software_ 23 (Dec. 2006), 44–52.  https://doi.org/10.1109/MS.2006.158
  * Krosnick (2015) R.P. Krosnick. 2015.  _Videodoc: Combining videos and lecture notes for a better learning experience_.  Ph. D. Dissertation. Massachusetts Institute of Technology.
  * Lafreniere et al. (2013) Benjamin Lafreniere, Tovi Grossman, and George Fitzmaurice. 2013.  Community enhanced tutorials: improving tutorials with multiple demonstrations. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ (Paris, FR). ACM, New York, NY, USA, 1779–1788.  https://doi.org/10.1145/2470654.2466235
  * Liu et al. (2022) Yuyang Liu, Ehsan Noei, and Kelly Lyons. 2022.  How ReadMe files are structured in open source Java projects.  _Information and Software Technology_ 148 (2022), 106924.  https://doi.org/10.1016/j.infsof.2022.106924
  * Mirhosseini and Parnin (2020) Samim Mirhosseini and Chris Parnin. 2020.  Docable: Evaluating the executability of software tutorials. In _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Virtual Event). ACM, New York, NY, USA, 375–385.  https://doi.org/10.5281/zenodo.3903727
  * Mysore and Guo (2017) Alok Mysore and Philip J Guo. 2017.  Torta: Generating mixed-media gui and command-line app tutorials using operating-system-wide activity tracing. In _Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology_ (Québec, CA). ACM, New York, NY, USA, 703–714.  https://doi.org/10.1145/3126594.3126628
  * Parnin and Treude (2011) Chris Parnin and Christoph Treude. 2011.  Measuring API documentation on the web. In _Proceedings of the 2nd international workshop on Web 2.0 for software engineering_ (Honolulu, HI, USA). ACM, New York, NY, USA, 25–30.  https://doi.org/10.1145/1984701.1984706
  * Pavel et al. (2014) Amy Pavel, Colorado Reed, Björn Hartmann, and Maneesh Agrawala. 2014.  Video digests: a browsable, skimmable format for informational lecture videos.. In _UIST_ (Honolulu, HI, USA). ACM, New York, NY, USA, 2642918–2647400.  http://dx.doi.org/10.1145/2642918.2647400
  * Pinto et al. (2019) Gustavo Pinto, Clarice Ferreira, Cleice Souza, Igor Steinmacher, and Paulo Meirelles. 2019.  Training software engineers using open-source software: the students’ perspective. In _2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET)_ (Montreal, CA). IEEE, Piscataway, New Jersey, 147–157.  https://doi.org/10.1109/ICSE-SEET.2019.00024
  * Ramírez-Mora et al. (2020) Sandra L Ramírez-Mora, Hanna Oktaba, and Helena Gómez-Adorno. 2020.  Descriptions of issues and comments for predicting issue success in software projects.  _Journal of Systems and Software_ 168 (Oct. 2020), 65.  https://doi.org/10.1016/j.jss.2020.110663
  * Robillard et al. (2017) Martin P Robillard, Andrian Marcus, Christoph Treude, Gabriele Bavota, Oscar Chaparro, Neil Ernst, Marco Aurélio Gerosa, Michael Godfrey, Michele Lanza, Mario Linares-Vásquez, et al. 2017\.  On-demand developer documentation. In _2017 IEEE International conference on software maintenance and evolution (ICSME)_ (Shangai, CN). IEEE, Piscataway, New Jersey, 479–483.  https://doi.org/10.1109/ICSME.2017.17
  * Robillard and Treude (2020) Martin P Robillard and Christoph Treude. 2020.  Understanding wikipedia as a resource for opportunistic learning of computing concepts. In _Proceedings of the 51st ACM Technical Symposium on Computer Science Education_ (Portland, USA). ACM, New York, NY, USA, 72–78.  https://doi.org/10.1145/3328778.3366832
  * Salman et al. (2015) Iflaah Salman, Ayse Tosun Misirli, and Natalia Juristo. 2015.  Are students representatives of professionals in software engineering experiments?. In _IEEE/ACM 37th IEEE International Conference on Software Engineering_ (Florence, IT). IEEE, Piscataway, New Jersey, 666–676.  https://doi.org/10.1109/ICSE.2015.82
  * Sjoberg et al. (2002) Dag IK Sjoberg, Bente Anda, Erik Arisholm, Tore Dyba, Magne Jorgensen, Amela Karahasanovic, Espen Frimann Koren, and Marek Vokác. 2002.  Conducting realistic experiments in software engineering. In _Proceedings international symposium on empirical software engineering_ (Nara, JP). IEEE, Piscataway, New Jersey, 17–26.  https://doi.org/10.1109/ISESE.2002.1166921
  * Steinmacher et al. (2015a) Igor Steinmacher, Tayana Conte, Marco Aurélio Gerosa, and David Redmiles. 2015a.  Social barriers faced by newcomers placing their first contribution in open source software projects. In _Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing_ (Vancouver, CA). ACM, New York, NY, USA, 1379–1392.  https://doi.org/10.1145/2675133.2675215
  * Steinmacher et al. (2016) Igor Steinmacher, Tayana Uchoa Conte, Christoph Treude, and Marco Aurélio Gerosa. 2016.  Overcoming open source project entry barriers with a portal for newcomers. In _Proceedings of the 38th International Conference on Software Engineering_ (Austin, Texas). ACM, New York, NY, USA, 273–284.  https://doi.org/10.1145/2884781.2884806
  * Steinmacher et al. (2015b) Igor Steinmacher, Marco Aurelio Graciotto Silva, Marco Aurelio Gerosa, and David F Redmiles. 2015b.  A systematic literature review on the barriers faced by newcomers to open source software projects.  _Information and Software Technology_ 59 (Mar. 2015), 67–85.  https://doi.org/10.1016/j.infsof.2014.11.001
  * Steinmacher et al. (2018) Igor Steinmacher, Christoph Treude, and Marco Aurelio Gerosa. 2018.  Let me in: Guidelines for the successful onboarding of newcomers to open source projects.  _IEEE Software_ 36, 4 (2018), 41–49.  https://doi.org/10.1109/MS.2018.110162131
  * Stol et al. (2010) Klaas-Jan Stol, Paris Avgeriou, and Muhammad Ali Babar. 2010.  Identifying architectural patterns used in open source software: approaches and challenges. In _14th International Conference on Evaluation and Assessment in Software Engineering (EASE)_ (Swindon, UK). ACM, New York, NY, USA, 1–10.
  * Teague and Lister (2014) Donna Teague and Raymond Lister. 2014.  Longitudinal think aloud study of a novice programmer. In _Conference in Research and Practice in Information Technology Series_ (Auckland, New Zealand). ACM, New York, NY, USA, 41–50.  https://opus.lib.uts.edu.au/handle/10453/33409
  * Thiselton and Treude (2019) Emillie Thiselton and Christoph Treude. 2019.  Enhancing python compiler error messages via stack. In _ACM/IEEE International Symposium on Empirical Software Engineering and Measurement_ (Porto de Galinhas, BR). IEEE, Piscataway, New Jersey, 1–12.  https://doi.org/10.1109/ESEM.2019.8870155
  * Treude and Aniche (2018) Christoph Treude and Maurício Aniche. 2018.  Where does google find api documentation?. In _Proceedings of the 2nd International Workshop on API Usage and Evolution_ (Gothenburg, SE). ACM, New York, NY, USA, 19–22.  https://doi.org/10.1145/3194793.3194796
  * Uddin and Robillard (2015) Gias Uddin and Martin P Robillard. 2015.  How API documentation fails.  _IEEE software_ 32, 4 (June 2015), 68–75.  https://doi.org/10.1109/MS.2014.80
  * Valez et al. (2020) Martin Valez, Michael Yen, Mathew Le, Zhendong Su, and Mohammad Amin Alipour. 2020.  Student Adoption and Perceptions of a Web Integrated Development Environment: An Experience Report. In _Proc. 51st ACM Technical Symposium on Computer Science Education_ (Portland, OR, USA). ACM, New York, NY, USA, 1172–1178.  https://doi.org/10.1145/3328778.3366949
  * Walker (2022) Ry Walker. 2022.  OSSRank.  https://ossrank.com/