11institutetext:  1 11email<EMAIL_ADDRESS>11email:
<EMAIL_ADDRESS>11email<EMAIL_ADDRESS>
Department of Software Engineering
Blekinge Institute of Technology, Sweden 211email<EMAIL_ADDRESS>
fortiss, Germany

# No Free Lunch: Research Software Testing in Teaching

Michael Dorner1  Andreas Bauer1  and Florian Angermeir1,2

###### Abstract

Software is at the core of most scientific discoveries today. Therefore, the
quality of research results highly depends on the quality of the research
software. Rigorous testing, as we know it from software engineering in the
industry, could ensure the quality of the research software but it also
requires a substantial effort that is often not rewarded in academia.
Therefore, this research explores the effects of research software testing
integrated into teaching on research software. In an _in-vivo_ experiment, we
integrated the engineering of a test suite for a large-scale network
simulation as group projects into a course on software testing at the Blekinge
Institute of Technology, Sweden, and qualitatively measured the effects of
this integration on the research software. We found that the research software
benefited from the integration through substantially improved documentation
and fewer hardware and software dependencies. However, this integration was
effortful and although the student teams developed elegant and thoughtful test
suites, no code by students went directly into the research software since we
were not able to make the integration back into the research software
obligatory or even remunerative. Although we strongly believe that integrating
research software engineering such as testing into teaching is not only
valuable for the research software itself but also for students, the research
of the next generation, as they get in touch with research software
engineering and bleeding-edge research in their field as part of their
education, the uncertainty about the intellectual properties of students’ code
substantially limits the potential of integrating research software testing
into teaching.

###### keywords:

research software engineering, testing, teaching, education, open source,
license, intellectual property, experiment

##  1 Introduction

Software underpins almost all research and is, in one way or another, at the
core of most scientific discoveries today. Researchers from different
disciplines develop new or customize existing software to collect, analyze,
and visualize data to push the limits of the unknown. Thus, the quality of
research results highly depends on the quality of the research software.

Although the significance of software testing for quality assurance is widely
acknowledged [GM16, Gar+17, SC18], efforts to ensure the quality of research
software are often not rewarded in academia [EC22] and are rarely included in
researchers’ training [NK17].

To address this major shortcoming of research software engineering, we
integrated research software testing into teaching aiming for synergy effects
between teaching and research software testing for improving the quality and
trustworthiness of real-world research software. In this article, we report
the effects of such an integration on research software. In detail, we set out
to answer the following research question:

RQ: What are the effects of integrating research software testing into
teaching on research software?

To answer our research question, we conducted an _in-vivo_ experiment on such
an integration. In the context of a course on software testing at Blekinge
Institute of Technology, Sweden, students developed a test suite for real-
world research software. We gather and analyze qualitative data on the effects
of the integration on the research software.

The remainder of this article is structured as follows: In Section 2, we
briefly discuss the background of our study. We then describe our experimental
design with its core components in Section 3 in detail. In Section 4, we
report the resulting effects of our experiment on the research software
(Section 4). After discussing the limitations of our study (Section 5), we
conclude our paper by raising a fundamental question in the context of
integrating research software engineering into teaching: How to handle the
intellectual property of students properly?

##  2 Background

Research on software-testing education has gained more attention over the last
years [Gar+20]. In light of our experiment on integrating research software
testing into teaching, we would like to highlight the current approaches and
initiatives to foster research software engineering in teaching (Section 2.1)
and using open-source software in software-testing education (Section 2.2).

###  2.1 Research Software Engineering in Teaching

As the research software engineering publication monitor shows, research
software engineering has gained more attention over the recent years [FS24]. A
proper education has become one of the key concerns in research software
engineering [Coh+21]. This need for education driven by different
organizations such as _HIFIS_
111https://hifis.net/services/software/training.html, _INTERSECT_
222https://intersect-training.org/, _CodeRefinery_
333https://coderefinery.org/, or _BSSw_ 444https://bssw.io/ address this
concern by providing learning materials, offering workshops or teaching, and
maintaining communities to share research software engineering teaching
experiences.

Despite its importance for modern science, surprisingly, research software
engineering is rarely a part of curricula. General courses on computer science
education (e.g., _The Missing Semester of Your CS Education_
555https://missing.csail.mit.edu/ at MIT or the _The Missing CS Class_ at
University of California, Davis, [Gil+22]) or courses specific to research
software engineering (e.g., University of Potsdam [BDL24], Technical
University of Dresden [Juc24] are notable and well-received exceptions.

Even though there is progress towards addressing education, we agree with
[Got+24] that a systemic institutionalization of research software engineering
education is needed [Got+24].

###  2.2 Open-Source Software for Teaching Software Testing

Providing authentic, real-world software quality assurance and testing
experiences within the context of a Computer Science or Software Engineering
curriculum is challenging [CML14, DDC20]. Different studies used open-source
projects for this reason [CML14, KMR14, DDC20, VA24]. Open-source software is
software that is available under a license that grants the right to use,
modify, and distribute the software—modified or not—to everyone free of charge
(adapted from [Rie23]).

None of the studies has addressed the ethical issue of the enforced use rights
to students’ intellectual property, as we will discover as part of our
experiment.

[VA24] proposed and discussed their teaching approach for a software testing
course that integrates theory and practical experience through the utilization
of both team-based learning and active contributions to open-source software
projects [VA24]. Although not in the context of research software testing, we
can partially confirm their reflections and recommendations in our experiment:
Almost all students had no experience with contributing to open-source
software. Therefore, the quality of documentation plays a pivotal role in the
integration of open-source software into testing courses.

In contrast to our experiment, [VA24] [VA24] awarded students for code
contributions via pull requests with extra points. Pull requests are a
convenient way for software developers to collaborate on open-source projects
by proposing changes [Inc24]. However, they observed that “this incentive
prompted students to submit pull requests lacking genuine contributions, which
inundated the [open-source software] community with superfluous requests”. As
a result, they discontinued offering extra points for pull requests. In our
work, we did not incentivize contributions and could not observe any code
contribution in the form of a pull request.

##  3 Experimental Design

In this section, we describe the design of our _in-vivo_ experiment that aims
to answer our research questions on the effects of integrating research
software engineering into teaching.

An _in-vivo_ experiment (or field experiment) is an experiment conducted in a
natural setting with a high degree of realism. The researcher manipulates some
properties or variables in the research setting to observe an effect of some
kind. The realistic research setting exists independent of the researcher,
which distinguishes it from the contrived research setting in a laboratory
experiment [SF18].

In our experiment, the natural setting of our integration is a course on
software testing, which is part of the regular curriculum for the Software
Engineering and Artificial Intelligence study programs at Blekinge Institute
of Technology, Sweden. This course comprises a project work in which the
students develop a test suite for software. Instead of a classical software
project or an open-source component, as it is commonly used in teaching
[CML14, KMR14, DDC20, VA24], we used a research software for our experiment
and observed the effects of this integration on this research software. In the
context of an experiment, we consider our research software as a type of real-
world example and not as a toy project or educational demonstrator that may
have limitations in terms of generalizability to real-world issues [Woh+12].
Thus, we see research software as a special type of real-world open-source
software. Figure 1 provides an overview of the setting of our experiment.

Software Develop a test suite for Group project Software Testing Course Course
Software Engineering BSc Artificial Intelligence MSc Curriculum Research
software Integration Figure 1: Overview of the research setting of our
experiment.

In the following sections, we describe the two core components of our
experimental design, the setting of our experiment and the subject, the
research software.

###  3.1 Setting

The setting for our experiment is a software testing course for students of
two study programs at Blekinge Institute of Technology: Bachelor of Science
(BSc) in Software Engineering and Master of Science (MSc) in Artificial
Intelligence.

This software testing course introduces hands-on testing and quality assurance
techniques for software systems. The course aims to make its participants
realize how testing can improve software quality if it is effectively
integrated into the software development processes and understand how this can
be achieved using both established and new techniques in software testing. The
aim is also to convey practical experience with tools that support and
automate these techniques.

As such, the course is split into two parts. In the first part, ten lectures
provide the theoretical foundations of software testing. Two practical
assignments deepen the understanding of the concepts introduced in the
lectures and introduce students to the technical aspects of software testing
in Python. In the second part of the course, students work in groups and
develop a test suite for a software project. In previous course iterations,
the software projects ranged from open-source to projects that students
developed. Afterward, the students summarize their findings as a written
report.

We conducted the experiment in the Spring of 2023 with $40$ enrolled students.
Twelve teams worked in three to four student groups on the given tasks.

###  3.2 Research Software

As part of the experiment, we asked the students to develop a test suite for
research software that we developed previously to simulate information
diffusion in code reviews at Microsoft, Spotify, and Trivago modeled as
communication networks [DB23].

The research software models the communication networks emerging from code
review as time-varying hypergraphs in which each vertex represents a developer
and each edge represents a code review, which can connect multiple developers
exchanging information. Hypergraphs are a generalization of traditional graphs
and capture time-dependent and higher-order interactions in social and
communication networks in which edges may connect more than two vertices.
Figure 2 depicts the concept of a time-varying hypergraph. We modeled the
information diffusion under best-case assumptions as minimal paths between
vertices through the time-varying hypergraphs.

$v_{1}$$v_{2}$$v_{3}$$v_{4}$$v_{5}$$v_{1}$$v_{2}$$v_{3}$$v_{4}$$v_{5}$$v_{1}$$v_{2}$$v_{3}$$v_{4}$$v_{5}$time
Figure 2: A time-varying hypergraph with five vertices and six hyperedges
which are available at different points in time.

The time dependency also extends the notion of a minimal distance in time-
varying hypergraphs: a path cannot only be the shortest in terms of its
topological distance but also the fastest or foremost path in terms of the
temporal distance. Dijkstra’s algorithm can be used to find the minimal paths
and, therefore, minimal distances in such graphs. However, Dijkstra’s
algorithm for time-varying hypergraphs has not been described in the
literature or implemented before.

The communication networks modeled as time-varying hypergraphs in our
simulation can become huge. For example, the communication network emerging
from Microsoft’s code review consists of over $309\,740$ code reviews (edges)
and $37\,103$ developers (vertices). The simulation is, therefore,
computationally expensive: On conventional hardware, it takes several days to
simulate the information diffusion and to process the results.

Before we started the experiment, the simulation had a minimal testing suite
covering tests for the graph data structure but not for the algorithm.
Furthermore, a minimal but complete README file contained instructions on the
installation and how to use it. As the course started, the research software
was already publicly available, but the related publication [Dor+24] was not
published.

The simulation fits well for our experiment for the following reasons:

  * •

The minimal existing test suite helped the students to get started without
revealing too much of the required solution. Yet, there is no single solution
to the problems, but different solution paths.

  * •

The algorithm is novel. The lack of a reference implementation makes thorough
testing non-trivial and yet very important.

  * •

Manual testing does not scale with the sheer size and complexity of the
communication networks. Also, brute-force testing approaches are not feasible
due to the computational complexity.

We defined requirements for the students to implement the test suite for the
research project as follows:

  * •

The test suite must comprise unit tests to automatically test the research
software.

  * •

The test suite must comprise unit tests to automatically test the research
software.

  * •

Test coverage metrics must be reported and discussed.

  * •

The test suite must cover both Python 3.8 and 3.11 environments, since there
is a significant performance improvement in the back then recently released
Python 3.11.

  * •

Beyond these minimal requirements, each team must select a focus of their
overall test suite. Possible areas to focus on are performance testing, unit
and integration testing, fuzzing, or visual testing.

Before the group-project phase of the course started, we informed the students
about the experiment and provided those instructions along with the README and
the scientific literature that introduced the mathematical background [Dor+22,
Dor+24] to the student teams. After the first week, the student teams decided
as a team for one focus area of the test suite.

##  4 Results

As a result of integrating software testing into teaching, we observed three
effects on our research software which we will discuss in the following
subsections.

###  4.1 Tests but no test code

The student teams tested the research software extensively and developed
diverse and well-engineered testing suites for our research software as part
of their projects. The implemented unit and integration tests validate the
expected behavior of the research software. Two project groups extended the
suite beyond classical testing to catch memory corruption and safety bugs
through fuzzy testing. Although no bugs were found, the different testing
approaches and diverse implementations of test suits provide a comprehensive
perspective on the quality of the research software. As authors of the
research software, we would not have been able to provide such in-depth
testing for the research software.

However, we could not integrate any test code developed by students into the
main project. This shortcoming is rooted in the way contributions to open-
source projects are handled. Our research software is open-source software
licensed under an MIT open-source license. Contributing to any open-source
project requires the contributor, as the owner of the intellectual property,
to grant permission, per the project’s open-source license, to use, copy,
distribute, or modify their contributions free of charge. That implies,
researchers are only allowed to use student contributions if the students make
their code available under the same or a compatible open-source license. In
our academic setting, an inherent conflict arises: Teachers evaluate the
students’ contributions while also benefiting from their contributions. Thus,
the teachers (as research software authors) exploit those contributions,
although not necessarily for financial gain. Hence, an ethical conflict arises
since students may feel forced to grant permission for their intellectual
property to avoid any potential negative impact on their grading. In
jurisdictions like Germany or Sweden, this also becomes a legal issue. To
avoid ethical and legal issues, we reached out for legal advice and to the
student council, seeking support. All parties advised us to underline the
optional and voluntary character of the potential contributions to the
research software.

Another obstacle for group projects is that all group members must agree to
publish their code under an open-source license. It would be nearly impossible
to unravel contributions from individual students without at least touching
the intellectual property of other team members. We did not explore how
willing individual group members would have been to contribute to the main
project while avoiding potential threats to the voluntary nature of
contributions.

Additionally, integrating code into the main software project is not
effortless. Very rarely, code is flawless and can be merged into the main
software project without further considerations, changes, or discussions. At
this point, we would like to emphasize that we feel sympathetic towards the
students: Spending additional and yet significant efforts that are not
rewarded and might be acknowledged in an academic context as they would become
co-authors of research software is not very compelling.

In summary, our research software was tested thoroughly in this course,
although we could not motivate the student groups to integrate their code into
the main research software.

###  4.2 Reduced hardware and software dependencies

We did not provide a unified hardware setup but relied on the students’
computers for testing the research software. To cover the large diversity in
hardware, operating systems, and software environments, we removed all
operating system and minimized hardware requirements and code dependencies.

Before the course, the research software was tested on macOS and Linux
(Ubuntu). In preparation for the course, we tested the research software on
the common operating systems Windows 11, macOS, and Linux (Ubuntu) to avoid
complications with students’ operating systems and development environments.

We also removed one software component and made it optional since. Although
the component orjson666https://github.com/ijl/orjson improves the performance
of the file processing substantially,
json777https://docs.python.org/3/library/json.html from the Python standard
library serves the same purpose and requires no additional steps to use.

Being agnostic on hardware and operating systems and having fewer external
dependencies contribute to easier use, maintenance, and extension of our
research software, facilitating a broader adoption among the research
community. Further, minimizing software dependencies can decrease the fault
proneness of a software system [Cat+09].

For us as developers of research software, minimizing hardware and software
created a small additional effort that is worthwhile to increase the re-
usability of research software.

###  4.3 Improved documentation

We found that scientific literature [Dor+22, Dor+24] and a minimal
documentation in the form of a README file were not sufficient for its use in
the context of teaching. To ease the barriers for students at the project
start and during the course, we continuously improved the documentation in
size and quality. We observed that continuously improving documentation
enabled efficient communication with the students.

Although the documentation improved, less experienced students requested
additional support to understand the domain and technical details of research
software, which is crucial for an efficient test suite. Therefore, we offered
each group weekly and individual students one-on-one sessions on request. In
particular, during the first weeks, this offer was well received, and all
groups booked meeting slots.

The continuous support through additional weekly meetings for all groups and
the continuous improvement of the documentation required significant
additional efforts, which exceeded the efforts that would have been required
for an open-source project, for example, where the domain and functionality
are more obvious.

We believe the improved documentation will help researchers reproduce or
replicate our simulation, ultimately fostering scientific growth through
traceability, transparency, and usability. This finding confirms the
reflection and recommendation by [VA24] [VA24].

##  5 Limitation

As for any _in-vivo_ experiment, there are two inherent limitations [SF18]:
First, our findings are limited in their transferability to other settings,
which means other courses and also other research software projects. Second,
although the setting of the experiment is realistic, it is subject to
confounding factors such as the specific type of research software or
previously attended courses that limit the precision of measurement. However,
we believe that our high-level findings are still transferrable to other
settings since the course introduces the students to the foundations of
software testing and does not require prior knowledge of software testing.
Additionally, two different curricula participated: Bachelor of Science (BSc)
in Software Engineering and Master of Science (MSc) in Artificial
Intelligence, which broadens the transferability to other curricula. Last, the
research software uses Python, including Jupyter Notebooks, which are common
in research software and teaching.

A considerable limitation of this study is that we do not quantify the
(additional) effort for the integration since we did not track the hours spent
on the integration or previous courses. Our estimates are based on personal
experiences and relative to previous instances of the courses. However, we
believe that a relative effort, as reported in this study, also provides
valuable insights for other research software developers and teachers, even
though a more precise effect size is missing.

We missed the chance to tailor the course evaluation after the course was
completed to our experiment and, therefore, to get a more profound insight
into the experiment from a student perspective. We plan to replicate our
experiment for the next iterations of the course and then collect student
feedback to better understand the challenges and benefits for students as
future researchers and future research software engineers.

##  6 Conclusion

In our experiment, we integrated the testing of research software into
teaching and observed the effects on the research software.

On the one hand, integrating research software engineering into teaching
substantially improved the research software and its usage and
trustworthiness. Some student teams developed outstanding and thoughtful test
suites to ensure the quality of the research software. The scale and extent of
testing would not have been possible within the research team that initially
developed the software.

Additionally, through the integration, we prepared the research software for
researchers who would like to reuse or build upon our research software in the
future. We improved the software documentation in preparation for the course
to ease the students’ onboarding. We also lowered hardware and software
dependencies, which made the initial setup easier, reduced the complexity of
the software, and, thereby, improved the maintainability of the software in
the future. This makes the integration into teaching an excellent incubator
for open science for research software engineering efforts.

On the other hand, those positive effects did not come for free. Due to the
complexity of the topic, we had to invest a significant amount of time in
preparing the research software for the course and in providing support to
student teams.

The most significant challenge we observed is the uncertainty about the
intellectual property of student code contributions to the project.
Contributing to open-source code requires copyright, which puts us researchers
and teachers in a conflicting position: We (teachers) grade students for work
that we (researchers) exploit—although maybe not financially. This conflict
raises at least ethical, but also—in jurisdictions like Germany or
Sweden—legal questions. We could not incentivize or motivate students to go
the extra mile to integrate their excellent work into the research software.
It remains unclear how this dilemma could be resolved. This uncertainty
regarding students’ contributions extends beyond research software and is
observable in contributions through bachelor’s and master’s theses.

Although there are not yet well-worn paths for integrating research software
testing into teaching, we believe that testing research software and teaching
can cross-fertilize. On the one hand, the research software gets an incubator
before being reused or extended in the scientific community. On the other
hand, students get in touch with the latest research in their respective
fields very early on as part of their curriculum. This hands-on experience
with research software engineering can help to train and prepare students, the
next generation of researchers.

We thank the participants of the deRSE’24 poster session for their insightful
discussions. This work was supported by the KKS Foundation through the SERT
Project (Research Profile Grant 2018/010) at Blekinge Institute of Technology.

## References

  * [BDL24] Nikolas Bertrand, Akshay Devkate and Anna-Lena Lamprecht  “Introducing an RSE Course at the University of Potsdam - First-Time Experiences and Plans for the Next Iteration”, deRSE24 - Conference for Research Software Engineering in Germany, 2024  URL: https://events.hifis.net/event/994/contributions/7832/
  * [Cat+09] Marcelo Cataldo, Audris Mockus, Jeffrey A. Roberts and James D. Herbsleb  “Software Dependencies, Work Dependencies, and Their Impact on Failures”  In _IEEE Transactions on Software Engineering_ 35.6, 2009, pp. 864–878  DOI: 10.1109/TSE.2009.42
  * [CML14] Zhenyu Chen, Atif Memon and Bin Luo  “Combining research and education of software testing”  In _Proceedings of the 29th Annual ACM Symposium on Applied Computing_ ACM, 2014, pp. 1179–1180  DOI: 10.1145/2554850.2555124
  * [Coh+21] Jeremy Cohen et al.  “The Four Pillars of Research Software Engineering”  In _IEEE Software_ 38.1, 2021, pp. 97–105  DOI: 10.1109/MS.2020.2973362
  * [DB23] Michael Dorner and Andreas Bauer  “michaeldorner/information-diffusion-boundaries-in- code-review: 1.0”  Zenodo, 2023  DOI: 10.5281/zenodo.10417852
  * [DDC20] Lin Deng, Josh Dehlinger and Suranjan Chakraborty  “Teaching Software Testing with Free and Open Source Software”  In _2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)_ IEEE, 2020, pp. 412–418  DOI: 10.1109/ICSTW50294.2020.00074
  * [Dor+22] Michael Dorner et al.  “Only Time Will Tell: Modelling Information Diffusion in Code Review with Time-Varying Hypergraphs”  In _ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)_ ACM, 2022, pp. 195–204  DOI: 10.1145/3544902.3546254
  * [Dor+24] Michael Dorner et al.  “The Upper Bound of Information Diffusion in Code Review”  In _Empirical Software Engineering_ , 2024
  * [EC22] Nasir U. Eisty and Jeffrey C. Carver  “Testing research software: a survey”  In _Empirical Software Engineering_ 27, 2022  DOI: 10.1007/s10664-022-10184-9
  * [FS24] Jakob Fritz and Robert Speck  “Do Scientists Write About Software? An RSE Publication Monitor”, 2024  DOI: 10.34734/FZJ-2024-01802
  * [Gar+17] Vahid Garousi, Michael Felderer, Marco Kuhrmann and Kadir Herkiloğlu  “What Industry Wants from Academia in Software Testing?: Hearing Practitioners’ Opinions”  In _Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering_ Karlskrona Sweden: ACM, 2017, pp. 65–69  DOI: 10.1145/3084226.3084264
  * [Gar+20] Vahid Garousi, Austen Rainer, Per Lauvås and Andrea Arcuri  “Software-testing education: A systematic literature mapping”  In _Journal of Systems and Software_ 165, 2020, pp. 110570  DOI: 10.1016/j.jss.2020.110570
  * [Gil+22] Grant Gilson et al.  “Design and Evaluation of ”The Missing CS Class,” a Student-led Undergraduate Course to Reduce the Academia-industry Gap”  In _Proceedings of the 53rd ACM Technical Symposium on Computer Science Education - Volume 1_ , SIGCSE  Providence, RI, USA: ACM, 2022, pp. 467–473  DOI: 10.1145/3478431.3499422
  * [GM16] Vahid Garousi and Mika V. Mäntylä  “A Systematic Literature Review of Literature Reviews in Software Testing”  In _Information and Software Technology_ 80, 2016, pp. 195–216  DOI: 10.1016/j.infsof.2016.09.002
  * [Got+24] Florian Goth et al.  “Foundational Competencies and Responsibilities of a Research Software Engineer”, 2024  arXiv:2311.11457
  * [Inc24] GitHub Inc  “GitHub: Pull Requests”, 2024  URL: https://docs.github.com/en/pull-requests
  * [Juc24] Guido Juckeland  “Introduction to Research Software Engineering - Experiences from first time RSE class at the Computer Science Faculty of TU Dresden”, deRSE24 - Conference for Research Software Engineering in Germany, 2024  URL: https://events.hifis.net/event/994/contributions/7943
  * [KMR14] Daniel E. Krutz, Samuel A. Malachowsky and Thomas Reichlmayr  “Using a real world project in a software testing course”  In _Proceedings of the 45th ACM technical symposium on Computer science education_ ACM, 2014, pp. 49–54  DOI: 10.1145/2538862.2538955
  * [NK17] Udit Nangia and Daniel S. Katz  “Track 1 Paper: Surveying the U.S. National Postdoctoral Association Regarding Software Use and Training in Research”, 2017  DOI: 10.6084/m9.figshare.5328442.v3
  * [Rie23] Dirk Riehle  “The Future of the Open Source Definition”  In _Computer_ 56, 2023, pp. 95–99  DOI: 10.1109/MC.2023.3311648
  * [SC18] Mary-Luz Sánchez-Gordón and Ricardo Colomo-Palacios  “From Certifications to International Standards in Software Testing: Mapping from ISQTB to ISO/IEC/IEEE 29119-2”  In _Systems, Software and Services Process Improvement_ 896 Cham, Switzerland: Springer International Publishing, 2018, pp. 43–55  DOI: 10.1007/978-3-319-97925-0˙4
  * [SF18] Klaas-Jan Stol and Brian Fitzgerald  “The ABC of Software Engineering Research”  In _ACM Transactions on Software Engineering and Methodology_ 27, 2018, pp. 1–51  DOI: 10.1145/3241743
  * [VA24] Elaine Venson and Reem Alfayez  “Bridging Theory to Practice in Software Testing Teaching through Team-based Learning (TBL) and Open Source Software (OSS) Contribution”  In _46th International Conference on Software Engineering Education and Training (ICSE-SEET’24)_ ACM, 2024  DOI: 10.1145/3639474.3640081
  * [Woh+12] Claes Wohlin et al.  “Experimentation in Software Engineering”  Berlin, Heidelberg: Springer Berlin Heidelberg, 2012  DOI: 10.1007/978-3-642-29044-2