Parinaz Barakhshan & Rudolf Eigenmann

*Parinaz Barakhshan,

University of Delaware, Evans Hall, 139 The Green, Newark, DE 19716

# Best Practices for Developing Computational and Data-Intensive (CDI)
Applications

Parinaz Barakhshan* (0000-0001-7232-3923)  Rudolf Eigenmann
(0000-0003-1651-827X)  Department of Electrical and Computer Engineering,
University of Delaware, Delaware, USA  Department of Electrical and Computer
Engineering, University of Delaware, Delaware, USA<EMAIL_ADDRESS>
(26 April 2016; 6 June 2016; 6 June 2016)

###### Abstract

[Summary] High-quality computational and data-intensive (CDI) applications are
critical for advancing research frontiers in almost all disciplines. Despite
their importance, there is a significant gap due to the lack of comprehensive
best practices for developing such applications. CDI projects, characterized
by specialized computational needs, high data volumes, and the necessity for
cross-disciplinary collaboration, often involve intricate scientific software
engineering processes. The interdisciplinary nature necessitates collaboration
between domain scientists and CDI professionals (Xperts), who may come from
diverse backgrounds.

This paper aims to close the above gap by describing practices specifically
applicable to CDI applications. They include general software engineering
practices to the extent that they exhibit substantial differences from those
already described in the literature as well as practices that have been called
pivotal by Xperts in the field.

The practices were evaluated using three main metrics: (1) participants’
experience with each practice, (2) their perceived impact, and (3) their ease
of application during development. The evaluations involved participants with
varying levels of experience in adopting these practices. Despite differing
experience levels, the evaluation results consistently showed high impact and
usability for all practices.

By establishing a best-practices guide for CDI research, the ultimate aim of
this paper is to enhance CDI software quality, improve approaches to
computational and data-intensive challenges, foster interdisciplinary
collaboration, and thus accelerate scientific innovation and discovery.

###### keywords:

Computational and Data-intensive (CDI) Applications, Best Practices, Xpert
Network, Research Software Engineers (RSEs), Research Facilitators, Research
Programmers

††articletype: Article Type

##  1 Introduction

Computational and data-intensive (CDI) applications are critical in modern
research for their ability to process and analyze vast amounts of data, as
well as execute complex computational tasks to address scientific problems.
CDI applications cater to the demands of data-driven and computation-heavy
research projects, requiring not only accuracy but also high levels of
efficiency and scalability. This paper aims to increase the productivity of
developers of such applications by describing and evaluating a set of best
practices, gathered from a diverse pool of researchers.

Best practices are guidelines or standards that outline the most effective and
efficient approaches to achieving desired outcomes in application development.
They provide a shared knowledge base and a common framework for practitioners,
researchers, and even experts to build upon. Although general software
engineering has a well-defined set of best practices for developers, which has
applicability to CDI research software, a specific guide is needed for the
following reasons.

CDI research distinguishes itself from general software engineering through
specific characteristics: (1) CDI problem domains often involve applying
computational methods to domain-specific challenges across various scientific
disciplines. (2) Interdisciplinary collaboration is often required, bringing
together diverse expertise to address complex problems. (3) CDI research is
inherently data-intensive, dealing with massive data volumes to extract
insights. (4) Significant computational resources are required for complex
computations and data processing.

Another key difference between CDI research and general software engineering
practices lies in the imperative for collaboration among domain scientists and
CDI professionals, emphasizing team dynamics beyond mere software development.
This is due to the interdisciplinary nature of such projects, which requires
collaboration between domain scientists and CDI support professionals —-
referred to as Xperts —- to tackle computer engineering challenges. Such
collaboration is less common in traditional software engineering, where the
focus is primarily on software development and engineering practices. In this
article, we use the terms Xperts, research software engineers (RSEs), research
facilitators, and research programmers interchangeably to denote professionals
who support CDI research software development.

Xperts come from a wide range of disciplines that extend beyond computer
science. Many are domain scientists who have delved into CDI application
development, eager to learn about new scientific fields as they support their
colleagues. Alternatively, some have developed their expertise through
practical experience, making significant contributions to CDI projects.
Irrespective of their origin, Xperts are expected to tackle computer
engineering issues in CDI research projects. They are instrumental in working
alongside domain scientists, lending their technical know-how, and navigating
the unique computational and data challenges of CDI applications.

The main goal of this article is to introduce a set of best practices for
developing computational and data-intensive (CDI) applications, aiming to
boost efficiency, effectiveness, and innovation in collaboration with science
teams. By promoting the adoption of these best practices that are tailored to
the characteristics of CDI research, CDI support professionals or developers
can enhance the development and optimization of CDI applications and improve
their efficiency and effectiveness.

The best practices highlighted in this article encompass newly introduced
practices tailored specifically for CDI applications, those substantially
diverging from conventional software engineering (SE) practices, or those
emphasized for their importance by the Xpert Network community 1, 2.

Several efforts contributed to identifying and evaluating best practices, as
described in Section 2: (i) Through the Xpert Network 2, 3, we collected
insights and best practices from CDI researchers, tool developers, and Xperts
in the CDI field. (ii) In the Atom project 4, 5, 6, 7, we implemented and
evaluated the practices identified via the Xpert network within a specific CDI
project. This effort also revealed additional practices and insights. (iii)
Through surveys we assessed the impact and usability of the practices
identified from these two sources. The responses also pointed to associated
challenges, benefits, limitations, and supporting tools.

The identified best practices are explained in Section 3. Each practice is
described in terms of a highlight, purpose, impact, recommended integration
method into the development process, significance in the context of CDI
development, challenges and limitations of implementation, as well as
supporting tools and resources.

Subsequently, Section 4 elaborates on our approach to evaluating CDI best
practices through three distinct surveys that contributed unique insights. (i)
Surveys targeted at CDI Researchers: We collected feedback from a diverse
group of researchers and practitioners across various disciplines, involving
46 participants utilizing HPC resources at the University of Delaware (UD).
This survey provided a broad perspective on how these best practices are
adopted and perceived in the research community. (ii) Case study evaluation
via the Atom Project: By applying the practices identified by the Xpert
Network in the Atom Project, we could observe their impact and usability. This
effort also led to the identification of additional practices. Feedback from
seven project participants, including CDI professionals and domain scientists,
informed our understanding of how these practices perform in a real-world
setting, highlighting their practical benefits and challenges. (iii) Expert
Reviews: We engaged 14 CDI support professionals, or Xperts, who were actively
engaged in Xpert Network activities, to obtain specialized assessments and
insights. Their significant experience with CDI applications allowed for a
detailed and expert analysis.

Our work is related to several key initiatives in the field of scientific
software development that have made significant contributions to improving the
quality and reliability of computational and data-intensive research across
various domains. Section 5 discusses these contributions. Notable efforts
include Better Scientific Software (BSSw) 8, the Research Software Engineering
(RSE) community 9, ACCESS (Advanced Cyberinfrastructure Coordination
Ecosystem: Services & Support) 10, 11, and MolSSI (Molecular Sciences Software
Institute) 12. Our effort distinguishes itself by providing and evaluating a
set of best practices tailored specifically for CDI support professionals and
developers. These guidelines address their unique requirements for working in
collaboration with science teams and developing CDI applications.

Key contributions of our research include the identification of seventeen best
practices specific to computational and data-intensive (CDI) research, along
with providing practical insights on challenges and limitations tied to each
practice within the CDI research landscape, and an overview of tools commonly
utilized in CDI projects to efficiently implement these practices.

##  2 How we collected our information: Identifying Best Practices

The information presented in this paper was gathered through three main
sources: the Xpert Network 2, the Atom Project case study 6, 7, 5, and surveys
that engaged a broad audience of researchers and practitioners. These efforts
served to identify relevant best practices and provide data for their
evaluation.

###  2.1 Xpert Network

The key initiative in this process was the establishment of the Xpert Network,
a collaborative platform catering to: (i) researchers involved in the
development and utilization of CDI applications, (ii) Xperts ( Facilitators
13, or Research Software Engineers (RSEs) 9, 14), professionals providing
support with CDI technology and methodologies, (iii) domain experts and
scientists from universities and research institutions, and (iv) developers
creating tools to facilitate the creation and usage of CDI applications.
Through webinars, workshops, and Birds-of-a-Feather (BoF) sessions, this
diverse community shared knowledge, fostered collaboration, and identified
best practices and essential tools for developing CDI applications 2, 15, 3,
1.

Participants of the Xpert Network activities represent over 60 distinct
projects, universities, national labs, and international institutions. Notable
projects include ACCESS (Advanced Cyberinfrastructure Coordination Ecosystem:
Services & Support) 10, CyVerse 16, MolSSI (Molecular Sciences Software
Institute) 12, and US-RSE (United States Research Software Engineer
Association) 9. These projects cover a broad spectrum of disciplines and
expertise, fostering collaboration and knowledge exchange among researchers,
domain experts, and computational professionals. More information on
participating resources and institutions is available on the project home page
17.

The collective expertise and varied experiences of participants in the Xpert
Network were instrumental in formulating the practices discussed later in
Section 3. This effort resulted in the formulation of the first 15 practices.

###  2.2 Atom Project Case Study

We used a CDI project in which we were directly involved, the Atom Project 4,
for gauging the applicability and completeness of the identified practices.
The Atom Project is a collaboration of physicists and computer scientists to
develop computational applications and a community web portal for calculating
and presenting data about certain properties of atoms and ions 6, 7, 5.
Examples of such properties are transition rates and polarizabilities. Among
the software development tasks were the creation of the computational
applications and the web portal server and client, including a database of the
requisite physics data.

Through this collaboration, the unique challenges and complexities of
interdisciplinary collaborations were highlighted. The effort led to the
identification of two additional best practices, described as the final two in
Section 3. The Atom Project participants also contributed to the surveys used
in Section 4.

###  2.3 Surveys

To enable the evaluation of the identified best practices in Section 4, the
primary means was the creation of surveys. We devised and circulated surveys
tailored to various participants, each serving distinct purposes. These
purposes include the validation of the practices, the assessment of their
impact and usability, as well as the understanding of practical applications
and challenges. We also gathered information about tools that support the
practices. We targeted three different groups with the surveys, including
general CDI researchers, Xperts (CDI support professionals), and the
participants of the Atom project case study.

##  3 Identified Best Practices

The following 17 subsections present each of the identified best practices in
terms of (i) a one-sentence recommendation, (ii) a brief description of the
practice, (iii) the expected impact of the practice, (iv) recommended methods
for integrating the practice into the development process, (v) an explanation
of how the practice differs from those found in general software engineering,
(vi) anticipated challenges and limitations of employing the practice, and
(vii) a list of tools and resources that help support the practice.

The first five practices enable interdisciplinary collaboration. They are
specific to the CDI context and rarely found in general software engineering
recommendations. The remaining 12 practices are software development oriented
but differentiate themselves from general software engineering practices
through distinct features or particular emphasis given by our Xpert Network
participants. Paragraphs “Difference from General Software Engineering
Practices” elaborate this point for each practice.

###  3.1  BP1 - Onboarding Xperts from Diverse Backgrounds

Devise an onboarding process that is tailored to the specific backgrounds of
individual Xperts.

#### The Practice

Xperts bring diverse backgrounds and a wide range of expertise to the team.
Members of Xpert groups can come from various fields, including domain
sciences and computer science, or they may have gained experience supporting
CDI science teams through on-the-job training. While individuals with a
computer science background often have a natural grasp of Unix commands,
parallel programming models, and version control, these very skills may be
critical best practices for Xperts with a domain-science background to
acquire. On the other hand, Xperts who have a domain degree are often well
acquainted with overcoming the terminology gap discussed in Section 3.5, which
greatly facilitates communication with new science collaborators. Training
programs for onboarding Xperts must take these differences into account,
allowing trainees to focus on unfamiliar best practices.

#### Impact

The main impact is a shortened onboarding process, yielding a productive team
member quickly.

#### Recommended Integration Methods

  * •

Conduct a thorough assessment of each Xpert’s knowledge, skills, and
background to identify their specific training needs.

  * •

Create custom training modules for specific skills and practices. Combine the
modules as suited for the training needs of individuals.

#### Difference from General Software Engineering Practices

While onboarding is a standard practice in general software engineering, a key
distinction in onboarding CDI research application developers arises from the
interdisciplinary nature of CDI projects and the diversity of Xpert
backgrounds.

#### Challenge(s) & Limitation(s)

Personalized onboarding as opposed to general onboarding, is initially more
expensive and resource-intensive. However, the long-term gains of increased
productivity and team performance validate the initial investment.

#### Supportive Tools and Resources

The Xpert Network discussions have pointed to a number of tools that assist
with general onboarding. They include Slack 18, Zoom 19, Microsoft Teams 20,
and Google Workspace 21 for team communication, file sharing, and project
collaboration.

###  3.2 BP2 - Understanding the Academic Environment

Familiarize Xperts with the academic reward system, research priorities, and
side activities that strengthen the science team’s academic standing.

#### The Practice

An issue for Xperts that is essentially absent in a general software
engineering context is the importance of understanding the academic
environment in order to navigate the unique dynamics and expectations of
science teams. This issue is especially relevant for software engineers with
backgrounds in industry, where hierarchical organization and the overriding
goal of creating a reliable product as rapidly as possible are the norm.
Understanding the academic reward system, research priorities, and the many
side activities that researchers may get engaged in to maintain the academic
standing of the science team will influence project decisions. For example,
the need to be innovative might take priority over delivering production-
quality applications; or publishing a research paper may compete with creating
periodic project reports. In fact, some of the best practices need to be
understood from this viewpoint, such as the issues of documentation and
testing, mentioned in Sections 3.12 and 3.11. Another distinction of the
academic environment is its dynamic nature, where project specifications may
change "overnight", driven by the unpredictability of scientific innovation.
Section 3.6 on understanding the domain problem and developing a project plan
will expand on this issue.

#### Impact

Comprehending academic factors will directly affect the setting of realistic
goals, expectations, and timelines. It will also improve productive
collaboration between Xperts and domain researchers.

#### Recommended Integration Methods

  * •

Devise an onboarding module that introduces new Xperts to the local academic
environment. Describe expectations and job performance criteria, such as
(co-)authoring scientific publications and grant writing, general goals of
science teams to be assisted, prototyping versus production-quality code
development, and the time devoted to tasks beyond software development, such
as documentation, research, presentations, mentoring, and training.

  * •

At the beginning of each new project, review these expectations and criteria
jointly with the domain science teams that will be supported.

#### Difference from General Software Engineering Practices

This practice is essentially absent in a general software engineering context.

#### Challenge(s) & Limitation(s)

The diversity of academic environments and research disciplines that Xperts
support makes it difficult to define general recommendations. It requires that
expectations be carefully defined for each team and project. An ability of
Xperts to be sensitive to and adapt to new environments is critical.

#### Supportive Tools and Resources

There are no specific resources recommended for this practice.

###  3.3 BP3 - Developing a Breadth of Skills for the Effective Handling of
Projects

Build partnerships with external Xperts to prepare for the diversity of
supported domain projects and their required skill sets.

#### The Practice

Modern CDI applications encompass a wide spectrum of technologies, including
computing paradigms, programming languages, architectures, and algorithms.
With the continuous evolution of CDI applications and their diverse
technological requirements, it becomes challenging for an individual or a
small team of Xperts to possess all essential skills. For instance, emerging
CDI applications might necessitate expertise in machine learning techniques,
an area that individual Xperts or small teams serving a vast CDI research
community may struggle to fully cover. Furthermore, these professionals often
engage in tasks spanning the entire software life cycle and contribute to
project management. Hence, it becomes imperative to identify instances where
seeking external advice is necessary. By establishing and nurturing
connections with other Xpert teams, individuals and teams can ensure access to
supplementary skills and knowledge that go beyond their current expertise.

#### Impact

Applying this practice and maintaining contacts with other expert teams
enables Xperts to enhance their readiness in handling the multitude of
emerging tasks throughout the life cycle of a CDI application and can
effectively navigate the complexities of the application development process,
ensuring successful project outcomes.

#### Recommended Integration Methods

  * •

Participate in communities of practice where Xperts with similar interests or
expertise share insights, address challenges, and collaborate on solutions. An
illustrative example is the Xpert Network 2, offering a dedicated space for
Xperts to connect, exchange experiences, discuss challenges and share ideas.
These interactions foster cross-team collaboration and the exchange of
valuable lessons learned.

  * •

Attend networking events, conferences, workshops, and seminars in the CDI
field to connect with professionals from diverse backgrounds and engage in
knowledge-sharing opportunities.

  * •

Join online forums, social media groups, and professional networks related to
the CDI domain to actively participate in discussions, share insights, and
connect with Xperts from various projects.

  * •

Consider outsourcing projects to Xpert teams with expertise in skills not
available locally.

#### Difference from General Software Engineering Practices

This practice is rarely found in general software engineering guides. We
attribute this to the smaller average size of academic research teams compared
to industry teams, making it more difficult to cover all the necessary skills.
Also, research is intrinsically exploratory, including the test of novel
technologies, thus expanding the range of skills Xpert groups need to be aware
of. Finally, intellectual property is guarded more closely in industry
settings, whereas seeking and providing help across groups and institutions is
often done freely in academic environments.

#### Challenge(s) & Limitation(s)

Limited resources, in terms of time, funding, and personnel, are a primary
challenge to building an Xpert team that covers the required skills and keeps
up with new technologies. This is a hard constraint. However, there is also a
cultural aspect of interdisciplinary collaboration and knowledge sharing among
Xpert teams and within the academic community, which can be addressed with
proper project management: Team members must be encouraged and given the time
to seek interactions with related groups, participate in professional
discussion forums and events, and continue their education.

#### Supportive Tools and Resources

While the practice is more reliant on personal connections, networking, and
maintaining contacts with other Xpert teams, there are a few tools and
resources that can support this practice in CDI development.

  * •

Professional networks and communities, such as LinkedIn 22, Twitter 23, Xpert
Network 2, and research forums, provide opportunities to connect with peers
and address open problems within the field.

  * •

Online forums and discussion boards provide platforms for connecting with a
wider community and learning from shared experiences. An example is Connect
Cyberinfrastructure (cnct.ci 24), which offers access to various portals
serving different segments of the research computing and data community. Users
can join and actively participate in threads to engage with others and gain
valuable insights.

###  3.4 BP4 - Collaborative Assistance Between Xperts and Domain Scientists

Assist domain scientists through short-term, close collaborations, rather than
from behind a service desk.

#### The Practice

A recommended form of interaction between Xperts and CDI domain scientists is
through collaborative assistance. For a period of one to several months,
Xperts work side-by-side (physically or virtually) with the domain scientists
whose projects they support. While collaboratively working on the problem,
each team member focuses on their respective areas of expertise, with Xperts
handling computer-engineering issues and domain scientists addressing
application science problems. This division of labor allows for immediate
problem-solving and efficient utilization of skills. It also reduces the need
for cross-training. Over the course of the joint work, the collaborators tend
to pick up each others’ knowledge, skills, and terminology, allowing the
domain researchers to continue the project independently, or with minimal
remote support, after the collaborative assistance period. This collaborative
model was initially pioneered by XSEDE’s ECSS group 25 and has since been
successfully implemented by other Xpert teams 3.

#### Impact

Through collaborative assistance, Xperts and domain scientists leverage each
other’s skills and expertise, leading to accelerated progress and development
of a higher-quality end product. This collaboration also facilitates the
exchange of knowledge and the mutual understanding of each other’s skills and
terminologies. Surveys by the XSEDE ECSS group 25 have found that
collaborative models, where domain scientists work with Xperts, can
significantly enhance efficiency 26, 27.

#### Recommended Integration Methods

  * •

Allocating projects to Xpert resources is a key step. Carefully match the
skills needed for the project with those available among the Xperts. A
competitive proposal process may be indicated if requested projects exceed
available resources.

  * •

The initial project phase is especially critical; in-person collaboration was
highly recommended by Xpert Network participants. In addition to the issues of
Section 3.2, devote sufficient time to the discussion of overall domain
science goals, specific collaboration goals, terminology, possible approaches,
project plan, and division of work between Xperts and domain scientists.

  * •

Xperts and their home teams need to keep in regular contact, such as weekly
meetings to brief the home team on progress and discuss open problems. This is
especially important for in-person collaborations at remote sites.

  * •

Be sure to plan your time into the schedule for professional development, even
when at remote sites.

#### Difference from General Software Engineering Practices

While this practice may also be applicable in an industrial setting, it has
not been covered in the general software engineering literature.

#### Challenge(s) & Limitation(s)

Adding a competitive process introduces overhead for setup, proposal writing,
and evaluation; simplified allocations, FIFO processing, and short
collaborations may be alternatives. Travel logistics and space constraints for
in-person collaborations are obvious challenges, with recent advances in
remote participation providing reasonable options. The hand-over from Xpert to
domain scientist at the end of the collaborative project can be an issue; a
schedule of weekly (or initially even daily) brief “check-in Zoom calls”, with
decreasing frequency is advisable.

#### Supportive Tools and Resources

Among the supporting tools mentioned by the participants are collaboration
platforms for information sharing, virtual meeting tools for real-time
interactions, as discussed in Section 3.1, and version control systems for
collaborative code management, which will be detailed in Section 3.9.

###  3.5 BP5 - Overcoming the Terminology Gap Between Computer and Domain
Sciences

Avoid domain-specific language, define key terminologies, identify and clarify
shared terminologies.

#### The Practice

The gap between computer science and domain science lingo is an often-
mentioned issue. The challenge can be big if the same term is used by both
collaborators but with different meanings. Xpert Network participants reported
significant confusion and even incorrect project executions due to this
challenge. Awareness of the issue and patience in trying to understand the
collaborators’ viewpoints is critical. Keeping the vocabulary to the
essentials and investing time in explaining new terms is key to successful
collaboration. Using many examples and frequent feedback from both sides will
help bridge this gap.

#### Impact

Overcoming terminology gaps reduces misunderstandings and errors in project
development, ensuring that the intended goals and objectives are accurately
translated and achieved. It brings clarity to reports and research papers, and
it facilitates the transfer of knowledge between computer science and domain
sciences.

#### Recommended Integration Methods

  * •

Encourage team members to look out for terminology gaps and resolve them
through frequent communication, especially at project begin.

  * •

Explain terms and concepts using simple language and examples.

  * •

Develop a glossary of terms, also serving for project documentation.

#### Difference from General Software Engineering Practices

The terminology gap is wider in CDI research contexts compared to general
software engineering. While understanding the client is important for all
software engineers, Xperts in CDI research contexts may need to become
familiar with highly specialized domain vocabularies. Similarly, domain
scientists need to acquire the necessary computer science terminology in order
to sustain the project after the collaboration is completed.

#### Challenge(s) & Limitation(s)

Time pressure and unawareness of the pitfalls of terminology differences may
prevent this practice from being given adequate attention. Project managers
are advised to alert team members of its importance and set aside dedicated
time at project begin.

#### Supportive Tools and Resources

While there are no specific tools that help identify and overcome terminology
gaps, general collaboration tools and platforms, such as Google Docs, Wikis,
GitHub pages 28, and Stack Overflow 29, have been mentioned as useful. They
help develop and share glossaries, leading to improved understanding of the
joint projects.

###  3.6 BP6 - Understanding the Domain Problem and Developing a Project Plan

Invest time in understanding the domain problem, turning possibly vague ideas
into a feasible solution approach, and creating a comprehensive project plan
to effectively address well-defined requirements.

#### The Practice

Many Xpert Network participants highlighted the importance of first contact
with the supported domain scientists and the approach taken to understand
problems and develop solutions. In addition to awareness of the terminology
gap, the relevance of investing time in understanding the goals and the
functional as well as non-functional requirements was emphasized. Showing
patience in the process is critical. The domain researcher needs to be helped
in transforming an idea that may be initially vague into a concrete plan.
Developing specific requirements for the computational application and the
underlying system is essential. These requirements need to be reviewed
periodically, as CDI research often evolves during the course of the project.

#### Impact

This practice provides clarity on project objectives, challenges, and scope,
establishing clear directions and focusing on critical aspects of the problem
at hand. It also helps identify necessary compute and storage resources.

#### Recommended Integration Methods

  * •

Devote significant time to understanding the domain problem and
discussing/developing solutions with domain scientists. Do so at project begin
as well as when research directions change.

  * •

Be sure to define and/or review goals, scope, deliverables, and milestones.
Identify and challenge assumptions. Explore alternatives. Plan ahead for
unexpected outcomes and contingencies.

  * •

When planning, begin with high-level ideas, gradually narrowing the focus and
refining details. (This point was emphasized, given different backgrounds of
Xperts versus domain researchers and the terminology issue of Section 3.5.)

#### Difference from General Software Engineering Practices

The collaborative relationship, with domain scientists being closely involved
in the project and eventually continuing without Xpert support, as described
in Section 3.4, makes this practice fundamentally different from general
software engineering, where a problem is solved for the client. Furthermore,
the dynamic nature of many research projects increases the importance of
planning not only at project begin, but revisions when emerging research
results prompt changes in project directions.

#### Challenge(s) & Limitation(s)

Deadline pressure can be a major impediment to this practice, as significant
time is needed for the teams to get familiar with each other and with project
plans. Additional time investment is needed when project directions change.
Recognizing the strength that the diversity of backgrounds as well as a
thorough review of goals, approaches, and assumptions can bring to the project
will help overcome this challenge. This practice and its challenges go hand-
in-hand with “Overcoming the Terminology Gap,” described in Section 3.5.

#### Supportive Tools and Resources

There are numerous project management tools available that can assist in
creating and managing project plans. Some popular ones include Microsoft
Project 30, Asana 31, Trello 32, and Jira 33. These tools allow for defining
project scope, creating milestones, setting deadlines, and allocating human
resources effectively.

###  3.7 BP7 - Prioritize Functional Requirements

Prioritize requirements based on project goals and stakeholder needs. Engage
with stakeholders throughout the course of the project to gather and
prioritize essential features as project needs evolve.

#### The Practice

The dilemma of a large number of desirable features but only a short project
duration can be substantial in the development of scientific software. It is
important to focus on those features that are essential for the core
objectives of the application and the requirements of the stakeholders.
Reprioritization may be needed when research outcomes lead to new insights or
stakeholder requirements change.

#### Impact

This practice increases the efficiency of the development process and the
adaptability to evolving research needs. It leads to the timely delivery of
critical functionality to users, enabling early feedback for iterative
improvements.

#### Recommended Integration Methods

  * •

Gather and document all features of the to-be-developed software. Categorize
the features by the source (project objectives, external stakeholders) and
priority (essential, desirable, optional).

  * •

Review the progressing project periodically for agreement with the set
priorities.

  * •

Review, and if needed revise, priorities as research outcomes evolve.

  * •

If external stakeholders are involved, provide frequent feedback on progress
and monitor requirements. Revise priorities when requirements change.

#### Difference from General Software Engineering Practices

Functional prioritization in a research context is especially important, as
there can be a large number of "nice-to-have" features of CDI applications,
which however are not essential for realizing and demonstrating the core
science contribution. At the same time, re-prioritization can be critical, as
the research contributions may change with new results and insights.

#### Challenge(s) & Limitation(s)

The tendency to implement "bells and whistles" or features that are easy to
realize often challenges most research software projects. Strict setting of
and adherence to priorities is advised. Categorizing features from essential
to optional is not always straightforward, however. For example, a seemingly
optional feature of a user interface may well be the one convincing a reviewer
that the developed system meets his/her bar for user-friendliness. Frequent
stakeholder engagement can help address this challenge.

#### Supportive Tools and Resources

Tools such as Jira 33, and Trello 32 can assist in capturing, organizing,
prioritizing, and tracking functional requirements.

###  3.8 BP8 - Issue Tracking

Choose a suitable tool or method to track project requirements and bug
reports, from their origin to their implementation.

#### The Practice

Issue tracking is about systematically recording and monitoring the status of
all project requirements and bug reports, from their origin to the
implementation. Issue tracking is especially important in mid- and large-size
CDI development projects, which may last several years and personnel may
change. It is essential to record feature requests, implementation status, and
the reasons for accepting or rejecting a request, so as to maintain a clear
record and facilitate re-assessment and re-prioritization of functional
requirements. This is an often overlooked practice.

#### Impact

A clear record of feature requests and their implementation status enables the
team to prioritize, assess, and adapt functional requirements to meet evolving
project needs and possibly conflicting requirements. Issue tracking tools also
prevent duplication of efforts, boost overall productivity, and reduce the
risk of overlooking crucial issues, ultimately contributing to successful
high-quality CDI applications. Traceability enhances transparency,
accountability, and quality control throughout the development process.

#### Recommended Integration Methods

  * •

Define a repository for collecting and tracking issues. Define the visibility
and access to the repository by stakeholder groups. Be sure to consider
informally communicated issues as well.

  * •

Define workflows and statuses for resolving issues, documenting implementation
decisions, tracking progress systematically, and reporting status updates to
stakeholder groups.

  * •

Periodically review unresolved issues, implementation plans, and
schedules/priorities.

#### Difference from General Software Engineering Practices

Recall that, unlike general software engineering practices, the development of
CDI applications often experiences evolving research perspectives and changing
requirements. Issue tracking in this dynamic environment goes beyond its
conventional purpose and becomes a means for recording the various paths
explored during the project development. This includes documenting reasons for
pivoting or persisting with certain paths. Essentially, it serves as a tool to
document team decisions and illuminate the project’s trajectory, adapting to
the unique challenges posed by CDI applications.

#### Challenge(s) & Limitation(s)

Scalability is a primary challenge. At the beginning of a project, it may seem
easy to track issues "in the head". However, the number of issues almost
always increases beyond the point where this is feasible. The team may miss
unresolved issues, fail to see conflicts between requirements, or forget to
inform stakeholders. Another challenge is that clients may make bug reports or
feature requests in ways most convenient for them, such as sending an email to
a staff member they happen to know, where they may get lost. Without a
systematic issue tracking mechanism, it is easy to get caught by such traps.

#### Supportive Tools and Resources

There are several methods for tracking issues in CDI development projects:

  * •

Manual Issue Tracking: This method involves using spreadsheets or Google
documents to manually track and record the status of issues. It requires
updating the tracking system by hand as issues are processed.

  * •

Visual Workflow Management: Kanban boards 34 provide a visual representation
of issues or tasks on a board with columns representing different stages of
progress (e.g., "To Do," "In Progress," "Done"). Issues are moved across the
board as they progress, providing a quick overview of their status.

  * •

Automated Issue Tracking Tools: Dedicated Issue-Tracking Tools provide a
structured and automated approach to tracking issues. These tools offer
features such as creating, assigning, and updating issues, attaching files,
setting priorities, and generating reports. Examples include Jira 33, GitHub
Issues 35, Bugzilla 36, and Trello 32.

The choice of tracking method depends on the size, complexity, team structure,
and requirements of the project. Automated issue-tracking tools are generally
recommended for larger projects or teams, as they offer more comprehensive
features and scalability. Smaller projects or teams may find manual tracking
or simpler tools, such as Kanban boards, sufficient.

###  3.9 BP9 - Source Code Management (SCM) and Version Control

Make use of source code management and version control systems to track the
evolution of your software throughout its development lifecycle.

#### The Practice

Many successful science software applications had their origin in a "toy
program" written by a graduate student. It got gradually expanded by several
authors, caught the attention of a wide audience, and ended up becoming an
important research tool. Without version control to track the evolution of
code, scripts, and data in scientific software, the history of the origin,
authors, relationship of features, and specific extensions may no longer be
known, making it difficult to extend further and obtain the needed
documentation. Therefore, it is crucial for Xperts and domain scientists to
learn version control methods and tools to facilitate collaborative software
development, enable software roll-back to a previous, well-defined state, and
start developing a new branch of the software for parallel experimentation and
development. This practice is well known in general software engineering.
Xpert Network participants emphasized it because of its importance and the
likelihood of being overlooked by Xperts with non-software engineering
backgrounds.

#### Impact

Version control systems provide safety nets for experimentation, increase
productivity, enhance reproducibility, and traceability of CDI applications,
and thus promote scientific rigor.

#### Recommended Integration Methods

Xperts with computer science/engineering background are well familiar with
this practice and its relevance 37, 38. For Xperts without formal software
engineering training, the inclusion of source code management and version
control in the onboarding process is essential.

#### Difference from General Software Engineering Practices

While the practice itself is the same as in general software engineering, it
plays a special role in research. The intrinsically exploratory nature, with
at times several research branches being tried, abandoned, merged, and down-
selected, fits well the ability of SCM to fork and merge software branches. In
this way, SCM tools can become important instruments for tracking, managing,
and documenting research directions. This is especially important in light of
the often limited time for documentation, discussed in Section 3.12.

#### Challenge(s) & Limitation(s)

Deadline pressure is also a major challenge in this practice, especially for
Xperts not familiar with SCM tools. Convincing them of the benefits during
onboarding is critical.

#### Supportive Tools and Resources

Among the tools are "Centralized Version Control Systems", such as Apache
Subversion (SVN) 39 and Perforce 40 as well as "Distributed Version Control
Systems", including Git 41 and Mercurial 42.

###  3.10 BP10 - Code Review

Domain scientists review the code written by the supporting Xperts, and vice
versa. Doing so will detect errors, verify common project understanding, and
transfer knowledge.

#### The Practice

Code review is a software development practice in which a second person reads
the newly written code to identify errors and improve the quality of the code.
In the context of CDI research projects, where interdisciplinary collaboration
is essential, code review by the partnering Xperts and domain scientists is
natural and essential. Next to the goal of error detection and code
improvement, an important role of code review is to operationally refine high-
level requirements and develop agreements on coding standards.

#### Impact

While the primary goal of this practice is to improve code quality and detect
bugs, code review also helps ensure a common understanding of requirements.
Furthermore, it facilitates communication as well as the transfer of knowledge
of the implementation, which is essential for enabling domain scientists to
continue the work after the collaboration completes.

#### Recommended Integration Methods

For basic code review methods, refer to general software engineering
guidelines 38. Specific recommendations for CDI projects with collaborative
partnerships of Xperts and domain scientists include:

  * •

At the beginning of the collaboration, establish the timing and frequency of
code reviews within the software development workflow.

  * •

Incorporate the practice into the onboarding process for new team members.

#### Difference from General Software Engineering Practices

In CDI research, code reviews are particularly valuable because of the diverse
backgrounds and expertise of Xperts and domain scientists. Mutual code review
by Xperts and domain scientists leverages their combined skills and prompts a
thorough examination of the code from scientific as well as technical
perspectives. Furthermore, given that research software often deals with
cutting-edge technologies and novel algorithms, code reviews provide an
additional safety net for these innovations. Finally, the practice helps to
pass on the expertise of an Xpert to the supported domain scientist, who will
often carry forward the work once the collaboration has been completed.

#### Challenge(s) & Limitation(s)

Developers tend to forego code review when faced with limited resources, such
as small team sizes, pressing deadlines, or lack of funding. This needs to be
carefully weighed against the potential of costly efforts to recover from and
fix undetected errors later in the project.

Differing views of coding styles and practices between developer and reviewer
may cost time. However, the diversity of opinions can be an opportunity as
well; it may prompt fruitful discussions about best practices for software
design, which may lead to guidelines that benefit the overall project.

#### Supportive Tools and Resources

This practice can be facilitated by using specialized code review tools such
as GitHub Pull Requests 43, and Gerrit 44, which offer features such as code
diffing, inline comments, and tracking of review comments.

###  3.11 BP11 - Test-driven Development

Define the test cases that the application should pass before developing the
application itself.

#### The Practice

Many Xpert Network participants advocated test-driven development (TDD), where
test cases that will need to be passed at the end are defined before coding
begins. This practice helps clarify requirements and how they will need to be
demonstrated or verified at the end. TDD helps developers navigate resource
constraints and prioritization, as mentioned in Section 3.7. TDD can lead to
software that is lean and maintainable, by omitting functionality that is not
prompted by requirements. Automated testing is highly desirable, so as to
ensure that tests continue to be passed after software updates (see also
Continuous Integration in Section 3.13.)

#### Impact

Comprehensive testing can detect potential issues, bugs, and inconsistencies
early in the development process. Testing can also serve as a form of
documentation, as it defines expected behavior.

#### Recommended Integration Methods

TDD is well documented in the software engineering literature 45, 46. The
essence is to clarify and specify the tests that requirements need to satisfy.
Complex requirements need to be broken down into components that will be
tested individually. This task should be done before coding starts, rather
than when it nears completion. Careful review of the tests is needed when
requirements or research directions change. Developing an automated regression
testing process that runs after every software update is highly recommended.

#### Difference from General Software Engineering Practices

This practice is similar to testing in a general software engineering context.
However, the intrinsically dynamic nature of research and academic priorities
pose additional challenges, as described next.

#### Challenge(s) & Limitation(s)

Testing challenges in CDI research mirror those in general software
engineering contexts – foremost the difficulty of translating requirements
into adequate tests and the time needed to do so. As mentioned above, the
dynamic nature and priorities (see Section 3.2) of research projects add to
these difficulties. It is natural for research directions or user requirements
to change as new scientific insights emerge. Adapting the tests that need to
be satisfied by the new requirements can be both difficult and time-consuming.
Academic priorities, such as a focus on demonstrating principles and concepts
rather than full functionality increase these challenges. Difficult tradeoffs
may need to be made between demonstrating and verifying prototype behavior
versus showing adequate functionality that renders research results credible.

#### Supportive Tools and Resources

Many tools support testing, including testing frameworks (JUnit 47, PyTest
48), test automation tools (Selenium 49, Appium 50, Cypress 51), mocking and
stubbing frameworks (Mockito 52, Sinon 53, PowerMock 54), test data generation
tools (Faker 55, DataFactory 56, SQL Data Generator 57), and Continuous
integration and testing platforms (Jenkins 58, Travis CI 59, CircleCI 60).

###  3.12 BP12 - Documentation

Document your project to ensure long-term success, enable reproducibility, and
obtain proper credit for your work.

#### The Practice

Proper documentation is crucial for long-term success, reproducibility, and
recognition of one’s work, particularly in academic software. Documentation is
essential in the joint work of Xperts with domain scientists, who will carry
on the project after the collaboration is complete. However, documentation is
often lacking due to time constraints and the academic reward system’s focus
on principles and prototypes rather than production-ready software. Navigating
these differing demands is key.

#### Impact

Proper documentation enhances reproducibility, usability, maintainability, and
requirement traceability. In academic settings and CDI contexts, documentation
is especially important due to frequent staff (students, postdocs) turnover,
project handover at the end of collaborative assistance periods, and academic
credit.

#### Recommended Implementation Methods

  * •

Define the minimal but sufficient documentation needed for the different
stakeholder groups: developers (Xperts and domain scientists involved in the
development), domain scientists sustaining the project long-term, and end
users. Include glossaries.

  * •

Focus the descriptions on the overall project structure and workflows. Make
use of in-line documentation and documentation-generation tools for code
details. Be sure to include references to existing publications about the
research.

#### Difference from General Software Engineering Practices

The need for reproducible scientific results is a key demand on documentation
that is not usually present in general software engineering. This need
exacerbates the problem of finding good tradeoffs between time investment in
documentation versus project advancement. On the other hand, available
publications about the underlying science can contribute to the documentation.

#### Challenge(s) & Limitation(s)

The academic reward system (Section 3.2) and time pressure are among the
primary challenges for this practice as well. Unawareness of the need for
targeting different audiences (Xperts, domain scientists, end users) and a
tendency to focus on details rather than high-level descriptions compound the
problem. To counteract these difficulties, descriptions should focus on high-
level architectures and workflows; implementation details can be explained
through in-line comments combined with documentation-generation tools. The
latter can also help address challenges posed by changing research directions,
which can render programs and their descriptions obsolete; documentation
generated from properly inserted in-line comments can more easily keep up with
code modifications.

#### Supportive Tools and Resources

Google Docs has been mentioned as a widely useful platform for collaborative
editing, including the creation of documentation. Other tools, such as Doxygen
61, Javadoc 62, Sphinx 63, and MkDocs 64, automatically generate documentation
from comments within the source code, producing documentation in formats such
as HTML, PDF, or Markdown. Furthermore, version control systems, such as Git,
are able to track and version documentation files effectively, making them
readily accessible alongside code changes.

###  3.13 BP13 - Continuous Integration

Regularly incorporate new software updates into the main application version
to share the latest features and identify potential issues.

#### The Practice

Continuous Integration (CI) involves regularly merging code changes into the
primary software version to make the latest updates available to other
developers and users. While this practice may be well understood in industrial
software development teams, Xpert Network participants reported several
examples in which graduate students "sat on their codes" until graduation.
Sharing the program and its features early could have benefited both end users
and fellow researchers. It also could have served to identify issues with
usability, compatibility with other features, miscommunicated requirements,
defined test cases, and integration. Identifying such issues late in the
development cycles, such as after the student has graduated and left, can be
very costly. The process of CI typically involves automated build and testing
procedures (see also Section 3.11), using a version control system for
managing the software versions (see Section 3.9). CI may be extended to
include continuous deployment (CD) 65, 66, where successful builds are
automatically deployed to production or staging environments.

#### Impact

Continuous integration can significantly improve team productivity by making
new features available to other developers. It also reduces the cost of late-
discovered issues with usability, compatibility with other features, broken
requirements, failed regression tests, and incompatibilities in integration.
Furthermore, getting accustomed to this practice can create a more iterative
software development mindset, frequently incorporating feedback from both
fellow developers and users into the project. This agility is particularly
valuable in scientific research, where requirements and understanding of the
problem domain can evolve rapidly.

#### Recommended Integration Methods

  * •

Define required integration periods as well as an integration workflow.
Familiarization with supporting tools and practices for tasks such as version
tracking and test generations is key. Include this material in the onboarding
process.

  * •

Automate the CI process as much as possible, including regression testing with
an evolving test suite.

#### Difference from General Software Engineering Practices

The importance of CI is less understood in research than in general software
engineering teams. However, the dynamic nature of science increases the
relevance of this practice, as changing requirements make the described issues
more likely and their late discovery more costly. CI can also prompt
communication among CDI application developers in useful ways, assisting the
practices outlined in Sections 3.5 and 3.6.

#### Challenge(s) & Limitation(s)

Unawareness of the importance of CI in academic settings and the resources
needed to set up and maintain the necessary infrastructure are the main
impediments to this practice.

#### Supportive Tools and Resources

Jenkins 58 is an open-source automation server for the automation of building,
testing, and deployment processes. Travis CI 59 offers a cloud-based CI
service with easy setup and integration with GitHub. CircleCI 60 provides a
user-friendly interface and supports building, testing, and deploying
applications across multiple platforms. GitLab CI/CD 67 integrates seamlessly
into the GitLab platform, enabling the definition of CI/CD pipelines using
configuration files. Bamboo 68 by Atlassian offers CI/CD capabilities and
integrates with other Atlassian tools, such as Jira 33. TeamCity 69 by
JetBrains supports the building, testing, and deployment of applications and
supports various programming languages and frameworks. GitHub Actions 70 allow
for the automation of workflows directly within GitHub repositories. Azure
DevOps 71 by Microsoft provides a comprehensive CI/CD platform with a range of
tools and services for building, testing, and deploying applications on the
Azure cloud platform.

###  3.14 BP14 - Reproducibility

Enable reproducibility and transparency of your research findings by capturing
and sharing the scientific method, workflow, and their associated parameters.

#### The Practice

Reproducibility of research results refers to the ability to replicate
research findings using the same methods and data that were used in the
original study. This is an important requirement for all scientific research,
particularly in the context of CDI research. By capturing the experiment
workflow and its parameters (including input data, software, hardware
platform, exact versions and options used, intermediate data, and results) and
sharing this information through available platforms, researchers can increase
transparency and enable others to reproduce their experiments. This practice
is of growing importance 72, 73, and platforms for sharing reproducibility
data are becoming increasingly widespread (e.g., conferences and journals that
allow for or even require reproducibility artifact descriptions.)

#### Impact

By providing access to the data, code, and methods used in a study,
reproducibility allows other researchers to independently verify and validate
the results. This promotes transparency, accountability, and trust in the
scientific process. Reproducibility also enables the replication and extension
of studies, fosters collaboration, and knowledge sharing, and supports the
advancement of scientific knowledge. Additionally, the practice promotes open
science principles and encourages the sharing of research artifacts.

#### Recommended Integration Method

Methods that promote reproducible research are well documented in the
literature 72, 73. They were emphasized by Xpert Network participants for
their growing importance. Implementation methods cover documentation, data
management, code versioning, techniques for capturing comprehensive
experimental contexts, open source distribution, as well as sharing and
archiving practices.

#### Difference from General Software Engineering Practices

Reproducibility is a key demand of scientific experimentation and hence unique
to software development in research. It is rarely found in general software
engineering.

#### Challenge(s) & Limitation(s)

Among the impediments to this practice are: (i) Capturing the full
environmental context can be very difficult. For example, there may be
settings in the operating system or architecture that the researcher is not
even aware of, but which may affect performance. There may also be external
dependencies on tools or data that the researcher has no control over. (ii)
Architectures (both hardware and software) change rapidly and may render the
exact reproduction of a computational experiment infeasible after only a few
years. (iii) There is intrinsic uncertainty and often variability in the
scientific observation and experiment. Their quantification is a challenge in
itself. (iv) Research involving sensitive data or proprietary software and
hardware may explicitly limit reproducibility.

#### Supportive Tools and Resources

Supportive tools and resources for the practice of reproducibility in
scientific research include:

  * •

Version control systems discussed in Section 3.9, which effectively manage and
share code, data, and documentation.

  * •

Containerization technologies, such as Docker 74 or Singularity 75, which
package software applications and their dependencies into self-contained
environments to ensure that software can run consistently across different
systems, enhancing reproducibility.

  * •

Platforms such as the Open Science Framework (OSF) 76, osf.io, provide
researchers with comprehensive tools for project management, version control,
and collaborative workspaces, enabling organized and transparent research
workflows. OSF is designed as a collaborative project management platform.

  * •

Zenodo 77 and figshare 78 are repositories that facilitate the sharing and
preservation of diverse research outputs, assigning Digital Object Identifiers
(DOIs) to ensure proper citation and permanent accessibility.

  * •

For computational reproducibility, Code Ocean 79 offers a cloud-based platform
that allows researchers to encapsulate code, data, and dependencies in
executable containers, promoting the transparency of computational
environments. Code Ocean enables researchers to create and share "capsules,"
which are self-contained environments containing code, data, and dependencies.
Users can execute code within the Code Ocean platform, eliminating issues
related to varying local environments and configurations.

###  3.15 BP15 - Parallelization

Create parallel code by writing fully debugged serial code first, then
parallelizing it.

#### The Practice

Parallel execution is a fundamental strategy for gaining high performance in
CDI applications and optimizing resource usage. The question of whether or not
to write parallel code directly or serial code first, followed by
parallelization, is decades old. While advocates of the latter point out that,
by writing serial code first, one may choose algorithms that limit
parallelism, the Xpert Network participants have clearly expressed a
preference for that practice. One argument was that the complexity of doing
both steps at once – writing correct code and parallelizing it – is highly
complex. Another reason was that there is a scarcity of software development
tools – debuggers in particular – for direct parallel programming. Performing
the same in two distinct steps also opens the door for differential debugging
methods, where an experimental program, or program section, is compared to one
that is known to be correct.

#### Impact

This practice was reported to not only save software engineering time but also
reduce unpredictably long debugging times. The latter may be caused by
combinations of algorithmic and parallel programming errors, which can be
excessively difficult to diagnose with state-of-the-art parallel programming
and error detection tools.

#### Recommended Integration Method

This paper does not discuss the first step of writing correct serial code. We
refer to the extensive software engineering literature, with Software
Carpentry 80 being a good starting point. To effectively integrate
parallelization into your project, consider the following approaches:

  * •

Start by profiling and analyzing the program’s performance to identify
critical sections that can benefit most from parallelization.

  * •

Choose a method for applying parallelization. (i) Automatic parallelization,
often built into compilers, is typically disabled by default due to its
unpredictable effects, requiring software engineers to experiment to optimize
its use. (ii) Manual parallelization is more labor-intensive and includes such
techniques as annotating program sections with parallel directives or
completely rewriting them in parallel. The availability of a correct, serial
program is important in this process, as it can be used to verify changes
incrementally. (iii) Semi-automatic parallelization combines these approaches,
utilizing automatic tools for sections amenable to automation and manual
optimization for others.

#### Difference from General Software Engineering Practices

With almost all processors today being parallel, this practice is equally
relevant for general software engineering tasks where performance matters. It
is central to CDI applications, however, as the processing of large data
volumes and high-performance execution are often primary concerns.

#### Challenge(s) & Limitation(s)

Parallelization requires comprehending such issues as thread safety, data
dependencies, load balancing, and scalability. Mastering these skills can
exacerbate the challenges related to complex algorithms and systems. The
skills include recognizing code that cannot be parallelized and thus needs to
be replaced with new, parallel algorithms. Covering these topics in the
onboarding process is crucial.

#### Supportive Tools and Resources

  * •

Parallel programming models and libraries include OpenMP 81 for shared-memory
parallelism, MPI 82 for distributed-memory parallelism, and CUDA 83 or OpenACC
84 for GPU programming.

  * •

Tools for performance analysis and debugging include a range of instruments,
such as profiling tools (GNU gprof 85, Scalasca 86, HPCToolkit 87), debugging
tools (88, TotalView 89), and performance anlysis tools (VTune 90, TAU 91,
NVIDIA Nsight 92).

  * •

Parallelization Tools include automatic parallelizers, such as ICC (Intel C
Compiler) 93, Rose 94, Cetus 95, Par4all 96, Pluto 97, Polly 98, ParaWise 99,
ParaGraph 100, SUIF (Stanford University Intermediate Format) 101, and Polaris
102. There are also interactive parallelizers, such as iCetus 103, SUIF
explorer 104, and ParTool 105. Many of these tools are supported by research
projects.

###  3.16 BP16 - Sustainability & Maintainability

Consider sustainability and maintainability as core software design goals to
ensure software longevity.

#### The Practice

Sustainability aims to ensure the longevity of CDI applications. This goal is
rarely in the mind of academic developers, such as graduate students or post-
doctoral researchers, who typically see time horizons of just a few years.
However, design for sustainability can substantially reduce costs and increase
long-term success. There is also increasing demand from research sponsors, who
typically fund novel developments through short-term grants, but expect long-
term benefits from their investments. Maintainability contributes to
sustainability; at a basic level, it keeps an existing application
operational, but it also aims to facilitate easy extensions and make an
application tolerant of future technology changes.

#### Impact

Next to the mentioned benefits of longevity, cost savings, and success, this
practice also yields generally higher-quality CDI applications.

#### Integration Strategies

Many of the described practices increase sustainability, including those
related to code readability, documentation, version control, continuous
integration, and testing. Many common software engineering practices also
contribute, such as modular software design methods and containerization
techniques 106, 107.

#### Difference from General Software Engineering Practices

While the practice itself is the same as in general software engineering
contexts, the generally shorter-term outlook of academic projects makes its
consideration especially important. Next to the increasing demand for
sustainability plans of funding agencies, the practice’s influence on reduced
long-term costs and increased success further adds to its relevance in
research projects.

#### Challenge(s) & Limitation(s)

Navigating the tradeoff between time investments for the long-term benefits of
this practice versus immediate project needs represents one difficulty.
Another challenge is to anticipate changes in technology for deciding how to
best "future-proof" the application.

#### Supportive Tools and Resources

As many of the mentioned practices contribute to sustainability and
maintainability, the tools and resources listed in those sections have
relevance here as well. Of particular interest are container platforms and
tools, such as Docker 74 and Kubernetes 108. They help "future-proof"
applications by packaging them with other software components they depend on,
thus enhancing portability across evolving platforms.

###  3.17 BP17 - User Community Engagement and Exchange

Enhance usability and adoption through frequent user feedback.

#### The Practice

The risk of creating application software that does not align with user
expectations is significant. This often occurs because users cannot clearly
articulate their needs without first engaging with the application or without
being aware of the possibilities that developers and emerging technologies can
bring to the table. Additionally, user requirements can differ widely among
clients. In the absence of close user engagement, developers tend to implement
their own interpretations of what users want, or they may focus on adding
“cool features”. Frequent user feedback is key, especially for applications
that have feature-rich user interfaces and broad audiences. Establishing
effective feedback collection mechanisms is therefore a crucial element of the
project plan. This practice aligns with the principles of Continuous
Integration of Section 3.13. By continuously integrating feedback into the
development process, software is more likely to meet user needs effectively
and evolve in tune with evolving demands.

#### Impact

The adoption of this practice helps ensure that the final application aligns
closely with user expectations, as a result mitigating the potential for
costly post-deployment modifications. Additionally, this approach promotes
community engagement, leading to a sense of ownership and endorsement among
users. The practice not only drives wider adoption but also encourages active
user participation in the development lifecycle, which can yield a more
refined and user-centric product.

#### Recommended Integration Methods

  * •

In the project plan, define the intended user communities and the methods for
collecting their requirements and feedback. These could take the form of
online community commenting tools, product demonstrations, workshops,
presentations at conferences and professional meetings, surveys, and direct
contacts.

  * •

Take advantage of the tight interaction of Xperts and domain scientists to
clarify and refine requirements.

  * •

Define periodic feedback reviews, potentially in conjunction with Continuous
Integration cycles, to analyze received community comments and incorporate
them into the evolving project plan.

#### Difference from General Software Engineering Practices

This practice is essential in industry software product developments but often
overlooked in academic settings. However, due to the dynamic nature of
science, with evolving research directions and changing requirements,
community engagement is vital.

#### Challenge(s) & Limitation(s)

The time investment needed for this practice is, again, a primary obstacle. It
needs to be weighed against the possible cost of broken or misunderstood
requirements. Another issue is that users often provide feedback through
informal channels that differ from those predefined by developers, such as
phone calls, casual remarks during meetings, or as side notes in emails that
cover a variety of topics. It is important to capture these pieces of feedback
and integrate them into the central feedback repository. A further issue is
the dynamic nature of science. As in several other practices, it is a
significant challenge and further increases the importance of keeping tight
links with the user community.

A factor that helps overcome these challenges is the collaborative environment
of Xperts and domain scientists (as detailed in Section 3.4), which forms a
natural context for collecting feedback and refining requirements. Another
natural form of community engagement results from presentations at
conferences, professional meetings, and peer networking, which are part of
academic culture. These venues can serve as a valuable source of feedback.

#### Supportive Tools and Resources

Feedback mechanism tools, such as UserVoice 109, SurveyMonkey 110, and Google
Forms 111, are instrumental in collecting feedback, preferences, and opinions
from users.

##  4 Evaluation

To assess the efficacy of best practices, we utilized three distinct
evaluation methods: (i) researcher surveys, (ii) a case study of the Atom
Project, and (iii) expert reviews. Each method evaluates the usability and
impact of the practices from a distinct perspective -— general CDI
researchers, a combination of Xperts and domain scientists involved in a
specific research project, and Xperts assisting in the development of CDI
applications, respectively. We also collected data on the experience levels of
the survey participants. For each evaluation, we developed survey forms
tailored to the specific needs of the participants and the study. All of these
forms included a brief description of the practices under review prior to
soliciting feedback from the participants. The metrics evaluated and the
questions asked varied across the different forms to suit each distinct
methodological approach. Sections 4.1 through 4.3 describe each survey,
including objectives, participants, and an analysis of the results.

###  4.1 Researcher Surveys

The researcher surveys gauge the research community’s familiarity with
identified practices and assess their impact on CDI applications. This
evaluation was conducted when the initial 15 practices were identified by the
Xpert Network effort. The survey sought feedback from CDI researchers across
various disciplines utilizing HPC resources at the University of Delaware.
Participants possessed domain knowledge and some experience in CDI application
development.

The survey process briefly introduced researchers to the 15 identified best
practices. The participants were then asked to assess the practices on two key
metrics: (i) Experience evaluates the research community’s experience in
adopting the practices. (ii) Impact measures the effects of implementing
practices, helping identify those that yield promising results and
significantly benefit the field. While “No Experience” indicates that the
practice has not been used, "Low Impact" and "High Impact" indicate the
corresponding experience with the practice.

#### Survey Data and Findings

$0$$10$$20$$30$$40$$50$$60$$70$$80$$90$$100$ BP1 BP2 BP3 BP4 BP5 BP6 BP7 BP8
BP9 BP10 BP11 BP12 BP13 BP14 BP15 $23.91$% $23.91$% $21.74$% $34.78$% $34.78$%
$17.39$% $26.09$% $28.26$% $30.43$% $8.7$% $23.91$% $23.91$% $4.35$% $4.35$%
$2.17$% $2.17$% $6.52$% $6.52$% $10.87$% $4.35$% $76.09$% $76.09$% $78.26$%
$65.22$% $65.22$% $76.09$% $76.09$% $80.43$% $71.74$% $65.22$% $63.04$%
$80.43$% $69.57$% $76.09$% $76.09$% Percentage (%)Best PracticesGood ImpactLow
ImpactNo Experience

Figure 1:  \- Researcher Survey Results (46 participants)

BP1 - Onboarding Xperts from Diverse Backgrounds; BP2 - Understanding the
Academic Environment; BP3 - Developing a Breadth of Skills for Effective
Handling of Projects; BP4 - Collaborative Assistance Between Xperts and Domain
scientists; BP5 - Overcoming the Terminology Gap; BP6 - Understanding the
Domain Problem; BP7 - Prioritize Functional Requirements; BP8 - Issue
Tracking; BP9 - Source Code Management; BP10 - Code Review; BP11 - Software
Testing; BP12 - Documentation; BP13 - Continuous Integration; BP14 -
Reproducibility; BP15 - Parallelization;

Figure 1 on page 1 presents data from the researcher surveys. We identify the
following common trends:

  * •

All practices show "Good Impact" (63.04% to 80.43%) on CDI software
development.

  * •

Correspondingly there is a low occurrence of "Low Impact" across all practices
(0% to 10.87%), indicating the positive effectiveness of the practices.

  * •

There is substantial variation in familiarity with the practices, with 8.7% to
34.78% reporting no prior experience with the practices. This result suggests
that additional educational outreach and training may increase adoption.

The survey data also point to potential areas for improvement:

  * •

Low Impacts: Despite the overall positive feedback, a small proportion of “Low
Impact” responses, particularly in Code Review (6.52%), Software Testing
(6.52%), and Documentation (10.87%), points to challenges that may be
mitigated through targeted training and better resource allocation for the
application of these practices.

  * •

Limited Adoption: The limited application of certain practices, notably “BP4 -
Collaborative Assistance Between Xperts and Domain Scientists” and “BP5 -
Overcoming the Terminology Gap Between Computer and Domain Sciences” (both at
34.78%), as well as “BP11 - Software Testing” (30.43%), indicate a lack of
exposure to interdisciplinary collaboration or early-stage research
engagement. These findings suggest the importance of fostering
interdisciplinary projects and integrating these practices more effectively
across research phases.

###  4.2 Case Study Survey of the Atom Project

To further evaluate the best practices, we applied them in a case study
referred to as the Atom Project 4, 6, 7, collecting feedback from the involved
participants. Recall from Section 2.2 that this study aimed to gauge the
effectiveness and completeness of the practices in a science project in which
the authors were directly involved; the study identified two additional,
relevant practices (BP16 - Sustainability & Maintainability and BP17 - User
Community Engagement and Exchange).

#### Survey Explanation

The case study survey evaluated all 17 practices identified by the Xpert
Network and the Atom project, targeting computational experts and developers
involved in the Atom project. We collected three metrics: (i) experience with
each practice, (ii) the practice’s impact, and (iii) its usability.
Additionally, we asked the participants about methods and tools they found
useful for employing the practices as well as about experiences with
challenges, outcomes, benefits, and limitations of the practices.

#### Survey Data and Findings

((a)) Average Experience

((b)) Average Impact

((c)) Average Usability

Figure 2: \- Average Percentage Ratings for All Practices in the Case Study
Survey by Metric Overview

Figure 2 on page 2 shows the average percentage ratings for all practices
across each survey metric. The overall average “Good” rating for each metric
significantly exceeds the “Low” rating portion. This response indicates the
substantial expertise of the participants and their recognition of the
practices’ benefits and ease of integration into the development workflow.
Figure 3 on page 3 allows a more in-depth analysis of the data.

  * •

Figure 3(a) on page 3(a) shows that most practices have experience ratings
exceeding 50%, suggesting that survey participants exhibit a “Good” level of
familiarity and competency with the practices. However, certain practices,
such as BP10 and BP17, show an experience level below 50%. This disparity may
be attributed to the diverse backgrounds and skill sets of the domain
scientists and Xperts participating in the survey.

  * •

Figure 3(b) on page 3(b) illustrates a predominantly positive impact, with
"Good impact" ratings reaching 100% for nearly all practices. Practice 10 –
Code Review – is an exception, with a "Good impact" of 85.7%. We attribute
this lower rating to lesser awareness of the benefits of code review.

  * •

Figure 3(c) on page 3(c) indicates a high level of "Good usability" for
practices (above 71.5%), suggesting that the practices have been relatively
easy to integrate into the development process and have been well-utilized by
the participants.

BP1BP2BP3BP4BP5BP6BP7BP8BP9BP10BP11BP12BP13BP14BP15BP16BP17$0$$50$$100$ 28.5
28.5 14.3 14.3 14.3 28.5 71.5 14.3 28.5 42.8 14.3 28.5 28.6 57.1 71.5 71.5 100
100 100 85.7 85.7 85.7 71.5 28.5 85.7 71.5 57.2 85.7 71.5 71.5 42.9 Percentage
(%)Good ExperienceLow Experience ((a)) Participants’ Experience with Practices

BP1BP2BP3BP4BP5BP6BP7BP8BP9BP10BP11BP12BP13BP14BP15BP16BP17$0$$50$$100$ 14.3
100 100 100 100 100 100 100 100 100 85.7 100 100 100 100 100 100 100
Percentage (%)Good ImpactLow Impact ((b)) Impact of Practices

BP1BP2BP3BP4BP5BP6BP7BP8BP9BP10BP11BP12BP13BP14BP15BP16BP17$0$$50$$100$ 14.3
14.3 28.5 14.3 14.3 14.3 28.5 14.3 28.5 28.5 28.5 14.3 14.3 85.8 100 85.7 71.5
85.7 100 85.7 85.7 100 71.5 85.7 71.5 71.5 100 71.5 85.8 85.7 Percentage
(%)Good UsabilityLow Usability ((c)) Usability of Practices

Figure 3:  \- Collected Data from the Case Study Surveys

BP1 - Onboarding Xperts from Diverse Backgrounds; BP2 - Understanding the
Academic Environment; BP3 - Developing a Breadth of Skills for Effective
Handling of Projects; BP4 - Collaborative Assistance Between Xperts and Domain
scientists; BP5 - Overcoming the Terminology Gap; BP6 - Understanding the
Domain Problem; BP7 - Prioritize Functional Requirements; BP8 - Issue
Tracking; BP9 - Source Code Management; BP10 - Code Review; BP11 - Software
Testing; BP12 - Documentation; BP13 - Continuous Integration; BP14 -
Reproducibility; BP15 - Parallelization; BP16 - Maintainability &
Sustainability; BP17 - User Community Engagement and Exchange;

The following practices have low usability above the 15% average; each of
these practices reported low usability at 28.5%:

  * •

Practice 4 - Collaborative Assistance Between Xperts and Domain Scientists:
While short-term collaborations are adequate for many projects, Atom Project
participants have highlighted the limitations of this approach for the
project. They preferred long-term partnerships, diverging from the advice
provided by Xperts who reported effective short-term collaborations.

  * •

Practice 10 - Code Review: Participants expressed difficulties in integrating
code review into the development process, especially when it falls outside
their domain expertise. Understanding each other’s code can be challenging,
hindering the smooth integration of code review. However, gaining a good
understanding of each other’s code is crucial, especially when the project
will eventually be continued by the domain scientists once the collaboration
concludes.

  * •

Practice 12 - Documentation: The lower usability of this practice was linked
to the fact that, although it was relatively straightforward to keep developer
documentation up to date by adding comments within the code, maintaining
separate documentation tailored specifically for domain scientists, in this
case physicists, required additional effort. Ensuring that the documentation
met the needs and understanding of domain scientists demanded more resources
and attention.

  * •

Practice 13 - Continuous Integration: The low usability rate of this practice
within the project can be attributed to multiple factors, including the
challenges of developing comprehensive test cases, automating the testing
process, and setting up the necessary infrastructure, tools, and automated
workflows.

  * •

Practice 15 - Parallelization: The reported low usability, as noted by some
participants, is attributed to the requirement for specialized skills beyond
the expertise of domain scientists in effectively implementing parallelization
techniques in CDI projects.

###  4.3 Expert Reviews

This survey targeted CDI Xperts, possessing advanced knowledge, experience,
and practical insights in the field of CDI research. We selected these experts
from among the most active participants in the Xpert Network. The survey
included all 17 practices of Section 3. The survey began with a brief
introduction of each practice, followed by evaluations from the experts across
three metrics: (i) their experience level with the practice, (ii) the
practice’s realized impact, and (iii) usability, specifically focusing on ease
and efficiency of implementation in real-life scenarios. Respondents were
asked to rate each metric as either “Good” or “Low”. Additionally, they were
encouraged to share the methods and tools they used in applying the practices.
This information was presented in the tools paragraphs of Section 3.

#### Survey Data and Findings

Figure 4 on page 4 displays an average percentage rating of all practices for
each metric, depicted through pie charts. The sum of the “Good” rating in each
metric significantly outweighs the “Low” rating. This shows that the
participants possess substantial expertise in the subject matter, assess the
practices as having positive impacts, and find them usable and well-integrated
with the development process. A more detailed analysis of each metric follows.

((a)) Average Experience

((b)) Average Impact

((c)) Average Usability

Figure 4: \- Average Percentage Ratings for All Practices in the Expert Review
Survey by Metric Overview

#### Evaluation of Experience

BP1BP2BP3BP4BP5BP6BP7BP8BP9BP10BP11BP12BP13BP14BP15BP16BP17$0$$50$$100$ 14.2
21.4 7.1 7.1 7.1 7.1 14.2 21.4 14.2 21.4 14.2 35.7 7.1 14.2 21.4 14.2 100 85.8
78.6 92.9 92.9 92.9 92.9 85.8 78.6 85.8 78.6 85.8 64.3 92.9 85.8 78.6 85.8
Percentage (%)Good ExpLow Exp ((a)) Experience of Xperts with Practices

BP1BP2BP3BP4BP5BP6BP7BP8BP9BP10BP11BP12BP13BP14BP15BP16BP17$0$$50$$100$ 7.1
7.1 7.1 7.1 7.1 7.1 7.1 7.1 100 100 92.9 100 92.9 92.9 92.9 100 92.9 92.9 92.9
100 100 92.9 100 100 100 Percentage (%)Good ImpactLow Impact ((b)) Impact of
Best Practices

BP1BP2BP3BP4BP5BP6BP7BP8BP9BP10BP11BP12BP13BP14BP15BP16BP17$0$$50$$100$ 14.2
7.1 7.1 7.1 7.1 7.1 28.6 14.2 7.1 21.4 14.2 28.6 7.1 14.2 100 85.8 92.9 92.9
100 92.9 92.9 92.9 100 71.4 85.8 92.9 78.6 85.8 71.4 92.9 85.8 Percentage
(%)Good UsabilityLow Usability ((c)) Usability of Best Practices

Figure 5:  \- Collected Data from Expert Reviews Surveys

BP1 - Onboarding Xperts from Diverse Backgrounds; BP2 - Understanding the
Academic Environment; BP3 - Developing a Breadth of Skills for Effective
Handling of Projects; BP4 - Collaborative Assistance Between Xperts and Domain
scientists; BP5 - Overcoming the Terminology Gap; BP6 - Understanding the
Domain Problem; BP7 - Prioritize Functional Requirements; BP8 - Issue
Tracking; BP9 - Source Code Management; BP10 - Code Review; BP11 - Software
Testing; BP12 - Documentation; BP13 - Continuous Integration; BP14 -
Reproducibility; BP15 - Parallelization; BP16 - Maintainability &
Sustainability; BP17 - User Community Engagement and Exchange;

  * •

Figure 5(a) on page 5(a) shows that, overall, the respondents have a moderate
to high level of “Good” experience in applying the practices. Variations in
experience levels may stem from the experts’ diverse backgrounds, the range of
project goals and sizes, and the different stages at which these experts were
involved in their respective projects.

  * •

Figure 5(b) on page 5(b) shows a high positive impact, with “Good” exceeding
92% for each practice. This implies that, with effective implementation, these
practices hold the promise of delivering substantial positive impacts and
results.

  * •

Figure 5(c) on page 5(c) shows a positive usability evaluation, with “Good”
usability ratings consistently above 71%. Certain practices exhibit lower
usability due to factors such as resource limitations, and specific
application challenges.

The following practices had lower usability ratings. Contributory factors to
these ratings were derived from supplementary comments provided by the
participants:

  * •

Practice 15- Parallelization & Practice 10- Code Review: show considerable low
usability ratings (28.6%), due to the complexity and expertise required.

  * •

Practice 13- Continuous Integration: faces usability challenges (21.4% low
usability), possibly due to resource and training gaps.

  * •

Practice 11- Software Testing, Practice 14- Reproducibility & Practice 17-
User Community Engagement and Exchange: encounter usability obstacles (14.2%
low usability), linked to insufficient resources or time constraints,
insufficient guidance, and engagement difficulties.

  * •

Practice 2- Understanding the Academic Environment: also displays usability
concerns (14.2% low usability), reflecting the complexity of navigating
diverse academic settings.

These insights highlight areas for focused improvement to enhance the adoption
and effectiveness of the practices. Addressing the identified challenges,
through enhanced training, better resource allocation, and clearer guidance,
can significantly improve usability and overall impact. Further strategies to
address these challenges are discussed in Section 3.

###  4.4 Comparing the Results of the Three Surveys

#### Impact Assessment

Figure 6(a) on page 6(a) compares the three surveys, showing the average of
“Good Impact”, “Low Impact”, and “No Experience” for all practices. “No
Experience” is reported for practices that researchers have not applied in
their projects. While participants in the case study had experience
implementing the practices, and participants in the expert review, who are
computational experts, apply the practices daily, none reported having no
experience with the practices. By contrast, the researcher survey covers a
broader community, including respondents with limited experience in these
practices.

All surveys consistently show high percentages of average “Good Impact” for
the practices. only one practice – BP10: Code Review – shows lower impact
across all surveys, as shown in Figure 1 on page 1, Figure 3(b) on page 3(b),
and Figure 5(b) on page 5(b). One reason for this result is a lack of clear
instructions on what constitutes good code and how code reviews should be
conducted.

$0$$10$$20$$30$$40$$50$$60$$70$$80$$90$$100$ Researcher Survey Expert Review
Survey Case Study Survey $24.2$% $2.7$% $3.4$% $0.8$% $73$% $96.6$% $99.2$%
PercentageSurveys on ImpactGood ImpactLow ImpactNo Experience ((a)) Comparing
the Averages of Good and Low Impact of Practices: Expert Review, Case Study,
and Researcher Survey

$0$$10$$20$$30$$40$$50$$60$$70$$80$$90$$100$ Expert Review Survey Case Study
Survey $10.9$% $15.1$% $89.1$% $84.9$% PercentageSurveys on UsabilityGood
UsabilityLow Usability ((b)) Comparing the Averages of Good and Low Usability
of Practices: Expert Review Survey, and Case Study Survey

Figure 6:  \- Comparing the Averages of Impact and Usability of Practices

#### Usability Evaluation

Usability assessments were conducted via the expert review and case study
surveys (the researcher survey omitted this question for brevity, aiming to
increase the response rates.) Figure 6(b) shows the average reported “Good
Usability” and “Low Usability”. There is high overall “Good Usability” across
practices, with a slightly higher value in the expert review survey. This
result is likely due to the participants’ expertise, which facilitates the
smoother integration of practices into their development workflows.

In Figure 3(c) on page 3(c) and Figure 5(c) on page 5(c), three practices show
lower usability in both surveys, despite their significant impact: BP10 - Code
Review, BP13 - Continuous Integration, and BP15 - Parallelization. We
attribute these results to the following reasons:

  * •

Code Review Training: Users and limited training on effective code review
strategies, including what constitutes high-quality code and how to identify
areas for improvement.

  * •

Continuous Integration Tools: While there are many tools supporting the
practice, there is a lack of overall environment, including for test case
generation.

  * •

Parallelization Knowledge: Parallelization skills are complex and often
require extensive training.

Despite the participants’ diverse experiences and perspectives, Figure 6 shows
consistent positive impact and usability of the practices.

##  5 Related work

General research software engineering efforts: A multitude of initiatives and
organizations have contributed to advancing software engineering in general,
which include the development of best practices:
Software Carpentry 80 is a widely recognized non-profit organization that
offers workshops and online resources to teach researchers foundational
software development skills. Their focus is on providing practical training on
tools, programming languages, and software development practices relevant to
scientific application development. Software Carpentry emphasizes
reproducibility, version control, data management, and automated testing as
key aspects of scientific software development.
The Turing Way 112 is an open-source community-driven project that provides a
handbook on best practices in reproducible data science. It offers practical
guidance and resources for researchers and software developers covering
reproducible research. The Turing Way places a strong emphasis on open
science, inclusivity, and fostering a collaborative research culture.
Better Scientific Software (BSSw) 8 is an initiative aimed at improving the
quality and sustainability of scientific software. BSSw focuses on fostering
community-driven discussions, sharing expertise, and advocating for better
software practices. Their community-driven articles encompass various
subjects, such as software engineering techniques, performance optimization,
software licensing, and software citation.
The Research Software Engineering (RSE) community 113 brings together
professionals who are specialized in developing software for research
purposes. This community focuses on advocating for the recognition and career
development of research software engineers, sharing expertise, and promoting
best practices in research software development. RSE publications often
discuss topics such as software sustainability, collaborative development, and
the role of research software engineers in interdisciplinary research.
ACCESS – Advanced Cyberinfrastructure Coordination Ecosystem: Services &
Support 10 Builds upon the foundation laid by XSEDE 114. It focuses on
providing researchers with advanced cyberinfrastructure resources, services,
and support to enhance the capabilities of scientific communities. Through
ACCESS, researchers gain access to high-performance computing, data
management, visualization tools, and networking infrastructure. ACCESS
actively collaborates with researchers across diverse domains to understand
their computational needs, enabling the development and deployment of
customized cyberinfrastructure solutions. By bridging the gap between
researchers and advanced computing resources, ACCESS empowers scientific
communities to engage in large-scale simulations, data analysis, and modeling,
leading to accelerated scientific discovery and fostering innovation.
The Molecular Sciences Software Institute (MolSSI) 12 tackles software
challenges in molecular sciences, offering resources and training to improve
scientific workflows’ productivity and reproducibility. It focuses on
advancing the development, maintenance, and sustainability of tools in
molecular simulations and quantum chemistry. By offering resources, training,
and community support, MolSII fosters best practices in software engineering,
data management, and software citation, thus improving the robustness and
impact of molecular sciences research.
While these efforts share the common goal of improving scientific software
development practices, each has its unique focus and approach. Software
Carpentry and The Turing Way concentrate on providing practical training and
resources for researchers to acquire essential software development skills and
adopt best practices. Better Scientific Software focuses on community
engagement, sharing expertise, and advocating for better software practices in
the scientific research community. The RSE community focuses on
professionalizing the role of research software engineers and promoting
collaboration between disciplines. ACCESS provides cyberinfrastructure
resources, services, and support for researchers conducting CDI research.
MolSSI addresses software challenges in molecular sciences, offering resources
and training to enhance productivity and reproducibility. By contrast, this
paper, while collaborating with these initiatives, aims to develop a
comprehensive set of guidelines tailored for CDI application development. The
guidelines include strategies for implementation, supportive tools, and
insights into overcoming potential challenges, evaluated by a diverse pool of
participants.
Specific contributions towards best practices: Wilson et al. 115 address
common challenges in scientific computing and propose eight best practices to
improve the reliability of scientific software. They emphasize version
control, testing, documentation, pair programming, and code review. The same
authors introduce simple and practical steps that can improve the efficiency
and reliability of scientific software 116. The steps represent a minimum set
of tools and techniques that researchers should consider adopting. Both
contributions are based on the collective experience of researchers involved
in Software Carpentry and Data Carpentry. There has been no evaluation of the
recommended practices.

Heroux et al. propose ten practices to improve computational science and
engineering (CSE) software 117. They address the challenges and constraints
faced by CSE software developers and aim to enhance the software quality,
reliability, and maintainability. The practices cover requirements gathering,
version control, testing, documentation, and collaborative development. They
were derived from the Trilinos project 118. There is also no evaluation of the
practices provided.

Our contribution stands out from other initiatives in that it delivers
guidelines specifically crafted for Xperts who assist domain scientists in
their Computational and Data Intensive (CDI) research projects. In contrast to
the above-mentioned efforts, which concentrate on providing guidelines for CDI
application developers, our approach highlights the importance of forming
collaboration between domain scientists and Xperts, which is similar to the
approach taken by the US-RSE 9 organization. Our study includes an evaluation
of the practices, which is absent in all prior contributions.

Our work also discusses challenges, limitations, and implementation methods.
Furthermore, we mention tools and resources that support practices. By
offering a resource that addresses both technical and practical aspects, our
aim is to empower Xperts to effectively navigate the complexities of CDI
applications.

##  6 Conclusions and Future Directions

Through the collective efforts of the Xpert Network and the Atom project, we
have identified and evaluated a set of best practices for developing
computational and data-intensive (CDI) applications. These guidelines aim to
assist CDI support professionals, referred to as Xperts or Research Software
Engineers, but also serve computational scientists in general. We have
examined the impacts, limitations, and challenges associated with these
practices, as well as recommended tools that support their effective
application.

We have evaluated the practices through three distinct methods tailored to
participants with varying levels of experience and perspectives. The results
confirm both high impact and usability in CDI application development
projects. A few of the practices are less familiar to general users; we have
discussed methods to increase their adoption.

We consider this paper a live document that needs to evolve together with new
technology and CDI applications. By continuing to engage a large user
community to refine and extend the practices, we aim to maintain a useful
guide that can be put in the hands of Xperts assisting CDI domain researchers
and, in this way, help push the frontiers of computational and data science.

## Acknowledgments

This work was supported in part by the National Science Foundation under
Awards No. OAC-1931339 and OAC-2209639.

## References

  * 1 Barakhshan, Parinaz and Eigenmann, Rudolf  “The Xpert Network: Exchanging Best Practices and Tools for Computational and Data-intensive Research”, https://sites.udel.edu/xpert-cdi/
  * 2 Parinaz Barakhshan and Rudolf Eigenmann  “Exchanging Best Practices for Supporting Computational and Data-Intensive Research, The Xpert Network”  In _Practice and Experience in Advanced Research Computing_ , PEARC ’22  Boston, MA, USA: Association for Computing Machinery, 2022  DOI: https://dx.doi.org/10.1145/3491418.353029310.1145/3491418.3530293
  * 3 Parinaz Barakhshan and Rudolf Eigenmann  “The Xpert Network, Workshop on best practices and tools for Computational and data, In conjunction with the International Conference on Supercomputing (ICS 2019)” https://doi.org/10.5281/zenodo.8428055, 2019
  * 4 Barakhshan, Parinaz and Marrs, Adam and Arora, Bindiya and Eigenmann, Rudolf and Safronova, Marianna S  “Portal for High-Precision Atomic Data and Computation”, https://www1.udel.edu/atom/
  * 5 Parinaz Barakhshan and Akshay Bhosale and Amani Kiruga and Rudolf Eigenmann and Marianna S. Safronova and Bindiya Arora  “A Portal for High-Precision Atomic Data and Computation: Design and Best Practices”, 2022  arXiv:https://arxiv.org/abs/2212.106652212.10665 [physics.atom-ph]
  * 6 Barakhshan, Parinaz and Marrs, Adam and Arora, Bindiya and Eigenmann, Rudolf and Safronova, Marianna S  “Portal for high-precision atomic data and computation”  In _Science Gateways_ 2021, 2021  DOI: https://dx.doi.org/10.5281/zenodo.557061410.5281/zenodo.5570614
  * 7 Parinaz Barakhshan and Rudolf Eigenmann and Bindiya Arora and Marianna S. Safronova  “Application of Software Engineering in Building the Portal for High-Precision Atomic Data and Computation”  In _Science Gateways_ , 2022  DOI: https://dx.doi.org/10.5281/zenodo.708947710.5281/zenodo.7089477
  * 8 “Better Scientific Software (BSSw)”, https://bssw.io/
  * 9 US-RSE  “US Research Software Engineer Association (US-RSE)”, https://us-rse.org/
  * 10 Manish Parashar  “Democratizing Science Through Advanced Cyberinfrastructure”  In _Computer_ 55.9, 2022, pp. 79–84  DOI: https://dx.doi.org/10.1109/MC.2022.317492810.1109/MC.2022.3174928
  * 11 ACCESS  “Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support”, https://access-ci.org/
  * 12 Molecular Sciences Software Institute  “MOLSII (Molecular Sciences Software Institute)”, https://molssi.org/
  * 13 Henry Neeman, Marisa Brazil and Dana Brunson  “The Virtual Residency: A Training Program for Research Computing Facilitators”, https://bpb-us-w2.wpmucdn.com/sites.udel.edu/dist/6/8980/files/2019/03/The-Virtual-Residency-A-Training-Program-for-Research-Computing-Facilitators.pdf, 2019
  * 14 Jeremy Cohen et al.  “The Four Pillars of Research Software Engineering”  In _IEEE Software_ 38.1, 2021, pp. 97–105  DOI: https://dx.doi.org/10.1109/MS.2020.297336210.1109/MS.2020.2973362
  * 15 Parinaz Barakhshan and Rudolf Eigenmann  “Exchanging Best Practices and Tools for Supporting Computational and Data-Intensive Research, The Xpert Network”, 2021  arXiv:https://arxiv.org/abs/2102.093732102.09373 [cs.CY]
  * 16 “CyVerse - The Open Science Workspace for Collaborative Data-driven Discovery ”, https://cyverse.org/
  * 17 “Participant Organizations in the Xpert Network for Computational and Data-Intensive Research” 2024, https://sites.udel.edu/xpert-cdi/participantorganizations/
  * 18 “Slack”, https://slack.com
  * 19 “Zoom”, https://zoom.us
  * 20 “Microsoft Teams”, https://www.microsoft.com/en-us/microsoft-365/microsoft-teams/group-chat-software
  * 21 “Google Workspace”, https://workspace.google.com
  * 22 “LinkedIn”, https://www.linkedin.com
  * 23 “twitter”, https://twitter.com/
  * 24 “Connect.Cybinfrastructure (Connect.CI)”, https://cnct.ci/
  * 25 Wilkins-Diehr, Nancy and Sanielevici, Sergiu and Alameda, Jay and Cazes, John and Crosby, Lonnie and Pierce, Marlon and Roskies, Ralph  “An overview of the XSEDE extended collaborative support program”  In _High Performance Computer Applications: 6th International Conference, ISUM 2015, Mexico City, Mexico, March 9-13, 2015, Revised Selected Papers 6_ , 2016, pp. 3–13  Springer
  * 26 John Towns et al.  “XSEDE: Accelerating Scientific Discovery”  In _Computing in Science & Engineering_ 16.5, 2014, pp. 62–74  DOI: https://dx.doi.org/10.1109/MCSE.2014.8010.1109/MCSE.2014.80
  * 27 Melissa Romanus et al.  “The anatomy of successful ECSS projects: Lessons of supporting high-throughput high-performance ensembles on XSEDE”  In _Proceedings of the 1st Conference of the Extreme Science and Engineering Discovery Environment: Bridging from the eXtreme to the campus and beyond_ , 2012, pp. 1–9
  * 28 “GitHub Pages, Websites for you and your projects”, https://pages.github.com/
  * 29 “Stack Overflow: Every developer has a tab open to Stack Overflow”, https://stackoverflow.com
  * 30 “Microsoft Project”, https://www.microsoft.com/en-us/microsoft-365/project/project-management-software
  * 31 “Asana”, https://asana.com
  * 32 “Trello”, https://trello.com
  * 33 “Jira”, https://www.atlassian.com/software/jira
  * 34 “Kanban”, https://kanbanize.com/kanban-resources/getting-started/what-is-kanban
  * 35 “GitHub Issues”, https://docs.github.com/en/issues
  * 36 “Bugzilla”, https://www.bugzilla.org
  * 37 Roger Pressman  “Software Engineering: A Practitioner’s Approach”  USA: McGraw-Hill, Inc., 2007
  * 38 Fowler, Martin and Highsmith, Jim  “The agile manifesto”  In _Software development_ 9.8 [San Francisco, CA: Miller Freeman, Inc., 1993-, 2001, pp. 28–35
  * 39 “Apache Subversion (SVN)”, https://subversion.apache.org
  * 40 “Perforce”, https://www.perforce.com
  * 41 “Git”, https://git-scm.com
  * 42 “Mercurial”, https://www.mercurial-scm.org
  * 43 “GitHub Pull Requests”, https://docs.github.com/en/pull-requests
  * 44 “Gerrit”, https://www.gerritcodereview.com
  * 45 Kent Beck  “Test driven development: By example”  Addison-Wesley Professional, 2022
  * 46 Dave Astels  “Test driven development: A practical guide”  Prentice Hall Professional Technical Reference, 2003
  * 47 “JUnit”, https://junit.org/junit5
  * 48 “PyTest”, https://docs.pytest.org
  * 49 “Selenium”, https://www.selenium.dev
  * 50 “Appium”, https://appium.io
  * 51 “Cypress”, https://www.cypress.io
  * 52 “Mockito”, https://site.mockito.org
  * 53 “Sinon”, https://sinonjs.org
  * 54 “PowerMock”, https://powermock.github.io
  * 55 “Faker”, https://faker.readthedocs.io
  * 56 “DataFactory”, https://github.com/marak/Faker.js/
  * 57 “SQL Data Generator”, https://www.red-gate.com/products/sql-development/sql-data-generator/
  * 58 “Jenkins”, https://www.jenkins.io
  * 59 “Travis CI”, https://www.travis-ci.com
  * 60 “CircleCI”, https://circleci.com
  * 61 “Doxygen”, https://www.doxygen.nl
  * 62 “Javadoc”, https://docs.oracle.com/javase/8/docs/technotes/tools/windows/javadoc.html
  * 63 “Sphinx”, https://www.sphinx-doc.org
  * 64 “MkDocs”, https://www.mkdocs.org
  * 65 Mojtaba Shahin, Muhammad Ali Babar and Liming Zhu  “Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices”  In _IEEE Access_ 5, 2017, pp. 3909–3943  DOI: https://dx.doi.org/10.1109/ACCESS.2017.268562910.1109/ACCESS.2017.2685629
  * 66 Keheliya Gallaba  “Improving the Robustness and Efficiency of Continuous Integration and Deployment”  In _2019 IEEE International Conference on Software Maintenance and Evolution (ICSME)_ , 2019, pp. 619–623  DOI: https://dx.doi.org/10.1109/ICSME.2019.0009910.1109/ICSME.2019.00099
  * 67 “GitLab CI/CD”, https://docs.gitlab.com/ee/ci/
  * 68 “Bamboo”, https://www.atlassian.com/software/bamboo
  * 69 “TeamCity”, https://www.jetbrains.com/teamcity
  * 70 “GitHub Actions”, https://github.com/features/actions
  * 71 “Microsoft Azure DevOps”, https://azure.microsoft.com/en-us/services/devops/
  * 72 Wilkinson, Mark D and Dumontier, Michel and Aalbersberg, IJsbrand Jan and Appleton, Gabrielle and Axton, Myles and Baak, Arie and Blomberg, Niklas and Boiten, Jan-Willem and da Silva Santos, Luiz Bonino and Bourne, Philip E and others  “The FAIR Guiding Principles for scientific data management and stewardship”  In _Scientific data_ 3.1 Nature Publishing Group, 2016, pp. 1–9
  * 73 Engineering National Academies of Sciences and Medicine  “Reproducibility and replicability in science”  National Academies Press, 2019
  * 74 “Docker”, https://www.docker.com
  * 75 “Singularity”, https://www.sylabs.io/singularity
  * 76 “OSF”, https://osf.io/
  * 77 “Zenodo”, https://zenodo.org
  * 78 “Figshare”, https://figshare.com
  * 79 “CODE OCEAN”, https://codeocean.com/
  * 80 Greg Wilson  “Software carpentry: getting scientists to write better code by making them more productive”  In _Computing in science & engineering_ 8.6 IEEE, 2006, pp. 66–69
  * 81 “OpenMP”, https://www.openmp.org
  * 82 “MPI”, https://www.mpi-forum.org
  * 83 “CUDA”, https://developer.nvidia.com/cuda-zone
  * 84 Rob Farber  “Parallel programming with OpenACC”  Newnes, 2016
  * 85 “GNU gprof”, https://sourceware.org/binutils/docs/gprof/
  * 86 “Scalasca”, https://www.scalasca.org
  * 87 “HPCToolkit”, https://hpctoolkit.org
  * 88 “Allinea DDT”, https://www.allinea.com/products/ddt
  * 89 “TotalView”, https://www.roguewave.com/products-services/totalview
  * 90 “Intel VTune”, https://software.intel.com/content/www/us/en/develop/tools/vtune-profiler.html
  * 91 “TAU”, https://www.cs.uoregon.edu/research/tau
  * 92 “NVIDIA Nsight”, https://developer.nvidia.com/nsight
  * 93 Tian, Xinmin and Bik, Aart and Girkar, Milind and Grey, Paul and Saito, Hideki and Su, Ernesto  “Intel® OpenMP C++/Fortran Compiler for Hyper-Threading Technology: Implementation and Performance.”  In _Intel Technology Journal_ 6.1, 2002
  * 94 “Rose”, https://rosecompiler.org
  * 95 “Cetus, A Parallelizing Source-to-Source Compiler for C Programs”, https://sites.udel.edu/cetus-cid/
  * 96 Amini, Mehdi and Creusillet, Béatrice and Even, Stéphanie and Keryell, Ronan and Goubier, Onig and Guelton, Serge and McMahon, Janice Onanian and Pasquier, François-Xavier and Péan, Grégoire and Villalon, Pierre  “Par4all: From convex array regions to heterogeneous computing”  In _IMPACT 2012: Second International Workshop on Polyhedral Compilation Techniques HiPEAC 2012_ , 2012
  * 97 Bondhugula, Uday and Hartono, Albert and Ramanujam, J and Sadayappan, P  “Pluto: A practical and fully automatic polyhedral program optimization system”  In _Proceedings of the ACM SIGPLAN 2008 Conference on Programming Language Design and Implementation (PLDI 08), Tucson, AZ (June 2008)_ , 2008  Citeseer
  * 98 Grosser, Tobias and Zheng, Hongbin and Aloor, Raghesh and Simbürger, Andreas and Größlinger, Armin and Pouchet, Louis-Noël  “Polly-Polyhedral optimization in LLVM”  In _Proceedings of the First International Workshop on Polyhedral Compilation Techniques (IMPACT)_ 2011, 2011, pp. 1
  * 99 Johnson, Stephen and Evans, Emyr and Jin, Haoqiang and Ierotheou, Constantinos  “The parawise expert assistant–widening accessibility to efficient and scalable tool generated openmp code”  In _Shared Memory Parallel Programming with Open MP: 5th International Workshop on Open MP Applications and Tools, WOMPAT 2004, Houston, TX, USA, May 17-18, 2004, Revised Selected Papers 5_ , 2005, pp. 67–82  Springer
  * 100 Ilona Bluemke and Joanna Fugas  “C code parallelization with paragraph”  In _2010 2nd International Conference on Information Technology,(2010 ICIT)_ , 2010, pp. 163–166  IEEE
  * 101 Wilson, Robert P and French, Robert S and Wilson, Christopher S and Amarasinghe, Saman P and Anderson, Jennifer M and Tjiang, Steve WK and Liao, Shih-Wei and Tseng, Chau-Wen and Hall, Mary W and Lam, Monica S and others  “SUIF: An infrastructure for research on parallelizing and optimizing compilers”  In _ACM Sigplan Notices_ 29.12 ACM New York, NY, USA, 1994, pp. 31–37
  * 102 Blume, William and Eigenmann, Rudolf and Faigin, Keith and Grout, John and Hoeflinger, Jay and Padua, David and Petersen, Paul and Pottenger, William and Rauchwerger, Lawrence and Tu, Peng and others  “Polaris: Improving the effectiveness of parallelizing compilers”  In _Languages and Compilers for Parallel Computing: 7th International Workshop Ithaca, NY, USA, August 8–10, 1994 Proceedings 7_ , 1995, pp. 141–154  Springer
  * 103 “iCetus, A Source-to-Source Compiler Infrastructure for C Programs”, http://icetus.ece.udel.edu/cetusWeb/
  * 104 Liao, Shih-Wei and Diwan, Amer and Bosch Jr, Robert P and Ghuloum, Anwar and Lam, Monica S  “SUIF explorer: An interactive and interprocedural parallelizer”  In _Proceedings of the seventh ACM SIGPLAN symposium on Principles and practice of parallel programming_ , 1999, pp. 37–48
  * 105 Varun Mishra and Sanjeev K Aggarwal  “ParTool: a feedback-directed parallelizer”  In _Advanced Parallel Processing Technologies: 9th International Symposium, APPT 2011, Shanghai, China, September 26-27, 2011. Proceedings 9_ , 2011, pp. 157–171  Springer
  * 106 IBM  “IBM Software Engineering for Sustainability”, https://www.ibm.com/design/practices/engineering-for-sustainability/IBM-engineering-for-sustainability.pdf, 2023
  * 107 Venters, Colin C and Capilla, Rafael and Betz, Stefanie and Penzenstadler, Birgit and Crick, Tom and Crouch, Steve and Nakagawa, Elisa Yumi and Becker, Christoph and Carrillo, Carlos  “Software sustainability: Research and practice from a software architecture viewpoint”  In _Journal of Systems and Software_ 138 Elsevier, 2018, pp. 174–188
  * 108 “kubernetes”, https://kubernetes.io/
  * 109 “UserVoice”, https://www.uservoice.com
  * 110 “SurveyMonkey”, https://www.surveymonkey.com
  * 111 “Google Forms”, https://www.google.com/forms/about
  * 112 Becky Arnold et al.  “The turing Way: a handbook for reproducible data science”  In _Zenodo_ , 2019  DOI: https://dx.doi.org/https://zenodo.org/records/3233986https://zenodo.org/records/3233986
  * 113 Cohen, Jeremy and Katz, Daniel S and Barker, Michelle and Hong, Neil Chue and Haines, Robert and Jay, Caroline  “The four pillars of research software engineering”  In _IEEE Software_ 38.1 IEEE, 2020, pp. 97–105
  * 114 Towns, John and Cockerill, Timothy and Dahan, Maytal and Foster, Ian and Gaither, Kelly and Grimshaw, Andrew and Hazlewood, Victor and Lathrop, Scott and Lifka, Dave and Peterson, Gregory D and others  “XSEDE: accelerating scientific discovery”  In _Computing in science & engineering_ 16.5 IEEE, 2014, pp. 62–74
  * 115 Wilson, Greg and Aruliah, Dhavide A and Brown, C Titus and Chue Hong, Neil P and Davis, Matt and Guy, Richard T and Haddock, Steven HD and Huff, Kathryn D and Mitchell, Ian M and Plumbley, Mark D and others  “Best practices for scientific computing”  In _PLoS biology_ 12.1 Public Library of Science San Francisco, USA, 2014, pp. e1001745
  * 116 Wilson, Greg and Bryan, Jennifer and Cranston, Karen and Kitzes, Justin and Nederbragt, Lex and Teal, Tracy K  “Good enough practices in scientific computing”  In _PLoS computational biology_ 13.6 Public Library of Science, 2017, pp. e1005510
  * 117 Michael A. Heroux and James M. Willenbring  “Barely sufficient software engineering: 10 practices to improve your CSE software”  In _2009 ICSE Workshop on Software Engineering for Computational Science and Engineering_ , 2009, pp. 15–21  DOI: https://dx.doi.org/10.1109/SECSE.2009.506915710.1109/SECSE.2009.5069157
  * 118 “Center for Computing Research (CCR)”, https://www.sandia.gov/ccr/software/trilinos/

## Author Biography

Parinaz Barakhshan. is a Ph.D. candidate and Research Assistant at the
University of Delaware, Department of Electrical and Computer Engineering. Her
research interests include optimizing compilers and performance evaluation for
high-performance computing systems. Contact her at<EMAIL_ADDRESS>

Rudolf Eigenmann. is a Professor at the Department of Electrical and Computer
Engineering, University of Delaware, USA. His research interests include
optimizing compilers, programming methodologies, tools, and performance
evaluation for high-performance computing, as well as the design of
cyberinfrastructure. Contact him at<EMAIL_ADDRESS>