# What do you need to consider when designing mobile health intervention?

Aneta Lisowska  Szymon Wilk  Mor Peleg

###### Abstract

Designing theory-driven digital health interventions is a challenging task
that needs support. We created a guide for the incomers in the field on how to
design digital health interventions with case studies from the Cancer Better
Life Experience (CAPABLE) European project. The guide explains how behaviour
change theories can inform customisation and personalisation of the
intervention. The proposed SATO (ideaS expAnded wiTh bciO) design workflow is
based on the IDEAS (Integrate, Design, Assess, and Share) framework and is
aligned with the Behaviour Change Intervention Ontology (BCIO). We provide a
checklist of the activities that should be performed during intervention
planning as well as app design templates which bundle together relevant
behaviour change techniques. In the process of creating this guide, we found
the necessity to extend the BCIO to support the scenarios of multiple clinical
goals in the same application. The extension utilizes existing classes and
properties where possible.

###### keywords:

digital behavior change intervention (DBCI); m-health; personalization;
machine learning; wellbeing; cancer

††journal: Journal of Biomedical Informatics

[inst1]organization=Sano Centre for Computational Medicine,city=Cracow,
country=Poland

[inst2]organization=Institute of Computing Science, Poznan University of
Technology,city=Poznan, country=Poland

[inst3]organization=Department of Information Systems, University of
Haifa,city=Haifa, country=Israel

## Statement of significance

Problem or issue

Design of digital health behaviour change interventions is a complex process,
yet there are few published methodologies that offer a theory-based solution
that delves into the practical level of app design.

What is already known

Murray et al. proposed the IDEAS (Integrate, Design, Assess, and Share)
framework to support the development life-cycle of effective digital health
interventions.

Michie et al. developed classifications of behavioural change techniques (BCT)
and the Behaviour Change Intervention Ontology (BCIO) to facilitate knowledge
sharing and enable discovery and validation of links between successful
behavioural change and particular behavioural techniques.

What this paper adds

We propose SATO (ideaS expAnded wiTh bciO) a digital health intervention
design workflow aligned with the BCIO and the IDEAS frameworks. We extend BCIO
to support the scenarios of multiple behaviour change interventions (BCIs)
addressing several clinical goals and extend IDEAS through integrating BCIO.
Moreover, we provide a concrete example on how to build a goal hierarchy and
choose evaluation measures at each level, to enable evaluation of the BCIs’
effectiveness. We also prepare reusable design templates bundling together
several behaviour change techniques and demonstrate how behaviour change
theories can drive customisation and personalisation of the intervention.

##  1 Introduction

The amount of literature on digital health interventions (DHI)[1] is vast [2,
3], with studies aiming to improve mental [4] and physical health [5].
Interventions cover applications from different stages of the health
management cycle and target populations across different age, social status
and cultures. Some DHIs aim to support patients with adherence to
pharmacological treatment, dose management and side effects reporting. Other
DHIs aid patients with modulation of health risk behaviours such as:
inactivity, poor nutrition, or substance abuse. The design of the latter is
the focus of this paper.

Many of the industry-based DHIs do not incorporate “theory-based strategies
known to drive changes in health behaviours or undergo systematic testing to
demonstrate their effectiveness” [6]. Some researchers, including our group,
are trying to develop structured theory-based methodologies for supporting the
development of effective DHIs. IDEAS (Integrate, Design, Assess, and Share)
[6] is a framework and toolkit of strategies for the development of more
effective digital interventions to change health behavior, integrating methods
from behavioral theory, design thinking, and intervention evaluation and
dissemination. Murray et al. draw from biomedical, behavioral, computing, and
engineering research methods to develop an evaluation approach for DHIs [7].
They define a set of key research questions that should form the basis for an
appraisal of a DHI, focusing on the target population and health need, and on
factors impacting the likelihood of benefit from the DHI, considering
causality, potential adverse effects, tailoring to participants, and
comparison to alternative interventions.

Another source of inspiration is from Michie and colleagues. This group has
been working for many years on creating a standardization of the behavior
change domain; much research has been done in this domain and many behavior
change interventions (BCI) have been developed. But evaluation of BCIs and
comparison of theories, studies, and trials that incorporate behavior change
is not feasible without a standard vocabulary and ontology. Such a structure
that organizes the basic techniques that are implemented in BCIs is needed in
order to attribute the success of behavioural change to particular behavioural
techniques. Michie’s group developed classifications of behavioural change
techniques (BCT)[8] and the Behaviour Change Intervention Ontology (BCIO) [9].

This paper intends to provide a detailed actionable systematic guide for DHI
development drawing from IDEAS [6] and in alignment with the BCIO[9] and
behaviour change techniques[8]. The SATO (ideaS expAnded wiTh bciO) guide
includes a design workflow and checklist (See Figure 1). We provide examples
taken from the Horizon 2020 project Cancer Patient Better Life Experience
(CAPABLE, https://capable-project.eu/) [10], where we follow a multi-
stakeholder and evidence-based iterative development cycle for a DHI, in teams
of informaticians, clinicians, patients, and engineers, and also utilize
feasibility studies on synthetic and public datasets. The main contributions
of this research are: 1) application and extension of the BCIO to multi-BCIs
mHealth applications that span several clinical goals, 2) BCI design guide and
checklist validated with examples from the CAPABLE project, and 3) reusable
design templates bundling together several behaviour change techniques.

##  2 Related Work

We review the IDEAS framework and Behaviour Change Intervention (BCI) Ontology
upon which we base our proposed DBCI development guide.

###  2.1 IDEAS

IDEAS [6] provides a disciplined way to incrementally translate behavioural
theories into highly relevant and practical interventions. Its ten steps are
organized into a four-phase process: Integrate phase, including (1) empathize
with target users, (2) specify target behaviour, (3) ground in behavioural
theory; DEsign phase, including (4) ideate implementation strategies, (5)
prototype potential products, (6) gather user feedback, (7) build a minimum
viable product; Assess phase, including (8) pilot test to assess potential
efficacy and usability, and (9) evaluation of the efficacy in an RCT; and
Share phase, with (10) share intervention and findings.

We previously extended IDEAS with an ontology [11] that structures the target
behaviour change intervention as a class derived from HL7 Fast Healthcare
Interoperability Resources (FHIR) standard [12] and demonstrated its
application to a case study taken from the CAPABLE project, that uses Fogg’s
Tiny Habits behavioural model [13] to improve the sleep of cancer patients via
Tai Chi. In another work [14], we extended IDEAS Ideate step from Design phase
by providing concrete backend architectural components and graphical user-
interface designs that implement behavioural interventions.

###  2.2 Behaviour Change Intervention Ontology

Ontology is an organization and representation of the entities in a domain
according to their properties and relations to one another [15]. It provides
shared and agreed upon definitions and descriptions of the important concepts
in a domain of interests [15]. Michie et al. [9] designed the Behaviour Change
Intervention Ontology (BCIO) following the principles of the Open Biological
and Biomedical Ontology (OBO) Foundry (https://obofoundry.org) by extending
the Basic Formal Ontology[16]. There are six main classes in the BCIO. The
first five are parts of a Behaviour Change Intervention (BCI) Scenario, in
which a (Behaviour Change) Intervention, developed for a Context (ie, target
population and setting), is exposed to the population via an Exposure, and via
a Mechanism of Action yields an Outcome Behaviour, which is the intended new
behaviour that should form a habit. The Scenario’s Outcome Behaviour –the
sixth entity– can be estimated in a clinical study. As an example for a BCI
Scenario, consider a BCI of Tai Chi practice (Tai Chi Capsule), developed for
the Context of fatigued cancer patients at home (see Table LABEL:tab:bci-def).
The BCI/Capsule is exposed by the introduction of an app supporting the BCI
that is advocated by the patients’ physician. The BCI’s Content implements a
Behaviour Change Technique (BCT) [8] of demonstrating the behavior. Observing
the video in which a person performs Tai Chi makes the target Outcome
Behaviour clear and increases the patient’s ability to perform it (hence this
is the Mechanism of Action). In a BCI Evaluation Study of a BCI scenario (see
the end of Table LABEL:tab:bci-def), we use an Evaluation Finding to measure
the BCI Scenario’s Outcome Behaviour as well as the Intervention outcome. An
Intervention outcome is a process that is influenced by an intervention,
including mental activity and physiological activity (eg., sleep, stress,
fatigue) and Outcome Behaviour is an Individual Human Behaviour (eg,
performing Tai Chi). The Evaluation Finding objects used to measure these
outcomes include a decrease in the reported Fatigue Severity Scale (FSS), a
log of the number of times that a patient watched and rated Tai Chi videos,
and the stress level and the number of hours slept at night, as detected from
a Smartwatch. The evaluation findings at the level of BCI scenario, such as
FSS, change slowly – therefore we treat them as lag measures and check rather
infrequently. FSS measures/estimates111We use the terms ”measure” and
”estimate” interchangeably; ”estimate” to be consistent with BCIO [9] and
”measure” to be aligned with the 4DX terminology [17] a patient’s fatigue
(current state). The change in the fatigue rating is the intervention outcome
estimate/lag measure. The number of times a patient played a Tai Chi video
reflects the process of engaging with the intervention and could be a lead
measure for intervention outcome. Lead measures track the critical activities
that drive or lead to the lag (outcome) measure [17].

Table 1: BCIO examples for the Fatigue Reduction Scenario Class with properties | Individual
---|---
BCI_Scenario | Fatigue Reduction Scenario
has_BCI_context [BCI_Context] |  Cancer patient with fatigue problems at home
has_occurant_part [BCI] |  {Tai_Chi_Capsule, Walk_Capsule, Deep_Breathing_Capsule, Imagery_Training_Capsule}
|  *has_goal [Clinical_Goal - information content entity]
---
Less fatigue (as measured by Fatigue Severity Scale <36)
*has_occurant_part [Intervention Outcome] | Change in reported Fatigue Severity Scale
BCI | Tai_Chi_Capsule
has_occurent_part [BCI_Content] |  Tai_Chi_Capsule_Content
has_occurant_part [BCI_Delivery] |  Tai_Chi_Delivery
|  *has_BCI_Source [BCI_Source]
---
Source: Song et al. [18]
*has_occurant_part [Outcome Behaviour] | Performing Tai Chi regularly
*has_occurant_part [BCI_Mechanism_of_Action] |  | Performing Tai Chi habitually relaxes the mind and body
---
and allows the person to fall asleep more rapidly and to
stay sleeping for longer stretches of time
a) |
BCI_Content | Habit_planning_for_Tai_Chi_Content
has_profile [BCI_Dose] |  1/week (review)
has_occurant_part [BC_Technique] |  {[BCT4];[BCT10]; [BCT15]; [BCT26]}
BC_Technique |  [BCT4]Prompt intention formation
|  *has_occurent_part [BCI_Mechanism_of_Action]
---
Behavioural resolution reinforces goal acceptance
BC_Technique |  [BCT10]Prompt specific goal setting
|  *has_occurent_part [BCI_Mechanism_of_Action]
---
Specification of at least one context, that is, where, when, how makes the
brain establish a ”memory” that makes the practice easier
BC_Technique |  [BCT15]Teach to use prompts or cues
|  *has_occurent_part [BCI_Mechanism_of_Action]
---
Learning to identify a prompt and react to it makes the response more
instinctive
BC_Technique |  [BCT26]Time management
|  *has_occurent_part [BCI_Mechanism_of_Action]
---
Freeing time for the behaviour makes it possible to practice it
*has_occurant_part [BCI_Engagement] | {set the goal, the prompt and the time for the activity}
*has_occurant_part [BCI_Delivery] |  | Habit_planning_for_Tai_Chi_Delivery
---
BCI_Delivery |
|  has_BCI_Source [BCI_Source]
---
|  has_profile [BCI_Mode_of_delivery]
---
mobile_app_mode_of_delivery
|  has_profile [BCI_Schedule_of_delivery]
---
@onboarding
|  has_profile [BCI_Style_of_delivery]
---
b) |
BCI_Content | Tai_Chi_Lesson_Content
has_profile [BCI_Dose] |  1/day
has_occurant_part [BC_Technique] |  {[BCT9], [BCT12]}
BC_Technique |  [BCT9]Model or demonstrate the behaviour
|  *has_occurent_part [BCI_Mechanism_of_Action]
---
Observing a person perform the activity makes it clear and increases patient’s
ability to perform it
BC_Technique |  [BCT12] Prompt self-monitoring of behaviour.
|  *has_occurent_part [BCI_Mechanism_of_Action]
---
The ticking of the box after video rewards behaviour and feed-backs lesson
completion
*has_occurant_part [BCI_Engagement] | {clicking play the video}
*has_occurant_part [BCI_Delivery] |  | Demonstrating_Tai_Chi_Delivery
---
BCI_Delivery |
|  has_BCI_Source [BCI_Source]
---
LeiaCohen@Taiflow
|  has_profile [BCI_Mode_of_delivery]
---
playable_electronic_storage_mode_of_delivery
|  has_profile [BCI_Schedule_of_delivery]
---
preferred_time_of_day
|  has_profile [BCI_Style_of_delivery]
---
warm_and_accepting
Intervention_Evaluation_Study | Fatigue Reduction Scenario Intervention Study
evaluates [BCI_Scenario] |  Fatigue Reduction Intervention Scenario
has_output [Evaluation_Finding] |
Evaluation_Finding |  Decreased Fatigue as measured by Severity Scale-Lag
BCI_Evaluation_Study | Tai Chi Intervention Study
evaluates [BCI] |  Tai_Chi_Capsule
has_output [Evaluation_Finding] |
Evaluation_Finding |  {Num_times_Tai_chi_Video_Pressed-Lead; Stress_decreased_Smartwatch; Num_hours_slept_a_night_increased_Smartwatch}

##  3 Digital Intervention Design Steps

An important contribution of this study is to suggest a SATO (ideaS expAnded
wiTh bciO) workflow for the development of DHI apps that create desirable
behaviour change. We consider DHI that corresponds to BCI Scenario and is a
collection of BCIs. Figure 1 summarizes the workflow and the sections below
elaborate on the different activities of the workflow.

We focus the SATO workflow on methodological construction of a Behaviour
Change Intervention Scenario Plan for the Behaviour Change Intervention
Scenario process. We highlight our extensions of the BCIO based on our need to
define clinical goals for the BCI Context (population and setting) – and for
each goal -– to recommend, specify and develop BCIs meeting the goal. Based on
our experience of implementing several BCIs as part of the CAPABLE project,
our running example is a BCI Scenario in which the Behaviour Outcome is
practice of Tai Chi and the desired Intervention Outcome is a reduction of
fatigue, as measured by the standard Fatigue Severity Scale [19] (Figure 2).

Figure 1: SATO workflow for development of DHI apps that create desirable
behaviour change.

###  3.1 BCI Scenario

Figure 2: BCI Scenario, BCI, and BCT examples from the CAPABLE project

Following the IDEAS framework for developing effective digital interventions
to change health behaviour [6], we recommend that the DHI will be developed by
a multi-stakeholder development team consisting of the clinicians whose
patients will use the DHI, the patients themselves, and the technical
developers (engineers, informaticians), which would meet regularly during the
DHI development phase.

####  3.1.1 Understand the Context

The first step that this team does is defining the BCI Scenario’s patient
population. For the CAPABLE project, it is cancer patients treated with
immunotherapy. For the fatigue improvement intervention scenario, we focus on
the subset of patients who experience fatigue.

The intervention setting (eg, patients treated at home) determines the
possible mode of delivery of the BCI and influences the barriers to
engagement. Considering that the patients are treated at home imposes a
constraint on the delivery mode, which is limited to digital interventions.
Potential barriers to engagement with DHI in this context could be a patient’s
lack of access or lack of ability to use the digital technology (such as
wearable devices, eg smartwatches).

####  3.1.2 Identify Clinical Goal (extension of BCIO)

A crucial step in developing a DHI is to select the important clinical
objective or goal of the DHI. The clinical goal will impact the choice of
intervention and evaluation metrics.

The clinical goal can be applicable to a diverse group of patients because it
may be a condition or avoiding the adverse effect that is typical for many
kinds of patients, like reducing sleep problems, anxiety, or increasing
physical activity. However, prior definition of a more specific context, ie,
the target population (eg, cancer patients) and the physical and social
setting for the behaviour change, may help in the creation of a targeted DHI
that can address the wellbeing needs specific to the population, recognize
their disease burden and support personalization of the interventions.
Therefore, to select the clinical goals of the DHI for the target population,
research should be done to establish what wellbeing dimensions [20] are the
ones most impacted by the patients’ condition, fitting with IDEAS’ Empathize
step of the Integrate phase, that integrates insights from users and theory.
This step should be led by the clinical researchers via a literature search
supplemented by questionnaires to the target population. For example, in the
CAPABLE project, our goal is to improve the mental wellbeing of cancer
patients treated at home. The main affected wellbeing dimensions were
identified from the literature and from questionnaires answered by renal
cancer patients in Pavia, Italy, melanoma cancer patients in Amsterdam, The
Netherlands, and patients from the Italian Cancer Patient organization, AIMAC.
They include reduced sleep quality [21], fatigue [22], stress [23], low mood
[24] and decreased physical activity [25]. Improvements in all these
dimensions are then posed as clinical goals (See top of Figure 2). The
clinical goals might be intertwined, (e.g. increasing physical activity could
be a goal in itself but also benefits sleep quality) and should be targeted by
the interventions provided by the digital health app.

Goal ontologies can be used to standardize the specification of clinical
goals. For example, in the Goal-based Comorbidity decision-support method
[26], goals specification follows the goal ontology developed by Fox et al.
[27] that includes a verb and a noun phrase (eg, manage hypertension, prevent
cardiovascular disease, treat fatigue), the HL7 Fast Healthcare
Interoperability Resources (FHIR) [12] Goal resource, and relationships from
the National Drug File - Reference Terminology (NDF-RT) ontology [28], such as
physiological effects of medications (eg, decrease platelet-aggregation,
increased sleep). Figure 3 shows an example of such a goal hierarchy that goes
all the way down to a leaf BCI meeting the goal. Alternatively, in the Asbru
[29] clinical guideline formalism, process goals (e.g., monitor blood
pressure) or state goals (e.g., normal blood pressure) can be specified as
temporal patterns that are meant to be maintained, avoided or achieved (e.g.,
achieve systolic blood pressure $<$140 within 1 month of starting
antihypertensive medication). In our fatigue reduction intervention scenario,
”no clinically significant fatigue” would be a state goal and daily
performance of Tai Chi a process (behaviour) goal.

Figure 3: Goal hierarchy for the Fatigue Treatment Goal

####  3.1.3 Define Lag Measures

How do we measure the intervention’s effectiveness? According to Michie et al
[9], intervention outcomes are processes influenced by interventions which
include individual human behaviour, mental activity and physiological
activity, including unintended side effects. They are measured as BCI
evaluation finding, including BCI effect estimate and BCI outcome estimate.
During the enrollment and at the termination of the CAPABLE study, patients
are asked to fill multiple standard patient-reported outcome measures (PROMs)
[30], such as Insomnia Severity Index (ISI) [31], Fatigue Severity Scale (FSS)
[19], GAD-7 [32] and PHQ-9 [33], which enable assessment of patients’ mental
health prior and post intervention. Each of these questionnaires is linked
with a clinical goal. Changes in scores on these questionnaires do not occur
rapidly; therefore the changes in scores between enrolment and exit
questionnaires are our lag measures[17]. For example, for the goal of improved
fatigue, the patients might score above 36 on the FSS at the enrollment, which
indicates clinical fatigue. Ideally after a few months, at the termination of
the intervention, the patient would have FSS less than 36, corresponding to no
clinically significant fatigue. Note that for the BCI Scenario we measure the
change in patients’ physiological/emotional state related to their clinical
goal and the performance of the target behaviours is considered at the BCI
level (See Table LABEL:tab:bci-def).

###  3.2 Identify BCIs for the clinical goals

To refine the clinical goals and identify the interventions that can meet the
goals, we can turn to clinical practice guidelines. Following the evidence-
based medicine (EBM) movement, “clinical practice guidelines that we can
trust” are defined as “statements that include recommendations intended to
optimize patient care, that are informed by a systematic review of evidence
and an assessment of the benefit and harms of alternative care options for a
clinical condition” (clinical objective/goal) [34]. Clinical guidelines
usually address a specific clinical condition. Unfortunately, most of the
clinical guidelines refer to medication-based care options, and evidence for
non-medication interventions is usually limited. However, non-pharmacological
life-style, exercise, and psycho-behavioural interventions are a promising way
to care for mental wellbeing, including for example, chronic pain[35] and
fatigue [36], and include evidence grades [37] based on cohort studies, and in
some cases on randomized controlled trials and meta-analyses, which provide a
higher grade of evidence.

####  3.2.1 Search for evidence-based intervention

Clinical goal(s) is an important extension of the BCIO and searching for
evidence-based intervention options that meet it fits the “specify target
behaviour” step of the Integrate phase of IDEAS. As mentioned above, clinical
practice guidelines, and other clinical sources following the EBM pyramid, are
the best way to search for evidence-based interventions. For example, the ESMO
cancer-related fatigue guideline [36] suggests mindfulness-based stress
reduction techniques including deep breathing, imagery training, Hatha yoga,
walking in nature, and physical exercise. Additional evidence-based meta-
reviews [37] suggests Tai Chi [18] as a further option. Figure 2 shows
interventions that meet the different clinical goals. We also added the source
property, which originally is a property of a BCT, to BCI to specify its
evidence-based source (See Table LABEL:tab:bci-def).

####  3.2.2 Define Lead Measures

To ensure that the BCI is effective in reaching the target clinical goal,
ideally we would be able to check if a patient is on track of reaching their
goal, and if not, modify the intervention. However, daily assessment through
patient-reported outcome measure (PROM) questionnaires is not feasible long
term, especially given that previous studies found that frequent surveys were
not perceived favorable by the study participants [38] and could negatively
impact engagement with the intervention. Therefore it is important to identify
measures that are related to the outcome but can also be captured frequently
and automatically. In order to reduce the patient burden of self-reporting, we
opted out of daily wellbeing surveys and settled on tracking proxies of
wellbeing, such as physical activity (e.g., number of steps) and quality and
duration of sleep and stress that can be passively measured by a smartwatch
[39]. In our pilot study on a healthy population, we found that the number of
times participants went for a walk correlated with the magnitude of fatigue
reduction [40]. Given that each BCI has a target behaviour, whose performance
should support patients in reaching their clinical goal, the number of times
that a target behaviour has been performed could be a good lead measure. For
example, the number of times a patient performs Tai Chi may serve as a lead
measure for reduction of fatigue. For some goals (e.g., sleep improvement as
measured by Insomnia Severity Index) there could also be automatic daily
measurement of number of hours slept a night, which may indicate early if the
intervention leads towards the achievement of the clinical goal.

####  3.2.3 Select BCI supporting technologies

To monitor intervention adherence, it might be helpful to pair the DHI app
with a wearable device. The choice of the device will depend on the
population, their clinical goals, the selected BCIs and the lead measures. In
the CAPABLE project, the wearable device (ASUS VivoWatch) was selected by the
clinical team based on the feature of automatic measurement of Pulse Transit
Time index [41] and accessibility of the data by a web-based API.

Detecting performance of the target behaviours using wearable devices could be
straightforward for some BCIs (e.g., walk); for other behaviours, e.g., Tai
Chi or Deep Breathing exercises, detection might be more challenging.
Consumer-grade smartwatches are frequently equipped with a photoplethysmogram
(PPG) sensor that could capture blood volume pulse (BVP) signal [42].
Therefore, based on experiments on a public dataset and a literature review,
we hypothesized that we could detect when a patient performed deep breathing
exercise from the changes to their BVP [43]. Nevertheless, during initial
evaluation of the device we found that the ASUS API exposes only extracted
heart rate variability features rather than the raw PPG signal. The feature is
captured in 15 minutes time steps, which results in its limited usefulness for
assessment of adherence to 5 minute-long deep breathing exercises. When
selecting a wearable device we suggest to consider not only type of captured
data but also the frequency and to test the candidate devices early in the
application development cycle.

For deep breathing and Tai Chi, an easier step for adherence monitoring is to
record mobile application’s usage data, including information such as if and
when the patient started the exercise recording embedded within the
application. This captures engagement with the application; not necessarily
the performance of the behaviour [44]. Yet it is a simple solution, not
requiring a specialized wearable device and could serve as a BCI adherence
proxy without posing additional reporting burden on the patient.

###  3.3 Create BCI Content

####  3.3.1 Create user stories

Once the patient population (e.g., cancer patients who are fatigued), their
context (e.g., at home), clinical goals and some indication of evidence-based
BCIs meeting the goals are identified from evidence-based sources, the
development team use agile software development methods [45], including
development of stereotypical persona and user stories, to create a shared
understanding of the anticipated user experience with the DHI. Figure 4
presents a set of user stories for a patient persona called Maria, and her
physician Giulio, that were developed for the CAPABLE project that demonstrate
in a high-level way the anticipated user-experience. These user stories form
the starting point for app screen mockups (e.g., Figure 5) and are refined
with users’ feedback.

Figure 4: A set of user stories for a patient persona called Maria and her
physician Giulio involving the Tai Chi BCI for the fatigue goal and following
BCTs.

(a) Goal-setting in physicians’ app

(b) Goals screen in patient app

(c) Goals progress tracking

Figure 5: (a) Mockup of a goal-setting screen in the physicians’ app that is
used during shared decision-making to set up goals for the patient (agree on a
behavioural contract and review behavioural goals); (b) Mockup of the patient
app showing BCIs. (c) Mockup of the goal-review screen with feedback on how
many of the behaviour goals were achieved compared to the set target. Mock-ups
were created by Bitsens UAB, a partner of the CAPABLE Consortium.

####  3.3.2 Find intervention protocols and source

In further iterations of the Ideate step, the informaticians on the team
searched for specific existing sources, i.e., implementations for the
different interventions, in the form of narrative instructions for deep
breathing [46] or videos demonstrating physical exercise such as Tai Chi [38].
These were accepted by the clinicians and patients of the multidisciplinary
app-design team.

####  3.3.3 Select Behaviour Change Techniques

Abraham and Michie [8] created a taxonomy of behaviour change techniques,
which initially included 26 distinct BCTs and later was extended to 93 BCTs
[47]. In BCI content, we bundle together multiple BCTs to create modules
reusable in different BCIs (See Table LABEL:tab:bci-def ”BCI_Content”). For
the sake of simplicity, we use the original set of 26 BCTs in our examples.

Note that some BCT bundles could be applied on clinical goal level as well as
at single BCI level. For example, during onboarding educational materials
providing information about the clinical goal and the overall approach of
addressing it via behaviour change is explained along with illustrative
examples of some of the BCIs. This content fits with BCTs of providing
information regarding behaviour-health link as well as information on
consequences (Education_Content). In the BCIO, BCTs are defined for BCIs and
not for the entire BCI Scenario. However, we found it as an essential part of
the BCI Scenario Plan to introduce such information about the overall DHI
approach (See Figure 6a) in addition to providing such information at the
level of the BCIs (See Figure 6b) of the BCI Scenario.

Although development of the BCTs for a given BCI can be done at any order, we
present the BCTs according to the order in which a user is usually exposed to
the BCTs over their use of a BCI delivered via the DHI: providing information
and instruction (Education_Content), goal setting and habit planning
(Habit_Planning_Content), behaviour demonstration/instructions and self-
monitoring (Behaviour_Lesson_Content e.g. Tai Chi), review of behaviour goals
and progress_tracking (Review_Content), prompting, and providing encouragement
and rewards (Notification_Content). Most of the app modules supporting these
BCTs are directly suitable for reuse in different BCIs with little adaptation.

Table LABEL:tab:bci-def provides BCIO examples for the Fatigue Improvement
Scenario of the Context ”fatigued cancer patients at home”. For the goal of
less fatigue, several BCIs are developed, each around one Outcome Behaviour.
For Tai Chi BCI, we show the details of different BCI_Content objects,
bundling one or more BC Techniques that support habit formation of the Outcome
behaviour. The BCI_Content objects shown include (a) habit planning for Tai
Chi (bundling 4 BCTs of intention formation, goal setting, learning to
identify a prompt, and time management) and (b) Tai Chi Lesson (bundling the
BCTs for demonstrating the behaviour and prompting self-monitoring)

Figure 6c shows Tai_Chi_Lesson_Content mockup with video demonstrations of the
activity and ticks to monitor and reward lesson completion. Figure 5c shows
Review_Content, prompting review of the weekly goals and providing feedback on
progress toward the set goal.

(a) App on-boarding

(b) Tai Chi Information

(c) Tai Chi Lessons

Figure 6: Mockup screens for education content at the application and
individual BCI level. Mock-ups were created by Bitsens UAB, a partner of the
CAPABLE Consortium

###  3.4 Customise and Personalise

Customisation (or customisability) refers to a creation of predefined set of
options during the design step, whereas personalisation is a process of
matching patients to the options best suited for them. Personalisation is one
of the most commonly used techniques in mobile health interventions [48] and
it plays important role in influencing their effectiveness [49].

Tong et al.[50] conducted a systematic review of personalized mobile BCIs and
highlighted that personalisation might be applied to: BCI_Content (e.g.,
demonstration video), BCI_Mode_of_delivery (e.g., voice message, game,
wearable), BCI_Dose (e.g., number of daily notifications),
BCI_Schedule_of_delivery. Automatic, data-driven personalisation depends also
on the type and source of the collected data, the frequency of data
collection, and the personalisation algorithm. These should be also already
considered at the customization option-generation step.

####  3.4.1 Define set of customisation options

To maximize the impact of education information on patients’ outcome
behaviour, it is important that patients perceive the information to be
personally relevant. Ghalibaf et al. conducted a systematic review of
computer-based health information tailoring and identified six dimensions
according to which patients could be characterized, these are: 1) socio-
demographic (e.g., age, level of education), 2) medical history(e.g.,
comorbidities), 3) health state (e.g., disease severity), 4) psycho-
behavioural determinants (e.g., attitude, self-efficacy) 5) knowledge level,
6) history of interactions (e.g., visited pages) [51]. In practice, majority
of studies used three or fewer dimensions for user profiling with socio-
demographic and psycho-behavioural features being the most popular.

The choice of the user categorisation dimensions depends on the BCI Content.
Some parts are static ( ie, selected once prior to interventions
commencement), other are dynamic (ie, change depending on the user interaction
with the application). For example, Michalowski et al. [52] developed
precompiled educational plans depending on patients education level and their
preferred learning style (e.g., visual or audio). To further enhance patients’
understanding of their care the authors included Question and Answer (Q&A)
components in the education plan. The tailoring of Q&A difficulty level was
based on Blooms taxonomy of educational objectives [53] and depended on the
patient’s learning progress captured in the response log.

Another interesting mode of delivery supporting long-term engagement with BCI,
and worth considering during the customisation step, is gamification [54]. DHI
gamification frequently relies on social cognitive theory (SCT), which states
that people learn new behaviours through observation of others [55] and the
behavioural change goal is realized through self-observation, self-evaluation,
self-reaction and self-efficacy. The BCI Engagement could be created to align
with SCT and include gamification elements, e.g. the notification template for
Tai Chi Notification Content might read as follows: ”The majority of patients
felt relief of fatigue symptoms after one month of daily Tai Chi practice. You
have completed X Tai Chi lessons this week (Y% of your weekly target and Z
lesson less/more than patients similar to you.” A potential problem with this
template is that people vary in response to social comparison [56]; some get
motivated others may get discouraged [57].

Tondello et al. created the Hexad Player Type Framework [58], which identifies
the following six player types: Disruptor, Free Spirit, Achiever, Player,
Socialiser, Philanthropist. Each player type is associated with a set of game
elements that motivates them. For example Socialiser will get motivated by
social competition, or team activities, whereas Achiever would prefer
challenges and progression through levels. Profiling patients according to
their player type could help to match them to adequate notification and weekly
goal review templates and facilitate their adherence to BCIs.

Finally, DHI should be customised for different goals. Goal setting is
associated with personalisation, but we should be ready up-front for
specifying different goals. Baretta et al. [59] reviewed multiple apps
directed towards encouraging physical activity in the context of goal setting
theory. In particular, they evaluated components of goal setting: specificity,
difficulty, time-frame, action planning, evaluation and reevaluation. The
authors found that the majority of the apps enabled specific goal setting and
many considered time-frame but few allowed the tailoring of goal difficulty to
the users’ ability level and non considered re-evaluating the goals based on
individuals achievements. In CAPABLE the goal setting is spread across
multiple BCT bundles. In Habit Planning Content, patients set the time in
which they want to perform the target behaviour. In Behaviour Lesson Content,
the user selects the length of the lesson and in Review Content, the user
modifies their weekly goals. Note that the latter two could be also set
automatically by an algorithm that selects and enumerates the customisable
templates for varying goals and difficulty, which are created during the
customisation step.

####  3.4.2 Develop personalisation methods

The goal of the personalisation algorithm is to match the patients to one of
available customization options and maximise the probability that they perform
the target behaviour. According to Fogg’s Behaviour Model (FBM) [60], three
factors impact behaviour completion: motivation, ability and trigger.
Tailoring of Notification Content and Education Content may increase patients’
motivation, Behaviour Lesson Dose, the user’s ability to perform target
behaviour, and schedule of prompt delivery the user’s responsiveness to the
notification.

The personalisation algorithms are either knowledge-based and data-based.
Sadasivam et al. [61] showed that using Bayesian probabilistic matrix
factorization, trained on previous message ratings, yields message tailoring
that is more effective at influencing users to quit smoking than a rule-based
method (Notification Content). Manuvinakurike et al. [62] utilized an adaptive
boosting model to select the best suited health behaviour-change story
presented to a user, given their response to questionnaires measuring
demographics, self-efficacy, decisional balance (i.e. balance between
perceived positive and negative consequences of selecting a new behaviour),
and the stage of change, according to the transtheoretical model (TTM) [63].
TTM identifies five stages of change: precontemplation, contemplation,
preparation, action and maintenance. Individuals are assumed to move through
this stages during the process of behaviour change. The model resulting from
considering these data was used for personalized story selection (Education
Content) and had a significant impact on users’ self-efficacy when compared to
randomly assigned stories.

ML models have also been utilized for tailoring of the notification timing
(BCI Schedule of delivery). Morris et al. [64] used information from GPS,
accelerometer and time of the day to train a Naive Bayes classifier to predict
if the participant will respond to the notification. When ML-powered push-
notifications were compared with daily notification within a predefined time
frame, there was no difference in user responsiveness. Nevertheless, other ML
approaches that aimed at maximization of responsiveness incorporated richer
information, such as mobile application usage [65] or user personality traits
[66] into model training and reported good performance in responsiveness
prediction on retrospective data.

We considered using the patient’s internal context such as stress [43] and
cognitive load [67], derived from data captured by the watch, alongside
external context information such as location and time of the day for learning
the best time to intervene. Given that prior to the intervention we had no
data about patients’ responsiveness in varying contexts, we conducted an
experiment on simulated data [68] to evaluate different ML approaches to
notification schedule personalisation and found adaptive supervised learning
approaches to be more effective than reinforcement learning methods.

Rule-based approaches might be well suited for automatic BCI Goal Setting. For
example, when a patient has short and inconsistent sleep, a rule may suggest a
BCI supporting sleep quality improvement. The BCI Dose might be also
implemented as a set of rules, starting from the recommended value for the
target population (eg, Tai Chi 3 times per week) and adjusted according to the
number of times the patient performed the activity during the last week (eg,
reduced to 2 times a week if the goal was not met) to fit the patient’s
ability.

##  4 Evaluation

We evaluated the applicability of our workflow and checklist by considering
multiple clinical goals/BCI Scenarios, BCIs (Capsules), and BCT Bundles. We
found that our methods were appropriate for all of these scenarios, BCIs and
BCT bundles. In Table 2 we summarise goals and interventions for which we
defined content as part of CAPABLE app.

Table 2: Goals and content defined in CAPABLE app | Names | Total Number
---|---|---
| Goals
---
BCI Scenarios
| Fatigue Reduction
---
Sleep Improvment
Stress/Anxiety Reduction
Mood Improvment
Physical Activity Increase
5
| BCIs (Capsules)
---
| Deep Brathing, Imagery Training,
---
Tai Chi, Yoga, Garden Bowl,
Gratitude Journal, Photo Collage
7
BCT Bundles |  | Education_Content
---
Habit_Planning_Content
Behaviour_Lesson_Content
Review_Content
Notification_Content
5

##  5 Discussion

We proposed SATO, a DHI design workflow aligned with the BCIO and the IDEAS
framework, which we evaluated as being comprehensive for designing several BCI
Scenarios for a CAPABLE System app for cancer patients, as part of the CAPABLE
project. To our knowledge, this is a first DHI guide which considers multiple
BCIs and builds a Goal hierarchy with defined evaluation metrics at each
level. We also provided examples on utilising behaviour change theories, such
as SCT and FBM, when considering customisation and personalisation of the
intervention for maximum engagement and adherence.

Although we presented the design steps consecutively, in practice they are
iterative (See Figure 1). For example, at the point of defining the target
population and setting, we started creating the user stories for the DHI and
later refined them for each BCI Scenario and BCI. Similarly, the lead measures
were changed after testing the real capabilities of the selected smartwatches.
Moreover, the knowledge engineers on our team suggested to incorporate a wide
range of BCTs; nevertheless some content elements (eg, providing feedback
through progress visualisation) were not included in the final app because the
psychologists who were part of the multi-stakeholder development team raised
concern that the ability of cancer patients to perform target behaviour might
actually deteriorate with time due to the toxic effects of the cancer therapy
and the course of illness; in such cases, visualizing progress data might
negatively impact their emotional well-being. This example highlights the need
for iterative redesign with the end users being continuously kept in mind.

The proposed guide focuses on DHI design, therefore the Share step from IDEAS
framework is not comprehensively addressed. We have however utilized BCIO and
extended it to facilitate knowledge sharing. The study with the cancer
patients has not yet commenced, hence we could not provide concrete examples
of Evaluation Findings. We will address this limitation and also evaluate the
suitability of our chosen lead measures in future work, when conducting the
clinical study of the usage of the CAPABLE system by cancer patients.

##  6 Conclusions

We described a process of designing digital behaviour change intervention
which incorporates a range of behaviour change techniques and addresses
multiple clinical goals. The step-by-step SATO guide that we created extends
BCIO to support the scenarios of multiple intertwined and hierarchical goals
and therefore could be used for design of any non-pharmacological digital
health intervention. We aimed to keep the process simple and provide concrete
examples of: technology-independent system captured lead metrics, application
modules bundling several behaviour change techniques, and customisation
templates based on behaviour change theories which could be readily reused in
other DHIs.

## Acknowledgments

The CAPABLE project has received funding from the European Union’s Horizon
2020 research and innovation programme under grant agreement No 875052. This
work has been also partly supported by the EU H2020 grant Sano No. 857533 and
the IRAP Plus programme of the Foundation for Polish Science.

The authors also wish to thank Vitali Gisko and Valentina Ganicheva from
Bitsens UAB for provision of the app mock-ups.

## References

  * [1] FDA. What is Digital Health?;.  Accessed: 2022-03-28.  https://www.fda.gov/medical-devices/digital-health-center-excellence/what-digital-health.
  * [2] Klasnja P, Pratt W.  Healthcare in the pocket: mapping the space of mobile-phone health interventions.  Journal of biomedical informatics. 2012;45(1):184-98.
  * [3] Silva BM, Rodrigues JJ, de la Torre Díez I, López-Coronado M, Saleem K.  Mobile-health: A review of current state in 2015.  Journal of biomedical informatics. 2015;56:265-72.
  * [4] Jameel L, Valmaggia L, Barnes G, Cella M.  mHealth technology to assess, monitor and treat daily functioning difficulties in people with severe mental illness: A systematic review.  Journal of psychiatric research. 2021.
  * [5] Emberson MA, Lalande A, Wang D, McDonough DJ, Liu W, Gao Z.  Effectiveness of Smartphone-Based Physical Activity Interventions on Individuals’ Health Outcomes: A Systematic Review.  BioMed Research International. 2021;2021.
  * [6] Mummah SA, Robinson TN, King AC, Gardner CD, Sutton S.  IDEAS (Integrate, Design, Assess, and Share): a framework and toolkit of strategies for the development of more effective digital interventions to change health behavior.  Journal of medical Internet research. 2016;18(12):e5927.
  * [7] Murray E, Hekler EB, Andersson G, Collins LM, Doherty A, Hollis C, et al.. Evaluating digital health interventions: key questions and approaches. Elsevier; 2016.
  * [8] Abraham C, Michie S.  A taxonomy of behavior change techniques used in interventions.  Health psychology. 2008;27(3):379.
  * [9] Michie S, West R, Finnerty AN, Norris E, Wright AJ, Marques MM, et al.  Representation of behaviour change interventions and their evaluation: Development of the Upper Level of the Behaviour Change Intervention Ontology.  Wellcome open research. 2020;5.
  * [10] Parimbelli E, Gabetta M, Lanzola G, Polce F, Wilk S, Glasspool D, et al.  CAncer PAtients Better Life Experience (CAPABLE) First Proof-of-Concept Demonstration.  In: International Conference on Artificial Intelligence in Medicine. Springer; 2021. p. 298-303.
  * [11] Veggiotti N, Sacchi L, Peleg M.  Enhancing the IDEAS Framework with Ontology: Designing Digital Interventions for Improving Cancer Patients’ Wellbeing.  In: AMIA Annual Symposium Proceedings. vol. 2021. American Medical Informatics Association; 2021. p. 1186.
  * [12] HL7. Fast Healthcare Interoperability Resources Release 3 Specification; 2017.  Accessed: 2022-03-20.  https://www.hl7.org/fhir/overview.html.
  * [13] Fogg BJ.  Tiny habits: The small changes that change everything.  Eamon Dolan Books; 2019.
  * [14] Peleg M, Michalowski W, Wilk S, Parimbelli E, Bonaccio S, O’Sullivan D, et al.  Ideating mobile health behavioral support for compliance to therapy for patients with chronic disease: a case study of atrial fibrillation management.  Journal of medical systems. 2018;42(11):1-15.
  * [15] Gruber TR.  The role of common ontology in achieving sharable, reusable knowledge bases.  Kr. 1991;91:601-2.
  * [16] Arp R, Smith B, Spear AD.  Building ontologies with basic formal ontology.  Mit Press; 2015.
  * [17] McChesney C, Covey S, Huling J.  The 4 disciplines of execution: Achieving your wildly important goals. vol. 34.  Simon and Schuster; 2012.
  * [18] Song S, Yu J, Ruan Y, Liu X, Xiu L, Yue X.  Ameliorative effects of Tai Chi on cancer-related fatigue: a meta-analysis of randomized controlled trials.  Supportive Care in Cancer. 2018;26(7):2091-102.
  * [19] Krupp LB, LaRocca NG, Muir-Nash J, Steinberg AD.  The fatigue severity scale: application to patients with multiple sclerosis and systemic lupus erythematosus.  Archives of neurology. 1989;46(10):1121-3.
  * [20] Linton MJ, Dieppe P, Medina-Lara A.  Review of 99 self-report measures for assessing well-being in adults: exploring dimensions of well-being and developments over time.  BMJ open. 2016;6(7):e010641.
  * [21] Mystakidou K, Parpa E, Tsilika E, Pathiaki M, Patiraki E, Galanos A, et al.  Sleep quality in advanced cancer patients.  Journal of psychosomatic research. 2007;62(5):527-33.
  * [22] Hofman M, Ryan JL, Figueroa-Moseley CD, Jean-Pierre P, Morrow GR.  Cancer-related fatigue: the scale of the problem.  The oncologist. 2007;12(S1):4-10.
  * [23] Zabora J, BrintzenhofeSzoc K, Curbow B, Hooker C, Piantadosi S.  The prevalence of psychological distress by cancer site.  Psycho-Oncology: Journal of the Psychological, Social and Behavioral Dimensions of Cancer. 2001;10(1):19-28.
  * [24] Spiegel D, Giese-Davis J.  Depression and cancer: mechanisms and disease progression.  Biological psychiatry. 2003;54(3):269-82.
  * [25] Littman AJ, Tang MT, Rossing MA.  Longitudinal study of recreational physical activity in breast cancer survivors.  Journal of Cancer Survivorship. 2010;4(2):119-27.
  * [26] Kogan A, Peleg M, Tu SW, Allon R, Khaitov N, Hochberg I.  Towards a goal-oriented methodology for clinical-guideline-based management recommendations for patients with multimorbidity: GoCom and its preliminary evaluation.  Journal of Biomedical Informatics. 2020;112:103587.
  * [27] Fox J, Alabassi A, Patkar V, Rose T, Black E.  An ontological approach to modelling tasks and goals.  Computers in biology and Medicine. 2006;36(7-8):837-56.
  * [28] US Dept. of Veterans Affairs, National Drug File – Reference Terminology (NDF- RT TM) Documentation U.S. Department of Veterans Affairs; 2015.
  * [29] Shahar Y, Miksch S, Johnson P.  A task-specific ontology for the application and critiquing of time-oriented clinical guidelines.  In: Conference on Artificial Intelligence in Medicine in Europe. Springer; 1997. p. 51-61.
  * [30] Cella D, Hahn EA, Jensen SE, Butt Z, Nowinski CJ, Rothrock N, et al.  Patient-reported outcomes in performance measurement. 2015.
  * [31] Morin CM, Belleville G, Bélanger L, Ivers H.  The Insomnia Severity Index: psychometric indicators to detect insomnia cases and evaluate treatment response.  Sleep. 2011;34(5):601-8.
  * [32] Williams N.  The GAD-7 questionnaire.  Occupational medicine. 2014;64(3):224-4.
  * [33] Kroenke K, Spitzer RL, Williams JB.  The PHQ-9: validity of a brief depression severity measure.  Journal of general internal medicine. 2001;16(9):606-13.
  * [34] Graham R, Mancher M, Wolman D, Greenfield S, Steinberg E.  Institute of Medicine. Clinical practice guidelines we can trust.  Natl Acad Press. 2011.
  * [35] Chou R, Huffman LH.  Nonpharmacologic therapies for acute and chronic low back pain: a review of the evidence for an American Pain Society/American College of Physicians clinical practice guideline.  Annals of internal medicine. 2007;147(7):492-504.
  * [36] Fabi A, Bhargava R, Fatigoni S, Guglielmo M, Horneber M, Roila F, et al.  Cancer-related fatigue: ESMO Clinical Practice Guidelines for diagnosis and treatment.  Annals of Oncology. 2020;31(6):713-23.
  * [37] Murad M, Asi N, Alsawas M, Alahdab F. New evidence pyramid. BMJ Evidence-Based Medicine. 2016; 21: 125–127;.
  * [38] Amorim AB, Pappas E, Simic M, Ferreira ML, Jennings M, Tiedemann A, et al.  Integrating Mobile-health, health coaching, and physical activity to reduce the burden of chronic low back pain trial (IMPACT): a pilot randomised controlled trial.  BMC musculoskeletal disorders. 2019;20(1):1-14.
  * [39] Cornet VP, Holden RJ.  Systematic review of smartphone-based passive sensing for health and wellbeing.  Journal of biomedical informatics. 2018;77:120-32.
  * [40] Lisowska A, Lavy S, Wilk S, Peleg M.  How to Improve Digital Wellbeing Interventions? Preliminary Study of Factors Affecting Intervention Engagement, Impact, and Habit Formation.  In: AMIA Annual Symposium Proceedings. American Medical Informatics Association; 2022. In press.
  * [41] Ghosh S, Banerjee A, Ray N, Wood PW, Boulanger P, Padwal R.  Continuous blood pressure prediction from pulse transit time using ECG and PPG signals.  In: 2016 IEEE Healthcare Innovation Point-Of-Care Technologies Conference (HI-POCT). IEEE; 2016. p. 188-91.
  * [42] Saganowski S, Kazienko P, Dzieżyc M, Jakimów P, Komoszyńska J, Michalska W, et al.  Review of Consumer Wearables in Emotion, Stress, Meditation, Sleep, and Activity Detection and Analysis.  arXiv preprint arXiv:200500093. 2020.
  * [43] Lisowska A, Wilk S, Peleg M.  Catching patient’s attention at the right time to help them undergo behavioural change: Stress classification experiment from blood volume pulse.  In: International Conference on Artificial Intelligence in Medicine. Springer; 2021. p. 72-82.
  * [44] Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, et al.  Understanding and promoting effective engagement with digital behavior change interventions.  American journal of preventive medicine. 2016;51(5):833-42.
  * [45] Lucassen G, Dalpiaz F, van der Werf JME, Brinkkemper S.  Improving agile requirements: the quality user story framework and tool.  Requirements engineering. 2016;21(3):383-403.
  * [46] WEbMD. Breathing Techniques for Stress Relief.;.  Accessed: 2021-09-30.  https://www.webmd.com/balance/stress-management/stress-relief-breathing-techniques#1.
  * [47] Michie S, Richardson M, Johnston M, Abraham C, Francis J, Hardeman W, et al.  The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions.  Annals of behavioral medicine. 2013;46(1):81-95.
  * [48] Dugas M, Gao G, Agarwal R.  Unpacking mHealth interventions: a systematic review of behavior change techniques used in randomized controlled trials assessing mHealth effectiveness.  Digital health. 2020;6:2055207620905411.
  * [49] Gosetto L, Ehrler F, Falquet G.  Personalization Dimensions for MHealth to Improve Behavior Change: A Scoping Review.  Integrated Citizen Centered Digital Health and Social Care. 2020:77-81.
  * [50] Tong HL, Quiroz JC, Kocaballi AB, Fat SCM, Dao KP, Gehringer H, et al.  Personalized mobile technologies for lifestyle behavior change: A systematic review, meta-analysis, and meta-regression.  Preventive Medicine. 2021:106532.
  * [51] Ghalibaf AK, Nazari E, Gholian-Aval M, Tara M.  Comprehensive overview of computer-based health information tailoring: a systematic scoping review.  BMJ open. 2019;9(1):e021022.
  * [52] Michalowski M, Wilk S, Michalowski W, O’sullivan D, Bonaccio S, Parimbelli E, et al.  A Health eLearning Ontology and Procedural Reasoning Approach for Developing Personalized Courses to Teach Patients about Their Medical Condition and Treatment.  International journal of environmental research and public health. 2021;18(14):7355.
  * [53] Bloom BS.  Taxonomy of educational objectives: The classification of educational goals.  Cognitive domain. 1956.
  * [54] Cugelman B.  Gamification: what it is and why it matters to digital health behavior change developers.  JMIR serious games. 2013;1(1):e3139.
  * [55] Bandura A, Walters RH.  Social learning theory. vol. 1.  Englewood cliffs Prentice Hall; 1977.
  * [56] Lisowska A, Lavy S, Wilk S, Peleg M.  Personality and habit formation: Is there a link?  In: Proceedings of the AIxIA 2021 SMARTERCARE Workshop, CEUR-WS; 2021\. p. 42-7.
  * [57] Schmidt-Kraepelin M, Thiebes S, Stepanovic S, Mettler T, Sunyaev A.  Gamification in health behavior change support systems-A synthesis of unintended side effects.  In: Proceedings of the 14th International Conference on Wirtschaftsinformatik; 2019. p. 1032-46.
  * [58] Tondello GF, Wehbe RR, Diamond L, Busch M, Marczewski A, Nacke LE.  The gamification user types hexad scale.  In: Proceedings of the 2016 annual symposium on computer-human interaction in play; 2016. p. 229-43.
  * [59] Baretta D, Bondaronek P, Direito A, Steca P.  Implementation of the goal-setting components in popular physical activity apps: Review and content analysis.  Digital health. 2019;5:2055207619862706.
  * [60] Fogg BJ.  A behavior model for persuasive design.  In: Proceedings of the 4th International Conference on Persuasive Technology; 2009. p. 1-7.
  * [61] Sadasivam RS, Borglund EM, Adams R, Marlin BM, Houston TK.  Impact of a collective intelligence tailored messaging system on smoking cessation: the Perspect randomized experiment.  Journal of medical Internet research. 2016;18(11):e6465.
  * [62] Manuvinakurike R, Velicer WF, Bickmore TW, et al.  Automated indexing of Internet stories for health behavior change: weight loss attitude pilot study.  Journal of medical Internet research. 2014;16(12):e3702.
  * [63] Prochaska JO, Redding CA, Evers KE, et al.  The transtheoretical model and stages of change.  Health behavior: Theory, research, and practice. 2015;97.
  * [64] Morrison LG, Hargood C, Pejovic V, Geraghty AW, Lloyd S, Goodman N, et al.  The effect of timing and frequency of push notifications on usage of a smartphone-based stress management intervention: an exploratory trial.  PloS one. 2017;12(1):e0169162.
  * [65] Pielot M, Cardoso B, Katevas K, Serrà J, Matic A, Oliver N.  Beyond interruptibility: Predicting opportune moments to engage mobile phone users.  Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. 2017;1(3):1-25.
  * [66] Künzler F, Mishra V, Kramer JN, Kotz D, Fleisch E, Kowatsch T.  Exploring the state-of-receptivity for mhealth interventions.  Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. 2019;3(4):1-27.
  * [67] Lisowska A, Wilk S, Peleg M.  Is it a good time to survey you? Cognitive load classification from blood volume pulse.  In: 2021 IEEE 34th International Symposium on Computer-Based Medical Systems (CBMS). IEEE; 2021. p. 137-41.
  * [68] Lisowska A, Wilk S, Peleg M.  From personalized timely notification to healthy habit formation: A feasibility study of reinforcement learning approaches on synthetic data.  In: Proceedings of the AIxIA 2021 SMARTERCARE Workshop, CEUR-WS; 2021\. p. 7-18.


# On the Use of Mutation in Injecting Test Order-Dependency††thanks: Accepted
at the MSR 2021 Registered Reports Track.

Sarra Habchi  University of Luxembourg
<EMAIL_ADDRESS>Maxime Cordy  University of Luxembourg
<EMAIL_ADDRESS>Mike Papadakis  University of Luxembourg
<EMAIL_ADDRESS>Yves Le Traon  University of Luxembourg
<EMAIL_ADDRESS>
###### Abstract

Background: Test flakiness is identified as a major issue that compromises the
regression testing process of complex software systems. Flaky tests manifest
non-deterministic behaviour, send confusing signals to developers, and break
their trust in test suites. Both industrial reports and research studies
highlighted the negative impact of flakiness on software quality and
developers’ productivity. While researchers strive to devise solutions that
could help developers addressing test flakiness, the elaboration and
assessment of these solutions are hindered by the lack of datasets large
enough to leverage learning techniques. Aim: To address this lack, we conduct
an exploratory study that investigates a new mean for producing datasets of
flaky tests. Method: We propose an approach that relies on program mutation to
inject flakiness in software tests. In particular, we plan to delete helper
statements from tests to make their outcomes order-dependent, i.e., pass in
certain running orders but fail in other orders. We intend to apply our
mutation-based approach to a set of 14 Java projects to assess the
effectiveness of test mutation in injecting order-dependency and generate a
new dataset that could be used to study flakiness.

###### Index Terms:

Test flakiness, order-dependency, regression testing, mutation analysis.

##  I Introduction

Regression testing is a critical activity of modern software development. It
enables simultaneous code development by validating program changes, _i.e.,_
providing confidence that these changes do not break existing functionality.
Nonetheless, this process can be hindered whenever non-deterministic
behaviours are exhibited. In particular, flaky tests, which pass and fail for
the same code version, send false alerts to developers and break their trust
in the test suite. Developers waste considerable amounts of time investigating
flaky test failures and this can slow down the continuous integration [1].
Besides, oft-repeated flaky failures may lead developers to disregard real
test failures or even dispels them from writing tests. For these reasons,
flakiness is widely discussed among software practitioners [2, 3, 4, 5].

Several reports have shown that developers observe recurrently flaky tests and
they struggle to address them. This struggle stems from the non-deterministic
nature of flaky tests and the lack of suitable tools to cope with them.
Specifically, developers are ill-equipped to distinguish flaky failures from
real faults. Thus, they tend to rerun tests several times looking for
persistent failures or passes. However, the rerun strategy is costly both
computation and time-wise and its efficiency at identifying and reproducing
flaky tests is questionable [6].

Responding to this need, researchers have started putting increasing amounts
of efforts in understanding flaky tests and forming efficient techniques to
detect them. Notably, the DeFlaker tool was proposed to detect flaky tests
based on historical regression test executions [7]. Also, Lam _et al_. [8]
proposed the iDFlakies tool that targets a different type of flakiness, tests
for which the outcome changes depending on their order in the suite, _a.k.a._
order-dependent tests. Furthermore, researchers envision the usage of machine
learning to address flakiness. Indeed, learning techniques are already used
for predicting flaky tests statically [9, 10, 11]. Moreover, machine learning
could assist in detecting, reproducing, and fixing flaky tests. Nevertheless,
these applications are limited by the lack of large sets of flaky tests that
could serve the purposes of learning and evaluating. Hence, there is a strong
need for novel, large and diverse datasets of flaky tests.

Empirical studies rely on flakiness detection tools to build datasets from
open source projects [7, 8]. For instance, a dataset was built using the
DeFlaker tool, and it remains, to the best of our knowledge, the largest set
with 1,874 flaky tests [7]. Yet, later studies showed that this set is hardly
reproducible, which makes it difficult to use for learning fixes or root
causes [10]. Nevertheless, it is hard to collect large, reproducible datasets
from open source projects since flakiness appears in rapidly evolved (and
tested) development environments. An additional root cause of flakiness
regards the infrastructure used, which pose additional questions on the
validity, responsibility and feasibility of the developed methods and their
results.

In this paper, we present an exploratory study that aims to address the lack
of test flakiness datasets, in general, and order-dependent tests in
particular, from a different perspective. Instead of trying to mine test
flakiness from projects, we propose Flaker, an approach that injects order-
dependency flakiness in software tests. Flaker relies on program mutation to
make tests order-dependent while it does not influence the test results. Our
focus on order-dependent tests is motivated by the well-founded evidence about
the sources and fixes of this type of flaky tests.

In particular, previous studies showed that order-dependency manifests through
recurrent patterns of interactions between tests, such as polluters, victims,
and brittles [12, 8]. Similarly, order-dependent tests can also be fixed
thanks to helper statements that appear in cleaners and state-setters [13].
These helpers set or reset the shared state to lay the ground for a successful
and stable test execution. The objective of our approach is to remove these
helpers and make stable tests order-dependent. For this, we rely on program
mutation, which is well-suited and widely used for these purposes.

We intend to evaluate Flaker on 14 open-source Java projects that we collect
from a public dataset. The objective of this evaluation is to answer the
following research questions:

  * •

RQ1: How effective is mutation in injecting order-dependent tests?
Objective: We aim to assess the suitability of test mutation for generating
datasets of order-dependent tests.
Evaluation: We compare the number of order-dependent tests generated by
mutation to the numbers of order-dependent tests that could be mined using
detection tools like iDFlakies.

  * •

RQ2: What are the characteristics of injected order-dependent tests?
Objective: We aim to characterise the injected order-dependent tests.
Evaluation: Following the terminology of previous research [13], we classify
the generated tests in terms of victims, and brittles. We also investigate the
nature of failures caused by the mutants, _e.g.,_ Assertion errors or other
exceptions.

  * •

RQ3: What projects are more receptive to order-dependency injection?
Objective: We aim to identify and characterise testing systems that are prone
to order-dependency injection.
Evaluation: We explore test suite properties that could explain the proneness
to test order-dependency, _e.g.,_ shared states, fixture usages, etc.

##  II Background

Order-dependent tests are a category of flaky tests that manifests a non-
deterministic behaviour because of the execution order of the test suite. Such
a situation may arise if the regression testing is performed using some form
of test selection [14]. A test $t$ is order-dependent if, for the same program
and test suite, there exists a passing order $O_{p}$ for which:

$Run(t,O_{p})=Pass$

And a failing execution order $O_{f}$ for which:

$Run(t,O_{f})=Fail$

The problem of order-dependency has been particularly highlighted when
projects updated their Java Development Kit version from $6$ to $7$. The new
Java version modified the reflection implementation and therefore the order
adopted by JUnit for running class tests evolved. Hence, test suites that used
to pass correctly, started failing because of the new JUnit orders [15, 16].
Accordingly, developers realised that their suites are order-dependent.

As with other categories of flaky tests, order-dependent tests hinder
regression testing as they waste development time and introduce uncertainty in
the test suite. Besides these effects, order-dependent tests also impact the
effectiveness of techniques that aspire to optimise regression testing. The
study of Lam _et al_. [17] showed that order-dependent tests should be
accounted for in test selection, prioritisation, and parallelisation
techniques. The dependency between tests makes it difficult to reduce or
rearrange the test suite without leading to non-deterministic failures.

Listing 1: Example of state polluters and cleaners with a victim test.

⬇

1

2//Polluter

3@Test

4public void customConnectionFactory() {

5 handler = new RequestHandler() {...};

6 ConnectionFactory factory =

7 new ConnectionFactory(){

8 ... };

9 HttpRequest.setConnectionFactory(factory);

10 int code = get("http://not/a/real/url").code();

11 assertEquals(200, code);

12}

13

14//Cleaner

15@Test

16public void nullConnectionFactory() {

17 handler = new RequestHandler() {...};

18 HttpRequest.setConnectionFactory(null);

19 int code = get(url).code();

20 assertEquals(200, code);

21}

22

23//Victim

24@Test

25public void postWithNumericQueryParams() {

26 Map<> inputParams = new HashMap<>();

27 inputParams.put(1, 2);

28 inputParams.put(3, 4);

29 final Map<> outputParams = new HashMap<>();

30 handler = new RequestHandler() {...};

31 HttpRequest request = post(url,inputParams,false);

32 assertTrue(request.ok());

33 assertEquals("2", outputParams.get("1"));

34 assertEquals("4", outputParams.get("3"));

35}

Previous studies showed that order-dependency is a result of interactions
between different types of tests [12, 18, 13]. In the following, we define and
explain these types of tests and their role in the passing and failing orders
of flaky tests.

###  II-A Polluters, victims, and cleaners

The study of Gyori _et al_. [12] identified polluters as a source of order-
dependent tests. These tests are not order-dependent themselves but they
pollute the state shared between tests and lead to the failure of tests that
are executed after them, _a.k.a._ victims. Accordingly, a victim is a test
that passes when run alone but fails when run with polluting tests. Following
the formalisation of Shi _et al_. [13], a test is defined as a victim $v$, if
there is a non-empty set of polluters $P$ for which:

$\displaystyle Run(v)=Pass$ $\displaystyle Run(P,v)=Fail$

In the same vein, cleaners are tests that contain helper statements that clean
the shared state and allow victims to pass. A set of tests $C$ are considered
cleaners if there exists a victim $v$ and a set of polluters $P$ for which:

$\displaystyle Run(P,C,v)=Pass$

Listing 1 shows an example of a test class from the HTTP Request project that
contains a polluter, a cleaner, and a
victim111https://github.com/kevinsawicki/http-request. In this example,
customConnectionFactory() pollutes the shared state by creating a
ConnectionFactory and setting it in the static field of the HttpRequest, line
9. On the other hand, the test nullConnectionFactory() cleans that polluted
state by resetting the factory of the HttpRequest to null in line 18. The test
postWithNumericQueryParams() presents a victim because it uses the HttpRequest
class without ensuring that its set properly. As a result, when the victim is
run after the polluter, the ConnectionFactory is not null and the assertion in
line 32 fails. However, when the same test is run alone or after the cleaner,
the state is properly set and the assertions pass.

###  II-B Brittles

Huo and Clause [18] identified brittles as another form of order-dependent
tests. A brittle is a test that fails when run alone but passes when run after
other tests. Brittles pass after other tests because they rely on other tests,
called state-setters, to prepare the state required for their success.
Specifically, a test $b$ is considered as a brittle if there is a non-empty
set of state-setters $S$ that ensures that:

$\displaystyle Run(b)=Fail$ $\displaystyle Run(S,b)=Pass$

##  III Execution Plan

In this section, we explain our execution plan by first presenting Flaker, our
approach for injecting order-dependent tests. Then, we present the evaluation
protocol that we intend to follow in order to answer our research questions.

###  III-A Flaker

Figure 1: Overview of the Flaker approach.

Figure 1 presents an overview of the Flaker approach. Flaker takes as input a
Java program with its test suite and outputs a database of order-dependent
tests. First, based on different rerun strategies, Flaker selects stable tests
from the program under analysis. Then, it mutates the selected tests to
generate mutants that have potential order-dependency. Finally, it explores
different running orders to evaluate and confirm the order-dependency in the
mutant classes.

####  III-A1 Step 1: Select stable tests

The objective of this step is to filter out tests that are already flaky
before proceeding to flakiness injection. For this purpose, Flaker extracts
all tests from the program under analysis and checks their flakiness following
a two-step process.

##### Detect non-order-dependent flaky tests

Non-order-dependent flaky tests represent tests that are non-deterministic
because of general flakiness root causes, _e.g.,_ concurrency and randomness,
and not because of order-dependency. Flaker identifies these tests by
rerunning every test individually $100$ times to check that its outcome is
consistent. If a test result shifts in one of the execution, _i.e.,_
$Pass\rightarrow Fail$ or $Fail\rightarrow Pass$, the test is considered flaky
and we exclude it.

##### Detect order-dependent tests

We rely on randomised running orders to detect order-dependent tests. This
strategy allows us to reveal dependencies between tests while avoiding the
combinatorial explosion. Previous studies showed that randomised orders are
the most effective for detecting dependency as they outperform reversal,
exhaustive bounded, and dependence-aware bounded algorithms [19]. Since our
tool targets Java programs, we only generate orders for tests that are in the
same class and we do not consider the whole suite. We generate $20$ random
orders for each test class. This choice is based on the experiments of Lam _et
al_. [8], which showed that 20 class orders are sufficient for detecting most
flaky tests.

We run each generated order while keeping a record of test outcomes. Once all
orders are run, we analyse the records to spot order-dependent tests, _i.e.,_
tests that yielded different results over the executions. We exclude these
tests because their non-determinism could influence the conduct of the
upcoming steps.

####  III-A2 Step 2: Mutate tests

The aim of this step is to generate test mutants that could lead to order-
dependency. The study of Shi _et al_. [13] highlighted the presence of helper
statements that reset the shared state and make tests order-independent. The
objective of Flaker is to remove these helpers and make tests order-dependent.
For this, we relied on the mutation operator del_statement, which allows us to
delete any statement from the test code. With this mutation, we can create
mutant tests that are brittles or victims.

##### Generate brittles

Listing 2 presents an example of helpers found in the open-source project
Undertow222https://github.com/undertow-io/undertow. The example shows that the
two tests testIdleTimeout() and testCloseReason() have an identical helper
statement in lines 3 and 11 respectively. These helpers reset the state of the
Endpoint and ensure that session connections are successful in both tests.

With mutation, we can remove the helper in line 11 and make testCloseReason()
order-dependent. Indeed, without resetting the Endpoint, testCloseReason()
would fail when run alone but would pass when run after the testIdleTimeout().
Consequently, testCloseReason() becomes a brittle and testIdleTimeout()
becomes a state-setter.

Listing 2: Example of a helper statement.

⬇

1@Test

2public void testIdleTimeout() throws Exception {

3 Endpoint.reset();

4 Session session = deployment.connectToServer();

5 Assert.assertEquals(..., Endpoint.message());

6 ...

7}

8

9@Test

10public void testCloseReason() throws Exception {

11 Endpoint.reset();

12 MessageEndpoint.reset();

13 Session session = deployment.connectToServer();

14 session.close();

15 assertEquals("CLOSED",Endpoint.message());

16 ...

17}

##### Generate victims

Listing 3 presents a modified example of potential polluters and victims in
the Hadoop project333https://github.com/apache/hadoop/. In this example, the
class has a test fixture method that runs once before all class tests. This
method initialises the static variable testDir to prepare for test runs. On
the other hand, testWithStringAndConfForBuggyPath() modifies this variable in
line 11 and pollutes the shared state. However, testAbsoluteGlob() is
protected from this pollution thanks to the helper in line 19, which reassures
that this shared variable is set to the right value before using it in
assertions. Hence, testAbsoluteGlob() is order-independent and it passes even
when run after testWithStringAndConfForBuggyPath().

With mutation, Flaker can remove the helper from testAbsoluteGlob() and
transform it into a victim. The test would pass when run alone because the
initialisation sets the required state but would fail when run after
testWithStringAndConfForBuggyPath(), which becomes a polluter.

Listing 3: Example of potential for a polluter and a victim.

⬇

1

2

3@BeforeClass

4public static void initialize() {

5 ...

6 testDir = new Path(System.getProperty(..)+"..");

7}

8

9@Test

10public void testWithStringAndConfForBuggyPath() {

11 dirString = "file:///tmp";

12 testDir = new Path(dirString);

13 assertEquals("file:/tmp", testDir.toString());

14 checkPathData();

15}

16

17@Test

18public void testAbsoluteGlob() {

19 testDir = new Path(System.getProperty(..)+"..");

20 PathData[] items = PathData.expandAsGlob(...);

21 assertEquals(sortedString(

22 testDir+"/d1/f1", testDir+"/d1/f1.1"),

23 sortedString());

24}

As we cannot identify the helper statement beforehand, the mutation is applied
to all test statements, except the assertions. That is, for each test with $m$
non-assert statements, Flaker generates $m$ mutants. Each mutant test is
reinserted in its mother class to form a mutant test class. If the mutant
class does not compile, the mutant is considered invalid and therefore
excluded from the analysis.

####  III-A3 Step 3: Evaluate mutants

The aim of this step is to assess the impact of mutants on order-dependency.
Algorithm 1 describes the process of identifying order-dependency in each
mutant class.

Inputs: mutantClass

Outputs: flakyTests[]

Procedure _identifyFlakiness(mutantClass)_

classTests[] = extractClassTests(mutantClass)

classOrders[] = generateTestOrders(mutantClass)

foreach  _classOrder O $\in$ classOrders[]_ do

passingTests[], failingTests[] = run(mutantClass, O)

foreach  _test T $\in$ classTests[]_ do

if  _T $\in$ passingTests[]_ then

T.addPassingOrder(O)

else

T.addFailingOrder(O)

end if

end foreach

end foreach

foreach  _test T $\in$ classTests[] _ do

if  _T.passingOrders $\neq\emptyset$ and T.failingOrders $\neq\emptyset$_ then

flakyTests.addTest(T)

if  _rerun(T) == Pass_ then

T.isVictim = True

else if  _rerun(T) == Fail_ then

T.isBrittle = True

else

T.isNonOrderDependent = True

end if

end if

end foreach

return flakyTests[]

Algorithm 1 Detect order-dependent tests in mutant.

First, we extract all the tests of the mutant class and we generate random
running orders for them. Then, we run each of these orders and we save the
execution results to keep track of orders that led to a test pass or failure.
After executing all the generated orders, we verify if any of the class tests
have both passing and failing orders. If so, this test is flaky and we need to
classify it. For this purpose, we rerun the test 100 times alone to check if
it is flaky regardless of the execution order. In such a case, the test is
flaky but not order-dependent and we label it as such. If the test passes
consistently through the reruns, this means that it is a victim of other
polluting tests, which makes it fail when run with other tests. However, if
the test fails when run alone, we consider it as a brittle that needs to run
with other state-setters to pass.

###  III-B Evaluation

TABLE I: Evaluation metrics. |  Metric
---|---
RQ1 |  #stable tests: The number of tests selected after filtering out tests that are already flaky in the first step of Flaker.
#mutants: The total number of mutants generated by Flaker at step 2 (before
validation).
#valid mutants: The number of mutants that were successfully compiled after
their reinsertion in the test class. This metric provides insights about the
effects of mutation on the test correctness.
#OD mutants: The number of mutants that show order-dependency at the
evaluation step.
%OD tests: The ratio between the number of order-dependent tests and the total
number of tests in a project.
%OD classes: The ratio between the number of order-dependent classes and the
total number of test classes in a project.
run time: The average execution time for Flaker on different projects and test
classes.
RQ2 |  %brittles: The percentage of order-dependent mutants that could not pass if run alone.
%victims: The percentage of order-dependent mutants that could pass when run
alone.
%assertion: The percentage of order-dependent mutants that fail due to
assertion errors.
%exceptions:The percentage of order-dependent mutants that fail due to other
exceptions or errors.
RQ3 |  class size: The number of tests in a test class.
#static fields: The number of static fields in a test class.
fixture: A boolean indicating the presence of fixture, _i.e.,_ setUp() and
tearDown() methods, in a test class.

Table I presents the metrics that we intend to compute after applying Flaker.
In the following, we explain how we leverage these metrics to answer our
research questions.

####  III-B1 RQ1: How effective is mutation in injecting order-dependent
tests?

First, we compute the proportion of order-dependent mutants compared to the
number of stable tests. This allows us to assess the capacity of mutation in
injecting order-dependent tests. Also, to highlight the success rate of the
mutation, we calculate the proportion of order-dependent mutants compared to
all the generated and valid mutants.

To evaluate the suitability of the mutation approach for generating order-
dependency datasets, we compare the numbers of generated order-dependent tests
to the numbers of order-dependent tests that could be mined using other tools.
Specifically, we compute the metrics %OD tests and %OD classes for the order-
dependent mutants generated by Flaker. Then, we compare these values with the
proportions of %OD tests and %OD classes that could be detected using the tool
iDFlakies [8]. These metrics can be retrieved from the results of previous
studies, which applied iDFlakies on a set of Java projects. This comparison
allows us to determine whether the mutation approach is a sound alternative
for creating order-dependency datasets or not.

Furthermore, to assess the feasibility of the mutation approach, we report the
average time consumed by Flaker to perform the mutation and evaluate the
order-dependency of mutants.

####  III-B2 RQ2: What are the characteristics of injected order-dependent
tests?

We categorise the order-dependent mutants into brittles and victims to explain
the nature of order-dependent tests that we can inject with mutation. We also
report the causes of failure, %assertion and %exceptions to show the impact of
mutation on tests. These four metrics are also useful for the characterisation
of the generated dataset and can be useful for studies that rely on it. On top
of that, these metrics can help in comparing our injected mutants with real
order-dependent tests that are observed in the wild.

####  III-B3 RQ3: What projects are more receptive to order-dependency
injection?

We aim to inspect the $14$ projects under study and identify characteristics
that could explain their level of proneness to flakiness injection. For this
purpose, we analyse test suite properties, which could be related to order-
dependency. For instance, we compute the size of test classes and analyse its
impact on the success of flakiness injection. We also investigate the effect
of shared resources, _e.g.,_ static fields, which could be a sign of hidden
states. Finally, we inspect the presence of test fixtures, _i.e.,_ setUp() and
tearDown() methods, which could protect tests against our mutation. To assess
the effect of these metrics on the number of injected order-dependent mutants,
we use the following statistical tests:

  * •

Spearman’s rank coefficient [20]: a non-parametric measure that assesses how
well the relationship between two variables can be described using a monotonic
function. We use this coefficient to assess the effect of class size and
static fields on the number of injected order-dependent mutants. This measure
does not require the normality of the variables and does not assess the
linearity, hence its adequacy for our analysis.

  * •

Mann-Whitney U test [21]: we use this test to check if the distributions of
injected order-dependent tests are identical in the classes that have fixtures
or not.

  * •

Cliff’s $\delta$ [22]: a non-parametric effect size measure, which is suitable
for ordinal data. We use this measure to quantify the effect size of the
presumed difference between the classes based on the presence of fixture.

It is worth noting that this question is exploratory and its analysis will
depend on the injection results and the project properties. Hence, the axes
presented here are only preliminary and other metrics can be added later.

##  IV Dataset

In this section, we present the dataset that we aim to use in our study.

TABLE II: Evaluation set.

Project | Link
---|---
Alien4Cloud | https://github.com/alien4cloud/alien4cloud
DropWizard | https://github.com/dropwizard/dropwizard
Dubbo | https://github.com/apache/incubator-dubbo
Elastic-Job-Lite | https://github.com/elasticjob/elastic-job-lite
Fastjson | https://github.com/alibaba/fastjson
Hsac-Fitness-Fixture | https://github.com/fhoeben/hsac-fitnesse-fixtures
Junit-quickcheck | https://github.com/pholser/junit-quickcheck
Marine API | https://github.com/ktuukkan/marine-api
OpenPOJO | https://github.com/OpenPojo/openpojo
Spring-data-envers | https://github.com/spring-projects/spring-data-envers
Struts | https://github.com/apache/struts
Undertow | https://github.com/undertow-io/undertow
Wikidata-Toolkit | https://github.com/Wikidata/Wikidata-Toolkit
Wildfly | https://github.com/wildfly/wildfly

Table II shows the projects that we intend to use in our evaluation. These
projects are collected from a public
dataset444https://sites.google.com/view/flakytestdataset that was used in
previous order-dependency studies. The original set of projects was first used
to identify flaky tests with iDFlakies [8]. Later, Shi _et al_. used the tool
iFixFlakies to patch order-dependent tests in 14 projects of this dataset
[13]. In our study, we intend to focus on the same 14 projects to facilitate
the early validation of our tooled approach. Specifically, if our mutation
approach is capable of removing helpers and evaluating order-dependency
accurately, it should be capable of reversing the fixes that were performed by
iFixFlakies in these projects. On top of that, the former existence of order-
dependency in these projects could be a favouring factor for our injection.
For instance, these projects can already have shared states or polluting tests
but their impact on tests is not visible yet because of the helpers. Our
approach could reveal these dependencies by removing helper statements.

##  V Threats to Validity

A potential threat to our internal validity stems from our strategy for
detecting order-dependency. We opted for randomised orders to detect order-
dependent tests in both Step 1 and Step 3 of Flaker. This strategy is not
complete in that it does not explore all possible permutations of tests and it
could miss some dependencies. Thus, it is possible that some order-dependent
tests are not filtered at Step 1 and could interfere with our mutant
evaluation. Similarly, some flaky tests could be missed in our mutant
evaluation and hinder our study results. Nevertheless, achieving a complete
order-dependency detection requires the exploration of all test permutations,
which is impossible due to combinatorial explosion. The randomised algorithm
showed its superiority compared to other incomplete algorithms like order
reversal, exhaustive bounded search, and dependence-aware bounded search [19].
Besides, our configuration and choice of the number of reruns align with the
positive results observed in previous studies on order-dependency detection
[8].

Another possible threat arises from the use of mutation to generate a dataset
of flaky tests. It is possible that the injected tests differ from the ones
detected in the wild. To alleviate this threat, we report several metrics on
the injected tests, _e.g.,_ brittle, victim, and raised exception, to allow
the comparison between our injected order-dependent tests and tests from other
datasets.

##  VI Conclusion

This paper presented our planned exploratory study on the injection of
flakiness in software tests. The objective of this study is to evaluate the
suitability of test mutation for generating datasets of order-dependent tests.
To conduct this study, we proposed Flaker, a tooled approach that removes
helper statements from stable test suites to create order-dependency. This
paper also presented the software projects and analysis protocol that we
intend to adopt in our planned study.

## References

  * [1] M. Eck, M. Castelluccio, F. Palomba, and A. Bacchelli, “Understanding Flaky Tests: The Developer’s Perspective,” _arXiv_ , pp. 830–840, 2019.
  * [2] J. Palmer, “Test flakiness – methods for identifying and dealing with flaky tests : Spotify engineering,” https://engineering.atspotify.com/2019/11/18/test-flakiness-methods-for-identifying-and-dealing-with-flaky-tests/, November 2019, (Accessed on 01/12/2021).
  * [3] J. Listfield, “Google testing blog: Where do our flaky tests come from?” https://testing.googleblog.com/2017/04/where-do-our-flaky-tests-come-from.html, April 2017, (Accessed on 01/12/2021).
  * [4] J. Micco, “The State of Continuous Integration Testing Google,” _Icst 2017_ , 2017.
  * [5] M. contributors, “What’s new in python 3.1 - python 3.9.1 documentation,” https://developer.mozilla.org/en-US/docs/Mozilla/QA/Avoiding_intermittent_oranges, 2019, (Accessed on 01/20/2021).
  * [6] W. Lam, S. Winter, A. Astorga, V. Stodden, and D. Marinov, “Understanding reproducibility and characteristics of flaky tests through test reruns in java projects,” in _2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE)_. IEEE, 2020, pp. 403–413.
  * [7] J. Bell, O. Legunsen, M. Hilton, L. Eloussi, T. Yung, and D. Marinov, “DeFlaker: Automatically Detecting Flaky Tests,” in _Proceedings of the 40th International Conference on Software Engineering - ICSE ’18_. New York, New York, USA: ACM Press, 2018, pp. 433–444. [Online]. Available: http://dl.acm.org/citation.cfm?doid=3180155.3180164
  * [8] W. Lam, R. Oei, A. Shi, D. Marinov, and T. Xie, “IDFlakies: A framework for detecting and partially classifying flaky tests,” _Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019_ , pp. 312–322, 2019.
  * [9] T. M. King, D. Santiago, J. Phillips, and P. J. Clarke, “Towards a Bayesian Network Model for Predicting Flaky Automated Tests,” _2018 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C)_ , pp. 100–107, 2018.
  * [10] G. Pinto, B. Miranda, S. Dissanayake, M. D’Amorim, C. Treude, and A. Bertolino, “What is the Vocabulary of Flaky Tests?” _Proceedings - 2020 IEEE/ACM 17th International Conference on Mining Software Repositories, MSR 2020_ , pp. 492–502, 2020.
  * [11] A. Bertolino, E. Cruciani, B. Miranda, and R. Verdecchia, “Know Your Neighbor: Fast Static Prediction of Test Flakiness,” _Proceedings of the International Conference on Software Engineering (ICSE)_ , 2020. [Online]. Available: https://ieeexplore.ieee.org
  * [12] A. Gyori, A. Shi, F. Hariri, and D. Marinov, “Reliable testing: detecting state-polluting tests to prevent test dependency,” in _Proceedings of the 2015 International Symposium on Software Testing and Analysis - ISSTA 2015_. New York, New York, USA: ACM Press, 2015, pp. 223–233. [Online]. Available: http://dl.acm.org/citation.cfm?doid=2771783.2771793
  * [13] A. Shi, W. Lam, R. Oei, T. Xie, and D. Marinov, “iFixFlakies : A Framework for Automatically Fixing Order-Dependent Flaky Tests,” in _27th ACM Joint European Software Engineering Conference and Symposium on the Foundations ofSoftware Engineering (ESEC/FSE ’19)_ , 2019.
  * [14] C. Leong, A. Singh, M. Papadakis, Y. L. Traon, and J. Micco, “Assessing transition-based test selection algorithms at google,” in _Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice, ICSE (SEIP) 2019, Montreal, QC, Canada, May 25-31, 2019_ , H. Sharp and M. Whalen, Eds. IEEE / ACM, 2019, pp. 101–110. [Online]. Available: https://doi.org/10.1109/ICSE-SEIP.2019.00019
  * [15] Intellijava, “Java dev notes.: Junit and java 7.” http://intellijava.blogspot.com/2012/05/junit-and-java-7.html, March 2012, (Accessed on 03/19/2021).
  * [16] B. Kunjummen, “all and sundry: Junit test method ordering,” http://www.java-allandsundry.com/2013/01/junit-test-method-ordering.html, January 2013, (Accessed on 03/19/2021).
  * [17] W. Lam, A. Shi, R. Oei, S. Zhang, M. D. Ernst, and T. Xie, “Dependent-test-aware regression testing techniques,” in _Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis_ , 2020, pp. 298–311.
  * [18] C. Huo and J. Clause, “Improving oracle quality by detecting brittle assertions and unused inputs in tests,” in _Proceedings of the 22Nd ACM SIGSOFT International Symposium on Foundations of Software Engineering_ , 2014, pp. 621–631.
  * [19] S. Zhang, D. Jalali, J. Wuttke, K. Muşlu, W. Lam, M. D. Ernst, and D. Notkin, “Empirically revisiting the test independence assumption,” in _Proceedings of the 2014 International Symposium on Software Testing and Analysis_ , 2014, pp. 385–396.
  * [20] J. H. Zar, “Spearman rank correlation,” _Encyclopedia of biostatistics_ , vol. 7, 2005.
  * [21] D. J. Sheskin, _Handbook of parametric and nonparametric statistical procedures_. crc Press, 2003.
  * [22] J. Romano, J. D. Kromrey, J. Coraggio, and J. Skowronek, “Appropriate statistics for ordinal level data: Should we really be using t-test and cohen’sd for evaluating group differences on the nsse and other surveys,” in _annual meeting of the Florida Association of Institutional Research_ , 2006, pp. 1–33.