# Translating Behavioral Theory into Technological Interventions: Case Study
of an mHealth App to Increase Self-reporting of Substance-Use Related Data

Mashfiqui Rabbi  Harvard University1 Oxford StreetCambridgeMA02134USA
<EMAIL_ADDRESS>,  Meredith Philyaw-Kotov  University of MichiganAnn
Arbor, MIUSA<EMAIL_ADDRESS>,  Jinseok Li  University of MichiganAnn
Arbor, MIUSA<EMAIL_ADDRESS>,  Katherine Li  University of MichiganAnn
Arbor, MIUSA<EMAIL_ADDRESS>,  Bess Rothman  University of MichiganAnn Arbor,
MIUSA<EMAIL_ADDRESS>,  Lexa Giragosian  Yale UniversityAnn Arbor, MIUSA
<EMAIL_ADDRESS>,  Maya Reyes  University of MichiganAnn Arbor, MIUSA
<EMAIL_ADDRESS>,  Hannah Gadway  University of MichiganAnn Arbor, MIUSA
<EMAIL_ADDRESS>,  Rebecca Cunningham  University of MichiganAnn Arbor,
MIUSA<EMAIL_ADDRESS>,  Erin Bonar  University of MichiganAnn Arbor,
MIUSA<EMAIL_ADDRESS>,  Inbal Nahum-Shani  University of MichiganAnn
Arbor, MIUSA<EMAIL_ADDRESS>,  Maureen Walton  University of MichiganAnn
Arbor, MIUSA<EMAIL_ADDRESS>,  Susan Murphy  Harvard
UniversityCambridge, MAUSA<EMAIL_ADDRESS>and  Predrag Klasnja
University of MichiganAnn Arbor, MIUSA<EMAIL_ADDRESS>
###### Abstract.

Mobile health (mHealth) applications are a powerful medium for providing
behavioral interventions, and systematic reviews suggest that theory-based
interventions are more effective. However, how exactly theoretical concepts
should be translated into features of technological interventions is often not
clear. There is a gulf between the abstract nature of psychological theory and
the concreteness of the designs needed to build health technologies. In this
paper, we use SARA, a mobile app we developed to support substance-use
research among adolescents and young adults, as a case study of a process of
translating behavioral theory into mHealth intervention design. SARA was
designed to increase adherence to daily self-report in longitudinal
epidemiological studies. To achieve this goal, we implemented a number of
constructs from the operant conditioning theory. We describe our design
process and discuss how we operationalized theoretical constructs in the light
of design constraints, user feedback, and empirical data from four formative
studies.

H5.2 User Interfaces: User Design; Theory & Methods

††ccs: Computer systems organization Embedded systems††ccs: Computer systems
organization Redundancy††ccs: Computer systems organization Robotics††ccs:
Networks Network reliability

##  1\. Introduction

HCI research on health often focuses on the development of novel technologies
for health behavior change (Hekler et al., 2013; Rabbi et al., 2015; Consolvo
et al., 2008b; Lane et al., 2011; Lee and Dey, 2014; Rabbi et al., 2018). Such
technologies use various intervention strategies—goal-setting, feedback,
rewards, etc.—to try to actuate mechanisms of change (e.g., habit formation,
operant learning) that can lead to desired changes in behavior. Commonly, both
the intervention strategies that these technologies implement and the
mechanisms they target are drawn from behavioral theories. These
theories—social cognitive theory, theory of planned behavior, operant
conditioning, etc.—are supported by extensive empirical evidence, and many
have been successfully used by behavioral scientists to guide behavior-change
interventions for decades (Glanz et al., 2008; Redding et al., 2000; Rothman,
2000). HCI researchers often follow a similar path, adopting widely used
theories to design their technological interventions. Since these theories are
supported by extensive evidence base, theory-based interventions enable
designers to maximize the likelihood of successfully influencing behavior
change (Rabbi et al., 2015; Consolvo et al., 2008b, 2009).

Despite the wide use of theory in intervention development, its translation
into features of a concrete technological intervention is rarely
straightforward (Kok et al., 2004). One challenge is that theoretical
constructs—i.e., the basic determinants postulated by a theory to influence
behavior (Hekler et al., 2013; Michie et al., 2014b)—are formulated at an
abstract level that does not afford straightforward implementation. Take, for
instance, a seemingly simple case of the construct “goal.” There is extensive
evidence that concrete goals work better than abstract goals (e.g., “do your
best”), that challenging but doable goals work better than easy goals, and
that goals work best when an individual feels ownership over them (Locke and
Latham, 2006; Gulotta et al., 2016). Yet, to use this evidence for
intervention design—say, a mobile app to support physical activity—requires a
great deal of specification. The designer has to decide how to make the goal
concrete by specifying what units to use (steps, minutes of moderate-to-
vigorous physical activity, number of exercise sessions?), what time frame
(day, week, open-ended?), and what kinds of activities will count toward the
goal (all physical activity, only sessions longer than 10 minutes, only
activities above a heart-rate threshold?). Defining a “challenging goal” is
still more complicated: Should the designer use national guidelines? Some
percentage increase over baseline activity level (if so, what percentage)?
Just let the user set what he/she thinks the right challenging goal is? And so
on. Even for well specified constructs, implementation requires a myriad of
decisions.

Another challenge arises from design constraints. Coherence of the user
experience, development resources, requirements of target population, cultural
norms, intended duration of use, among other factors, all come into play when
a new intervention is being designed. Such considerations can create design
tensions and limit how a particular theory can be implemented. Finally, user
reactions and feedback can override even the most careful theoretical and
design thinking. If users are turned off by a feature, keeping it risks poor
adoption or even abandonment.

Designing theoretically-based interventions thus has to be an intricate
balancing act where theoretical constructs are iteratively concretized and
operationalized in the light of constraints and user feedback. How that
process proceeds is rarely made explicit. What is usually presented in papers
is the final artifact, accompanied by a list of theories or constructs that
the intervention embodies. What was involved in translating those theories
into individual features and how those features ended up in the form they did
typically remains unstated even in papers that report on the design process in
some detail. Yet, theory translation is an essential aspect of health
technology design and we need robust methods for doing it effectively.

The primary methodological contribution of this paper is a process for theory
translation during technology development. We describe this process through a
case study—accompanied by empirical data from several formative studies—of the
theory translation process during the design of a new technology. We describe
the development of SARA (Substance Abuse Research Assistant), an app for
collecting self-report data in epidemiological studies on substance use among
adolescents (ages 14-17) and emerging adults (ages 18-24). We discuss how we
used concepts from operant conditioning theory (Skinner, 2011; Reynolds, 1975;
Staddon and Cerutti, 2003; Ferster and Skinner, 1957) to design a variety of
incentives that aim to increase self-report adherence over time while
minimizing the need for financial compensation. As we will see, even though
operant conditioning has a robust evidence base and its constructs are well
specified, the translation process was far from straightforward. We had to
undertake an extensive user-centered design process to iteratively build
numerous design elements that conformed to a large number of practical
constraints. By presenting our experience with SARA, we hope we can make
explicit the general issues that are involved in theory translation and begin
formulating a more systematic method for translating theoretical constructs
into high-fidelity technological interventions.

Our second contribution is the novel SARA app itself. SARA tackles the problem
of low self-report adherence in mHealth (mobile, 2018; Dorsey et al., 2017) in
the context of a uniquely new population for technical
interventions—adolescents and young adults at high risk of substance abuse.
The resulting self-report adherence rates from a 30-day SARA deployment are
encouraging, and the rates are similar to prior substance use epidemiological
studies that paid nearly seven times more money to participants to collect
data (details in section 3.10) (Bonar et al., 2018).

##  2\. Background and motivations

###  2.1. What we mean by theory translation

Every behavior change theory postulates a set of constructs and mechanisms by
which the constructs interact to change behavior. For example, the goal-
setting theory defines 14 constructs and four mechanisms (Locke and Latham,
1994; Michie et al., 2014b). Two of these 14 constructs are ‘directive
function’ and ‘goal specificity’, and these two constructs interact by the
following mechanism: a specific goal acts as a directive function that takes
attention away from irrelevant stimuli and behavior and by doing so increases
the frequency of target behavior. This is just one example of the relationship
of theory, constructs and mechanisms, and there are many more. For a fuller
definition of theory, constructs, and mechanisms, see Hekler et al. (Hekler et
al., 2013) and Michie et al. (Michie et al., 2014b). When designers use a
theory like goal-setting as the basis for one or more components of a
technological intervention, they implement constructs from that theory as
concrete design elements—say, a graph that shows goal progress, interface
through which users set their weekly activity goals, or the algorithm the
system uses to calculate the goal levels suggested to users (Hekler et al.,
2013)111Note that it is rare for an intervention to implement all constructs
from a theory. Most interventions, both technological and non-technological,
typically implement only a subset of constructs from a single theory and will
often mix and match constructs from multiple theories (Hekler et al., 2013;
Rabbi et al., 2018; Adams et al., 2014). The issue is that while the
constructs being implemented are general and abstract, the design elements
that embody the interventions have to be made concrete. We define the process
that designers go through to bridge this gap as the theory translation
process.

###  2.2. Theory translation in HCI literature

Theory translation happens every time designers incorporate an element in
their technology that is based on a theoretical construct.  Theory translation
is a central step of several intervention development frameworks. For example,
the Multiple Optimization Strategy (MOST) framework includes a “preparation”
phase (Collins, 2018) during which researchers construct theory-based
intervention components, and the Agile approach (Hekler et al., 2016) includes
a “behavior change module development” phase, where the goal is to build small
units of intervention based on theory, expert knowledge, and secondary
analysis of data.222 Note, other intervention optimization frameworks (e.g.,
micro-randomized trials (Klasnja et al., 2015),Just-in-time interventions
(Nahum-Shani et al., 2014b)) also provide stage-by-stage processes for
intervention development. However, these models and MOST largely focus on
optimization trials, where the goal is to gather data that can help the
researcher find an optimal combination and sequence of interventions. These
methods are quantitative and conducting the trials requires well-developed
intervention components. Our paper deals with how to create these well-
developed intervention components. Klasnja et al. (Klasnja et al., 2017) give
a process for defining a “proximal outcome” of an already developed
intervention. However, the Klasnja et al. does not address how to build an
intervention stage-by-stage. Theory is a key source of intervention components
in all these framework, and is one commonly used to develop HCI interventions
as well. One would expect, then, that best practices for theory translation
process are well understood in HCI, and that designers have principled methods
to engage in this process efficiently and effectively. This is not the case,
however. In our review of the HCI literature on behavior change, we found that
intentional theory translation in HCI is not the norm, and that even when it
does occur, it is often done without the benefit of a systematic and well-
described process.

To perform the review, we searched for the term “behavior change” in the ACM
Digital Library. The search returned 591 papers from CHI, Ubicomp, and CSCW
conferences. We read the abstracts of these papers, and we excluded papers
that did not contain interventions. We also excluded work-in-progress papers
because the design process is likely incomplete for the interventions
described in those papers. This filtering reduced the number of papers to 71.
The resulting set included papers that described behavior change interventions
in several domains, including health, environmental sustainability, and
internet use, among others (see Supplementary file 2 for a complete list). We
then read each of the 71 papers and we coded them for how they used theory to
support intervention design.333A limitation of our review is we only reviewed
HCI literature. Theory translation has been done in behavioral science, but
they often focus on organizational aspects of human-to-human interaction
(e.g., doctors, stake-holders, policymakers). HCI on the other hand focus on
technical interventions and interactions between humans and computers.

Five categories of theory use emerged from the review: (i) 43% (31/71) of the
papers did not explicitly mention theory or constructs at all. The
interventions described in those papers either have no theoretical basis or
they use theory implicitly, providing no information about what theories were
used or how. (ii) 7% (5/71) of the papers used theory to explain data. In
other words, these papers used theory post facto to make sense of the
findings, rather than to design the interventions. (iii) 17% (12/71) of the
papers referred to theories (usually multiple) as being the basis for the
intervention, but provided no description of how those theories were used for
intervention design. Theories and the system are presented separately, and
these papers provide no explicit description of which aspects of the system
are based on which theoretical constructs and how. (iv) 20% (14/71) of the
papers used theory to inform design and are explicit about which technology
features were derived from which construct. However, these papers provide no
information about the translation process—i.e., how and why the features ended
up in the form that they did. As such, this category also provides little help
for understanding how the translation process takes place. (v) Finally, a
small fraction of papers (12%, 9/71) described their theory translation
processes in at least some detail. Examining these papers revealed no
standardized ways of approaching theory translation, nor a presence of design
techniques to facilitate this process. Furthermore, none of the papers in this
category discuss if or how they ensured that the final design is a faithful
representation of the underlying constructs. For example, some of the papers
started with a theoretical construct and implemented something different than
that construct, making it unclear what theory is tested in the end.  In
summary our review reveals that the HCI literature on behavior change has so
far largely ignored theory translation as a methodological aspect of the
design of behavior change technologies.

###  2.3. Theory translation in behavioral science

Of course, we are not the first to think about the intricacies of applying
theory to intervention design. Literature in behavioral science deals with
many of these issues, although usually in non-technological settings. Several
models have been proposed, including Intervention Mapping (Bartholomew et al.,
1998; Kok et al., 2004), Behavior Change Wheel (Michie et al., 2011; Michie et
al., 2013; Michie et al., 2014a), and the logic model (Julian, 1997)), that
provide guidance on how to apply theory to develop interventions.  In
intervention mapping (IM), design a theory-based intervention includes several
phases. IM starts with a need assessment phase where focus groups, online
surveys, expert consensus gathering, etc. are held to identify what, if
anything, needs to be changed and for whom. This need assessment phase is
intended to develop objectives that the intervention has to achieve. Needs
assessment is followed by a literature search phase, which aims to identify
theories that can help the intervention achieve these objectives. Then the
process moves to a phase where the intervention is implemented. This is done
by using chosen theories to specify “strategies”—operations that the
intervention will use to bring about change—and developing intervention
materials that implement these strategies. However, IM provides little detail
about how this move from strategies to intervention materials takes place,
other than to note that the population and resource constraints need to be
taken into account when intervention materials are developed.

Logic model is typically used for the theory selection purpose (den Bakker et
al., 2019). Logic model asks designers to clearly and thoroughly specify the
theories/constructs and the mechanisms through which these theories/constructs
will influence various intended outcomes. Logic model’s highly detailed
description brings clarity to communicating the intervention and creating an
action plan to implement the intervention. Logic model has been extensively
used for non-technical intervention, and it is beginning to be used to design
technical interventions as well (Edwards et al., 2018). The process to derive
the logic model can greatly differ across studies. Some studies used focus
groups and online surveys (Geraghty et al., 2016), while others used
literature review and expert feedback to create the logic model (Cole-Lewis et
al., 2019; den Bakker et al., 2019; Theofanopoulou et al., 2019). In addition,
sometimes a logic model is used not to develop an intervention but to describe
the final intervention (Edwards et al., 2018). While it is extremely useful
for clarifying the causal pathway through which an intervention is supposed to
operate, logic model has on occasion been criticized for being too stringent
or rigid (Greenhalgh and Russell, 2010). For the purposes of this discussion,
however, the main weakness of the logic model is that it too does not specify
how exactly constructs in the model should be concretize into specific
intervention features.

The Behavior Change Wheel model (Michie et al., 2011; Michie et al., 2013;
Michie et al., 2014a) is another model that provides steps to guide the
creation of theory-based interventions. At a high level, Behavior Change Wheel
model proposes a three step process: (i) identify target behaviors in detail.
This is done via the COM-B model (Michie et al., 2011) that helps intervention
designers work through the target population’s capabilities, motivations, and
opportunities for change; (ii) once target behaviors are specified, use the
behavior change wheel framework to find appropriate intervention categories
(e.g., education, persuasion, coercion) for impacting those target behaviors
(Michie et al., 2011); and (iii) once intervention categories are chosen, use
the taxonomy of behavior change techniques (BCT) to specify more granular
intervention strategies (e.g., provide situation-specific rewards or
punishment, facilitate goal-setting) that are appropriate for targeting chosen
behaviors (Michie et al., 2013). The least abstract entity in this model are
the behavior change techniques. BCTs are theoretically-derived operations—such
as “facilitate planning,” “monitor behavior,” “monitor goal progress,”
etc.—that specify the functionality of intervention components that serve a
theoretically-motivated purpose, such as fostering the reactivity of self-
monitoring or increasing individuals’ self-efficacy (Michie et al., 2013).
Yet, BCTs are still highly abstract; they are consensus-based distillations of
the behavior change literature that attempt to systematize what exactly
different interventions _do_. BCTs are defined at a level of generality that
allows for a single BCT to be implemented in many different ways. As such,
BCTs are types of strategies for facilitating change, but these strategies
still need to be made concrete in any given intervention. Michie et al.’s
model tries to account for this by asking intervention designers to specify
the “mode” for each intervention component (delivered in person, via a text
message, etc.). This level of specificity is still far lower than what is
needed to effectively design concrete features of a technology for a mobile
app or a wearable activity tracker. Beyond specifying the mode of delivery,
the Behavior Change Wheel assumes that the specifics of implementation will be
guided by clinical judgment, and thus it provides little additional guidance
on how these specifics should be determined.

###  2.4. Why we need better theory translation methods

Since theory translation is something designers routinely do, one may wonder
if there is really a need for better methods for this process. Certainly, one
way to read the results of our review (section 2.2) is that designers get
along just fine without such methods. We believe there are several reasons why
such methods are important for increasing the impact of HCI work on health
behavior change. First, better methods for theory translation can improve how
we both describe our interventions and report findings, which can strengthen
the replicability of our research (Michie et al., 2013). Replicability is the
cornerstone of the scientific process, and it is especially important for the
health sciences (Dallery et al., 2013; Shneidermana, 2016). Second, better
theory translation methods can lead to interventions with higher theoretical
fidelity—i.e., how faithfully the intervention operationalizes theoretical
constructs or mechanisms it is trying to implement (Rovniak et al., [n.d.];
Raedeke and Dlugonski, 2017). Fidelity is important for both designers and
behavioral scientists. For designers interested in building systems, a higher
fidelity implementation has a higher likelihood to be effective at supporting
behavior change (Raedeke and Dlugonski, 2017). For behavioral scientists
interested in theory development, a higher fidelity implementation increases
the confidence that results from an evaluation of a technological intervention
can be interpreted as evidence for the operation of the hypothesized
theoretical mechanisms. Thus, high-fidelity implementations of theoretical
constructs greatly increase the utility of findings from evaluations of
behavior change technologies. Finally, standardizing theory translation can
speed up scientific progress (Michie et al., 2013). This is because careful
theory translation compels researchers and designers to specify exactly what
constructs they are implementing. Well-specified constructs and intervention
components facilitate comparisons of results across studies, as well as
synthetic analyses like systematic reviews and meta-analyses that can
characterize the effectiveness of different types of interventions for
different conditions and populations (Michie et al., 2013). In summary, robust
theory translation methods could greatly enhance the ability of HCI
researchers and designers to develop technologies that not only work well but
also contribute to the larger scientific evidence base about what works, for
whom, and in what contexts.

The current paper takes the first step toward the development of a
methodological framework for translating constructs and mechanisms from
behavioral theory into concrete features of technological interventions. We do
this by providing a detailed account of how we attempted this process in SARA,
our mHealth application for self-reporting substance use behaviors, in order
to surface the complexities of theory translation and the kinds of issues that
designers have to grapple with in order to design a usable technology which
also aims to implement a set of theoretical constructs with fidelity. Our hope
is that as more such accounts are published, we can begin to synthesize a
method for efficiently engaging in this important aspect of the design
process.

##  3\. Iterative design of SARA: A detailed case study of translating theory
into technological interventions

Following sections describe the iterative design of the SARA app. We
demonstrate how we handled the intricacies of balancing theoretical
considerations with project constraints and user feedback to create an app
deployable in a 30-day clinical trial. To make our theory translation process
more transparent, we first describe theoretical considerations and then the
design choices we made based on these considerations. Before we go into the
details of the design process, however, we describe the final design of the
SARA app to provide context for the later discussion.

Figure 1. Screenshots of SARA application. (a) daily survey, (b) tapping task
(c) virtual aquarium (d) visualization of past data

###  3.1. An overview of the SARA application

SARA (Substance Abuse Research Assistant) was created to support
observational, epidemiological studies on adolescents and young adults (AYA)
who are at high risk of substance abuse. We are interested in this problem
because substance use is a public health issue and AYA are at high risk. A
recent report shows that significant portion of AYA used substances in the
last month (Hedden, 2015)444In the past month, 6.1%,7.4%,7.4% of adolescents
and 37.7%,19.6%,2.8% of young adults respectively reported binge drinking (5
or more drinks), non-medical marijuana use, and misuse of prescription opioids
(Hedden, 2015). Consequences of substance use include hindered brain
development and lifelong compromised decision making (Gruber et al., 2012;
Lopez-Larson et al., 2011; Schepis et al., 2011). However, there is a lack of
fine-grained longitudinal data on how AYA use substances (Shrier et al., 2013;
Shrier et al., 2014; Bonar et al., 2018). While sensors can capture certain
fine-grained behavior (e.g., alcohol use (Bae et al., 2018; Marques and
McKnight, 2009; SCRAM, 2018), ketamine (You et al., 2016)), they cannot detect
poly-substance use (e.g., marijuana or opioids) and important predictors of
substance use (e.g., stress, mood, etc.) (Choe et al., 2017; Sinha, 2008).
Thus, self-report remains a central way to capture AYA substance use data.
However, long-term adherence to self-report is challenging for most mHealth
apps (including substance use apps) (mobile, 2018; Helander et al., 2014;
Badawy and Kuhns, 2017; Majeed-Ariss et al., 2015; Hoeppner et al., 2017;
Cordeiro et al., 2015). Self-reporting can be increased by financial
incentives (Van Berkel et al., 2017; Lynn, 2001) and/or frequent human support
by the research staff (Mohr et al., 2011). But these methods are costly.
Virtual rewards are cost-effective alternatives, but most published studies do
not change virtual rewards over time (Van Berkel et al., 2017; Werbach and
Hunter, 2012; Johnson et al., 2016), risking habituation and potentially
failing to deal with participants’ changing needs (Eysenbach, 2005; mobile,
2018; Lazar et al., 2015; Epstein et al., 2015). Furthermore, published
studies do not provide any principled ways to combine virtual rewards with
other incentives (e.g., money, data-visualizations). The goal of SARA was to
develop a low-cost way to provide the right incentives at the right time to
effectively support ongoing self-reporting.

The SARA version described in this paper was intended for use in a 30-day
study, which is a typical length for many observational substance use studies
(Suffoletto et al., 2012; Shrier et al., 2018; Clark et al., 2010). This
period is also sufficient to examine engagement, as compliance with daily
assessments declines over 30 days in samples of substance-using youth (Bonar
et al., 2018; Buu et al., 2017; Comulada et al., 2015; Suffoletto et al.,
2012; Wen et al., 2017). Thus, focusing on a 30-day study was an efficient way
to iterate on our design decisions while keeping the study duration in line
with other research in this area. Note also that SARA’s goal was not to
investigate whether self-reporting decreases substance use behaviors through
self-regulation or self-reflection (You et al., 2015), which are sometimes
referred to as assessment reactivity in the substance use literature. Although
self-reflection is well-theorized in HCI, such as to increase exercise, the
potential effects of self-reflection are less clear for AYA alcohol or
marijuana use. Most substance-use focused EMA studies focus on tobacco
cessation (Serre et al., 2015), with few studies showing evidence of
reactivity on smoking behavior (potentially due to habituation; (Shiffman,
2009)), and on samples with greater problem severity (e.g., injection drug
users; (Roth et al., 2017)). As reactivity may be lower among those with lower
problem severity (Wray et al., 2014), such as SARA’s target population, we did
not expect SARA to function as a therapeutic intervention. Rather, its goal
was to support regular self-reporting so that the dynamics and determinants of
AYA substance use can be better understood.

Self-reporting in SARA involves completing one survey and two active tasks
each day between 6 PM and midnight (see Fig 1a-b). The survey asks about
emotions (e.g., stress, mood), hopefulness, and reflections about the day
(e.g., amount of free time, level of excitement) (Cranford et al., 2006;
Ramirez-Valles et al., 1998; Hoyle et al., 2002; Lippman et al., 2014). On
Sundays, the survey asks an additional 14 questions about past week substance
use (i.e., alcohol, cannabis, tobacco) frequency and motives, perceived risk
of regular substance use, impulsivity, and behavioral intentions to avoid
substance use in the following week (Johnson et al., 2006; Patton et al.,
1995; Stephens et al., 2002; Simons et al., 1998; Grant et al., 2007). The two
active tasks (Inc., 2018; Mariakakis et al., 2018) in SARA are: a spatial
memory task, in which a random sequence of five seashells light up in a
2-dimensional grid of nine seashells and participants are asked to repeat the
sequence; and a reaction time task involving tapping two buttons alternately
for 10 seconds.  The reaction task measures motor speed and a number of past
studies have shown reaction time changes based on alcohol (Celio et al., 2014;
Nicholson et al., 1992) or marijuana use (Peeke et al., 1976). Spatial memory
task, on the other hand, measures executive function and visuo-spatial memory
(i.e., remembering different locations of objects as well as spatial relations
between objects), and past research has shown similar tasks can predict
alcohol intoxication (Celio et al., 2014) and cannabis use (Nestor et al.,
2008).

SARA uses the operant conditioning theory (Skinner, 2011) to provide a slew of
virtual and monetary incentives to reinforce self-reporting (see Fig 1c-d). A
virtual aquarium fills with fish as daily self-reports are completed. Other
incentives like visualizations of past data, funny or inspirational content,
and small amounts of money are provided periodically to further reinforce
self-reporting. With these various combined incentives, SARA’s self-report
adherence rate was similar to that of a prior AYA substance use study that
used seven times more money (more details in section 3.10). This figure is
especially significant because less than 1% of AYA substance users are
currently under treatment (Abuse et al., 2016) and a low-cost solution like
SARA can democratize data collection at scale.

This encouraging result was achieved largely due to our efforts to translate
the operant conditioning theory (Skinner, 2011) in order to optimize the
frequency and timing of incentives in SARA. This theory translation process,
however, was not straightforward, and we undertook a lengthy iterative design
process (five iteration and four studies) to pick and operationalize 16
constructs from the theory. We ended up working on 233 design elements while
balancing 17 constraints during our process of theory translation (see
Supplementary file 1 for a complete list of design choices, constructs, and
constraints). It is to the details of this process that we now turn.

###  3.2. An outline of our agile approach to translate theory into SARA

Figure 2. Iterative approach to translate theory in SARA. Left to right is the
start and end of the design cycle. As we cycled through several iterations, we
added, improved or removed constructs. Also, earlier evaluation strategies,
shown in the bottom row, used low-cost methods initially and higher cost
methods at later stages of the design process.

A key question for theory translation is what kind of development process the
designer should use. While one can follow the classic waterfall model, we
believe an agile approach is more appropriate for theory translation in HCI
(Hekler et al., 2016). Below, we describe waterfall and agile models, and why
agile models are more applicable for theory translation in the context of HCI.

The waterfall model, as the name suggests, flows in one direction: after a
need finding phase, a collection of features is identified and implemented.
Waterfall models are classic software development models, where each
development cycle is long and each release is feature rich. Waterfall’s
philosophy is “just-in-case” where a large number of features are implemented
to minimize situations where the system cannot support a particular scenario.
But, like any large systems, waterfall models are costly and they are less
adaptable to change. Agile, on the other hand, starts with a minimal viable
product with only a few features, and it then rapidly improves the product by
iterating, evaluating, and adapting to changes. So, agile is more adept at
dealing with newly discovered needs. Agile approaches are also more cost-
effective.

Most health intervention development models (e.g., logic models, COM-B),
including quite recent work (Vilardaga et al., 2018), use a form of a
waterfall model where theory selection and implementation of constructs is
done once, at the start of the design process. While such approaches can be
theoretically rigorous, they provide no built-in methods for early discovery
of failures of construct operationalization or the need to incorporate
additional constructs. Agile approaches are better suited for addressing such
issues, especially in the context of HCI work, for several reasons: (i)
development resources for novel HCI interventions are typically limited, and
agile is more cost-effective, (ii) the need for changes are commonly
discovered in HCI studies of behavior change applications: users get
habituated or bored, rewards as designed are not found to be sufficiently
rewarding by the target group, the application ends up creating more user
burden than anticipated, etc. Rapid iterations can efficiently address such
findings; and (iii) agile’s iterative approach is similar to the iterative
design approach in HCI, allowing HCI researchers and designers to align their
design and theory-translation work.

In SARA, we used agile principles in two important ways. First, we added,
removed, and refined theories or theoretical constructs iteratively. We did so
to address an important challenge of intervention design, choosing what
constructs to implement from a large collection of theories/constructs from
the behavioral change literature (nearly 83 theories and over 1600 constructs
(Michie et al., 2014b)). A concrete implementation of any abstract
theory/construct requires financial resources, requiring judicious selection
of intervention components, especially in the context of resource-constrained
HCI work. To deal with this issue, in SARA we started with a minimal set of
constructs. We then made low-fidelity prototypes of these constructs, and used
low-cost participant feedback methods to evaluate them. After the initial
evaluations, we identified which constructs should be carried forward for
higher fidelity implementations, how they should be revised, and which
additional constructs should be added to create a better intervention. The top
portion of Figure 2 shows the theory/construct selection process used in SARA.

The second challenge that an agile approach to theory translation helps
address is how to assess theoretical fidelity: i.e., to what degree the
implementation represents the underlying theory or construct. By focusing on
fast, iterative evaluation, agile provides guidance on how to sequence
different kinds of evaluations to check how effectively constructs have been
implemented and if there is a need to implement additional constructs. Here, a
tradeoff the designer has to make is between more informative user studies
(e.g., a pilot study measuring actual use and health behaviors) that are also
more costly and require more time, and faster, cheaper, but less rigorous
evaluations. Since uncertainty is high in early stages of the design process,
high cost methods can be wasteful. In SARA, we initially used low-cost methods
like design reflection, online surveys, and focus groups that did not involves
actual use of the app. For instance, in the SARA online survey, we asked users
to rate perceived benefits of gamification, rewards, etc., without showing the
app screenshots. In the focus group, we showed the app to users to get
feedback about the current design of different theory-based features, but the
participants did not use the app in their day-to-day lives. Only once several
design iterations were done based on low-cost feedback, that we moved to more
expensive means of validation, such as pilot and evaluation studies. These
studies focused on the actual use of the app, and we also gathered qualitative
feedback to gain insights for future improvements in our construct
operationalization.

A final consideration is sample size. The appropriate sample size depends on
multiple factors, including the level of evidence required at each stage of
the design process, the effort required by participants and study staff, and
the constraints on time and other resources. Table 1 shows the trade-offs of
different user studies, and the bottom portion of Figure 2 shows the different
user studies we used at different phases of SARA’s design.

Evaluation method/ | cost/ | participant | cost/study | uncertainty in | Actual | Sample | when used in
---|---|---|---|---|---|---|---
user study | participant | burden | staff | design cycle | use | size | design cycle
Desiger’s reflection | none | none | none | high | no | – | early on
Online survey | low | low | low | high | no | $N\geq 100$ | early on
Focus group | low | moderate | moderate | high | no | $N=20-30$ | early on
Pilot study | high | high | high | moderate | yes | $N=8-20$ | later on
Evaluation study | high | high | high | low | yes | $N\geq 30$ | later on
Table 1. Types and relative trade-off of different user study types

###  3.3. Iteration 1: A virtual aquarium

Theoretical considerations: The first challenge of developing SARA was to
choose a theory that can be useful for supporting self-report adherence.  We
looked at the substance use literature for theories to improve self-report
adherence. However, as a recent meta-analysis in substance use research
suggests, engagement theory is limited to the provision of monetary rewards
(Jones et al., 2019). While gamification (Boyle et al., 2017) and data-
visualization (You et al., 2015) have been used in substance use research,
they were used to reduce alcohol use behavior and not to improve self-report
adherence. Due to this lack of guidance on how to support self-report
adherence in substance use research, we decided to look at the theories from
psychology. While several theories are applicable (Fogg, 2009; Petty and
Cacioppo, 1986), we chose the Operant Conditioning Theory (OCT) (Skinner,
2011; Reynolds, 1975; Staddon and Cerutti, 2003) for two reasons: (i) OCT
provides a detailed account of how consequences of prior actions influence the
frequency of future behavior. Most importantly for our purposes, OCT describes
how positive reinforcement (valued consequences, rewards) can be used to
increase the frequency of a target behavior. This directly matches SARA’s aims
of increasing the frequency of self-report completion; and (ii) OCT is a well-
developed theory. First proposed in the 1930s, over the years, OCT has become
one of the most precise and well supported theoretical accounts of learning.
OCT has been successfully applied to a broad range of problems, from animal
training (Staddon and Cerutti, 2003; Ferster and Skinner, 1957), to education
(Weegar and Pacis, 2012; Phillips, 2012; Shield, 2000; Villamarín-Salomón and
Brustoloni, 2010), and psychotherapy (Wolpe, 1968; Wolf et al., 1963; Burchard
and Tyler Jr, 1964).

OCT describes a number of constructs that determine how reinforcement affects
a target behavior. Three of these core constructs are: (i) value of the
reinforcement: extensive literature on Matching Law (Hodos, 1961; Herrnstein,
1970) demonstrates that organisms allocate their behavior in proportion to the
perceived value of reinforcement associated with different behavioral choices.
An effective way of increasing the frequency of a target behavior is to
reinforce it with something the organism finds valuable; (ii) immediacy: a
reinforcement delivered immediately after a behavior is more effective than a
reinforcement that is more temporally distant. The main reason for this is
that organisms discount the value of reinforcements based on their temporal
distance; i.e., the same reinforcement is perceived to be more valuable if it
is received sooner (Simpson and Vuchinich, 2000; Ainslie, 2001); and (iii)
contingency: reinforcement that an organism can clearly associate with a
behavior influences that behavior more strongly than a reinforcement that
could have resulted from multiple behaviors. A way to ensure contingency is to
provide reinforcement only after the desired behavior (Miltenberger, 2011).

Figure 3. Iteration 1 of SARA. Left image shows the 6 fish and their stages of
growth. Middle and right columns show the aquarium.

Design considerations: In designing SARA, we first tried to translate the
above-mentioned three OCT constructs: value of reinforcement, immediacy and
contingency. Translating immediacy and contingency was straightforward: we
could provide the reinforcement immediately and only after self-report
completion. Translating the notion of a valuable reinforcement turned out to
be more complex, however. While in animal research a strong reinforcement is
routinely achieved by using food, the use of such strong reinforcement (food,
shelter, personal safety) is ethically unacceptable in most human-subjects
research.555Exceptions occur in special circumstances: e.g., in reinforcement-
based therapy, abstinence in drug-dependent patients is reinforced by
providing housing, job training, etc. (Tuten et al., 2012). In research with
people, the most common form of reinforcement is money (Ariely and Jones,
2013), and money has been used effectively to motivate self-report completion
(Van Berkel et al., 2017; Lynn, 2001). However, since SARA’s goal is to reduce
money in order to make data collection more scalable, we initially tried to
focus on non-financial reinforcements.

A common non-financial approach to reinforcement is gamification (Deterding,
2012; Deterding et al., 2011), where game-like elements such as points,
badges, progression, levels, and leaderboards are provided (Van Berkel et al.,
2017; Werbach and Hunter, 2012; Johnson et al., 2016; Ariely and Jones, 2013;
Deterding, 2012; Deterding et al., 2011; Zuckerman and Gal-Oz, 2014). Several
gamification features satisfied our design goals and we included them in SARA
(more details below). However, we chose not to use social features such as
leaderboards for two reasons. First, if study participants were recruited over
time, as is common in clinical trials, different participants would be
potentially exposed to a very different leaderboard based on when they were
recruited (i.e., it’s not clear that a leaderboard with 3 people and 150
people are the same intervention). Second, social features can have unintended
adverse effects among young substance users (e.g., social undermining (Fogg,
2009) or negative contagion (Joiner Jr and Katz, 1999; Prinstein, 2007)), and
planning for and managing these was beyond our financial resources.

Without a leaderboard, we suspected that points by themselves would not be
perceived as particularly valuable. However, since points were attractive for
other reasons—they could scale over time, they were free, and could be
adjusted to deferentially reinforce both individual acts of self-report
completion and patterns of adherence over time (see below)—we attempted to
increase the perceived value of points. To do so, we decided to create a way
to convert points into something that participants would find interesting and
attractive. After extensive design ideation on different types of
representations (e.g., virtual pets, avatars, growing tree, space objects), we
settled on the idea of creating a virtual aquarium that would be populated
with fish as points accumulate.

The aquarium representation had a number of attractive features that made it a
good candidate for reinforcement. First, unlike many other representations,
fish (and an aquarium) were positively received by both men and women, as well
as by individuals of varying ages (Lin et al., 2006; Lane et al., 2011).
Second, the representational language of the aquarium was quite rich, allowing
us to provide a large number of interesting reinforcements and, thus, scale
the representation over time. Finally, aquarium representations had already
been used successfully in the mHealth setting: Fish’n step (Lin et al., 2006)
and BeWell (Lane et al., 2011) used aquariums to promote healthy activity, and
Abyssrium, a mobile game that involves growing a fish population, has been
downloaded over 30 million times and received a game of the year award in 2016
(Abyssrium, 2018). As such, we had strong preliminary evidence for the
feasibility and acceptability of an aquarium representation. Figure 3 shows
the first prototype of the SARA application. We had six different fish and
each fish had four stages of growth. Each time a participant completed a self-
report, he or she would earn 200 points and a fish would go through one stage
of growth. Once a fish went through four stages of growth, we considered the
fish fully grown and added it to the aquarium.

Design reflections: Our initial prototype translated the concept of
reinforcement using points, aquarium, and fish. We also combined points with
fish to create what we hoped would be a more valuable reinforcement. These
reinforcements were to be provided immediately and only after self-report
completion to maximize their effect.  Regarding evaluation of these ideas, we
deferred evaluation using online surveys or focus groups at this stage because
we were at an early stage of design and the number of features in SARA was
small. We wanted to use more costly focus groups and online surveys when we
had more features to investigate, so at this stage we decided to only use
design reflection and feedback within the research group. During this
evaluation, we quickly realized that in this first iteration we did not
consider how our reinforcements should be scheduled so they remained effective
at promoting self-report completion over time (Ferster and Skinner, 1957). The
next iteration tried to address this issue.

###  3.4. Iteration 2: Reinforcement schedules

Theoretical considerations: Extensive research in OCT shows that different
schedules, or timing, of reinforcement can produce different behavioral
effects (Ferster and Skinner, 1957). Consider the simple case of satiation,
where a reinforcement is received too often in a short period of time and it
temporarily loses efficacy. For instance, satiation can happen when an animal
has eaten enough and is no longer hungry, and the availability of additional
food temporarily loses the ability to influence the animal. The same
phenomenon is found in humans: after binge-watching our favorite TV show, an
opportunity to watch more of it may not be perceived as particularly
desirable, no matter how much we love the show (Staddon and Cerutti, 2003). In
cases of satiation, temporarily reducing the frequency of reinforcement can
give an organism time to re-sensitize to it, and reinforcement then regains
its ability to influence behavior.

Another key idea is that reinforcing behavior intermittently can lead to high
levels of responding. Intermittent reinforcement can follow a fixed or
variable-rate schedule. A fixed-rate schedule means that reinforcement is
provided each time a fixed number of instances of the target behavior is
completed. A variable-rate schedule means reinforcement is given after a
variable number of target behaviors, but with the mean number of target
behaviors before reinforcement held constant. Reinforcing less often using a
variable schedule can generate similar frequencies of target behavior as a
fixed schedule that reinforces more often. The variable-rate schedules have
this effect due to uncertainty and anticipation (Ferster and Skinner, 1957).
However, when a new behavior is learned for the first time, reinforcing more
often using a fixed schedule can result in faster initial learning (Ferster
and Skinner, 1957). Furthermore, the perceived value of reinforcement matters.
A less valuable reinforcement needs to be used more often—i.e., after fewer
occurrences of target behavior—compared to a more valuable reinforcement
(Trosclair-Lasserre et al., 2008).

Design considerations: We translated the above mentioned theoretical insights
by adopting a fast, fixed-rate schedule, where we provided (almost) one fish
for each day of self-reporting. We chose this schedule because (i) self-
reporting is a new behavior and reinforcing more often induces faster
learning; and (ii) we suspected fish may not be perceived to be a very
valuable reinforcement, so we needed to use them more often to increase self-
report. Lacking evidence for how valuable fish would be perceived to be, we
opted to be conservative and assume they would be of low relative value and
would require a fast schedule. Now, a potential side-effect of frequent
reinforcement is that it can induce satiation. To prevent satiation, we
included a gap day after every 4th to 5th fish, much as games use satiation
and deprivation sequences to increase engagement (Zichermann and Cunningham,
2011).

We had to significantly redesign SARA in order to incorporate this fast
reinforcement schedule for fish. First, we retired the six fish from iteration
1 (Figure 3) because in order to follow a fast schedule of one-fish-a-day, we
would have needed to start reusing previously given fish after the first six
days. Recycling old fish would mean that the reinforcement would become less
novel and hence potentially less valuable. We replaced the six fish from
iteration 1 with 25 unique fish (see Figure 4) so that a different fish could
be awarded for almost each day of self-report completion in a 30-day study.

Figure 4. Top two rows show 25 different fish in Iteration 2. Bottom row shows
different stages for the growing aquarium. a,b,c,d,e respectively shows the
state of aquarium if a participant logs for 1,7,14, 24, 30 days.

In addition to reinforcement schedules, in this iteration we also attempted to
increase the value of fish in several ways: (i) we used animated and better
looking fish, because better aesthetics can improve reward value (Perski et
al., 2016); (ii) we displayed a fun fact about each fish when it was unlocked.
For example, when the goldfish was unlocked, we showed a fun fact “Do you know
goldfish can recognize faces?” The aim of the fun facts was to generate
curiosity and thus increase the reward value of the fish; and (iii) we
provided an outline for the next fish to be unlocked to increase anticipation
and feeling of progression. Finally, a potential challenge for the one-fish-a-
day schedule was that the aquarium could get over-populated. Therefore, we
introduced levels as is commonly done in games (Zichermann and Cunningham,
2011). For the 30-day planned study for SARA, we created two levels:
participants began with a fish bowl environment level and unlocked fish, such
as goldfish, which are commonly seen in household aquariums. After 15 days of
self-reporting, participants graduated to a sea environment and unlocked sea
creatures, such as sharks and dolphins. Figure 4 shows iteration 2 of SARA’s
design and a progression of the aquarium over a 30-day study.

Design reflections: The major improvement in this iteration, from a
theoretical perspective, was the explicit inclusion of a reinforcement
schedule. However, due to our lack of knowledge about how valuable the fish
would be perceived to be, we selected a fast, fixed-rate reinforcement
schedule. But the use of a fast schedule also created problems. Even for a
relatively short 30-day study, this schedule necessitated the inclusion of a
lot more fish, as well as a way to deal with aquarium overcrowding. The later
issue, in particular, is a good example of how theoretical concepts must be
considered in light of design constraints, in this case limitations related to
the number of fish that could be comfortably shown on a mobile phone screen.

For evaluation of the design ideas in Iteration 2, we brainstormed whether we
were missing any obvious features because adding those features would give us
an opportunity to ask about them in an online surveys and focus groups. So,
here again design evaluation was done via design reflection and feedback
within the research group. During the design reflection we realized that even
with our best efforts to increase the reward value of fish and include
reinforcement schedules, we would need additional forms of reinforcements due
to the burden of ongoing self-reporting. In mHealth, the use of multiple
reinforcement strategies is quite common. Nearly all commercial mHealth apps
and health-gamification research projects use two or more kinds of
reinforcements (e.g., stars, badges, etc.) to promote health behaviors
(Fitbit, Inc., 2013; Johnson et al., 2016). In addition, inclusion of
additional reinforcements would also allow us to introduce other types of
reinforcement schedules, which could more effectively target sustainability of
self-reporting behavior. Thus, for the next design iteration, we focused on
introducing additional forms of reinforcement.

###  3.5. Iteration 3: Additional reinforcements and conjoint schedules

Theoretical considerations: For guidelines on how to schedule multiple types
of reinforcement, we again turned to OCT. We found that OCT supports the idea
that multiple types of reinforcement can be more valuable together than a
single reinforcement. One piece of evidence for this came from Hursh (Hursh,
1980), who proposed a microeconomic conceptualization of OCT. Hursh posits
that demand for reinforcement is created after a target behavior happens; the
obtainable reinforcement is the supply for meeting that demand. Hursh argues
that multiple reinforcements are additive and increase the overall supply.
Another, less obvious part of Hursh’s argument is that multiple types of
reinforcement can create a substitution effect: if one type of reinforcement
is ineffective but other types of reinforcement are available, they can
substitute for it and still maintain the effectiveness of the overall
reinforcement supply. For instance, at the start, the fish may not be seen as
particularly rewarding since the aquarium is mostly empty and participants are
learning its value to them. Giving some money, which most people care about,
may reduce the demand for the fish to be immediately highly reinforcing.
Usefulness of multiple types of reinforcement was further supported by the OCT
literature on conjoint reinforcement schedules.This literature suggests that
conjoint schedules generally increase target behavior more than single
schedules, unless the rate of reinforcement is so slow that the amount of
reinforcement from single and conjoint schedules cannot be distinguished
(Weatherly et al., 1996).

Design considerations: As with the initial design iteration, a key design
question was what to use for this new form of reinforcement and how to
schedule it. Given that providing some form of financial incentive is common
in most studies, we decided to use small financial reinforcements that would
not compromise scaling up SARA for larger studies. However, we had to make the
financial reinforcements coherent with the overall gamified theme of the SARA
app. We decided to tie monetary reinforcements to badges: if participants
completed several back-to-back days of self-report, SARA would award them
badges that came with small monetary incentives. Specifically, if participants
completed daily surveys three days in a row, or they completed active tasks
three days in a row, they received a badge and 25 cents. For longer streaks,
participants could earn different badges and higher monetary incentives. SARA
could reward 3, 6, 12, 18, and 30-day streaks; for these streaks, participants
could earn 0.25, 0.50, 1, 2, and 3 dollars, respectively. Under this schedule,
if we assume 90% adherence, then there was less than 5% chance that
participants would earn more than 10 dollars in a 30-day study. This amount is
lower than most substance use studies that generally pay $1-4 per day of study
participation (Bonar et al., 2018; Buu et al., 2017). We hoped that this
conjoint schedule of being able to receive money every 3 days and fish nearly
every day (from iteration 2) would be fast enough so that its effect would be
greater than a single schedule of either money or fish (Weatherly et al.,
1996).

Design reflections: The primary challenge of this design iteration was how to
structure the schedule of financial reinforcements. We decided to provide more
money for longer streaks; this progressive monetary schedule was intended to
encourage longer stretches of daily self-reporting. At the same time, we
wanted to keep the application scalable, so we opted for the smallest amounts
of money that we thought might still be effective in reinforcing self-
reporting in the AYA population.

Up to this point, the app design was based on our best attempt to translate
several principles from OCT in a way that took into account various
constraints: target population, the app’s scalability etc. Along the way, we
made a number of assumptions and design choices that we thought would help us
to effectively translate OCT. When we got to this stage of design, we felt
that, given the considerable number of theory-driven features we had
developed, we needed to bring in user feedback to check our assumptions and
design choices. To do this, we conducted our first two formative studies–an
online survey and a focus group study.

### Formative user feedback

We conducted two formative studies to gather early developmental input on
SARA. The first study involved an online survey and the second study was a
focus group. Both studies targeted undergraduate students of University of
Anonymous who were within our target age group.  Since we were still early in
the design cycle, we did not opt to use higher-cost methods, such as a study
focusing on actual use. Furthermore, since online surveys are lower-
participant-burden and less expensive than focus groups, we decided to do an
online survey first. Given the low cost of conducting a survey, we aimed to
recruit at least 100 participants, to increase the confidence in any potential
findings. We further reduced evaluation costs by using the same online survey
to recruit focus group participants as we describe below.

###  3.6. User study 1: Online survey (N=124)

The online survey intended to gather information on perceptions of different
reinforcement types from a sample of our target population. After asking a set
of questions about demographics and prior experience with mHealth apps, the
survey asked the participants to rate the likelihood of how money,
progression, and unlockable features would motivate them to self-report
regularly. We initially also intended to ask questions about risky substance
use. However, since many responders could be under the age of 21, the IRB
needed special approval to include substance-use-related questions. Since
binge drinking and marijuana use is common in college students,666Among
undergraduates at University of A, 53.6% reported binge drinking and 34.4%
reported marijuana use in past 3 months [–] we decided to drop these
questions.

Results: We used the university registrar to send the survey to a random
sample of 2000 undergraduate students. Of the 280 students who started the
survey, 124 (38% male) answered the questions regarding different types of
reinforcement. Mean age of these participants was 19.9 (SD=2.59). There were
no significant differences in demographics between those starting the survey
and those completing the survey. A 4-point Likert scale was used to rate the
anticipated impact of reinforcements on self-report in mHealth apps. The scale
ranged from 1=not likely to 4=very likely. As we can see in Figure 5a-5c,
money was rated as the most likely reinforcement to influence self-reporting
$(\mu=3.72,\sigma=0.53)$, followed by gamification features such as points
$(\mu=2.57,\sigma=0.98)$ and unlockable features $(\mu=2.73,\sigma=0.93)$. The
high ratings of points, unlockable features, and money suggested that our
target population might perceive these reinforcements as valuable, providing
preliminary evidence for our efforts to operationalize reinforcement.

###  3.7. User study 2: Focus groups (N=21)

The second formative study focused on getting in-depth qualitative feedback on
SARA’s design. We invited 21 participants (47% male) from the 124 respondents
of the online survey for focus groups.  During the recruitment, we balanced
gender, ethnicity, and age. We held three one-hour-long, semi-structured,
mixed-gender focus group sessions $(N=7,5,9)$. All sessions were audio
recorded. Our initial goal was to recruit 30 participants as is often done in
focus group studies (Schueller et al., 2018; Metting et al., 2018). Given the
early stage of the design process, however, we decided to prioritize resource
efficiency. So, we kept the number of sessions to three and recruited 21
participants. During each focus group session, after a few ice-breaking
questions, participants received phones with the SARA app installed.
Moderators then demonstrated how to self-report in SARA and showed the
different reinforcements of SARA (aquarium, levels, money, etc.) using a
storyboard. We created temporary buttons which could be pressed to simulate
daily self-reports and give a feel for how the aquarium evolved and money was
rewarded. After participants played with the SARA app for some time, they
answered questions about which features they liked or disliked and what
additional features would make SARA feel more rewarding. After the focus
group, participants completed a debriefing survey on how likely they thought
money, aquarium, progression, and fish would be to affect self-reporting.
Since the debriefing survey was anonymous, participants also reported past
3-month alcohol and marijuana use.77761.9% and 28.5% focus group participants
reported they binge drank and used marijuana, respectively, in the past 3
months. Participants received 20 dollars for participation.

Results: We analyzed the focus group discussions using thematic analysis
(Braun and Clarke, 2006). In the following, we present a summary of the
results that are relevant to translating OCT.

Figure 5. Distribution of ratings for various reinforcement types. Top row is
for online survey (User study 1) with a 4-point Likert scale (1=not at all,
4=very much). Middle and bottom rows are for debriefing survey of focus group
(User study 2) and follow-up survey after a 30-day pilot trial (User study 3),
which use 3-point Likert scales (1=not at all, 3=very much)

.

Fish and aquarium: When asked about the fish and aquarium, participants
appreciated the aquarium theme and characterized the aquarium as relaxing and
calming. Participants gave mixed responses when asked how they would like the
aquarium to be improved. Some participants mentioned that the app’s appearance
was not as polished as other commercial apps; some participants wanted more
consistent animations of fish. As one participant stated, “I guess it’s mildly
off-putting to me that the fish are all kind of different kinds of animation,
some look more like clip art, some look more like actual fish kind of a deal.”
Other participants were more accepting of the fish animations; one participant
stated, “you could make them goldfish, like literally just pure gold, or
diamond fish, or platinum fish. [chuckle] Because people aren’t gonna care.
It’s an app.”. Participants also liked when the aquarium changed levels to the
sea environment. For example, as one participant stated,“I think it’s a good
concept. I think it’d be really cool… you graduate from this aquarium and go
to the next.” We also provided an outline of the next unlockable fish and
several participants mentioned the outline created anticipation. Like the
survey, these observations indicated that participants thought they would find
the aquarium and the fish to be rewarding, providing support for our efforts
to design non-financial reinforcements AYA found valuable.

Participants also appreciated how the aquarium progressed over time. One
participant mentioned that “it’s really good that there’s an objective you can
work towards and you can see what’s happening each day.” However, participants
found the aquarium more appealing when it was full. As one participant stated,
“If you don’t have that many [fish] then it’s kind of a bland thing to look
at. What I’m looking at right now, I only have one fish. If I have something
like that [fuller aquarium], it’s more pleasing to the eye.”. These comments
provided initial evidence for our decision to use a fast, fixed-ratio
schedule, but indicated that the fish schedule may not be sufficiently fast.

Money and streaks: When we asked about money, participants unanimously agreed
that money was a huge pull, particularly in their age group. As one
participant noted, “I think the monetary reward is one of the highest things I
think for our age group, that should definitely be pretty clear and specified
at the beginning. That is probably one of the main things that’ll keep people
coming back.” As expected, the focus groups supported the idea to use money
for reinforcement. Furthermore, participants provided no indication they
thought that money would conflict with the aquarium, providing preliminary
support for our decision to pair these forms of reinforcement (Iteration 3).
However, one participant pointed out that the streaks may be too hard to
complete and that, once interrupted, there was no way to recover a lost
streak. Given how monetary incentives were structured, he said, losing a
streak, especially a longer more valuable one, would feel like punishment.
Although only one participant commented on this issue, it raised the question
of whether the schedule of monetary reinforcements would behave as we intended
it to, encouraging regular, uninterrupted self-reporting.

Additional novelty: A majority of participants wanted more novelty.
Participants wanted additional themes such as gardens, jungles or car
collections. One participants said “It would be cool to be able to unlock
other little worlds… Maybe a little forest or something? Or a garden, you
could do butterflies or something. Everybody likes butterflies… Maybe for
people who like bugs or something, you can do a bug garden if that’s even a
thing…” When the moderators mentioned the possibility of having visualizations
of personal data, participants were enthusiastic about the idea: “It’d be cool
if there were like a reinforcements page and then also data but kind of… Like
how the Apple health app gives you graphs and stuff like that. I think that
would be kinda cool… Put data like mood, maybe as a graph for energy level.”
We interpreted these observations as indicating that we had not over-saturated
SARA with reinforcements and that AYAs thought there was still a need for
additional forms of reinforcement.

Choice and interactivity: When asked about what features they would like to
change, participants wanted more control over the aquarium and to be able to
interact with the fish. One participant wanted to change the background color
of the aquarium. Several participants wanted to feed the fish and for the fish
to do something interesting (e.g., tell a fun fact) when they touched the
fish. Participants also wanted to have control over their points and be able
to use their points to get the fish they preferred or restore a broken streak
of money. They also wanted to choose at what level they would start. These
findings raised two interesting possibilities for operationalizing
reinforcement: first, giving choice meant people could choose the
reinforcements they found valuable, providing a way to deal with the
heterogeneity in people’s preference. And second, participants’ comments
indicated that interactivity might increase their ownership over the aquarium,
which, in turn, would and make it feel more valuable (Sundar, 2007). We
considered these lessons in future design iterations.

Debriefing survey: Following the focus group, participants answered a
debriefing survey, where they reported how they thought different
reinforcements would increase a participant’s use of the app. All the ratings
were recorded on a three point Likert scale: 1=not at all, 2=somewhat, 3=very
much. Figures 5d-5f show the results. Money $(\mu=2.95,\sigma=0.22)$ was again
rated the highest, followed by unlockable fish $(\mu=2.45,\sigma=0.59)$, and
points $(\mu=2.38,\sigma=0.58)$.

Formative study summary and design reflections: The two formative studies
provided initial positive evidence for how we translated OCT constructs.
Participants generally liked the representations we developed (aquarium, fish,
etc.), as well as the financial incentives, which indicated that these
reinforcements held promise for reinforcing self-reporting. We also learned
that providing choice and interactivity were additional ways of increasing
reinforcement value. Furthermore, participants liked having several
reinforcement types, which provided support for Hursh’s reinforcement supply
idea (Hursh, 1980) and indicated that we might be able to effectively combine
different forms of reinforcement using a conjoint schedule. Participants
appreciated the fast schedule for fish, providing support for the use of such
a schedule at the start of the study, in line with theory suggestions.
However, we got an indication that the schedule of financial incentives might
be problematic since larger monetary reinforcements were tied to long
uninterrupted streaks which participants felt would be difficult to achieve.
Given, though, that focus participants were just trying to imagine how they
would experience the monetary schedule, it was difficult to tell how it would
perform in a deployment.

###  3.8. Iteration 4: Memes and life insights

Design constraints and theoretical considerations: While our formative studies
provided us with a number of insights about how to improve the perceived value
of reinforcement in SARA, we could not implement every request due to resource
constraints. In particular, both substantial improvements in the app’s
aesthetics and implementation of reinforcement choice and interactivity
required additional designers and developers, which were beyond our level of
development resources. Adding novelty, however, was easier to implement
because we could simply add more reinforcement types without extensive
development.

Design considerations: In this design iteration, we introduced two new types
of reinforcements to create additional novelty. The first kind was what we
called life insights; life insights are visualizations of past tracked data.
We decided to include life insights because focus-group participants wanted to
see their past data and prior mHealth work found that seeing patterns in one’s
own data was intrinsically motivating and could encourage regular self-
tracking (Vorauer, 2006; Van den Bos, 2009; Nahum-Shani et al., 2014a;
Leonardelli and Lakin, 2010; Weary and Jacobson, 1997). We created seven
different life insights that visualized past week’s (i) stress, (ii)
loneliness, (iii) level of fun, (iv) how new and exciting their days were, (v)
free hours each day, (vi) tap count, and (vii) the number of seconds required
to complete the spatial task. The data for life insights were pulled from the
daily surveys (i-v) and active tasks (vi-vii).

The second kind of reinforcement we introduced were memes. We included memes
because they are widely available on the internet and are popular among SARA’s
target age group. Memes can also make people laugh and evoke positive
emotions, which are powerful intrinsic rewards (Perski et al., 2016; Miller
and Rose, 2009). To enhance the novelty of this reinforcer, we incorporated
two types of memes: funny and inspirational. 120 memes were compiled and
filtered by Amazon MTurk workers and undergraduate research assistants to
increase their relevance to AYA.

A question that remained was how to schedule memes and life insights. We
decided to use a variable schedule, where (i) a meme would be provided with
0.5 probability if a daily survey was completed and (ii) a life insight would
be provided with 0.5 probability if the two active tasks were completed. We
decided on a variable schedule because we did not want to satiate participants
with too frequent reinforcement. In addition, using variable schedules for
these reinforcements let us micro-randomize their delivery to empirically
decide on the best schedule in a post-study analysis (Klasnja et al., 2015;
Boruvka et al., 2017).

Figure 6. Examples of funny memes (left), inspirational memes (middle) and a
life-insight (right).

###  3.9. User study 3: Pilot trial (N=13)

At this point, we pushed SARA’s design as far as we could without empirical
data on how participants would experience and respond to the various
components of SARA. To make further changes, we needed to understand how the
reinforcements we designed would affect self-reporting and how their effects
would change over time.  Furthermore, while we could have run focus groups to
test the features we added in iteration 3, we thought that the number of new
features from iteration 3 was too small to warrant investing in another set of
focus groups. Instead, we opted to move directly to a 30-day pilot study.
Since further changes may have been needed based on how people respond to
SARA, we first did a pilot study before conducting a larger scale evaluation
study. The sample size of such pilot studies are typically 8-20 people
(Consolvo et al., 2008b; Lane et al., 2011; Burns et al., 2011). We recruited
16 participants, but three initial participants dropped out due to a software
bug that was unrelated to the theory-based incentives in SARA. So, the final
sample size was 13.

Pilot study participants were recruited from the University of Anonymous
Hospital Pediatric and Adult Emergency Department. Patients were eligible (i)
if they were between the ages of 14-24, understood English, medically stable,
able to provide informed consent/assent (e.g., not cognitively
impaired/intoxicated), and accompanied by a parent/guardian (for participants
between the ages of 14-17), (ii) screened positive for binge drinking (¿4
drinks female, ¿5 drinks male; (Bush et al., 1998)) in the past month, or any
past-month cannabis use without a medical marijuana card. Then a research
assistant installed SARA on the participant’s phone and demonstrated how to
use the app. After the 30-day study, participants completed a 45-60 minute
telephone interview, where we asked a sequence of close and open-ended
questions about their experience with SARA (Hoeppner et al., 2010; Stoyanov et
al., 2015).

Results: Recruiters approached 241 individuals out of which 26 participants
met study inclusion criteria and 17 were enrolled in the study. The average
age of participants was 21.2 years $(\sigma=1.9$, range=18-24, 60% male). 45%
screened in for past-month binge drinking only, 35% for past-month marijuana
use only, and the rest for using both substances. Four participants dropped
out of the study due to software bugs or malfunctioning phones. We excluded
these participants from the following analyses since their adherence was not
related to the reinforcements in SARA.

Adherence: Figure 9a shows participants’ adherence over the 30-day study.
Participants counted as being adherent if they answered the daily survey and
active tasks for the day. We found adherence to decrease with time: adherence
was 63.8% for days 1 to 10, 49.2% for days 11 to 20, and 34.6% for days 21 to
30. We also identified 3 clusters of participants: (i) 5 out of 13
participants, represented as green in Figure 9a, were engaged for the entirety
of the study and self-reported for more than 20 days (mean=25.4 days); (ii) 4
participants, represented as blue in Figure 9a, self-reported on more than 10
days (mean=12.5 days). Two of these participants provided no data past the
15th day of study participation, while the other two provied no data after the
23rd day in the study; and (iii) 4 participants, represented as red in Figure
9a, self-reported on fewer than 10 days (mean=3.75 days), and only one of them
provided any data past the 13th day in the study. While SARA clearly did not
work for this last group of participants, the presence of both the second and
the third group indicated the need for further design refinements to reduce
these types of non-adherence.

Monetary incentives: On average, participants earned $4.60
$(\sigma=5.5,q_{50}=2.2)$. Most participants failed to complete longer streaks
and earned less money as a result: the total number of 3, 6, 12, 18, and 30
daily streaks completed was 22, 10, 6, 3, and 1, respectively.888Note that
when a longer streak was completed, we did not count its shorter constituent
streaks. For instance, for a 12 day streak, we did not count the 3 or 6-day
streaks that were completed on the way to the 12-day streak. The low number of
long self-reporting streaks suggested, in line with the focus-group
participant’s intuitions, that our initial schedule of financial
reinforcements was ineffective for encouraging long, uninterrupted periods of
self-reporting.

Between self-report distance: We also measured the gap between two successive
self-reports. If our implementation of schedules of reinforcement in Iteration
2 worked as intended, then we should see regular self-reporting and the intra-
day gap between self-reports would be small. Figure 9b shows the distribution
of day gaps between two successive self-reports; these numbers are for
participants prior to finishing the 30-day study or prior to their complete
disengagement (i.e., before they stopped using the app completely). The gap
was one day in 78.2% of cases, two days in 13.9% of cases and three days in
3.91% of cases. Of the remaining 3.91% of the cases where the gaps were 4 days
or longer, 71.4% of the gaps were for people who replied fewer than 10 times
in total during the study (i.e., the red participants in Figure 9 top). For
participants in blue and green clusters, the gaps between responses were low,
which indicates that SARA’s reinforcement schedules likely influenced regular
self-reporting as OCT suggests, at least to some extent.

Figure 7. (a) Number of days participants self-reported in a 30-day study. The
colors represent three clusters. The green, blue and red color participants
self-reported for ¿20, 11-19, and ¡10 days, respectively. (b) Distribution of
the gap between two consecutive self-report days.

Follow-up survey: After the study, we asked participants to rate how they
thought various reinforcements influenced their app use over the 30 days. For
the new reinforcements from Iteration 4, memes and life insights, participants
rated their perceived influence on a five point scale, from 1=not at all to
5=very much. Life insights $(\mu=3.92,\sigma=0.8)$ and memes
$(\mu=3.8,\sigma=0.5)$ were rated similarly. These results indicated that that
the delivery of memes and life insights was perceived as being rewarding after
they were experienced in use, whether or not they actually changed
participants’ behavior. For other reinforcements, we used the same three point
Likert scale we used in the debriefing survey after the focus groups, where
1=not at all, 2=somewhat, and 3=very much. Figure 5g-5i show the results.
Participants rated money most highly $(\mu=2.7,\sigma=0.49)$, followed by
points $(\mu=2.4,\sigma=0.9)$, and the fish $(\mu=2.2,\sigma=0.7)$. Note that
these ratings are on average lower than those obtained from the focus group
debriefing survey, where participants did not use SARA in their daily lives
(see Figure 5 for a visual comparison). There are several possible reasons for
these lower ratings after participants used SARA for 30 days: (i) they may
indicate that the reinforcements lost their value over time (e.g., due to
habituation or wearing off of novelty); or (ii) focus group participants could
not accurately predict how they would experience the reinforcements and the
burden of self-reporting over the long-term, and thus they misjudged how
valuable they would find SARA’s reinforcements in the future.

The Follow-up interview: After the 30-day study, we conducted semi-structured
telephone interviews with participants. Below, we discuss the interview themes
related to reinforcement design.

Aquarium and fish: Participants’ reactions to the aquarium mirrored those of
the focus group participants. One participant noted: “I liked the fish a lot.
I thought they were very cute, and I liked when the app moved from the fish
bowl to the ocean.” However, similarly to focus group participants, they also
indicated a need for better aesthetics and more interactivity.

Financial reinforcements: While participants liked receiving money for self-
reporting, they did not like financial reinforcements in small fractions:
“Bigger rewards would be more exciting… don’t have $0.25 rewards and instead
have $1.00 rewards every once in a while.” One participant also suggested that
monetary reinforcements be ramped up as participants reported more data in the
study. “I would prefer that the app gradually gave me more money as I took
more surveys and active tasks. So at the beginning, I would only receive a
small amount and gradually receive more.” Note that the schedule of financial
reinforcements actually increased incentives for longer streaks, but most
participants did not see this since streaks were broken even if they missed a
single day.

Memes and life insights: Some participants liked the memes, indicating that
they were funny, but others wanted memes to be personalized. One participant
said “Maybe you could let people choose what type of memes they want to see…
or add a like/don’t like button for the memes (like you have for the
inspirational messages) and then if someone doesn’t like one type of memes,
you could push a different type of meme.” Life insights were generally liked.
One participant said: “I like the tracking and life insights on a daily basis.
For people my age, [it is] totally helpful. [It is] awesome.” However,
participants also wanted life insights that combined different types of data.
Overall, the memes and life insights were generally well liked, suggesting
they may be effective as reinforcers for AYA.

Habituation: Participants mentioned that fish became “trivial” over time. Some
also mentioned that self-reporting became repetitive and they wanted more
variety in active tasks or more information about why certain information was
being collected. These observations indicate that participants might have
habituated to the reinforcements, and that self-reporting was increasingly
seen as boring. To improve the experience, participants suggested to add a
variety of new active tasks and add educational content on why the data were
being collected.

###  3.10. Iteration 5: Final improvements before the trial

Design constraints and theoretical considerations: While the overall results
were positive, our preliminary studies pointed to several shortcomings of our
operationalizations of concepts from OCT. Iteration 5 tried to address some of
these shortcomings. The first shortcoming was related to the schedule for
financial reinforcements. Our initial schedule was progressive, where more
money was paid for longer streaks. Progressive schedules work well for highly
valued reinforcements (Hodos, 1961), but our findings indicated they were not
well suited for the small amounts we were paying. Second, for participants who
disengaged early in the study, the available reinforcement might have been
insufficient to overcome self-reporting burden. OCT suggests fast
reinforcement schedules at the start because more reinforcement early on can
speed up learning (Ferster and Skinner, 1957). The amount of reinforcement we
provided early on might not have been high enough for these participants.
Finally, the interview data suggested that participants became habituated to
the fish and other reinforcements (habituation is the decrease in efficacy of
a reinforcement after repeated exposure (Thompson, 2015; Zang, 2007; Rankin et
al., 2009)). OCT suggests two ways to deal with habituation: (i) allow for
“break time” to resensitize to the existing reinforcements; and (ii) increase
reinforcement variety. Given that we were under time pressure to run a trial,
introducing “break time” was the more feasible way to proceed.

Figure 8. Left and middle are aquarium screenshots after 2-days of data
collection for iteration 2 and iteration 5 respectively. Right screenshot
shows adherence rates of User study 4 in comparison with Bonar et al. (Bonar
et al., 2018)

Design considerations: In iteration 5, we made three modifications to our
reinforcement design. The first modification was to use a simpler schedule for
monetary reinforcements where participants earned $1 whenever they completed a
3-day streak of self-reporting. This change addressed the difficulties of
achieving longer streaks and participants’ preference for reinforcements in
whole dollar amounts. Note that the new schedule did not substantially
increase projected participant earnings: 90% adherence would lead, on average,
to the earning of only $12 over a 30-day study. The second modification
attempted to increase reinforcement early in the study. To do this, we front-
loaded reinforcements at the start of the study: (i) we awarded participants
$1 after they self-reported the first day; and (ii) in the first two days, we
awarded two fish for each self-report, enabling participants to earn four fish
in two days (see Figure 8). The third modification meant to address
habituation. We implemented a simple protocol to try to bring participants
back to the study after they have had time to re-sensitize to the
reinforcements in SARA: we decided to send text messages to participants after
a few days of non-response. The first text message was to be sent after 2 days
of no self-reporting. If non-adherence continued, an additional text message
would be sent every three days. Text messaging stopped if participants did not
self-report for 3 weeks.

###  3.11. User study 4: An evaluation study of SARA (N=37)

We evaluated the final design of SARA’s adherence interventions in a 30-day
field study with 37 high-risk AYA substance users (49% male; 73.5% Caucasian;
age: $\mu$=20.4, SD=2.1; 53% binge drinking-only, 47% any marijuana use in
last month).  We did not test the new features from Iteration 5 in a focus
group or another pilot, because we had a deadline on our funding and the
number of new features was small to necessitate additional pilot evaluations.
For the field evaluation, we used the same 30-day study protocol as we used in
the user study 3. We recruited high-risk AYA who were admitted to  the
University of Anonymous Hospital Pediatric and Adult Emergency Department for
risky substance use. Potential participants were approached as they were being
discharged and were screened for eligibility and interest in the study.
Interested individuals were consented and SARA was installed on their personal
phones, after which they began 30 days of data collection. As we noted above,
the 30-day study duration is common for observational studies on substance use
(Suffoletto et al., 2012; Shrier et al., 2018; Clark et al., 2010; Bonar et
al., 2018; Buu et al., 2017; Comulada et al., 2015; Suffoletto et al., 2012;
Wen et al., 2017), so this duration allowed us to compare our adherence rates
to those in the literature.  Furthermore, the sample of $N=37$ is similar to
or higher than other mHealth deployment studies that focused on novel
intervention technologies (Consolvo et al., 2008a; Lane et al., 2014; Bardram
et al., 2013). The sample size is also sufficient to draw design insights
using both qualitative and quantitative measures.

The findings from Study 4 fall into several categories:

Adherence: We compared the adherence patterns from User Sstudy 4 with the
adherence rates in User Study 3 as well as with a similar study by Bonar et
al. (Bonar et al., 2018). User Study 3 used the same protocol as User Study 4,
and the Bonar et al. (Bonar et al., 2018) study involved answering a daily
survey for 30 days and the population is similar to SARA’s population of AYA
at high risk of substance use. Bonar et al.’s asked 27 questions in the daily
survey, but the average number of questions answered was 18. For SARA, average
number of questions are close to seven but SARA had two additional assessments
in terms of active tasks. Thus the overall self-report burden in SARA and
Bonar et al. (Bonar et al., 2018) are comparable.

SARA’s adherence rate in User Study 4 was 71.7% for the first 10 days, 64.4%
for days 11-20, and 50.8% for days 21-30. The adherence rate of 50.8% after 20
days is higher than for most self-report studies. These results suggest that
some of the changes in Iteration 5 had a positive effect: beyond the first ten
days, adherence rates increased substantially from User Study 3 ( 63.8% for
days 1 to 10, 49.2% for days 11 to 20, and 34.6% for days 21 to 30). Figure 8
shows the adherence rate of SARA and Bonar et al. (Bonar et al., 2018) on week
1,2,3,4 of the study. While the adherence rates are similar, Bonar et al.
(Bonar et al., 2018) paid significantly more money.

Figure 9. Distribution of ratings for various reinforcement types in user
study 4. Left two subfigure uses a three point likert scale (1=not all,
2=somewhat,3=very much) to rate how these reinforcement influenced self-
report. Righ two subfigure uses a five point likert scale (1=not all, 5=very
much) to rate how these reinforcement influenced self-report.

Monetary incentives: On average, participants earned $6.53 each
($\sigma=3.8,q_{50}=6.5$) in User Study 4 for completion of the assessments
over 30 days. In the exit survey, when we asked participants how money
affected their use of the SARA app (“How much did earning money bonuses
increase your use of the app?” on a 3-point Likert scale (1=not at all,
2=somewhat, and 3=very much), 83.3% reported very much and the rest reported
somewhat. No participant answered ‘not at all.’ In the exit interviews, both
the amount of money and the schedule were perceived positively: one
participant said “I also liked the money bonuses and thought the amount of
money offered was just right.” and another participant said “I liked earning
money because it kept me coming back for 3 days in a row.” These results
suggest that the changes in monetary schedule in Iteration 5 (i.e., $1 for
each three-day streak) had a positive impact.

Furthermore, the average amount of money earned, $6.53, is important. Bonar et
al., described earlier, used higher financial incentives to achieve similar
daily adherence rates to Iteration 5 of SARA. Bonar et al. paid $2 for each
daily survey completed and an additional $5 bonus when participants completed
at least 6 of 7 daily surveys each week (up to $20 total bonus). Participants
in Bonar et al.’s study earned on average $46.20 $(SD=\$24.62)$ in 28 days
(including bonuses), which is seven times more than what SARA participants
were paid.

Aquarium and fish: Participants on average spent 14.5 seconds
$(SD=17.6seconds)$ looking at the aquarium. However, proportion of
participants looked at the aquarium everyday other than self-reporting
decreased by 1% per day over the course of study which may mean participants
were likely losing interest. When we asked participants how unlocking fish
affected their use of the SARA app (“How much did unlockable fish increase
your use of the app?”) on a 3-point Likert scale (1=not at all, 2=somewhat,
and 3=very much), the average rating was 1.7 $(SD=2.7)$. Note that these
ratings are lower than those of the focus group (see section 3.7) whose
participants did not use the app for 30-days. The qualitative feedback for the
aquarium is similar to the feedback from the pilot study. Some participants
liked the appearance of the aquarium: “I also liked the appearance of it, it
was engaging. The contrasting colors were captivating and I also liked the
idea of an aquarium.” and some liked the reinforcement schedule of regularly
unlocking fish: “The app made me want to use it every day because I wanted to
unlock new fish.” However, as in earlier user studies, some participants did
not find the aquarium aesthetically pleasing: one participant said, “The color
composition is not aesthetically appealing.”. Another participant said, ‘The
images look like they’re from the 1990s.”.

Memes and life insight: Participants on average spent 5.6 seconds $(SD=7.7)$
looking at the memes and 7.1 seconds $(SD=8.9)$ looking at the life insights.
In the exit survey, participants rated the perceived influence of memes and
life-insight on a five-point scale, from 1=not at all to 5=very much. Life
insights $(\mu=4.2,\sigma=2.5)$ were rated higher than memes
$(\mu=3.6,\sigma=2.3)$. Participants generally liked the life insights and
they reported using life-insights to reflect on their week and to track their
progress in tapping or spatial tasks. One participant said, “I liked having a
way to record all of my days. I thought it was cool to look back on each day
at the graphs and see how my weeks have been.” Another participant said “[I]
enjoyed seeing how performance on tasks reflected in insights.” Adding more
life insights was one area of improvement identified by participants.
Regarding memes, some participants liked their variety while others had
recommendations for updating and personalizing them. For example, one
participant said “I liked how there were different memes and quotes every
day.” and another said “the memes weren’t actually memes. They were either
2009-style memes, which just look kinda bad, or they just looked like pictures
used to make memes.”

These results suggest that life insights are generally viewed positively,
which is consistent with past findings that younger adults like to explore
their data (Meng et al., 2018). However, the memes we selected using pilot
procedures were not as appealing; user feedback indicates that memes have a
temporal nature and people like memes on specific topics they care about. It
is harder to design memes as reinforcement than we had anticipated. In part,
because some memes were meant to appeal to user’s sense of humor, anticipating
and personalizing memes to such a highly personal trait such as sense of humor
is challenging. It may be that the inspirational memes used were more
appealing; however, this supposition requires future study.

Text messages and phone call: On average, participants received 5.1 $(SD=1.8)$
texts and phone calls. On 47% of days, participants self-reported on the same
day after they received a text message. Recall the text messages or phone
calls were sent after participants did not 2 to 3 days. So, the contacts from
study staff likely increased adherence as intended.

Future app improvement opportunities: In the exit interviews, participants
provided additional suggestions for improving the future version of the app.
The first type of feedback asked for options to choose the time of survey
completion. One participant said, “The survey starting at 6pm was a barrier to
adherence… it would’ve been better if it was even an hour earlier or earlier
in the day so there’s more time and more flexibility.” Another participant
said “I would’ve preferred to answer the daily survey first thing in the
morning about the previous day.” The second type of feedback suggested
categorizing the memes and letting participants choose which kind they see.
One participant said “I think if there was an option to choose a specific
category of memes/quotes to receive, then I would have been more captivated.”
Finally, participants asked for better instructions on using the app and asked
for more features to give reasons to go the app. We believe the issue with the
instructions can be resolved with additional user-centered design. Creating
more value can be achieved in different ways. We can use data to provide
therapeutic strategies to reduce substance use (e.g., coping skills). Another
way to create value could be a “viable research alliance” where participants
are informed of the scientific value of the data they are giving us
(Csikszentmihalyi and Larson, 2014). Another idea to improve design could be
to work with developmental psychologists to add more age appropriate content.
Another thing to note is that this user feedback deals with issues of choice,
interruptibility, and empowering users with knowledge. These issues, however,
are not well-covered by OCT, so we need to move beyond OCT and use other
theories to make SARA more engaging.

Nonetheless, the above results suggest that our theory translation of OCT and
design iterations led to improved effectiveness of the reinforcement used in
SARA. These results do not reach the level of rigor that a randomized
controlled trial would yield. But given their purpose—to provide empirical
data that can be triangulated with information from previous user studies to
assess the quality of our theory translation efforts— User Study 4’s results
indicate that constructs from operant conditioning may have been implemented
successfully.

##  4\. Theory translation in a nutshell

In the previous section, we gave a detailed description of how we translated a
set of constructs from OCT into a number of features of the SARA application.
In this section, we distill our experience and provide some general
methodological guidelines for the process of translating theory into technical
interventions. We hope these guidelines can act as a starting point for a more
structured process that can help designers do theory translation in a more
efficient and less error-prone way.

Theory translation is an iterative design process. Although iterative deign is
common in HCI, iterative design for theory translation is different in that
its aim is to improve the _theoretical fidelity_ of the intervention—i.e., how
faithfully the intervention features embody the constructs they are intended
to implement. For example, the animated fish from the iteration 2 of SARA
(section 3.3) are a higher-theoretical-fidelity representation of ‘valued
reinforcement’ than the static fish from Iteration 1. Improving theoretical
fidelity is important for reasons discussed in Section 2.4. However, improving
theoretical fidelity is a complex iterative process of balancing user
feedback, project constraints, theoretical insights, etc. Based on our
experience with designing SARA, we have identified eight key steps that
designers need to address during theory translation:

  1. (1)

Selecting a theory: First, select an appropriate theory that will provide an
account of how the technology under development is intended to influence the
target behavior in the target population. Intervention development frameworks
from behavioral science, such as the Behavior Change Wheel (Michie et al.,
2011; Michie et al., 2013; Michie et al., 2014a) and Intervention Mapping
(Bartholomew et al., 1998; Kok et al., 2004), can be used to facilitate this
step. A key thing to note at this step is that choosing a theory means that
the theory will act, at least in part, as a blueprint of the causal influence
that the technology is intended to have on the target behavior. It should
describe how exactly the technology is intended to change people’s behavior.
It thus follows that how well a theory is specified greatly influences both
how useful it is as the basis for an intervention and how easily it can be
implemented.

  2. (2)

Selecting an initial set of constructs: Once a theory is selected, select the
constructs that will form the foundation of the intervention’s functionality.
In SARA, our goal was to increase the frequency of responding to the daily
self-report and to maintain this responding over time. Thus, we started by
trying to implement positive reinforcement and schedules of reinforcement,
which are the core OCT constructs that describe the process that increases
behavior frequency and affects its dynamics over time (Section 3.2-3.3).
Additional constructs can be added later as needed (e.g., based on user
feedback or to account for a specific aspect of the behavior change process),
but we suggest trying to get the core functionality—and the corresponding
intended causal process—clearly specified first.

  3. (3)

Developing initial designs: For each initially selected construct, explore the
design space for how that construct can be operationalized in the particular
system that is being developed. The key goals of the ideation stage is to
develop designs that (a) embody key properties of the construct, and (b) match
the known constraints of the population, context of use, and system coherence.
A useful structure for exploring the design space is to develop both
alternative ways of implementing a construct and multiple variations of each
of those designs. Generated designs can then be evaluated for feasibility,
user acceptance, and preliminary theoretical fidelity.

  4. (4)

Specifying theoretical fidelity criteria: After the basic set of designs have
been developed, for each implemented construct articulate how you would be
able to tell if your implementation of the construct got it “right”—namely,
whether the technology feature has captured the essence of the construct. This
is arguably the hardest step in the whole theory translation process, as the
criteria for establishing theoretical fidelity may not be at all obvious. For
some constructs, theoretical fidelity may need to be established based on the
qualitative user feedback; for instance, to operationalize “reinforcement
value” we had to rely on participants’ comments about whether they found
proposed design features (e.g., fish) appealing and whether they thought those
features would motivate them to self-report. For other constructs, it may be
possible to articulate exactly what kinds of patterns of behavior one would
expect if the feature was operating as described by the construct (e.g., this
is possible for different types of reinforcement schedules). Despite the
difficulties inherent in this step, it is a crucial one, as it allows
designers to test whether their technology features, as designed, are in fact
embodying the intended construct.

  5. (5)

Preliminary testing of theoretical fidelity: After the basic set of constructs
have been selected and the criteria for theoretical fidelity articulated,
implement low-fi prototypes of the features that embody these constructs and
test their theoretical fidelity. This can be done in many ways, from informal
internal testing to small user studies depending on the nature of the fidelity
criteria. Whatever the method, the central task in this step is to get early
information about whether the technology feature, as envisioned, may be able
to act as described by the construct. As is typical for low-fi prototyping,
ineffective prototypes can be discarded and the rest of the prototypes quickly
improved to achieve higher-theoretical-fidelity implementation of the
underlying constructs. The purpose of our user studies 1 and 2 was precisely
to do this type of preliminary testing of theoretical fidelity, although, in
retrospect, these studies could have likely been done in a cleaner way to
achieve this goal.

  6. (6)

Adding constructs: As the intervention develops, it will often be necessary to
add additional constructs either from the same theory or from other theories.
New constructs should be chosen either because they may amplify the
functioning of the already implemented constructs or because designers believe
the intervention requires additional forms of behavioral support. In either
case, as the constructs are being chosen, it is important to articulate how
they would interact with the already implemented features, so designers can
assess, before development resources are expended, if the features based on
these constructs would strengthen or negatively impact the existing
functionality. Formal representations, such as causal diagrams, can often be
helpful for this, but less formal approaches—like our considerations of using
multiple reinforcers—may be sufficient.

  7. (7)

Selecting study methods for formative evaluations: Early tests of theoretical
fidelity, as well as formative evaluations of overall functionality will often
require some form of data collection. What types of user studies will be
useful will depend, of course on the specific questions they are intended to
address. However, during the formative stage, online surveys or focus groups
can often be cost-effective ways to assess user perceptions in order to
improve the theoretical fidelity of the intervention. Later on, field trials
will usually be needed to capture behavioral responses to confirm intended
functioning. The key thing to note is that the studies need to match the level
of evidence that is required to move to the next stage of the development
process. Early on, a designer might just need a sanity check on an idea, as we
did with the aquarium representation. The level of evidence needed at this
point is much lower—and can be achieved with a much simpler study—than the
evidence needed to establish the effect size for an intervention. Being clear
about what exactly a study is intended to achieve, and then selecting the
study design and sample size appropriately, is thus paramount both for doing
theory translation in a resource-efficient way and, down the line, for
producing reliable, trustworthy evidence on the completed intervention.
Similar considerations apply to selecting the study duration as well.
Behavioral interventions can have effects that change over time due to
learning, habituation, habit formation and so on. Real-world use can uncover
problems which may need to be addressed with additional constructs or fidelity
improvements. For example, the length of the User Study 3 allowed us to learn
about habituation and the problems with the initially implemented monetary
schedule. As with the study type, the study length for field studies thus
needs to be chosen thoughtfully, with the eye toward maximizing the designers’
ability to learn about problems that need to be addressed while minimizing the
need for resources and participant burden.

  8. (8)

Providing in-depth descriptions of intervention design: One goal of theory-
based interventions is to allow testing of theories in order to advance
science. To further this goal, it is essential for designers to report what
theoretical constructs were implemented in an intervention and how in detail.
Such descriptions can help both with the interpretation of findings from any
experimental trials of the intervention, as well as their comparison with
results from other studies of other interventions that embody the same
constructs. This level of description is currently unusual both in HCI and in
behavioral science, but we firmly believe that it is essential for advancing
our understanding of the behavior change process and the factors that
influence intervention response.

##  5\. Discussion and conclusions

In this paper, we provided a detailed account of our process of trying to
translate a set of concepts from OCT into features of SARA, a mobile
application for conducting substance use research with adolescents and young
adults. As a case study of theory translation, OCT is in many ways the best
case scenario. The constructs and processes postulated by the theory are well
specified and are supported by—and have been refined through—decades of
careful empirical work with both animals and humans. If there is a theory that
should allow for straightforward implementation, OCT is it. Indeed, some of
the concepts we could implement in a very straightforward way. We could
implement the notions of immediacy and contingency just by providing
reinforcement right after, and only after, a user provided self-report
(Section 3.2). Similarly, the extensive empirical data on schedules of
reinforcement allowed us to make informed decisions to use a fast fixed-ratio
schedule for fish and to include variable-rate schedules to make the effects
of reinforcement more sustainable (Section 3.3, 3.4, 3.7). OCT also provided
clear guidance on the question of whether multiple forms of reinforcement
could work together effectively, leading us to greatly expand the range of
rewards that SARA could provide to reinforce self-reporting (Section 3.4,
3.7). Much more directly than many theories, then, OCT could tell us what we
needed to do, at least in broad terms.

Yet, even with such a highly specified theory, how exactly the various forms
of reinforcement in SARA needed to work was left unclear. What forms of
reinforcement would be found to be valuable, how fast the fish schedule needed
to be, how exactly the schedule of financial reinforcements needed to be
structured were among the many decisions we had to make based on design
intuitions, user feedback, and resource constraints. The theory could tell us
what should work in principle, but the many design details we had to decide on
to operationalize its concepts for this application and this particular
population were left to us to work through. Yet, those design details mattered
greatly. The aesthetics of the fish or whether the financial rewards were
given in fractions or whole dollar amounts influenced how valuable—and thus
reinforcing—our participants perceived them to be. To get the details right,
we had to keep the theory in constant conversation with user feedback and
other constraints and to iteratively make gradual design changes until we
started to see reactions and behaviors that began to approach what OCT told us
should happen for the concepts we were trying to implement. In other words, we
could only tell that we implemented a construct from OCT adequately when we
could see the behavior postulated by the theory to result from the construct.

Which brings us to a key challenge of theory translation: determining if a
theoretical construct has been implemented with fidelity. Theory-based
interventions play a dual role. On the one hand, they are designed to address
a particular problem in a particular population. Insofar as an intervention
achieves this goal, it can be considered to be a success. On the other hand,
interventions act as tests of the theories they embody. For science to
progress, theories have to be tested and refined, and there is no way to do
that in the abstract. A behavioral theory can only be tested by studying the
behavior of real people and their responses to concrete interventions. But for
a study to provide evidence about a theory, the intervention used in the study
needs to have implemented that theory with fidelity.

How to ensure that a construct has, in fact, been implemented with high
fidelity is not trivial, however. As we noted, our test for theoretical
fidelity in SARA was to look for behaviors that the theory postulated. For a
precise theory like OCT that provides highly specific accounts of what should
happen under different circumstances, such a test makes sense. But many of the
theories commonly used to guide the development of technological interventions
are not nearly so specific in their accounts of what to expect beyond
assertions that a set of determinants influence behavior. For such theories, a
designer is left without clear criteria for evaluating theoretical fidelity.
Consulting with domain experts with experience in that theory can help with
articulating theoretical fidelity criteria, but even this strategy can
sometimes leave the designer with having to come up with something herself. As
we noted in Section 4, making an effort to articulate fidelity criteria is
paramount, however, and with the rapid move toward technological interventions
in behavior change research the issue of theoretical fidelity will only grow
in importance. As HCI researchers are increasingly participating in
interdisciplinary collaborations with behavioral scientists, and insofar as we
wish to more deeply understand why our technologies do or do not work, our
discipline will need to deal with this issue head-on.

All theories are by their nature abstract. Theories are formulated to account
for concrete behaviors and events, but a theoretical explanation is only
achieved by focusing on certain narrow aspects of a phenomenon, abstracting
out its general features, and stripping out everything else. The problem, from
the standpoint of technology design, is going in the opposite
direction—filling in all the concreteness that was left out in order to
achieve theoretical generality. How best to approach this process of
operationalizing theories in technological interventions is something that, we
believe, our community needs to think about carefully. In presenting this case
study, our goal was to surface the complexity and nuances involved in theory
translation. We are not arguing that the process we followed is the right one,
or that others should apply it to their work in the same form. Rather, we hope
that the process we went through can serve as an example that can be critiqued
and improved upon. Over time, we hope, we will arrive at a more robust and
efficient process for theory translation and will more fully understand its
challenges and pitfalls. There is much left to do to achieve this goal.

## References

  * (1)
  * Abuse et al. (2016) Substance Abuse, Mental Health Services Administration, et al. 2016\.  2015 National Survey on Drug Use and Health.  (2016).
  * Abyssrium (2018) Abyssrium. 2018\.  https://www.abyssrium.com/.  [Online; accessed 2 July 2018].
  * Adams et al. (2014) Phil Adams, Mashfiqui Rabbi, Tauhidur Rahman, Mark Matthews, Amy Voida, Geri Gay, Tanzeem Choudhury, and Stephen Voida. 2014\.  Towards personal stress informatics: Comparing minimally invasive techniques for measuring daily stress in the wild. In _Proceedings of the 8th International Conference on Pervasive Computing Technologies for Healthcare_. ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), 72–79.
  * Ainslie (2001) George Ainslie. 2001\.  _Breakdown of will_.  Cambridge University Press.
  * Ariely and Jones (2013) Dan Ariely and Simon Jones. 2013.  _The upside of irrationality_.  CNIB.
  * Badawy and Kuhns (2017) Sherif M Badawy and Lisa M Kuhns. 2017.  Texting and mobile phone app interventions for improving adherence to preventive behavior in adolescents: a systematic review.  _JMIR mHealth and uHealth_ 5, 4 (2017).
  * Bae et al. (2018) Sangwon Bae, Tammy Chung, Denzil Ferreira, Anind K Dey, and Brian Suffoletto. 2018.  Mobile phone sensors and supervised machine learning to identify alcohol use events in young adults: Implications for just-in-time adaptive interventions.  _Addictive behaviors_ 83 (2018), 42–47.
  * Bardram et al. (2013) Jakob E Bardram, Mads Frost, Károly Szántó, Maria Faurholt-Jepsen, Maj Vinberg, and Lars Vedel Kessing. 2013.  Designing mobile health technology for bipolar disorder: a field trial of the monarca system. In _Proceedings of the SIGCHI conference on human factors in computing systems_. ACM, 2627–2636.
  * Bartholomew et al. (1998) L Kay Bartholomew, Guy S Parcel, and Gerjo Kok. 1998.  Intervention mapping: a process for developing theory and evidence-based health education programs.  _Health Education & Behavior_ 25, 5 (1998), 545–563.
  * Bonar et al. (2018) Erin E Bonar, Rebecca M Cunningham, R Lorraine Collins, James A Cranford, Stephen T Chermack, Marc A Zimmerman, Frederic C Blow, and Maureen A Walton. 2018.  Feasibility and acceptability of text messaging to assess daily substance use and sexual behaviors among urban emerging adults.  _Addiction research & theory_ 26, 2 (2018), 103–113.
  * Boruvka et al. (2017) Audrey Boruvka, Daniel Almirall, Katie Witkiewitz, and Susan A Murphy. 2017.  Assessing time-varying causal effect moderation in mobile health.  _J. Amer. Statist. Assoc._ just-accepted (2017).
  * Boyle et al. (2017) Sarah C Boyle, Andrew M Earle, Joseph W LaBrie, and Daniel J Smith. 2017.  PNF 2.0? Initial evidence that gamification can increase the efficacy of brief, web-based personalized normative feedback alcohol interventions.  _Addictive behaviors_ 67 (2017), 8–17.
  * Braun and Clarke (2006) Virginia Braun and Victoria Clarke. 2006.  Using thematic analysis in psychology.  _Qualitative research in psychology_ 3, 2 (2006), 77–101.
  * Burchard and Tyler Jr (1964) John Burchard and Vernon Tyler Jr. 1964.  The modification of delinquent behaviour through operant conditioning.  _Behaviour Research and Therapy_ 2, 2-4 (1964), 245–250.
  * Burns et al. (2011) Michelle Nicole Burns, Mark Begale, Jennifer Duffecy, Darren Gergle, Chris J Karr, Emily Giangrande, and David C Mohr. 2011\.  Harnessing context sensing to develop a mobile intervention for depression.  _Journal of medical Internet research_ 13, 3 (2011).
  * Bush et al. (1998) Kristen Bush, Daniel R Kivlahan, Mary B McDonell, Stephan D Fihn, and Katharine A Bradley. 1998\.  The AUDIT alcohol consumption questions (AUDIT-C): an effective brief screening test for problem drinking.  _Archives of internal medicine_ 158, 16 (1998), 1789–1795.
  * Buu et al. (2017) Anne Buu, Lynn S Massey, Maureen A Walton, James A Cranford, Marc A Zimmerman, and Rebecca M Cunningham. 2017.  Assessment methods and schedules for collecting daily process data on substance use related health behaviors: A randomized control study.  _Drug and alcohol dependence_ 178 (2017), 159–164.
  * Celio et al. (2014) Mark A Celio, Julie M Usala, Stephen A Lisman, Gerard E Johansen, Courtney S Vetter-O’Hagen, and Linda P Spear. 2014.  Are we drunk yet? Motor versus cognitive cues of subjective intoxication.  _Alcoholism: clinical and experimental research_ 38, 2 (2014), 538–544.
  * Choe et al. (2017) Eun Kyoung Choe, James Fogarty, Bongshin Lee, Mark Matthews, Julie A Kientz, Saeed Abdullah, Mashfiqui Rabbi, Edison Thomaz, Daniel A Epstein, Felicia Cordeiro, et al. 2017\.  Semi-automated tracking: A balanced approach for self-monitoring applications.  _IEEE Pervasive Computing_ 1 (2017), 74–84.
  * Clark et al. (2010) Heddy Kovach Clark, Chris L Ringwalt, Sean Hanley, Stephen R Shamblen, Robert L Flewelling, and Mary C Hano. 2010.  Project SUCCESS’effects on the substance use of alternative high school students.  _Addictive behaviors_ 35, 3 (2010), 209–217.
  * Cole-Lewis et al. (2019) Heather Cole-Lewis, Nnamdi Ezeanochie, and Jennifer Turgiss. 2019\.  Understanding Health Behavior Technology Engagement: Pathway to Measuring Digital Behavior Change Interventions.  _JMIR formative research_ 3, 4 (2019), e14052.
  * Collins (2018) Linda M Collins. 2018\.  _Optimization of Behavioral, Biobehavioral, and Biomedical Interventions: The Multiphase Optimization Strategy (MOST)_.  Springer.
  * Comulada et al. (2015) W Scott Comulada, Marguerita Lightfoot, Dallas Swendeman, Christine Grella, and Nancy Wu. 2015\.  Compliance to cell phone-based EMA among Latino youth in outpatient treatment.  _Journal of ethnicity in substance abuse_ 14, 3 (2015), 232–250.
  * Consolvo et al. (2008a) Sunny Consolvo, Predrag Klasnja, David W McDonald, Daniel Avrahami, Jon Froehlich, Louis LeGrand, Ryan Libby, Keith Mosher, and James A Landay. 2008a.  Flowers or a robot army?: encouraging awareness & activity with personal, mobile displays. In _Proceedings of the 10th international conference on Ubiquitous computing_. ACM, 54–63.
  * Consolvo et al. (2009) Sunny Consolvo, David W McDonald, and James A Landay. 2009\.  Theory-driven design strategies for technologies that support behavior change in everyday life. In _Proceedings of the SIGCHI conference on human factors in computing systems_. ACM, 405–414.
  * Consolvo et al. (2008b) Sunny Consolvo, David W McDonald, Tammy Toscos, Mike Y Chen, Jon Froehlich, Beverly Harrison, Predrag Klasnja, Anthony LaMarca, Louis LeGrand, Ryan Libby, et al. 2008b.  Activity sensing in the wild: a field trial of ubifit garden. In _Proceedings of the twenty-sixth annual SIGCHI conference on Human factors in computing systems_. ACM, 1797–1806.
  * Cordeiro et al. (2015) Felicia Cordeiro, Daniel A Epstein, Edison Thomaz, Elizabeth Bales, Arvind K Jagannathan, Gregory D Abowd, and James Fogarty. 2015.  Barriers and negative nudges: Exploring challenges in food journaling. In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_. ACM, 1159–1162.
  * Cranford et al. (2006) James A Cranford, Patrick E Shrout, Masumi Iida, Eshkol Rafaeli, Tiffany Yip, and Niall Bolger. 2006.  A procedure for evaluating sensitivity to within-person change: Can mood measures in diary studies detect change reliably?  _Personality and Social Psychology Bulletin_ 32, 7 (2006), 917–929.
  * Csikszentmihalyi and Larson (2014) Mihaly Csikszentmihalyi and Reed Larson. 2014.  Validity and reliability of the experience-sampling method.  In _Flow and the foundations of positive psychology_. Springer, 35–54.
  * Dallery et al. (2013) Jesse Dallery, Rachel N Cassidy, and Bethany R Raiff. 2013\.  Single-case experimental designs to evaluate novel technology-based health interventions.  _Journal of medical Internet research_ 15, 2 (2013).
  * den Bakker et al. (2019) Chantal M den Bakker, Frederieke G Schaafsma, Eva van der Meij, Wilhelmus JHJ Meijerink, Baukje van den Heuvel, Astrid H Baan, Paul HP Davids, Petrus C Scholten, Suzan van der Meij, W Marchien van Baal, et al. 2019\.  Electronic Health Program to Empower Patients in Returning to Normal Activities After General Surgical and Gynecological Procedures: Intervention Mapping as a Useful Method for Further Development.  _Journal of medical Internet research_ 21, 2 (2019), e9938.
  * Deterding (2012) Sebastian Deterding. 2012\.  Gamification: designing for motivation.  _interactions_ 19, 4 (2012), 14–17.
  * Deterding et al. (2011) Sebastian Deterding, Dan Dixon, Rilla Khaled, and Lennart Nacke. 2011.  From game design elements to gamefulness: defining gamification. In _Proceedings of the 15th international academic MindTrek conference: Envisioning future media environments_. ACM, 9–15.
  * Dorsey et al. (2017) E Ray Dorsey, Michael V McConnell, Stanley Y Shaw, Andrew D Trister, Stephen H Friend, et al. 2017\.  The use of smartphones for health research.  _Academic Medicine_ 92, 2 (2017), 157–160.
  * Edwards et al. (2018) Elizabeth A Edwards, Hope Caton, Jim Lumsden, Carol Rivas, Liz Steed, Yutthana Pirunsarn, Sandra Jumbe, Chris Newby, Aditi Shenvi, Samaresh Mazumdar, et al. 2018\.  Creating a theoretically grounded, gamified health app: lessons from developing the Cigbreak smoking cessation mobile phone game.  _JMIR serious games_ 6, 4 (2018), e10252.
  * Epstein et al. (2015) Daniel A Epstein, An Ping, James Fogarty, and Sean A Munson. 2015.  A lived informatics model of personal informatics. In _Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing_. ACM, 731–742.
  * Eysenbach (2005) Gunther Eysenbach. 2005\.  The law of attrition.  _Journal of medical Internet research_ 7, 1 (2005).
  * Ferster and Skinner (1957) Charles B Ferster and Burrhus Frederic Skinner. 1957.  Schedules of reinforcement.  (1957).
  * Fitbit, Inc. (2013) Fitbit, Inc. 2013\.  http://www.fitbit.com/.  [Online; accessed 19 March 2013].
  * Fogg (2009) BJ Fogg. 2009.  A behavior model for persuasive design. In _Proceedings of the 4th international Conference on Persuasive Technology_. ACM, 40.
  * Geraghty et al. (2016) Adam WA Geraghty, Ricardo F Muñoz, Lucy Yardley, Jennifer Mc Sharry, Paul Little, and Michael Moore. 2016.  Developing an unguided Internet-delivered intervention for emotional distress in primary care patients: Applying common factor and person-based approaches.  _JMIR mental health_ 3, 4 (2016), e53.
  * Glanz et al. (2008) Karen Glanz, Barbara K Rimer, and Kasisomayajula Viswanath. 2008\.  _Health behavior and health education: theory, research, and practice_.  John Wiley & Sons.
  * Grant et al. (2007) Valerie V Grant, Sherry H Stewart, Roisin M O’Connor, Ekin Blackwell, and Patricia J Conrod. 2007\.  Psychometric evaluation of the five-factor Modified Drinking Motives Questionnaire-Revised in undergraduates.  _Addictive behaviors_ 32, 11 (2007), 2611–2632.
  * Greenhalgh and Russell (2010) Trisha Greenhalgh and Jill Russell. 2010.  Why do evaluations of eHealth programs fail? An alternative set of guiding principles.  _PLoS medicine_ 7, 11 (2010), e1000360.
  * Gruber et al. (2012) Staci A Gruber, Mary Kathryn Dahlgren, Kelly A Sagar, Atilla Gönenc, and William DS Killgore. 2012.  Age of onset of marijuana use impacts inhibitory processing.  _Neuroscience letters_ 511, 2 (2012), 89–94.
  * Gulotta et al. (2016) Rebecca Gulotta, Jodi Forlizzi, Rayoung Yang, and Mark Wah Newman. 2016.  Fostering engagement with personal informatics systems. In _Proceedings of the 2016 ACM Conference on Designing Interactive Systems_. ACM, 286–300.