# Mindless Attractor: A False-Positive Resistant Intervention for Drawing
Attention Using Auditory Perturbation

Riku Arakawa  0000-0001-7868-4754 The University of TokyoTokyoJapan arakawa-
<EMAIL_ADDRESS>and  Hiromu Yakura  0000-0002-2558-735X University
of TsukubaTsukubaJapan<EMAIL_ADDRESS>
(2021; 2021)

###### Abstract.

Explicitly alerting users is not always an optimal intervention, especially
when they are not motivated to obey. For example, in video-based learning,
learners who are distracted from the video would not follow an alert asking
them to pay attention. Inspired by the concept of _Mindless Computing_ , we
propose a novel intervention approach, _Mindless Attractor_ , that leverages
the nature of human speech communication to help learners refocus their
attention without relying on their motivation. Specifically, it perturbs the
voice in the video to direct their attention without consuming their conscious
awareness. Our experiments not only confirmed the validity of the proposed
approach but also emphasized its advantages in combination with a machine
learning-based sensing module. Namely, it would not frustrate users even
though the intervention is activated by false-positive detection of their
attentive state. Our intervention approach can be a reliable way to induce
behavioral change in human–AI symbiosis.

Mindless computing, Human attention, Computational intervention, Machine
learning-based sensing, Video-based learning

††copyright: none††journalyear: 2021††doi:
10.1145/xxxxxxx.xxxxxxxx††journalyear: 2021††copyright:
acmlicensed††conference: CHI Conference on Human Factors in Computing Systems;
May 8–13, 2021; Yokohama, Japan††booktitle: CHI Conference on Human Factors in
Computing Systems (CHI ’21), May 8–13, 2021, Yokohama, Japan††price:
15.00††doi: 10.1145/3411764.3445339††isbn: 978-1-4503-8096-6/21/05††ccs:
Human-centered computing Auditory feedback††ccs: Human-centered computing
Interaction techniques††ccs: Applied computing Interactive learning
environments

##  1\. Introduction

For decades, video-based communication has been expected to take over face-to-
face communication (Fish et al., 1992; Whittaker, 1995). In particular,
schools have leveraged video-based learning to provide educational
opportunities for distanced students, as massive open online courses have done
(Martin, 2012; Guo et al., 2014). Moreover, the recent COVID-19 pandemic has
precipitated the transition to video-based communication for the purpose of
preventing infection (Coombs, 2020; Kodama, 2020), especially in the context
of education (Kerres, 2020; Goldschmidt, 2020). However, it has been noted
that people often have trouble maintaining their attention in video-based
communications (Kuzminykh and Rintel, 2020a, b), as they can concurrently
perform other tasks, like texting or accessing social media using a smartphone
(Oeppen et al., 2020).

Considering the increasing demand for video-based learning, it would be
fruitful if computers can help learners pay attention to a video. Here, recent
advances in machine learning techniques have enabled the automatic estimation
of a user’s attention level from a video of their face (Thomas and Jayagopi,
2017). On the other hand, it is not trivial how to intervene with learners
using computers based on the estimation result. A straightforward approach is
to explicitly alert them when they seem to not be paying attention to the
video, as Xiao and Wang (Xiao and Wang, 2016) did. However, unlike the
critical situations targeted in conventional studies of alert designs (Cobus
et al., 2018; Saint-Lot et al., 2020), users of video-based learning systems
would not hesitate to ignore such alerts, especially when they are focused on
side tasks. For example, Xiao and Wang (Xiao and Wang, 2016) reported that
their intervention approach in their user study was described as unhelpful by
some participants who were less motivated. In other words, the efficacy of the
alerting approach would depend on the user’s motivation to actively take part,
and such interventions would not be an optimal intervention for inducing
behavioral change.

Looking back to the nature of human communications, we often change the tone
of our voices intentionally to draw listeners’ attention (Xu, 2005). Based on
this observation, we anticipate that we can help learners return their
attention to videos by computationally changing the tone of voice during
video-based learning situations. This approach is inspired by the concept of
Mindless Computing—behavior-changing technologies that leverage human biases
or unconscious behaviors—proposed by Adams et al. (Adams et al., 2015). Given
that Mindless Computing does not consume a user’s conscious awareness to be
effective, Adams et al. (Adams et al., 2015) stated that it does not rely on
the user’s motivation, whereas many of the current persuasive technologies
have a strong reliance on user motivation and are likely to fail. In addition,
the independence from the user’s conscious awareness enables such behavior
influencing to work without interfering with the user’s main task, which suits
our situation (i.e., use during video-based learning).

Furthermore, we argue that this mindless intervention approach has a high
affinity with sensing modules based on machine learning techniques. That is,
if we explicitly alert users, they can be distracted and frustrated by
misinformed alerts caused by erroneous false-positive detection, which can
lead them to ignore the result of a machine learning module (Dietvorst et al.,
2015; De-Arteaga et al., 2020). On the other hand, the mindless approach
designed based on human nature does not necessarily consume users’ conscious
awareness, and such negative effects due to false positives can thus be
mitigated.

In this paper, we propose a novel intervention approach, Mindless Attractor,
which computationally leverages the nature of our speech communication, and
examine its effectiveness in the case of helping users in video-based learning
return their attention to the video. For this purpose, we first determined its
requirements and possible designs so as to reduce the time that users are
distracted in a mindless manner. We then conducted an experiment to confirm
that the proposed intervention was effective in helping users refocus their
attention without consuming conscious awareness. We also combined this
mindless intervention with a machine learning-based sensing module and
evaluated its effectiveness in the context of false-positive detection, in
comparison to a conventional alerting approach. The series of experiments
presented the advantages of the proposed approach, especially in combination
with machine learning techniques. Based on the results, we discuss
implications for the HCI community, emphasizing the importance of the mindless
intervention approach in the era of human–AI symbiosis.

##  2\. Related Work

To situate our work, we first examine previous literature on interaction
techniques for video-based learning, particularly those focusing on learners’
attention. We then review conventional alert-based techniques for drawing
human attention and discuss why they would not fit our purposes. We also
explore previous studies regarding the nature of human speech communication,
as this is a foundation of our mindless approach for drawing users’ attention.

###  2.1. Attention-Related Interaction Techniques for Video-Based Learning

As mentioned in Section 1, opportunities for video-based communication are
increasing, and many interaction techniques have thus been proposed to enhance
the experience of such communications. Some prior studies have proposed
interaction techniques centering on the context of participants’ attention
(D’Mello et al., 2012; Sharma et al., 2016; Xiao and Wang, 2016), as it has
been pointed out that people often have difficulty maintaining their attention
during video-based communication (Kuzminykh and Rintel, 2020a, b). These
techniques benefit from the significant effort that has been devoted to
estimating participants’ attentiveness based on visual cues, such as face
movement (Thomas and Jayagopi, 2017), body postures (Zaletelj and Kosir,
2017), and gaze (Bidwell and Fuchs, 2011; Hutt et al., 2017; Veliyath et al.,
2019). They then use the estimation results to enhance learners’ performance,
for instance in the case of video-based learning, as it is widely acknowledged
that learners’ attention and engagement are strongly related to their learning
performance (Baker et al., 2010; D’Mello et al., 2012).

For example, Gaze Tutor is a gaze-reactive intelligent tutoring system for
video-based learning (D’Mello et al., 2012). Using a conventional eye tracker,
it estimates the learner’s attention level based on gaze direction by applying
a simple rule assuming that off-screen gaze patterns imply distraction. When
the system detects that the learner is not focusing on the video, the tutor
agent stops the video and alerts them explicitly (e.g., by saying “Please pay
attention”). Although their experiment showed its effectiveness in reorienting
participants’ attention, the intervention method left room for improvement, as
the authors mentioned in their discussion. Specifically, they found individual
differences in the efficacy of the alert-based intervention, including that
some participants never followed the alerts. Accordingly, the authors noted
that alternate intervention approaches, including indirect feedback, could be
implemented. Another example that computationally utilizes the estimated
attention level during video-based learning was provided by Sharma et al.
(Sharma et al., 2016). Similar to Gaze Tutor, their system provided users with
direct feedback, such as simple red rectangles on the screen, with the purpose
of improving users’ attention.

As can be inferred from these studies, previous research has mainly considered
explicit alerting as an intervention method for video-based learning. However,
the findings from these studies complement our concern, which is discussed in
Section 1 based on the results of Xiao et al. (Xiao and Wang, 2016). That is,
such interventions have a reliance on users’ motivation; they may not work
effectively when we cannot assume that all users are motivated to change their
behavior. In Section 2.2, we will explain why the reliance occurs based on the
discussion by Adams et al. (Adams et al., 2015), which in turn motivated us to
explore a better intervention approach for video-based learning situations.

###  2.2. Alerting Techniques for Drawing Human Attention

Drawing users’ attention is one of the crucial components of human-computer
interaction, not limited to video-based learning. Many researchers have dealt
with a wide range of topics in this area, such as Internet advertisements
(Nettelhorst and Brannon, 2012), smartphone notifications (Stothart et al.,
2015), and alerting systems (Guo et al., 2002). Consequently, previous studies
have developed many methods suitable for individual situations using diverse
perceptual modalities.

One of the most popular strategies is to provide users with visual
stimulation. For example, Red Alert is a visual alerting system which uses a
translucent orange-red flash to mask a screen, designed to warn pilots
potential collisions in air traffic control (Saint-Lot et al., 2020). Audio
stimuli have also been favorably employed as a means to alert users. BBeep is
a collision-avoidance system that can emit a beep sound to alert pedestrians
around a visually impaired user to clear the way (Kayukawa et al., 2019).
Another strategy is the use of the tactile modality. BuzzWear is a wrist-worn
tactile display to notify users on the go by combining different parameters of
the tactile stimulus (Lee and Starner, 2010). As can be observed in these
examples, most systems adopt explicit stimuli to notify users, assuming that
they will take action after their attention is drawn to the target.

However, Adams et al. (Adams et al., 2015) pointed out that such alerting
strategies would not be optimal when used within persuasive technologies
designed to influence user behavior. Unlike critical situations (e.g., air
traffic control) where it can be expected that users will be motivated to
follow an alert from a computer, not all scenarios for inducing behavioral
change can assume that users are motivated to do so. In such cases, an alert
that requires the user’s conscious awareness and effort to work effectively
would likely fail due to lack of motivation or potentially counteract positive
aspects of the intervention by frustrating them. Thus, the authors recommended
the Mindless Computing strategy of leveraging human biases or unconscious
behaviors, which diminishes reliance on users’ conscious awareness. It also
enables a user intervention without interfering with users’ ongoing activity,
whereas alerting users explicitly can interrupt such activity. Furthermore,
they complimented the advantage of the mindless approach by mentioning that
such interventions have long-term effectiveness, even though users are aware
of the biases behind the interventions (Wansink and van Ittersum, 2007).

This point is common to the previous studies for video-based learning in
regards to the reliance on learners’ motivation, which is mentioned in Section
2.1. That is, as conventional alerting approaches are requiring learners’
conscious awareness to be effective, they would have an option not to follow
the intervention. Therefore, for the purpose of helping learners return their
attention, we explore a new computational approach that intervenes without
consuming their conscious awareness. This led us to make use of the nature of
human speech communication.

###  2.3. Speech Communication Techniques for Drawing Human Attention

Speech is one of the most natural modalities of human communication. It
consists not only of linguistic aspects but also of paralinguistic aspects,
such as pitch, volume, and speed, which play an important role in conveying
nuance or emotion (Trager, 1958). Though the use of paralinguistic aspects is
a natural habit that does not necessarily require our conscious processes
(Poyatos, 1993), it is also a common practice to intentionally create changes
in such paralinguistic parameters while speaking so as to draw listeners’
attention (International, 2011). The relationship between speech parameters
and their effects in terms of drawing attention has generated considerable
research interest in understanding human speech communication. For example, Xu
(Xu, 2005) confirmed that an increase in pitch when starting a new topic can
draw listeners’ attention. Moreover, a similar effect of drawing attention has
also been observed in infants hearing the speech of their mothers, who
naturally vary their pitch (Sullivan and Horowitz, 1983). The idea that humans
unconsciously respond to paralinguistic cues is further supported by Zatoree
and Gandour (Zatorre and Gandour, 2007), who verified that human neural
mechanisms are sensitive to such spectral and temporal acoustical properties.

Based on these results, we speculate that leveraging this nature of human
speech communication by computationally varying speech parameters can draw
listeners’ attention in a natural manner. More specifically, if a person
losing their attention to a video hears speech with altered pitch or volume,
they will naturally respond to such a change, regardless of their motivation
to pay attention. Such an intervention approach is in line with the concept of
Mindless Computing (Adams et al., 2015) and thus is expected to work without
depending on users’ motivation. In the following section, we further elaborate
on the rationale for our design of using alterations of human speech to draw
attention in video-based learning situations.

##  3\. Mindless Attractor

In this paper, we propose Mindless Attractor for the purpose of helping users
in video-based learning situations return their attention to the video.
Inspired by the concept of Mindless Computing (Adams et al., 2015), it
leverages the nature of speech communication to intervene with users. In this
section, we present the details of Mindless Attractor, starting by discussing
why the mindless approach should be considered and what requirements should be
fulfilled.

###  3.1. Why Mindless?

As we stated in Section 1, our research aim is to support video-based
learning, given the growing demand for it, by establishing a suitable
computational intervention for users who are not paying attention to the
video. The difficulty is that we cannot assume all users to be highly
motivated to follow such an intervention for maintaining attention, which we
mentioned in Section 2.2 as the reason that conventional alerting approaches
would not be suitable. Thus, we need to consider an intervention approach that
does not rely on users’ motivations. In addition, even when a user is not
focusing on the video, intervention approaches that interrupt the user should
be avoided since such approaches might lead them to miss subsequent content.

These points led us to adopt an approach based on Mindless Computing (Adams et
al., 2015) that leverages human biases or unconscious behaviors to induce
behavioral change. Since such an intervention approach does not consume the
user’s conscious awareness to be effective, it is considered less reliant on
their motivation to pay attention. Moreover, it enables us to design a less
interruptive intervention than explicit alerts, as Adams et al. (Adams et al.,
2015) confirmed that their mindless approach using auditory feedback could
influence people’s behavior when talking without annoying them.

Furthermore, we presume that the mindless approach will reveal a new advantage
when integrated with a sensing module based on machine learning techniques, as
mentioned in Section 1. More specifically, although machine learning systems
enable various sensing scenarios, humans tend to evaluate such systems’
mistakes more severely than human mistakes (Dietvorst et al., 2015). In
addition, the trust that machine learning systems lose as a result of their
failure is usually greater than the trust they gain from their success (Yu et
al., 2017). Consequently, people often become less hesitant to override
outputs from machine learning systems after seeing their failures (De-Arteaga
et al., 2020). Moreover, it has been suggested that people with a high level
of cognitive load will have less trust in interactions with machine learning
systems (Zhou et al., 2017). These discussions imply the risk posed by the
false-positive detection of the sensing module in intervening with users—that
is, mistakenly alerting them in an explicit manner during video-based learning
situations would frustrate them and lead them to disregard the alerts. On the
other hand, since the mindless approach does not consume conscious awareness,
unlike the alerting approach, it might mitigate the negative effects caused by
false positives.

We therefore suppose that the mindless approach would be suitable as an
intervention in the context of video-based learning. In particular, we believe
that this is a plausible solution to the current situation where effective
interventions for video-based learning have not been well investigated, as
discussed in Section 2.1.

###  3.2. Designing Mindless Attractor

To design the mindless approach leveraging human biases or unconscious
behaviors, we exploited the nature of human speech communication. Our design
is based on the following requirements we considered in view of using the
mindless approach in video-based learning situations.

Avoid interruption due to interventions.:

Considering that video-based learning is sometimes delivered in the form of
live streams or in a synchronous manner (Baturay, 2015), interrupting users
due to interventions should be avoided, as it can cause them to miss
information and counteract our aim of helping them pay attention. This
requirement is one reason to eliminate the use of alerting approaches, as we
discussed their interruptive aspect in Section 3.1.

Use a modality that users will not neglect.:

To intervene with users who are not paying attention to the video, it is
important to use a modality that is always reachable for users. In this
regard, though it is possible to leverage human perceptual bias to design the
mindless approach by showing something on a display, this would not be
suitable because the user can take their eyes off the display, especially when
performing other tasks using a smartphone (Oeppen et al., 2020). On the other
hand, it seems more unlikely that the user would not hear the audio due to
muting it while in video-based learning situations.

Function without external devices.:

Though the use of external devices would extend the range of possible
interventions, such as using a tactile stimulus (Lee and Starner, 2010), it
raises an additional cost to utilize the interventions. Therefore, it is
desirable to design an intervention that could be integrated into video-based
learning situations without requiring external devices.

As we reviewed in Section 2.3, it has been suggested that humans unconsciously
respond to paralinguistic cues in speech, such as a change in pitch, volume,
and speed. In our case, we considered perturbing the pitch or volume of the
voice in the video to help users refocus their attention. We did not use speed
because it would be difficult to maintain time-series consistency when video-
based learning is conducted in a synchronous manner (e.g., live lectures
(Baturay, 2015)).

In addition, the perturbation is enabled and disabled repeatedly when the user
is seemingly not paying attention to the video, as Adams et al. (Adams et al.,
2015) emphasized the importance of cues to trigger different perceptions and
sensations in designing mindless approaches. Otherwise, if we activated the
perturbation once when the user became distracted and kept it thereafter, the
user would have less opportunity to refocus their attention as they became
acclimated to the changed pitch or volume.

###  3.3. Implementation

We used Python and PyAudio111https://people.csail.mit.edu/hubert/pyaudio/docs/
to perturb the audio signal in real time. The audio signal was captured in 16
kHz and the perturbation process was activated each 1/16 sec to ensure that
the perturbed signal was delivered without significant delay. The pitch shift
was performed using a library named
rubberband222https://github.com/breakfastquay/rubberband through time-shifting
and resampling the signal via Fourier transform. The volume change was
performed by directly multiplying the waveform double or halve. Our source
code is publicized at a GitHub
repository333https://github.com/hiromu/MindlessAttractor.

In addition, as we mentioned in Section 1 and Section 3.1, our mindless
intervention approach is expected to incorporate a sensing module that
monitors users’ behavior and detects when they are distracted. The detailed
implementation of the sensing module is later explained in Section 6.3.

##  4\. Hypotheses

Up to this point, we have introduced Mindless Attractor, which is designed as
an intervention for users during video-based learning that incorporates a
sensing module based on machine learning techniques. It computationally
perturbs the pitch and volume of the voice in the video in real time to
refocus users’ attention when they seem to be distracted from the video. Our
design rationale for the proposed approach, which we discussed in Section 3.2,
imposes the following hypotheses, which need to be verified to ensure the
validity and effectiveness of the proposed approach.

First, as we discussed in Section 3.1, our proposal is based on the concept of
Mindless Computing (Adams et al., 2015) so as to ensure that the intervention
works without relying on user motivation and without interrupting users. To
satisfy these points, we should examine whether Mindless Attractor can
influence users’ behavior in a mindless manner, i.e., without consuming their
conscious awareness.

> H1: Mindless Attractor is an effective means to refocus the attention of
> users in video-based learning situations without consuming their conscious
> awareness.

If H1 holds, we have two choices for inducing behavioral change in users
(i.e., drawing their attention back to the video): alerting users in an
explicit manner or intervening in a mindless manner. Here, as we discussed in
Section 3.1, we expect that the proposed approach will be favored over
alerting approaches when combined with a machine learning-based sensing module
that detects when users are losing attention. More specifically, the fact that
such a sensing module may produce false positives implies the risk of
mistakenly intervening in users, which can be annoying when we alert them
explicitly. Thus, we posit our second hypothesis:

> H2: Mindless Attractor is not only an effective means to refocus users’
> attention but is also preferred by users when combined with a machine
> learning-based sensing module, while the alerting approach is not accepted.

If these hypotheses are supported, we can pave the way for intervening with
users in real time to support their participation during video-based learning.
With this motivation, we evaluated these hypotheses by conducting a series of
experiments.

##  5\. Experiment I: Evaluation of H1

###  5.1. Design

To evaluate H1, we conducted an experiment that replicated video-based
learning situations. We used a within-participant design comparing a treatment
condition using Mindless Attractor with a control condition that did not
intervene in participants. Then, H1 is supported if the following two points
are confirmed: Mindless Attractor helps participants refocus their attention,
and Mindless Attractor does not consume participants’ conscious awareness.

###  5.2. Measure

We prepared two measures corresponding to the above two points to be
confirmed: recovery time and cognitive workload.

####  5.2.1. Recovery Time

This metric indicates the time that it took for participants to return their
attention to the video after losing focus. If Mindless Attractor helps
participants refocus their attention, the time that they are distracted should
be shortened in comparison to the case in which no intervention was taken.

To compute this metric, we collected human annotations for each participant
denoting whether the participant was paying attention or not. As we explain in
the detailed procedure description in Section 5.5, an experimenter observing
the state of the participants annotated in real time so that the recovery time
could be calculated later.

####  5.2.2. Cognitive Workload

This metric was used to evaluate whether Mindless Attractor consumed the
participants’ conscious awareness or not. Measuring cognitive workload is
common in the previous studies proposing alerting approaches (Saint-Lot et
al., 2020; Lee and Starner, 2010). Whereas they aimed to show that their
proposed approaches exhibited lower workload compared to other possible
approaches, we compared the metric between the control and treatment
conditions. If the cognitive workload in the treatment condition is not
significantly different from that in the control condition, it suggests that
Mindless Attractor does not consume participants’ conscious awareness. In our
study, we used the NASA-TLX questionnaire (Hart and Staveland, 1988; Byers et
al., 1989) to measure cognitive workload, in the same manner as the previous
studies (Saint-Lot et al., 2020; Lee and Starner, 2010).

We note that it would be possible to evaluate whether Mindless Attractor
consumes the participants’ conscious awareness by asking them whether they
noticed the perturbation. However, to do so, we would need to conceal from the
participants that they would be subject to an intervention, which would create
an unrealistic situation if we consider the practical applications of the
proposed approach. More specifically, it is unlikely that users in video-based
learning situations would be subject to interventions without opt-in consent;
that is, they would use Mindless Attractor of their own accord to focus on
videos or at least would be notified about the possibility of the
intervention. In addition, as we mentioned in Section 2.2, Adams et al. (Adams
et al., 2015) explained that the mindless approaches work regardless of
whether a user knows their mechanisms or not, as they do not depend on the
user’s conscious awareness. Thus, we used this measure based on NASA-TLX and
also notified the participants beforehand that they would be subject to
interventions.

###  5.3. Material

To replicate a video-based learning situation, we prepared a video recording
of a 30-minute lecture on urban sociology. As this experiment was conducted
remotely, the video was presented to the participants using the screen-sharing
function of Zoom444https://zoom.us/.

By following the implementation we described in Section 3.3, we also prepared
a client software that modifies Zoom’s audio output to perform our
intervention. This software captures and perturbs the audio output in real
time when it receives an activation command from a control server via
WebSocket. Here, we conducted a pilot study in the same manner as Adams et al.
(Adams et al., 2015) to find the best parameters for intervening without
causing distractions. Consequently, we implemented four perturbation patterns:
halving or doubling the volume and lowering or raising the pitch by one tone.
The software then activates one of the four patterns randomly so as to enable
the comparison of their effectiveness for helping the participants refocus
their attention. Since Zoom automatically removes noises and extracts voices,
we confirmed that our naïve implementation of pitch shifting based on fast
Fourier transform would be sufficient for the purposes of this experiment.

We further prepared an experimenter console in the control server to record
annotations concerning whether the participant was paying attention or not.
The console was implemented to enable sending the activation and deactivation
command to the client software when the participant started to divert their
attention from the video and refocused their attention, respectively.

###  5.4. Participants

This experiment involved 10 participants, three of whom were female. They were
recruited via online communication in a local community where over 100
university students gather. As described later in Section 5.5, our
experimental procedure required participants to be observed by a remote
experimenter so that their state of attention could be annotated. Therefore,
we asked them to prepare a PC with a webcam in a quiet room as well as to
enable their faces to be captured.

###  5.5. Procedure

Each participant underwent one session of watching the 30-minute video using a
computer connected over Zoom, as we mentioned in Section 5.3. To replicate the
usual situation of video-based learning, in which learners have some reasons
to watch the video, we told participants in advance that they would be asked
to write a few sentences summarizing the video. At the same time, we asked
them to bring their smartphones and told them that we would not prohibit the
use of smartphones so that they could be distracted as usual (Oeppen et al.,
2020).

As depicted in Figure 1, each session was divided into two parts of 15 minutes
each: one with no intervention and another involving interventions. To
normalize the order effect, we balanced the order of the two parts: five
participants first experienced the part with no intervention, and the others
first experienced the part involving interventions. After each part, the
participant was asked to write a summary and fill out the questionnaire
measuring cognitive workload. Note that these two parts do not correspond to
the control and treatment conditions, as explained in the following
paragraphs.

Figure 1. Example illustration of the procedure for our first experiment. (A)
Half of participants first experienced the part with no intervention and then
experienced the part involving interventions, and (B) the others followed the
reversed order.

Example illustration of the procedure for our first experiment. After each
participant installed the client with opt-in consent at the beginning, they
experienced two parts: one with no intervention and another involving
interventions. In the part involving interventions, an experimenter annotated
whether the participant was paying attention to the video. Half of
participants followed this order, and the others first experienced the part
involving interventions and then experienced the part with no intervention.

In the part involving interventions, an experimenter observed the state of a
participant, including their use of smartphones, and annotated whether they
were paying attention to the video or not. When the experimenter pressed a
button on the experimenter console to record the timestamp at which the
participant diverted their attention from the video, the console assigned
either the control or treatment condition with a 50% probability of each. Note
that the selected condition was concealed from the experimenter in order to
avoid the experimenter bias in the annotations. If the treatment condition was
assigned, the console sent the activation command to the client, and the
client then repeatedly enabled and disabled one of the four perturbation
patterns every 3 seconds, as explained in Section 3.2. This intervention
continued until the client received the deactivation command indicating that
the experimenter pressed another button to record the participant’s recovery
from the distraction. On the other hand, if the control condition was
assigned, no command was sent to the client. Consequently, based on the
assigned conditions and the recorded timestamps, the recovery time could be
calculated and compared.

The other part (with no intervention) was prepared to evaluate the cognitive
workload. We compared its cognitive workload score with that of the part
involving interventions, which were activated on a random basis. If the
intervention did not consume the participant’s conscious awareness, the scores
of the two parts would not be significantly different.

In addition, at the end of the session, we asked the participants for their
comments about their feelings or anything they noticed. In total, the entire
session took about an hour to complete.

###  5.6. Results

Table 1. Comparison of the recovery time and cognitive workload score between the control and treatment conditions. The treatment condition involved the mindless intervention. Measure | Treatment | Control |  $p$-value
---|---|---|---
Recovery time |  $17.71~{}s$ ($\pm 10.52~{}s$) |  $32.25~{}s$ ($\pm 16.92~{}s$) | ¡ 0.0001
Cognitive workload |  $26.00$ ($\pm 10.32$) |  $27.00$ ($\pm 9.13$) | 0.5212

####  5.6.1. Recovery Time

Figure 2. Distribution of the recovery time across each participant and the
experimental conditions.

Distribution of the recovery time across each participant and the experimental
conditions. The participants tended to spend more time recovering their
attention in the control condition.

As shown in Table 1, the proposed intervention significantly shortened the
recovery time according to the unpaired $t$-test (Cohen’s $d=1.0044$,
$p<0.0001$). The distribution of the recovery time is shown in Figure 2, which
also confirms this reduction. This result supports that Mindless Attractor
helped participants refocus their attention.

We also investigated which of the four perturbation patterns (i.e., halving or
doubling the volume and lowering or raising the pitch by one tone) effectively
helped participants refocus their attention. We examined the last perturbation
pattern before each time the participant returned their attention and counted
their occurrence, as shown in Table 2. This examination is based on our
assumption that the intervention just before the participant’s attention
returned is the cause of the change in the participant’s state. According to
the $\chi^{2}$-test comparing with the total occurrence, the results were not
significantly different in that each pattern equally helped participants
recover their attention (Cramer’s $V=0.1220$, $p=0.2794$). In other words, we
can conclude that there was no significant difference in the effectiveness of
the four perturbation patterns.

Table 2. Occurrence of the four perturbation patterns that were executed just before participants returned their attention. The comparison with the total occurrence suggests that there was no significant difference in effectiveness ($p=0.2794$). Perturbation | Halve the volume | Double the volume | Lower the pitch | Raise the pitch
---|---|---|---|---
Occurrence just before | 19 | 7 | 14 | 16
participants returned their attention
Total occurrence | 50 | 47 | 50 | 55

####  5.6.2. Cognitive Workload

We also could not find a significant difference in participants’ cognitive
load scores according to the paired $t$-test (Cohen’s $d=0.2110$, $p=0.5212$),
as presented in Table 1. That is, it is suggested that Mindless Attractor did
not consume participants’ conscious awareness or at least did not negatively
affect participants’ cognitive load by consuming their conscious awareness.
Thus, in combination with the effect on the recovery time, H1 was supported.

####  5.6.3. Comments

We also examined the comments that the participants wrote at the end of the
experiment. At first, we realized that three participants mentioned that they
did not notice any intervention, although they were informed of the
intervention beforehand. Interestingly, the recovery time for these three
participants also showed a significant difference (Cohen’s $d=0.8105$,
$p=0.0122$) between the treatment ($15.88~{}s$ on average) and control
($28.34~{}s$ on average) conditions. Thus, it is suggested that the mindless
approach worked even when it was not noticed by participants, further
supporting that Mindless Attractor did not consume the participants’ conscious
awareness. This point not only corroborates H1 but also shows consistency with
the discussion by Adams et al. (Adams et al., 2015).

It was also interesting that, although five participants mentioned that they
noticed the changes in volume, no participant recognized the changes in pitch.
That is, although no significant difference was found between the
effectiveness of the four perturbation patterns in Table 2, their
noticeability varied, suggesting further room for investigation.

Nevertheless, no participants regarded the mindless intervention as disruptive
or annoying; rather, two participants made positive comments about it:

> I found it useful because it naturally brought my attention back to the
> video when I thought something might have changed in the speech. (P1)

> It was nice as it made me feel like…the computer was recommending me to
> concentrate, rather than warning me. (P4)

In particular, the latter comment suggested that the mindless approach can
mitigate the negative effect that might be caused by false-positive detection
when combined with a machine learning-based sensing module. These results
motivated us to conduct a second experiment to evaluate this possibility, as
discussed in Section 4 when posing H2.

##  6\. Experiment II: Evaluation of H2

###  6.1. Design

To evaluate H2, we conducted an experiment that replicated a video-based
learning situation in the same manner as Section 5. However, in this case, we
combined a machine learning-based sensing module rather than manually
activating interventions and compared the effects of the mindless approach and
the alerting approach. Here, we used a within-participant design over three
conditions: mindless, alerting, and control (no intervention). We added the
control condition to confirm that the proposed approach was at least effective
in contributing to refocusing users’ attention as an automated system
controlled by a machine learning-based sensing module. H2 is thus supported if
the following two points are confirmed: Mindless Attractor helps participants
refocus their attention, and participants favor Mindless Attractor over the
alerting approach.

###  6.2. Measure

Similar to the first experiment, we measured time with regards to whether
participants were paying attention. However, we introduced a different
approach for evaluating the time factor, i.e., total distracted time instead
of the recovery time. In addition to this, we introduced a measure for
behavioral intention.

####  6.2.1. Total Distracted Time

Although we have confirmed that Mindless Attractor can help participants
return their attention, it is desirable to investigate whether the total time
that they are distracted during video-based learning is decreased. In other
words, it may be possible that, though the mindless approach shortened the
recovery time, the participants were distracted more frequently, especially
when the mindless approach was combined with a machine learning-based sensing
module having a risk of false positives.

To compute this metric, we collected human annotations for each participant,
as we did in Section 5, and aggregated the duration when the participants were
not paying attention. If the total distracted time in the mindless condition
is significantly shorter than in the control condition, it is suggested that
Mindless Attractor can make users more likely to pay attention, even in
combination with a machine learning-based sensing module.

It should be noted that, due to the false negatives of such a sensing module,
there would be a case when the intervention is not triggered even when the
participant is actually losing their attention and a case when the
intervention is deactivated before the participant refocus. Therefore,
calculating the recovery time as in Section 5.6 is not appropriate in this
second experiment, further rationalizing the introduction of the total
distracted time as a different metric.

####  6.2.2. Behavioral Intention

This metric was prepared to evaluate whether the mindless approach was favored
over the alerting approach. The concept of behavioral intention is guided by
the Technology Acceptance Model (Davis, 1989), which explains users’ attitudes
towards technologies, and is frequently used to evaluate how likely
individuals are to use the technologies. We used the questionnaire to measure
behavioral intention in the same manner as the previous studies (Venkatesh et
al., 2003). If this score in the mindless condition is significantly better
than that in the alerting condition, we can confirm that Mindless Attractor
can be favored over the alerting approach, especially when it works as an
automated system with a sensing module.

###  6.3. Material

Similar to our first experiment, we prepared a video recording of a 30-minutes
lecture on social sciences. The experiment was conducted remotely and the
video was presented using Zoom’s screen-sharing function, as in the first
experiment. However, in this second experiment, we developed a system that
automatically detected the status of participants’ attention. To implement
this sensing module, we followed previous studies that estimated participants’
attentiveness based on their visual cues, which we reviewed in Section 2.1.
Specifically, we analyzed the video stream of face images of each participant
by leveraging machine learning techniques that can detect their head pose in
real time. If the module detected that the participant was looking off the
screen, the system judged that the participant was failing to pay attention to
the video lecture, and activated an intervention.

Figure 3 illustrates how the system processed the video streams of
participants and intervened in them. Videos were processed in a frame-by-frame
manner. First, a human face was detected and located in each frame using a
deep learning model, RetinaFace (Deng et al., 2020). We used this model
because it achieves state-of-the-art performance and its pretrained model is
publicly released. Face alignment was then performed to obtain facial
keypoints using a deep learning model proposed by Fang et al. (Fang et al.,
2017) that is also known to estimate keypoints with high accuracy. Finally,
based on the estimated facial keypoints, the head pose was calculated by
solving a perspective-n-point problem. These calculations were performed using
a dedicated computation server with an NVIDIA V100 Tensor Core GPU.

Next, the estimated head pose was passed to the experimenter’s PC, a
conventional laptop with a 2.2 GHz Intel Core i7 processor. This PC checked
whether the passed head direction was off-screen or not. The experimenter had
conducted a calibration process beforehand to calculate the threshold for this
judgment, in which participants were asked to track a red circle that appeared
and moved along the edge of the screen. Participants were told to track the
circle by moving their head, i.e., not following it only by moving their gaze.
We then calculated the maximum head rotations for each direction (top–down and
left–right) and regarded them as the range where the head is toward the
screen. In other words, when the estimated head pose was out of this range,
then the system judged that the participant was looking off the screen, and
thus, losing their attention.

While the participants were watching the video, changes in their state–i.e.,
whether they were looking at the screen or not–were shared with another
control server maintaining a WebSocket connection with the client software.
The control server then correspondingly sent activation or deactivation
commands in the same manner as the first experiment. All of the above
processes were performed in real time with a frame rate of 15 FPS.

Figure 3. Architecture of the entire system we implemented for the second
experiment.

Architecture of the entire system we implemented for the second experiment.
From each frame of the video capturing the participant’s face, a GPU server
detects the face position, calculates the face alignment, and estimates the
head pose. Based on the head pose, an experimenter PC activates interventions
via a control server.

In addition to the sensing module, we implemented an intervention to
explicitly alert users in the client software, to be compared with our
proposed approach. In this case, the client software played a short beep for
0.1 seconds, which followed the previous study’s use of a beep alert (Kayukawa
et al., 2019), rather than perturbing the audio output. Once the alert was
activated, it replayed the same beep every 3 seconds until it received the
deactivation command, in the same manner as the mindless condition.

###  6.4. Participants

This experiment involved 20 participants, five of whom were female. They were
recruited in the same manner as we did in the first experiment. Eight of the
participants participated in our first experiment, which had been held at
least two weeks before this experiment. The participants were asked to prepare
a PC in a quiet room and to enable their faces to be captured with a webcam,
as in the first experiment.

###  6.5. Procedure

Similar to the first experiment, each participant experienced a session of
watching the 30-minute video using a computer connected over Zoom. As before,
we told participants in advance that they would be asked to write a few
sentences summarizing the video and also allowed them to bring and use their
smartphones in the session.

As illustrated in Figure 4, each session consisted of three parts lasting 10
minutes each: one with no intervention, another with the mindless approach,
and a third with the alerting approach. The order of these three parts was
automatically randomized among participants, as we will describe later in this
section. After each session, participants were asked to write the summary.
They were also asked to fill out the questionnaire measuring behavioral
intention when they finished a part with either the mindless or alerting
approach. We compared the scores between the two conditions to examine which
approach participants favored.

Before starting the first session, the experimenter performed a calibration
process to determine the threshold for whether the participant’s head pose was
out of the screen, as described in Section 6.3. The experimenter explained
that the participants should not move their PC until the entire process was
complete and advised them to find a comfortable position before the
calibration process started.

In each of the three parts, the experimenter manually annotated whether the
participant was paying attention to the video lecture, similar to the first
experiment. To avoid bias, the experimenter was blind to which of the three
conditions had been applied to the participant. Specifically, the control
server (see Figure 3) decided the order of conditions in each session, and the
experimenter did not have access to this information until the session ended.
The obtained annotations were used to calculate the total distracted time for
each part.

In addition, our developed machine learning-based sensing module triggered
interventions to the participants in either the alerting or mindless
condition, as described in Section 6.3. In the alerting condition,
participants were exposed to the beep sound when the system judged that they
were losing attention, whereas they were exposed to perturbations in the
speech in the mindless condition. In the control condition (i.e., that with no
intervention), the client system did not intervene. In each part, the sequence
of the system’s judgment was recorded along with timestamps, which we later
used to assess the accuracy of the sensing module by comparing it with the
human annotations.

Finally, at the end of the session, we asked the participants for their
comments about their feelings or anything they noticed. In total, the entire
session took about an hour to complete.

Figure 4. Example illustration of the procedure for our second experiment.
Each participant was randomly assigned to one of six possible orders of the
three conditions.

Example illustration of the procedure for our second experiment. After each
participant installed the client and performed the calibration process at the
beginning, they experienced three conditions: mindless, alerting, and control.
Each participant was randomly assigned to one of six possible orders of the
three conditions.

###  6.6. Results

####  6.6.1. Sensing Accuracy

Table 3. Confusion matrix between the human annotations and the detection results of the machine learning-based module in regard to participants’ attentive state. |  | Detection result
---|---|---
|  | Attentive | Distracted
Human annotations | Attentive | 435.4 min (68.5 %) | 78.0 min (12.3 %)
Distracted | 51.4 min (8.1 %) | 70.7 min (11.1 %)

We first examined the accuracy of our machine learning-based sensing module in
detecting participants’ attentive state. We compared the human annotations and
the detection results of the module and obtained Table 3. Though our aim is
not to develop a detection system, the accuracy across all the participants
was 79.6 %, which was relatively close to the previous study (Thomas and
Jayagopi, 2017) that achieved the accuracy of 82–85 % using only head pose. We
note that the accuracy varied among participants (64.9–93.0 %), which implies
that some environmental factors (e.g., the distance to camera or lighting
conditions) might largely affect the detection results. At the same time, the
sensing module exhibited a lot of false-positive detection, as its precision
was 47.6 %, which suited our aim to investigate the effect of Mindless
Attracter while having a risk of false positives.

####  6.6.2. Total Distracted Time

Figure 5. Comparison of participants’ total distracted time. We found
significant differences between the control condition and the other
conditions.

This bar chart with standard errors shows participant’s total distracted time
during each session, comparing the mindless, alerting, and control condition.
The time for the control condition is significantly longer than that for the
mindless and alerting condition (**). The time for the mindless and alerting
condition is not significant.

Figure 6. Comparison of how many times participants got distracted. We found
no significant difference between the three conditions.

This bar chart with standard errors shows how many times participants got
distracted during each session, comparing the mindless, alerting, and control
condition. There is no significant difference between each condition.

Figure 7. Comparison of participants’ scores of the behavioral intention. We
found a significant difference between the mindless and alerting conditions.

This bar chart with standard errors shows the scores of the behavioral
intention, comparing the mindless and alerting condition. The score for the
mindless condition is significantly higher than that for the alerting
condition (**).

Next, based on the human annotations, we calculated the total distracted time
for each participant, as presented in Figure 7. We found a significant
difference among the three conditions according to ANOVA ($F(2,57)=8.5773$,
$\eta^{2}=0.2313$, $p=0.0005$), and thus conducted a post-hoc test. As a
result, the control condition showed significant differences against the
mindless and alerting conditions (Cohen’s $d=1.1795$, $p=0.0013$ and Cohen’s
$d=1.0828$, $p=0.0032$, respectively). On the other hand, we found no
significant difference between the mindless and alerting conditions.

From this result, it was confirmed that Mindless Attractor is an effective
means to refocus users’ attention even when combined with a machine learning-
based sensing module, as the mindless condition significantly reduced the
total distracted time than the control condition. In addition, it is notable
that Mindless Attractor would work effectively as well as the conventional
alerting approaches since the mindless and alerting conditions showed similar
distracted times.

We also examined how many times the participants got distracted because it was
possible that our interventions increased the frequency even though the total
distracted time was reduced. As shown in Figure 7, we did not find significant
differences among the three conditions ($F(2,57)=0.1796$, $\eta^{2}=0.0062$,
$p=0.8360$). It can be explained as follows: the participants were almost
equally likely to lose focus in all the three conditions; but, if there was an
intervention, they often refocused their attention to the video earlier, as
confirmed in the first experiment; as a result, their distraction time in the
mindless and alerting conditions was significantly reduced than the control
conditions. From these results, we conclude that H2 was supported in terms of
the effectiveness of Mindless Attractor.

####  6.6.3. Behavioral Intention

Lastly, we compared participants’ scores of the behavioral intention between
the mindless and alerting conditions. As presented in Figure 7, we found a
significant difference (Cohen’s $d=0.7025$, $p=0.0054$) according to the
paired $t$-test. That is, compared to the alerting approach, the participants
showed their stronger intentions to use the implemented system when it is
combined with the mindless approach. This result supports that Mindless
Attractor is much preferred by users than the alerting approach, as we
hypothesized as H2.

####  6.6.4. Comments

The above results coincided with H2; that is, Mindless Attractor helps
participants refocus their attention and it is favored over a conventional
alerting approach. In addition, the comments obtained at the end of the
experiment corroborated H2, especially in regard to the unacceptability of the
alerting approach.

> I felt like the beep sound made me lose focus. It was frustrating,
> especially when I was concentrating. (P9)

> The beep felt like noise because it overlapped the speech though I wanted to
> listen to what was being said. As a result, my concentration was more
> disrupted than the case that I had not used the system. (P12)

> I thought the one with the beep sound might be a good signal until halfway
> through, but then it came to ring repeatedly even though I was
> concentrating. As a result, I stopped caring about the sound. (P2)

These comments confirmed our anticipation; that is, explicitly alerting users
based on false-positive detection makes them distracted and frustrated, which
can lead them to ignore the intervention. In addition, one participant
suggested that such negative effects can be caused even when the intervention
was activated by accurate detection:

> I was disgusted by the alarm, which rang when I was using my smartphone for
> googling a word I never heard. (P8)

In contrast, the mindless condition was totally favored, as follows:

> In the part [of the mindless condition], I felt like I was able to focus on
> the lecture relatively well. (P12)

> I did not notice much of a change in the audio, but when I compare the three
> parts, I seemed to be able to maintain my concentration the most. I think
> having such a system that brings back my attention without making a big deal
> will help me stay focused in usual situations. (P3)

> When the pitch of the speech became higher, I paid attention to the video as
> I felt strange a little. It did not provide a sense of being angry, compared
> to the beep alarm. (P11)

These comments corresponded to the comparison of the scores of behavioral
intention (Figure 7).

Furthermore, 17 of 20 participants agreed they often have trouble maintaining
their attention and computationally solving it would be beneficial, like:

> I find it difficult to maintain my attention in such online situations
> because of the lack of eyes around. (P1)

In addition, they suggested that the proposed approach can be used outside
video-based learning situations.

> I thought it would be nice to be able to introduce a similar system in
> offline situations. I will appreciate it if some device such as a smartwatch
> helps me refocus when I am losing my attention from an important
> conversation. (P4)

The obtained comments not only supported the effectiveness of Mindless
Attractor through supporting H2 but also highlighted the further potential of
the proposed approach.

##  7\. Discussion

So far, by verifying H1 and H2, we have demonstrated that Mindless Attractor
works effectively as a novel intervention approach to support users’
participation during video-based learning. In this section, we contemplate the
findings of our study, envision future application scenarios, and discuss
limitations and directions for future work to further pave the way for
supporting users in video-based communication.

###  7.1. Necessity of Mindless Intervention in Machine Learning-Based Systems

The results of our second experiment supported H2: Participants favored the
proposed mindless approach, while the alerting approach was not accepted.
Specifically, the obtained comments suggested that participants were annoyed
by the alerts when they were triggered by false positives of the sensing
module. In other words, mistakenly intervening in an explicit manner while
users are concentrated on the main task can unnecessarily consume their
conscious awareness and eventually disrupt their experience.

Indeed, such failures in designing automated systems based on machine
learning-based sensing modules have been pointed out in a recent guideline for
human–AI interaction (Amershi et al., 2019). That guideline emphasized the
importance of considering that such AI-infused systems may demonstrate
unpredictable behaviors due to false positives and false negatives.
Consequently, it was suggested that an effective approach in designing AI-
infused systems is to enable users to dismiss the undesired functions
instantly. In light of this, our proposed mindless approach can be a promising
direction that follows this guideline, as it does not consume users’ conscious
awareness, letting them not mind the mistakenly triggered interventions
without much cognitive workload. Therefore, we believe that Mindless Attractor
can support users as a novel intervention method integrated with machine
learning-based systems in various cases, not limited to the presented case
(i.e., video-based learning).

###  7.2. Application Scenarios

As mentioned in Section 1, the importance of helping participants be attentive
during video-based communication has been emphasized in various contexts. In
this regard, we believe that Mindless Attractor can be used effectively not
only in video-based learning but also in other situations using video-based
communication. For example, it can be employed to help participants in video-
based meetings be more attentive in the same manner as shown in this study.
Here, we note that a few studies have aimed to provide real-time feedback to
participants in meetings (Schiavo et al., 2014; Samrose et al., 2017). For
example, CoCo is a system designed to achieve balanced participation through
feedback, such as showing a pie chart representing the participation ratio
that can be estimated from speaking length and frequency (Samrose et al.,
2017). Similar to the discussion we had with regard to video-based learning,
these techniques of providing explicit feedback require participants to be
motivated to change their behavior, i.e., to be more attentive to the meetings
based on the feedback. Therefore, we can expect that Mindless Attractor will
be a promising alternative approach in that it does not consume participants’
conscious awareness during meetings, even when combined with machine learning-
based sensing systems.

Furthermore, we envision a future where Mindless Attractor can be utilized in
everyday interpersonal interactions. If we can assume that wearing earphones
in our daily life become more popular, it is possible to perturb the sound
they hear to utilize Mindless Attractor. For example, once the system detects
that the user is failing to pay attention during a conversation based on their
behavioral or physiological data, the envisioned system can intervene in a
mindless manner by modifying the voice they hear. Note that such demand for
offline use was indeed observed in one participant’s comment (P4) in our
second experiment.

It is noteworthy that we verified the effectiveness of Mindless Attractor in
the experiments in which users used it with prior consent. This lets us
imagine further practical applications utilizing Mindless Attractor as an opt-
in function. More specifically, it would allow users to selectively turn the
system on and off on their own, according to their situations and motivations.
For example, if a user attends an important lecture or meeting and thinks that
they need the assistance, they can actively allow themselves to be exposed to
the mindless intervention by turning on the system. In other words, our
results, which showed that the mindless approach worked with opt-in consent,
will pave the way for the user-centered exploitation of computational
interventions with which users can augment their levels of attention.

###  7.3. Limitations and Future Work

Though our experiments have demonstrated that Mindless Attractor is a
promising approach, there are some limitations. Initially, further
investigations involving a greater number of participants and diverse lecture
content are desirable to generalize our results. For example, if a lecture is
so attractive that learners are not distracted from the video, the proposed
approach would not be necessary, while at worst it would not be harmful, as
its impact on cognitive load was not observed in Section 5.6.

Secondly, our approach and evaluations are based on the discussion of Mindless
Computing proposed by Adams et al. (Adams et al., 2015), considering users
whose motivation for obeying the intervention is not always assumed. In fact,
we have given some consideration to the experimental designs so that the
participants would not become much motivated to the video, like allowing the
use of smartphones. Thus, we skipped the measurement of the participants’
motivation in our studies. However, this means that their results would not
necessarily guarantee the universal effectiveness of the proposed method for
users with any levels of motivation. Thus, evaluating participants’ motivation
and exploring its correlation with the efficacy of Mindless Attractor can be a
promising future work.

In addition, the accuracy of the machine learning-based sensing module in the
second experiment can be improved using the latest techniques (Thomas and
Jayagopi, 2017; Zaletelj and Kosir, 2017; Hutt et al., 2017; Veliyath et al.,
2019). In this study, we used a naïve approach based on head pose to
investigate the effect of the proposed approach with false-positive detection.
Although our sensing approach achieved a certain level of accuracy, as
discussed in Section 6.6, there is room to further sophisticate the algorithm.
It remains to be explored how users would feel if the alerting approach is
combined with a much more accurate sensing module. Nevertheless, we believe
that our mindless approach can be an effective intervention because false
positives will still remain.

In relation to this, it is noteworthy that recent works have proposed methods
for drowsiness detection from human visual cues (Ghoddoosian et al., 2019).
Thus, it can be explored in future work whether Mindless Attractor can help
participants who get sleepy during video-based learning, by integrating such a
detection technique in the sensing module. Examining the boundary of the
effectiveness of the proposed approach in such a situation would inform us of
further possible approaches, such as a hybrid of the mindless and alerting
interventions.

We also acknowledge that refining the design of alerts can mitigate the
negative impact suggested in the second experiment. While we used a simple
beep as an alert, alternative methods to inform users in less annoying manners
are possible. In particular, Weiser and Brown conceptualized “calm technology”
as a more acceptable communication channel from computers (Weiser and Brown,
1995, 1997). For example, alerting users with less explicit sounds (e.g.,
birds chirping) could be preferred to a simple beep sound. In addition, if we
ignore the requirement of using the auditory modality, showing a status lamp
on display is an alternative to inform users that they are losing attention.
However, as Adams et al. pointed out, these techniques require users’
conscious awareness (e.g., interpreting the status based on the lamp) to
induce behavioral change (Adams et al., 2015), while mindless computing does
not. Therefore, Mindless Attractor can be differentiated from alerting
approaches in that it can work without consuming users’ conscious awareness,
as suggested in the first experiment (see Section 5.6). That said, it is
desirable to explore sophisticated alerting approaches to draw further
implications in comparison to our mindless approach.

At the same time, the design of the mindless intervention has also room for
exploration. Currently, as explained in Section 3.2, we decided to perturb the
pitch or volume of the voice based on the nature of human speech
communication. Though we did not statistically examine the results due to the
small number of perturbations activated for each participant, there were
individual differences in terms of their effectiveness, which would imply the
possibility of personalizing the intervention patterns. Moreover, human brains
are known to show a special response to a self-voice (Conde et al., 2015) or a
familiar voice (Beauchemin et al., 2006). Thus, a possible intervention might
involve computationally modifying a voice so as to be similar to a self-voice
or familiar voice when learners are not paying attention. This can be achieved
through recent techniques for high-fidelity real-time voice conversion (Toda
et al., 2012; Arakawa et al., 2019).

Looking toward production deployment, investigating whether the proposed
approach that helps learners pay attention contributes to their learning
performance could be a future study. Considering that previous studies
adopting explicit feedback to help learners pay attention have shown a
positive impact on performance (Baker et al., 2010; Xiao and Wang, 2016), our
mindless approach can be expected to have a positive effect. This is because
the mindless approach exhibited an effect on distracted time comparable to
that of the alerting approach in Section 6.6, while showing no significant
impact on the cognitive load in Section 5.6. Examining the long-term effect of
the proposed approach is also suggested for future work. Though our design is
based on the concept of Mindless Computing, which Adams et al. (Adams et al.,
2015) have described as having long-term effectiveness, it is difficult to
deny, without further investigation, the possibility that users will become
acclimated to the perturbations. However, even in this case, the combination
with voice conversion we mentioned above could be a remedy, as it enables as
many patterns of interventions as the number of conversion targets.

##  8\. Conclusion

We presented a novel intervention approach, Mindless Attractor, which helps
users refocus their attention in a mindless manner. The approach leverages the
nature of human speech communication and perturbs the voice that users hear
when they are losing their attention. Our first experiment confirmed the
effectiveness of Mindless Attractor in a video-based learning context by
showing that it helped users refocus their attention without consuming their
conscious awareness. Moreover, through a comparison with a conventional
alerting approach, our second experiment further supported the efficacy of our
proposed mindless approach when integrated as an automated system with a
machine learning-based sensing module. Based on the results of the
experiments, we discussed implications for utilizing mindless interventions,
especially in tandem with machine learning-based sensing modules, and
envisioned future application scenarios. Our findings and discussion pave the
way for developing novel mindless interventions that can be harnessed in
human–AI symbiosis.

###### Acknowledgements.

This work is partially supported by JST ACT-X, Grant Number JPMJAX200R, Japan.
Several components of the sensing module used in this study were offered by
ACES Inc., Japan.

## References

  * (1)
  * Adams et al. (2015) Alexander Travis Adams, Jean Marcel dos Reis Costa, Malte F. Jung, and Tanzeem Choudhury. 2015\.  Mindless computing: designing technologies to subtly influence behavior. In _Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing_. ACM, New York, NY, 719–730.  https://doi.org/10.1145/2750858.2805843
  * Amershi et al. (2019) Saleema Amershi, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, Eric Horvitz, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, and Paul N. Bennett. 2019.  Guidelines for Human-AI Interaction. In _Proceedings of the 37th ACM CHI Conference on Human Factors in Computing Systems_ , Vol. 3. ACM, New York, NY, 1–13.  https://doi.org/10.1145/3290605.3300233
  * Arakawa et al. (2019) Riku Arakawa, Shinnnosuke Takamichi, and Hiroshi Saruwatari. 2019\.  Implementation of DNN-based real-time voice conversion and its improvements by audio data augmentation and mask-shaped device. In _Proceedings of the 10th ISCA Speech Synthesis Workshop_. ISCA, Baixas, France, 93–98.  https://doi.org/10.21437/SSW.2019-17
  * Baker et al. (2010) Ryan Shaun J. D. Baker, Sidney K. D’Mello, Ma. Mercedes T. Rodrigo, and Arthur C. Graesser. 2010\.  Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments.  _International Journal of Human-Computer Studies_ 68, 4 (2010), 223–241.  https://doi.org/10.1016/j.ijhcs.2009.12.003
  * Baturay (2015) Meltem Huri Baturay. 2015\.  An Overview of the World of MOOCs.  _Procedia – Social and Behavioral Sciences_ 174 (2015), 427–433.  https://doi.org/10.1016/j.sbspro.2015.01.685
  * Beauchemin et al. (2006) Maude Beauchemin, Louis De Beaumont, Phetsamone Vannasing, Aline Turcotte, Claudine Arcand, Pascal Belin, and Maryse Lassonde. 2006.  Electrophysiological markers of voice familiarity.  _The European Journal of Neuroscience_ 23, 11 (2006), 3081–—3086.  https://doi.org/10.1111/j.1460-9568.2006.04856.x
  * Bidwell and Fuchs (2011) Jonathan Bidwell and Henry Fuchs. 2011.  _Classroom analytics: Measuring student engagement with automated gaze tracking_.  Technical Report. Department of Computer, University of North Carolina at Chapel Hill, Chapel Hill, NC. 1–17 pages.  https://doi.org/10.13140/RG.2.1.4865.6242
  * Byers et al. (1989) James C. Byers, Alvah C. Bittner, and Susan G. Hill. 1989\.  Traditional and raw task load index (TLX) correlations: Are paired comparisons necessary?. In _Proceedings of the 1989 Annual International Industrial Ergonomics and Safety Conference_. Taylor & Francis, Philadelphia, PA, 481–485.
  * Cobus et al. (2018) Vanessa Cobus, Hannah Meyer, Swamy Ananthanarayan, Susanne Boll, and Wilko Heuten. 2018\.  Towards reducing alarm fatigue: peripheral light pattern design for critical care alarms. In _Proceedings of the 10th Nordic Conference on Human-Computer Interaction_. ACM, New York, NY, 654–663.  https://doi.org/10.1145/3240167.3240218
  * Conde et al. (2015) Tatiana Conde, Óscar F. Gonçalves, and Ana P. Pinheiro. 2015\.  Paying attention to my voice or yours: An ERP study with words.  _Biological Psychology_ 111 (2015), 40–52.  https://doi.org/10.1016/j.biopsycho.2015.07.014
  * Coombs (2020) Crispin Coombs. 2020\.  Will COVID-19 be the tipping point for the Intelligent Automation of work? A review of the debate and implications for research.  _International Journal of Information Management_ 102182 (2020), 1–4.  https://doi.org/10.1016/j.ijinfomgt.2020.102182
  * Davis (1989) Fred D. Davis. 1989\.  Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology.  _MIS Quarterly_ 13, 3 (1989), 319–340.  https://doi.org/10.2307/249008
  * De-Arteaga et al. (2020) Maria De-Arteaga, Riccardo Fogliato, and Alexandra Chouldechova. 2020.  A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores. In _Proceedings of the 38th ACM CHI Conference on Human Factors in Computing Systems_ , Vol. 509. ACM, New York, NY, 1–12.  https://doi.org/10.1145/3313831.3376638
  * Deng et al. (2020) Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. 2020.  RetinaFace: Single-Shot Multi-Level Face Localisation in the Wild. In _Proceeedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition_. IEEE, New York, NY, 5202–5211.  https://doi.org/10.1109/CVPR42600.2020.00525
  * Dietvorst et al. (2015) Berkeley J. Dietvorst, Joseph P. Simmons, and Cade Massey. 2015\.  Algorithm aversion: People erroneously avoid algorithms after seeing them err.  _Journal of Experimental Psychology: General_ 144, 1 (2015), 114–126.  https://doi.org/10.1037/xge0000033
  * D’Mello et al. (2012) Sidney D’Mello, Andrew Olney, Claire Williams, and Patrick Hays. 2012.  Gaze tutor: A gaze-reactive intelligent tutoring system.  _International Journal of Human-Computer Studies_ 70, 5 (2012), 377–398.  https://doi.org/10.1016/j.ijhcs.2012.01.004
  * Fang et al. (2017) Haoshu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu. 2017\.  RMPE: Regional Multi-person Pose Estimation. In _Proceedings of the 2017 IEEE International Conference on Computer Vision_. IEEE, New York, NY, 2353–2362.  https://doi.org/10.1109/ICCV.2017.256
  * Fish et al. (1992) Robert S. Fish, Robert E. Kraut, Robert W. Root, and Ronald E. Rice. 1992.  Evaluating Video as a Technology for Informal Communication. In _Procedings of the 10th ACM CHI Conference on Human Factors in Computing Systems_. ACM, New York, NY, 37–48.  https://doi.org/10.1145/142750.142755
  * Ghoddoosian et al. (2019) Reza Ghoddoosian, Marnim Galib, and Vassilis Athitsos. 2019\.  A Realistic Dataset and Baseline Temporal Model for Early Drowsiness Detection. In _Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_. IEEE, New York, NY, 178–187.  https://doi.org/10.1109/cvprw.2019.00027
  * Goldschmidt (2020) Karen Goldschmidt. 2020\.  The COVID-19 Pandemic: Technology use to Support the Wellbeing of Children.  _Journal of pediatric nursing_ 53 (2020), 88–90.  https://doi.org/10.1016/j.pedn.2020.04.013
  * Guo et al. (2002) Linsong Guo, Qin Zhang, and Shuxia Han. 2002.  Agricultural Machinery Safety Alert System Using Ultrasonic Sensors.  _Journal of Agricultural Safety and Health_ 8, 4 (2002), 385–396.  https://doi.org/10.13031/2013.10219
  * Guo et al. (2014) Philip J. Guo, Juho Kim, and Rob Rubin. 2014.  How video production affects student engagement: an empirical study of MOOC videos. In _Proceedings of the 1st ACM Conference on Learning @ Scale_. ACM, New York, NY, 41–50.  https://doi.org/10.1145/2556325.2566239
  * Hart and Staveland (1988) Sandra G. Hart and Lowell E. Staveland. 1988.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research.  _Advances in Psychology_ 52 (1988), 139–183.  https://doi.org/10.1016/s0166-4115(08)62386-9
  * Hutt et al. (2017) Stephen Hutt, Caitlin Mills, Nigel Bosch, Kristina Krasich, James R. Brockmole, and Sidney K. D’Mello. 2017.  “Out of the Fr-Eye-ing Pan”: Towards Gaze-Based Models of Attention during Learning with Technology in the Classroom. In _Proceedings of the 25th ACM International Conference on User Modeling, Adaptation and Personalization_. ACM, New York, NY, 94–103.  https://doi.org/10.1145/3079628.3079669
  * International (2011) Toastmasters International. 2011\.  Your Speaking Voice: Tips for Adding Strength and Authority to Your Speaking Voice.  https://www.toastmasters.org/resources/your-speaking-voice.  Accessed: August 19, 2019.
  * Kayukawa et al. (2019) Seita Kayukawa, Keita Higuchi, João Guerreiro, Shigeo Morishima, Yoichi Sato, Kris Kitani, and Chieko Asakawa. 2019.  BBeep: A Sonic Collision Avoidance System for Blind Travellers and Nearby Pedestrians. In _Proceedings of the 37th ACM CHI Conference on Human Factors in Computing Systems_ , Vol. 52. ACM, New York, NY, 1–12.  https://doi.org/10.1145/3290605.3300282
  * Kerres (2020) Michael Kerres. 2020\.  Against All Odds: Education in Germany Coping with COVID-19.  _Postdigital Science and Education_ 2, 3 (2020), 690–694.  https://doi.org/10.1007/s42438-020-00130-7
  * Kodama (2020) Mitsuru Kodama. 2020\.  Digitally transforming work styles in an era of infectious disease.  _International Journal of Information Management_ 102172 (2020), 1–6.  https://doi.org/10.1016/j.ijinfomgt.2020.102172
  * Kuzminykh and Rintel (2020a) Anastasia Kuzminykh and Sean Rintel. 2020a.  Classification of Functional Attention in Video Meetings. In _Proceedings of the 38th ACM CHI Conference on Human Factors in Computing Systems_ , Vol. 419. ACM, New York, NY, 1–13.  https://doi.org/10.1145/3313831.3376546
  * Kuzminykh and Rintel (2020b) Anastasia Kuzminykh and Sean Rintel. 2020b.  Low Engagement As a Deliberate Practice of Remote Participants in Video Meetings. In _Extended Abstracts of the 38th ACM CHI Conference on Human Factors in Computing Systems_ , Vol. 321. ACM, New York, NY, 1–9.  https://doi.org/10.1145/3334480.3383080
  * Lee and Starner (2010) Seungyon Claire Lee and Thad Starner. 2010.  BuzzWear: alert perception in wearable tactile displays on the wrist. In _Proceedings of the 28th ACM CHI International Conference on Human Factors in Computing Systems_. ACM, New York, NY, 433–442.  https://doi.org/10.1145/1753326.1753392
  * Martin (2012) Fred G. Martin. 2012\.  Will massive open online courses change how we teach?  _Commununications of the ACM_ 55, 8 (2012), 26–28.  https://doi.org/10.1145/2240236.2240246
  * Nettelhorst and Brannon (2012) Stephen C. Nettelhorst and Laura A. Brannon. 2012.  The effect of advertisement choice, sex, and need for cognition on attention.  _Computers in Human Behavior_ 28, 4 (2012), 1315–1320.  https://doi.org/10.1016/j.chb.2012.02.015
  * Oeppen et al. (2020) Rachel S. Oeppen, Graham Shaw, and Peter A. Brennan. 2020\.  Human factors recognition at virtual meetings and video conferencing: how to get the best performance from yourself and others.  _British Journal of Oral and Maxillofacial Surgery_ 58, 6 (2020), 643–646.  https://doi.org/10.1016/j.bjoms.2020.04.046
  * Poyatos (1993) Fernando Poyatos. 1993\.  _Paralanguage: A Linguistic and Interdisciplinary Approach to Interactive Speech and Sounds_.  John Benjamins Publishing Company, Amsterdam, Netherlands.
  * Saint-Lot et al. (2020) Julie Saint-Lot, Jean-Paul Imbert, and Frédéric Dehais. 2020\.  Red Alert: A Cognitive Countermeasure to Mitigate Attentional Tunneling. In _Proceedings of the 38th ACM CHI Conference on Human Factors in Computing Systems_ , Vol. 580. ACM, New York, NY, 1–6.  https://doi.org/10.1145/3313831.3376709
  * Samrose et al. (2017) Samiha Samrose, Ru Zhao, Jeffery White, Vivian Li, Luis Nova, Yichen Lu, Mohammad Rafayet Ali, and Mohammed E. Hoque. 2017.  CoCo: Collaboration Coach for Understanding Team Dynamics during Video Conferencing.  _Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies_ 1, 4 (2017), 160:1–160:24.  https://doi.org/10.1145/3161186
  * Schiavo et al. (2014) Gianluca Schiavo, Alessandro Cappelletti, Eleonora Mencarini, Oliviero Stock, and Massimo Zancanaro. 2014.  Overt or subtlefi: supporting group conversations with automatically targeted directives. In _Proceedings of the 19th ACM International Conference on Intelligent User Interfaces_. ACM, New York, NY, 225–234.  https://doi.org/10.1145/2557500.2557507
  * Sharma et al. (2016) Kshitij Sharma, Hamed S. Alavi, Patrick Jermann, and Pierre Dillenbourg. 2016.  A gaze-based learning analytics model: in-video visual feedback to improve learner’s attention in MOOCs. In _Proceedings of the 6th ACM International Conference on Learning Analytics & Knowledge_. ACM, New York, NY, 417–421.  https://doi.org/10.1145/2883851.2883902
  * Stothart et al. (2015) Cary Stothart, Ainsley Mitchum, and Courtney Yehnert. 2015\.  The attentional cost of receiving a cell phone notification.  _Journal of Experimental Psychology: Human Perception and Performance_ 41, 4 (2015), 893–897.  https://doi.org/10.1037/xhp0000100
  * Sullivan and Horowitz (1983) Joseph W. Sullivan and Frances Degen Horowitz. 1983.  The effects of intonation on infant attention: the role of the rising intonation contour.  _Journal of Child Language_ 10, 3 (1983), 521–534.  https://doi.org/10.1017/s0305000900005341
  * Thomas and Jayagopi (2017) Chinchu Thomas and Dinesh Babu Jayagopi. 2017.  Predicting student engagement in classrooms using facial behavioral cues. In _Proceedings of the 1st ACM SIGCHI International Workshop on Multimodal Interaction for Education_. ACM, New York, NY, 33–40.  https://doi.org/10.1145/3139513.3139514
  * Toda et al. (2012) Tomoki Toda, Takashi Muramatsu, and Hideki Banno. 2012\.  Implementation of Computationally Efficient Real-Time Voice Conversion. In _Proceedings of the 13th Annual Conference of the International Speech Communication Association_. ISCA, Baixas, France, 94–97.
  * Trager (1958) George L. Trager. 1958\.  Paralanguage: A first approximation.  _Studies in Linguistics_ 13 (1958), 1–12.
  * Veliyath et al. (2019) Narayanan Veliyath, Pradipta De, Andrew A. Allen, Charles B. Hodges, and Aniruddha Mitra. 2019.  Modeling Students’ Attention in the Classroom using Eyetrackers. In _Proceedings of the 2019 ACM Southeast Regional Conference_. ACM, New York, NY, 2–9.  https://doi.org/10.1145/3299815.3314424
  * Venkatesh et al. (2003) Viswanath Venkatesh, Michael G. Morris, Gordon B. Davis, and Fred D. Davis. 2003.  User Acceptance of Information Technology: Toward a Unified View.  _MIS Quarterly_ 27, 3 (2003), 425–478.  https://doi.org/10.2307/30036540
  * Wansink and van Ittersum (2007) Brian Wansink and Koert van Ittersum. 2007.  Portion Size Me: Downsizing Our Consumption Norms.  _Journal of the American Dietetic Association_ 107, 7 (2007), 1103–1106.  https://doi.org/10.1016/j.jada.2007.05.019
  * Weiser and Brown (1995) Mark Weiser and John Seely Brown. 1995.  _Designing calm technology_.  Technical Report. Xerox PARC, Palo Alto, CA. 1–5 pages.
  * Weiser and Brown (1997) Mark Weiser and John Seely Brown. 1997.  The coming age of calm technology.  In _Beyond calculation_. Springer, New York, NY, 75–85.  https://doi.org/10.1007/978-1-4612-0685-9_6
  * Whittaker (1995) Steve Whittaker. 1995\.  Rethinking video as a technology for interpersonal communications: theory and design implications.  _International Journal of Human-Computer Studies_ 42, 5 (1995), 501–529.  https://doi.org/10.1006/ijhc.1995.1022
  * Xiao and Wang (2016) Xiang Xiao and Jingtao Wang. 2016.  Context and cognitive state triggered interventions for mobile MOOC learning. In _Proceedings of the 18th ACM International Conference on Multimodal Interaction6_. ACM, New York, NY, 378–385.  https://doi.org/10.1145/2993148.2993177
  * Xu (2005) Yi Xu. 2005.  Speech melody as articulatorily implemented communicative functions.  _Speech Communication_ 46, 3-4 (2005), 220–251.  https://doi.org/10.1016/j.specom.2005.02.014
  * Yu et al. (2017) Kun Yu, Shlomo Berkovsky, Ronnie Taib, Dan Conway, Jianlong Zhou, and Fang Chen. 2017\.  User Trust Dynamics: An Investigation Driven by Differences in System Performance. In _Proceedings of the 22nd International Conference on Intelligent User Interfaces_. ACM, New York, NY, 307–317.  https://doi.org/10.1145/3025171.3025219
  * Zaletelj and Kosir (2017) Janez Zaletelj and Andrej Kosir. 2017.  Predicting students’ attention in the classroom from Kinect facial and body features.  _EURASIP Journal on Image and Video Processing_ 2017 (2017), 80\.  https://doi.org/10.1186/s13640-017-0228-8
  * Zatorre and Gandour (2007) Robert J. Zatorre and Jackson T. Gandour. 2007.  Neural specializations for speech and pitch: moving beyond the dichotomies.  _Philosophical Transactions of the Royal Society B: Biological Sciences_ 363, 1493 (2007), 1087–1104.  https://doi.org/10.1098/rstb.2007.2161
  * Zhou et al. (2017) Jianlong Zhou, Syed Z. Arshad, Simon Luo, and Fang Chen. 2017\.  Effects of Uncertainty and Cognitive Load on User Trust in Predictive Decision Making. In _Proceeding of the 16th IFIP TC 13 International Conference on Human-Computer Interaction_. Springer, Cham, Switzerland, 23–39.  https://doi.org/10.1007/978-3-319-68059-0_2