# Situated Understanding of Older Adults’ Interactions with Voice Assistants:
A Month-long In-home Study

Amama Mahmood<EMAIL_ADDRESS>Johns Hopkins University3400 N. Charles
StBaltimoreMarylandUSA21218 ,  Junxiang Wang<EMAIL_ADDRESS>Johns Hopkins
University3400 N. Charles StBaltimoreMarylandUSA21218 and  Chien-Ming Huang
<EMAIL_ADDRESS>Johns Hopkins University3400 N. Charles
StBaltimoreMarylandUSA21218

(20 February 2007; 12 March 2009; 5 June 2009)

###### Abstract.

Our work addresses the challenges older adults face with commercial Voice
Assistants (VAs), notably in conversation breakdowns and error handling.
Traditional methods of collecting user experiences—usage logs and post-hoc
interviews—do not fully capture the intricacies of older adults’ interactions
with VAs, particularly regarding their reactions to errors. To bridge this
gap, we equipped 15 older adults’ homes with Amazon smart speakers integrated
with custom audio recorders to collect “in-the-wild” audio interaction data
for detailed error analysis. Recognizing the conversational limitations of
current VAs, our study also explored the capabilities of Large Language Models
(LLMs) to handle natural and imperfect text for improving VAs. Midway through
our study, we deployed ChatGPT-powered Alexa skill to investigate its efficacy
for older adults. Our research suggests leveraging vocal and verbal responses
combined with LLMs’ contextual capabilities for enhanced error prevention and
management in VAs, while proposing design considerations to align VA
capabilities with older adults’ expectations.

voice assistant, older adults, human-agent interaction, large language models,
personal assistant, conversational AI, errors, conversational breakdowns

††ccs: Human-centered computing Empirical studies in HCI††ccs: Computing
Methodologies Artificial intelligence Figure 1. We investigate the
interactions of older adults with commercial voice assistants (VAs) in their
homes over a period of four weeks. To augment our collected interaction data,
we record their audio using the recording device depicted. Each participant’s
interaction with the VA is recorded in its entirety, along with an additional
10 seconds after the interaction concludes, to capture their spontaneous
reactions and responses to the VA.

##  1\. Introduction

Commercial Voice Assistants (VAs), despite their growing popularity, are not
designed for special populations such as older adults (Stigall et al., 2019).
Nevertheless, they hold the potential to significantly enhance the quality of
life and autonomy for older adults (Smith et al., 2023; Blair and Abdullah,
2019). Additionally, it is important to recognize that data-driven intelligent
voice assistants are not immune to errors (Pearl, 2016). The adoption and
sustained use of VAs by older adults are impeded by AI failures and errors
(Kim and Choudhury, 2021; Pradhan et al., 2020), which are particularly more
prevalent for older adults partly because of their verbose and disfluent
speech patterns (Mortensen et al., 2006). Consequently, it is crucial to
investigate and analyze the conversational breakdowns experienced by older
adults when interacting with these systems.

While researchers have investigated older adults’ long-term interactions with
VAs at home, focusing on their adoption, benefits, challenges, and perceptions
(Kim and Choudhury, 2021; Pradhan et al., 2019, 2020; Purao et al., 2021), a
significant research gap exists in comprehensive error analysis. This happens
largely due to insufficient error logging in usage data (“text not available”
or “unknown”) (Pradhan et al., 2020) often leaving conversational context and
real-time user reactions unrecorded. Furthermore, speech recognition errors
are often not identifiable solely from transcribed usage logs (Kim and
Choudhury, 2021). Supplementary audio recordings are needed to capture nuanced
aspects of how smart speakers are embedded into conversational settings
(Porcheron et al., 2018). Given that commercial VAs do not provide a mechanism
to record complete audio interactions, they fall short in capturing users’
immediate thoughts and reactions during their interactions. Audio data can be
particularly crucial in instances where errors disrupt the conversational
flow. Documenting how users respond or adapt to continue the conversation can
offers insights into not only the nature, frequency, and occurrence of errors
but also users’ perceptions of VAs and the strategies they employ to advance
the conversation. To overcome the lack of complete interaction data, we
designed and developed a custom audio recorder to capture and log user
interactions with a commercial VA (Alexa) as audio files. We recorded each
interaction from start to finish and an additional 10 seconds post-interaction
to document users’ verbal and vocal responses during and immediately after
their interaction with Alexa. Our recording device enabled us to shed light on
older adults interaction dynamics and providing a comprehensive analysis of
conversational breakdowns.

So, RQ1: How can additional audio interaction data captured “in the wild”
offer deeper, context-rich insights into the behaviors and interaction
dynamics of older adults with VAs, particularly in identifying errors and
advancing conversations? To investigate RQ1, we deployed Amazon echo dot smart
speakers along with our recording device at 15 homes of older adults for
4-weeks. Our study highlighted key issues in older adults’ use of voice
assistants, including frequent intent recognition errors primarily due to the
VA’s limited understanding of user intentions and failing to accommodate
specific needs(_e.g.,_ forgetfulness and a preference for natural speech
styles). Furthermore, we found that errors often escalated when older adults
attempted to correct them, as these attempts typically failed to address the
initial misunderstandings, resulting in numerous unresolved errors.
Additionally, we highlight the potential of leveraging users’ immediate
reactions or responses as implicit cues for VAs to identify errors, thereby
improving error management.

While the conversational breakdowns need to be explored in further detail for
older adults, evidence points to current VAs’ lack in ability to understand
user inputs beyond scripted or structured formats, which is particularly
inadequate for older adults who favor more natural, conversational forms of
communication (Pradhan et al., 2020, 2018). Moreover, these VAs struggle with
understanding context often restricting interactions to a single exchange
hindering natural progression of conversation (Purao and Meng, 2019; Purao et
al., 2021). These issues are further exacerbated in older demographics due to
the prevalence of speech recognition errors stemming from factors such as
slower speech, stuttering, pauses, repeats, and flawed syntactic structures
(Kim and Choudhury, 2021). In this context, large language models (LLMs) have
demonstrated potential in interpreting natural, flawed, and conversational
speech (Mahmood et al., 2023), which could significantly improve older adults’
experiences with VAs.

RQ2: What potential benefits and challenges does an LLM-powered VA present in
enhancing the interaction quality for older adults, addressing the limitations
of current commercial voice assistants? To investigate RQ2, we integrated
ChatGPT into Alexa skills and deployed as a technology probe (Hutchinson et
al., 2003) in the latter half of our four-week study. The integration of LLMs
into VAs in publicly available platforms such as Alexa (Rausch, 2023)
underscores the relevance of this exploration in understanding how advanced
VAs can be tailored for special populations such as older adults. Our findings
emphasize the resilience of LLMs in conversation and the learning curve
necessary for older adults to engage effectively with such sophisticated VAs,
providing crucial insights for the design of future LLM-powered VAs.

Our work’s contribution is three-folds:

  1. (1)

Data collection tool (methodology): We introduce a novel method for collecting
“in-the-wild” audio data, enabling a deeper understanding of how users
interact with and perceive VAs within their personal environments (see Fig. 1
and 2).

  2. (2)

Analysis of real-world interaction data (empirical): We qualitatively analyzed
“in-the-wild” audio interaction data to understand how older adults
incorporate VAs into their lives, especially focusing on the nuances of
erroneous interactions. Our findings reveal both opportunities and challenges,
offering valuable design guidelines for creating more robust and fluid VA
interactions.

  3. (3)

LLM-powered VA as a technology probe (empirical): We develop, deploy, and
evaluate an LLM-powered VA as a technology probe to understand how older
adults engage with conversationally adept futuristic VAs to identify
opportunities and limitations of LLM integration into VA technology for older
adults.

##  2\. Related Work

The objective of our study is to explore the integration of voice
assistants111Throughout this manuscript and in our review of prior work, we
have specifically defined the term “voice assistant” in the context of their
embodiment as smart-speaker based. Conversely, when referring to agents that
extend beyond smart speakers to include web-based platforms, smartphones, and
watches, we use the term “voice-based.” Furthermore, the term “voice
interface” is employed to describe the modality of the system, focusing on how
users interact with it, rather than referring to the agent or assistant
itself. into the daily activities of older adults, examine the challenges
leading to erroneous interactions, and investigate how the integration of LLMs
may enhance their experience in seeking information. Below, we review relevant
prior work.

###  2.1. Voice Assistants in Older Adults’ Life

Voice assistants have become a staple in many households (Lis, 2022); their
adoption among older adults has been notably high (Lifestyle, 2022; Kakulla,
2021). Voice assistants not only show significant potential to enhance the
quality of life for older adults (Smith et al., 2023; Blair and Abdullah,
2019) but also have been shown to uplift spirits, provide mental engagement
opportunities, and aid in fostering social connections (Upadhyay et al., 2023;
Oewel et al., 2023). Common uses of VAs are seeking weather updates, getting
information, listening to music, and setting reminders (Arnold et al., 2022;
O’Brien et al., 2020). Voice-based agents have also been explored to assist
older adults for specific tasks—for example, interactions with complex user
interfaces (UIs) (_e.g.,_ locate interface features while navigating UIs (Yu
et al., 2023) and VAs controlling home appliances (Choi et al., 2020; Kowalski
et al., 2019))—highlighting their preference for voice interfaces (Kowalski et
al., 2019; Wulf et al., 2014; Liu et al., 2020). Moreover, verbal game play
with VAs is explored as a potential way to increase usage and engagement which
is received positively by older adults (Nacimiento-Garcia et al., 2021).

While initial interactions are often entertainment-oriented, there is a
notable shift towards practical applications as users become more familiar
with the technology, particularly for health-related queries (Pradhan et al.,
2020; Brewer et al., 2022). In terms of such task-specific applications, VAs
have shown significant promise as health aids. Prior work focusing on specific
needs, such as pain management, have found VAs to be particularly beneficial
for older adults (Shade et al., 2020). The ability of VAs to set reminders and
promote accountability in health routines has been appreciated (Shade et al.,
2020; Bolaños et al., 2020). Research has also explored healthcare information
seeking (Brewer et al., 2022; Harrington et al., 2022; Sanders and Martin-
Hammond, 2019), preventive care, (Yamada et al., 2018) and use of reminders
for medicines (Chen et al., 2021; Bolaños et al., 2020; Jesús-Azabal et al.,
2020).

Apart from practical usage, the social perceptions of VAs by older adults are
influenced by various factors such as their familiarity with technology,
experiences with VA, and preferences. For instance, the user’s emotional state
and even the smart speaker’s physical location can affect whether they view
the assistant as a machine or a more human-like companion (Pradhan et al.,
2019). As users grow more accustomed to VA technology, they begin to value not
just their operational ease but also the sense of digital companionship they
offer (Kim and Choudhury, 2021).

###  2.2. Conversational Challenges Faced by Older Adults While Interacting
with Voice Assistants

Although older adults recognize the benefits of using VAs, some potential
barriers to the adoption of VA technology for the aging population remain. For
instance, older adults’ inability to find essential utility and their habitual
reliance on other ways of completing tasks are mentioned as main reasons for
disuse of smart speakers such as the Amazon Echo (Trajkova and Martin-Hammond,
2020). Older adults also point out data protection concerns and fear of being
patronized and manipulated by VAs (Horstmann et al., 2023). Additionally,
older adults face conversational breakdowns (Kim and Choudhury, 2021) that may
impact their adoption and continued use of VA technology due to additional
frustration. While VAs have the potential to support daily activities, well-
being, and healthcare needs of older adults, addressing these challenges is
essential for their effective integration into the daily lives of older
adults.

Older adults often require more time to complete tasks and experience higher
error rates compared to younger adults when interacting with phone-based
virtual assistants using voice (Chiaradia et al., 2019; Barros and Seabra,
2020), underscoring the increased conversational challenges they face due to
factors such as loss of cognitive and motor skills and limited prior
technology use (Barros and Seabra, 2020). Although older adults encounter more
challenges with voice-based agents compared to younger adults, they still
generally hold positive impressions of the interactions (Chiaradia et al.,
2019; Barros and Seabra, 2020).

While research shows that older adults “perceive” that they have a good mental
model of the VA’s capabilities (such as asking question a certain way)
(Pradhan et al., 2020), other research points out that they are not fully sure
of its capabilities (Kim and Choudhury, 2021). Such mismatch can cause
conversational breakdowns negatively affecting user experience and causing
frustration. Additionally, older adults often find it challenging to remember
commands associated with VA features and skills (Pradhan et al., 2020; Stigall
et al., 2019)—as simple as remembering wakeword at times (Cuadra et al., 2022;
Angelini et al., 2021; Pradhan et al., 2020). They frequently resort to trial
and error methods following conversational breakdowns (Kim and Choudhury,
2021) and may eventually give up trying to reformulate their queries after
multiple failed attempts, leading to an inability to recover from errors and
thus limiting their use of VAs (Pradhan et al., 2020).

Although older adults are somewhat cognizant of the limitations of VAs, such
as their lack of follow-up and contextual understanding, they prefer and
initially attempt a conversational style (Pradhan et al., 2019). Nevertheless,
the frustration arising from VAs’ inability to remember context (Pradhan et
al., 2019) often forces them to modify their approach. This leads to a
breakdown especially in information retrieval (_e.g.,_ health-related),
causing older adults to reluctantly shift from their preferred conversational
style to more scripted and self-contained queries (Brewer et al., 2022).
Consequently, many interactions with VAs end up being command-based—indicating
a transactional rather than conversational relationship (Purao et al., 2021;
Purao and Meng, 2019)—and usually lack proactive interactions (Yamada et al.,
2018). Such shift to transactional dynamics has led researchers to debate
whether these interactions should truly be considered “conversations”
(Porcheron et al., 2018).

The lack of holistic analysis of daily user-VA interactions, due to the
absence of real-world audio interaction data, makes it difficult to concretely
interpret the extent of older adults’ awareness of VA capabilities and their
efforts to recover from conversational breakdowns. These breakdowns often stem
from mismatches in mental models, limitations of VAs, and other factors such
as speech recognition errors that are exacerbated by speech difficulties
common in older adults (Kim and Choudhury, 2021). Therefore, further in-depth
analysis of VA failures with older adults is necessary to better understand
the complexities of their daily interactions

###  2.3. Research Tools and Methodologies for Longitudinal Field Studies on
VAs with Older Adults

Several rigorous methodologies have been adopted to gather data about the use
and perceptions of voice assistants among older adults for longitudinal at-
home studies offering insights into changing perceptions and interaction
dynamics over time. One primary approach is the combination of semi-structured
interviews (Pradhan et al., 2019, 2020; Kim and Choudhury, 2021) and daily
diary entries (Pradhan et al., 2020), enabling researchers to gather
subjective user perceptions of their experiences with VAs. Second approach is
to complement the understanding of user perceptions with a detailed analysis
of usage logs to reveal frequency and evolving patterns of the VA utilization
(Purao et al., 2021; Pradhan et al., 2020; Kim and Choudhury, 2021; Oewel et
al., 2023). However, this method is constrained by transcription inaccuracies,
which complicates comprehending errors based solely on usage logs (Kim and
Choudhury, 2021) suggesting that the usage logs are not always objective
either—not always truly reflective of user interactions with VAs. Moreover,
the usage logs are unable to capture organic vocal reactions, interruptions
and overlapping speech during the interaction. Longitudinal studies on VA
technologies for older adults lack real audio interaction data, that has shown
to be beneficial for understanding user behavior (Porcheron et al., 2018).
Interaction data can particularly be useful for understanding the nature of
conversational breakdowns. Prior research has also pointed out that further
research is needed to incorporate such objective audio data to deepen our
understanding of these interactions (Purao et al., 2021). Therefore, in this
study, we developed a tool to augment interaction data with real-world audio
recordings, capturing older adults verbal and vocal reactions and responses to
Alexa including interruptions and overlapping speech, not logged in
traditional usage data.

###  2.4. LLM-Powered Voice Assistants: A Technology Probe

Despite older adults wanting to have conversational interactions with VAs,
current commercial VA capabilities are limited to achieve that (Pradhan et
al., 2019), as discussed in section 2.2. Large language models, on the other
hand, have shown enhanced conversational capabilities yielding robust and
fluid interactions with users (Mahmood et al., 2023; Chan et al., 2023).
Similarly, recent efforts to incorporate LLMs into voice-based interactions
for older adults has shown improvement in conversational aspects. For
instance, CareCall, a chatbot powered by LLM, facilitates large-scale public
health monitoring through open-ended dialogues, surpassing the capabilities of
traditional rule-based systems (Jo et al., 2023). Similarly, Talk2Care, an
LLM-powered VA designed for older adults, enhances health information
gathering and mental support in situations such as symptom reporting and post-
operative care (Yang et al., 2023), opening up venues of future research.

In our study, we develop and deploy an LLM-powered VA for older adults.
Designed as a general information retrieval tool and deployed as a technology
probe, this exploration aims to uncover the potential benefits of LLM-powered
VAs. By adopting the technology probe framework (Hutchinson et al., 2003), our
focus is on assessing the usability and effectiveness of an LLM-powered VA in
real-world settings. Moreover, this approach is intended to gain insights into
the needs and desires of older adults regarding advanced VAs and to
conceptualize new opportunities for tailoring VAs to meet their needs and
expectations.

##  3\. Methods

To understand older adults’ perceptions and interactions with voice
assistants, we conducted a 4-week field deployment of a smart speaker (Amazon
Echo Dot $5^{th}$ generation). Our method collects data through usage logs,
audio interactions from our recording device, and semi-structured interviews
at start and end of deployment. This section outlines the implementation of
our recording device and ChatGPT integration into Alexa, followed by a
description of the longitudinal deployment, participant details, and our data
analysis approach.

###  3.1. Data Collection: Recording Device

In our study, we utilized a specially developed recording device to capture
user’s interaction with Alexa including their immediate responses and
reactions for an additional 10 seconds following each interaction. The
recording device comprises of three modules:

  1. (1)

Detector: The recording device detects the activation of Alexa, indicated by
the blue light ring on the bottom of Amazon Echo Dot utilizing the light color
detector on Arduino UNO BLE SENSE 33.

  2. (2)

Recorder: Triggered by the detection of the Echo Dot’s blue light, the Arduino
begins recording audio onto a MicroSD card inserted into an Adafruit microSD
card breakout board connected to the Arduino. Recording stops 10 seconds after
the blue light goes off. However, if the user reactivates Alexa within this
10-second window, the device continues recording until the end of this new
interaction, followed by an additional 10-second period.

  3. (3)

Privacy indicator: To address privacy concerns and enhance user comfort, our
device does not continuously buffer audio for keyword detection, as described
in prior research (Porcheron et al., 2018). Instead, we only buffer audio when
the blue light is detected at the start of interaction. To ensure
transparency, a red LED light, positioned on top of the recording device,
illuminates to indicate active audio recording allowing user to monitor the
recording status.

The recording device is adaptable for use with other smart speakers as
integration with wake word detection is also feasible. We designed and
3D-printed a compact enclosure for the recording device, which can be mounted
on a base plate with a designated spot for the Echo Dot, as shown in Fig. 1
and 2. The code for the recording device, 3D printing sketch, and assembly
instructions will be publicly available222Link to code and design for
recording device and study materials: https://bit.ly/3UIgAri.. To validate the
functionality of this recording device, we conducted a pilot test with one
young adult and one older adult, deploying the smart speaker and our recording
device in their homes for one week. This testing phase confirmed that the
device operated as intended, fulfilling its designed purpose effectively.
Interaction data from these pilot participants were not included in our
analysis.

Figure 2. Design of enclosing and base plate for placing Echo Dot smart
speaker and recording device. The CAD files are available online at
https://bit.ly/3UIgAri for 3D printing.

###  3.2. Technology Probe: Integrating ChatGPT into Alexa Using Alexa Skills

To investigate older adults’ interactions with more conversational voice
assistants and examine the benefits of integrating LLMs into VAs, we
incorporated ChatGPT 3.5 into an Alexa skill. To enhance conversational
fluidity and address the issue of Alexa skills timing out during longer
ChatGPT API responses, we employed a conversational framework using fillers
(“searching”) and small talk (VA asking “How was your day?”) through a
middleman API mechanism, similar to the one used in prior work (Mahmood et
al., 2023). To enable intuitive activation by older adults, we used natural
activation phrases for the ChatGPT skill, such as “Alexa, let’s chat” or
“Alexa, question.” Upon user activation, the ChatGPT-powered VA introduces
itself. Users then can have conversations without repeatedly using the wake
word “Alexa.” We refer to this skill as “ChatGPT+Alexa skill” or simply
“ChatGPT skill”.

###  3.3. Longitudinal Study: Procedure

The study comprised of 4-week field deployment that included an initial
interview, device set up, demo session, check-in at week 1, deployment of LLM-
Alexa skill at week 2, check-in at week 3 to set up a time for pick up, and a
final session at week 4 that involved a semi-structured exit interview. We
also had a detailed demographics questionnaire at the beginning. The first
session lasted about 90 minutes.

####  3.3.1. Week 0: Initial interview and smart speaker setup.

First session consisted of following in the given order:

  1. (1)

Introduction and consent. Experimenter explained the study and obtained
informed consent. Experimenter informed the participants that they could
either keep the smart speaker or be compensated equivalent to the market price
of Amazon Echo Dot _i.e.,_ $50\$$.

  2. (2)

Demographics and quality of life survey. Detailed demographics questions
including socioeconomic status were filled out by the participant (see Table
3.4). Moreover, participants filled out a short quality of life survey
(Bowling et al., 2013).

  3. (3)

Pre-study interview.  Experimenter conducted a brief semi-structured interview
to gauge participants’ awareness and perception of smart technologies in
general and voice assistants such as Alexa and Siri, specifically.

  4. (4)

Set up the smart speaker. Experimenter then set up the smart speaker at
participants’ desired location (see Table 2). Experimenter told participants
that they are allowed to move the smart speaker if they wanted and showed how
it can be done. Experimenter also gave participants an instruction page about
how they can debug internet issues if needed. Participants could contact the
experimenter at any time through email, text, or phone call. Amazon Alexa
application is installed in their device of choice (smart phone or tablet).
Experimenter set up participants’ name pronunciations, voice profile, updates
the device location, and turn on the in-built adaptive listening feature and
follow-up mode in settings. Adaptive listening provides the user with an
extended duration to complete their speech before Alexa initiates its
response. Follow-up mode allows them to ask Alexa follow-up questions without
repeating the wake word. Free Amazon music subscription was made available for
participants.

  5. (5)

Demonstration of Alexa and recording device. The experimenter demonstrated the
participants through an introductory tutorial, which covered fundamental VA
capabilities. The tutorial included tasks like setting alarms, reminders, and
timers, creating shopping and to-do lists, playing music, asking for jokes,
posing questions, and engaging in casual, unstructured question answers with
the VA. Furthermore, the experimenter showcased the process of accessing and
utilizing Alexa skills. A printed copy of said Alexa introductory tutorial was
left with the participants22footnotemark: 2. Alongside showcasing Alexa’s
capabilities, the experimenter explained that the audio recorder is only
recording while the red LED light is on. During the demo, the experimenter
also removed the micro SD card from the recording device and played a few
recordings for the participants to ensure their comprehension of what gets
recorded during their interactions with the speaker.

####  3.3.2. Week 1: Check-in.

At the end of Week 1, the experimenter reached out to the participants to
inquire briefly about their experiences thus far mainly to see if the
participants have any technical difficulties or questions. The experimenter
also verified whether they continued to observe the activation of the red LED
light on the recording device while interacting with Alexa.

####  3.3.3. Week 2: ChatGPT deployment.

We implemented the Alexa skill that facilitates participants in engaging with
ChatGPT 3.5. The skill was remotely deployed, and comprehensive instructions
along with illustrative examples and suggested conversation prompts were
provided to the participants22footnotemark: 2. These instructions were
communicated either through email or in printed form via our liaison at the
community center. The experimenter then contacted all participants via phone
to introduce the skill as a means to hold natural conversations with a voice
assistant using Alexa that is capable of responding to follow-up questions and
statements. Subsequently, the experimenter conducted a demonstration of the
skill over the phone.

####  3.3.4. Week 3: Finalizing the smart speaker collection.

Concluding Week 3, the experimenter reached out to the participants either
through phone or text message to ascertain their decision regarding keeping
the smart speaker. Additionally, arrangements were made to collect the
recording device at a mutually convenient time.

####  3.3.5. Week 4: Collection of smart speaker

The study concluded with the last session for collecting the smart speaker and
exit semi-structured interview. The exit semi-structured interview aimed to
gather detailed insights from participants regarding their 4-week experience
with the smart speaker. Participants were encouraged to provide comprehensive
information about their overall experiences, the usefulness of the VA in daily
activities, specific tasks that were facilitated by the VA, enjoyable and
useful features, and any challenges they encountered. The interview delved
into aspects such as voice interactions and ease of use, challenges faced,
concerns about privacy and security, social interaction and communication, and
the learning curve. Participants were also asked about their willingness to
continue using the VA beyond the study and their recommendations for
improvements. The interview provided a comprehensive view of participants’
past experiences, preferences, challenges, and suggestions related to their
interactions with the VA.

###  3.4. Participants

We deployed 15 smart speakers in three waves in the greater Baltimore Area,
Maryland, USA. Data for four weeks was analyzed for each device. Recruitment
was done from a community center that has both an independent and assisted
living community. We also recruited community dwelling adults from our
previous connections. Tables 3.4 and 2 summarize their demographics and
awareness of voice assistant technology respectively. The participants were
compensated for the study. They could either keep the smart speaker or be
compensated equivalent to the market price of Amazon Echo Dot _i.e.,_ $50\$$.
All the participants except three—P11 couple (wanted a different colored
speaker), P5, and P10 couple—kept the smart speaker. The study was approved by
our Institutional Review Board (IRB).

Table 1. Demographics for field study: All participants were fluent in
English. All were retired except P10a (employed full time) and P11a (employed
part time). Note: eq means equivalent diploma

P | Gender | Age | Ethnicity | Highest degree | Profession | Disabilities | QOL overall
---|---|---|---|---|---|---|---
Community Center - Assisted Living (By Self)
2 | M | 66 | Prefer not to say | Bachelor’s degree | Physician assistant | Wheelchair | Alright
3 | F | 82 | Caucasian | High school or eq | Accountant clerk | Hearing aid | Alright
4 | M | 79 | Caucasian | Master’s degree | Actuary | Wheelchair | Good
Community Center - Independent Living (By Self)
5 | F | 77 | Caucasian | Bachelor’s degree | Social work |  | Good
6 | F | 81 | Caucasian | Master’s degree | CIA | Cane/Rollator | Alright
7 | F | 74 | African American | Bachelor’s degree | Administrator |  |
14 | F | 73 | African American | Bachelor’s degree | Senior Claims |  | Good
15 | F | 73 | African American | Master’s degree | Educator/Pastor |  | Good
Homeowner (By Self)
8 | F | 84 | Caucasian | Bachelor’s degree | Writer/editor | Walker | Alright
9 | M | 75 | Caucasian | Bachelor’s degree | Case management |  | Very good
13 | M | 81 | Caucasian |  |  | Vision issues |
Homeowner (Couple, With Spouse)
1a | M | 76 | Caucasian | Bachelor’s degree | Business owner |  | Good
1b | F | 75 | Caucasian | Nursing diploma | Nurse |  |
1-8 10a | M | 68 | Caucasian | Bachelor’s degree | Staff engineer |  | Good
10b | F | 94 | Caucasian | Master’s degree | Education | Walker/scooter | Bad
1-8 11a | M | 72 | Caucasian | Master’s degree | Urban planning |  | Very good
11b | F | 71 | Asian | Master’s degree | Social work/teacher |  | Very good
1-8 12a | M | 75 | Caucasian | Bachelor’s degree |  |  | Very good
12b | F | 72 | Caucasian | Nursing diploma | Registered nurse |  | Good
1-8 |  |  |  |  |  |  |

Table 2. Demographics for field study: Voice assistant technology use and familiarity P | Placement |  Technology use
---|---|---
Community Center - Assisted Living (By Self)
2 | Studio |  Alexa in the dining room of community center. Have Alexa app on iPad already.
3 | Studio |  Does not have smart phone. Never used VA before.
4 | Studio |  Had Alexa long time ago not anymore.
Community Center - Independent Living (By Self)
5 | Living room |  Use speech to text for messaging. Owns tablet.
6 | Living room |  Use TV remote using voice.
7 | Living room |  Not used before.
14 | Living room |  Used Siri on phone to ask questions, directions and locations. Sometimes rating of a restaurant. Owns apple watch and iPad as well.
15 | Living room |  Not used before.
Homeowner (By Self)
8 | Bedroom |  No smart phone but own iPad. Not familiar with VAs.
9 | Dining room |  Owns a speaker compatible with Alexa, mostly for music. Used for asking weather but usually uses phone for that.
13 | Dining room |  Owns tablet as well. Not familiar with VAs
Homeowner (Couple, With Spouse)
1 | Living room |  Have Alexa smart speakers in living room, kitchen, bedroom, study, at daughter’s home next door. Use for communicating between rooms, and timers in kitchen while cooking, etc.
10 | Living room |  Not familiar with VAs.
11 | Kitchen |  Use Siri on phone for calling (especially when driving), sending messages and asking questions. Also use with google maps.
12 | Living room |  Have Alexa smart speakers in kitchen (will move to laundry) and bedroom. Use for communicating (drop ins) between rooms, music and radio.

###  3.5. Data and Analysis

The audio recordings from the device were transcribed and matched to usage
logs from the Alexa dashboard. We also note non-verbal and vocal cues such as
laughter, “huh” and “hmm” in our transcripts from the recording device. We
take note of any interruptions, overlapping conversations, comments or remarks
to interactions—specifically errors, and other private conversations involving
Alexa. For usage logs and audio recordings from the device, after excluding
commands spoken during experimenter’s demonstration, 2552 user query-VA
response pairs (also referred to as “turns”) were found across all
participants. Initially, the first author examined the interaction data, using
an inductive approach to identify preliminary codes to gain a broader
understanding of usage, conversational breakdowns, and social aspects of
interactions. Following this, two researchers independently coded the data for
two participants (approximately 10%) using these initial codes. They then
convened to discuss and resolve any conflicts, resulting in the formulation of
a final codebook. All transcripts were subsequently analyzed, with the
established codes from this codebook being deductively applied to each user
query-VA response pair. Each query-response pair was coded for its purpose
(type of interaction as informed from prior work (Pradhan et al., 2020; Kim
and Choudhury, 2021)), presence of errors, type of error (as defined in Table
3), whether the error was resolved, and the user’s recovery method (_e.g.,_
moving on, repetition, clarification, rephrasing). We also note whether an
error led to a conversational breakdown and if it was evident that the
participant recognized the error through their reactions or by initiating
recovery.

To identify whether people engage in multi-turn conversations with voice
assistants, we grouped single user query-VA response pairs into multi-turn
interactions, provided the topic remained the same or the participant
continued the interaction. This multi-turn aspect was identified from the
recorded audio interactions. The initial codebook for this analysis was
developed by the first author, with subsequent discussions and resolutions of
disagreements by the first two authors. The final codebook is provided in
Appendix A. All the interviews were transcribed and participants quotes were
used to support, explain, and contextualize our findings with participants’
accounts of their experiences.

Table 3. Definitions of errors and their occurance in interaction data Error |  Definition | Count
---|---|---
Not listened |  User query was not listened by Alexa | 69
Wrong wakeword |  User used the wrong wakeword | 49
Trigger |  Alexa was triggered when user did not intend to | 15
Listened partial |  Alexa captured user query partially | 29
Interruption |  Alexa interrupted the user query | 1
Partial query |  User did not complete their query | 23
Transcription |  User query is transcribed inaccurately | 40
System |  User query is captured accurately but Alexa fails due to unknown reasons and responds such as “I’m having trouble getting that right now. Please try again.” | 15
Not handled |  User query is captured accurately but Alexa did not respond | 57
Limitation |  User query is captured accurately but Alexa is not capable of fulfilling the action and responds such as “Sorry, I don’t know about that” | 130
Intent |  User query is captured accurately but Alexa fails to recognize user intent and responds inaccurately | 204

##  4\. Findings

The dataset has total 2552 one-turn interactions. However, upon analyzing
interactions that were contiguous and centered around the same topic, such as
sequences of back-and-forth questions, we identified a substantial number of
multi-turn interactions. Such continuity in conversations was effectively
captured by our recording device, which also helped in discerning whether
participants had continued interaction. Although the majority of interactions
out of 2552 total turns were single-turn ($n=1173$), a substantial number
involved more than one turn, with 348 instances of 2-turn interactions. Out of
the total 2552 turns, we identified remaining 1379 instances as part of multi-
turn interactions. Fig. 3 illustrates the number of turns in these
interactions and their frequency. Our interaction data highlights the varying
nature of participant engagement with the voice assistant, ranging from brief
single commands to more extended dialogues. In this section, first we present
our findings on participants usage patterns followed by detailed analysis of
their erroneous interactions with the VA, and finally, our observations on the
social nature of interactions.

Figure 3. Total 2552 one turn interactions, 1379 ($348\times 2+57\times
3+\dots$) turns were part of multi turn interactions

###  4.1. Trends in Voice Assistant Usage Over Time

Most of our findings regarding usage patterns of the participants confirm
prior work (Pradhan et al., 2020; Kim and Choudhury, 2021). Below, we provide
a comprehensive overview of user engagement with the VA, highlighting trends
in usage that emerged over time.

Table 4. Interaction Categories and Types with Success Rates

Category | Type | Count | Success | Category | Type | Count | Success
---|---|---|---|---|---|---|---
Functional | reminder | 251 | 48.60% | Entertainment | music | 324 | 79.01%
weather | 205 | 83.41% | radio | 232 | 73.70%
timer | 174 | 90.23% | Alexa skills | 89 | 58.43%
alarm | 98 | 80.61% | joke | 22 | 86.36%
command | 94 | 94.68% | tv | 10 | 20.00%
list | 90 | 77.78% | story | 7 | 71.43%
communication | 71 | 98.59% | poem | 3 | 33.33%
6-8 | Alexa capabilities | 69 | 52.17% | Total | 753 | 71.84%
| time | 46 | 100.00% | Information-seeking | question | 414 | 57.25%
| repeat | 16 | 62.50% | Social | greetings | 174 | 93.68%
| calendar | 12 | 58.33% |  | and gratitude |  |
| notes | 1 | 100.00% | ChatGPT | ChatGPT | 64 | 68.75%
2-4 | Total | 1130 | 75.93% | Other | misc. | 17 | 0.00%

Table 5. Information seeking: Type of questions asked. n represents count and ER represents error rate (percentage of type of question queries that ran into error). For instance, $59.62\%$ of total local information questions resulted in error ($n=31$ out of 52). Type of question | n | ER | Type of question | n | ER | Type of question | n | ER
---|---|---|---|---|---|---|---|---
Entertainment | 98 | 28.57% | Miscellaneous | 38 | 44.74% | STEM related | 16 | 31.25%
Local information | 52 | 59.62% | Food & drink | 37 | 21.62% | Specific places | 14 | 64.29%
Meaning & definitions | 45 | 42.68% | Famous personalities | 31 | 58.06% | History | 10 | 60.00%
Health | 43 | 55.81% | Alexa related | 21 | 52.38% | Travel | 9 | 11.11%

####  4.1.1. Features in use

In our study, we analyzed the types of interactions older adults had with
Alexa. We categorized them based on the nature and goal of the interaction, as
shown in Table 4.1. Interactions were primarily functional ($n=1130$), aiding
in memory-related tasks like reminders, timers, calendar updates, and list
creation, as well as inquiring about weather and Alexa’s capabilities. For
instance, P15 regularly set reminders for her nightly medication and
occasionally queried Alexa to confirm whether she had taken her medication.
Entertainment uses ($n=753$), particularly music and radio, were also
prevalent. Some participants also engaged Alexa for jokes, stories, and poems.
The most frequent interaction type was question-asking ($n=414$), primarily
for information seeking. These inquiries were further categorized (see Table
5) to understand the nature of information sought (Pradhan et al., 2020), with
entertainment-related questions (_e.g.,_ “is the oriole game televised today?”
and “when was the children’s book room on the broom published?”) ($n=98$)
being the most common. Participants also queried about Alexa itself; for
instance, “how old are you?” and “are you a spy?”. Table LABEL:tab:alexa-
dialogue-questions has example conversations of such interactions. Our
findings align with existing literature on older adults’ usage of smart
speakers (Arnold et al., 2022).

From exploratory behavior to selective use. A notable observation is the
decline in overall usage with time which is consistent with prior work
(Pradhan et al., 2020; Lopatovska et al., 2019). The initial high usage,
characterized by exploratory behavior where participants experimented with
various features, gradually settled into a pattern of utilizing select
features they found beneficial in daily life. Prior work has discussed factors
influencing such decline to include usability issues, forgetting the wakeword,
and feeling of discomfort talking to a ‘machine’ (Pradhan et al., 2020). Usage
variation across interaction categories. While looking at interaction purpose,
we found that functional interactions, despite a gradual decline, remained the
most consistent category over the four-week period. As these interactions are
need driven and support daily activities, they tend to integrate more
seamlessly into users’ routines. In contrast, entertainment-oriented
interactions, such as listening to music, radio, or asking for jokes, showed
more variability. These desire driven interactions may be influenced by
factors such as mood, available leisure time, and personal preferences,
leading to their inconsistent frequency. Spike in usage during experimenter
check-ins and ChatGPT deployment phase. Interestingly, we observed spikes in
collective usage during specific periods, such as the week 1 check-in with the
experimenter and the ChatGPT deployment phase. These instances suggest that
indirect reminders of the VA’s presence can temporarily boost usage, although
this effect diminishes quickly (as evidenced by the usage peaks in Fig. 4).
During the ChatGPT deployment phase, we noticed a spike usage, that can be
attributed to participants exploring this new feature. This phase also saw an
increase in information retrieval requests using Alexa, possibly for two
reasons: 1) failure in activating the ChatGPT skill (_i.e.,_ forgetting to use
the ChatGPT activation phrase _e.g.,_ “Let’s talk”), which resulted in system
defaulting to query Alexa and 2) a reminder of Alexa’s capabilities to answer
questions and provide information.

####  4.1.3. Usage over times of days

In examining the cumulative four-week data, we analyzed the hours of the day
during which users interacted with Alexa, as illustrated in Fig. 5. Consistent
with previous research (Kim and Choudhury, 2021), we identified three primary
usage peaks: around 8 a.m. (morning), 12 p.m. (noon), and 6 p.m. (evening-
night). Interactions related to the exploration of ChatGPT were predominantly
confined to the afternoon, indicating that users tended to experiment with
this feature during their spare time, possibly when other commitments were
less pressing.

The usage patterns changed across participants as depicted in prior work
(Pradhan et al., 2020). The detailed results are presented in Appendix B.1.

Figure 4. Interaction categories usage change over days averaged over all
participants. Figure 5. Interaction categories usage change over hours of day
averaged over 4 weeks.

###  4.2. Analysis of Erroneous Interactions

In this study, we conduct a detailed analysis of errors, user experience, and
behavior surrounding these errors, an area less explored in prior research. By
examining users’ immediate responses and efforts to rectify errors—captured
through our recording device—we explore the dynamics of user interactions
during conversational breakdowns. Error rate in VA interactions. Our study
reveals that a considerable 24.76% of Alexa turns (632 out of 2552) led to
errors, equating to almost one in every four queries failing. Of these, 98.10%
manifested as conversational breakdowns.

Below, we outline our findings on several key aspects: how participants’
behavior indicates their recognition of errors, error types, the distribution
of errors across different interaction categories, participants’ strategies
for error recovery, and the compounding nature of errors.

Table 7. Example conversations related to errors. “…” represents truncation of conversation for brevity.  Tag |  Participant |  Alexa
---|---|---
C10
P2 |  Alexa do you have a memory of what I say to you |  I didn’t find any stickies. If you’d like to create one, just ask me to make a new sticky note.
|  Alexa I don’t want a sticky note I want you to remember the conversation do you have any kind of memory about what I asked you |  Okay, I’ve saved your sticky. [intent error]
|  Alexa never mind |
|  Alexa in the most recent literature produced by the FDA what kind of cancer did Aspartame causes? |  According to Mayo Clinic: Cancer is caused by changes, or mutations, to the DNA …[intent error]
|  Alexa, you’re a circular idiot |  [No response from Alexa]
C11
P11a |  Alexa, tell me something wonderful |  I wasn’t able to understand the question I heard [limitation error].
2-3 |  It’s ok [laughter] It’s amazing |  Great! [‘‘It’s amazing’’ not listened by Alexa]
C12 |  Alexa I took my medicine |  Sorry, I don’t know that [intent error]
2-3 P15 |  Yes I took my medicine |  Hmm, I don’t know that one. [intent error]
2-3 |  Cancel medicine |  {P15’s name}, I can’t find medicine, what’s the date or time of the event you want to delete? [intent error]
2-3 |  Cancel the alert |  [reminder stops ringing]

####  4.2.1. Frequency of various types of VA errors

Our interaction data encompassed a diverse range of errors—including
activation, intent recognition, limitation, speech recognition, and human
errors (detailed in Table 3.) Capturing activation errors for true error rate.
In our interaction data, 632 participant queries resulted in errors (24.76%)
which represents the true error rate, contrasting with prior work (Kim and
Choudhury, 2021), where activation errors were not accounted for, as they are
typically not logged in user logs. However, our methodology, which included
the use of a recording device, enabled us to capture these activation errors
(wrong wakeword, trigger, not listened, and partial listened, etc.) providing
a more comprehensive understanding of the error rate in voice assistant
interactions. Prevalence of intent recognition and Alexa limitation errors. We
found that the most frequent errors encountered by participants while
interacting with Alexa were intent recognition errors, accounting for $32.3\%$
($n=204$) of all errors (632), see Table 3. In these cases, although Alexa
accurately transcribed the user’s query, the response or action provided did
not align with the participant’s intention. The next most common error
category was Alexa’s limitations in handling user requests ($n=130,20.6\%$).
Errors associated with speech recognition system. Some of user queries
intended for Alexa were either not listened to ($n=69,10.9\%$) or not handled
by the system ($n=57,9.0\%$) due to users not using the wake word in their
follow up queries or Alexa mistakenly assuming that the user’s speech was not
a query. Other speech recognition errors such as transcription errors
($n=40,6.3\%$), partially listened queries ($n=29,4.6\%$), triggering when not
intended by user ($n=15,2.4\%$), and system errors ($n=15,2.4\%$) were less
frequent. This finding suggests that the primary challenges in smart speakers
appear in understanding user intent and VA limitations, rather than in
listening inaccuracies or system issues. Human error. On 23 occasions
($3.6\%$), error happened because participant did not complete their query.
Participants sometimes incorrectly used “Alexis” instead of “Alexa”
($n=49,7.8\%$), a confusion also noted previously (Pradhan et al., 2020;
Cuadra et al., 2022; Angelini et al., 2021). Alexa was able to pick up the
wrong wake word as a potential trigger but marked it as “not intended for
Alexa”.

####  4.2.2. Error resolution rate

While coding the data, we noted if an error was resolved in immediate retry or
not. Only $25.47\%$ ($n=160$) of all errors were resolved in the immediate
next attempt by participants indicating that many errors either remained
unresolved and required multiple attempts to rectify. Intent recognition and
limitation errors, the most frequent types, had low resolution rates of
$27.5\%$ and $10.8\%$, respectively (see Fig. 6), further highlighting Alexa’s
limited capabilities in understanding and handling user queries.

####  4.2.3. Occurrence of errors by interaction type

Frequency of errors varied across interaction types. Information retrieval.
Most errors occurred during information retrieval _i.e.,_ question type
interactions ($n=176,27.85\%$ of total errors, see Fig. 7). Significant number
of question type interactions resulted in errors ($42.51\%$, 176 of 414) while
only in $8.7\%$ (36 of 414) of interactions, errors were resolved (see Fig. 7
and Table 5). Within information seeking, certain query types had higher error
rates, such as specific places ($64.29\%$), history ($60.00\%$), local
information ($59.62\%$), famous personalities ($58.06\%$) and health-related
queries ($52.38\%$), compared to travel (11.11%), food and drink ($21.62\%$),
and entertainment ($28.57\%$) related queries. Such distinction is critical,
as errors in local (grocery stores, restaurants etc.) information and health-
related queries, which typically carry higher stakes, can lead to increased
friction in the user-VA relationship making inaccuracies in these areas are
more impactful and potentially more frustrating for users, especially since
participants asked such questions more frequently. Additionally, some question
types that were less frequently asked by participants (see Table 5) such as
specific places ($64.29\%$ error rate) and historical information ($60.00\%$)
or did not fit into any category (miscellaneous: $44,74\%$) demonstrated high
error rates—further highlighting a limitation of the VA in handling less
common or unique edge queries. Functional. Frequent errors were also noted in
functional interactions—reminders ($n=88,13.92\%$), radio ($n=59,9.34\%$),
music ($n=56,8.86\%$), and queries about Alexa’s capabilities ($n=32,5.06\%$).
For functional tasks such as reminders, a significant number of queries
failed: $35\%$ $(27+8)$ of 251 interactions (see Fig. 7), despite being
comparatively straight-forward tasks. Further examining the errors related to
Alexa’s reminder functionality, we noted that the majority of errors ($n=69$,
$78.41\%$ of erroneous reminder interactions) occurred when participants tried
to stop a ringing reminder. The interaction data revealed that participants
often used naturalistic phrases like “Thank you” or “I got it” to indicate
their intent to stop the reminders, timers, and alarms. However, Alexa did not
recognize or interpret these phrases as commands to stop the timer, as it
typically responds to more scripted commands like “stop”, “cancel”, or “I’m
done”. C12 in Table 7 is an illustrative example. This finding also hints at
the disparity between participants’ mental model of the VA’s capabilities and
its actual capabilities.

Our observations indicated that participants initially communicated with Alexa
in a natural manner, but after encountering errors, they switched to more
scripted commands. Query reformation upon errors is observed in prior work as
well (Kim and Choudhury, 2021; Pradhan et al., 2020). We also noticed that
despite experiencing these failures, participants often reverted to natural
language in later interactions, indicating a preference for intuitive
communication. Entertainment. Significant errors were also found in popular
entertainment tasks—music ($17.28\%$ error rate out of 324 interactions),
radio ($25.43\%$ out of 232), and news ($31.82\%$ out of 66). Additionally,
edge case queries such as accessing TV channels through the smart speaker
($80\%$ error rate), asking for poems ($66.7\%$), and other miscellaneous
queries ($100\%$ error rate) predominantly resulted in errors, highlighting
VAs’ limitations in handling less common requests.

Figure 6. Error types and their resolution upon immediate participant retry
attempt Figure 7. For each interaction type, percentage of interactions that
resulted in errors and percentage of interactions in which error were resolved
or not. The interaction types from left to right are in descending order of
their frequency in the interaction data. Number of errors on each bar shows
total errors in that interaction type. For instance, 414 interactions were of
question type and 176, $44\%(34\%+9\%)$ of those resulted in error and only
$9\%$ of 414 were resolved after first retry. Note: not all errors had retries
and some errors had multiple retries as discussed later.

####  4.2.4. Error identification and reactions to errors

We observed that 79.84% of conversational breakdowns were recognized by
participants, either through verbal and vocal reactions and/or by immediate
actions taken to rectify the error. The error identification rate for all
participants was at least 50%, with many showing even higher rates of error
identification (see Fig. 8). We captured participants’ immediate verbal and
vocal responses to Alexa’s errors using our recording device (see Table 8).
Overall, participants reacted verbally or vocally to $13.77\%$ of errors
($n=87$ out of 632 errors, with 545 errors not eliciting a verbal or vocal
reaction). Articulating dissatisfaction through negative responses. The most
frequent type of reaction we observed was negation, occurring 27 times. This
involved participants saying ‘no’ once or repeatedly in response to an error.
Additionally, participants sometimes took immediate action upon encountering
an error, including interrupting Alexa in 21 instances. Beyond negation and
interruptions, there were 10 instances where participants expressed negative
comments towards Alexa. These comments typically reflected frustration or
dissatisfaction with the VA’s performance. For example, C10 in Table 7
illustrated P2’s increasing frustration with Alexa as the errors compounded.
Forgiving attitude reflected in positive or neutral reactions. Instances of
laughter were recorded 14 times, and there were seven acknowledgments of
incorrect responses or actions by Alexa, such as saying “okay” or “hmm”
reflecting neutral or positive expressions. Some participants responded more
positively to errors. Expressions of gratitude (“Thank you”) were noted six
times and positive comments were made five times, illustrating a forgiving
attitude towards the VA, see C11 in Table 7.

Table 8. Verbal reactions to errors (not including attempts to correct the error): 545 errors had no reactions. Rest of the 87 errors, participants reacted to. 6 errors had compound reactions that we separated and counted under each type of reactions. Verbal Reaction to Error | Count | Verbal Reaction to Error | Count | Verbal Reaction to Error | Count
---|---|---|---|---|---
no reaction | 545 | laughter | 14 | gratitude | 6
negation | 27 | negative comment | 10 | positive comment | 5
interruption | 21 | acknowledgement | 7 | vocal (sighs, shout) | 3
Figure 8. Participants’ responses and explicit actions indicate identification
of most of the errors. For y-axis error manifestation-identification pair,
“no-no” indicates that the error did not manifest and was not identified by
the participant, “yes-no” indicates that the error manifested but was there
was no indication of its identification by the participant, and “yes-yes”
shows that the error manifested and its identified by the participant was
evident either in their immediate response or action to rectify the error.

####  4.2.5. Recovery strategies

In our study, the most prevalent strategy observed among participants when
encountering errors with Alexa was to “move on”, meaning they would give up
either after the initial error or after unsuccessful attempts to rectify it.
In their efforts to recover from errors, participants employed various
strategies. These included repeating or reformulating their query (such as
rephrasing, repeating with additional details), providing clarification,
correcting the wake word, and re-initiating the task or conversation. Fig. 9
illustrates which recovery strategies were most commonly used for different
types of errors. P15 highlighted this need for reformulating queries: “She
doesn’t understand and you have to be more creative and ask the question in a
different way. It’s like teaching English to non-English speakers; you have to
say the same thing in 50 million ways for them to get it.” For errors where
Alexa failed to understand or deliver what the participant wanted such as
intent recognition, not listened to, not handled, transcription, and system
errors, participants often chose to repeat or reformulate their queries. When
the issue was related to using the wrong wake word, the straightforward
solution was to correct the wake word. In some instances, Alexa itself
initiated a recovery sequence, particularly in cases where the participant’s
query was incomplete, and the system prompted the user to complete or clarify
their request. The findings regarding recovery strategies align with prior
research that investigated error recovery patterns during interactions with a
voice-based calendar (functional tasks only) (Myers et al., 2018). Our
analysis (Fig. 9) presents a holistic picture of the recovery strategies
employed by older adults in response to various errors encountered while using
VAs for broad spectrum of interaction purposes including functional tasks,
entertainment, and information retrieval.

Figure 9. Recovery strategies used by participants to address various types of
errors. Cumulative totals are shown below the strategy on x-axis.

####  4.2.6. Compound errors and user retries to fix conversational breakdowns

Although previous research has identified error types, estimated error rates,
and examined recovery strategies in older adults’ use of VAs (Pradhan et al.,
2020; Kim and Choudhury, 2021), these studies do not address the sequential
nature of how errors compound when users attempt to correct them. Our study
fills this gap by utilizing recorded interaction data, which captures
participants’ immediate actions (recovery attempts) after conversational
breakdowns, allowing us to analyze the ‘snowball effect’ of errors and
providing a deeper understanding of the dynamics involved in users’ erroneous
interactions with VAs.

We observed that errors with Alexa often compounded, with participants
typically attempting at least one recovery effort (see Fig. 10). While for 108
errors participants did not retry, in 238 instances they made at least one
attempt to correct the error. Of these, 107 errors ($44.96\%$) were resolved
after the first retry, 43 instances ($15.19\%$) led to participants giving up,
and in 88 cases ($36.97\%$), users attempted further retries (see Fig. 10). In
contrast to our findings in section 4.2.1 showing only $25.32\%$ (160 out of
632) of errors were resolved eventually, we shift our focus to compound
errors, which are errors emerging from participants’ reattempts to rectify the
original error. By categorizing these compound errors as a single error
instance, we discovered that the actual count of distinct errors stands at
346—this figure is the aggregate of both resolved ($n=160$) and unresolved
errors ($n=186$). This adjustment in calculation indicates a greater level of
user resilience and persistence in addressing error resolution, as the actual
resolution rate is recalculated to be higher at $46.24\%$ (160 out of 346), in
contrast to the previously noted $25.32\%$ (160 out of 632). Our analysis
revealed that while a significant number of errors ($30.92\%$, 107 out of 346)
were resolved with just one retry (see Fig. 10), many errors persisted despite
multiple retries.

Figure 10. Error retries and their resolution. The errors that were retried
were either resolved, not resolved and retried again in the next turn. Total
resolved errors are 160 ($107+43+15+...$) and unresolved errors after all
retries are 186 ($108+43+18+\dots$) . The total erroneous one-turn
interactions are 632, which is the sum of unresolved errors ($186$) and number
of retries ($446=238+88+44\dots$). So, total compound errors are sum of
resolved ($160$) and unresolved errors ($186$) which is 346.

###  4.3. Social Nature of Voice Interactions

Our audio recordings uncovered social aspects of interactions that extend
beyond typical VA usage, aspects which are not typically captured in usage
logs. These recordings enhance situated understanding of users’ interactions
with the VA and the VA’s presence in their daily lives.

####  4.3.1. Social responses and reactions

Our analysis revealed that participants’ interactions with Alexa also included
social responses, as detailed in Table 9. These interactions extended beyond
mere task completion. Valence of social interactions. People expressed
gratitude ($n=179$, “Thank you”), greeted Alexa ($n=14$, “Good morning”, “Good
night”, etc.), and laughed ($N=13$) at its responses. Participants associated
human characteristics to Alexa because of the way it responded to social
interactions, as P14 mentioned: “When I say: ‘thank you Alexa’, Alexa will
say: ‘oh, you are welcome, I’m here to serve you.’ I don’t know why I like to
hear that as if it’s a real person or something.” Moreover, we observed verbal
backchannel acknowledgements ($n=20$, “Hmm”, “Okay”). Participants also
expressed both positive and negative feedback (backchannel exemplification);
positive examples included exclamations of wonder such as “Isn’t that
amazing?” and direct compliments such as “Alexa, I really like you.” On the
other hand, negative feedback was sometimes expressed sarcastically, as in
“hell of a job,” or as critical remarks, “Alexa is much more stupider.” In 21
cases, participants also commented on Alexa’s responses or actions (see C3 and
C4 in Table 10). Variance across participants. Participants’ social
interactions with Alexa, such as expressing gratitude or greetings, varied.
While some (P6 and P15) frequently thanked or greeted Alexa, others (P1, P7,
P9, P12) did not engage in such social responses. This variance might reflect
differing perceptions of Alexa as either a tool or a social entity or it might
reflect the individual personalities of the participants. Social responses as
implicit ways of communication. An interesting new observation was P15 using a
social response (“Thank you”) instead of a command (“Stop”) with reminders,
suggesting a preference for more socially ‘polite’ interaction with the VA in
general, also highlighted in prior work (Pradhan et al., 2019). Another
illustrative example is C12 in Table 7 where P15 responds naturally (“I took
my medicine”) to a ringing reminder to cancel it but fails. Using a social or
natural response instead of direct command further underscores the tendency of
individuals to apply implicit communication cues in their interactions with
VAs, akin to those used in human interactions. Similar behavior is also
evident when participants expressed gratitude to the VA when it failed to
fulfill their request (illustrated in Table 8). The implicit and natural ways
of communication suggests that users may attribute social characteristics to
voice assistants, influencing the nature of their communication. However, we
observe that Alexa often fails to comprehend these implicit communication
methods (see C12), while participants continue to use them, indicating a
mismatch between the participants’ mental model of the VA’s capabilities and
its actual functionality.

####  4.3.2. Voice assistants in social conversations

In our study, we not only observed social responses and reactions to Alexa
during participant interactions but also identified social conversations
surrounding Alexa that signify the integration of voice assistants into users’
lives (see Table 11 left). We captured participants’ discussions about Alexa,
instructional conversations on its use, and dialogues stemming from topics
initiated with Alexa. A notable finding was participants talking about Alexa
($n=50$) either immediately after or during its use as depicted in C5 in Table
10.

Additionally, there were 21 instances where participants engaged in
discussions on how to operate Alexa with others—more frequently couples trying
to navigate its functionality together (C6, Table 10) and less frequently when
participants were showcasing Alexa to visitors (C7, Table 10). Participants
also engaged in conversations related to topics initiated with Alexa
highlighting need-based use indicating deeper integration of the VA into their
lives. An illustrative example is a couple (P11a and P11b) discussing the
history of Tony Bennett, where P11a relied on Alexa for information, only to
find that the VA did not provide the expected response (see C8 in Table 10).
Another example is C9 where P1a verifies Alexa’s answer to P1b’s question.

####  4.3.3. Social indicators of friction in interactions

As previously highlighted in section 4.2.4, Alexa elicited verbal reactions of
social nature when conversational breakdowns happened due to errors. The
social reactions encompassed a broad spectrum, ranging from acknowledgement
and laughter to negation, negative commentary, and even positive remarks and
expressions of gratitude, as detailed in Table 8. Such a diverse array of
social responses suggests that interactions with the VA extend beyond mere
transactional exchanges.

In addition to reactions to errors, we also captured and coded aspects of
participants’ interactions that indicate friction in the conversations and are
usually not captured through usage logs. For instance, interruptions and
overlap in Alexa and participants’ speech (see Table 11 right). It is
noteworthy that most of the interruptions in the whole interaction data (21
out of 31, as referenced in Tables 8 and 11) were a direct result of errors
made by Alexa. Despite encountering issues during interactions with Alexa and
Alexa’s tendency for errors as discussed in section 4.2, our observations
revealed minimal overlap and interruptions in participants’ speech when
interacting with Alexa, suggesting a degree of patience and forgiveness from
our participants towards the VA despite conversational failures. Prior work
also reflects a similar sentiment in the elderly users _i.e.,_ despite
experiencing more errors, elderly users tend to exhibit a higher willingness
to continue working with voice-based agents in the future compared to younger
adults (Barros and Seabra, 2020).

Table 9. Social responses and reactions by users to Alexa Social Response/Reaction | Count | Social Response/Reaction | Count
---|---|---|---
no | 2301 | greeting | 14
gratitude | 179 | laughter | 13
remarks | 21 | verbal backchannel exemplification negative | 12
verbal backchannel acknowledgement | 20 | verbal backchannel exemplification positive | 9
Table 10. Example conversations of social aspects beyond simple interactions. “…” represents truncation of conversation for brevity. Full conversations are presented in supplementary materials. Tag |  Participant |  Alexa
---|---|---
C3 |  Alexa what is Donald Trump’s educational background? |  Donald Trump was educated at the UPenn.
2-3 P2 |  That’s it, no Harvard no Yale. [remark] |  [not listened by Alexa]
C4 |  [User initiates incomplete reminder request] |  When should I remind you?
2-3 P11b |  [No user response] |  What’s the reminder for?
2-3 |  She Remembers it! |
C5 |  Alexa put hard candy on my shopping list. |  You already have hard candy on your shopping list.
2-3 P6 |  Oh. Did you hear her? I said Alexa …[P6 to visitor] |
C6 |  Alexa, give me a 10 minute warning please. |  10 minutes, starting now [it worked]
2-3 P11a |  |  [P11b to P11a: No you need to say timer for 10 minutes [overlapping Alexa speech]
2-3 |  What wait wait what {P11b’s name} |  [P11b to P11a]: Set a time for 10 minutes
2-3 |  Okay [To P11b] Alexa, set timer for 10 minutes |  Second timer, 10 minutes, starting now
2-3 |  Set a time for 10 minutes [P11b to P11a] |
C7 |  Alexa add milk to my shopping list. |  I added milk to your Shopping list.
2-3 P8 |  I don’t know where the list is. Oh wait a minute may be it’s in Amazon. …[Visitor to P8] |
C8 |  Alexa play the history of Tony Bennett. |  [P11b to P11a]: What’s the history of Tony Bennett
2-3 P11a |  [To P11b] Oh she’ll tell you |  [P11b chuckles mockingly]
2-3 |  She didn’t tell you [to P11b] |  [P11b to P11a]: Because it makes no sense
C9
P1b |  Did the actress who played the voice of howard wolowitz on third rock from the sun really die? |  From fun107.com: But sadly the actress behind that memorable voice has passed away.
2-3 |  "That is true. She did pass away" [P1a to P1b] |
Table 11. Participant Interaction with Alexa: other aspects Description | Count | Description | Count
---|---|---|---
Private conversation about Alexa | 50 | Overlap: participant starts talking over Alexa | 12
Private conversation about how to use Alexa | 21 | Overlap: Alexa starts talking over participant | 4
Private conversation about same topic | 15 | Participant interrupts Alexa | 31
|  | Alexa interrupts participant | 15

###  4.4. Interactions with LLM-powered VA

After the second week, we introduced the ChatGPT+Alexa skill to participants
to explore adoption and initial perceptions of older adults towards LLM-
powered conversational VAs.

####  4.4.1. Barriers to adoption

We observed that only a few participants (P1, P3, P11, P15) interacted with
ChatGPT skill, primarily around the day of deployment, as indicated by
activity peaks in Fig. 4. Participants reported challenges in learning and
accessing the skill as their reasoning to not try it out—P9 pointed out, “I
don’t like the skill part of it. It’s too complicated, too cumbersome. I don’t
like it at all.” and further found the idea of chatting with a machine
unusual, “I don’t want to try to have an artificial conversation. It’s just
repugnant to me.” This pattern highlights the reluctance of our participants
to interact with advanced conversational VAs.

####  4.4.2. Nature of questions asked

Participants asked a variety of questions such as health related ($n=7$,
example C16 in Table 4.4.4), history ($n=6$, C13 in Table 4.4.4), VA related
($n=3$, “I wanna know your age”), specific places ($n=2$), asked for a story
($n=2$), recipe ($n=1$, C15 in Table 4.4.4), and two other questions shown in
C17 Table 4.4.4.

####  4.4.3. Conversational breakdowns

Compared to traditional information seeking through Alexa, the ChatGPT skill
showed a lower percentage of errors (Fig. 7), with a higher resolution rate of
$14.1\%$, versus $8.7\%$ for standard questions. Most errors stemmed from the
integration of ChatGPT into the Alexa skill, involving system issues,
listening, transcription, and intent recognition for activating and exiting
the skill. Table 12 details the distribution and resolution of errors during
ChatGPT skill interactions. All intent recognition errors were related to
difficulties in exiting the skill (_e.g.,_ C16 in Table 7) and were resolved
successfully. Partial listening errors ($n=3$), often due to participant
taking longer to formulate their queries, were not resolved as participants
did not retry to fix those. P15 also expressed the need to be fast to interact
with the ChatGPT skill “I think I wasn’t thinking fast enough”.

####  4.4.4. Learning curve

Successful interactions with ChatGPT (C15-C17 in Table 4.4.4) showed a
learning curve for users, especially in managing multiple interaction layers,
such as initiating questions, asking follow ups, and closing the conversation.
For instance, in C16, P15 initially faced challenges in navigating the
conversation with the ChatGPT-powered Alexa skill. However, with practice, P15
was able to engage in a much more fluid interaction, demonstrating
adaptability and learning over time. Despite initial successes, repeated
interactions with the ChatGPT-powered skill were limited in the latter two
weeks of the study.

Table 12. Errors in ChatGPT Alexa skill interactions Individual | Total | Resolved |  Compound errors
---|---|---|---
intent | 5 | 3 |  All the intent errors stemmed from difficulties exiting an Alexa skill, attributed to the skill itself, not ChatGPT. Participants attempted a few retries: two retries for two of the errors and one retry for the remaining error eventually resolving all intent errors.
system | 4 | 3 |  System errors compounded (once with not handled error) but were resolved on the second try. One system error remained unresolved as the participant chose to move on without attempting a retry.
listened partial | 3 | 0 |  None of the listened partial errors were resolved.
not handled | 3 | 1 |  Only one of the errors was resolved in one retry.
transcription | 3 | 1 |  Two transcription errors compounded, but were resolved on the second retry. However, another one remained unresolved as the participant did not attempt a retry.
interrupted | 1 | 1 |  On retry the error was fixed.
limitation | 1 | 0 |  Participant did not retry to fix the error.
Table 13. Example conversations of ChatGPT+Alexa skill. “…” represents
truncation of conversation for brevity. Full conversations are presented in
supplementary materials.

Tag |  Participant |  ChatGPT+Alexa skill
---|---|---
C13
P3 |  What are the interesting a historical event [cut off by Alexa] in 19 …[partial listening error] |  There are countless interesting and significant historical events. Here are a few examples: 1. The signing of the Declaration of Independence in 1776 …
C14
P11a |  Alexa I understand that you now can do chatGPT, is that correct? |  Hmm, I don’t know that. [P11b to P11a:] You have to do certain commands
2-3 |  Okay I have to read it then [P11b to P11a] Alexa what do you know about chatGPT |  ChatGPT is an AI-powered chat tool released in November 2022. It uses OpenAI’s large language models …
C15
P1b: |  Alexa let’s chat |  Hey! I am an intelligent virtual agent programmed to provide you with the requested information …What do you wanna know?
2-3 |  What’s the best way for to make hard boiled eggs? |  Searching... [filler + small talk] How do you unwind?
2-3 |  I take a nap
[P1b thanked Alexa afterwards] |  Here it is... To make hard-boiled eggs, follow these steps: 1. Place your desired number of eggs …
2-3 C16
P15 |  I’m having trouble falling asleep. [cut off by Alexa] what can you do to help me |  [After small talk] Sorry, I had trouble doing what you asked. Please try again. [system (skill) error]
2-3 |  I’m having trouble falling asleep I take medicine but it doesn’t help what should I use to help me fall asleep [repeat-details] |  Here it is... I’m sorry to hear that you’re having trouble falling asleep. Here are a few techniques that might help: 1. Create a bedtime routine: Establish a relaxing routine …Make sure your bedroom is quiet, dark , and at a comfortable temperature …
2-3 |  Thank you a lot of good information |  What else can I help you with?
2-3 |  Why should i keep the room dark [cut off by Alexa] to help with sleep |  Searching [P15 did not wait for VA response and repeated herself]
2-3 |  Alexa why should I keep the room dark to help with sleep [not listened by Alexa] |  "Here it is... Keeping the room dark can promote better sleep because darkness signals to your body that it’s nighttime and time to rest. Our bodies naturally produce a hormone called melatonin, …
2-3 |  Ok thank you |  What else can I help you with?
2-3 |  ummm nothing else thank you [intent error, Alexa skill cannot detect intent to end the conversation] |  Sorry, I don’t know about that. Is there something else you would like to know? Try saying echo or chatbot before your question if you are trying to ask ChatGPT something.
2-3 |  no nothing [intent error, Alexa skill cannot detect intent to end] |  "Here it is... If you feel that none of the suggestions are helpful or applicable to your situation …
2-3 |  Alexa stop [P15 interrupted Alexa] |  Goodbye! [scripted end intent worked]
2-3 C17
P15 |  Is the extreme heat ummm natural to weather |  Here you go...Extreme heat is a natural occurrence in weather patterns. It can be caused by various natural factors such as high-pressure systems, heatwaves, or the position of the sun …
2-3 |  Ok thank you |  What else can I help you with?
2-3 |  hmmm can you tell me a story |  [not handled by Alexa error] Any other questions for me?
2-3 |  can you tell me a story [repeat] |  Here it is... Certainly! Here’s a short story for you: Once upon a time, in a small village nestled among rolling hills, there lived a curious young girl named Maya. Maya loved exploring …
2-3 |  Umm thank you good story |  You’re welcome! I’m glad you enjoyed the story. If you have any more requests or questions, feel free to ask.
2-3 |  I have a question what’s good music to help me relax |  Here you go...The choice of relaxing music varies from person to person , as everyone has different preferences. However, here are a few genres and artists known for creating calming and soothing …

##  5\. Discussion

Table 14. Design considerations mapped out across various themes of user-VA interaction. Main Theme | Sub-Theme |  Corresponding Design Considerations
---|---|---
Older Adults Challenges | General VA |  DC1, DC2
2-3 | Forgetfulness |  DC4
2-3 | Speech difficulties |  DC5, DC6
2-3 | Varying Familiarity and Expectations |  DC3, DC7
2-3 | Varying Conversation Style |  DC8
Error Category | Automatic Speech Recognition Errors |  DC5, DC6
2-3 | VA Limitation or Human Errors |  DC2, DC3, DC7
2-3 | Intent Recognition Errors |  DC8
Error Management Step | Avoiding Error |  DC2, DC4, DC5, DC6, DC7, DC8
2-3 | Error Detection |  DC1
2-3 | Recovery and Repair |  DC1, DC3
Implementation Level | Voice User Interface |  DC5
2-3 | Model |  DC1, DC8 (Modeling vocal and verbal cues) DC2, DC6, DC7 (Using LLMs)
2-3 | Interaction |  DC1 (Mitigation)
DC4 (Proactive)
DC3, DC7 (Educational)

Our study delves into the nuanced dynamics of older adults interacting with
smart speaker based voice assistant Alexa captured through real-world audio
interaction data. We investigate various aspects, including how Alexa is
situated and socially present in their conversations and lives, the nature of
conversational breakdowns due to accumulating errors, and the struggle of
older adults to recover from these errors, such as needing multiple attempts
and rigid query formation. These insights reveal the limitations of current
commercial VAs and uncover opportunities for improvement, leading to specific
design considerations. Furthermore, our study extends to interactions with an
LLM-powered VA for informational retrieval, highlighting barriers to adoption,
older adults’ initial perceptions of such VA, its potential to improve the
interaction and the challenges older adults faced during the interaction to
offer design insights for such advanced and highly conversational VAs.

In this section, first we discuss social presence of VAs in older adults’
life. Secondly, we pinpoint missed opportunities for developing resilient VAs
in general. Thirdly, we highlight age-specific challenges older adults face
that result in automatic speech recognition errors. Finally, we discuss
mismatch between older adults understanding of VA capabilities and VAs’
limited comprehension of user intent resulting in limitation and intent
recognition failures respectively. We offer design considerations covering
various themes of older adult interactions with VA (see Table 14).

Table 15. Additional illustrative conversations. “…” represents truncation of conversation for brevity.  Tag |  Participant |  Alexa
---|---|---
C18
P15 |  Alexa tell me a joke |  Sometimes people ask me if I speak lizard. I don’t, but iguana learn someday!
2-3 |  [laughter] Okay, so you still working |  [not listened by Alexa]
C19
P6 |  Alexa what is a meme |  Internet memes are jokes within Internet culture that have achieved a certain level of celebrity, and are generally recognized and shared independently of their original source material. Noteworthy memes are often sarcastically praised for being dank.
2-3 |  Can you Alexa give an example of a meme? |  Here is a meme: What Does This Monster Have to Mourn?
C20 |  Alexa can you keep a recipe for me |  Sure I can search by dish name and ingredients. What would you like?
2-3 P1b |  No, I wanna keep one of my own recipes |  Sure I can search by dish name and ingredients. What would you like?
2-3 |  Never mind |

###  5.1. Voice Assistant in Older Adults Life

Our findings corroborate prior studies in demonstrating the utility of voice
assistants in the lives of older adults, particularly in supporting memory
through functional tasks, providing entertainment, and delivering necessary
information (Kim and Choudhury, 2021; Pradhan et al., 2020). Participants
explore a variety of VA features, reflecting the VA’s multifaceted role. For
instance, playing music and listening to news indicating the use of VAs beyond
need-based tasks. Many engaged in creating reminders, shopping lists, pointing
to a blend of functional and entertainment purposes. Some participants used VA
for diverse purposes such as attempting to connect their Audible account for
listening to audio books, syncing their personal calendar to stay updated,
asking Alexa to tell jokes, stories, and poems—suggesting that VAs are
perceived not merely as tools but as ‘digital companions’, a perspective that
differs from that of younger adults (Chung et al., 2019). Our study also
reveals the creative ways in which different participants interact with VAs.
P15, for example, demonstrated an inventive approach by asking Alexa to tell a
joke to check whether the VA is working after malfunctioning (C18 in Table
15). This interaction is an illustration of the novel ways in which users
adapt their interactions with VAs, transforming routine troubleshooting into a
more interactive and enjoyable experience. Similarly, questions such as
“Alexa, are you there?” or “Alexa, are you working?” show a form of human-
agent interaction that resembles a ‘checking up on’ behavior.

Our findings expand on existing knowledge by illustrating how smart speaker-
based VAs can become integrated and situated within the daily lives of older
adults. Captured social interactions with VAs reveal significant insights;
older adults often engage in social niceties with VAs, such as expressing
gratitude or greetings (section 4.3.1) which is also evident in prior work
(Pradhan et al., 2019). As prior work has established that older adults are
more likely to anthropomorphize the agent, often using courteous language to
show gratitude, whereas younger adults generally viewed it more as a
utilitarian tool, focusing on its convenience (Oh et al., 2020; Chung et al.,
2019). Expressions of affection towards Alexa—“I really like you” or “I love
you,”—not only show an emotional attachment that may result in raised
expectations but also raise ethical concerns such as deception,
infantilization, and privacy about the nature and impact of such bonds
(Sharkey and Sharkey, 2012; Wachsmuth, 2018). Given that older adults are a
more vulnerable demographic, it is crucial that AI assistance is designed with
these repercussions in mind.

Furthermore, the use of VAs in situated conversations and as part of social
interaction with others (as shown in section 4.3.2) points to their evolving
role. Instances where couples involve Alexa in their discussions (C8 and C9 in
Table 10), or participants such as P6 showcase Alexa to visitors (C5 in Table
10), reflect a perception of VAs as active, social entities. The changing use
and perception of VAs among older adults point to their significant role in
daily and social activities beyond basic utility raising questions about their
role that transcends simple assistance. Such shift calls for more discussion
on designing VAs to suit the specific needs and preferences of this
demographic, considering both ethical aspects and possible improvements.

###  5.2. Designing Context-Aware Resilient VAs for Improved Error Management

Our findings underscore the potential of leveraging users’ immediate
reactions, responses, actions to enhance older adults’ interactions with VAs.
Additionally, we highlight the potential of utilizing LLMs’ ability to
maintain contextual information and conversation history from our results. In
the following discussion, we explore these missed opportunities identified in
our study and suggest design considerations for creating more resilient VAs
with improved error management and conversational capabilities.

####  5.2.1. User reactions and responses to conversational breakdowns: a
missed opportunity

Our research indicates that about 80% of errors in VA interactions can be
identified through participants’ immediate verbal reactions (_e.g.,_ negation,
remarks and comments by users) or their attempts at recovery (_e.g.,_ users
reformulating their queries) as illustrated in section 4.2.4. However, such
valuable feedback is not being effectively leveraged by current commercial VAs
even though they may capture immediate user reactions and follow up queries.
Alexa’s follow-up mode, for instance, captures a portion of immediate user
responses but often fails to categorize them properly, labeling many as “Audio
not intended for Alexa,” or “Sorry, I don’t know about that” and does not
utilize these cues to improve interaction. Effectively incorporating users
immediate responses or reactions, particularly in multi-turn interactions,
could greatly enhance the VA’s error detection process. After an error has
been identified and rectified, the VA can engage in verbal trust repair such
as apologizing for the mistake (Mahmood et al., 2022) to improve the recovery
process.

Opportunity 1: Users react, respond or attempt to fix most errors made by VAs.

Design Consideration 1: Refine VAs to better utilize user immediate vocal and
verbal reactions and responses for improved error detection, recovery, and
trust repair.

####  5.2.2. Mitigating conversational friction: leveraging LLM capabilities
to improve VA interactions

A more holistic approach—involving the maintenance of conversation history, a
deeper contextual understanding, and better understanding of humans’ verbal
cues—could significantly improve VAs’ ability to discern the reasons behind
misunderstandings. Adopting such approach could not only result in more
precise error identification but also offer users improved assistance,
effectively employing user feedback to enhance voice assistant functionality.
One way is to augment the existing voice assistants’ natural language
understanding capabilities by integrating LLMs to better recognize user’s
intentions behind their spoken words. Prior work has show that the integration
of LLMs into VAs significantly enhances their conversational abilities for
information retrieval; such LLM-powered VA maintains context and
conversational history, enabling users, to ask follow-up questions and engage
in multi-turn interactions with reduced friction (Mahmood et al., 2023). We
also observed similar improvement in conversation flow in our exploration of
ChatGPT-integrated Alexa skill with older adults. For example, P15
demonstrated such enhanced interaction by asking a follow-up question, “Why
should I keep the room dark,” after receiving the said advice on falling
asleep (refer to C16 in Table 4.4.4). LLMs’ ability to maintain context and
respond to vague queries represents a substantial improvement in user-VA
interaction for older adults.

Opportunity 2: Older adults often issue vague queries to VAs due to their
assumption of an existing conversational context and history.

Design Consideration 2: Employ LLMs to enhance the contextualization of multi-
turn interactions with the VA, thereby improving its understanding of vague
user queries.

Despite the improvement in conversational flow, we did observe that older
adults faced difficulties interacting with the ChatGPT-integrated Alexa skill
as described in section 4.4.3 but these failures were related to Alexa’s
speech technology and skill activation, particularly during attempts to exit
the skill, yet the overall intent was accurately identified in most cases (see
section 4.4.3). Additionally, many errors in ChatGPT skill interactions were
often resolved within a maximum of two retry attempts by participants (see
Table 12, showcasing better error resolution compared to standard Alexa
interactions where many errors remained unresolved despite multiple retries as
shown in section 4.2.6. While system and skill level errors persisted, we
observed a learning curve as users became more adept at interacting with the
LLM-powered VA with practice (section 4.4.4). For instance, comparing P15’s
initial interaction in the afternoon (C15 in Table 4.4.4) with a later one in
the evening of the same day (C17 in Table 4.4.4) reveals a more fluid
conversation. Improvement after trial and error in user interaction with such
advanced VAs highlights the critical role of guidance for older adults in
their initial usage stages, akin to any new technology. This learning process
can be further improved by the VA offering insights into why an error may have
occurred and suggesting alternative ways to phrase requests or retry commands.
Such proactive assistance can create a more intuitive and supportive
experience, aiding users, particularly older adults, in effectively navigating
and correcting errors.

Limitation 1: System level challenges can arise when older adults interact
with advanced conversational LLM-powered VAs due to unfamiliarity with
technology.

Opportunity 3: Older adults show a learning curve by getting better at using
advanced LLM-powered VAs through practice.

Design Consideration 3: Design VAs to offer error-specific feedback and
guidance on recovery, enhancing usability during initial and ongoing
interactions.

###  5.3. Tailoring VAs for Older Adults: Understanding and Addressing
Suboptimal Interactions Due to Age-Related Factors

Voice Assistants are often not designed with the aging population in mind
(Stigall et al., 2019), leading to unique challenges and errors for older
adults in their interactions with VAs. Factors such as unawareness of
technology and cognitive decline causing forgetfulness, and speech
difficulties causing speech detection and recognition failures, are unique
challenges. In the following sections, we explore how various speech
recognition failures (partial, listening, transcription, and wrong wakeword
errors) reflect these challenges and propose design considerations to enhance
the experience for older adults.

####  5.3.1. Forgetfulness

In exploring the continued use of voice assistants by older adults, a
significant limitation observed is forgetfulness which manifests in three
principal ways. Firstly, participants often forget the VA’s utility or even
its existence, a reflection of either their cognitive decline or their
ingrained habits of not depending on VAs for tasks also pointed out in prior
work on older adults interactions with VAs (Trajkova and Martin-Hammond,
2020). While the current reactive nature of VAs necessitates user initiation
for activation, our findings indicate that when participants are reminded of
the VA, their engagement with the VA increases (see section LABEL:sec:usage-
over-days and Fig. 4). Such sudden and short lasting increase suggests a
potential benefit in transitioning towards more proactive interactions. VAs
could initiate actions based on user habits or daily routines, such as
reminding about medication or daily tasks, especially if the user is prone to
forgetting these activities. For example, since P15 habitually sets reminder
almost every day for medication at 6 p.m., the VA could proactively provide or
suggest these reminders to be repeated daily, compensating for any lapse in
the user’s memory to set the reminder everyday. Such forgetfulness is also
iterated by older adults in prior work (Pradhan et al., 2020) further
highlighting the need for proactive interactions. While proactive VA behavior
in critical settings is appreciated by users, it is essential for such VAs to
take social and environmental context into account to ensure users’ agency and
control over interactions (Zargham et al., 2022). For instance, timing of
proactive interactions can be personalized by mapping out their interactions
throughout the day (similar to Fig. 5) to estimate their preferred time for
certain interactions through creation of user profiles. In less critical
situations, VAs should seek user permission while starting proactive
interactions and let the user know the purpose and reason explicitly (Zargham
et al., 2022). Therefore, considering when and to what extent interactions
should be proactive is essential. Mixed interactions—balancing reactive and
proactive—for older adults needs further exploration.

Secondly, we observed is participants mistakenly using the incorrect wake word
’Alexis’ instead of ’Alexa’, leading to conversational breakdowns as Alexa did
not respond despite detecting the similar-sounding trigger word (refer to
section 4.2.1). Such human error, often a result of their forgetfulness, could
potentially be addressed by system-generated messages and reminders such as
(“Did you mean to say ’Alexa’ I’m not certain if you were addressing me.”)
upon error detection, rather than the VA remaining unresponsive. However, the
practice of the system assuming the correct wake word when it has not been
clearly articulated can be perceived as intrusive, thereby raising concerns
about user privacy.

Thirdly, another aspect of forgetfulness pertains to the inability to recall
the purpose of interactions such as setting timers, reminders, or alarms.
Participants often find themselves forgetting and questioning the intent
behind an alert—saying “what am I supposed to do?”, “what should I do?” when
the reminder rings without a message—indicating a mismatch between VA
functionalities and the needs of older adults. To address this, VAs could be
designed to solicit more detailed information at the time of setting reminders
or alarms, thus making them more effective and tailored to the user’s needs.

Limitation 2: Reactive nature of VAs fails to accommodate forgetfulness of
older adults.

Design Consideration 4: Design VAs to proactively suggest actions and prompt
older adults for further details to help them formulate their queries, rather
than merely reacting to their initial requests.

####  5.3.2. Speech difficulties and challenges

While most of the errors faced by the older adults in our interaction data
stemmed from the VA’s failure to recognize user intent or fulfill requests, a
considerable portion of errors are related to automatic speech recognition
technology such as partial listening and transcription errors as outlined in
section 4.2.1 and depicted in C1 of Table LABEL:tab:alexa-dialogue-questions.
Moreover, these errors exhibited low resolution rates (refer to Table 6). Such
speech recognition system inaccuracies are particularly prevalent among older
adults as they may take longer to formulate their queries—due to unfamiliarity
with technology and speech difficulties associated with aging—suggesting a
need for improvement in VAs to better accommodate users with slower speech
patterns and speech difficulties. Since future VAs are visioned to be more
conversational, it is crucial for these systems to allow older users
sufficient time to process information and articulate their queries. Such
adaptations would make voice user interfaces (VUIs) more responsive to the
needs and communication styles of older adults, enhancing their overall
experience with the technology.

Limitation 3: Limited adaptability of VUIs for current assistants to slower
speech patterns and extended speaking time needs of older adults especially
when VAs are more conversational.

Design Consideration 5: Tailor VUIs to better suit the speech pace of older
adults, ensuring fewer speech recognition errors.

We observed that partial listening, and transcription errors also occurred in
older adults’ interactions with LLM-powered VAs, as shown in C13 and C16 of
Table 4.4.4. These errors, related to speech detection and recognition, can
impede the understanding of user intent. However, previous research indicates
that LLMs can potentially mitigate such errors by absorbing
incomplete/incorrect inputs or initiating proactive recovery sequences to
prevent conversational breakdowns (Mahmood et al., 2023). An illustrative
example from our study (C16 in Table 4.4.4) showcases an instance where
ChatGPT successfully managed a situation of partial listening due to an
interruption by Alexa. The user’s incomplete sentence, “Why should I keep the
room dark [cut off by Alexa] to help with sleep,” was not fully captured,
missing the latter part. Moreover, when unable to estimate user intent, LLM-
powered VAs can initiate error recovery by prompting user for clarification or
to repeat themselves to maintain conversational flow (Mahmood et al., 2023).
Additionally, Amazon Alexa’s newly proposed speech-to-speech LLM, designed to
improve the entire pipeline from user speech to VA response, could be
advantageous in addressing speech capturing
inaccuracies333https://www.amazon.science/blog/alexa-unveils-new-speech-
recognition-text-to-speech-technologies. All in all, LLMs’ ability to
comprehend user intent despite partial or partially incorrect speech inputs
can significantly reduce conversational breakdowns, improving interaction
quality, especially for older adults who might struggle with clear and concise
speech. Future research should explore the benefits of integrating LLMs into
voice assistants for older adults in detail.

Opportunity 4: LLMs are capable of absorbing speech recognition errors such as
partial listening and transcription inaccuracies to reduce conversational
breakdowns.

Design Consideration 6: Leverage LLMs’ context awareness to reduce speech
recognition errors for older adults to further accommodate for their unique
speech patterns and characteristics.

###  5.4. Aligning Older Adults’ Expectations and Voice Assistant Functional
and Comprehension Abilities

Our findings indicate that participants often overestimate or underestimate VA
capabilities resulting in limitation errors. Moreover, VAs struggle to
understand the natural speech patterns of older adults causing intent
recognition failures highlighting a need to enhance VAs’ ability to understand
user intentions. The continuous advancement in VAs’ functional and
comprehension capabilities, necessitates an active recalibration of older
adults’ mental models for optimal technology use. In the following discussion,
we explore and present design considerations evident from our findings for
both adjusting the mental models of older adults and improving VA
comprehension of their intent.

####  5.4.1. Inaccurate mental model of VA capabilities

Our study reveals a notable discrepancy in participants’ understanding of
voice assistant capabilities, leading to an inaccurate mental model. The high
incidence of limitation errors ($n=130,20.6\%$), where the VA fails to fulfill
user intents and responds with “I don’t know about that” (see Table 3),
illustrates that Alexa often cannot meet older adults’ requests. Occurrence of
limitation errors, thus, suggest that older adults may not be fully aware of
the VA’s capabilities, especially regarding how to formulate their queries
that Alexa can comprehend. Additionally, certain interaction types with VAs
exhibit notably higher error rates, particularly in less frequent, ‘edge’
categories such as miscellaneous queries, TV-related commands, stories, and
poems (see Fig. 7). Such trend extends to less commonly asked questions
pertaining to famous people, Alexa itself, history and specific places that
had error rates exceeding 50% (refer to Table 5). Similarly, miscellaneous
queries that did not fit other categories also had high error rate of 44.47%.
The high error rates in less common interactions and questions further
emphasize the mismatch between user expectations and VA capabilities
underscoring the necessity for VAs to either adapt more effectively to a
broader range of user queries and commands, or to communicate their
capabilities in a clearer manner.

The disconnect between user mental models and VA capabilities becomes more
apparent upon examining example interactions between couples with divergent
mental models. For instance, in the case of P11, the couple demonstrated
differing perceptions: P11a overestimated the capabilities of Alexa (C8 Table
10 and C11 Table 7), whereas P11b underestimated them (C6). Such disparities
in understanding were not isolated; several participants either overestimated
or underestimated the VA’s functionalities. Instances of overestimating were
also observed with individual participants such as P8 attempting to access TV
channels through the VA and P2 asking the VA to remember their conversations
(C10 in Table 7). Moreover, P6 requested an example of a meme, highlighting a
misconception about the VA’s ability to convey visual information verbally
(C19 Table 15). Questions about Alexa’s capabilities, such as P11b asking if
it could speak in a British accent or P2 questioning the extent of its
conversational memory (C10), further illustrate how participants are unaware
of its capabilities and tend to directly ask the VA.

Regular users of multiple smart speakers for many years still had
misconceptions about VA’s capabilities. For instance, P1 attempted to control
non-smart, unconnected lights via Alexa reflects a fundamental
misunderstanding of the technology’s requirements and limitations.
Additionally, the same participant’s habit of commanding Alexa to stop talking
even after it had finished indicates a cautious approach (underestimating VA
capabilities), stemming from uncertainty about controlling the VA. Such
behavior was also noted in the participants who are new users, suggesting a
broader trend of over-caution (saying “stop” when not necessary or saying
“stop” multiple times) due to unclear understanding of the VA’s operation or
their experience during erroneous interactions. Previous research indicates
that users often rely on trial and error to develop their mental model of the
VA (Kim and Choudhury, 2021). However, our finding of mental model mismatch in
‘experienced’ users suggests that this method might not always correct their
mental model; instead, it tends to reinforce behaviors that ‘work’ rather than
those that are more effective and accurate. Consequently, providing both
novice and experienced users—particularly older adults who may have a limited
grasp of the technology—with accurate information is crucial. Furthermore,
with rapid advancements in the field with the integration of LLMs in voice
assistance technology (OpenAI, 2023; Rausch, 2023), the capabilities of VAs
are evolving too quickly for older adults to maintain a fixed mental
model—which calls for dynamic approaches to continually update their
understanding of these technologies. To promote continued use of VAs, it is
essential to develop user-focused instructional protocols that offer clear
initial guidance and continued learning support across various formats such as
online tutorials and in-device instructions (Koon et al., 2020). Therefore,
dedicated tutorials with real-time updates or corrective actions by VA
embedded in daily interactions can be employed to effectively convey the VA’s
capabilities and optimize older adults use of this technology.

Moreover, VAs should be designed to detect instances of user misconceptions,
responding with tailored information that helps rectify these
misunderstandings. Instead of defaulting to generic and uninformative
responses such as “Sorry, I don’t know that,” the VA should provide context-
specific explanations (Pradhan et al., 2020; Kim and Choudhury, 2021). For
instance, if a user attempts to interact with a non-smart appliance, the VA
should explain the prerequisites for smart home integration, rather than
merely stating, “Sorry, I couldn’t find a connected light.” Such approach not
only clarifies the VA’s limitations but also educates the user about the
necessary conditions for optimal use. Responses should be tailored to
accommodate varying levels of technological familiarity, specifically for
older adults.

Limitation 4: Lack of conveyance of its capabilities by the VA continually
results in inaccurate mental model for older adults even in long-time users.

Limitation 5: Rapid advancements in VA technology further exacerbates the gap
between the new capabilities and the relatively static mental models of older
adults.

Design Consideration 7: VAs should employ context-aware responses with dynamic
tutorials and real-time guidance in daily interactions to better align older
adults’ mental models with the evolving VA technology.

####  5.4.2. Lack of VA’s comprehension of users’ natural way of conversing

Our findings highlight a significant gap in Alexa’s ability to comprehend the
natural conversational patterns reflected in high rate of intent recognition
errors, where the VA incorrectly responds to user queries ($32.3\%$,
$n=204/632$, as detailed in section 4.2.1). For instance, participants often
preferred to use more conversational phrases such “Thank you” instead of
command-like terms such as “stop,” likely perceiving the latter as
interruptive or rude. This could stem from the fact that older adults tend to
favor polite interactions, both in their own communication with VAs (Pradhan
et al., 2019) and in receiving responses from VAs (Hu et al., 2022). Moreover,
participants consistently reverted to natural phrases (“Thank you, I have
taken my medicine”) instead of specific commands (“cancel alert”) as
illustrated in C12 Table 7 underscoring a desire for more intuitive
interaction with VAs. Such intent recognition failures become particularly
critical in high-stakes scenarios, such as medication reminders. P15 relied on
Alexa for medication reminders but struggled with informing the VA when the
task was completed (C12 Table 7), often leading to confusion and frustration
captured during one of her interactions during the study: “I don’t know what I
am supposed to say to it to tell it that I have taken it”. Despite the VA
having the functionality to track completed reminders, it often failed to
understand P15’s queries such as “Did I take my medicine today?” Older adults
find the need to use specific keywords or commands annoying (Pradhan et al.,
2020, 2018).

Compound errors (refer to section 4.2.6) further escalates the friction
between the user and the VA, as illustrated in C1 Table LABEL:tab:alexa-
dialogue-questions where P2 keeps reformulating his queries to get the
information he wants. Failure to recognize user intents can also be a reason
for decreased usage over time as indicated in our findings (see Fig. 4). For
instance, P1b discontinued attempts to communicate with Alexa after repeated
misunderstandings (see C20 in Table 15).

Requiring users to pose scripted commands and well-formulated queries ties
back to the previously discussed mismatch in user mental model (section 5.4.1)
and is exacerbated by the observed forgetfulness and speech difficulties in
older adults (section 5.3). Older adults reverting to use of their natural
style of conversing highlights their expectation and preference for VA to be
more social and conversational as opposed to younger adults (Oh et al., 2020).
Therefore, adapting voice assistants to understand and respond to more
naturalistic conversational patterns—especially considering the age-related
differences for older adults, and them being slow in adopting to new
technologies (Vaportzis et al., 2017; Czaja et al., 2006)—is crucial for
enhancing their user experience and ensuring effective communication.

Therefore, intent recognition should be human-aware and context-aware;
especially considering the expectations of older adults for VAs to be more
socially capable, which differ both from other user groups (Oh et al., 2020)
and within the older adult demographic itself (Harrington et al., 2022). For
instance, leveraging people’s social cues, verbal and vocal in this case, to
infer their implicit intent and creating personas to form a mental model of
users can improve the VA’s ability to accurately recognize their intent. Being
agnostic to users’ state—body language, audio-prosodic features and other
contextual information—results in voice assistants failing to appropriately
respond or react to older adults (Cuadra et al., 2022). Moreover, as evidenced
in our exploration of LLM-powered VAs (see section 4.4.3) and as discussed in
sections 5.2.2 and 5.3.2, utilizing LLMs contextual capabilities (retaining
task information, conversation history, and user personas, etc.) can improve
error management, particularly in adapting to flexible user speech
characteristics and preferences. Such integration of LLMs demonstrates a more
effective approach in deciphering user intent (Mahmood et al., 2023).

Limitation 6: Lack of VAs’ comprehension of naturalistic and diverse
conversational styles of older adults leads to intent recognition failures and
subpar user experience.

Design Consideration 8: Design VAs to be human-aware and context-aware to
detect implicit intent through verbal and vocal cues.

###  5.5. Limitations and Future Work

While our four-week study provided insights into the behavior of older adults
towards VAs, especially during interaction breakdowns, extended studies
spanning several months could provide deeper understanding. Longer duration
interactions could serve as a rich data source for training VAs to better
recognize and recover from errors. Currently, our analysis focuses
qualitatively on queries and responses to assess their potential utility.
However, a more detailed query analysis (Brewer et al., 2022) would offer
further insights into the nuances of older adults’ interactions with VAs. In
this study, we investigated a generic application of LLM-powered VAs. Future
research should consider specific use cases, such as healthcare information
retrieval or support tailored for older adults with disabilities and
impairments. Additionally, our participant sample is relatively small and
limited in scope such as location (one community center and local residents),
abilities (no disclosed memory problems), and ethnicity. Future studies should
aim for a larger and more diverse sample to encompass a wider range of
abilities and experiences.

##  6\. Conclusion

Traditional voice assistants often fail to effectively utilize immediate user
reactions, explicit user actions of error correction, and conversation history
in assisting users or recovering from errors themselves. In our qualitative
longitudinal study, we collected immediate user responses and reactions
through an audio recorder, demonstrating that such social cues can often lead
to identification of most errors. Additionally, we found that the need for
scripted commands and well-formed queries exacerbates conversational
breakdowns for older adults, attributed to their inaccurate mental model of VA
capabilities, forgetfulness of effective query styles, and a preference for
more naturalistic communication. We also explored the potential of integrating
an LLM into Alexa to enhance VAs’ conversational capabilities for older
adults. Our findings provide insights and considerations for the design and
research of nuanced interaction dynamics between older adults and futuristic
conversational voice assistants, emphasizing the value of “in-the-wild” data
collection to better comprehend such dynamics.

## Acknowledgements

This work was supported by the National Science Foundation award #1840088, the
Johns Hopkins University Institute for Assured Autonomy, and Malone Center for
Engineering in Healthcare.

## References

  * (1)
  * Angelini et al. (2021) Leonardo Angelini, Maurizio Caon, Emmanuel Michielan, Omar Abou Khaled, and Elena Mugellini. 2021.  Seniors’ perception of smart speakers: challenges and opportunities elicited in the Silver&Home Living Lab. In _Congress of the International Ergonomics Association_. Springer, 137–144.
  * Arnold et al. (2022) Anneliese Arnold, Stephanie Kolody, Aidan Comeau, and Antonio Miguel Cruz. 2022.  What does the literature say about the use of personal voice assistants in older adults? A scoping review.  _Disability and Rehabilitation: Assistive Technology_ (2022), 1–12.
  * Barros and Seabra (2020) Tiago Carneiro Gorgulho Mendes Barros and Rodrigo Duarte Seabra. 2020.  Usability assessment of google assistant and Siri virtual assistants focusing on elderly users. In _17th International Conference on Information Technology–New Generations (ITNG 2020)_. Springer, 653–657.
  * Blair and Abdullah (2019) Johnna Blair and Saeed Abdullah. 2019.  Understanding the needs and challenges of using conversational agents for deaf older adults. In _Conference Companion Publication of the 2019 On computer supported cooperative work and social computing_. 161–165.
  * Bolaños et al. (2020) Manuel Bolaños, César A Collazos, and Francisco L Gutiérrez. 2020.  Adapting a Virtual Assistant Device to Support the Interaction with Elderly People.. In _ICT4AWE_. 291–298.
  * Bowling et al. (2013) Ann Bowling, Matthew Hankins, Gill Windle, Claudio Bilotta, and Robert Grant. 2013.  A short measure of quality of life in older age: The performance of the brief Older People’s Quality of Life questionnaire (OPQOL-brief).  _Archives of gerontology and geriatrics_ 56, 1 (2013), 181–187.
  * Brewer et al. (2022) Robin Brewer, Casey Pierce, Pooja Upadhyay, and Leeseul Park. 2022.  An empirical study of older adult’s voice assistant use for health information seeking.  _ACM Transactions on Interactive Intelligent Systems (TiiS)_ 12, 2 (2022), 1–32.
  * Chan et al. (2023) Szeyi Chan, Jiachen Li, Bingsheng Yao, Amama Mahmood, Chien-Ming Huang, Holly Jimison, Elizabeth D Mynatt, and Dakuo Wang. 2023.  ” Mango Mango, How to Let The Lettuce Dry Without A Spinner?”: Exploring User Perceptions of Using An LLM-Based Conversational Assistant Toward Cooking Partner.  _arXiv preprint arXiv:2310.05853_ (2023).
  * Chen et al. (2021) Chen Chen, Janet G Johnson, Kemeberly Charles, Alice Lee, Ella T Lifset, Michael Hogarth, Alison A Moore, Emilia Farcas, and Nadir Weibel. 2021.  Understanding barriers and design opportunities to improve healthcare and QOL for older adults through voice assistants. In _Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility_. 1–16.
  * Chiaradia et al. (2019) Thiago Silva Chiaradia, Rodrigo Duarte Seabra, and Adriana Prest Mattedi. 2019.  Evaluating the usability of the Siri virtual assistant on mobile devices with emphasis on Brazilian elderly users. In _16th International Conference on Information Technology-New Generations (ITNG 2019)_. Springer, 437–441.
  * Choi et al. (2020) Yong K Choi, Hilaire J Thompson, and George Demiris. 2020.  Use of an internet-of-things smart home system for healthy aging in older adults in residential settings: Pilot feasibility study.  _JMIR aging_ 3, 2 (2020), e21964.
  * Chung et al. (2019) Kyungjin Chung, Young Hoon Oh, and Da Young Ju. 2019.  Elderly users’ interaction with conversational agent. In _Proceedings of the 7th International Conference on Human-Agent Interaction_. 277–279.
  * Cuadra et al. (2022) Andrea Cuadra, Hyein Baek, Deborah Estrin, Malte Jung, and Nicola Dell. 2022.  On Inclusion: Video Analysis of Older Adult Interactions with a Multi-Modal Voice Assistant in a Public Setting. In _Proceedings of the 2022 International Conference on Information and Communication Technologies and Development_. 1–17.
  * Czaja et al. (2006) Sara J Czaja, Neil Charness, Arthur D Fisk, Christopher Hertzog, Sankaran N Nair, Wendy A Rogers, and Joseph Sharit. 2006.  Factors predicting the use of technology: findings from the Center for Research and Education on Aging and Technology Enhancement (CREATE).  _Psychology and aging_ 21, 2 (2006), 333.
  * Harrington et al. (2022) Christina N Harrington, Radhika Garg, Amanda Woodward, and Dimitri Williams. 2022.  “It’s Kind of Like Code-Switching”: Black Older Adults’ Experiences with a Voice Assistant for Health Information Seeking. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_. 1–15.
  * Horstmann et al. (2023) Aike C Horstmann, Till Schubert, Lea Lambrich, and Clara Strathmann. 2023.  Alexa, I Do Not Want to Be Patronized: A Qualitative Interview Study to Explore Older Adults’ Attitudes Towards Intelligent Voice Assistants. In _Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents_. 1–10.
  * Hu et al. (2022) Yaxin Hu, Yuxiao Qu, Adam Maus, and Bilge Mutlu. 2022.  Polite or direct? Conversation design of a smart display for older adults based on politeness theory. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_. 1–15.
  * Hutchinson et al. (2003) Hilary Hutchinson, Wendy Mackay, Bo Westerlund, Benjamin B Bederson, Allison Druin, Catherine Plaisant, Michel Beaudouin-Lafon, Stéphane Conversy, Helen Evans, Heiko Hansen, et al. 2003\.  Technology probes: inspiring design for and with families. In _Proceedings of the SIGCHI conference on Human factors in computing systems_. 17–24.
  * Jesús-Azabal et al. (2020) Manuel Jesús-Azabal, Javier Rojo, Enrique Moguel, Daniel Flores-Martin, Javier Berrocal, José García-Alonso, and Juan M Murillo. 2020.  Voice assistant to remind pharmacologic treatment in elders. In _Gerontechnology: Second International Workshop, IWoG 2019, Cáceres, Spain, September 4–5, 2019, Revised Selected Papers 2_. Springer, 123–133.
  * Jo et al. (2023) Eunkyung Jo, Daniel A. Epstein, Hyunhoon Jung, and Young-Ho Kim. 2023.  Understanding the Benefits and Challenges of Deploying Conversational AI Leveraging Large Language Models for Public Health Intervention. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, Article 18, 16 pages.  https://doi.org/10.1145/3544548.3581503
  * Kakulla (2021) Brittne Kakulla. 2021.  _2021 tech trends and the 50+: Top 10 biggest trends_.  Technical Report. AARP, Washington, DC.
  * Kim and Choudhury (2021) Sunyoung Kim and Abhishek Choudhury. 2021.  Exploring older adults’ perception and use of smart speaker-based voice assistants: A longitudinal study.  _Computers in Human Behavior_ 124 (2021), 106914.
  * Koon et al. (2020) Lyndsie M Koon, Sean A McGlynn, Kenneth A Blocker, and Wendy A Rogers. 2020.  Perceptions of digital assistants from early adopters aged 55+.  _Ergonomics in Design_ 28, 1 (2020), 16–23.
  * Kowalski et al. (2019) Jarosław Kowalski, Anna Jaskulska, Kinga Skorupska, Katarzyna Abramczuk, Cezary Biele, Wiesław Kopeć, and Krzysztof Marasek. 2019.  Older adults and voice interaction: A pilot study with google home. In _Extended Abstracts of the 2019 CHI Conference on human factors in computing systems_. 1–6.
  * Lifestyle (2022) Senior Lifestyle. 2022.  Digital assistants for seniors.  https://www.seniorlifestyle.com/resources/blog/digital-assistants-for-seniors/.  Accessed: 2023-11-9.
  * Lis (2022) Jessica Lis. 2022.  US Voice Assistants and Smart Speakers Forecast 2022.  https://www.insiderintelligence.com/content/us-voice-assistants-smart-speakers-forecast-2022/.  Accessed: 2023-11-9.
  * Liu et al. (2020) Ying-Chieh Liu, Chien-Hung Chen, Yu-Sheng Lin, Hsin-Yun Chen, Denisa Irianti, Ting-Ni Jen, Jou-Yin Yeh, Sherry Yueh-Hsia Chiu, et al. 2020\.  Design and usability evaluation of mobile voice-added food reporting for elderly people: randomized controlled trial.  _JMIR mHealth and uHealth_ 8, 9 (2020), e20317.
  * Lopatovska et al. (2019) Irene Lopatovska, Katrina Rink, Ian Knight, Kieran Raines, Kevin Cosenza, Harriet Williams, Perachya Sorsche, David Hirsch, Qi Li, and Adrianna Martinez. 2019.  Talk to me: Exploring user interactions with the Amazon Alexa.  _Journal of Librarianship and Information Science_ 51, 4 (2019), 984–997.
  * Mahmood et al. (2022) Amama Mahmood, Jeanie W Fung, Isabel Won, and Chien-Ming Huang. 2022.  Owning mistakes sincerely: Strategies for mitigating AI errors. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_. 1–11.
  * Mahmood et al. (2023) Amama Mahmood, Junxiang Wang, Bingsheng Yao, Dakuo Wang, and Chien-Ming Huang. 2023.  LLM-Powered Conversational Voice Assistants: Interaction Patterns, Opportunities, Challenges, and Design Guidelines.  _arXiv preprint arXiv:2309.13879_ (2023).
  * Mortensen et al. (2006) Linda Mortensen, Antje S Meyer, and Glyn W Humphreys. 2006.  Age-related effects on speech production: A review.  _Language and Cognitive Processes_ 21, 1-3 (2006), 238–290.
  * Myers et al. (2018) Chelsea Myers, Anushay Furqan, Jessica Nebolsky, Karina Caro, and Jichen Zhu. 2018.  Patterns for how users overcome obstacles in voice user interfaces. In _Proceedings of the 2018 CHI conference on human factors in computing systems_. 1–7.
  * Nacimiento-Garcia et al. (2021) Eduardo Nacimiento-Garcia, Carina Gonzalez-González, Daniel Domínguez-Gutiérrez, and Francisco Gutiérrez-Vela. 2021.  Pervasive gaming experiences to promote active aging using the virtual voice assistant Alexa. In _Ninth International Conference on Technological Ecosystems for Enhancing Multiculturality (TEEM’21)_. 189–194.
  * O’Brien et al. (2020) Katherine O’Brien, Anna Liggett, Vanessa Ramirez-Zohfeld, Priya Sunkara, and Lee A Lindquist. 2020.  Voice-controlled intelligent personal assistants to support aging in place.  _Journal of the American Geriatrics Society_ 68, 1 (2020), 176–179.
  * Oewel et al. (2023) Bruna Oewel, Tawfiq Ammari, and Robin N Brewer. 2023.  Voice Assistant Use in Long-Term Care. In _Proceedings of the 5th International Conference on Conversational User Interfaces_. 1–10.
  * Oh et al. (2020) Young Hoon Oh, Kyungjin Chung, and Da Young Ju. 2020.  Differences in interactions with a conversational agent.  _International journal of environmental research and public health_ 17, 9 (2020), 3189.
  * OpenAI (2023) OpenAI. 2023.  ChatGPT can now see, hear, and speak.  https://openai.com/blog/chatgpt-can-now-see-hear-and-speak
  * Pearl (2016) Cathy Pearl. 2016.  _Designing voice user interfaces: Principles of conversational experiences_.  ” O’Reilly Media, Inc.”.
  * Porcheron et al. (2018) Martin Porcheron, Joel E Fischer, Stuart Reeves, and Sarah Sharples. 2018.  Voice interfaces in everyday life. In _proceedings of the 2018 CHI conference on human factors in computing systems_. 1–12.
  * Pradhan et al. (2019) Alisha Pradhan, Leah Findlater, and Amanda Lazar. 2019.  ” Phantom Friend” or” Just a Box with Information” Personification and Ontological Categorization of Smart Speaker-based Voice Assistants by Older Adults.  _Proceedings of the ACM on Human-Computer Interaction_ 3, CSCW (2019), 1–21.
  * Pradhan et al. (2020) Alisha Pradhan, Amanda Lazar, and Leah Findlater. 2020.  Use of intelligent voice assistants by older adults with low technology use.  _ACM Transactions on Computer-Human Interaction (TOCHI)_ 27, 4 (2020), 1–27.
  * Pradhan et al. (2018) Alisha Pradhan, Kanika Mehta, and Leah Findlater. 2018.  ” Accessibility Came by Accident” Use of Voice-Controlled Intelligent Personal Assistants by People with Disabilities. In _Proceedings of the 2018 CHI Conference on human factors in computing systems_. 1–13.
  * Purao et al. (2021) Sandeep Purao, Haijing Hao, and Chenhang Meng. 2021.  The use of smart home speakers by the elderly: exploratory analyses and potential for big data.  _Big Data Research_ 25 (2021), 100224.
  * Purao and Meng (2019) Sandeep Purao and Chenhang Meng. 2019.  Data capture and analyses from conversational devices in the homes of the elderly. In _Advances in Conceptual Modeling: ER 2019 Workshops FAIR, MREBA, EmpER, MoBiD, OntoCom, and ER Doctoral Symposium Papers, Salvador, Brazil, November 4–7, 2019, Proceedings 38_. Springer, 157–166.
  * Rausch (2023) Daniel Rausch. 2023.  Previewing the future of Alexa.  https://www.aboutamazon.com/news/devices/amazon-alexa-generative-ai