:
LEAVE UNSET 2023 LEAVE UNSET LEAVE UNSET Machine Learning for Health (ML4H)
2023

# MemoryCompanion: A Smart Healthcare Solution to Empower Efficient
Alzheimer’s Care Via Unleashing Generative AI

Lifei Zheng<EMAIL_ADDRESS>
New York University Abu Dhabi These authors contributed equally UAE  Yeonie
Heo11footnotemark: 1<EMAIL_ADDRESS>
New York University Abu Dhabi  UAE  Yi Fang<EMAIL_ADDRESS>
New York University Abu Dhabi  UAE

###### Abstract

With the rise of Large Language Models (LLMs), notably characterized by GPT
frameworks, there emerges a catalyst for novel healthcare applications.
Earlier iterations of chatbot caregivers, though existent, have yet to achieve
a dimension of human-like authenticity. This paper unveils ‘MemoryCompanion’ a
pioneering digital health solution explicitly tailored for Alzheimer’s disease
(AD) patients and their caregivers. Drawing upon the nuances of GPT technology
and prompt engineering, MemoryCompanion manifests a personalized caregiving
paradigm, fostering interactions via voice-cloning and talking-face mechanisms
that resonate with the familiarity of known companions. Using advanced prompt-
engineering, the system intricately adapts to each patient’s distinct profile,
curating its content and communication style accordingly. This approach
strives to counteract prevalent issues of social isolation and loneliness
frequently observed in AD demographics. Our methodology, grounded in its
innovative design, addresses both the caregiving and technological challenges
intrinsic to this domain.

###### keywords:

Smart Caregiving, Generative AI, Large Language Models, Digital Health
Solutions, Alzheimer’s Disease

Figure 1: MemoryCompanion’s Overview. The Verbal Interaction Module enables
real-time vocal communication between AD patients and a GPT model, improving
efficiency for those with cognitive deficits. The transcribed input of the
patient is then processed into the tailored AD-Patient-GPT Language Model that
supports in personalized conversations and reminiscence therapy, enhancing
cognitive function and daily communication for AD patients in GPT-Powered
Intelligent Module. Lastly, Affective Face-Driven Dialogue Module uses
emotion-sensitive talking-face interface for authentic human-machine
interaction, reducing feelings of loneliness and social isolation. Figure 1 is
generated by Midjourney (2023).

##  1 Introduction

Alzheimer’s disease (AD) is the foremost neurodegenerative disorder
Association (2023b); of Environmental Health Sciences (2022), impacting 1 in 9
individuals (10.8%) aged 65 or older Picanco et al. (2018), with global
prevalence projected to increase by 62% by 2030, reaching approximately 47
million patients Rajan et al. (2021). Amidst this growing concern, present
treatment options are restricted Ralph and Espinet (2018). Given the increased
risk of stroke and death associated with antipsychotics in AD patients Maust
et al. (2015), non-pharmacological approaches are prioritized to emphasize
activities and social interactions for enhancing cognitive function and
decelerating memory loss progression Organization (2023); of Medicine (2017).
However, the provision of continuous interaction and companionship for AD
patients encounters serious hurdles of limited caregiver availability van den
Kieboom et al. (2020), isolation, and daily socialization challenges
Association (2023a).

To alleviate the demanding physical and mental strains faced by caregivers, we
introduce MemoryCompanion, a digital healthcare solution leveraging Generative
Pre-trained Transformer (GPT) technology OpenAI (2021). GPT serves as the core
technology behind ChatGPT, enabling MemoryCompanion to be a continuous
companion for AD patients. MemoryCompanion integrates GPT with multimedia
technologies, including speech-to-text transcription Hannun et al. (2014),
voice cloning Coqui.ai (2023), and talking-face Ma et al. (2023), to establish
an efficient caregiving system for AD patients. The integration of voice
cloning and talking-face technologies within GPT responses offers AD patients
the sensation of communicating with familiar individuals at their convenience.
As a robust caregiving alternative, MemoryCompanion consistently delivers
care, mitigating the challenges encountered by human caregivers Ohno et al.
(2021); Koskie (2018). MemoryCompanion’s scope extends beyond AD, yielding
benefits across diverse health and educational domains, thus generating a
broader social impact.

Figure 1 presents the architectural blueprint of MemoryCompanion, focusing on
three integral modules: the Verbal Interaction Module, the GPT-Powered
Intelligent Module, and the Affective Face-Driven Dialogue Module. Together,
these components facilitate a unified and user-friendly experience for AD
patients. In this study, our contributions are delineated in three pivotal
aspects, each of which highlights the significant advancement brought forth by
MemoryCompanion:

Patient-Centric Language Learning Model We introduce an innovative patient-
centric Language Learning Model (LLM) designed for engaging AD patients in not
only day-to-day conversations and offering emotional support but assisting in
memory retrieval based on their personal experiences. This approach
contributes to personalized healthcare technologies, specifically engineered
to tailor the distinctive requirements of AD patients, thereby significantly
enhancing the quality of care provided.

Structured AD Patient Profile Paradigm To tackle the issues of limited data
availability and data sensitivity in smart health solutions for AD patients,
we create a well-structured AD patient profile paradigm. Simultaneously, we
construct a synthetic dataset, serving as a foundational resource for future
research endeavors. This initiative fulfills the imperative for extensive data
in this domain while fostering future developments in AD patient support
technologies and research.

AD Caregiving Evaluation Metrics We present a tailored set of metrics for
evaluating the effectiveness of smart caregiving systems for AD patients.
These metrics establish a rigorous benchmark for future research, facilitating
systematic assessments of system reliability and effectiveness. This
contribution establishes a standardized framework for advancing smart
caregiving technologies, thereby enhancing the quality of care and support for
AD patients and their caregivers.

Figure 2: Methodology Overview. The four color-highlighted boxes represent the
principal modules within our pipeline. Detailed descriptions of each module
can be found in Modules 3.1 Real-Time Speech-to-text Recognition, 3.2 Patient-
Centric MemoryCompanion Model Fine-Tuning, 3.3 Text-to-Speech Synthesization,
and 3.4 Talking Face Construction.

##  2 Related Work

Many smart health solutions have been developed to address the complex
caregiving needs of patients with AD and on a broader scale, dementia. Prior
studies were mostly developing applications that engaged in the domain of
patient monitoring, health care service support, and patient education. Since
2012, Robotics-based applications have been investigated for assisting AD
patients in tasks such as food preparation and consumption Derek et al.
(2012), and participating in recreational activities within care homes Louie
et al. (2015). These robotic innovations have demonstrated considerable
potential in mitigating psychological distress among AD patients and their
caregivers by reducing agitation, depression, anxiety, impulsiveness,
restlessness, and chronic pain through “robotherapy” Bhargava and Baths
(2022). However, there was a noticeable paucity of applications concerning
caregiving skills and activities due to constraints in artificial intelligence
and achieving an authentic human-like interaction Team (2023).

The introduction of Large Language Models (LLMs) has catalyzed research to
overcome these limitations and enhance the human-like interactivity of these
systems. Studies exploring the use of LLM-based chatbots, taking into account
user personality profiles and emotional states to support dementia caregivers,
have become increasingly prevalent Nasiri (2023). Similar studies, for
example, MultiMedQA, presented in Nature in 2023, propose a large language
model encoded with clinical knowledge trained through prompt engineering
strategies and clinically relevant datasets Singhal et al. (2023).

Yet, a gap in the literature remains with no reported solutions integrating
both LLM and visual or audio components to create a fully immersive caregiving
environment for AD patients. In response to this gap, we propose
MemoryCompanion, a revolutionary autonomous technology that combines GPT-based
models with speech-to-text transcription, voice-cloning, and talking-face
technologies. This innovative approach represents a significant leap forward
in the domain of AD patient care, offering a more genuine, human-like
interaction that not only improves the quality of life but also mitigates the
sense of social isolation.

##  3 Methodology

The principal purpose of MemoryCompanion is to develop a digital GPT agent
that accurately replicates the user’s beloved one’s visage, tonal nuances, and
linguistic style in order to prompt a natural conversation flow and maintain
high authenticity in human-computer interaction. To achieve this objective,
MemoryCompanion comprises four distinct components. The overview of the
methodology, as illustrated in Figure 2, illustrates the overall architecture
of MemoryCompanion, with sections 3.1, 3.2, 3.3, and 3.4 indicating each
module. Each module and its submodules is explained in detail below.

###  3.1 Real-Time Speech-to-text Recognition

To enable direct user-chatbot interaction, Module 3.1 Real-Time Speech-to-text
Recognition is developed to transcribe the speech from AD patients into text.
This module processes an audio stream as input and transforms it into a
sequence of words and punctuations. The transcribed text is then forwarded to
Module 3.2, a fine-tuned GPT Language model, which allows for smooth text-
response processing. Our real-time speech recognition module is implemented
using an approach similar to that described in the DeepSpeech paper Hannun et
al. (2014).

###  3.2 Patient-Centric MemoryCompanion Model Fine-Tuning

Figure 3: Patient-Centric LLM Fine-Tuning Diagram

Upon receiving the transcribed speech from Module 3.1, the subsequent stage
entails processing the text and generating responses using our novel AD-
patient-centric large language model. Our objective is to equip
MemoryCompanion with ChatGPT-level OpenAI (2021) language comprehension and
processing capabilities, specifically fine-tuned to interact effectively with
AD patients within a healthcare context.

To meet these goals of advanced language comprehensive and customized
responses for AD patients, we develop a new MemoryCompanion Generative
Language Model. This is achieved by fine-tuning the GPT-3.5 Turbo model OpenAI
(2021) using domain-specific patient cases data. Through this fine-tuning
process, we transition from a general-purpose GPT model to a highly
specialized one, thereby enabling MemoryCompanion to comprehend the AD
patients and generate text that is closely aligned with the unique
requirements of these patients Solulab (2023). As shown in Figure 3, the fine-
tuning process is described as:

GPT Model Training The GPT-3.5 Turbo model is pretrained on a varied corpus.
During this phase, the model’s parameters, $\theta$, are adjusted to optimize
the log-likelihood of the observed data, as denoted by:

$\theta^{*}=\arg\max_{\theta}\sum_{i=1}^{N}\log P(y_{i}|x_{i},\theta)$ (1)

In this equation, $x_{i}$ represents the input sequence, while $y_{i}$
signifies the target output distribution. Given an input sequence, the model’s
response is generated by randomly sampling from this target distribution. This
optimization seeks to find the configuration of $\theta^{*}$ that makes the
model’s predictions most consistent with the observed data. Upon obtaining the
optimal parameters $\theta^{*}$, as described in Equation 1 and part (2) in
Figure 4, the model can generate responses by sampling from a probability
distribution conditioned on a given query $X_{\text{query}}$:

$P(y|X=X_{\text{query}},\theta^{*})$ (2)

Contextual Refinement for AD Patients Despite the parameter optimization in
the general GPT model in Equation 2, the resulting probability distribution
may lack the requisite specificity for effective interactions with AD
patients. To ensure tailored interactions, we implement prompt engineering,
concatenating the query $X_{\text{query}}$ with the patient profile
$X_{\text{patient}}$ as input sequence X, as visualized in Equation 3 and part
(3) in Figure 4:

$P(y|X=X_{\text{query}}+X_{\text{patient}},\theta^{*})$ (3)

Figure 4: GPT vs Patient-centric GPT Figure 5: Talking Face Construction
Diagram. The left side presents our talking face technology synthesization
diagram. The right talking face figure presents a screenshot of our
synthesized MemoryCompanion, designated as ‘Sam’, delivering the output
response from the patient-centric LLM model in an emotion-oriented and
interactive way through our advanced talking-face multimedia technology. For a
comprehensive visual representation of additional responses from
MemoryCompanion, please refer to the supplementary videos in Appendix A. ‘Sam’
is synthesized based on an image generated from Midjourney (2023).

This concatentation refines responses by leveraging the pretrained GPT model’s
knowledge and tailoring it to the specific conditions and queries of each AD
patient. This process generates a more precise conditional probability
distribution, reducing variance $\sigma^{2}$. This variance reduction enhances
system efficacy, yielding focused, accurate, and contextually aware responses.
The conditional probability distribution minimizes erratic or irrelevant
outputs, critical in maintaining clarity and relevance with AD patients, while
sharpening the model’s precision, aligning output closely with the specific
queries and conditions presented. This enhancement allows for accurate,
contextually aware, and personalized responses that address the nuanced needs
of AD patients. Notably, the efficacy of this approach is directly correlated
with the diversity and granularity of the patient profile dataset. A more
comprehensive dataset leads to a more refined and AD-patient-centric set of
likely responses, underlying the significance of detailed patient profiles in
this endeavor.

###  3.3 Text-to-Speech Synthesization

To enhance user interaction, it’s essential to convert text responses from the
language model in Module 3.2 Patient-Centric MemoryCompanion Model Fine-Tuning
into authentic speech. Module 3.3 Text-to-Speech Synthesization Module, refer
to Figure 2, achieves this by utilizing Text-to-Speech (TTS) technology
Coqui.ai (2023) which is designed expressly to transform text into speech. A
standout feature is voice cloning, which mimics the conversational speech
patterns of the beloved one chosen by the user, thus elevating the user
experience. This voice cloning comprises two critical phases:

Speaker Voice Reference Module This phase focuses on collecting targeted voice
data to train a model that captures unique auditory characteristics of
specific speakers, representing the characteristics as an embedding vector.

Speech Synthesizer Module In this phase, we develop a synthesizer. Integrated
with the embedding vector from last phase, it accurately translates textual
input to speech. The paired text-audio datasets guarantee the synthesis of
exceptional fidelity, which finally output a synthesized speech that will be
used as input for Module 3.4 Talking Face Construction.

###  3.4 Talking Face Construction

Given the synthesized speech in Module 3.3, the integration of a visually
expressive talking face with emotional nuances becomes essential for enhancing
user experience authenticity, as shown in Figure 5. Existing studies emphasize
the importance of nonverbal cues and facial expressions in effective emotional
communication Mehrabian (1971); Argyle et al. (1971). Observing a human figure
with synchronized facial features bolsters communication effectiveness by
reinforcing verbal discourse and tonality. Our methodology is inspired by
StyleTalk Ma et al. (2023). Our emotionally enriched talking face comprises
two phases, as shown in Figure 5.

Talking Face Audio & Image Feature Extraction Synthesized speech, referring
Figure 5, is transformed into word embeddings for vocal generation, while
facial attributes from the user’s input image, presented in Figure 5, are
distilled into a vector. Both data forms merge through a self-attention
pooling layer, resulting in the final style code.

Talking Face Synthesization Utilizing style data from last phase, a standard
transformer produces accurate lip syncing and authentic expressions. A talking
face transformer, referring Figure 5, then adjusts the network, ensuring a
talking face with well-coordinated emotional expressions and speaking style,
enhancing overall user experience.

Figure 6: Chat History of Linda with MemoryCompanion and ChatGPT

##  4 Experiment & Evaluation

In the experiement section, we meticulously prepare the data our
MemoryCompanion’s ability to respond to patient queries, both qualitatively
and quantitatively. After generating a synthetic patient case, Linda Williams
in Figure 6, we qualitatively evaluate MemoryCompanion in comparison to
OpenAI’s ChatGPT 4.0, a general GPT model to highlight the unique
contributions of our patient-centric approach. Next, various evaluation
critera are employed to provide quantitative understanding of our system’s
performance for 100 synthetic patient cases.

###  4.1 Data Preparation

The key to fine-tuning an AD patient language model lies in applying patient
profile data. Recognizing the sensitivity and confidentiality of patient data,
we have been cautious in our data collection. We sourced a dataset of profiles
of AD patients from Beattie and Mackenzie (2005); Yoo et al. (2017); Honig and
Chin (2001). Due to the limited availability of real patient data, we generate
10,000 synthetic patient cases whose structure follows a sourced dataset of
profiles of AD patients from Beattie and Mackenzie (2005); Yoo et al. (2017);
Honig and Chin (2001). The synthetic cases generation process consists of two
phases.

Patient Profile Case Structure Definition The case structure starts with
demographics like age and ethnicity, it transitions into medical history,
familial background, and professional and personal pursuits. Notably, it
delves into the patient’s current challenges, future aspirations, and
emotional well-being. This structured profile provides a holistic
understanding of AD patients, amalgamating both quantitative and qualitative
facets. It is imperative for ensuring nuanced and tailored interaction with AD
patients. A detailed patient profile example can be found in Appendix B.

Synthetic Cases Data Generation After establishing the case structure
framework, the subsequent step involves leveraging the ChatGPT-4 model OpenAI
(2023) to synthesize patient cases conforming to the specified structure.
Utilizing the OpenAI ChatGPT platform, we fed this pre-defined case structure
into the GPT-4 model, and instruct GPT model to comprehend the intricacies of
the provided case structure and to generate 10,000 high-fidelity, available in
Appendix B, realistic synthetic patient cases that are analogous to the
original dataset Beattie and Mackenzie (2005); Yoo et al. (2017); Honig and
Chin (2001).

###  4.2 Qualitative Evaluation

When evaluating interactions with AD patients, the tailored, patient-specific
approach of MemoryCompanion in Figure 6 presents distinct advantages over
ChatGPT in three aspects in their responses.

Ability to Initialize Conversation Initiating dialogues fosters patient
engagement and eases their cognitive load in sustaining interactions.
MemoryCompanion proactively starts dialogues, evident from its greeting “How
are you?” to Linda in Figure 6. In contrast, ChatGPT tends to reactively await
patient input, potentially making it harder for AD patients to maintain
conversations.

Provision of accurate and personalized information Providing specific,
accurate information is important for AD patients, ensuring clarity and
reducing confusion. MemoryCompanion effectively offers precise details about
Linda’s son, his residence and occupation, leveraging patient-specific data,
as seen in Figure 6. Conversely, ChatGPT, without this granularity,
generalizes its advice, potentially complicating matters for AD patients.

Tailored responses for personalized interaction Beyond accuracy, personalized
recommendations enhance the quality of care, addressing both emotional and
cognitive needs of AD patients. In Figure 6, when Linda seeks suggestions
about interacting with her son, MemoryCompanion integrates her interests, like
baking, into its advice. This level of customization deepens the patient’s
connection and trust in the system’s caregiving capabilities.

###  4.3 Quantitative Evaluation

Quantitative evaluation of MemoryCompanion is pivotal to validate its efficacy
across diverse patient demographics. This ensures both broad applicability and
tailored responsiveness in Alzheimer’s care. Using criteria such as Accuracy,
Ability To Lead Casual Conversation, Empathy & Tone, and Error Handling, we
examined its performance across varied, realistic patient interactions. From a
dataset of 10,000 synthetic patient instances, we randomly selected 100 cases.
Each was subjected to a question-answering process using nine tailored
questions, according to Table 1, designed to mirror real-world interactions
MemoryCompanion might face, especially with AD patients. Responses were
evaluated on a scale from 1 (ineffective) to 5 (exceptional).

| Evaluation
---
Criteria
Questions
Accuracy | Q1 | What is my name?
Q2 |  | How is my family? Can
---
you talk a bit about them?
Q3 | Where do I live?
Ability to Lead Casual Conversations | Q4 | How are you today?
Q5 | What should I do today?
Q6 |  | How was the weather
---
today?
Empathy & Tone | Q7 |  | My friend passed away.
---
What should I do?
Q8 |  | I missed my family
---
members so much.
Error Handling | Q9 |  | Tell me about my time as
---
an astronaut?
Table 1: Evaluation Criteria Questions Figure 7: Average Ratings of Questions

####  4.3.1 Accuracy

Accuracy is essential to MemoryCompanion, ensuring dependable information for
AD patients. In Figure 7, MemoryCompanion consistently delivers accurate
details about the patient’s name, family, and residence. Variability in
location-related responses remains acceptable, given the GPT model’s
complexities. However, while accurate, MemoryCompanion could enhance user
experience by promoting more engaging continuity. For instance, asking about a
patient’s past, the origin of their name, or family activities might foster a
more dynamic interaction, aligning with AD patients’ and caregivers’ needs.

####  4.3.2 Ability to lead casual conversations

This criterion evaluates MemoryCompanion’s friendliness and its capability to
create a natural and comforting interaction environment for AD patients. The
ability to initiate and sustain casual conversations is of significant
importance, as it contributes to the overall user experience and emotional
well-being of the patients. As depicted in Figure 7, MemoryCompanion offers
warm greetings and recommends enjoyable activities, fostering a reassuring
atmosphere. However, it occasionally falls short of achieving a completely
human-like interaction, which is crucial for emulating conversations with
familiar individuals. Importantly, disclosing its AI nature can disrupt the
intended illusion. To ensure an optimal user experience, responses should
consistently align with the expected personality of familiar individuals

####  4.3.3 Empathy & Tone

Empathy and tone are essential criteria, particularly in the context of AD
patients who may be dealing with sensitive and emotional challenges.
MemoryCompanion’s ability to demonstrate empathy and a supportive tone is
crucial for providing emotional support and encouragement. This criterion
ensures that the system can effectively acknowledge the feelings and address
the needs of AD patients, enhancing their overall well-being and satisfaction.
According to Figure 7, MemoryCompanion excels in demonstrating empathy and a
supportive tone in responses to sensitive and emotional questions. It
consistently provides compassionate and comforting guidance to AD patients,
acknowledging their feelings and offering valuable suggestions for coping with
emotional challenges.

####  4.3.4 Error Handling

Error handling is a pivotal criterion as it assesses MemoryCompanion’s
capacity to manage incorrect or out-of-scope questions. This capability is
essential for maintaining a coherent and meaningful conversation with AD
patients, who may occasionally make memory-related errors or ask questions
beyond the system’s scope. As seen from Figure 7, MemoryCompanion demonstrates
exceptional error handling in response to the out-of-scope question about
being an astronaut. It accurately recognizes the error in the patient’s memory
and provides gentle correction while offering meaningful details about the
patient’s actual past as another professional occupation.

##  5 Limitation and Future Work

While MemoryCompanion offers a groundbreaking approach to supporting AD
patients, several crucial concerns arise:

Ethical Dilemma MemoryCompanion integrates familiar faces and tonal nuances of
patients’ loved ones, raising ethical questions about how patients perceive
such intimate simulations. The emotional impact is profound, requiring a
delicate balance between comfort and the risk of emotional dependency or
memory distortion. Comprehensive ethical evaluations in collaboration with
experts are essential to assess these design choices. A framework that
prioritizes patients’ emotional and cognitive well-being, ensuring the tool
aids rather than distorts their genuine memories, is imperative.

Data Equilibrium The efficacy of our patient-centric model relies on data,
creating a tension between improving MemoryCompanion’s performance through
more data and potential privacy breaches from data over-reliance. Striking an
optimal balance is crucial. Exploring a phased data acquisition strategy with
robust encryption and anonymization protocols is essential. Implementing a
dynamic consent framework, allowing patients or guardians to control data
sharing, can enhance protection.

Authenticity of Representation While current video-based interactions are
innovative, there’s room for improvement as demands for more immersive mediums
grow. Future iterations may explore augmented reality (AR) or holographic
representations for a lifelike and dynamic interaction. Collaboration with
multimedia technology experts can guide the development of immersive patient
experiences.

Resolving these limitations requires an approach centered around the patient,
one that seamlessly incorporates ethical considerations and harnesses the
potential of cutting-edge technological advancements.

## References

  * Argyle et al. (1971) Michael Argyle, Florisse Alkema, and Robin Gilmour.  The communication of friendly and hostile attitudes by verbal and non-verbal signals.  _American Psychological Association_ , 1971.  URL https://psycnet.apa.org/record/1973-04431-000.
  * Association (2023a) Alzheimer’s Association.  Caregiver stress, 2023a.  URL https://www.alz.org/help-support/caregiving/caregiver-health/caregiver-stress.  Accessed 12 May 2023.
  * Association (2023b) Alzheimer’s Association.  2023 alzheimer’s disease facts and figures.  2023b.  URL https://www.alz.org/media/Documents/alzheimers-facts-and-figures.pdf.
  * Beattie and Mackenzie (2005) B. Lynn Beattie and Ian R.A. Mackenzie.  Dementia diagnosis: Case presentations and neuropathology.  _BC Medical Journal_ , 2005.
  * Bhargava and Baths (2022) Yesoda Bhargava and Veeky Baths.  Technology for dementia care: benefits, opportunities and concerns.  _Journal of Global Health Reports_ , 2022.  URL https://www.joghr.org/article/39606-technology-for-dementia-care-benefits-opportunities-and-concerns.
  * Coqui.ai (2023) Coqui.ai.  Tts: Deep learning based text-to-speech, 2023.  URL https://github.com/coqui-ai/TTS.  GitHub repository.
  * Derek et al. (2012) McColl Derek, Jeanie Chan, and Goldie Nejat.  A socially assistive robot for meal-time cognitive interventions.  _The American Society of Mechanical Engineers_ , 2012.
  * Hannun et al. (2014) Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, and Andrew Y. Ng.  Deep speech: Scaling up end-to-end speech recognition.  _Cornell University_ , 2014.  URL https://arxiv.org/abs/1412.5567.
  * Honig and Chin (2001) Lawrence S. Honig and Steven S. Chin.  Alzheimer’s disease: A case study.  _Science_ , 2001.  URL https://www.science.org/doi/10.1126/sageke.2001.1.dn2.
  * Koskie (2018) Brandi Koskie.  The state of caregiving for alzheimer’s and related dementia 2018, 2018\.  URL https://www.healthline.com/health/state-of-alzheimers#key-findings.
  * Louie et al. (2015) Wing-Yue G. Louie, Jacob Li, Chris Mohamed, Francis Despond, Vincent Lee, and Goldie Nejat.  Tangy the robot bingo facilitator: A performance review.  _J. Med. Devices_ , 2015.
  * Ma et al. (2023) Yifeng Ma, Suzhen Wang, Zhipeng Hu, Changjie Fan, Tangjie Lv, Yu Ding, Zhidong Deng, and Xin Yu.  Styletalk: One-shot talking head generation with controllable speaking styles.  _AAAI2023_ , 2023.  URL https://arxiv.org/abs/2301.01081.
  * Maust et al. (2015) Donovan T Maust, Hyungjin Myra Kim, Lisa S Seyfried, Claire Chiang, Janet Kavanagh, Lon S Schneider, and Helen C Kales.  Antipsychotics, other psychotropics, and the risk of death in patients with dementia: Number needed to harm, May 2015.  URL https://pubmed.ncbi.nlm.nih.gov/25786075/.
  * Mehrabian (1971) Albert Mehrabian.  Silent messages.  _American Psychological Association_ , 1971.  URL https://psycnet.apa.org/record/1973-04431-000.
  * Midjourney (2023) Midjourney.  Photograph credits.  Image produced by Midjourney., 2023.
  * Nasiri (2023) Yeganeh Nasiri.  A language-model-based chatbot that considers the user’s personality profile and emotions to support caregivers of people with dementia.  2023\.  URL https://scholarsarchive.byu.edu/etd/9862/.
  * of Environmental Health Sciences (2022) National Institute of Environmental Health Sciences.  Neurodegenerative diseases, 2022.  URL https://www.niehs.nih.gov/research/supported/health/neurodegenerative/index.cfm.
  * of Medicine (2017) National Library of Medicine.  Non-drug interventions for alzheimer’s disease, July 2017.  URL https://www.ncbi.nlm.nih.gov/books/NBK279355/.
  * Ohno et al. (2021) Shinya Ohno et al.  Humanistic burden among caregivers of patients with alzheimer’s disease or dementia in japan: A large-scale cross-sectional survey.  _Journal of Medical Economics_ , 2021.  URL https://pubmed.ncbi.nlm.nih.gov/33467967/.
  * OpenAI (2021) OpenAI.  Introducing the openai api, 2021.  URL https://openai.com/blog/openai-api.
  * OpenAI (2023) OpenAI.  Openai chat, 2023.  URL https://chat.openai.com/.
  * Organization (2023) World Health Organization.  Dementia.  https://www.who.int/news-room/fact-sheets/detail/dementia, March 2023.  Accessed: 15 Mar 2023.
  * Picanco et al. (2018) Leide Caroline Dos Santos Picanco, Priscilla F Ozela, Maiara de Fatima de Brito Brito, Abraao A Pinheiro, Elias C Padilha, Francinaldo S Braga, Carlos Henrique Tomich de Paula da Silva, Cleydson Breno Rodrigues Dos Santos, Joaquín M C Rosa, and Lorane Izabel da Silva Hage-Melim.  Alzheimer’s disease: A review from the pathophysiology to diagnosis, new perspectives for pharmacological treatment, 2018.  URL https://pubmed.ncbi.nlm.nih.gov/30191777/.
  * Rajan et al. (2021) Kumar Rajan, Jennifer Weuve, Lisa L. Barnes, Elizabeth A. McAninch, Robert S. Wilson, and Denis A. Evans.  Population estimate of people with clinical alzheimer’s disease and mild cognitive impairment in the united states (2020-2060).  _Alzheimer’s & Dementia: The Journal of the Alzheimer’s Association_, December 2021.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9013315/.
  * Ralph and Espinet (2018) Stephen J Ralph and Anthony J Espinet.  Increased all-cause mortality by antipsychotic drugs: Updated review and meta-analysis in dementia and general mental health care.  _Journal of Alzheimer’s Disease Reports_ , 2018.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6159703/.
  * Singhal et al. (2023) Karan Singhal, Shekoofeh Azizi, and Vivek Natarajan.  Large language models encode clinical knowledge.  _Nature_ , 2023.  URL https://www.nature.com/articles/s41586-023-06291-2.
  * Solulab (2023) Solulab.  How to fine-tune gpt models for specific niche domains, 2023.  URL https://www.linkedin.com/pulse/how-fine-tune-gpt-models-specific-niche-domains-solulab/.  Accessed: 2023-09-06.
  * Team (2023) Vicuna Team.  Vicuna: An open-source chatbot impressing gpt-4 with 90quality, 2023.  URL https://lmsys.org/blog/2023-03-30-vicuna/.
  * unsplash (2023) unsplash.  Image credits.  https://unsplash.com/, 2023.
  * van den Kieboom et al. (2020) Robin van den Kieboom, Liselore Snaphaan, Ruth Mark, and Inge Bongers.  The trajectory of caregiver burden and risk factors in dementia progression: A systematic review.  _Journal of Alzheimer’s Disease: JAD_ , 2020.  URL https://pubmed.ncbi.nlm.nih.gov/32804093/.
  * Yoo et al. (2017) Hyun Um Yoo, Woo Hee Choi, Won Sang Jung, Young Ha Park, Chang-Uk Lee, and Hyun Kook Lim.  A case report of a 37-year-old alzheimer’s disease patient with prominent striatum amyloid retention.  _National Library of Medicine_ , 2017.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5561413/.

##  Appendix A MemoryCompanion Experiments Supplementary Video

In this section, we present three experiments of MemoryCompanion across three
distinct AD patients with different conditions. We provide concise
introductions for each of the three AD patients as follows. The three provided
instances exemplify the output of the MemoryCompanion pipeline for patients
with varied demographic profiles. Comprehensive experiments are accessible
within our pipeline repository on GitHub in Appendix C.

Margaret Thompson: A 92-year-old woman who dedicated her life to teaching. The
MemoryCompanion tailored for Margaret incorporates the image of her favorite
former student from her teaching career, Amanda. A visual representation of
their interaction through the talking face interface is depicted in Figure 8.
The full conversation in video between Margaret and MemoryCompanion (Amanda)
can be found here.

David Johnson: A 57-year-old who established himself as an entrepreneur.
David’s MemoryCompanion features his wife, Eva, as the conversational partner.
A visual representation of their interaction through the talking face
interface is depicted in Figure 8. The full conversation in video between
David and MemoryCompanion (Eva) can be found here.

Emily Anderson: A 24-year-old woman who recently completed her university
education. Emily’s MemoryCompanion features her closed friend, Julie. A visual
representation of their interaction through the talking face interface is
depicted in Figure 8. The full conversation in video between Emily and
MemoryCompanion (Julie) can be found here.

Figure 8: MemoryCompanion Talking Interface. In the order from left to right,
first is Margaret’s MemoryCompanion (Amanda), second is David’s
MemoryCompanion (Eva), and third is Emily’s MemoryCompanion (Julie). These
talking face interfaces are sourced from unsplash (2023).

##  Appendix B Patient Synthetic Cases Dataset

To foster transparency, reproducibility, and to offer a comprehensive dataset
for further research, we are releasing our 10,000 synthetic patient profiles
utilized in the MemoryCompanion research project. This dataset has been
designed to emulate realistic patient profiles while ensuring no real patient
data or personally identifiable information is included. It serves as a
valuable resource for researchers and developers aiming to test, validate, or
extend methods in related domains. We encourage researchers and practitioners
to leverage this dataset for their experiments, ensuring that findings in this
domain can be benchmarked and compared against a consistent data backdrop.
This dataset is created by the following phases:

###  B.1 Collected Patient Profile Data

We initially gathered genuine patient profile data from public resources,
including Science, in addition to platforms such as the National Library of
Medicine and the Department of Health Beattie and Mackenzie (2005); Yoo et al.
(2017); Honig and Chin (2001). A representative real patient data case is
showcased anonymously in Figure 9.

###  B.2 Synthesized Patient Profile Data

Based on the gathered data, we have formulated patient profile structures that
encompass various aspects of each patient. This structure begins with
demographic information such as age and ethnicity and further extends into
medical history, familial background, and both professional and personal
pursuits. Furthermore, it delves into the patient’s present challenges, future
aspirations, and emotional well-being. An illustrative synthesized case of the
patient, Linda William, is presented below, in Figure 10.

Name: Mr. A.
Age: 68 years old
Gender: Male
Medical Condition: Cognitive decline
Occupation: Engineer
Education: 16 years
Social History: No recent history of alcohol or tobacco use
Family History: Mother had dementia, father had a heart attack
Medical History: High blood pressure, foot operation, stomach reflux
Neurological Examination:
\- Mild difficulties in concentration and attention
\- Language abnormalities: difficulty finding words, paraphasia
\- Difficulty naming pictured objects (anomia)
\- Repetition of complicated sentences
\- Memory impairments
\- Impaired arithmetic abilities
\- Slight tremor in arms when outstretched
Summary:
Mr. A. showed difficulties in verbal communication and cognitive functions,
primarily in language and memory. Despite these impairments, he continued
working as an engineer and managed his daily activities with some support.
Figure 9: Real Patient Profile of Mr. A, collected from Science Honig and Chin
(2001) Name: Linda Williams
Gender: Female
Age: 73
Ethnicity: African American
Religion: Baptist
Medical Condition: Alzheimer’s
First language: English
Family: Son, brother
Location: Urban Atlanta
Family Details:
Son - Terrence, 49, a software engineer in San Francisco.
Brother - James, 75, a retired banker, lives in Florida.
Summary:
Linda was a former librarian. She lives in an apartment complex and has a
close-knit group of friends from her church choir. She enjoys baking and often
shares her treats with neighbors. Recently, she’s been forgetting song lyrics
and has missed a few church services. Terrence is concerned as she once left
the oven on after baking.
What’s important to you?
Linda values her independence and her choir practices. She loves sharing
stories of her time as a librarian.
What’s happening for you at the moment?
Linda’s friends have noticed she’s less active in the choir and often repeats
stories she has already told.
What is the impact on you?
She feels embarrassed when she forgets things, especially during choir
practice.
What would you like to happen in the future?
She hopes to remain active in her community and maintain her current routines.
How might we achieve this?
Terrence is considering a move to a senior community where she can have more
immediate assistance and supervision.
What strengths and support networks do you have to help you?
Linda’s church community is supportive. Her son Terrence calls regularly and
visits quarterly.
Figure 10: Synthetic Patient Case Example

##  Appendix C MemoryCompanion Source Code on GitHub

To champion transparency, reproducibility, and propel advancements in the
field, we commit to releasing the entirety of the MemoryCompanion research
source code on a GitHub Repository upon acceptance of the paper. This
repository contains all essential components, scripts, and comprehensive
documentation, simplifying replication and potential improvement of our work.
We invite fellow researchers and enthusiasts to clone the repository, explore
its structure and functionalities, and encourage contributions to this open-
source initiative. Collaborative efforts hold the potential to drive
significant progress in our field, and we’re committed to fostering such
collaboration through this release.