# Empathic Grounding: Explorations using Multimodal Interaction and Large
Language Models with Conversational Agents

Mehdi Arjmand  Northeastern UniversityBoston, MA<EMAIL_ADDRESS>,
Farnaz Nouraei  Northeastern UniversityBoston, MA<EMAIL_ADDRESS>,
Ian Steenstra  Northeastern UniversityBoston, MA<EMAIL_ADDRESS>and  Timothy Bickmore  Northeastern UniversityBoston, MA
<EMAIL_ADDRESS>
(2024; April 2024)

###### Abstract.

We introduce the concept of “empathic grounding” in conversational agents as
an extension of Clark’s conceptualization of grounding in conversation in
which the grounding criterion includes listener empathy for the speaker’s
affective state. Empathic grounding is generally required whenever the
speaker’s emotions are foregrounded and can make the grounding process more
efficient and reliable by communicating both propositional and affective
understanding. Both speaker expressions of affect and listener empathic
grounding can be multimodal, including facial expressions and other nonverbal
displays. Thus, models of empathic grounding for embodied agents should be
multimodal to facilitate natural and efficient communication. We describe a
multimodal model that takes as input user speech and facial expression to
generate multimodal grounding moves for a listening agent using a large
language model. We also describe a testbed to evaluate approaches to empathic
grounding, in which a humanoid robot interviews a user about a past episode of
pain and then has the user rate their perception of the robot’s empathy. We
compare our proposed model to one that only generates non-affective grounding
cues in a between-subjects experiment. Findings demonstrate that empathic
grounding increases user perceptions of empathy, understanding, emotional
intelligence, and trust. Our work highlights the role of emotion awareness and
multimodality in generating appropriate grounding moves for conversational
agents.

Affective Computing, Conversational Agents, Multimodal Interaction, Empathy,
Grounding, Social Robots

††copyright: acmlicensed††journalyear: 2024††doi: 3652988.3673949††conference:
; June 2024; ††booktitle: Preprint††isbn: 979-8-4007-0625-7/24/09††ccs: Human-
centered computing User studies††ccs: Human-centered computing Empirical
studies in HCI Figure 1. Computational Model of Multimodal Empathic Grounding

##  1\. Introduction

Empathy is the understanding and/or feeling of the affective experiences of
another person (Wispe, 1986). At least three kinds of empathy have been
identified in the literature: cognitive empathy, which is the perception and
understanding of another’s affective states, affective empathy, which is the
experiential alignment with another’s affective states (i.e., actually feeling
what the other is feeling), and narrative empathy, which is empathy for others
in related narratives and experiences (Keen, 2010). Empathy is hard-wired into
the brain; the mere act of observing affective facial displays triggers neural
responses in the observer’s own emotional circuitry, including activation of
premotor areas related to the observer’s own facial displays mirroring those
of the observed (Carr et al., 2003). A closely-related concept to empathy is
sympathy, which is the awareness of another’s negative affective state as
something to be alleviated (Wispe, 1986), often by the listener conveying
their feeling about the speaker’s affective state, such as concern or
compassion. However, most recent work in psychology and the helping
professions conflate sympathy and empathy, with the latter term having
dominant use.

The literature on empathy is voluminous, but can be roughly categorized into
two levels of analysis. At the macro level, empathic communication encompasses
intentional, overt conveyance of understanding of another’s emotional state to
satisfy instrumental goals of the listener, and can span many turns of dialog.
Examples include acknowledgment of patient distress by healthcare providers,
”reflections” in counseling (e.g., motivational interviewing (Miller and
Rollnick, 2023)), and active listening. In medicine, physician empathy for
patients has been associated with greater patient satisfaction, better health
outcomes, and fewer malpractice suits (Lorié et al., 2017), and is important
enough that training in empathic communication has been mandated as a learning
objective for medical school education (Riess et al., 2012).

At the micro level, empathy conveys understanding of some aspect of the
speaker’s current or related affective state that the they just communicated.
This process is largely unconscious and the empathic display can be very
brief, consisting of as little as a transitory facial display by the listener.
In this role, empathic cues play a role in grounding: the process
interlocutors use to incrementally update their mutual knowledge during
conversation (Clark and Brennan, 1991). For an utterance to be understood, the
interlocutors must arrive at a situation in which they mutually believe that
the listener understood what the speaker meant to a criterion sufficient for
current purpose, something referred to as the grounding criterion. For
utterances in which the speaker’s current or related affective state is
communicated or implied, the grounding criterion may include listener empathy.

Clark and Brennan identified five types of evidence that human listeners can
provide to demonstrate that they understood a speaker’s last utterance (Table
1). Listener empathy can serve, in whole or in part, as grounding cues for
levels 2-5.

Level | Evidence | Description
---|---|---
1 | Continued attention | H shows he/she is continuing to attend and therefore remains satisfied with S’s presentation.
2 | Initiation of the relevant next contribution | H starts in on the next relevant contribution.
3 | Acknowledgment | H nods or says ”uh huh” (backchannel).
4 | Demonstration | H demonstrates all or part of what he/she has understood S to mean.
5 | Display | H displays verbatim all or part of S’s presentation.

Table 1. Evidence of Understanding in Grounding, ranked from weakest to
strongest, for Speaker S and Hearer H. Based on (Clark and Brennan, 1991)

.

Clark extended the Gricean maxims of Quantity and Manner (Grice, 1975) to
define the principle of least collaborative effort, where interlocutors
attempt to minimize the joint effort they invest in making a conversation
work. Since empathy or sympathy can be displayed nonverbally (e.g., via facial
display), these cues facilitate grounding by allowing grounding evidence to be
distributed across channels simultaneously. For example, brief affect displays
can be used as acknowledgments (“backchannel” cues) that do not interrupt the
speaker, but convey more information than affectively-neutral cues such as
headnods. Smiles have been studied as backchannel cues (Bertrand et al.,
2007), (Bilalpur et al., 2024), as have nonverbal displays of positive and
negative affect (Shahverdi et al., 2023), but we argue that all affective
displays can not only play this role, but also provide more grounding evidence
than affectively neutral backchannels, regardless of modality.

We introduce the concept of empathic grounding as an extension of Clark’s
conceptualization of grounding in which the grounding criterion includes
listener empathy for the speaker’s affective state. This encompasses both
micro-empathy and affective backchannels, but also macro-empathy and sympathy.
Empathic grounding is not always required, since affectively neutral
information exchanges can be grounded without affect. However, it facilitates
communication and the grounding process whenever the speaker’s emotions are
foregrounded or when the speaker is relating affectively-charged narratives or
information. In addition, empathic grounding is essential for macro-empathy.

Relative to Clark and Brennan’s model (Table 1), empathic communication
comprises demonstration or display grounding cues, by definition. We note that
brief empathic facial displays, with no verbal accompaniment, fit between
acknowledgments and demonstrations in strength, since they convey valenced
information–beyond simple acknowledgment–that shows some level of
understanding. We agree with (Allwood et al., 1992) that demonstrations
typically provide stronger information than displays, at least for empathic
grounding, since demonstrations require a level of interpretation that
displays do not.

We differentiate empathic grounding from Jung’s “Affective Grounding” in HRI
(Jung, 2017). Jung’s work focused on emotion communication and regulation
among interlocutors, but as a separate track from non-emotional communication
and grounding processes. Here, we argue that propositional and affective
communication are entagled and support each other in empathic grounding. In
particular, we are interested in the role of affective communication in
enhancing propositional communication and grounding processes.

In this paper, we present a computational model of multimodal empathic
grounding for an embodied conversational agent that uses a large language
model (LLM) to generate grounding responses for user utterances. Prior studies
indicate that LLMs may be good at generating empathic language, with one study
demonstrating that LLM-generated responses to medical questions from
laypersons were rated higher on empathy than those generated by physicians
(Ayers et al., 2023). Our model goes beyond text, incorporating information
from user speech and nonverbal displays of affect, and produces brief agent
utterances of empathy and sympathy accompanied by appropriate affective and
interactional nonverbal behavior.

We also present a testbed for evaluating empathic grounding models in the
domain of interviewing participants about a prior painful experience, that
isolates the effects of grounding responses for comparison.

We report results from a between subjects experiment in which our empathic
grounding model (EMPATHIC GROUNDING) is compared against a baseline control
model (BACKCHANNEL) that only generates affectively-neutral backchannels.

We hypothesize that participants relating their pain experience in the
EMPATHIC GROUNDING condition, compared to participants in the BACKCHANNEL
condition, will rate the agent higher on (H1) perception of being understood,
(H2) empathy and empathic listening, and (H3) trust. In addition, we
hypothesize that (H4) recognition and incorporation of user affect from
multimodal cues will lead to better empathic grounding.

Figure 2. User interacting with the robot testbed for the empathic grounding
model

##  2\. Related Work

###  2.1. Grounding in Human-Agent Interactions

Previous work on conversational agents has seen long-standing interest in the
notion of grounding (Clark and Brennan, 1991). Task-based dialog systems
require feedback from users to understand the result of their actions (Wiener,
2019). Similarly, dialog systems that provide feedback to users to signal
understanding are able to create more responsive and efficient interactions
(Ward, 1996). In this realm, Traum’s computational model of grounding (Traum,
1994) provides grounding acts, such as acknowledgments and repairs, to enable
grounding in multi-turn human-agent dialog. Building on this model, (Nakano et
al., 2003) proposed a conversational agent design that maps the user’s verbal
and non-verbal behavior to the agent’s dialog state to perform grounding acts.
However, such a system can only fulfill the propositional aspect of dialog
through grounding acts, as both the input and output of the system are unaware
of emotion cues.

Among forms of grounding moves, backchanneling has been widely studied in
virtual agents. Most work on listener backchannel generation has used rules
based on prosodic and linguistic cues of the speaker (Ward and Tsukahara,
2000; Lala et al., 2017). The appropriate generation of backchannels by
listening agents has been shown to be effective in establishing rapport and
increasing engagement with users (Gratch et al., 2006; Huang et al., 2011;
Heylen, 2007). For example, Gratch et al. (Gratch et al., 2007) developed a
virtual agent that expressed listening and backchanneling behaviors and found
that contingent backchanneling with respect to user utterances can effectively
build rapport compared to non-contingent feedback (Gratch et al., 2007).
Compared to their approach to providing “envelope feedback” and following
prior research showing the needs of speakers for relevant listening behaviors
in particularly emotional contexts (Bavelas et al., 2000), we explore whether
more contentful grounding moves, including semantically- and affectively-
relevant feedback can establish a higher level of trust and empathy with
users. Further, while most prior work has focused on generating appropriate
non-verbal backchanneling behaviors, less attention has been given to
producing contingent verbal utterances alongside non-verbal cues (Heylen,
2007). Our work bridges this gap by introducing an LLM grounding move
generation module that takes multimodal user input and generates multimodal
agent output in real-time.

Another thread of research has focused on multimodal emotion processing and
delivering feedback to users via empathic listening agents (Buschmeier and
Kopp, 2014; Schroder et al., 2011; DeVault et al., 2014). For example,
(DeVault et al., 2014) proposed a framework to incorporate users’ multimodal
input and provide listening feedback, including verbal and nonverbal
backchannels. The authors found that the system was able to build rapport with
users in a mental health screening context, and led to more self-disclosure.
However, due to limitations in speech recognition, meaningful grounding moves
were sporadically generated in practice (Gratch and Lucas, 2021), and empathic
utterances were limited. In this work, we propose a novel multimodal empathic
grounding component that incorporates the user’s verbal and non-verbal inputs
and generates appropriate empathic grounding behaviors using multiple
modalities, including utterances, emotional states, and movements.

###  2.2. Multimodal Human-Agent Interaction

Human face-to-face conversation is highly multimodal, with facial displays,
hand gesture, and proxemics playing important roles in structuring
interactions and conveying information beyond speech. Thus, previous research
has attempted to use multimodal interaction in conversational agents,
including social robots and virtual agents (Pelachaud et al., 2021; Lugrin et
al., 2022).

Several ways of modeling multimodal conversational behavior have been used,
including theory and rule-based (Cassell et al., 2001), and machine learning
and corpora-based approaches (Lee and Marsella, 2006).

Our model of multimodal empathic grounding can be characterized by the
computational framework for multimodal interactions in (Kopp and Hassan,
2022), which outlines three primary components: processing, mapping, and
generating multimodal signals. Processing involves signal sensing,
recognition, and interpretation, while mapping focuses on planning and rule
application and generating entails producing output signals. In our model, we
perform the processing component explicitly, while mapping and generating are
performed using an LLM. We describe the details of our model architecture in
section 3.

###  2.3. LLMs for Social Agents

Pre-trained large language models (LLMs) such as GPT-based models (Brown et
al., 2020; Achiam et al., 2023), and LlaMA (Touvron et al., 2023) have
revolutionized natural language processing, advancing tasks related to
understanding and generating language. Specifically, the perception of empathy
and emotional intelligence in LLMs has been of rising interest among
researchers in social dialogue systems. For instance, studies have shown that
LLM-generated responses to social media posts describing common life
experiences are consistently perceived as more empathic than human-written
ones (Lee et al., 2024). Cuadra et al. demonstrate a systematic analysis of
LLMs display and perception of empathy (Cuadra et al., 2024). Moreover,
multiple studies demonstrated the capabilities and limitations of LLMs in
text-based emotion recognition tasks (Tak and Gratch, 2023; Broekens et al.,
2023; Zhan et al., 2023). Recently, conversational agents have used LLMs to
improve natural language understanding and generation. LLMs have also been
used in emotion recognition, dialog act classification, and utterance
generation in negotiation tasks (Lin et al., 2023).

Several studies have explored generating backchanneling and expressive
behavior for social robots. For example, generating robotic backchannelling
behaviors have been explored by (Mahadevan et al., 2024), who used few-shot
chain-of-thought prompting to translate human language instructions into
parameterized control code.LLMs have also been utilized to generate expressive
reactions and mapped them to a robot behavior using a fine-tuned model to be
shown on a social robot, leading to more engaging and empathetic interactions
(Wang et al., 2024).

##  3\. Computational Model of Multimodal Empathic Grounding

We assume a structured, agent-initiated dialog, in which each discourse
segment (Grosz and Sidner, 1986) is comprised of 1) an agent’s question to the
user, 2) the user’s response to the question, and 3) a grounding response by
the agent (Figure 3). Both user and agent contributions to the dialog can be
multimodal, comprised of verbal utterances and/or emotional state(s) exhibited
during the contribution. Our computational model is illustrated in Figure 1.
The individual components of the model are described below.

###  3.1. Multimodal User Affect Sensing and Recognition

At each turn of the conversation, the first step of the model is to capture
the user’s visual and speech input using a camera and microphone (sensing).
The user’s facial expressions and verbal utterances are derived by processing
the visual and speech inputs (recognition). The facial recognition input is
then summarized and converted into two most commonly-appeared emotion labels.
These processed inputs are then fed into the Interpretation module to generate
empathic grounding actions.

###  3.2. Multimodal Interpretation

Previous work has demonstrated the effectiveness of LLMs in understanding the
emotional tone of text (Tak and Gratch, 2023; Broekens et al., 2023). We
expand this approach and prompt an LLM as a multimodal emotion interpretation
model that captures the underlying emotional state of the user based on their
verbal utterances and facial expressions at each turn. Thus, the LLM can be
prompted to use the agent’s question and the user’s multimodal response to
predict the user’s dominant emotion label (i.e., single-class classification)
as interpreted from combined verbal and visual channels in the present turn,
along with the valance, arousal, and dominance components of their emotional
state (Russell and Mehrabian, 1977).

###  3.3. Multimodal Empathic Grounding Generation

After predicting the user’s dominant emotion, the LLM can be prompted to
generate an appropriate empathic grounding move. Unlike previous rule-based
approaches in affective computing (e.g., (Lin et al., 2023)), which often
provide explicit rules for mapping emotional states to behavior generation, we
do not define explicit mapping rules and leave decision-making to the LLM to
select among multiple available emotional states (e.g., neutral, sad, and
happy). Likewise, a set of body movements (e.g., head-nods) can be provided as
explicit option sets for the LLM to choose from. Finally, the LLM generates an
appropriate verbal utterance.

Figure 3. Example Discourse Segment. 1. Agent question. 2. User response. 3.
Agent grounding move.

###  3.4. Implementation Details

We used the LibreFace Model (Chang et al., 2024) in the OpenSense application
(Stefanov et al., 2020) to extract user facial expression features. This model
provides eight categorical facial expressions corresponding to neutral,
surprise, fear, happiness, sadness, disgust, anger, and contempt emotional
states. We export labels corresponding to the user’s dominant facial
expressions at a rate of 15 frames per second and filter out noise using a
maximum pool window for every five labels. We then take the two most common
labels (excluding neutral) from each user utterance and use them in our
multimodal interpreter.

We used “gpt-3.5-turbo-0125” (April 2024 version) as the LLM for the
multimodal empathic grounding module. We implemented both the multimodal
interpreter and generator in one zero-shot prompt. The input to and output
from the LLM were structured in JSON format, which was then parsed by the
agent for behavior realization. We instructed the LLM not to give medical
advice or inappropriate recommendations, and we did not define a persona for
the agent, unlike prior work on affective listeners (McKeown et al., 2011).
More comprehensive details of the prompt can be found in Table 2, and the
complete prompt is included in the supplementary material
111https://github.com/arminarj/empathic-grounding.

As input, the LLM received the agent’s question (text only) and the user’s
multimodal response. Our implementation of the LLM prompt outputs four
components for the generated agent behavior (grounding move): 1) emotional
state, 2) head movement, 3) verbal utterance, and 4) an explanation for its
decisions. Below is an overview of these components:

  * •

Emotional State: We provided the LLM with a list of possible emotional states
for the agent that fit the pain assessment scenario. These include the agent
being neutral, sad, happy, concerned, or surprised.

  * •

Head Movement: The LLM chose from among ‘no_movement’ and “head-nod” actions
for agent movements.

  * •

Verbal Utterance: The LLM was instructed to generate a brief empathic response
to the user’s action. We prompted LLM to generate a short non-generic
utterance to make the user feel heard and reflect on their emotion in an
appropriate way. We defined some rules for not generating any medical
recommendations and not asking any questions from the user.

  * •

Explanation: We requested that LLM clarify the reason for choosing its
outputs.

Table 2. Elements of LLM Prompt Used to Generate Empathic Grounding Moves

Subsection |  Description |  Part of the prompt
---|---|---
Defining the role |  Describes the role of the LLM as a Multimodal grounding generator. |  ”You are performing multimodal grounding moves and back-channeling(BC) for a conversational agent. For input, you will receive…”
Input channel descriptions |  Details of each value input to or output from the LLM, including input channels, output options, and rules. |  ”Description of the input channels: { ”agent_utterance”: ”the question that has been asked from the user”, …}”
Multimodal Affect Recognition |  Instructs LLM to use both input modalities to predict the user’s current affective state and generate the associated emotion label and circumplex scores. |  ”It is only possible to understand users’ meaning by focusing on both their verbal and facial expressions…facial expression can be noisy…predict user’s dominant emotion and give VAD scores…”
Multimodal Interpretation |  Use the user’s predicted dominant emotion and generate an appropriate response. |  ”Based on the user dominant emotion you guessed, you need to generate a proper multimodal and empathic grounding move…”
Human-agent conversation description |  Describes the structure of the discourse segments used in the model. |  ”Agent:agent_utterance_1, User: user_utterance_1, Agent: <Your generated moves, including verbal and non-verbal parts>, Agent: agent_utterances_2, …”
Rules |  Rules that LLM needs to use in general include the following 1) options, 2) generation rules, and 3) format rules. |  ”RULES: 1) For the channels you receive options for [affect and movements], only choose your response between the provided options ….2)…”
Input and output format |  Illustrates example JSON inputs and outputs. |  ”input format: {”agent_utterance”:…, ”BC_verbal_rules”: …, ”BC_nonverbal_options”: …}, output format {”user_dominant_emotion”: ”Classify the user’s emotional state into…”}”

##  4\. Evaluation Testbed

We developed a testbed to evaluate the empathic grounding model (Figure 2).
Our task domain involves interviewing participants about a prior painful
experience, inspired by (Murali et al., 2023). This domain was chosen because
it can involve an extended dialog to probe different aspects of pain (e.g.,
intensity, location, temporal pattern, etc.) and provides many opportunities
for empathic grounding. There are validated self-report measures that user
responses can be validated against, and painful incidents are relatively
common, facilitating study recruitment.

The overall conversation comprised a greeting, a few turns of social chat, an
initial open-ended question about the user’s pain experience, a series of
follow-up questions probing different aspects of pain (some open-ended and
some closed-ended), and a farewell. All dialog is agent-initiated, and
grounding moves are generated for every user response, including social chat.

In our testbed, only grounding responses by the agent were fully automated.
The agent’s questions and task-based follow-up utterances (e.g., ”Tell me
more.”, ”Could you repeat that?”) were controlled by a human confederate in a
wizard-of-oz setup to ensure human-level understanding of the task-related
parts of the conversation. This strategy ensured that errors in understanding
user task responses did not derail the conversation and lead to poor
perceptions of the agent, even if its grounding moves were flawless. In this
setup, the agent grounding moves fit into a very constrained role in the
dialog, allowing the effects of different grounding models to be isolated and
compared.

###  4.1. Embodied Converational Agent

We used a Furhat humanoid robot (Al Moubayed et al., 2012) as our embodied
conversational agent, which is a human-like head capable of expressing verbal
utterances, facial expressions, gaze, and head movements. Furhat uses a
projector to show an animated face and sensors that can track the user’s head
movements. This setup is optimal for our empathic grounding investigation, as
it focuses the user’s attention on the robot’s face during the interaction.
Furhat is equipped with a Google-based automatic speech recognition (ASR)
model to listen to users’ speech. Realistic speech is generated using Amazon
Polly Neural Voices and includes automatic lip-synchronization. We used a
female’s face and voice as our robot character. For non-verbal behaviors, we
used the robot’s default surprise and sad faces and created new
implementations of happy and concerned facial expressions and head nods.

###  4.2. Wizard-of-Oz Framework

In order to provide accurate dialog management and understanding of the effect
of empathic grounding moves in a conversation, we designed a Wizard-of-Oz
control setup (Figure 3). The wizard hears and sees the participant’s
interaction with the robot via closed-circuit video and can initiate the next
pain assessment question by pushing a button on a control user interface.
After a user responds to the question, and the robot’s grounding move is
automatically produced and performed, the wizard can decide to either proceed
to the next question or to perform other actions as described in
LABEL:tab:wizard. However, the wizard does not control the robot’s grounding
responses, emotional states, or head movements, as the testbed automatically
generates these responses using the generation module.

Table 3. Wizard-of-Oz Control Actions Button |  Action and Descriptions
---|---
User Repeat Response |  Ask the user to repeat a response if the robot could not capture their utterance.
Interrupt |  | If robot interrupts, apologize to the user and
---
encourage them to continue.
Irrelevant |  | Tells the user that the robot cannot answer
---
their question.
Listen-only |  | Lets the robot listen to the user’s
---
speech without saying anything.

##  5\. Comparative Evaluation Experiment

We conducted a between-subjects experiment to assess the effect of our agent’s
grounding moves on users while they expressed their recent episodes of pain,
comparing our empathic grounding model to a baseline backchannel-only model.
In this setup, we hypothesized that using empathic grounding would help the
user feel more heard and build more trust with the agent, compared to non-
empathic grounding. We designed a between-subjects experiment comparing the
following conditions:

  * •

BACKCHANNEL (acknowledgment): In this condition, the agent performs non-
empathic grounding moves. The agent a neutral facial expression, a head
movement randomly chosen from among “head-nod” and “no_movement”, as well as
an acknowledgment utterance selected at random from a set of neutral grounding
utterances, such as ”Noted.” or ”OK.”, representing a clinical standard of
care in pain assessment interviews based on our analysis of online mock
interactions.

  * •

EMPATHIC GROUNDING: In this condition, the agent performs empathic grounding
moves to react to the user’s responses based on the model described in section
3.

In both conditions, the wizard controls the task conversation flow and
encourages the user to keep sharing their thoughts until they stop speaking or
explicitly mention they have nothing to add.

###  5.1. Recruitment

We recruited participants via an online job posting site and social media
within our institution. We had three recruitment criteria: over 18 years old,
a native speaker of English, and having experienced an episode of pain in the
past two weeks. The study was approved by our Institutional Review Board, and
participants were compensated for their time and transportation costs.

###  5.2. Measures

We captured the participant’s speech and visual interaction with the robot
using an RGB camera and microphone using the OpenSense (Stefanov et al., 2020)
platform in the Microsoft \psi data format (Bohus et al., 2021).

In addition to sociodemographics, we collected the following self-report
measures from participants following their interaction with the robot: the
Active-Empathic Listening Scale (AELS) (Bodie, 2011) and the Responsiveness of
Other Scale (Utami and Bickmore, 2019) to assess perceptions of the robot’s
empathy, the MSCEIT-Based Perceived Emotional Intelligence Questionnaire
(MSCEIT) (Mayer et al., 2003), the Bond subscale of the Working Alliance
Inventory (WAI) (Horvath and Greenberg, 1989) to assess trust and confidence
in working with the robot, the Godspeed Agent Attitude Questionnaires (GS)
(Bartneck et al., 2009), and a subset of the General Agent Rating items
(Bowman et al., 2024) (Table 4).

Additionally, in order to assess the validity of pain reports to the robot,
participants provided self-reports on their pain experiences using validated
measures, including PROMIS (Bevans et al., 2014) and the Visual Analog Scale
(Langley and Sheppeard, 1985).

Finally, semi-structured interviews were conducted with participants at the
end of their session to obtain an overall understanding of their experience.
Interviews were recorded, transcribed, and analyzed using thematic analysis
techniques (Braun and Clarke, 2006).

Question | Anchor 1 | Anchor 7 | BACKCHANNEL | EMPATHIC | p
---|---|---|---|---|---
How satisfied are you with the conversational agent? | Not at all | Very satisfied | 4.00 (0.93) | 5.11 (1.54) | 0.074
How much would you like to continue working with the conversational agent? | Not at all | Very much | 4.00 (0.76) | 5.56 (1.24) | 0.015∗
How much do you like the conversational agent? | Not at all | Very much | 3.63 (1.06) | 5.78 (1.39) | 0.006∗∗
How would you characterize your relationship with the agent? | Stranger | Close friend | 2.88 (1.36) | 4.22 (1.92) | 0.139
How much do you feel the conversational agent cares about you? | Not at all | Very | 3.50 (2.27) | 5.00 (1.50) | 0.200
How much do you feel that you and the conversational agent understand each other? | Not at all | Very much | 3.13 (1.36) | 4.78 (1.39) | 0.036∗

Table 4. Self-Report Attitudes towards Robot by Study Condition. Wilcoxon
significance tests on single items.

###  5.3. Study Protocol

Following informed consent, we randomly assigned participants to one of our
two groups (BACKCHANNEL and EMPATHIC GROUNDING). We then introduced the robot
as an automated agent that can see and hear people and ask them about a recent
painful experience. Participants were informed that they should respond to the
robot’s questions with natural speech. A researcher (wizard) controlled the
robot from a different room, using the method described in subsection 4.2.

Following the pain interview with the robot, participants filled out post-
study questionnaires and participated in the semi-structured interview
subsection 5.2. Finally, participants received a debriefing about the system
and the WoZ deception for controlling the robot.

###  5.4. Results

####  5.4.1. Participants

A total of 18 participants completed the experiment, including 12 males and 6
females. Participants were aged between 21 and 64 (Med =30, IQR=23.5). In
terms of race, 50% of participants identified as White, 33.3% as Asian, and
5.5% as African American, while others were multi-racial or preferred not to
disclose. Most had completed higher education (high school=2, bachelor’s
degree=7, graduate degree=9) and had experience with computers (regularly
using computer=5, expert using computer=13). Lastly, participants reported the
intensity of their reported pain experience an average of
$\mu=5.32,\sigma=1.61$ on the Visual Analog Scale (ranging from 0 for no pain
to 10 as “worst pain”). We excluded one participant’s data due to a technical
error that led to incomplete interaction.

###  5.5. Quantitative Analysis

Although we had a small sample size, Shapiro-Wilk normality tests showed the
normality of the distributions of our measurements WAI, MSCEIT, AELS, and
Responsiveness. As demonstrated in Figure 4, we performed an independent
sample t-test for composite measures between two conditions. Additionally, we
compared the single-item measures of satisfaction using Wilcoxon (Table 4.)

Compared to those in the BACKCHANNEL condition, participants in the EMPATHIC
GROUNDING condition rated the robot significantly higher on empathy, based on
AELS, $t(15)=2.173,p<.05$, significantly higher on emotional intelligence,
based on MSCEIT, $t(15)=2,186,p<.05$, and significantly higher on working
alliance, based on WAI-Bond, $t(13)=3.438,p<0.01$. Differences in
Responsiveness (RESP) were in the hypothesized direction but were only
trending in significance, $t(15)=1.886,p=.079$. On the Godspeed measures, the
EMPATHIC GROUNDING robot was found to be significantly more animated (GS-
ANIM), $t(16)=2.53,p=.022$, and likable (GS-LIKE), $t(16)=2.18,p=.045$,
compared to the BACKCHANNEL robot. These results are illustrated in Figure 4.

Based on Wilcoxon tests on the General Agent Rating items (Table 4),
participants in the EMPATHIC GROUNDING condition rated the robot significantly
higher on the degree they felt the robot understood them, satisfaction with
the robot, desire to continue working with the robot, and liking of the robot.

We checked the validity of participant pain reports to the robot against the
PROMIS and Visual Analog Scale measures. Those in the EMPATHIC GROUNDING group
reported pain intensity that was significantly correlated with a numeric scale
of pain intensity, Spearman $rho=0.932,p<.001$, but whose correlation with the
Visual Analog Scale of pain intensity was only trending, Spearman
$rho=0.635,p=.066$. Those in the BACKCHANNEL group reported pain intensity
that was significantly correlated with both standard measures, Spearman
$rho=0.897,p=.003$ for the numeric scale, and Spearman $rho=0.830,p=.011$ for
the Visual Analog Scale.

For reports of the degree to which pain was interfering with their life, those
in the EMPATHIC GROUNDING group reported interference that was significantly
correlated with the corresponding PROMIS measure, Spearman $rho=0.805,p=.016$,
while for those in the BACKCHANNEL group, this correlation was not
significant, Spearman $rho=0.506,p=.201$.

Figure 4. Ratings of BACKCHANNEL and EMPATHIC GROUNDING Robots. See subsection
5.2 for other measure descriptions. (*) shows $p<0.05$ and (**) shows $p<0.01$

###  5.6. Qualitative Analysis

From semi-structured interviews, we analyzed a total of 171 minutes of
transcripts using thematic analysis to understand user perceptions of the
interaction with the robot. The most prominent themes that emerged from the
data are described below.

####  5.6.1. Perceptions of Empathic Grounding

Participants in the EMPATHIC GROUNDING group (8 out of 9 participants)
commented on features explicitly related to the empathic grounding moves in
their feedback. Some liked the expressions and noticed instances of facial
expressions and emotions: ”When it was saying something very sad, it [looked]
down…It understands and gives a sad expression.” [P12]. Another participant
highlighted: ”it was sensing my emotion in whatever I was [saying], it was
[interpreting] the context and if I speak positively, negatively, sadly
happily it was getting that”. However, some found these responses partially
scripted: ”There were a couple of times I thought that it did [listen to
me]…But for the majority of the time it [was] more pre-programmed, scripted
response.” [P4].

####  5.6.2. Feeling heard by the robot

Only 3 out of 8 participants in our BACKCHANNEL group mentioned being heard by
the agent, while 9 out of 9 participants in EMPATHIC GROUNDING said the robot
listened to them during the conversation. Although in the BACKCHANNEL group,
the robot acknowledged what the participant had mentioned, some participants
felt that the interaction was programmed: ”It listened. I mean it recorded me.
I didn’t really get the feeling it was listening as much as recording” [P15,
BACKCHANNEL]. This was in contrast to most participants in EMPATHIC GROUNDING,
who felt that the robot genuinely heard them. Additionally, multiple
participants in this group emphasized that the interaction felt natural,
possibly contributing to the feeling of being heard.

####  5.6.3. Being interrupted by the Robot

The majority of our participants (6 out of 8) in the BACKCHANNEL group and 4
out of 9 participants in the EMPATHIC GROUNDING group mentioned that the robot
interrupted them in the middle of their conversation. Although we used the
same robot setup in both groups, the rate of reporting the interruption during
the open-ended interviewing was different.

###  5.7. Case Analysis: LLM as Multimodal Reasoner

Throughout the experiment, we observed several interactions in which
information from the user’s affective state, as recognized from their facial
display, was used to provide more empathically-accurate grounding than would
have been otherwise possible for those in the EMPATHIC GROUNDING condition.
For example, P16’s response to the question ”How’s the weather today?” was
”cloudy.” with a smiling facial display, and the robot used the empathic
grounding move: ”I’m glad you find joy in it”. The LLM’s explanation for this
action was ”The user’s verbal response ’cloudy’ combined with facial
expressions of ’Happiness’ indicate a positive and cheerful mood”. Based on a
post-experiment analysis, we found that had the user affect information not
been provided to the LLM, it would have generated the more neutral sentence
”It looks like a cloudy day.”

Another example was during P14’s social chat. When asked about the weather,
they responded, ”it’s raining outside and sometimes it is showing to be snowy
and I like it personally but it is a little bit uh not that comfortable but
it’s all right” while smiling. In response, the agent’s reaction was a head-
nod, happy face, and a ”It sounds like a lovely mix!” utterance. However, if
user affect information had been removed, the response would have been an
agent with a neutral face saying, ”Sounds like a mixed weather day”.

As part of the conversation with P18’s interaction, when the agent asked if
they had experienced a painful experience recently, they said, ”that is
correct” while smiling. The agent’s grounding move was ”I appreciate your
honesty and strength” with a neutral face and a head nod. The LLM’s rationale
for this was ”The user confirmed a painful experience but expressed happiness,
indicating a resilient attitude, so a supportive and appreciative response is
suitable”. However, if LLM hadn’t used the facial expression, the generated
sentence would have been ”I understand your pain. Take care” with a concerned
facial expression and no head movement.

Participants in the EMPATHIC GROUNDING condition indicated that they and the
robot understood each other to a significantly greater degree compared to
those in the BACKCHANNEL condition, supporting H1. Participants in the
EMPATHIC GROUNDING condition also rated the robot significantly higher on the
Active-Empathic Listening Scale (AELS), supporting H2. Participants in the
EMPATHIC GROUNDING condition also rated the robot significantly higher on the
Bond dimension of the Working Alliance Inventory, indicating a higher degree
of trust, supporting H3. Finally, we identified several instances in which our
empathic grounding model incorporated information about the user’s affective
state to produce responses that were more appropriate than they would have
been otherwise, providing partial support for H4. That said, a more systematic
approach is required to fully investigate the impact of multimodal input
channels on the quality of empathic grounding moves.

Participants in the EMPATHIC GROUNDING condition also rated the robot
significantly higher on Perceived Emotional Intelligence. This scale has items
spanning the identification of emotions, appropriate responses to emotions,
and the appearance of being emotionally self-aware.

Our analysis suggests that empathic grounding can alter the user’s perception
of the interaction and make them feel heard more often. Not every participant
in our EMPATHIC GROUNDING study felt their emotions were being used in the
same way, but some noticed that the agent was able to convey emotions in
accordance with what they had expressed.

Interestingly, even in conversational exchanges that were not intended to
require empathic grounding–such as social chat about the weather–the empathic
grounding model still provided more appropriate and affectively-tailored
responses to user utterances, compared to a model that only responded with
neutral backchannel responses.

##  6\. Conclusion

We introduced the concept of ”empathic grounding” in conversational agents,
and described how it fits into existing theories of macro- and micro-empathy,
as well as grounding in face-to-face communication. We described a model for
automatically generating multimodal empathic grounding responses for an
embodied conversational agent using LLMs, and a testbed for evaluating such
models in the task domain of interviewing people about past pain experiences.
Finally, we presented results from an experiment, demonstrating that our
empathic grounding model leads to significantly higher ratings of empathic
listening, emotional intelligence, and working alliance for an agent compared
to an agent that only uses neutral backchannels for grounding.

###  6.1. Limitations

There are many limitations to our study, beyond the small convenience sample
used. We evaluated our empathic grounding model in a highly constrained
scenario, and our results may not generalize to other kinds of conversations
or even other dialog structures.

The two conditions we compared in our study may not have been ideal for
isolating a single factor for testing. However, we feel they represent a
pragmatic trial in which the BACKCHANNEL condition is a good representation of
’standard of care’ in clinical pain assessment, and the EMPATHIC-GROUNDING
condition reflects the behavior of an affectively-attuned, empathic caregiver.

###  6.2. Future Work

There are many interesting directions for future work. Grounding includes much
more than providing positive evidence of understanding, and the use of LLMs to
generate other kinds of grounding moves–including negative evidence and
repairs–remains an important avenue of investigation. We only used participant
facial display for affect recognition, and there are many other modalities and
approaches that could provide more accurate and nuanced assessments that could
be included.

We only used one turn of conversation to generate empathic grounding moves,
but future work can assess the role of including more discourse context in
move generation.

Due to a fixed audio threshold timeout to determine end of user utterance in
Furhat, the robot frequently interrupted users who paused during long
contributions. Even though the robot interrupted with a grounding move, it
often disrupted and annoyed users. Future work should explore the use of
better turn-taking mechanisms, including the incorporation of multimodal cues
such as user gaze, as well as a more nuanced generation of grounding moves
that are sensitive to whether the user is trying to hold the floor or not
(e.g., using only backchannels during pauses in long user contributions).

Finally, the nonverbal cues used to express empathy can be culturally-
dependent (Lorié et al., 2017), raising the question of how grounding differs
across cultures, and this represents another important area of inquiry and
experimentation.

## References

  * (1)
  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023\.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ (2023).
  * Al Moubayed et al. (2012) Samer Al Moubayed, Jonas Beskow, Gabriel Skantze, and Björn Granström. 2012.  Furhat: a back-projected human-like robot head for multiparty human-machine interaction. In _Cognitive Behavioural Systems: COST 2102 International Training School, Dresden, Germany, February 21-26, 2011, Revised Selected Papers_. Springer, 114–130.
  * Allwood et al. (1992) J Allwood, J Nivre, and E Ahlsen. 1992.  On the semantics and pragmatics of linguistic feedback.  _Journal of Semantics_ 9 (1992).
  * Ayers et al. (2023) J. W. Ayers, A. Poliak, M. Dredze, E. C. Leas, Z. Zhu, J. B. Kelley, D. J. Faix, A. M. Goodman, C. A. Longhurst, M. Hogarth, and D. M. Smith. 2023.  Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum.  _JAMA Intern Med_ 183, 6 (2023), 589–596.
  * Bartneck et al. (2009) Christoph Bartneck, Dana Kulić, Elizabeth Croft, and Susana Zoghbi. 2009.  Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots.  _International journal of social robotics_ 1, 1 (2009), 71–81.
  * Bavelas et al. (2000) Janet B Bavelas, Linda Coates, and Trudy Johnson. 2000.  Listeners as co-narrators.  _Journal of personality and social psychology_ 79, 6 (2000), 941.
  * Bertrand et al. (2007) R Bertrand, G Ferré, P Blache, R Espesser, and S Rauzy. 2007.  _Backchannels revisited from a multimodal perspective_.  1–5.
  * Bevans et al. (2014) Margaret Bevans, Alyson Ross, and David Cella. 2014.  Patient-Reported Outcomes Measurement Information System (PROMIS): efficient, standardized tools to measure self-reported health and quality of life.  _Nursing outlook_ 62, 5 (2014), 339–345.
  * Bilalpur et al. (2024) M Bilalpur, M Inan, D Zeinali, J Cohn, and M Alikhani. 2024.  Learning to Generate Context-Sensitive Backchannel Smiles for Embodied AI Agents with Applications in Mental Health Dialogues.  https://arxiv.org/html/2402.08837v1
  * Bodie (2011) Graham D Bodie. 2011.  The Active-Empathic Listening Scale (AELS): Conceptualization and evidence of validity within the interpersonal domain.  _Communication Quarterly_ 59, 3 (2011), 277–295.
  * Bohus et al. (2021) Dan Bohus, Sean Andrist, Ashley Feniello, Nick Saw, Mihai Jalobeanu, Patrick Sweeney, Anne Loomis Thompson, and Eric Horvitz. 2021.  Platform for Situated Intelligence.  arXiv:2103.15975 [cs.AI]
  * Bowman et al. (2024) R Bowman, O Cooney, J Newbold, A Thieme, L Clark, G Doherty, and Cowan B. 2024.  Exploring how politeness impacts the user experience of chatbots for mental health support.  _International Journal of Human-Computer Studies_ 184 (2024).
  * Braun and Clarke (2006) Virginia Braun and Victoria Clarke. 2006.  Using thematic analysis in psychology.  _Qualitative research in psychology_ 3, 2 (2006), 77–101.
  * Broekens et al. (2023) Joost Broekens, Bernhard Hilpert, Suzan Verberne, Kim Baraka, Patrick Gebhard, and Aske Plaat. 2023.  Fine-grained Affective Processing Capabilities Emerging from Large Language Models. In _2023 11th International Conference on Affective Computing and Intelligent Interaction (ACII)_. IEEE, 1–8.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Buschmeier and Kopp (2014) Hendrik Buschmeier and Stefan Kopp. 2014.  When to elicit feedback in dialogue: Towards a model based on the information needs of speakers. In _Intelligent Virtual Agents: 14th International Conference, IVA 2014, Boston, MA, USA, August 27-29, 2014. Proceedings 14_. Springer, 71–80.
  * Carr et al. (2003) L. Carr, M. Iacoboni, M. C. Dubeau, J. C. Mazziotta, and G. L. Lenzi. 2003.  Neural mechanisms of empathy in humans: a relay from neural systems for imitation to limbic areas.  _Proc Natl Acad Sci U S A_ 100, 9 (2003), 5497–502.
  * Cassell et al. (2001) Justine Cassell, Hannes Högni Vilhjálmsson, and Timothy Bickmore. 2001.  Beat: the behavior expression animation toolkit. In _Proceedings of the 28th annual conference on Computer graphics and interactive techniques_. 477–486.
  * Chang et al. (2024) Di Chang, Yufeng Yin, Zongjian Li, Minh Tran, and Mohammad Soleymani. 2024.  LibreFace: An Open-Source Toolkit for Deep Facial Expression Analysis. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_.  To appear.
  * Clark and Brennan (1991) Herbert H. Clark and Susan E. Brennan. 1991.  _Grounding in Communication_.  American Psychological Association, Washington, 127–149.  read.
  * Cuadra et al. (2024) Andrea Cuadra, Maria Wang, Lynn Andrea Stein, Malte F Jung, Nicola Dell, Deborah Estrin, and James A Landay. 2024.  The Illusion of Empathy? Notes on Displays of Emotion in Human-Computer Interaction. In _ACM Conference on Human Factors in Computing Systems (CHI)_.
  * DeVault et al. (2014) David DeVault, Ron Artstein, Grace Benn, Teresa Dey, Ed Fast, Alesia Gainer, Kallirroi Georgila, Jon Gratch, Arno Hartholt, Margaux Lhommet, et al. 2014\.  SimSensei Kiosk: A virtual human interviewer for healthcare decision support. In _Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems_. 1061–1068.
  * Gratch and Lucas (2021) Jonathan Gratch and Gale Lucas. 2021.  _Rapport Between Humans and Socially Interactive Agents_ (1 ed.).  Association for Computing Machinery, New York, NY, USA, 433–462.  https://doi.org/10.1145/3477322.3477335
  * Gratch et al. (2006) Jonathan Gratch, Anna Okhmatovskaia, Francois Lamothe, Stacy Marsella, Mathieu Morales, Rick J van der Werf, and Louis-Philippe Morency. 2006.  Virtual rapport. In _Intelligent Virtual Agents: 6th International Conference, IVA 2006, Marina Del Rey, CA, USA, August 21-23, 2006. Proceedings 6_. Springer, 14–27.
  * Gratch et al. (2007) Jonathan Gratch, Ning Wang, Jillian Gerten, Edward Fast, and Robin Duffy. 2007.  Creating rapport with virtual agents. In _Intelligent Virtual Agents: 7th International Conference, IVA 2007 Paris, France, September 17-19, 2007 Proceedings 7_. Springer, 125–138.
  * Grice (1975) H Grice. 1975.  _Logic and conversation_.  Academic Press, New York.
  * Grosz and Sidner (1986) Barbara Grosz and C. Sidner. 1986.  Attention, Intentions, and the Structure of Discourse.  _Computational Linguistics_ 12, 3 (1986), 175–204.
  * Heylen (2007) Dirk Heylen. 2007.  Multimodal backchannel generation for conversational agents. In _Workshop on Multimodal Output Generation, MOG 2007_. Centre for Telematics and Information Technology (CTIT), 81–92.
  * Horvath and Greenberg (1989) Adam O Horvath and Leslie S Greenberg. 1989.  Development and validation of the Working Alliance Inventory.  _Journal of counseling psychology_ 36, 2 (1989), 223.
  * Huang et al. (2011) Lixing Huang, Louis-Philippe Morency, and Jonathan Gratch. 2011.  Virtual Rapport 2.0. In _Intelligent Virtual Agents: 10th International Conference, IVA 2011, Reykjavik, Iceland, September 15-17, 2011. Proceedings 11_. Springer, 68–79.
  * Jung (2017) M Jung. 2017.  Affective Grounding in Human-Robot Interaction. In _HRI_.
  * Keen (2010) S Keen. 2010.  _Narrative empathy_.  University of Texas Press, 61–94.
  * Kopp and Hassan (2022) Stefan Kopp and Teena Hassan. 2022.  _The Fabric of Socially Interactive Agents: Multimodal Interaction Architectures_ (1 ed.).  Association for Computing Machinery, New York, NY, USA, 77–112.  https://doi.org/10.1145/3563659.3563664
  * Lala et al. (2017) Divesh Lala, Pierrick Milhorat, Koji Inoue, Masanari Ishida, Katsuya Takanashi, and Tatsuya Kawahara. 2017.  Attentive listening system with backchanneling, response generation and flexible turn-taking. In _Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue_. 127–136.
  * Langley and Sheppeard (1985) GB Langley and H Sheppeard. 1985.  The visual analogue scale: its use in pain measurement.  _Rheumatology international_ 5, 4 (1985), 145–148.
  * Lee and Marsella (2006) Jina Lee and Stacy Marsella. 2006.  Nonverbal behavior generator for embodied conversational agents. In _International Workshop on Intelligent Virtual Agents_. Springer, 243–255.
  * Lee et al. (2024) Yoon Kyung Lee, Jina Suh, Hongli Zhan, Junyi Jessy Li, and Desmond C Ong. 2024.  Large Language Models Produce Responses Perceived to be Empathic.  _arXiv preprint arXiv:2403.18148_ (2024).
  * Lin et al. (2023) Eleanor Lin, James Hale, and Jonathan Gratch. 2023.  Toward a Better Understanding of the Emotional Dynamics of Negotiation with Large Language Models. In _Proceedings of the Twenty-fourth International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing_. 545–550.
  * Lorié et al. (2017) Á Lorié, D. A. Reinero, M. Phillips, L. Zhang, and H. Riess. 2017.  Culture and nonverbal expressions of empathy in clinical settings: A systematic review.  _Patient Educ Couns_ 100, 3 (2017), 411–424.
  * Lugrin et al. (2022) Birgit Lugrin, Catherine Pelachaud, and David Traum (Eds.). 2022.  _The Handbook on Socially Interactive Agents: 20 years of Research on Embodied Conversational Agents, Intelligent Virtual Agents, and Social Robotics Volume 2: Interactivity, Platforms, Application_ (1 ed.). Vol. 48.  Association for Computing Machinery, New York, NY, USA.
  * Mahadevan et al. (2024) Karthik Mahadevan, Jonathan Chien, Noah Brown, Zhuo Xu, Carolina Parada, Fei Xia, Andy Zeng, Leila Takayama, and Dorsa Sadigh. 2024.  Generative expressive robot behaviors using large language models.  _arXiv preprint arXiv:2401.14673_ (2024).
  * Mayer et al. (2003) John D Mayer, Peter Salovey, David R Caruso, and Gill Sitarenios. 2003.  Measuring emotional intelligence with the MSCEIT V2. 0.  _Emotion_ 3, 1 (2003), 97.
  * McKeown et al. (2011) Gary McKeown, Michel Valstar, Roddy Cowie, Maja Pantic, and Marc Schroder. 2011.  The semaine database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent.  _IEEE transactions on affective computing_ 3, 1 (2011), 5–17.
  * Miller and Rollnick (2023) W Miller and S Rollnick. 2023.  _Motivational Interviewing: Helping People Change and Grow_ (fourth ed.).  Guilford Press, New York.
  * Murali et al. (2023) P Murali, M Arjmand, M Volonte, Z Li, J Griffith, M Paasche-Orlow, and T Bickmore. 2023.  Towards Automated Pain Assessment using Embodied Conversational Agents. In _The Third International Workshop on Automated Assessment of Pain (AAP)_.
  * Nakano et al. (2003) Yukiko I Nakano, Gabe Reinstein, Tom Stocky, and Justine Cassell. 2003.  Towards a model of face-to-face grounding. In _Proceedings of the 41st annual meeting of the Association for Computational Linguistics_. 553–561.
  * Pelachaud et al. (2021) C Pelachaud, C Busso, and D Heylen. 2021.  _Multimodal Behavior Modeling for Socially Interactive Agents_. Vol. 1.  Association for Computing Machinery, 259–310.
  * Riess et al. (2012) H. Riess, J. M. Kelley, R. W. Bailey, E. J. Dunn, and M. Phillips. 2012.  Empathy training for resident physicians: a randomized controlled trial of a neuroscience-informed curriculum.  _J Gen Intern Med_ 27, 10 (2012), 1280–6.
  * Russell and Mehrabian (1977) James A Russell and Albert Mehrabian. 1977.  Evidence for a three-factor theory of emotions.  _Journal of research in Personality_ 11, 3 (1977), 273–294.
  * Schroder et al. (2011) Marc Schroder, Elisabetta Bevacqua, Roddy Cowie, Florian Eyben, Hatice Gunes, Dirk Heylen, Mark Ter Maat, Gary McKeown, Sathish Pammi, Maja Pantic, et al. 2011\.  Building autonomous sensitive artificial listeners.  _IEEE transactions on affective computing_ 3, 2 (2011), 165–183.
  * Shahverdi et al. (2023) P Shahverdi, K Rousso, J Klotz, I Bakhoda, M Zribi, and W-Y Louie. 2023.  Emotionally Specific Backchanneling in Social Human-Robot Interaction and Human-Human Interaction. In _IROS_.
  * Stefanov et al. (2020) Kalin Stefanov, Baiyu Huang, Zongjian Li, and Mohammad Soleymani. 2020.  OpenSense: A Platform for Multimodal Data Acquisition and Behavior Perception. In _Proceedings of the 2020 International Conference on Multimodal Interaction_. Association for Computing Machinery, 660–664.
  * Tak and Gratch (2023) Ala N Tak and Jonathan Gratch. 2023.  Is GPT a Computational Model of Emotion?. In _2023 11th International Conference on Affective Computing and Intelligent Interaction (ACII)_. IEEE, 1–8.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023\.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ (2023).
  * Traum (1994) David Traum. 1994.  A computational theory of grounding in natural language conversation.  (1994).
  * Utami and Bickmore (2019) Dina Utami and Timothy Bickmore. 2019.  Collaborative user responses in multiparty interaction with a couples counselor robot. In _2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_. IEEE, 294–303.
  * Wang et al. (2024) Zining Wang, Paul Reisert, Eric Nichols, and Randy Gomez. 2024.  Ain’t Misbehavin’–Using LLMs to Generate Expressive Robot Behavior in Conversations with the Tabletop Robot Haru.  _arXiv preprint arXiv:2402.11571_ (2024).
  * Ward (1996) Nigel Ward. 1996.  Using prosodic clues to decide when to produce back-channel utterances. In _Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP’96_ , Vol. 3. IEEE, 1728–1731.
  * Ward and Tsukahara (2000) N Ward and W Tsukahara. 2000.  Prosodic features which cue back-channel responses in English and Japanese.  _J Pragmatics_ 32, 8 (2000), 1177–1207.
  * Wiener (2019) Norbert Wiener. 2019.  _Cybernetics or Control and Communication in the Animal and the Machine_.  MIT press.
  * Wispe (1986) L Wispe. 1986.  The distinction between sympathy and empathy: To call forth a concept, a word is needed.  _Journal of personality and social psychology_ 50, 2 (1986), 314–321.
  * Zhan et al. (2023) Hongli Zhan, Desmond C Ong, and Junyi Jessy Li. 2023.  Evaluating subjective cognitive appraisals of emotions from large language models.  _arXiv preprint arXiv:2310.14389_ (2023).


# A Non-Hermitian Moiré Valley Filter

Kai Shao  National Laboratory of Solid State Microstructures, School of
Physics, and Collaborative Innovation Center of Advanced Microstructures,
Nanjing University, Nanjing, 210093, China  Hao Geng  National Laboratory of
Solid State Microstructures, School of Physics, and Collaborative Innovation
Center of Advanced Microstructures, Nanjing University, Nanjing, 210093, China
Erfu Liu  National Laboratory of Solid State Microstructures, School of
Physics, and Collaborative Innovation Center of Advanced Microstructures,
Nanjing University, Nanjing, 210093, China  Jose L. Lado  Department of
Applied Physics, Aalto University, 02150 Espoo, Finland  Wei Chen
Corresponding author<EMAIL_ADDRESS>National Laboratory of Solid State
Microstructures, School of Physics, and Collaborative Innovation Center of
Advanced Microstructures, Nanjing University, Nanjing, 210093, China  D. Y.
Xing  National Laboratory of Solid State Microstructures, School of Physics,
and Collaborative Innovation Center of Advanced Microstructures, Nanjing
University, Nanjing, 210093, China

###### Abstract

A valley filter capable of generating a valley-polarized current is a crucial
element in valleytronics, yet its implementation remains challenging. Here, we
propose a valley filter made of a graphene bilayer which exhibits a 1D moiré
pattern in the overlapping region of the two layers controlled by
heterostrain. In the presence of a lattice modulation between layers,
electrons propagating in one layer can have valley-dependent dissipation due
to valley asymmetric interlayer coupling, thus giving rise to a valley-
polarized current. Such a process can be described by an effective non-
Hermitian theory, in which the valley filter is driven by a valley-resolved
non-Hermitian skin effect. Nearly 100% valley-polarization can be achieved
within a wide parameter range and the functionality of the valley filter is
electrically tunable. The non-Hermitian topological scenario of the valley
filter ensures high tolerance against imperfections such as disorder and edge
defects. Our work opens a new route for efficient and robust valley filters
while significantly relaxing the stringent implementation requirements.

_Introduction_.-Valleytronics Rycerz  _et al._ (2007); Xu  _et al._ (2014);
Schaibley  _et al._ (2016); Vitale  _et al._ (2018), exploits the valley
degree of freedom for electronic information processing and storage similar to
the use of spin degree of freedom in spintronics Žutić  _et al._ (2004). As
the _K_ and _K’_ valleys are distinctly separated in reciprocal space,
graphene and other hexagonal 2D materials have become promising candidates for
the valleytronic devices. Various valley-resolved phenomena have been
explored, including the valley Hall effect Xiao  _et al._ (2007); Yamamoto
_et al._ (2015); Gorbachev  _et al._ (2014); Mak  _et al._ (2014), optical
control of valley polarization Mak  _et al._ (2012); Zeng  _et al._ (2012);
Cao  _et al._ (2012); Jones  _et al._ (2013), and valley Zeeman effect
Srivastava  _et al._ (2015); Aivazian  _et al._ (2015); Li  _et al._ (2014);
MacNeill  _et al._ (2015); Stier  _et al._ (2016). This demonstrations
exemplify the feasibility to manipulate the valley degree of freedom. Just as
spin-polarized current serves as a key ingredient in spintronics Žutić  _et
al._ (2004), the implementation of valley-polarized current is a prerequisite
for any valleytronic applications. A variety of proposals have been put
forward to generate valley-polarized current in graphene, including using the
zeroth mode of the zigzag ribbon Rycerz  _et al._ (2007), topological line
defects Gunlycke and White (2011); Liu  _et al._ (2013); Chen  _et al._
(2014), zero-line modes Martin  _et al._ (2008); Qiao  _et al._ (2014, 2011);
Li  _et al._ (2018), mechanical strain Fujita  _et al._ (2010); Jiang  _et
al._ (2013); Zhai  _et al._ (2011) and nanobubbles Settnes  _et al._ (2016).
Despite intense efforts, achieving an ideal valley filter capable of
generating fully valley-polarized current with controllable valley
polarization remains an open challenge Schaibley  _et al._ (2016). Therefore,
it is crucial to explore new strategies that can achieve a highly efficient
valley filter, and at the same time, tolerate various device imperfections.

Physical effects that are robust against imperfections are commonly found in
systems with nontrivial topological properties. One such example is the
occurrence of unidirectional transport in the chiral edge states of the
quantum Hall phase, which are robust to disorder Girvin and Yang (2019). In
Hermitian systems, however, nonreciprocal transport can only occur at the
boundaries of the sample. To achieve unidirectional transport in the bulk of
the sample, it is necessary to disrupt the Hermiticity of the system Lee  _et
al._ (2019); Bessho and Sato (2021); Chen  _et al._ (2023), or turn it open.
Recently, great progress has been made in this direction. The unidirectional
transport can be achieved inside the bulk, which causes accumulation of states
at open boundaries, known as the non-Hermitian skin effect Yao and Wang
(2018); Yao  _et al._ (2018); Kunst  _et al._ (2018); Yokomizo and Murakami
(2019); Yang  _et al._ (2020). Such nonreciprocal bulk transport also has a
topological origin Borgnia  _et al._ (2020); Okuma  _et al._ (2020); Zhang
_et al._ (2020); Bergholtz  _et al._ (2021), the point gap topology of the
complex spectrum, indicating its robustness against imperfections. Inspired by
the analysis above, a robust valley filter was suggested to be implemented
through valley-resolved nonreciprocal transport in the non-Hermitian regime,
which can be engineered in mesoscopic electronic systems Geng  _et al._
(2023).

In this Letter, we propose a robust valley filter using the 1D moiré pattern
formed in a graphene homobilayer, as illustrated in Fig. 1(a), referred to as
the moiré valley filter. The setup has an interface structure composed of the
upper layer (system S) and the lower layer (reservoir R), along with four
terminals (1-4). If layer R is subjected to a uniaxial tension, the coupling
between S an R can be valley-dependent. Due to such a lattice modulation, the
electrons in S that flow from terminal 1 to 2 initially occupy both valleys
and then undergo a valley selective filtering by R, thus generating a valley-
polarized current [cf. Fig. 1(a)]. Physically, the valley-dependent coupling
with R introduces an effective non-Hermitian self-energy to S, which leads to
valley-resolved nonreciprocal transport. Specifically, the electrons in the
$K$ $(K^{\prime})$ valley can only propagate in the $x$ $(-x)$ direction,
while its propagation in the opposite direction will have a strong decay. The
scenario can be interpreted by the valley-resolved complex spectral winding,
which induces nonreciprocity within each valley. Such non-Hermitian
topological effect enables the valley filter to be resilient against
imperfections such as disorder and edge defects, features of major importance
towards its practical implementation.

Figure 1: (a) Schematic diagram of the proposed interface system composed of
an upper graphene layer S and a lower graphene layer R, with four terminals
labeled by 1-4. Electrons in the $K$ valley can transmit from 1 to 2 while
those in the $K^{\prime}$ valley are filtered out by entering the R layer. (b)
The effective complex bands of S with the interlayer hopping strength set to
one percent of its original value for better illustration. (c,d) Complex
energy spectra of the two lowest sub-conduction bands (above the bands with
zero energy) with the directions of spectral winding marked by arrows, which
corresponds to the change of $ka$ (c) from $-\pi$ to $0$ and (d) from $0$ to
$\pi$, respectively. The relevant parameters are $a_{0}=0.142$ nm,
$W_{S}=29a_{0}$, $d_{0}=0.335$ nm, $t^{0}_{\perp}=0.48$ eV and
$\delta_{0}=0.319a_{0}$.

_Model_.-We take the region S as a zigzag graphene nanoribbon with a width of
$W_{S}$ along the $x$ direction, stacked on top of the R layer with a length
of $W_{R}$ in such a way that their zigzag edges are aligned with each other.
The overlapping area of the two layers is $W_{S}$ $\times$ $W_{R}$. The
bilayer system can be captured by the tight-binding Hamiltonian as

$\begin{split}H=-t\sum_{\langle
i,j\rangle}c_{i}^{\dagger}c_{j}-\sum_{i,j}t_{\perp}(\mathbf{r}_{i},\mathbf{r}_{j})c_{i}^{\dagger}c_{j},\end{split}$
(1)

where $c_{i}^{\dagger}$ $(c_{i})$ creates (annihilates) electrons at site $i$.
The first term describes the intralayer nearest-neighbor hopping with strength
$t=2.7$ eV, and the second term corresponds to the interlayer hopping Trambly
de Laissardière  _et al._ (2010); Uryu (2004); Nakanishi and Ando (2001)

$t_{\perp}(\mathbf{r}_{i},\mathbf{r}_{j})=te^{-\frac{r-a_{0}}{\delta_{0}}}\left[1-\left(\mathbf{r}\cdot\hat{z}/{r}\right)^{2}\right]-t^{0}_{\perp}e^{-\frac{r-d_{0}}{\delta_{0}}}\left({\mathbf{r}\cdot\hat{z}}/{r}\right)^{2}.$
(2)

Here, $\mathbf{r}=\mathbf{r}_{i}-\mathbf{r}_{j}$ is the displacement vector
between two sites, $\hat{z}$ is the unit vector in the $z$-direction, $a_{0}$
is the intralayer site spacing, $d_{0}$ is the interlayer spacing, and
$t^{0}_{\perp}$ is the vertical hopping between two layers. The hopping decays
exponentially within a typical scale $\delta_{0}$.

The 1D moiré superlattice Timmel and Mele (2020) can be achieved by including
a uniaxial strain on the R layer in the $x$-direction. Accordingly, the scale
of R layer is changed by a factor $\varepsilon$ in the $x$-direction and the
$x$-coordinate of the lattice sites changes through
$x_{i}\rightarrow(1+\varepsilon)x_{i}$. It is assumed that the $y$-coordinate
remains unchanged by neglecting the Poisson’s ratio of graphene Pereira  _et
al._ (2009). Such deformation of R gives rise to a moiré pattern in the
$x$-direction; see Fig. 1(a). The main effect of the deformation of the R
layer can be clearly seen in the momentum space, which drives the two Dirac
points $K,K^{\prime}$ moving towards the $\Gamma$ point in the projected 1D
Brillouin zone foo . As a result, a relative shift takes place between the
Dirac cones in the two layers, which breaks the reciprocity within each
valley. The moiré superlattice is assumed to be commensurate for simplicity
such that $1+\varepsilon=m/n$ is a rational number with $m,n\in\mathbb{N}$.
The primitive wavevectors of the S layer, R layer and the whole moiré
superlattice are given by $\mathbf{b}_{S}=(2\pi/a)\hat{x}$,
$\mathbf{b}_{R}=n\mathbf{b}_{S}/m$ and $\mathbf{b}_{M}=\mathbf{b}_{S}/m$,
respectively, with $a=\sqrt{3}a_{0}$.

_Complex energy bands_.-In contrast to the typical scenario in periodic moiré
systems, our study instead concentrates on the non-Hermitian physics within
the S layer, whose electronic property is engineered by the R layer Geng  _et
al._ (2023). Electrons in S can penetrate into R and finally escape into leads
3 and 4, thus undergoing an effective loss [cf. Fig. 1(a)]. The physics within
S can be described by the Green’s function in a general form as

$\begin{split}g_{S}^{r}(\omega)=\left(\omega-H_{S}^{\rm{eff}}\right)^{-1},\ \
H_{S}^{\rm{eff}}(\omega)=H_{S}+\Sigma^{r}_{R}(\omega),\end{split}$ (3)

where the effective Hamiltonian $H_{S}^{\rm{eff}}$ consists of the bare
Hamiltonian $H_{S}$ of the S layer and the retarded self-energy
$\Sigma^{r}_{R}(\omega)$, which is non-Hermitian. The self-energy is given by
$\Sigma^{r}_{R}(\omega)=\hat{V}g_{R}^{r}(\omega)\hat{V}^{\dagger}$, where
$\hat{V}$ is the interlayer hopping matrix and
$g_{R}^{r}(\omega)=\left(\omega-
H_{R}-\Sigma^{r}_{3}-\Sigma^{r}_{4}\right)^{-1}$ is the retarded Green’s
function of the R layer. Here, $H_{R}$ is the Hamiltonian of the R layer and
the effect of leads 3 and 4 is absorbed into the surface self-energy
$\Sigma^{r}_{3,4}$, which is calculated using the iterative method sm .

The electron dynamics in S is governed by $H_{S}^{\rm{eff}}(\omega)$, making
it insightful to explore its complex energy bands. For this purpose, we first
assume that both S and R layers extend to infinity in the $x$-direction. For
the commensurate moiré modulation, the interlayer coupling takes place between
the Bloch states with their wave vectors differing by a reciprocal
superlattice vector $\mathbf{G}_{\alpha}=l\mathbf{b}_{M}$, where $\alpha=S,R$
is the layer index and $l\in\mathbb{Z}$. The Bloch Hamiltonian for the bilayer
is expressed as

$\begin{split}H(\mathbf{k})&=\sum_{\alpha,\beta=S,R}\sum_{\mathbf{G}_{\alpha},\mathbf{G}^{\prime}_{\beta}}c^{{\dagger}}_{\mathbf{k}+\mathbf{G}_{\alpha}}H_{\mathbf{G}_{\alpha},\mathbf{G}^{\prime}_{\beta}}(\mathbf{k})c_{\mathbf{k}+\mathbf{G}^{\prime}_{\beta}},\end{split}$
(4)

where the Bloch wave vector
$\mathbf{k}\in[-\mathbf{b}_{M}/2,\mathbf{b}_{M}/2]$ lie in the moiré Brillouin
zone and the summation over the reciprocal superlattice vector is restricted
by the condition that the total wave vector
$\textbf{k}+\textbf{G}_{\alpha}=\textbf{q}_{\alpha}$ lies in the first
Brillouin zone of the $\alpha$ layer. As such, the reciprocal vectors
$\\{\textbf{G}_{\alpha}\\}$ become band labels and all bands are folded up
into the moiré Brillouin zone. The Fermi annihilation operator is expressed by
the sublattice components as
$c_{\mathbf{q}_{\alpha}}=(c^{1}_{\mathbf{q}_{\alpha}},c^{2}_{\mathbf{q}_{\alpha}},\cdots,c^{\chi}_{\mathbf{q}_{\alpha}},\cdots)^{\text{T}}$
with
$c^{\chi}_{\mathbf{q}_{\alpha}}=N_{\alpha}^{-1/2}\sum_{\mathbf{r}_{\alpha}^{\chi}}e^{i{\mathbf{q}_{\alpha}}\cdot\mathbf{r}_{\alpha}^{\chi}}c_{\mathbf{r}_{\alpha}^{\chi}}$,
where the lattice site is specified by its location
$\mathbf{r}_{\alpha}^{\chi}$ and $N_{\alpha}$ is the number of unit cells. The
matrix elements in Eq. (4) are specified as follows.
$H_{\textbf{G}_{S},\textbf{G}^{\prime}_{S}}=H_{S}(\textbf{k}+\textbf{G}_{S})\delta_{\textbf{G}_{S},\textbf{G}^{\prime}_{S}}$
is diagonal in the $\textbf{G}_{S}$ space with
$H_{S}(\textbf{k}+\textbf{G}_{S})$ the Bloch Hamiltonian of the S layer.
Similarly,
$H_{\textbf{G}_{R},\textbf{G}^{\prime}_{R}}=\tilde{H}_{R}(\textbf{k}+\textbf{G}_{R})\delta_{\textbf{G}_{R},\textbf{G}^{\prime}_{R}}$
is diagonal in the $\textbf{G}_{R}$ space with
$\tilde{H}_{R}(\mathbf{k}+\mathbf{G}_{R})=H_{R}(\mathbf{k}+\mathbf{G}_{R})+\Sigma^{r}_{3}(\mathbf{k}+\mathbf{G}_{R})+\Sigma^{r}_{4}(\mathbf{k}+\mathbf{G}_{R})$
the effective Hamiltonian of the R layer composed of the bare Hamiltonian and
the self-energies due to leads 3 and 4 sm . Finally, the interlayer coupling
matrix is calculated by
$H_{\textbf{G}_{S},\textbf{G}^{\prime}_{R}}=H^{\dagger}_{\textbf{G}^{\prime}_{R},\textbf{G}_{S}}=\langle\Phi_{\mathbf{k}+\mathbf{G}_{S}}^{S}|\hat{V}|\Phi_{\mathbf{k}+\mathbf{G}^{\prime}_{R}}^{R}\rangle$,
in which $|\Phi_{\mathbf{q}_{\alpha}}^{\alpha}\rangle$ is the Bloch function
of the $\alpha$ layer.

Integrating out the R layer following Eq. (3) yields the self-energy matrix
$\Sigma^{r}_{R}(\omega,\textbf{k};\textbf{G}_{S},\textbf{G}^{\prime}_{S})=\sum_{\textbf{G}_{R}}H_{\textbf{G}_{S},\textbf{G}_{R}}[\omega-\tilde{H}_{R}(\textbf{k}+\textbf{G}_{R})]^{-1}H^{\dagger}_{\textbf{G}^{\prime}_{S},\textbf{G}_{R}}$.
The complex band structures can be obtained by diagonalizing the effective
Hamiltonian of the S layer
$H_{S}^{\text{eff}}(\omega,\textbf{k})=H_{S}+\Sigma^{r}_{R}$. Due to the
folding of the complex bands, the information regarding the two valleys is
hidden. To extract the valley flavor of the states, we unfold the multiple
bands back to the original Brillouin zone of the S layer by tracking the
momentum index $\mathbf{G}_{S}$. Futhermore, since the self-energy
$\Sigma_{R}^{r}(\omega,\mathbf{k})$ is itself a function of energy $\omega$,
only the eigenvalues of $H_{S}^{\rm{eff}}(\omega,\textbf{k})$ that fulfill
$\text{Re}[E(\mathbf{k})]=\omega$ have physical meanings and are considered.
The complex band structure, shown in Fig. 1(b), highlights both the real and
imaginary parts of the energy spectrum, for which the valley-resolved
nonreciprocal transport can be clearly revealed. Specifically, the lifetime
$\tau(\omega,\mathbf{k})$ of electrons in S is determined by
$\tau^{-1}=-\text{Im}[E(\omega,\mathbf{k})]$. From Fig. 1(b). It is observed
that the right-moving electrons around the $K$ valley possess a longer
lifetime compared to the left-moving electrons within the same valley. The
opposite situation applies to the $K^{\prime}$ valley as expected from time-
reversal symmetry, thus exhibiting a valley-resolved nonreciprocity. Such
scenario can be further illustrated by the complex spectral winding Borgnia
_et al._ (2020); Okuma  _et al._ (2020); Zhang  _et al._ (2020); Bergholtz
_et al._ (2021). Here, we have separated the spectral winding for the two
valleys depicted in Figs. 1(c) and 1(d), where the arrows represent the
momentum increase. The complex energy spectra in Figs. 1(c) and 1(d) are
identical, but they exhibit opposite winding directions, indicating a valley-
resolved non-Hermitian skin effect. The spectral winding, along with the
distinct separation of the two valleys in the momentum space, indicates the
robustness of this effect.

Figure 2: (a) Valley-resolved differential conductance as a function of
$W_{R}$ with $\varepsilon=5\%$. (b) Valley polarization with different
distortion factors $\varepsilon$. (c) Valley polarization adjusted by the gate
voltages with $\varepsilon=2\%$. The bias voltage is $eV=0.81$ eV and the
other parameters are the same as those in Fig. 1.

_Non-Hermitian moiré valley filter_.-The valley-resolved nonreciprocal
transport can be utilized to create a novel type of valley filter. In contrast
to previous proposals, the current scenario exploits a non-Hermitian strategy.
We numerically study the electron transport between lead 1 and 2 through the
moiré valley filter using the scattering matrix approach based on the lattice
model. To reveal the valley-resolved scattering, we label different transport
channels with the wave vector k for a specific incident energy $\omega$ sm .
In particular, the amplitude $t_{\mathbf{k}_{2},\mathbf{k}_{1}}(\omega)$
corresponds to the transmission from channel $\mathbf{k}_{1}$ in lead 1 into
channel $\mathbf{k}_{2}$ in lead 2. Within the energy window of interest where
the valley degree of freedom is well defined, electrons that lie in the $K$
and $K^{\prime}$ valley can be selected by the conditions of $k\in[0,\pi/a]$
and $k\in[-\pi/a,0]$, respectively sm .

Figure 3: Valley-resolved conductance and valley polarization as a function of
(a) disorder strength, (b) number of edge defects. The relevant parameters are
$\varepsilon=5\%$, $W_{R}=141a$, and the other parameters are the same as
those in Fig. 1.

By applying a bias voltage $V$ on lead 1 and keeping the other three leads
grounded, the differential conductance with the valley index
$\kappa=K,K^{\prime}$ is defined as

$\begin{split}G_{\kappa}(eV)=\frac{e^{2}}{h}\sum_{\mathbf{k}_{1}}\sum_{\mathbf{k}_{2}\in\kappa}|t_{\mathbf{k}_{2},\mathbf{k}_{1}}|^{2},\end{split}$
(5)

which is contributed by the transmitted electrons in the $\kappa$ valley in
lead 2, while the incident electrons from lead 1 occupy both valleys. The
numerical results of $G_{K}$ and $G_{K^{\prime}}$ as a function of $W_{R}$ are
shown in Fig. 2(a). Because of the valley-resolved nonreciprocity,
$G_{K^{\prime}}$ decays much faster than $G_{K}$ as $W_{R}$ increases.
Furthermore, both conductances exhibit additional oscillations arising from
the intravalley interference.

The valley polarization of the current is quantified by Rycerz  _et al._
(2007)

$P=\frac{G_{K}-G_{K^{\prime}}}{G_{K}+G_{K^{\prime}}},$ (6)

where $P>0$ and $P<0$ correspond to valley polarization at $K$ and
$K^{\prime}$, respectively. The valley polarization corresponding to the
conductance in Fig. 2(a) is depicted in Fig. 2(b) by the solid line. It can be
observed that the current becomes $K$-valley polarized as $W_{R}$ exceeds a
certain value. Notably, a polarization of $P\simeq 100\%$ can be achieved
around the plateaus. The results with different deformation factors
$\varepsilon$ are compared in Fig. 2(b), where the contrast in $P$ that
appears first for small $W_{R}$ decreases and eventually disappears as $W_{R}$
increases. A similar situation occurs for the setup with different widths
$W_{S}$ sm , indicating that the decrease of the valley polarization for a
larger $W_{S}$ can be compensated by a larger $W_{R}$. These results highlight
the universality of our proposal. The positive valley polarization here
underscores the main distinction between the current scenario and the one in
Ref. Rycerz  _et al._ (2007). For the latter, the presence of an additional
right-moving mode at the $K^{\prime}$ valley in the conduction band would lead
to a negative polarization with the same parameters.

The valley polarization can be further enhanced by applying gate voltages
Oostinga  _et al._ (2008); Castro  _et al._ (2007); Chen  _et al._ (2020); Sui
_et al._ (2015); Shimazaki  _et al._ (2015). Their effects can be simulated by
two on-site potential $U_{t}$ and $U_{b}$ in the respective layer as
$H_{g}=\sum_{i\in S}U_{t}c_{i}^{\dagger}c_{i}+\sum_{i\in
R}U_{b}c_{i}^{\dagger}c_{i}$. The impact of the gate voltages is illustrated
in Fig. 2(c), which results in a suppression of the oscillation in $P$ and
effectively expands the parameter region with $P\simeq 100\%$. This property
alleviates the stringent requirement for a specific $W_{R}$ to achieve an
efficient valley filter.

_Robustness.-_ In contrast to previous proposals for the valley filter Rycerz
_et al._ (2007); Gunlycke and White (2011); Liu  _et al._ (2013); Chen  _et
al._ (2014); Martin  _et al._ (2008); Qiao  _et al._ (2014, 2011); Li  _et
al._ (2018); Fujita  _et al._ (2010); Jiang  _et al._ (2013); Zhai  _et al._
(2011); Settnes  _et al._ (2016), our proposal capitalizes on the non-
Hermitian topological effect, making it more robust against various lattice
imperfections. The valley-resolved nonreciprocity diminishes the impact of
multiple scattering on the valley polarization. This happens because when
electrons are scattered into the opposite valley while maintaining their
propagating direction, their propagation experiences a strong decay.
Therefore, the valley polarization of the transmitted electrons remains
unchanged.

Here, we examine two types of imperfections commonly encountered in the
system: local disorder and edge defects. We first consider the effect of
Anderson disorder depicted by the inset of Fig. 3(b). It is described by the
random on-site potential $H_{\text{dis}}=\sum_{i\in
S}U_{i}c_{i}^{\dagger}c_{i}$ in the S layer, where
$U_{i}\in[-W_{D}/2,W_{D}/2]$ with $W_{D}$ the disorder strength. The valley-
resolved transport is shown in Figs. 3(a) and 3(b), in which the high valley
polarization persists despite the disorder of a moderate strength. Moving on
to the edge defects, we consider vacancies that randomly exist at the zigzag
edges of the S layer; refer to the inset of Fig. 3(d). Similar to the disorder
effect, the valley polarization remains high even in the presence of a large
number of edge defects as shown in Figs. 3(c) and 3(d). Moreover, the impact
of edge defects is expected to diminish as the width $W_{S}$ increases because
the valley filter relies primarily on bulk transport.

During the fabrication a small twist between the two layers can appear
accidentally. This is simulated by a rotation of the R layer and consequently,
the interlayer hopping changes its magnitude according to Eq. (2) by using the
twisted geometry. By calculating the conductance $G_{K,K^{\prime}}$ and valley
polarization $P$ as a function of the twist angle, we find that the valley
polarization is always positive and exhibits an interesting oscillation
behavior. Furthermore, nearly perfect valley polarization can be realized
around different plateaus; see Supplemental Material sm  for more details.

_Discussion and outlook.-_ We have demonstrated an efficient strategy to
create a valley filter in a heterostrained graphene bilayer stemming from the
moiré modulation. The valley filter effect relies on the 1D moiré pattern,
which can be realized by a differential and uniaxial strain Si  _et al._
(2016); Bai  _et al._ (2020). The underlying non-Hermitian topological
scenario guarantees its robustness against lattice imperfections, lifting the
stringent implementation requirements of previous proposals. The universality
of the mechanism implies that the reduction of nonreciprocity due to increased
$W_{S}$ can be compensated by using a larger $W_{R}$. Moreover, the device
with a larger scale effectively reduces the energy spacing between transverse
modes and so a small bias voltage is sufficient to drive the valley-resolved
transport. Due to the bulk origin of our mechanism, the impact of irregular
edges becomes negligible for wide devices. Our moiré valley filter mechanism
can be readily extended to other hexagonal 2D materials, including transition
metal dichalcogenides Schaibley  _et al._ (2016); Vitale  _et al._ (2018).
Furthermore, a similar strategy can be applied to create the valley-resolved
non-Hermitian skin effect in metamaterials, such as photonic and phononic
crystals Weidemann  _et al._ (2020); Ghatak  _et al._ (2020); Zhang  _et al._
(2021a, b); Zhou  _et al._ (2023), where the nonreciprocal wave propagation
can exhibit richer structures with valley dependence. Our results pave the way
for exploring the valley-resolved non-Hermitian skin effect, establishing an
overlooked connection between non-Hermiticity and moiré physics Andrei and
MacDonald (2020); Balents  _et al._ (2020); Andrei  _et al._ (2021); Kennes
_et al._ (2021); Lau  _et al._ (2022); Mak and Shan (2022).

_Acknowledgments._ -This work was supported by the National Natural Science
Foundation of China under Grant No. 12074172 (W.C.), No. 12222406 (W.C.) and
No. 12174182 (D.Y.X.), the State Key Program for Basic Researches of China
under Grants No. 2021YFA1400403 (D.Y.X.), the Fundamental Research Funds for
the Central Universities (W.C.). J.L.L acknowledges the support from Aalto
Science-IT project, the Academy of Finland Projects No. 331342, No. 336243 and
No. 358088, and the Jane and Aatos Erkko Foundation.

## References

  * Rycerz  _et al._ (2007) A. Rycerz, J. Tworzydło, and C. Beenakker, Nature Physics 3, 172 (2007).
  * Xu  _et al._ (2014) X. Xu, W. Yao, D. Xiao, and T. F. Heinz, Nature Physics 10, 343 (2014).
  * Schaibley  _et al._ (2016) J. R. Schaibley, H. Yu, G. Clark, P. Rivera, J. S. Ross, K. L. Seyler, W. Yao, and X. Xu, Nature Reviews Materials 1, 1 (2016).
  * Vitale  _et al._ (2018) S. A. Vitale, D. Nezich, J. O. Varghese, P. Kim, N. Gedik, P. Jarillo-Herrero, D. Xiao, and M. Rothschild, Small 14, 1801483 (2018).
  * Žutić  _et al._ (2004) I. Žutić, J. Fabian, and S. D. Sarma, Reviews of modern physics 76, 323 (2004).
  * Xiao  _et al._ (2007) D. Xiao, W. Yao, and Q. Niu, Physical review letters 99, 236809 (2007).
  * Yamamoto  _et al._ (2015) M. Yamamoto, Y. Shimazaki, I. V. Borzenets, and S. Tarucha, Journal of the Physical Society of Japan 84, 121006 (2015).
  * Gorbachev  _et al._ (2014) R. Gorbachev, J. Song, G. Yu, A. Kretinin, F. Withers, Y. Cao, A. Mishchenko, I. Grigorieva, K. S. Novoselov, L. Levitov, _et al._ , Science 346, 448 (2014).
  * Mak  _et al._ (2014) K. F. Mak, K. L. McGill, J. Park, and P. L. McEuen, Science 344, 1489 (2014).
  * Mak  _et al._ (2012) K. F. Mak, K. He, J. Shan, and T. F. Heinz, Nature nanotechnology 7, 494 (2012).
  * Zeng  _et al._ (2012) H. Zeng, J. Dai, W. Yao, D. Xiao, and X. Cui, Nature nanotechnology 7, 490 (2012).
  * Cao  _et al._ (2012) T. Cao, G. Wang, W. Han, H. Ye, C. Zhu, J. Shi, Q. Niu, P. Tan, E. Wang, B. Liu, _et al._ , Nature communications 3, 887 (2012).
  * Jones  _et al._ (2013) A. M. Jones, H. Yu, N. J. Ghimire, S. Wu, G. Aivazian, J. S. Ross, B. Zhao, J. Yan, D. G. Mandrus, D. Xiao, _et al._ , Nature nanotechnology 8, 634 (2013).
  * Srivastava  _et al._ (2015) A. Srivastava, M. Sidler, A. V. Allain, D. S. Lembke, A. Kis, and A. Imamoğlu, Nature Physics 11, 141 (2015).
  * Aivazian  _et al._ (2015) G. Aivazian, Z. Gong, A. M. Jones, R.-L. Chu, J. Yan, D. G. Mandrus, C. Zhang, D. Cobden, W. Yao, and X. Xu, Nature Physics 11, 148 (2015).
  * Li  _et al._ (2014) Y. Li, J. Ludwig, T. Low, A. Chernikov, X. Cui, G. Arefe, Y. D. Kim, A. M. van der Zande, A. Rigosi, H. M. Hill, S. H. Kim, J. Hone, Z. Li, D. Smirnov, and T. F. Heinz, Phys. Rev. Lett. 113, 266804 (2014).
  * MacNeill  _et al._ (2015) D. MacNeill, C. Heikes, K. F. Mak, Z. Anderson, A. Kormányos, V. Zólyomi, J. Park, and D. C. Ralph, Phys. Rev. Lett. 114, 037401 (2015).
  * Stier  _et al._ (2016) A. V. Stier, K. M. McCreary, B. T. Jonker, J. Kono, and S. A. Crooker, Nature communications 7, 10643 (2016).
  * Gunlycke and White (2011) D. Gunlycke and C. T. White, Phys. Rev. Lett. 106, 136806 (2011).
  * Liu  _et al._ (2013) Y. Liu, J. Song, Y. Li, Y. Liu, and Q.-f. Sun, Phys. Rev. B 87, 195445 (2013).
  * Chen  _et al._ (2014) J.-H. Chen, G. Autès, N. Alem, F. Gargiulo, A. Gautam, M. Linck, C. Kisielowski, O. V. Yazyev, S. G. Louie, and A. Zettl, Phys. Rev. B 89, 121407 (2014).
  * Martin  _et al._ (2008) I. Martin, Y. M. Blanter, and A. F. Morpurgo, Phys. Rev. Lett. 100, 036804 (2008).
  * Qiao  _et al._ (2014) Z. Qiao, J. Jung, C. Lin, Y. Ren, A. H. MacDonald, and Q. Niu, Phys. Rev. Lett. 112, 206601 (2014).
  * Qiao  _et al._ (2011) Z. Qiao, J. Jung, Q. Niu, and A. H. MacDonald, Nano letters 11, 3453 (2011).
  * Li  _et al._ (2018) J. Li, R.-X. Zhang, Z. Yin, J. Zhang, K. Watanabe, T. Taniguchi, C. Liu, and J. Zhu, Science 362, 1149 (2018).
  * Fujita  _et al._ (2010) T. Fujita, M. Jalil, and S. Tan, Applied Physics Letters 97, 043508 (2010).
  * Jiang  _et al._ (2013) Y. Jiang, T. Low, K. Chang, M. I. Katsnelson, and F. Guinea, Phys. Rev. Lett. 110, 046601 (2013).
  * Zhai  _et al._ (2011) F. Zhai, Y. Ma, and K. Chang, New Journal of Physics 13, 083029 (2011).
  * Settnes  _et al._ (2016) M. Settnes, S. R. Power, M. Brandbyge, and A.-P. Jauho, Phys. Rev. Lett. 117, 276801 (2016).
  * Girvin and Yang (2019) S. M. Girvin and K. Yang,  _Modern condensed matter physics_ (Cambridge University Press, 2019).
  * Lee  _et al._ (2019) J. Y. Lee, J. Ahn, H. Zhou, and A. Vishwanath, Phys. Rev. Lett. 123, 206404 (2019).
  * Bessho and Sato (2021) T. Bessho and M. Sato, Phys. Rev. Lett. 127, 196404 (2021).
  * Chen  _et al._ (2023) W.-Q. Chen, Y.-S. Wu, W. Xi, W.-Z. Yi, and G. Yue, Journal of High Energy Physics 2023, 1 (2023).
  * Yao and Wang (2018) S. Yao and Z. Wang, Phys. Rev. Lett. 121, 086803 (2018).
  * Yao  _et al._ (2018) S. Yao, F. Song, and Z. Wang, Phys. Rev. Lett. 121, 136802 (2018).
  * Kunst  _et al._ (2018) F. K. Kunst, E. Edvardsson, J. C. Budich, and E. J. Bergholtz, Phys. Rev. Lett. 121, 026808 (2018).
  * Yokomizo and Murakami (2019) K. Yokomizo and S. Murakami, Phys. Rev. Lett. 123, 066404 (2019).
  * Yang  _et al._ (2020) Z. Yang, K. Zhang, C. Fang, and J. Hu, Phys. Rev. Lett. 125, 226402 (2020).
  * Borgnia  _et al._ (2020) D. S. Borgnia, A. J. Kruchkov, and R.-J. Slager, Phys. Rev. Lett. 124, 056802 (2020).
  * Okuma  _et al._ (2020) N. Okuma, K. Kawabata, K. Shiozaki, and M. Sato, Phys. Rev. Lett. 124, 086801 (2020).
  * Zhang  _et al._ (2020) K. Zhang, Z. Yang, and C. Fang, Phys. Rev. Lett. 125, 126402 (2020).
  * Bergholtz  _et al._ (2021) E. J. Bergholtz, J. C. Budich, and F. K. Kunst, Rev. Mod. Phys. 93, 015005 (2021).
  * Geng  _et al._ (2023) H. Geng, J. Y. Wei, M. H. Zou, L. Sheng, W. Chen, and D. Y. Xing, Phys. Rev. B 107, 035306 (2023).
  * Trambly de Laissardière  _et al._ (2010) G. Trambly de Laissardière, D. Mayou, and L. Magaud, Nano letters 10, 804 (2010).
  * Uryu (2004) S. Uryu, Phys. Rev. B 69, 075402 (2004).
  * Nakanishi and Ando (2001) T. Nakanishi and T. Ando, Journal of the Physical Society of Japan 70, 1647 (2001).
  * Timmel and Mele (2020) A. Timmel and E. J. Mele, Phys. Rev. Lett. 125, 166803 (2020).
  * Pereira  _et al._ (2009) V. M. Pereira, A. H. Castro Neto, and N. M. R. Peres, Phys. Rev. B 80, 045401 (2009).
  * (49) The deformation of the R layer slightly changes the intralayer nearest-neighbor hopping as well. This modification of intralayer hopping is also governed by Eq. (2), but its impact is negligible.
  * (50) See Supplemental Material at xxxx for the calculation of self-energies due to leads, derivation of the Bloch Hamiltonian for the bilayer system, numerical calculation of the transmission coefficients, valley polarization for different $W_{S}$, and the effect due to small twist, which includes Ref. Datta (1997); Nardelli (1999); Sancho  _et al._ (1985); Ando (1991); Wakabayashi (2000); Khomyakov  _et al._ (2005).
  * Oostinga  _et al._ (2008) J. B. Oostinga, H. B. Heersche, X. Liu, A. F. Morpurgo, and L. M. Vandersypen, Nature materials 7, 151 (2008).
  * Castro  _et al._ (2007) E. V. Castro, K. S. Novoselov, S. V. Morozov, N. M. R. Peres, J. M. B. L. dos Santos, J. Nilsson, F. Guinea, A. K. Geim, and A. H. C. Neto, Phys. Rev. Lett. 99, 216802 (2007).
  * Chen  _et al._ (2020) H. Chen, P. Zhou, J. Liu, J. Qiao, B. Oezyilmaz, and J. Martin, Nature communications 11, 1202 (2020).
  * Sui  _et al._ (2015) M. Sui, G. Chen, L. Ma, W.-Y. Shan, D. Tian, K. Watanabe, T. Taniguchi, X. Jin, W. Yao, D. Xiao, _et al._ , Nature Physics 11, 1027 (2015).
  * Shimazaki  _et al._ (2015) Y. Shimazaki, M. Yamamoto, I. V. Borzenets, K. Watanabe, T. Taniguchi, and S. Tarucha, Nature Physics 11, 1032 (2015).
  * Si  _et al._ (2016) C. Si, Z. Sun, and F. Liu, Nanoscale 8, 3207 (2016).
  * Bai  _et al._ (2020) Y. Bai, L. Zhou, J. Wang, W. Wu, L. J. McGilly, D. Halbertal, C. F. B. Lo, F. Liu, J. Ardelean, P. Rivera, _et al._ , Nature Materials 19, 1068 (2020).
  * Weidemann  _et al._ (2020) S. Weidemann, M. Kremer, T. Helbig, T. Hofmann, A. Stegmaier, M. Greiter, R. Thomale, and A. Szameit, Science 368, 311 (2020).
  * Ghatak  _et al._ (2020) A. Ghatak, M. Brandenbourger, J. Van Wezel, and C. Coulais, Proceedings of the National Academy of Sciences 117, 29561 (2020).
  * Zhang  _et al._ (2021a) X. Zhang, Y. Tian, J.-H. Jiang, M.-H. Lu, and Y.-F. Chen, Nature communications 12, 5377 (2021a).
  * Zhang  _et al._ (2021b) L. Zhang, Y. Yang, Y. Ge, Y.-J. Guan, Q. Chen, Q. Yan, F. Chen, R. Xi, Y. Li, D. Jia, _et al._ , Nature communications 12, 6297 (2021b).
  * Zhou  _et al._ (2023) Q. Zhou, J. Wu, Z. Pu, J. Lu, X. Huang, W. Deng, M. Ke, and Z. Liu, Nature Communications 14, 4569 (2023).
  * Andrei and MacDonald (2020) E. Y. Andrei and A. H. MacDonald, Nature materials 19, 1265 (2020).
  * Balents  _et al._ (2020) L. Balents, C. R. Dean, D. K. Efetov, and A. F. Young, Nature Physics 16, 725 (2020).
  * Andrei  _et al._ (2021) E. Y. Andrei, D. K. Efetov, P. Jarillo-Herrero, A. H. MacDonald, K. F. Mak, T. Senthil, E. Tutuc, A. Yazdani, and A. F. Young, Nature Reviews Materials 6, 201 (2021).
  * Kennes  _et al._ (2021) D. M. Kennes, M. Claassen, L. Xian, A. Georges, A. J. Millis, J. Hone, C. R. Dean, D. Basov, A. N. Pasupathy, and A. Rubio, Nature Physics 17, 155 (2021).
  * Lau  _et al._ (2022) C. N. Lau, M. W. Bockrath, K. F. Mak, and F. Zhang, Nature 602, 41 (2022).
  * Mak and Shan (2022) K. F. Mak and J. Shan, Nature Nanotechnology 17, 686 (2022).
  * Datta (1997) S. Datta,  _Electronic transport in mesoscopic systems_ (Cambridge university press, 1997).
  * Nardelli (1999) M. B. Nardelli, Phys. Rev. B 60, 7828 (1999).
  * Sancho  _et al._ (1985) M. P. L. Sancho, J. M. L. Sancho, J. M. L. Sancho, and J. Rubio, Journal of Physics F: Metal Physics 15, 851 (1985).
  * Ando (1991) T. Ando, Phys. Rev. B 44, 8017 (1991).
  * Wakabayashi (2000) K. Wakabayashi,  _Low-Energy Physical Properties of Edge States in Nano-Graphites_ , Ph.D. thesis, University of Tsukuba (2000).
  * Khomyakov  _et al._ (2005) P. A. Khomyakov, G. Brocks, V. Karpan, M. Zwierzycki, and P. J. Kelly, Phys. Rev. B 72, 035450 (2005).