# A quantum interferometer for quartets in superconducting three-terminal
Josephson junctions

Régis Mélin  Université Grenoble-Alpes, Institut Néel, BP 166, F-38042
Grenoble Cedex 9, France  CNRS, Institut Néel, BP 166, F-38042 Grenoble Cedex
9, France  Denis Feinberg  Université Grenoble-Alpes, Institut Néel, BP 166,
F-38042 Grenoble Cedex 9, France  CNRS, Institut Néel, BP 166, F-38042
Grenoble Cedex 9, France

###### Abstract

An interferometric device is proposed in order to analyze the quartet mode in
biased three-terminal Josephson junctions (TTJs), and to provide experimental
evidence for emergence of a single stationary phase, the so-called quartet
phase. In such a quartet-Superconducting Quantum Interference Device (quartet-
SQUID), the flux sensitivity exhibits period ${hc}/{4e}$, which is the
fingerprint of a transient intermediate state involving two entangled Cooper
pairs. The quartet-SQUID provides two informations: an amplitude that measures
a total “quartet critical current”, and a phase lapse coming from the
superposition of the following two current components: the quartet
supercurrent that is odd in the quartet phase, and the phase-sensitive
multiple Andreev reflection (phase-MAR) quasiparticle current, that is even in
the quartet phase. This makes a TTJ a generically ”$\theta$-junction”.
Evidence for phase-MARs plays against conservative scenarii involving
synchronization of AC Josephson currents, based on “adiabatic” phase dynamics
and RSJ-like models.

Introduction:  Multiterminal Josephson junctions (MTJs) Omelyanchouk1 ;
Omelyanchouk2 ; Omelyanchouk3 ; Omelyanchouk4  appear as a very fertile
evolution in the field of superconductivity. While unbiased MTJs offer
prospects as platforms for controllable topological properties vanHeck ;
Padurariu ; Nazarov1 ; Nazarov2 ; topo0 ; Feinberg1 ; Feinberg2 ; topo1 ;
topo2 ; topo3 ; topo4 ; topo4-bis ; topo5 ; Berry ; topo1-plus-Floquet ;
Levchenko1 ; Levchenko2 ; Akh2 ; Gavensky , biased MTJs reveal new channels
for both superconducting phase-sensitive and quantum mechanical DC currents,
as predicted by theory Cuevas-Pothier ; Houzet-Samuelsson ; Freyn ; Melin1 ;
Jonckheere ; biSQUID ; FWS ; Sotto ; engineering ; Nowak ; paperI ; paperII ;
long-distance ; PRB2022 ; Akh2 ; Pillet ; EPJB ; Pillet2 ; Nazarov-PRR ;
Nazarov-PRB-AM  and confirmed in experiments Lefloch ; Heiblum ; HGE ;
multiterminal-exp1 ; multiterminal-exp2 ; multiterminal-exp3 ; multiterminal-
exp4 ; multiterminal-exp5 ; multiterminal-exp6 ; multiterminal-exp7 ;
multiterminal-exp8 ; multiterminal-exp9 ; multiterminal-exp10 . A paradigm of
multiterminal Josephson junction Freyn  involves three superconductors biased
at the opposite voltages $0,V,-V$, this making the junction host Cooper
quartets Freyn . Those transient quartets are made of entangled pairs of
Cooper pairs and flowing from the unbiased terminal towards the two others
simultaneously. This voltage configuration ensures energy conservation, a
necessary condition for having DC Josephson currents. The quartet mechanism
goes together with emergence of a stationary phase combination of the three
terminal phases, the so-called quartet phase $\varphi_{Q}$. At the microscopic
level, the minimal process appearing in perturbation theory in the tunnel
amplitudes consists of four Andreev reflections. Quartets (as well as higher-
order multipairs such as sextets, octets, ….) therefore constitute a genuine
quantum mechanical mesoscopic phenomenon, not occurring in simple classical
Josephson arrays but instead in truly multiterminal junctions.

Besides this quartet supercurrent, another current component happens to depend
on the quartet phase. It originates from multiple Andreev reflections (MAR),
which promote quasiparticles across the superconducting gap $2\Delta$ with the
help of Cooper pair transfers, each one gaining energy $2eV$ MAR . New
channels open in a three-terminal Josephson junction (TTJ) Nowak , where all
pairs of terminals are simultaneously involved. Among those, specific
processes involve emission of quartets at zero energy but with phase
$\varphi_{Q}$: the energy cost for promoting a quasiparticle between two
terminals, say $S_{1}$, $S_{0}$ (with $V_{1}-V_{0}=V$), instead of
transferring a pair between terminals $S_{1}$, $S_{0}$, can be provided by
transferring a pair between terminals $S_{0}$, $S_{2}$ (with $V_{0}-V_{2}=V$)
plus absorbing simultaneously a quartet from ($S_{1}$, $S_{2}$) to $S_{0}$
(Figure 1). This quartet carries a phase $\varphi_{Q}$ and these MAR processes
become phase-dependent subgap quasiparticle currents Jonckheere . Detailed
calculations about the phase and voltage sensitivities of both quartet and
phase-MAR currents can be found in Refs. Sotto, ; Jonckheere2, . While quartet
supercurrents are truly nondissipative, the phase-MAR currents are
dissipative. Both of them depend on the control variables $(\varphi_{Q},V)$
but with different symmetries Jonckheere ; Sotto . Owing to time inversion
symmetry, the quartet and phase-MAR currents have to be antisymmetric with
respect to inverting both variables $\varphi_{Q}$ and $V$. The quartet current
is antisymmetric in phase and symmetric in voltage, but the phase-MAR current
is instead symmetric in phase and antisymmetric in voltage. This duality is
reminiscent of the tunnel junction treated by Josephson Josephson  in his
seminal work, concerning the DC current and the phase-sensitive quasiparticle
current. The latter is AC in a two-terminal junction, but can become DC in a
multiterminal one.

Figure 1: Diagram (a) pictures a quasiparticle promoted through the gap and a
Cooper pair (CP) transferred from $S_{1}$ to $S_{0}$, via two Andreev
reflections (AR). Diagram (b) pictures a quartet (Q) formed of two entangled
Cooper pairs, transferred from $S_{0}$ to $S_{1}$ and $S_{2}$ simultaneously,
with two AR and two crossed Andreev reflections (CAR). Diagram (c) pictures a
quasiparticle promoted through the gap, transferred from $S_{1}$ to $S_{0}$,
while a Cooper pair is instead transferred from $S_{0}$ to $S_{2}$. Diagram
(c) can be formally obtained by superimposing the lines of diagrams (a) and
(b), showing that phase-dependent MARs involve quartets.

Regarding experiments, an important question is about the interpretation of
transport anomalies observed when a TTJ is biased at the voltages $0,V,-V$
Lefloch ; Heiblum ; HGE ; multiterminal-exp8 ; multiterminal-exp9 ;
multiterminal-exp10 . A conservative explanation involves the synchronization
of AC Josephson currents flowing across each of the junctions polarized at $V$
and $-V$ respectively Jain . This mechanism is electromagnetic in nature and
it involves the impedance (or the photon modes) of the whole circuit including
the junction.

Minimal models involve an adiabatic dependence of the currents with time-
dependent phases, in a way similar to the standard treatment of Shapiro
resonances Tinkham . This can be done in the presence of an external
environment described by a circuit impedance, that includes the resistive part
of the junction itself, within RSJ-related models Jain . This qualitatively
accounts for the DC-current features observed in TTJs multiterminal-exp3 ;
multiterminal-exp2 ; multiterminal-exp4 ; multiterminal-exp8 ; multiterminal-
exp5 , but is not a proof of the physical relevance of such a description. For
instance, the zero-frequency current-current cross-correlations Heiblum  can
hardly receive interpretation in terms of the RSJ model, and quite specific
frequency dependence of the device external circuit impedance should be
advocated to interpret a recent four-terminal experiment as originating from a
RSJ model HGE . Still, complementary experiments would be important to
ascertain the mesoscopic nature of multipair processes.

A first requisite is the control over the quartet phase, which can be used to
prove coherence of the multipair supercurrent. Such a phase coherence might be
present in an extrinsic synchronization scenario, although hampered by
decoherence mechanisms due to the environment itself. On the contrary, phase
coherence of the quartets is expected to be much more robust. To go further in
the discrimination between extrinsic and intrinsic mechanisms, one must take
into account the high transparency of the junctions, necessary to produce a
mesoscopic multipair transport. The consequence is the existence of MAR
processes, which in the standard two-terminal case have no explanation but
with the help of subgap Andreev reflections, and thus go well beyond a
phenomenological RSJ modeling. Specifically, in MTJs, the observation of the
phase-sensitive MARs can be taken as evidence for truly mesoscopic processes
involving quartets, thus disproving any classical synchronization scenario.

In this work, we propose an interferometric scheme able to control the quartet
phase and, at the same time, reveal the phase-MAR component, thus proving both
the phase coherence of multipair processes and their truly subgap mesoscopic
nature.

Following Josephson’s discovery that a current must flow in an unbiased
junction and depends on the phase difference between the contacts Josephson ,
SQUID setups were invented in order to control and analyze this phase
sensitivity Tinkham . The flux dependence exhibits period ${hc}/{2e}$ that
directly proves supercurrents carried by Cooper pairs with charge $2e$.
Similarly, one expects that interferometry also helps elucidating the
mechanism of quartets in TTJs, in particular proving that they carry a charge
4e. Yet, this simple expectation meets a difficulty : a TTJ involves three
terminals, two of them being biased. This prevents from building a trivial
generalization of the original two-terminal SQUID which is fully
equipotential. Such a device must necessarily be different from those already
proposed for multijunctions at equilibrium multiterminal-exp1 ; biSQUID .

In this work, we describe a four-terminal scheme building a true quartet-
SQUID. The clue is to connect two TTJs in parallel by their unbiased as well
as their biased terminals, in order to close them in a double-TTJ loop. Cooper
pairs injected in the quartet-SQUID at voltage $V=0$ can cross either TTJ as
quartets, picking up the quartet phase of each TTJ, and recombine in the
common outputs at voltages $V$ and $-V$. The design encloses two loops instead
of one. Generalizing the standard SQUID argument in the presence of magnetic
flux shows that this imposes a difference between the quartet phases of the
two TTJs, thus achieving a perfect parallel with an ordinary SQUID.

This scheme allows analyzing the sensitivity of the quartet mode on voltage,
as a new control parameter for a DC supercurrent. Microscopic models show that
it is not monotonous, owing to nonadiabatic transitions between Andreev
levels. Moreover it can switch from a generic $\pi$-junction behavior
(perturbative and low voltage case) to a $0$-junction one. Such an evidence
goes beyond classical synchronization scenarii unless assuming ad hoc an
unlikely voltage (i.e. AC Josephson frequency) dependence of the circuit
impedance.

The proposed quartet-SQUID also allows exploiting the interplay between
quartets and phase-sensitive MARs. Separation between those two distinct
processes could in principle be achieved in ideally symmetrical TJJs. More
generally, the different phase symmetry of quartet and phase-sensitive MAR
currents results in a phase lapse in the periodic flux response of the
quartet-SQUID. Measuring this phase lapse quantifies the presence of phase-
MARs in transparent enough junctions. Phase-MARs are mesoscopic and they
involve quartet excitation amplitudes, therefore they bring the necessary
proof of a truly new physics being involved in TTJs.

Figure 2: Scheme of a quartet-SQUID based on two TTJs, with a main loop
threaded by a flux $\Phi$ and a secondary loop threaded by a flux $\phi$. The
current is injected into the large loop such as to make the two TTJs interfere
with each other. The current exits through the biased leads at voltages $V,-V$
of the branches $L,R$ of the secondary loop. The phases are mentioned in red
within a simple gauge convention, see text. Here $\varphi_{1M}=0$ and
$\varphi_{2M}=\Phi^{*}+{\phi^{*}}/{2}$.

Three-terminal junction quartet-SQUID: The principle of the quartet-SQUID is
to make two TTJs interfere with each other by joining their biased arms in a
secondary circuit, as pictured in Figure 2. The two TTJs thus enclose a
secondary loop with two branches respectively at the voltages $V$ (hereafter
denoted as “$L$-branch”), $-V$ (hereafter denoted as “$R$-branch”), threaded
by a flux $\phi$. The main loop is threaded by a flux $\Phi$. Both loops are
separated by the $L$ branch (see Figure 2). The total current is injected as
$I_{tot}=I_{1M}+I_{2M}$ where $I_{1M}$ and $I_{2M}$ are the currents entering
each of the TTJs from the unbiased branch, and eventually exiting in the
biased branches (second circuit) as $I_{L}$ and $I_{R}$. Current conservation
reads:

$I_{tot}=I_{1M}+I_{2M}=I_{L}+I_{R}$ (1)

Let us define the phases at the unbiased branch of TTJ$1$ and TTJ$2$ as
$\varphi_{1M},\varphi_{2M}$ respectively, and the phases at the biased
branches of the TTJs as $(\varphi_{1L},\varphi_{1R})$ and
$(\varphi_{2L},\varphi_{2R})$ respectively. From previous works Freyn  one
knows that the stationary quartet phase components are

$\varphi_{Qi}=\varphi_{iL}+\varphi_{iR}-2\varphi_{iM},$ (2)

while the oscillating phase components (at frequency ${4eV}/{\hbar}$) are
$\varphi_{iL}-\varphi_{iR}$ ($i=1,2$). Let us define the normalized fluxes
between $0$ and $2\pi$ as $\Phi^{*}=({2\pi}/{\phi_{0}})\Phi$ and
$\phi^{*}=({2\pi}/{\phi_{0}})\phi$, with $\phi_{0}={hc}/{2e}$. The fluxoid
argument is applied to the main loop containing the $L$ branch, then to the
main plus secondary loop containing the $R$ branch. This is perfectly allowed,
in spite of the main loop and the biased branches not being at the same
potential. In fact, the fluxoid argument takes care of the phase variation
inside each superconductor, whatever its potential. The supercurrent
circulation in the bulk of each superconductor is assumed to be zero as for a
thick superconductor, see Ref. Tinkham, . The presence of voltage biases
between the different superconductors only enters in the phase difference at
the junctions, that can depend on time in the present scheme, with frequency
${2eV}/{\hbar}$. The fluxoid argument Tinkham  amounts to equating on both
paths the sum of the phase differences at the junctions to the normalized flux
in the loop (modulo $2\pi$), which yields:

$\displaystyle\Phi^{*}$ $\displaystyle=$
$\displaystyle\varphi_{1L}-\varphi_{1M}+\varphi_{2M}-\varphi_{2L}$ (3)
$\displaystyle\Phi^{*}+\phi^{*}$ $\displaystyle=$
$\displaystyle\varphi_{1R}-\varphi_{1M}+\varphi_{2M}-\varphi_{2R}.$ (4)

Taking the difference between these two equations, one obtains a relation
between the oscillating phases components at the two TTJs:

$(\varphi_{1R}-\varphi_{1L})-(\varphi_{2R}-\varphi_{2L})=\phi^{*},$ (5)

expressing that these time-dependent components are perfectly synchronized. On
the other hand, taking the sum of Eqs. (3)-(4) yields a relation between the
quartet phases of the two TTJs [see Eq. (2)]:

$\varphi_{1Q}-\varphi_{2Q}=2(\Phi^{*}+{\phi^{*}}/{2})$ (6)

This central result shows that, like an ordinary SQUID, the interferometer
imposes a phase difference between the stationary quartet phases at the two
TTJs. Because of the ($L$, $R$) symmetry of the quartet current, the
corresponding flux is the arithmetic mean of the fluxes delimited by the $L$
(i.e. $\Phi^{*}$) and the $R$ (i.e. $\Phi^{*}+\phi^{*}$) branches.

Interestingly, if the TTJs are symmetric by exchanging their contacts to
branches $L,R$, the currents $I_{1,2M}$ entering the TTJs are pure quartet
currents i.e. $I_{1M}=I_{1Q}(\varphi_{1Q}),I_{2M}=I_{2Q}(\varphi_{2Q})$. In
turn, a pure MAR current $I_{L}-I_{R}$ flows between branches $L$ and $R$ thus
in the secondary circuit, and one can write:

$\displaystyle I_{L}$ $\displaystyle=$
$\displaystyle\frac{1}{2}\big{[}I_{1Q}(\varphi_{1Q})+I_{2Q}(\varphi_{2Q})\big{]}$
$\displaystyle+$ $\displaystyle I_{1MAR}(\varphi_{1Q})+I_{2MAR}(\varphi_{2Q})$
$\displaystyle I_{R}$ $\displaystyle=$
$\displaystyle\frac{1}{2}\big{[}I_{1Q}(\varphi_{1Q})+I_{2Q}(\varphi_{2Q})\big{]}$
$\displaystyle-$ $\displaystyle
I_{1MAR}(\varphi_{1Q})-I_{2MAR}(\varphi_{2Q}).$

In this case, a Cooper pair current circulates in the main loop, while the
secondary one contains a superposition of a quartet current – flowing as
parallel Cooper pair currents in the $L$ and $R$ branches, thus insensitive to
the flux $\phi^{*}$ – and a circulating MAR current – sensitive to $\phi^{*}$
via its phase-MAR component.

In the general case of asymmetric TTJs, all currents $I_{1,2M},I_{L,R}$
contain components of both quartet and MAR origins. One can carry the analysis
further in the simplifying case of weak transparencies. Noting that the
quartet and MAR components are respectively odd and even in the quartet
phases, and one can write:

$\displaystyle I_{1M}$ $\displaystyle=$ $\displaystyle
I_{Qc1}(V)\sin\varphi_{1Q}+\bar{I}_{MAR1}(V)+I_{MARc1}(V)\cos\varphi_{1Q}$ (9)
$\displaystyle I_{2M}$ $\displaystyle=$ $\displaystyle
I_{Qc2}(V)\sin\varphi_{2Q}+\bar{I}_{MAR2}(V)+I_{MARc2}(V)\cos\varphi_{2Q},$
(10)

where the first terms in Eqs. (9)-(10) are the quartet currents, with
“critical currents” $I_{Qci}$. The second terms are the phase-averaged MAR
components, including the phase-independent two-terminal MAR processes, and
the last terms contain the phase-sensitive MAR components, with “critical
currents” $I_{MARci}$. The critical current values defining the amplitude of
the phase oscillations of the quartet and MAR currents are voltage-sensitive,
and have in general nonmonotonous variations with $V$ Sotto ; paperI ; paperII
; PRB2022 ; Jonckheere2 . The sine and cosine dependences of the respective
quartet and MAR currents stem from their symmetry in phase. Such expressions
can be checked by microscopic calculations in the low transparency case Sotto
.

From Eqs. (1), (6), (9) and (10), the total current injected in this quartet-
SQUID can be written as (omitting the voltage sensitivities):

$\displaystyle I_{tot}$ $\displaystyle=$
$\displaystyle\bar{I}_{MAR1}+\bar{I}_{MAR2}+I_{c1}\sin(\varphi_{1Q}+\alpha_{1})$
(11) $\displaystyle+$ $\displaystyle
I_{c2}\sin\big{(}\varphi_{1Q}-2(\Phi^{*}+{\phi^{*}}/{2})+\alpha_{2}\big{)},$
(12)

with ($i=1,2$):

$\displaystyle I_{ci}$ $\displaystyle=$
$\displaystyle\sqrt{I_{Qci}^{2}+I_{MARci}^{2}}$
$\displaystyle\tan(\alpha_{i})$ $\displaystyle=$
$\displaystyle{I_{MARci}}/{I_{Qci}}.$ (13)

The total current appears as the sum of (i) a phase-independent MAR current
and (ii) a typical SQUID current, which depends on the quartet phase
$\varphi_{1Q}$, and on the effective flux $\Phi^{*}+{\phi^{*}}/{2}$, with
phase lapses $\alpha_{1,2}$ that measure the ratio of phase-sensitive MAR
currents to quartet currents. As in a usual SQUID, maximizing the total
current with respect to the (quartet) phase yields the following expression
for the critical current:

$\displaystyle I_{tot}$ $\displaystyle=$
$\displaystyle\bar{I}_{MAR1}+\bar{I}_{MAR2}$ $\displaystyle+$
$\displaystyle\Big{[}I_{c1}^{2}+I_{c2}^{2}+2I_{c1}I_{c2}\cos\big{(}2(\Phi^{*}+{\phi^{*}}/{2})+\alpha_{1}-\alpha_{2}\big{)}\big{]}^{1/2}.$

This relation achieves the goal of building a quartet-SQUID. As a first
result, the factor 2 in the flux sensitivity, that results in a ${hc}/{4e}$
periodicity, manifests the fact that quartets are made of two entangled Cooper
pairs and carry charge $4e$. Second, the phase lapses $\alpha_{1,2}$ directly
contain the information about the presence or not of phase-MARs. These phase
lapses disappear in the case of TTJs with symmetric branches $V,-V$
($\alpha_{1,2}=0$) or in the unlikely case of identical TTJs
($\alpha_{1}=\alpha_{2}$).

In experiments performed at low voltage and in incoherent diffusive regimes,
the MAR currents are negligible, and the quartet-SQUID gives direct access to
the pure quartet currents.

The above discussion is not restricted to harmonic dependences of the quartet
and MAR current with phase. In resonant dot models, nonharmonic behavior is
easily obtained and the quartet current can be quite large, actually
comparable to the ordinary Josephson current of a two-terminal junction in the
same conditions Jonckheere ; Sotto .

Exploring the voltage dependence: from “$0-$” to “$\pi-$” junction: Having a
quartet-SQUID in hands allows a thorough study of the dependence of the
quartet (and phase-MAR) currents with voltage, as a new control parameter for
DC Josephson currents. Focusing on the quartet current, different models,
suited to different types of junctions (single or many level quantum dot, or
diffusive metallic) lead to the same conclusions: the quartet current-phase
characteristics changes sign several times with voltage, owing to nonadiabatic
transitions between Andreev levels, triggered by the voltage via the running
phase $(\varphi_{L}-\varphi_{R})(t)$ at frequency ${4eV}/{\hbar}$ Sotto ;
paperI ; paperII ; PRB2022 ; Jonckheere2 . This means that, in terms of the
quartet current component, a TTJ can be either a “$0-$” or a “$\pi-$”
junction, with respect to the quartet phase $\varphi_{Q}$. The same occurs
with the phase-MAR current component that also changes sign but at different
voltages. Superposition of quartet and phase-MAR components actually makes a
generic TTJ a “$\theta$-junction”.

More generally, the characteristics of a TTJ (transparency, asymmetries
between the three contacts, degree of decoherence) all conspire to shift or
even suppress the sign changes. For instance, if the couplings to the biased
terminals are much smaller than the one to the unbiased terminal, and the
quartet TTJ current keeps a “$\pi-$” junction character at low and
intermediate voltages Jonckheere2 . Focusing on quartets only, in the case
where backgates allow to separately control the transparency of the different
contacts, one can reach a situation where, for a given voltage, the pair of
TTJs of the SQUID can be both “$0-$” (or both “$\pi-$”) junctions, or one
being a “$0-$” and the other a “$\pi-$” junction. This strongly reminds the
experiments performed with carbon nanotubes Wernsdorfer  (nanoSQUID) where the
mechanism for “$0-$” to “$\pi-$” transition is instead the Coulomb interaction
and the gate control of the nanotube junctions. In addition, “$0-$” to
“$\pi-$” transitions have also been observed in superconductor-ferromagnet-
superconductor Josephson junctions SFS .

To illustrate the possibilities of such a quartet-SQUID, let us assume that
TTJ1 is fully symmetric and resonant, with high quartet critical currents and
several sign changes as $V$ is increased from $0$ to $2\Delta$. On the
contrary, TTJ2 couples weakly but equally to the $L$, $R$ terminals. This
suppresses the MAR component in the SQUID current, and leaves us with a very
asymmetric quartet-SQUID, with (neglecting the anharmonicity in this example):

$I_{tot}(V)=I_{c1}(V)\sin(\varphi_{1Q})+I_{c2}(V)\sin\big{(}\varphi_{1Q}-2(\Phi^{*}+{\phi^{*}}/{2})\big{)}$
(15)

and $|I_{c1}(V)|>>|I_{c2}(V)|$. As said above, TTJ2 remains a “$\pi-$”
junction so that $I_{c2}<0$, while the sign of $I_{c1}$ depends on $V$.
Following the classical argument of an asymmetric SQUID, one first maximizes
$I_{tot}\sim I_{c1}\sin(\varphi_{1Q})$ with respect to $\varphi_{1Q}$, which
yields $\varphi_{1Q}\sim{\pi}/{2}$ if $I_{c1}>0$ and $\varphi_{1Q}\sim
3{\pi}/{2}$ if $I_{c1}<0$. Inserting this value into the – small – second term
of Eq. (15) yields

$I_{tot}\sim|I_{c1}|\pm I_{c2}\cos 2(\Phi^{*}+{\phi^{*}}/{2}),$ (16)

with $\pm$ sign depending on the “$0-$” or “$\pi-$” character of TTJ1. First,
this reconstructs the current-phase relation of TTJ2, including the sign of
$I_{c2}$. Second, as $V$ is swept upwards from $0$, the sign changes of TTJ1
reflect themselves in $\pi-$ shifts in the flux dependence of $I_{tot}$.

As another example, Eq. (A quantum interferometer for quartets in
superconducting three-terminal Josephson junctions) shows that phase-MARs can
be investigated in TTJ1 only, provided TTJ2 is symmetric in ($L$, $R$) thus
$\alpha_{2}=0$. The relative amplitude of phase-MARs and quartets in TTJ1
reflects directly in the shift $\alpha_{1}$ of the total current vs flux
dependence. In the generic case of non-symmetric TTJs, one instead measures
the difference $\alpha_{1}-\alpha_{2}$ of the two TTJ phase lapses.

An additional piece of information can also be gained by measuring the
currents in the biased loop, i. e. the combination $I_{L}-I_{R}$ which,
contrarily to $I_{tot}$, eliminates the quartet components at low transparency
and is thus sensitive to MAR currents only.

Conclusion: We have proposed a quartet-SQUID generalizing the standard SQUID
geometry to make quartet and phase-sensitive MAR current interfere under
control of a magnetic flux. The periodicity in the flux dependence of the
total critical current through the SQUID reflects the quartet charge $4e$. In
addition, the distinguishing phase symmetries of both current components imply
a phase lapse in the flux sensitivity of the critical current of the
interferometer, which allows to quantify the phase-MARs with respect to the
quartet current. Finally, phase-MARs are a consequence of both quartet
emission and coherent subgap transport. Thus, they provide evidence against
scenarii based on extrinsic synchronization via the outer circuit of junctions
described by an adiabatic current-phase relation. A full quantitative analysis
and comparison with future experiments requires to inject into the present
description the expressions for quartet and MAR currents of each TTJ, obtained
from microscopic theories.The principle of the present quartet-SQUID can
obviously be generalized to higher order multipair transport in MTJs with four
or more terminals.

R.M. acknowledges support from the French National Research Agency (ANR) in
the framework of the Graphmon project (ANR-19-CE47-0007).

## References

  * (1) R. de Bruyn Ouboter and A. Omelyanchouk, Multi-terminal squid controlled by the transport current, Physica B: Condensed Matter 205, 153 (1995).
  * (2) M. Amin, A. Omelyanchouk, and A. Zagoskin, Mesoscopic multiterminal Josephson structures. i. effects of nonlocal weak coupling, Low Temperature Physics 27, 616 (2001).
  * (3) M. Amin, A. Omelyanchouk, and A. Zagoskin, Dc squid based on the mesoscopic multiterminal Josephson junction, Physica C: Superconductivity 372, 178 (2002).
  * (4) M. Amin, A. Omelyanchouk, A. Blais, A. M. van den Brink, G. Rose, T. Duty, and A. Zagoskin, Multi-terminal superconducting phase qubit, Physica C: Superconductivity 368, 310 (2002).
  * (5) B. van Heck, S. Mi, and A.R. Akhmerov, Single fermion manipulation via superconducting phase differences in multiterminal Josephson junctions, Phys. Rev. B 90, 155450 (2014).
  * (6) C. Padurariu, T. Jonckheere, J. Rech, R. Mélin, D. Feinberg, T. Martin, and Yu.V. Nazarov, Closing the proximity gap in a metallic Josephson junction between three superconductors, Phys. Rev. B 92, 205409 (2015).
  * (7) R.-P. Riwar, M. Houzet, J.S. Meyer, and Y.V. Nazarov, Multi-terminal Josephson junctions as topological materials, Nat. Commun. 7, 11167 (2016).
  * (8) E. Eriksson, R.-P. Riwar, M. Houzet, J. S. Meyer, and Y. V. Nazarov, Topological transconductance quantization in a four-terminal Josephson junction, Phys. Rev. B 95, 075417 (2017).
  * (9) H.-Y. Xie, M.G. Vavilov and A. Levchenko, Topological Andreev bands in three-terminal Josephson junctions, Phys. Rev. B 96, 161406 (2017).
  * (10) H.-Y. Xie, M.G. Vavilov and A. Levchenko, Weyl nodes in Andreev spectra of multiterminal Josephson junctions: Chern numbers, conductances and supercurrents, Phys. Rev. B 97, 035443 (2018).
  * (11) O. Deb, K. Sengupta and D. Sen, Josephson junctions of multiple superconducting wires, Phys. Rev. B 97, 174518 (2018).
  * (12) B. Venitucci, D. Feinberg, R. Mélin, B. Douçot, Nonadiabatic Josephson current pumping by microwave irradiation, Phys. Rev. B 97, 195423 (2018).
  * (13) L.P. Gavensky, G. Usaj, D. Feinberg and C.A. Balseiro, Berry curvature tomography and realization of topological Haldane model in driven three-terminal Josephson junctions, Phys. Rev. B 97, 220505 (2018).
  * (14) R. L. Klees, G. Rastelli, J. C. Cuevas, and W. Belzig, Microwave Spectroscopy Reveals the Quantum Geometric Tensor of Topological Josephson Matter, Phys. Rev. Lett. 124, 197002 (2020).
  * (15) B. Douçot, R. Danneau, K. Yang, J.-G. Caputo and R. Mélin, Berry phase in superconducting multiterminal quantum dots, Phys. Rev. B 101, 035411 (2020).
  * (16) V. Fatemi, A.R. Akhmerov and L. Bretheau, Weyl Josepshon circuits, Phys. Rev. Research 3, 013288 (2021).
  * (17) L. Peyruchat, J. Griesmar, J.-D. Pillet and Ç.Ö Girit, Transconductance quantization in a topological Josephson tunnel junction circuit, Phys. Rev. Research 3, 013289 (2021).
  * (18) H. Weisbrich, R.L. Klees, G. Rastelli and W. Belzig, Second Chern Number and Non-Abelian Berry Phase in Topological Superconducting Systems, PRX Quantum 2, 010310 (2021).
  * (19) Y. Chen and Y.V. Nazarov, Weyl point immersed in a continuous spectrum: an example from superconducting nanostructures, Phys. Rev. B 104, 104506 (2021).
  * (20) Y. Chen and Y.V. Nazarov, Spin-Weyl quantum unit: theoretical proposal, Phys. Rev. B 103, 045410 (2021).
  * (21) E.V. Repin and Y.V. Nazarov, Weyl points in the multi-terminal Hybrid Superconductor-Semiconductor Nanowire devices, Phys. Rev. B 105, L041405 (2022)
  * (22) A. Melo, V. Fatemi and A.R. Akhmerov, Multiplet supercurrent in Josephson tunneling circuits, SciPost Phys. 12, 017 (2022)
  * (23) L. Peralta Gavensky, G. Usaj and C.A. Balseiro, Multiterminal Josephson junctions: a road to topological flux networks, arXiv:2211.12524 (2022).
  * (24) J. C. Cuevas and H. Pothier, Voltage-induced Shapiro steps in a superconducting multiterminal structure, Phys. Rev. B 75, 174513 (2007).
  * (25) M. Houzet and P. Samuelsson, Multiple Andreev reflections in hybrid multiterminal junctions, Phys. Rev. B 82, 060517(R) (2010).
  * (26) A. Freyn, B. Douçot, D. Feinberg, and R. Mélin, Production of non-local quartets and phase-sensitive entanglement in a superconducting beam splitter, Phys. Rev. Lett. 106, 257005 (2011).
  * (27) T. Jonckheere, J. Rech, T. Martin, B. Douçot, D. Feinberg, and R. Mélin, Multipair DC Josephson resonances in a biased allsuperconducting bijunction, Phys. Rev. B 87, 214501 (2013).
  * (28) J. Rech, T. Jonckheere, T. Martin, B. Douçot, D. Feinberg and R. Mélin, Phys. Rev. B 90, 075419 (2014).
  * (29) D. Feinberg, T. Jonckheere, J. Rech, T. Martin, B. Douçot and R. Mélin, Quartets and the current-phase structure of a double quantum dot superconducting bijunction at equilibrium, Eur. Phys. J. B 88, 99 (2015).
  * (30) R. Mélin, D. Feinberg, and B. Douçot, Partially resummed perturbation theory for multiple Andreev reflections in a short three-terminal Josephson junction, Eur. Phys. J. B 89, 67 (2016).
  * (31) R. Mélin, M. Sotto, D. Feinberg, J.-G. Caputo and B. Douçot, Gate-tunable zero-frequency current cross-correlations of the quartet mode in a voltage-biased three-terminal Josephson junction, Phys. Rev. B 93, 115436 (2016).
  * (32) R. Mélin, J.-G. Caputo, K. Yang and B. Douçot, Simple Floquet-Wannier-Stark-Andreev viewpoint and emergence of low-energy scales in a voltage-biased three-terminal Josephson junction, Phys. Rev. B 95, 085415 (2017).
  * (33) R. Mélin, R. Danneau, K. Yang, J.-G. Caputo, and B. Douçot, Engineering the Floquet spectrum of superconducting multiterminal quantum dots, Phys. Rev. B 100, 035450 (2019).
  * (34) Nowak, M. P., Wimmer, M., Akhmerov, A., Supercurrent carried by nonequilibrium quasiparticles in a multiterminal Josephson junction, Phys. Rev. B 99, 075516 (2019).
  * (35) J.D. Pillet, V. Benzoni, J. Griesmar, J.-L. Smirr, and Ç.Ö. Girit, Nonlocal Josephson Effect in Andreev Molecules Nano Lett. 19, 7138 (2019).
  * (36) V. Kornich, H.S. Barakov, and Yu.V. Nazarov, Fine energy splitting of overlapping Andreev bound states in multiterminal superconducting nanostructures, Phys. Rev. Research 1, 033004 (2019).
  * (37) R. Mélin, Inversion in a four terminal superconducting device on the quartet line. I. Two-dimensional metal and the quartet beam splitter, Phys. Rev. B 102, 245435 (2020).
  * (38) R. Mélin and B. Douçot, Inversion in a four terminal superconducting device on the quartet line. II. Quantum dot and Floquet theory, Phys. Rev. B 102, 245436 (2020).
  * (39) J.-D. Pillet, V. Benzoni, J. Griesmar, J.-L. Smirr, and Ç Ö Girit, Scattering description of Andreev molecules, SciPost Phys. Core 2, 009 (2020).
  * (40) V. Kornich, H. S. Barakov and Yu. V. Nazarov, Overlapping Andreev states in semiconducting nanowires: competition of 1D and 3D propagation, Phys. Rev. B 101, 195430 (2020).
  * (41) R. Mélin, Ultralong-distance quantum correlations in three-terminal Josephson junctions, Phys. Rev. B 104, 075402 (2021).
  * (42) R. Mélin, Multiterminal ballistic Josephson junctions coupled to normal leads, Phys. Rev. B 105, 155418 (2022).
  * (43) A.H. Pfeffer, J.E. Duvauchelle, H. Courtois, R. Mélin, D. Feinberg, and F. Lefloch, Subgap structure in the conductance of a three-terminal Josephson junction, Phys. Rev. B 90, 075401 (2014).
  * (44) E. Strambini, S. D’Ambrosio, F. Vischi, F.S. Bergeret, Yu.V. Nazarov, and F. Giazotto, The $\omega$-SQUIPT as a tool to phase-engineer Josephson topological materials, Nat. Nanotechnol. 11, 1055 (2016).
  * (45) Y. Cohen, Y. Ronen, J.H. Kang, M. Heiblum, D. Feinberg, R. Mélin, and H. Strikman, Non-local supercurrent of quartets in a three-terminal Josephson junction, Proc. Natl. Acad. Sci. U. S. A. 115, 6991 (2018).
  * (46) A.W. Draelos, M.-T. Wei, A. Seredinski, H. Li, Y. Mehta, K. Watanabe, T. Taniguchi, I.V. Borzenets, F. Amet, and G. Finkelstein, Supercurrent flow in multiterminal graphene Josephson junctions, Nano Lett. 19, 1039 (2019).
  * (47) K.F. Huang, Y. Ronen, R. Mélin, D. Feinberg, K. Watanabe, T. Taniguchi, and P. Kim, Evidence for 4e charge of Cooper quartets in a biased multi-terminal graphene-based Josephson junction, Nat. Comm. 13, 3032 (2022).
  * (48) N. Pankratova, H. Lee, R. Kuzmin, K. Wickramasinghe, W. Mayer,J. Yuan,M. Vavilov,J. Shabani and V. Manucharyan, The multi-terminal Josephson effect, Phys. Rev. X 10, 031051 (2020).
  * (49) G.V. Graziano, J.S. Lee, M. Pendharkar, C. Palmstrom and V.S. Pribiag, Transport Studies in a Gate-Tunable Three-Terminal Josephson Junction, Phys. Rev. B 101, 054510 (2020).
  * (50) E.G. Arnault, T. Larson, A. Seredinski, L. Zhao, H. Li, K. Watanabe, T. Taniguchi, I. Borzenets, F. Amet and G. Finkelstein, The multiterminal inverse AC Josephson effect, Nano Lett. 21, 9668 (2021).
  * (51) S.A. Khan, L. Stampfer, T. Mutas, J.-H. Kang, P. Krogstrup and T.S. Jespersen, Multiterminal Quantized Conductance in InSb Nanocrosses, Advanced Materials 33, 2100078 (2021).
  * (52) O. Kürtössy, Z. Scherübl, G. Fülöp, I. E. Lukács, T. Kanne, J. Nygard, P. Makk and S. Csonka, Andreev molecule in parallel InAs nanowires, Nano Lett. 21, 7929 (2021).
  * (53) G. V. Graziano, M. Gupta, M. Pendharkar, J. T. Dong, C. P. Dempsey, C. Palmstrøm and V. S. Pribiag, Selective control of conductance modes in multi-terminal Josephson junctions, Nat. Comm. 13, 5933 (2022).
  * (54) E.G. Arnault, S. Idris, A. McConnell, L. Zhao, T.F.Q. Larson, K. Watanabe, T. Taniguchi, G. Finkelstein, F. Amet, Dynamical stabilization of multiplet supercurrents in multi-terminal Josephson junctions, Nano Lett. 22, 7073 (2022).
  * (55) F. Zhang, A.S. Rashid, M.T. Ahari, W. Zhang, K.M. Ananthanarayanan, R. Xiao, G.J. de Coster, M.J. Gilbert, N. Samarth and M. Kayyalha, Andreev processes in mesoscopic multi-terminal graphene Josephson junctions, arXiv: 2210.04408v2 (2022).
  * (56) T. M. Klapwjik, G. E. Blonder, and M. Tinkham, Explanation of subharmonic energy gap structure in superconducting contacts, Physica B+C 109-110, 1657 (1982); M. Octavio, M. Tinkham, G. E. Blonder, and T. M. Klapwijk, Subharmonic energy-gap structure in superconducting constrictions, Phys. Rev. B 27, 6739 (1983).
  * (57) T. Jonckheere, J. Rech, C. Padurariu, L. Raymond, T. Martin, D. Feinberg, in preparation.
  * (58) B.D. Josephson, Possible new effects in superconductive tunnelling, Physics Letters 1, 251 (1962).
  * (59) Jain, A. K., Likharev, K. K., Lukens J. E., and Sauvageau, J. E., Mutual Phase-locking in Josephson junction arrays. Phys. Rep. 109, 309-426 (1984).
  * (60) M. Tinkham, Introduction to Superconductivity, 2nd ed. (McGraw-Hill, New York, 1996).
  * (61) J.-P. Cleuziou, W. Wernsdorfer, V. Bouchiat, T. Ondarcçuhu, and M. Monthioux, Carbon Nanotube Superconducting Quantum Interference Device, Nature Nanotech. 1, 53 (2006).
  * (62) W. Guichard, M. Aprili, O. Bourgeois, T. Kontos, J. Lesueur, and P. Gandit, Phase Sensitive Experiments in Ferromagnetic-Based Josephson Junctions, Phys. Rev. Lett. 90, 167001 (2003).


#  Empathy Through Multimodality in Conversational Interfaces

Mahyar Abbasian∗1, Iman Azimi1, Mohammad Feli2, Amir M. Rahmani1, Ramesh Jain1
1University of California, Irvine
2University of Turku ∗Corresponding author<EMAIL_ADDRESS>
###### Abstract

Agents represent one of the most emerging applications of Large Language
Models (LLMs) and Generative AI, with their effectiveness hinging on
multimodal capabilities to navigate complex user environments. Conversational
Health Agents (CHAs), a prime example of this, are redefining healthcare by
offering nuanced support that transcends textual analysis to incorporate
emotional intelligence. This paper introduces an LLM-based CHA engineered for
rich, multimodal dialogue—especially in the realm of mental health support. It
adeptly interprets and responds to users’ emotional states by analyzing
multimodal cues, thus delivering contextually aware and empathetically
resonant verbal responses. Our implementation leverages the versatile openCHA
framework, and our comprehensive evaluation involves neutral prompts expressed
in diverse emotional tones: sadness, anger, and joy. We evaluate the
consistency and repeatability of the planning capability of the proposed CHA.
Furthermore, human evaluators critique the CHA’s empathic delivery, with
findings revealing a striking concordance between the CHA’s outputs and
evaluators’ assessments. These results affirm the indispensable role of vocal
(soon multimodal) emotion recognition in strengthening the empathetic
connection built by CHAs, cementing their place at the forefront of
interactive, compassionate digital health solutions.

##  I Introduction

Human conversations transcend mere words, orchestrated as a multimedia
experience where tonal inflections, facial dynamics, and gestural semantics
are interwoven. These non-verbal cues enrich the emotional and contextual
semantics of our exchanges, serving a role analogous to metadata in digital
content. Echoing Socrates’ ancient apprehensions about written language, we
recognize the imperative to resurrect the soul of conversation within our
digital interactions.

The advent of mobile technology, replete with sophisticated biometric sensors
and capabilities for environmental data capture, has ushered in a
transformative shift in communication. Physiological signatures, measured
through technologies such as photoplethysmography, accelerometers, and
transdermal optical imaging, now provide integral data streams, enriching the
field of emotional analytics.

This integration of multimodal sensory data with computational intelligence,
especially when interfaced with cutting-edge Generative AI and Large Language
Models (LLMs), marks the dawn of a new era in human-computer interaction.
Harnessing complex pattern recognition and affective computing capabilities,
we envision digital agents capable of providing interactions as nuanced and
empathetically resonant as those between humans.

In multimedia computing, the challenge extends beyond crafting algorithms for
optimal information fidelity to engineering systems endowed with emotional
intelligence. The synergy of LLMs, sensor fusion algorithms, and context-aware
computing empowers us to create digital assistants that transcend information
delivery to offer genuine relational engagement, fostering trust and
personalized user experiences.

Our paper delves into the nexus of empathetic computing and multimedia
technology. We investigate the potential of combining LLMs with a suite of
contemporary sensing modalities, aiming to develop Conversational Health
Agents (CHAs) that redefine traditional paradigms of human-agent interaction.
Our goal is to endow these agents with the capacity to decipher and resonate
with emotional cues, thus initiating a new chapter in empathetic, human-
centric digital communication.

We believe that the multimedia technology community is ready to
reconceptualize digital dialogue’s future. Our rigorous experimentation and
innovation seek to close the gap between technological advancement and
authentic human experience, championing AI interactions replete with the depth
and empathy synonymous with human connection.

Recent studies have commenced the exploration of LLM-based solutions designed
to generate empathetic responses to users’ emotional cues. “CharacterChat”
provided a framework for social support in emotional distress [1]. Lei et al.
[2] introduced an LLM-based Emotionally Responsive Conversation (ERC) model,
utilizing a retrieval template module for contextual relevance. Similarly,
Zheng et al. fine-tuned LLaMA for emotional support dialogues [3], while Nie
et al. [4] developed a conversational AI therapist incorporating LLMs and
smart devices for mental health interventions.

Yet, these early LLM-based approaches are predominantly text-centric,
overlooking the vital speech and gestural modalities intrinsic to human
interaction. As a result, they fail to capture the full spectrum of contextual
and emotional information inherent in conversations, and their responses are
limited to textual formats. CHAs, however, stand at the vanguard of LLM and
Generative AI applications. Their multimodal capacities are pivotal in
navigating intricate user environments, fusing LLMs with diverse external data
and AI models to create a holistic experience.

We posit that CHAs have the capability to transcend LLM limitations in
conveying empathy by integrating a rich array of multimodal data
channels—including textual, speech, video (for facial and gesture analysis),
and physiological biomarkers (like heart rate variability). This paper is
dedicated to pioneering the integration of speech modalities to surmount these
challenges.

In this paper, we introduce an LLM-powered multimodal CHA, designed for rich
dialogues within mental health support contexts. This agent discerns emotional
cues from speech patterns to provide context-aware and empathetic verbal
responses. Utilizing the openCHA framework [5], we integrate an LLM with
speech-to-text, speech emotion detection, Internet search, and text-to-speech
tools. Our evaluation includes two stages: 1) the consistency and
repeatability of the planning capability and 2) inquiring questions with
varied emotional tones—sadness, anger, and joy and analyzing responses by
human evaluators in terms of empathetic resonance.

##  II Related Work

In this section, we present an overview of state-of-the-art conversational
methods that take emotions into account, including both conventional and LLM-
based approaches.

###  II-A Emotion Recognition in Conversation

There has been a growing interest in leveraging emotion recognition in
conversation (ERC) to support mental well-being [6]. For example, Morris et
al. [7] proposed a conversational agent aimed at providing empathetic support
to users. Their proposed system utilized preexisting emotional support
statements drawn from a large corpus of online interactions. In this
framework, users shared stressful situations and negative thoughts, receiving
feedback selected from the existing corpus of support interactions based on
similarity and user ratings. In another study, Adikari et al. [8] introduced
an conversational agent framework for real-time monitoring and co-facilitation
of mental health. This framework consisted of four components: patient
emotions analysis using natural language processing (NLP) techniques, group
emotion detection employing multiple machine learning approaches, the capture
of patient behavioral metrics according to the content shared by the patient
within a conversational setting, and a rule-based response generator.
Moreover, several studies [9, 10, 11] have delved into the utilization of
various machine-learning and deep-learning methods to develop ERC models for
conversational agents (chatbots).

In addition, studies have harnessed data from multiple modalities to enhance
ERC in conversational support systems [12]. Tavabi et al. [13] developed a
multimodal deep-learning approach leveraging textual, audio, and visual
features to identify opportunities when an agent should convey positive or
negative empathetic responses. These modalities were mapped to specific
feature representations and then fused for classification. This method
achieved an f1-score of 0.71 in discerning when empathetic responses should be
delivered by the agent to the user. In a study by Lian et al. [14], a
multimodal ERC framework was introduced utilizing a transformer-based
structure to model intra-modal and cross-modal interactions on word-level
lexical and acoustic features. The authors further presented two additional
multimodal ERC frameworks: a semi-supervised approach utilizing an auto-
encoder [15], and an attention-based bi-directional gated recurrent unit (GRU)
method [16].

Despite the advancements, these studies often struggle with open-ended
dialogues when the conversation flows freely without specific prompts or
questions. These methods are more tailored with predefined responses or
prompts, making them better suited for Q&A interactions in chatbots. Their
struggle to handle the complexity and unpredictability of open-ended
conversations may limit their effectiveness in providing sufficient mental
health support. Additionally, they tend to adopt a discriminative approach,
focusing on classification based on predefined categories or labels for
emotion recognition. While these methods can identify specific emotions in
isolated moments within a conversation, they may overlook emotional tendencies
and context that heavily rely on historical utterances. Such approaches
prioritize classification accuracy over understanding the emotional
expression, potentially leading to misinterpretations or incomplete responses.

Figure 1: LLM-based CHA for multimodal speech-based emotional support

###  II-B LLM-powered Conversational Methods

Recently, LLM-based solutions have been proposed for mental health support,
wherein these models aim to generate empathetic responses to users’ emotional
cues. Tu et al. [1] proposed CharacterChat, a personalized social support
conversation framework for individuals dealing with emotional troubles. Their
approach harnessed advanced language models, such as LLaMA [3], as the
response generation backbone, while employing BERT [17] as the memory
selection backbone, trained on a dataset created based on MBTI personality
types. This study emphasized the importance of interpersonal matching in
mental health conversational support systems. In a comparable approach, an
LLM-based ERC approach was introduced in [2], incorporating a retrieval
template module to ensure that the model considers the context of the
conversation. Additionally, subject identification and emotion prediction
tasks were integrated to model conversation flow and anticipate future
emotional tendencies. Zheng et al. [18] developed another emotional support
conversational system by fine-tuning LLaMA [3] on an emotional support
dialogue dataset created using ChatGPT. Furthermore, Nie et al. [4] introduced
a conversational AI therapist leveraging LLMs and smart devices to address
mental health challenges. This platform monitors day-to-day functioning and
provides psychotherapeutic interventions through reinforcement learning.

While these studies [19, 20, 21, 22, 23] have explored LLM-based emotional
support conversation for mental health, their objectives were mainly focused
on textual data within conversations. This limited focus neglects the
potential contributions of other modalities like audio, which can provide
valuable insights into users’ emotional states and enhance the effectiveness
of mental health support systems. Additionally, relying solely on text
communication can lead to ambiguity. Textual expressions of emotions might
often be ambiguous, making it challenging for models to accurately interpret
and respond to users’ emotional states. Moreover, textual responses are the
primary output of these systems and may not be the most effective means of
communication to users in all situations. In contrast to audio or visual
modalities, text lacks the richness of tone and other nonverbal cues that can
significantly convey empathy and understanding.

##  III LLM-powered Multimodal CHA

We develop an LLM-powered CHA aimed to offer multimodal emotional support
through conversations. This agent leverages LLMs and multimodal conversation
and emotion detection modules to interact with users via speech. To achieve
this, we adopt an agent-based framework, entitled openCHA [5]. Our proposed
CHA consists of an orchestrator that interact with these components to
generate empathetic responses based on the user’s emotion. For example, when
users inquire about mental health issues, the CHA assesses their emotional
state by analyzing vocal cues and tailors responses accordingly. These
responses are then delivered in speech format. Our proposed CHA comprises five
key components: Interface, Multimodal Conversation, Orchestrator, Multimodal
Emotion Detection, and Reliable Mental Health Sources (see Figure 1). We
outline these components in the following.

###  III-A Interface

The interface facilitates multimodal interactions through a web chat, offering
features for voice recording and playback. Recorded voice messages are sent to
the Multimodal Conversation component, where they are transcribed into text
for further analysis. Additionally, the final response is conveyed back to the
interface via the Multimodal Conversation component, where it is converted
into spoken voice.

###  III-B Multimodal Conversation

Multimodal Conversation component plays a crucial role in facilitating
multimodal communication within the CHA. It consists of two primary modules,
as speech-to-text and text-to-speech, to facilitate speech-enabled
conversations.

For speech-to-text, our tasks utilize the openAI’s whisper-base model [24].
Whisper is a versatile speech recognition model that stands out for its
ability to handle a range of speech processing tasks, including multilingual
recognition, translation, and language identification, thanks to its training
on a vast and varied dataset. It employs a Transformer sequence-to-sequence
framework, where tasks like speech recognition across multiple languages,
translation, language identification, and voice activity detection are
integrated into a single workflow.

Meanwhile, for text-to-speech, we leverage the gTTS GitHub library [25]. gTTS,
is a versatile Python library and command-line tool that connects to Google
Translate’s text-to-speech API. It features a customizable sentence tokenizer
tailored for speech, enabling it to read texts of any length while maintaining
correct intonation, handling abbreviations, decimals, and other nuances.
Additionally, it offers customizable text pre-processors that can adjust
pronunciation as needed.

###  III-C Orchestrator

The Orchestrator sits at the heart of our CHA, tasked with problem-solving,
devising action plans, and generating responses tailored to the user based on
their inquiries. This element collaborates with the Multimodal Emotion
Detection component to capture the most relevant user information related to
the query dynamically. It also collects data from the Reliable Mental Health
Sources component to secure the latest and most personalized information.
Moreover, by synthesizing the collected data and utilizing LLMs, it extracts
insights to formulate the user’s final response in text format. The
Orchestrator have four important capabilities as Planning, Execution, Short-
term Memory, and Response Generator.

The Planning capability, fueled by an LLM, acts as the decision-making and
cognitive nucleus of the Orchestrator. It is responsible to collate all the
information needed to resolve user inquiries effectively. This involves
analyzing the user’s question to identify the necessary steps for execution we
call them tasks.

Figure 2: Examples of developed CHA answering a user voice query with Sad (a)
and Happy (b) emotions

To convert user questions into actionable tasks, we employ the Tree of Thought
[26] prompting techniques for strategic planning. This method requires the LLM
to undertake three key actions: first, to devise three separate strategies,
each a sequence of tasks with specified inputs; second, to outline the pros
and cons of these strategies; and third, to determine the most suitable
strategy for the query at hand. For the implementation, we use OpenAI’s [27]
GPT-3.5-turbo model.

The Execution phase within the Orchestrator implements the tasks outlined
during the planning stage. These tasks encompass determining the user’s
current emotional state and conducting searches for information that align
with the user’s query and perceived emotional condition. The sequence in which
these tasks are carried out, along with the specifics of what information to
seek, are directed by the Planning capability of the Orchestrator.

The Short-term Memory acts as a storage for information gathered from the
Multimodal Emotion Detection and Reliable Mental Health Sources components
throughout conversational interactions, essential for enabling multimodality.
It holds onto intermediate data that might be too voluminous or complex for
the Planning LLM or the Response Generator’s LLM to process directly.

The Response Generator, powered by an LLM, utilizes information compiled by
the Planning component to craft clear and empathetic responses personalized to
the user. For this module, we employ OpenAI’s GPT-3.5-turbo model, serving as
the foundational LLM for the Response Generator.

For the Orchestrator implementation, we customize and leverage the
architecture presented in [28].

###  III-D Multimodal Emotion Detection

This component enables multimodality within LLM-healthcare integration,
enhancing trustworthiness, personalization, and empathy through data insight
extraction [29, 30]. Given the limitations of LLMs in extensive computations,
this component will facilitate emotion extraction from various sources like
video, audio, or biomarkers analysis. Our current implementation contains only
speech emotion recognition.

Our approach for identifying emotional states in speech utilizes the wav2vec2
[31] model, fine-tuned on the IEMOCAP dataset [32]. This model, known for
learning from speech audio to outperform existing semi-supervised methods with
greater simplicity, is applied to recognize emotions from speech. The IEMOCAP
dataset [32], enriched with facial expressions and hand movements data from
actors in various emotional scenarios, supports this fine-tuning. We adopt the
SpeechBrain [33] version of wav2vec2 [31], specifically adjusted with the
IEMOCAP [32] dataset, to serve as our speech emotion detection tool.

###  III-E Reliable Mental Health Sources

This component retrieves the latest and most relevant data from healthcare
sources like healthcare literature and reputable websites through search
engines [34, 35, 36, 37] to avoid hallucination and bias. We incorporated
Google Search API (called SerpAPI [38]) and Playwright [39] Extract Text to
conduct website searches based on user queries and extract relevant context to
accurately address users’ questions. For more details see [5].

##  IV Demonstration and Evaluation

In this section, we evaluate the performance of our proposed CHA in providing
empathetic responses to user queries based on the perceived emotion from their
voice. Our aim is to assess the CHA’s capability to tailor responses according
to the user’s current emotional state.

Figure 2 indicates two examples of how user interaction unfolds with our
system, detailing the process from initial voice query to the final audio
response tailored to the user’s emotional state. Initially, the user’s voice
query is captured and converted into text by the Speech To Text component.
This text is then forwarded to the Orchestrator, which coordinates the
planning and formulation of an appropriate response. The planning involves
using the Speech Emotion Recognition component to detect emotion from the
speech. Once the emotion is identified, it is used to retrieve relevant and
reliable answers from the Internet sources. These answers are then sent to the
Response Generator, which crafts the final response that is subsequently
converted back into audio for the user. Figure 2.a illustrates the response
generated when the user’s emotion is identified as “Sad,” directing them
towards serious support resources to help mitigate any potential harm. Figure
2.b, on the other hand, shows a different response suited to a “Happy”
emotional state, where the CHA uses a motivational tone and suggests resources
to help the user address the issue.

For the evaluation, we chose five neutral questions related to mental health
(see Table I). The questions were also tagged as neutral in tone by GPT-3.5.
Our evaluation of CHA includes two steps.

In the first step, we focused on the consistency and repeatability of our CHA
planning capability in task selection. This involved the planner’s ability to
accurately extract user emotion from voice data and to search and retrieve
information relevant to the user’s query. To do this, we input a randomly
selected question, chosen from the five voice questions infused with one of
three emotions, into the CHA. We repeated this process 500 times.

The performance of the planner is measured using two metrics. The first metric
examined how often, out of the 500 tests, the planner successfully identified
the emotional state from the voice and retrieved related information pertinent
to the user’s query. The second metric focused more narrowly on the planner’s
use of the extracted emotion to conduct Internet searches for relevant data.
We obtained the accuracy of %89 and %61 for the first and second metrics,
respectively.

TABLE I: List of five questions used for the evaluation

| I’ve noticed that I’ve been experiencing
---|---
Question 1 | some difficulty concentrating lately.
| Could this just be due to stress, or should
| I be concerned about something more?
| I’ve been feeling a bit more irritable than
Question 2 | usual lately, especially at work. Could this
| be a sign of burnout, or is it just a phase?
| I’ve been experiencing some difficulty
| sleeping, but I’m not sure if it’s related to
Question 3 | stress or if there could be other underlying
| causes. How can I determine the root cause?
| I’ve been feeling a bit disconnected from my
Question 4 | emotions lately. Are there any exercises or
| practices I can try to become more in tune
| with how I’m feeling?
| I’ve been feeling overwhelmed by the
| constant stream of negative news lately.
Question 5 | How can I maintain a healthy balance
| between staying informed and protecting
| my mental well-being?

Note that we observed that in cases where the extracted emotion was not used
to guide Internet searches, the planner still forwarded the emotional data
along with the search results to the response generator. The response
generator then used this information to craft an empathetic response. However,
when the planner did engage in more sophisticated, emotion-informed searching,
it performed a more targeted search based on both the user’s query and their
emotional state to fetch more personalized information. Regardless of the
approach, the final response was tailored to reflect the user’s emotional
state. Any deviations from these two defined planning paths were considered
incorrect.

TABLE II: Scores from human evaluators that reflect how well the responses align with and show empathy towards the user’s questions and Happy, Sad, and Angry emotional states | Happy | Sad | Angry
---|---|---|---
Question 1 | 6 | 8.3 | 6
Question 2 | 6.3 | 6.3 | 7.6
Question 3 | 5.3 | 6 | 5.3
Question 4 | 5.6 | 8.6 | 6.6
Question 5 | 8 | 7 | 7.3
Total Average | 6.24 | 7.24 | 6.56

In the second evaluation step, our goal is to measure how well the responses
matched the emotional state of the user and the level of empathy they
conveyed, given the identified question and extracted user emotion. We asked
the five questions to our CHA with three different emotional tones: Happy,
Sad, and Angry. This was done to assess how the emotional tone of a question
influences CHA’s responses.

Then, three external evaluators have reviewed each response, scoring each on
how well it aligned with the user’s current emotional state and its empathetic
quality on a scale from 0 to 10. A score of 0 meant there was no alignment or
empathy, and a score of 10 indicated a high degree of both. We then calculated
the average scores for each answer-emotion pair. These averages, which reveal
the performance of responses across each emotional category, are detailed in
Table II. The evaluators agreed that scores of 6 or higher indicated a
reasonable alignment. Additionally, they reported that responses to questions
posed in a state of sadness were more empathetic and aligned compared to those
in the other two emotional states.

Consequently, the effective planning capability and commendable evaluation
scores for responses indicate the success of the proposed CHA in delivering
empathetic answers based on user’s emotional state. Our future work will
extend the capabilities of CHAs to embrace a broader spectrum of computational
empathy. This research has laid the foundation by integrating the first two
major modalities of user interactions (i.e., text and voice) providing nuanced
emotional understanding and response. We will include other modalities to
capture facial and physiological cues and integrate them into CHA’s responses.
This integration will enable a more holistic empathetic communication
framework, driving us towards the objective of CHAs that can engage with and
support users with a depth and sensitivity akin to human caregivers.

##  V Conclusion

Our exploration into the realm of multimodal CHAs using LLMs offers a
promising avenue towards revolutionizing human-computer interaction. In this
paper, we introduced an LLM-powered multimodal CHA, tailored for in-depth
dialogues within health support environments. This agent was capable of
interpreting emotional cues from speech patterns to provide context-aware and
empathetic verbal responses. Employing the openCHA framework, we integrated an
LLM with speech-to-text, speech emotion detection, Internet search, and text-
to-speech tools. Our evaluation was conducted in two stages. We, first,
assessed the planning capabilities of the agent. Our findings showed that the
planner obtained %89 accuracy to identify the emotional state from the voice
and retrieve related information pertinent to the user’s query. It also
obtained %61 accuracy to correctly call the Internet searches tool based on
the emotion states. Then, we evaluated the responses in terms of empathy. We
posed questions with varied emotional tones (i.e., sadness, anger, and joy)
and analyzed the responses with the assistance of external human evaluators
for empathetic resonance. The external evaluators confirmed that the empathy
of the response had reasonable alignment with the three emotions. We observed
that responses to questions asked in a state of sadness were deemed more
empathetic and better aligned with user expectations compared to those in
other emotional states.

## References

  * [1] Q. Tu, C. Chen, J. Li, Y. Li, S. Shang, D. Zhao, R. Wang, and R. Yan, “Characterchat: Learning towards conversational ai with personalized social support,” _arXiv preprint arXiv:2308.10278_ , 2023.
  * [2] S. Lei, G. Dong, X. Wang, K. Wang, and S. Wang, “Instructerc: Reforming emotion recognition in conversation with a retrieval multi-task llms framework,” _arXiv preprint arXiv:2309.11911_ , 2023.
  * [3] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale _et al._ , “Llama 2: Open foundation and fine-tuned chat models,” _arXiv preprint arXiv:2307.09288_ , 2023.
  * [4] J. Nie, H. Shao, Y. Fan, Q. Shao, H. You, M. Preindl, and X. Jiang, “Llm-based conversational ai therapist for daily functioning screening and psychotherapeutic intervention via everyday smart devices,” _arXiv preprint arXiv:2403.10779_ , 2024.
  * [5] M. Abbasian _et al._ , “Conversational health agents: A personalized llm-powered agent framework,” _arXiv preprint arXiv:2310.02374_ , 2023.
  * [6] A. S. Raamkumar and Y. Yang, “Empathetic conversational systems: a review of current advances, gaps, and opportunities,” _IEEE Transactions on Affective Computing_ , vol. 14, no. 4, pp. 2722–2739, 2022.
  * [7] R. R. Morris, K. Kouddous, R. Kshirsagar, and S. M. Schueller, “Towards an artificially empathic conversational agent for mental health applications: system design and user perceptions,” _Journal of medical Internet research_ , vol. 20, no. 6, p. e10148, 2018.
  * [8] A. Adikari, D. De Silva, H. Moraliyage, D. Alahakoon, J. Wong, M. Gancarz, S. Chackochan, B. Park, R. Heo, and Y. Leung, “Empathic conversational agents for real-time monitoring and co-facilitation of patient-centered healthcare,” _Future Generation Computer Systems_ , vol. 126, pp. 318–329, 2022.
  * [9] S. Poria, N. Majumder, R. Mihalcea, and E. Hovy, “Emotion recognition in conversation: Research challenges, datasets, and recent advances,” _IEEE access_ , vol. 7, pp. 100 943–100 953, 2019.
  * [10] D. Bertero, F. B. Siddique, C.-S. Wu, Y. Wan, R. H. Y. Chan, and P. Fung, “Real-time speech emotion and sentiment recognition for interactive dialogue systems,” in _Proceedings of the 2016 conference on empirical methods in natural language processing_ , 2016, pp. 1042–1047.
  * [11] W. Ying, R. Xiang, and Q. Lu, “Improving multi-label emotion classification by integrating both general and domain-specific knowledge,” in _Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)_ , 2019, pp. 316–321.
  * [12] Y. Fu, S. Yuan, C. Zhang, and J. Cao, “Emotion recognition in conversations: A survey focusing on context, speaker dependencies, and fusion methods,” _Electronics_ , vol. 12, no. 22, p. 4714, 2023.
  * [13] L. Tavabi, K. Stefanov, S. Nasihati Gilani, D. Traum, and M. Soleymani, “Multimodal learning for identifying opportunities for empathetic responses,” in _2019 International Conference on Multimodal Interaction_ , 2019, pp. 95–104.
  * [14] Z. Lian, B. Liu, and J. Tao, “Ctnet: Conversational transformer network for emotion recognition,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 985–1000, 2021.
  * [15] ——, “Smin: Semi-supervised multi-modal interaction network for conversational emotion recognition,” _IEEE Transactions on Affective Computing_ , 2022.
  * [16] Z. Lian, J. Tao, B. Liu, and J. Huang, “Conversational emotion analysis via attention mechanisms,” _arXiv preprint arXiv:1910.11263_ , 2019.
  * [17] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [18] Z. Zheng, L. Liao, Y. Deng, and L. Nie, “Building emotional support chatbots in the era of llms,” _arXiv preprint arXiv:2308.11584_ , 2023.
  * [19] N. Madani, S. Saha, and R. Srihari, “Steering conversational large language models for long emotional support conversations,” _arXiv preprint arXiv:2402.10453_ , 2024.
  * [20] S. Lissak, N. Calderon, G. Shenkman, Y. Ophir, E. Fruchter, A. B. Klomek, and R. Reichart, “The colorful future of llms: Evaluating and improving llms as emotional supporters for queer youth,” _arXiv preprint arXiv:2402.11886_ , 2024.
  * [21] D. Kang, S. Kim, T. Kwon, S. Moon, H. Cho, Y. Yu, D. Lee, and J. Yeo, “Can large language models be good emotional supporter? mitigating preference bias on emotional support conversation,” _arXiv preprint arXiv:2402.13211_ , 2024.
  * [22] H. Zhang, Y. Chen, M. Wang, and S. Feng, “Feel: A framework for evaluating emotional support capability with large language models,” _arXiv preprint arXiv:2403.15699_ , 2024.
  * [23] T. Lai, Y. Shi, Z. Du, J. Wu, K. Fu, Y. Dou, and Z. Wang, “Supporting the demand on mental health services with ai-based conversational large language models (llms),” _BioMedInformatics_ , vol. 4, no. 1, pp. 8–33, 2023.
  * [24] OpenAI, “Whisper: Openai’s automatic speech recognition system,” https://openai.com/research/whisper, Accessed: Apr 2024.
  * [25] Pierre Nicolas Durette, “Google translator text-to-speech github python library.” https://github.com/pndurette/gTTS, Accessed: Apr 2024.
  * [26] S. Yao _et al._ , “Tree of thoughts: Deliberate problem solving with large language models,” _arXiv preprint arXiv:2305.10601_ , 2023.
  * [27] OpenAI, “Chatgpt: Openai’s conversational ai model,” https://openai.com/chatgpt, Accessed: Apr 2024.
  * [28] M. Abbasian and I. Azimi, “openCHA,” https://github.com/Institute4FutureHealth/CHA, Accessed: Apr 2024.
  * [29] E. J. Topol, “As artificial intelligence goes multimodal, medical applications multiply,” _Science_ , vol. 381, no. 6663, 2023.
  * [30] R. Han, J. N. Acosta, Z. Shakeri, J. Ioannidis, E. Topol, and P. Rajpurkar, “Randomized controlled trials evaluating ai in clinical practice: A scoping evaluation,” _medRxiv_ , pp. 2023–09, 2023.
  * [31] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” _Advances in neural information processing systems_ , vol. 33, pp. 12 449–12 460, 2020.
  * [32] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “Iemocap: Interactive emotional dyadic motion capture database,” _Language resources and evaluation_ , vol. 42, pp. 335–359, 2008.
  * [33] M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cornell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong, J.-C. Chou, S.-L. Yeh, S.-W. Fu, C.-F. Liao, E. Rastorgueva, F. Grondin, W. Aris, H. Na, Y. Gao, R. D. Mori, and Y. Bengio, “SpeechBrain: A general-purpose speech toolkit,” 2021, arXiv:2106.04624.
  * [34] H. Nori, Y. T. Lee, S. Zhang, D. Carignan, R. Edgar, N. Fusi, N. King, J. Larson, Y. Li, W. Liu _et al._ , “Can generalist foundation models outcompete special-purpose tuning? case study in medicine,” _arXiv preprint arXiv:2311.16452_ , 2023.
  * [35] D. Hiemstra, “Information retrieval models,” _Information Retrieval: searching in the 21st Century_ , pp. 1–19, 2009.
  * [36] H. R. Turtle and W. B. Croft, “A comparison of text retrieval models,” _The computer journal_ , vol. 35, no. 3, pp. 279–290, 1992.
  * [37] H. Nori, Y. Lee, S. Zhang _et al._ , “Can generalist foundation models outcompete special-purpose tuning? case study in medicine. arxiv. 2023.”
  * [38] SerpAPI, “Google search api,” https://serpapi.com, Accessed: Apr 2024.
  * [39] Playwright, “Playwright for python,” https://github.com/microsoft/playwright-python, Accessed: Apr 2024.