# Wave packet dynamics in a non-Hermitian exciton-polariton system

Y.-M. Robin Hu  ARC Centre of Excellence in Future Low-Energy Electronics
Technologies and Department of Quantum Science and Technology, Research School
of Physics, The Australian National University, Canberra, ACT 2601 Australia
Elena A. Ostrovskaya  ARC Centre of Excellence in Future Low-Energy
Electronics Technologies and Department of Quantum Science and Technology,
Research School of Physics, The Australian National University, Canberra, ACT
2601 Australia  Eliezer Estrecho  ARC Centre of Excellence in Future Low-
Energy Electronics Technologies and Department of Quantum Science and
Technology, Research School of Physics, The Australian National University,
Canberra, ACT 2601 Australia

###### Abstract

We theoretically investigate the dynamics of wave packets in a generic, non-
Hermitian, optically anisotropic exciton-polariton system that exhibits
degeneracies of its complex-valued eigenenergies in the form of pairs of
exceptional points in momentum space. We observe the self-acceleration and
reshaping of the wave packets governed by their eigenenergies. We further find
that the exciton-polariton wave packets tend to self-organize into the
eigenstate with the smaller decay rate, then propagate towards the minima of
the decay rates in momentum space, resulting in directional transport in real
space. We also describe the formation of pseudospin topological defects on the
imaginary Fermi arc, where the decay rates of the two eigenstate coincide in
momentum space. These effects of non-Hermiticity on the dynamics of exciton
polaritons can be observed experimentally in a microcavity with optically
anisotropic cavity spacer or exciton-hosting materials.

††preprint: APS/123-QED

##  I Introduction

Open dissipative systems described by non-Hermitian Hamiltonian operators
exhibit a special type of spectral degeneracies called exceptional points [1,
2, 3, 4, 5, 6], leading to the emergence of novel topological invariants [1,
2, 6], new topological states [1, 2, 3, 7, 8, 9], nontrivial lasing [4, 10,
11], non-reciprocal transmission [4, 10] and unidirectional transport [12,
13]. Microcavity exciton polaritons, created when the electron-hole pairs
(excitons) are strongly coupled to photons in an optical microcavity [6, 14,
15, 16], represent an accessible solid-state platform for studies of non-
Hermitian physics due to their inherent open-dissipative character. Non-
Hermitian spectral degeneracies, both in parameter and momentum space, as well
as the associated topological invariant, have been observed in this system [5,
6, 17].

In a Hermitian system, the motion of a wave packet is described by a pair of
semi-classical equations of motion, and its centre-of-mass motion in momentum
space is governed by external forces [18, 19, 20]. Recently, it was discovered
that a wave packet in a system described by a non-Hermitian Dirac model can
move in momentum space without the presence of an external force as a result
of the growths and decay of its components corresponding to different
eigenenergy branches [21]. The trajectories of the wave packets under this
self-acceleration are polarization-dependent and the centre-of-mass momenta
for certain initial conditions follow the gradient of the imaginary part of
the eigenenergy. Similar effects were also described in the context of an one-
dimensional non-Hermitian lattice [22]. In the absence of an out-of-plane
magnetic field, the model of exciton polaritons in a planar microcavity has a
band structure similar to the non-Hermitian Dirac model investigated in Ref.
[21], which is characterized by pairs of exceptional points connected by the
so-called Fermi arcs [6]. Therefore, we can expect similar wave packet
dynamics to emerge in a non-Hermitian optically anisotropic exciton-polariton
system.

In this work, we theoretically investigate the non-Hermitian wave packet
dynamics in a microcavity exciton-polariton system. Apart from the motion in
the absence of an external force, we also find that for some initial
conditions, the wave packets tend to split into multiple components that
propagate towards different directions. Moreover, these wave packets tend to
self-organize into different eigenstates as they evolve and propagate towards
the maxima of the imaginary part of the corresponding eigenenergy. Finally, we
examine the exciton-polariton pseudospin textures resulting from the wave
packet evolution and describe the emergent pseudospin anti-merons [23, 24, 25,
26, 27, 28] on the imaginary Fermi arc in momentum space. The anti-merons are
non-singular topological point defects that are characterized by half-integer
topological invariant. They have been studied both in real space [23, 29, 30,
28] and in momentum space [25, 31] in a variety of physical systems. Their
detection on the imaginary Fermi arc in an exciton-polariton system would
signify a clear signature of the non-Hermitian wave packet dynamics.

This work is organized as follows. In Section II, we present the non-Hermitian
exciton-polariton model considered in this work. In Section III.1, we discuss
the wave-packet self-acceleration and splitting in momentum space. In Section
III.2, we describe the asymptotic behaviour of the exciton-polariton wave
packets in momentum space and the unidirectional propagation in real space.
Finally, in Section III.3, we present our investigation on the dynamics of the
exciton-polariton pseudospins, including the formation of the pseudospin
defects on the imaginary Fermi arc without (Section III.3.1) and with (Section
III.3.2) the presence of an out-of-plane field, and the defects formation in
real space (Section III.3.3). In Appendix F, we compare the results for the
exciton-polariton model presented here to those of the non-Hermitian Dirac
model and the non-Hermitian Chern insulator.

##  II Theoretical Model

In this work, we focus on a non-Hermitian exciton-polariton model exhibiting
an effective non-Abelian gauge field. The general form of this 2$\times$2,
two-band Hamiltonian in momentum space is
$\mathbf{H}(\mathbf{k})={H}_{0}(\mathbf{k}){\mathbf{I}}+\overrightarrow{\mathbf{G}}(\mathbf{k})\cdot\overrightarrow{\sigma},$
where
$\overrightarrow{\mathbf{G}}(\mathbf{k})=[G_{x}(\mathbf{k}),G_{y}(\mathbf{k}),G_{z}(\mathbf{k})]$
is the gauge field, $\overrightarrow{\sigma}$ is a vector of Pauli matrices,
${\mathbf{I}}$ is the $2\times 2$ identity matrix, and
$\mathbf{k}=(k_{x},k_{y})$. The Hamiltonian is written in the basis of
circular polarization of the cavity photon or the spin projection of the
excitons on the $z$-direction normal to the plane of the exciton-hosting
material embedded in the microcavity. This basis forms the psuedospin of the
exciton polaritons. Dropping the $k$ dependence for brevity, the complex
energy spectrum can be written simply as $E_{\pm}=H_{0}\pm G$.

We model the exciton polaritons in an optical microcavity using the non-
Hermitian Hamiltonian presented in Ref. [6], which has the components:

$\begin{split}H_{0}(\mathbf{k})&=E_{0}-i\gamma_{0}+\frac{\hbar^{2}k^{2}}{2m}-i\chi{k}^{4}\\\
\overrightarrow{\mathbf{G}}(\mathbf{k})&=[\alpha-ia+(\beta-
ib)(k_{x}^{2}-k_{y}^{2}),2k_{x}k_{y}(\beta-ib),\Delta].\end{split}$ (1)

The expression for the complex energy can be written as:

$\begin{split}E_{\pm}=&E_{0}-i\gamma_{0}+\frac{\hbar^{2}k^{2}}{2m}-i\chi{k}^{4}\pm
G\\\ G=&\bigg{(}\Big{(}\alpha-ia+(\beta-ib)(k_{x}+ik_{y})^{2}\Big{)}\\\
&\times\Big{(}\alpha-ia+(\beta-
ib)(k_{x}-ik_{y})^{2}\Big{)}+\Delta^{2}\bigg{)}^{1/2}\end{split}$ (2)

where $G$ denotes the mean-subtracted eigenenergies. This model is a non-
Hermitian generalization of the one describing the microcavity exciton
polaritons in Ref. [32]. Here, $\alpha$ denotes the microcavity anisotropy
which leads to energy splitting between the linearly-polarized modes, $\beta$
describes the photonic spin-orbit coupling which splits the TE (transverse
electric) and TM (transverse magnetic) modes [32, 6]. The parameters $a$, $b$
describe the photonic losses that are polarization-dependent due to
microcavity anisotropy and the TE-TM splitting [6]. The presence of these
parameters results in the non-Hermitian nature of the Hamiltonian. The term
$\Delta\sigma_{z}$ in the Hamiltonian describes the effects of Zeeman
splitting induced by an out-of-plane magnetic field which can be introduced
into the experimental setup [33, 6]. The term $\frac{\hbar^{2}k^{2}}{2m}$
approximates the kinetic energies of the exciton polaritons, where $m$ is the
effective polariton mass. The two terms $-i\gamma_{0}-i\chi{k}^{4}$ ensure
that the imaginary part of the eigenenergies $\operatorname{Im}E_{\pm}$ are
always negative. This is required since this model describes the effects of
both gain (e.g., an optical pump) and loss (e.g., radiative decay) by an
effective loss term which determines the linewidth of the exciton-polariton
spectrum. Positive values of $\operatorname{Im}E_{\pm}$ would suggest a net
gain, which is not physical for this system.

Figure 1: The (a) real part and (b) imaginary part of the eigenenergies of
exciton-polariton model with cuts (denoted by the orange and blue lines) to
show the degeneracies and the branch cuts. The green dots mark the exceptional
points, the pink lines are the bulk Fermi arc where the two
$\operatorname{Re}E$ surfaces touch and the purple lines are the imaginary
Fermi arcs where the two $\operatorname{Im}E$ surfaces cross. (c): The
simplified structure of the bulk Fermi arcs, the imaginary Fermi arcs and the
exceptional points in momentum space. (d,e): The imaginary parts of the
eigenenergies at $k_{y}=0$ and $k_{x}=0$, respectively. The black dots
highlight the maxima of $\operatorname{Im}E_{\pm}$. The values of the
parameters used here and the rest of the work are described in Appendix A
unless specified otherwise.

The eigenenergies $E_{\pm}$ are plotted in Fig. 1(a,b). The two exciton-
polariton energy bands exhibit four exceptional points, each pair connected by
the bulk Fermi arc where the real energy surfaces cross, and by the imaginary
Fermi arc where the linewidth surfaces cross [see Fig. 1(a-c)] [6, 34]. A non-
zero Zeeman splitting $\Delta$ would shrink the bulk Fermi arc, moving the
exceptional points in a pair towards each other. A sufficiently strong Zeeman
splitting, $|\Delta|>|(a\beta-b\alpha)/\beta|$, will destroy the exceptional
points and open a gap. This shows that the exceptional points are more stable
than the Dirac points (Hermitian spectral degeneracies) which annihilate for
any non-zero value of $\Delta$.

The imaginary part of each eigenenergy branch has two local maxima. As shown
in Fig. 1(b) and highlighted in Fig. 1(d,e), $\operatorname{Im}E_{+}$
(represented by the orange surfaces and the orange lines) has two maxima lying
on the $k_{y}$-axis, while $\operatorname{Im}E_{-}$ (represented by the blue
surfaces and the blue lines) has two maxima at the $k_{x}$-axis. These points
play an important role in the dynamics of the exciton-polariton wave packets
as discussed in the next Section.

##  III Results and Discussion

Inspired by previous works [21, 22], we investigate the dynamics of Gaussian
wave packets in a non-Hermitian exciton-polariton systems in the absence of an
external potential. Since the Hamiltonian (1) contains $k$-components only,
i.e., no external potential or spatial components, the time evolution of wave
packets in real and momentum space and the resulting pseudospin textures can
be calculated exactly (see Appendix B for details).

The initial exciton-polariton wave packet is Gaussian in momentum space and
can be represented as a superposition of multiple $k$-components of the
pseudospin eigenstates. The time evolution of the contributions from these
components can be described as:

$\begin{split}|\psi^{R}_{\pm}(\mathbf{k},t)\rangle=e^{-i\operatorname{Re}E_{\pm}(\mathbf{k})t}e^{\operatorname{Im}E_{\pm}(\mathbf{k})t}|\psi^{R}_{\pm}(\mathbf{k})\rangle,\end{split}$
(3)

where the superscript $R$ denotes the right-eigenstates (see Appendix C for
details). The real part of energies $\operatorname{Re}E_{\pm}$ describes the
motion of the wave packet centre of mass in real space while the imaginary
part $\operatorname{Im}E_{\pm}$ governs the growth and decay of the
corresponding eigenstate. If $\operatorname{Im}E_{\pm}>0$,
$\operatorname{Im}E_{\pm}$ corresponds to the growth rate of the corresponding
eigenstate. Similarly, if $\operatorname{Im}E_{\pm}<0$,
$-\operatorname{Im}E_{\pm}$ corresponds to the decay rate. The variation in
the imaginary part, as shown below, is responsible for the peculiar effects
described in this work.

###  III.1 Self-Acceleration and Splitting of Exciton-Polariton Wave Packets

Similar to the dynamics of the wave packets in a non-Hermitian Dirac model
[21], we observe the acceleration of the exciton-polariton wave packets in the
absence of an external potential. Furthermore, the self-accelerating wave
packet trajectories are sensitive to the initial polarization as shown by the
centre-of-mass motion in Fig. 2(a-c). Note, however, that regardless of the
initial polarization, self-accelerating wave packets tend to move towards the
same point in momentum space or towards the same direction in real space. This
acceleration arises from the gradient of the imaginary part of eigenenergy
$\nabla\operatorname{Im}E_{\pm}$ [see Fig. 1(b,d,e)]. Namely, some
$k$-components are decaying faster than others, resulting in an effective
displacement of the wave packet center of mass towards $k$-components with
larger imaginary part (or lesser decay rates).

Surprisingly, when the exciton-polariton wave packet is initialized on the
imaginary Fermi arc, it splits into two components that propagate away from
each other as seen in Fig. 3(a-c). Furthermore, when the wave packet is
initialized at the origin $\mathbf{k}=0$, while overlapping with the two
imaginary Fermi arcs, it splits into four components. The four components
accelerate away from each other along the $\pm k_{x}$ and $\pm k_{y}$
directions [see Fig. 3(d-f)].

Figure 2: The polarization-dependent trajectories of the exciton-polariton
wave-packet centre-of-mass in (a,b) the momentum space and (b) the real space,
where (b) is the zoom-in of (a). Note that the trajectories of the
horizontally-polarized and the vertically-polarized modes overlap. The white
dots denote the initial wave packet centre of mass in both momentum and real
space and the black dots denote the maxima of $\operatorname{Im}E_{\pm}$ in
momentum space. Figure 3: (a-c): The splitting of the exciton-polariton wave
packet initially placed on the imaginary Fermi arc. (d-f): The splitting of
the exciton-polariton wave packet initially placed at the origin. The black
dots denote $\max(\operatorname{Im}E_{\pm})$. The plots and the colorbar
represent the evolution of the normalized probability density function (PDF).
Note that this is not equal to the wave packet amplitude $|\psi|^{2}$, which
decreases over time due to the continuous decay of the exciton polaritons.

The splitting of the wave packets is due to the different imaginary parts (or
decay rates) of the two pseudospin components. Since
$\operatorname{Im}E_{\pm}$ corresponds to the growth/decay rates of the
corresponding eigenstate, as time evolves, it is natural that the wave packet
will self-organize towards the eigenstate with the larger growth rate (or the
smaller decay rate) [see Fig. 3(a-f)]. In our model,
$\operatorname{Im}E_{+}(\mathbf{k})$ is larger inside the region bounded by
the Fermi arcs, while $\operatorname{Im}E_{-}(\mathbf{k})$ is larger outside
of this region. Therefore, in the process of evolution, the components of the
wave packets bounded by the Fermi arcs in momentum space will eventually self-
organize into the upper eigenstate $|\psi_{+}^{R}(\mathbf{k})\rangle$, while
the components outside the region tend to self-organize into the lower
eigenstate $|\psi_{-}^{R}(\mathbf{k})\rangle$. Hence, when the wave packet
splits in Fig. 3(a-c), the components belong to different eigenstates with the
one inside the region bounded by the Fermi arcs belonging to the
$|\psi_{+}^{R}(\mathbf{k})\rangle$, while the other to
$|\psi_{-}^{R}(\mathbf{k})\rangle$. In a similar manner, for the case shown in
Fig. 3(d-f), the two components inside (outside) the region bounded by the
Fermi arcs belong to $|\psi_{+}^{R}(\mathbf{k})\rangle$
($|\psi_{-}^{R}(\mathbf{k})\rangle$).

Note that since the difference in the imaginary parts of the two
eigenenergies, $\Delta\operatorname{Im}E$, is larger for large $k$ [see Fig.
1], the components of the wave packets with large $k$ will self-organize into
the two eigenstates faster than the components near $\mathbf{k}=0$.

Figure 4: Evolution of the normalized (a-c) $|c_{+}|^{2}$ and (d-f)
$|c_{-}|^{2}$ of the same wave packet, where the initial state of the wave
packet is an equal superposition of the two eigenstate
$|\psi\rangle=c_{+}|\psi_{+}^{R}\rangle+c_{-}|\psi_{-}^{R}\rangle$. (see
Appendix C for details.) The green dots denote the exceptional points.

###  III.2 Directional Propagation of Exciton-Polariton Wave Packets

After the wave packet self-organizes into the eigenstates with the larger
$\operatorname{Im}E$, its center-of-mass will then propagate towards the
maxima of the corresponding $\operatorname{Im}E$ in momentum space. In real
space, the wave packets will exhibit directional transport with a group
velocity corresponding to the $\nabla_{\mathbf{k}}\operatorname{Re}E$ at
$\mathbf{k}^{*}=\max(\operatorname{Im}E)$, similarly to the dynamics described
in Ref. [22].

In our model, $\operatorname{Im}E_{+}$ has two maxima at $(0,\pm k^{*}_{+})$
inside the region bounded by the Fermi arcs, while $\operatorname{Im}E_{-}$
has two maxima at $(\pm k^{*}_{-},0)$ outside this region [see Fig. 1(c)]. The
exciton-polariton wave packets that are inside the region bounded by the Fermi
arcs will self-organize into the upper eigenstate and their center-of-mass
momenta will reach the two fixed points at
$\mathbf{k}^{*}=(0,\pm\sqrt{b/2\chi})$. On the other hand, the wave packets
that are outside this region will self-organize into the lower eigenstate and
their center-of-mass momenta will reach the other two fixed points at
$\mathbf{k}^{*}=(\pm\sqrt{b/2\chi},0)$. In the real space, the wave packets in
the upper eigenstate will asymptotically reach non-vanishing group velocities
of

$\langle\mathbf{v_{+}}\rangle=\bigg{(}0,\pm\Big{(}\frac{\hbar}{2m}-\beta/\hbar\Big{)}\sqrt{\frac{2b}{\chi}}\bigg{)}$
(4)

while the wave packets in the lower eigenstate will reach group velocities of

$\langle\mathbf{v_{-}}\rangle=\bigg{(}\pm\Big{(}\frac{\hbar}{2m}-\beta/\hbar\Big{)}\sqrt{\frac{2b}{\chi}},0\bigg{)}.$
(5)

We note that these asymptotic group velocities of the exciton-polariton wave
packets only depend on the mean linewidth $\chi$, the spin-orbit coupling
parameter $\beta$, and the polarization-dependent loss $b$.

Figure 5: The time-evolution of a tight wave packet in (a-c) momentum space
and (d-f) real space. (g) and (h) are the fixed points coordinates as a
function $\Delta$ where $\mathbf{k}^{*}=(0,\pm k_{+}^{*}),(\pm k_{-}^{*},0)$.
(i) and (j) are the asymptotic group velocities where
$\langle\mathbf{v}_{+}\rangle=(0,\pm v_{+}^{*})$ and
$\langle\mathbf{v}_{-}\rangle=(\pm v_{-}^{*},0)$. The black lines in (i,j)
represent the group velocities
$\mathbf{v}=\nabla_{\mathbf{k}}\operatorname{Re}E_{\pm}$ calculated from
$\mathbf{k}^{*}$ in (g,h), which show excellent agreement with the results
from numerical simulation of wave packet dynamics (black dots).

Multiple extremum points $\mathbf{k}^{*}$ can be covered when an exciton-
polariton wave packet is tightly focused in real space, since it is broad in
momentum space. In this case, the wave packet will split into four parts with
each part funnelling to the vicinity of $\mathbf{k}^{*}$ that it covered. In
real space, the wave packet splits and spreads over time, eventually evolving
into four broad wave packets in the two eigenstates. Each of the smaller wave
packets exhibits directional transport with the corresponding non-vanishing
asymptotic group velocities $\langle\mathbf{v_{\pm}}\rangle$ [see Fig.
5(a-f)].

In the presence of Zeeman splitting, i.e., $\Delta\neq 0$, the fixed points
can be found numerically. Increasing $\Delta$ will move the fixed points
towards $\mathbf{k}=0$ [see Fig. 5(g,h)]. Since the fixed points stay on the
$k_{x}$ and $k_{y}$ axes, $\Delta$ will only change the magnitude of the final
group velocities, but not the directions. The final group velocities depend on
$\Delta$ non-monotonically, as seen in Fig. 5(i,j).

It is worth mentioning that the above results are derived under the assumption
that the exciton-polariton wave packet was initially in a superposition of the
two eigenstates,
$|\psi\rangle=c_{+}|\psi_{+}^{R}\rangle+c_{-}|\psi_{-}^{R}\rangle$, and have
nonzero $|c_{\pm}|$ at all of $\mathbf{k}^{*}$. In contrast, if the initial
state was in one of the eigenstate $|\psi^{R}_{j}\rangle$, where $j=+,-$, the
exciton-polariton wave packet will only propagate towards the corresponding
$\max(\operatorname{Im}E_{j})$. Furthermore, if the components of the
eigenstate $|c_{j}|^{2}$ in the initial state is zero at the corresponding
$\max(\operatorname{Im}E_{j})$, the wave packets will not funnel to
$\max(\operatorname{Im}E_{j})$ asymptotically. For example, when $\Delta=0$,
the upper eigenstate at the fixed points
$\mathbf{k}^{*}=(0,\pm\sqrt{b/2\chi})$ inside the region bounded by the Fermi
arcs is horizontally polarized. Therefore, if the initial state is vertically
polarized, which has no horizontal component, the exciton polaritons at
$\mathbf{k}^{*}=(0,\pm\sqrt{b/2\chi})$ cannot self-organize into
$|\psi_{+}^{R}\rangle$ as shown in Fig. 6(a-f). Consequently, the wave packet
will never funnel to those fixed points [see Fig. 6(g-i)]. Instead, the wave
packet will funnel to the other two fixed points at
$\mathbf{k}^{*}=(\pm\sqrt{b/2\chi},0)$ and exhibit directional transport in
real space at the constant velocities $\langle\mathbf{v_{-}}\rangle$.

Figure 6: Time evolution of the normalized $|c_{+}|^{2}$ (a-c) and
$|c_{-}|^{2}$ (d-f) when the initial wave packet is vertically-polarized. Also
showing the time evolution of the wave packet (g-i) with black dots denoting
$\mathbf{k}^{*}=\max(\operatorname{Im}E_{\pm})$ and the green dots denote the
exceptional points.

###  III.3 Dynamics of Exciton-Polariton Pseudospin Textures

The self-acceleration of the exciton-polariton wave packets is accompanied by
a non-trivial pseudospin dynamics. The exciton-polariton pseudospin can be
characterized by the Stokes vector $\mathbf{S}=[S_{x},S_{y},S_{z}]$ calculated
using the polarization components as described in Appendix D, and is analogous
to the electronic spins in a solid state system [29, 23, 24, 25, 26, 35]. We
note that the definitions of the in-plane Stokes vectors $[S_{x},S_{y}]$ can
be arbitrary, and the results in this Section are derived using the convention
specified in Appendix D.

As described above, the exciton-polariton wave packets tend to self-organize
into different eigenstates in different regions in momentum space. Since these
regions are separated by the Fermi arcs, it is natural to investigate what
will happen on these Fermi arcs. We find that, despite the pseudospin textures
of each eigenstate exhibiting discontinuous jumps on the bulk Fermi arcs, the
pseudospin textures of one eigenstate can smoothly transition to the
pseudospin textures of the other eigenstate across these arcs [see Fig.
7(a-c)]. On the other hand, the pseudospin textures of the two eigenstate are
both continuous at the imaginary Fermi arcs, but they point in the opposite
directions on these arcs [see Fig. 7(a-c)]. Furthermore, as the exciton-
polariton pseudospins self-organize into different eigenstates, we observe the
emergence of pseudospin defects resulting from the frustration.

Note that the evolution of the pseudospin textures and the emergence of the
defects depend only on the initial polarization of the wave packet, but not
the initial wave packet widths, center-of-mass momenta, or the center-of-mass
positions. Therefore, the pseudospin textures can evolve independently to the
probability density function of the wave packet. Therefore, the defects can be
generated even in the regions not covered by the bulk of the wave packet in
momentum space.

Figure 7: The pseudospin textures of (a) the upper eigenstate, (b) the lower
eigenstate and (c) an exciton-polariton wave packet at $t=3$ ps with
$\Delta=0$. (d) The pseudospin textures of the exciton-polariton wave packet
at $t=6$ ps. (e-h): The zoom-in near the pseudospin defects on the imaginary
Fermi arcs.

####  III.3.1 Pseudospin Defects

The pseudospin defects on the imaginary Fermi arcs are non-singular point
defects similar to skyrmions [36, 27, 24, 37, 29, 23, 25, 26, 38, 35, 31]. As
the wave packet evolves, these defects first emerge at the segments of the
imaginary Fermi arcs with large $k$, where the difference between the decay
rates of the two bands, $\operatorname{Im}(\Delta E)$, is the largest. Then,
more defects are generated at segments with smaller $k$ on the imaginary Fermi
arc where $\operatorname{Im}(\Delta E)$ is smaller. The defects then move
towards the exceptional points along the imaginary Fermi arc. This results in
the defects becoming more densely packed as more defects are generated.

Topological point defects can be characterized by a winding number called the
skyrmion number, $N_{sk}$, which counts how many times the pseudospin wraps
around the unit sphere [23, 24, 25]:

$\begin{split}N_{sk}&=\frac{1}{4\pi}\iint\mathbf{S}\cdot\Big{(}\partial_{x}\mathbf{S}\times\partial_{y}\mathbf{S}\Big{)}d^{2}\mathbf{r}.\end{split}$
(6)

The skyrmion number can also be expressed in terms of two other topological
invariants, the vorticity $w$ and the polarity $p$, as $N_{sk}=wp/2$. The
vorticity measures the in-plane winding of the Stoke vector $\mathbf{S}$,
where the angle of the in-plane Stoke vectors $[S_{x},S_{y}]$ changes by $2\pi
w$ along a closed curve encircling the core counter-clockwise [27, 25, 28,
39]. For example, a spin vortex has a vorticity of $w=+1$, while an anti-
vortex has a vorticity of $w=-1$ [39]. On the other hand, the polarity
measures the out-of-plane winding of $\mathbf{S}$ and quantifies the
continuous change of $S_{z}$ from the centre to the edge of the defect [27,
28, 25, 38, 39, 37]. The Stoke vectors $\mathbf{S}$ in a skyrmion exhibits
both the in-plane winding as well as the out-of-plane winding. The in-plane
Stoke vectors of a skyrmion are similar to the ones of a vortex, resulting in
a vorticity of $w=+1$. Furthermore, a core-up (core-down) skyrmion has an out-
of-plane Stoke vector $S_{z}=+1$ ($S_{z}=-1$) at the centre, which
continuously transforms into $S_{z}=-1$ ($S_{z}=+1$) at the edge, giving it a
polarity of $p=+2$ ($p=-2$). [36, 27, 24, 37, 29, 23, 25, 26, 38, 35].
Consequently, a core-up (core-down) skyrmion has a skyrmion number of
$N_{sk}=+1$ ($N_{sk}=-1$). Similarly, an anti-skyrmion has in-plane Stoke
vectors $[S_{x},S_{y}]$ similar to the ones in an anti-vortex and a core-up
(core-down) anti-skyrmion has a polarity of $p=+2$ ($p=-2$), giving it a
skyrmion number of $N_{sk}=-1$ ($N_{sk}=+1$) [25].

We observe core-up and core-down defects where the pseudospins lie on the
$xy$-plane, that is $S_{z}=0$, on the edge of these defects [See Fig. 7(d-h)].
These defects have vorticities of $w=-1$ and the $z$-components of their
pseudospins only perform a $\pi/2$-rotation instead of a $\pi$-rotation [23].
Consequently, these core-up (core-down) defects have polarities of $p=+1$
($p=-1$), giving them skyrmion numbers of $N_{sk}=-1/2$ ($N_{sk}=+1/2$). We
therefore conclude that they are core-up and core-down anti-merons, i.e.,
anti-skyrmions with half-integer winding numbers. In the vicinity of the
exceptional points in the upper-half plane ($k_{y}\geq 0$), core-up and core-
down anti-merons form on the right-hand side ($k_{x}\geq 0$) and left-hand
side ($k_{x}\leq 0$) of the exceptional points pair, respectively. On the
other hand, in the vicinity of the exceptional points in the lower-half plane
($k_{y}\leq 0$), core-up and core-down anti-merons form on the left-hand side
($k_{x}\leq 0$) and right-hand side ($k_{x}\geq 0$) of the exceptional points
pair, respectively. We also observe pseudospin vortices with core polarization
$S_{z}=+1$ ($S_{z}=-1$) forming between the core-down (core-up) anti-merons on
the imaginary Fermi arcs [see Fig. 7(d-h)]. The in-plane pseudospins
$[S_{x},S_{y}]$ wind around the vortex cores, giving them vorticities of
$w=+1$. However, the pseudospin $S_{z}=\pm 1$ at their cores does not
transform into $S_{z}=\mp 1$ or $S_{z}=0$ on the edge. Consequently, they do
not have $p=\pm 2$ like a skyrmion or $p=\pm 1$ like a meron, and therefore do
not have integer-valued or half-integer-valued skyrmion numbers.

####  III.3.2 Pseudospin Defects with Zeeman Splitting

Although the Zeeman splitting will not drastically change the in-plane
textures $[S_{x},S_{y}]$ of the defects, it has a huge impact on the
$S_{z}$-textures of the eigenstates and the defects. As the strength of the
Zeeman splitting increases, the pseudospin textures of one eigenstate will
gain more positive $S_{z}$ contribution, while those of the other eigenstate
will gain more negative $S_{z}$ contribution. As seen in Fig. 8, as $|\Delta|$
increases, $S_{z}$ becomes more negative on one side of the imaginary Fermi
arc, and more positive on the other side. Consequently, as $|\Delta|$
increases, some of the defects will be slightly pushed away from the imaginary
Fermi arcs. Which defects are pushed away from the imaginary Fermi arcs
depends on the initial polarization of the exciton-polariton wave packet as
well as the sign of the Zeeman splitting. For example, when the initial wave
packet is in a circularly polarized mode, $\psi^{+}$, the core-down anti-
merons and vortices are pushed away from the imaginary Fermi arcs, while the
core-up defects stay on the arcs. As shown in Fig. 8(a-c), a positive Zeeman
splitting pushes the core-down defects out of the regions bounded by Fermi
arcs as they gain more negative $S_{z}$ in this region. Similarly, as seen in
Fig. 8(d-f), a negative Zeeman splitting pushes the core-down defects into the
regions bounded by Fermi arcs as the defects gain more negative $S_{z}$. If
the wave packet is initially in the other circularly polarized mode,
$\psi^{-}$, the opposite would happen, and the core-down defects would stay on
the arc while the core-up defects would be pushed away.

Figure 8: Pseudospin textures for (a-c) positive and (d-f) negative values of
Zeeman splittings. Also showing the plots of the pseudospin defects in (h) a
gapless phase with non-zero $\Delta$, (i) the phase where the two exceptional
points merge and (j) a fully gapped phase. The wave packets used in these
panels were initially circularly-polarized.

Another consequence of the Zeeman splitting is that the core-up and core-down
defects also gain more positive and negative $S_{z}$ contribution,
respectively. If $|\Delta|$ continues to increase, eventually the pseudospin
on the edge of the anti-merons will no longer lie on the $xy$-plane.
Consequently, $S_{z}$ in these defects will no longer perform $\pi/2$
rotations with respects to the $xy$-plane, and they lose their anti-meron
character as well. This is different from the self-acceleration, splitting and
directional transport of the wave packets, which can persist at large
$|\Delta|$.

####  III.3.3 Pseudospin Defects in Real Space

The pseudospin anti-merons on the imaginary Fermi arc in momentum space can
also be measured in the real space. When the wave packet covers the anti-
merons in momentum space, the anti-meron textures also emerge in the real
space [see Fig. 9]. We should emphasise that the emergence of these anti-
merons is a direct consequence of the frustration of pseudospins on the
imaginary Fermi arc as the wave packet self-organizes into different
eigenstates. This is in contrast to the previous studies of purely Hermitian
two-dimensional systems, where the topological defects emerge due to the
cavity anisotropy and TE-TM splitting [23, 30, 29, 28]. Our results show a new
way to generate topological defects in the psuedospin structures that arise
from the growth and decay of wave packets in a non-Hermitian system. These
defects also represent a clear signature of the effects of the imaginary Fermi
arc on the exciton-polariton dynamics in the real space.

Figure 9: An exciton-polariton wave packet and its pseudospin textures in the
momentum (a,b) and the real space (c,d).

Moreover, our preliminary estimates indicate that these defects can be
observed experimentally in an exciton-polariton system based, e.g., on a
perovskite embedded in a high quality microcavity [6], which would serve as a
clear signature of the non-Hermitian effects on the wave packet dynamics (see
Appendix E).

##  IV Conclusion and outlook

To conclude, we have investigated the dynamics of wave packets in a non-
Hermitian exciton-polariton model. We found that the wave packets tend to
self-organize into the eigenstate with the larger imaginary part of energy
$\operatorname{Im}E$, then propagate towards the momentum with the
corresponding $\max(\operatorname{Im}E)$. This results in splitting of the
wave packets and a directional transport. We also observed core-up and core-
down anti-merons on the imaginary Fermi arc in momentum space, which are
destroyed by a sufficiently large Zeeman splitting.

However, there are still many open questions. It is yet unclear if the
observables in the propagating wave packets, including the width, the centre-
of-mass momentum and centre-of-mass positions, have simple analytic dependence
on time, i.e., if they can be effectively described by a semi-classical
equation of motion [19, 18, 20]. Furthermore, despite the fact that the
trajectories of the exciton-polariton wave packets are aligned with the
gradient of the loss rate in the system $\nabla\operatorname{Im}E$, this
quantity does not always correctly describe the wave-packet trajectories under
self-acceleration (see Appendix F). The origin of the pseudospin defects and
their connections to the Fermi arcs also require further investigation.

Our results shed new light on the dynamics of wave packets and pseudospins in
a non-Hermitian exciton-polariton system in momentum space as well as
transport in real space. Furthermore, we believe that the emergent momentum
space anti-meron textures in exciton-polariton pseudospins will enable the
direct observation of the influence of the imaginary Fermi arcs on the
dynamics of the system. All of the effects described above do not require an
external potential and can be measured in an exciton-polariton system with
optical anisotropy arising from the material properties of the semiconductor,
e.g., perovskite [6] and the organics [17], or in cavity photon systems with
an anisotropic spacer [40, 41]. Our results highlight the excellent potential
of exciton polaritons as a platform to study non-Hermitian dynamics.

###### Acknowledgements.

We acknowledge support from the Australian Research Council (ARC) through the
Centre of Excellence Grant CE170100039 and the Discovery Early Career
Researcher Award DE220100712, and Australian Government Research Training
Program (RTP) Scholarship.

##  Appendix A Values of Parameters

For the figures presented in this work, we chose the parameters such that the
exceptional points are well separated and the bulk Fermi arcs can be clearly
seen by the readers. We set $\chi=3.75\times 10^{-4}$ $\mu$m4eV, and choose
the mean energy, linewidth and effective polariton mass to be
$E_{0}-i\gamma_{0}=2.306-4.5\times 10^{-4}i$ eV and $\hbar^{2}/2m\approx
2.3\times 10^{-3}$ $\mu$m2eV. We then set the optical anisotropy and the
photonic spin-orbit coupling to be $\alpha=8\times 10^{-3}$ eV and
$\beta=10^{-3}$ $\mu$m2eV. We then set the photonic losses to be $a=10^{-5}$
eV, $b=7.5\times 10^{-4}$ $\mu$m2eV and assume there is no Zeeman splitting,
$\Delta=0$ eV. Using this set of parameters, the exceptional points have
momenta of $k\approx 2.5$ $\mu$m-1 and each pair is separated by $\approx 1.6$
$\mu$m-1 in momentum space. In a realistic system such as the perovskite-based
exciton polaritons described in Ref. [6], the exceptional points would have
momenta of $k\approx 6.7$ $\mu$m-1 and each pair is separated by $\approx
0.24$ $\mu$m-1 in momentum space, which would be difficult for the readers to
observe.

##  Appendix B Time-Evolution Operator in Exciton-Polariton Model

We consider a general two-band non-Hermitian Hamiltonian of the form

$\mathbf{H}(\mathbf{k})={H}_{0}(\mathbf{k}){\mathbf{I}}+\overrightarrow{\mathbf{G}}(\mathbf{k})\cdot\overrightarrow{\sigma},$

where
$\overrightarrow{\mathbf{G}}(\mathbf{k})=[G_{x}(\mathbf{k}),G_{y}(\mathbf{k}),G_{z}(\mathbf{k})]$
is the gauge field, $\overrightarrow{\sigma}$ is a vector of Pauli matrices,
and ${\mathbf{I}}$ is the $2\times 2$ identity matrix. Dropping the
$\mathbf{k}$ dependence for brevity, the time-evolution operator can then be
written as

$e^{-i\mathbf{H}t/\hbar}=e^{-i{H_{0}}t/\hbar}\left({\mathbf{I}}\cos{\frac{Gt}{\hbar}}-i\frac{\overrightarrow{\mathbf{G}}}{G}\cdot\overrightarrow{\sigma}\sin{\frac{Gt}{\hbar}}\right),$
(7)

where $G=\sqrt{G_{x}^{2}+G_{y}^{2}+G_{z}^{2}}$. Note that the complex energy
spectrum can be written simply as $E_{\pm}=H_{0}\pm G$.

We should also mention that this formalism does not hold at the exceptional
points where the Hamiltonian can no longer be diagonalized and the two
eigenstates will coalesce. However, since this formalism holds in the vicinity
of the exceptional points as well as for the rest of momentum space, we
believe that it is an accurate approximation for the time-evolution of the
exciton-polariton wave packets.

##  Appendix C Overlap with Eigenstates

In a quantum system described by a non-Hermitian Hamiltonian operator, the
right and left eigenstates are generally not equal and can be written as [1,
2]:

$\begin{split}\mathbf{H}|\psi^{R}_{\pm}\rangle&=E_{\pm}|\psi_{\pm}^{R}\rangle\\\
\mathbf{H}^{\dagger}|\psi^{L}_{\pm}\rangle&=E^{*}_{\pm}|\psi_{\pm}^{L}\rangle.\end{split}$
(8)

Instead of the usual orthonormal condition, the eigenstates obey the
biorthogonal condition

$\langle\psi_{i}^{L}|\psi_{j}^{R}\rangle=\delta_{i,j}.$ (9)

Therefore, when the system is in a superposition of the two right eigenstates
$|\psi\rangle=c_{+}|\psi_{+}^{R}\rangle+|\psi_{-}^{R}\rangle$, the
coefficients can be determined as

$c_{\pm}=\langle\psi_{\pm}^{L}|\psi\rangle.$

By taking the normalized modulus squared
$|c_{\pm}|^{2}/(|c_{+}|^{2}+|c_{-}|^{2})$, we were able to calculate the ratio
of each eigenstate in the wave packet. Similarly to the time-evolution
operator in Appendix B, this approach fail at the exceptional points where the
two eigenstates coalesce. Therefore, we believe that the approximation will be
accurate as it will hold in the vicinity of the exceptional points.

##  Appendix D Exciton-Polariton Pseudospins

The pseudospins of the exciton polaritons can be calculated from the
polarization components as

$\begin{split}S_{x}&=\frac{|\psi^{H}|^{2}-|\psi^{V}|^{2}}{|\psi^{H}|^{2}+|\psi^{V}|^{2}}\\\
S_{y}&=\frac{|\psi^{D}|^{2}-|\psi^{A}|^{2}}{|\psi^{D}|^{2}+|\psi^{A}|^{2}}\\\
S_{z}&=\frac{|\psi^{+}|^{2}-|\psi^{-}|^{2}}{|\psi^{+}|^{2}+|\psi^{-}|^{2}}.\end{split}$
(10)

Here, $|\psi^{H,V}|^{2}$ represents the horizontal and vertical polarization
intensities, $|\psi^{D,A}|^{2}$ represents the diagonal and anti-diagonal
polarization intensities and $|\psi^{\pm}|^{2}$ represents the circular
polarization intensities [42].

##  Appendix E Proposed Experimental Measurement of the Pseudospin Defects

In experiments, the probability density distribution and the pseudospin
textures can be measured using a streak camera which can take a screen shot of
the wave packets in both real and momentum space at a resolution of $\approx
2$ ps. Since the measurements are made by taking multiple screen shots and
then averaging over time resolution, we aim to investigate whether the
pseudospin defects on the imaginary Fermi arc will still appear when the
fields are averaged.

Figure 10: The averaged wave packet and pseudospin textures of the exciton-
polariton in momentum space (a,b) and in the real space (e,f). The probability
density distribution and the pseudospin take similar forms to the wave packet
and pseudospin textures at $t=5$ ps in (c,d) and (g,h). The purple line
denotes the imaginary Fermi arc.

Since the pseudospin of the wave packets cannot be measured in the regions
with small probability density, we need to make sure that the exciton-
polariton density at the location of the pseudospin defects is non-negligible.
Previous work [21] shows that the larger the wave packets are, the faster they
will move away from the initial centre-of-mass momentum. Therefore, we need a
wave packet that is well localized in momentum space, so it will stay near the
defect for a long time. However, the wave packet cannot be too focused since
we still need to measure the pseudospin texture of the entire defect.

Furthermore, although it is unknown whether the change in the pseudospin
textures will reach a steady state, our numerical calculations indicate that
the change of the pseudospin slows down with time. Therefore, the best way to
measure the pseudospin defects is to take measurements over a long time,
taking care to not exceed the time after which the wave packet moves away from
the imaginary Fermi arc.

To simulate an experiment, we choose a circularly-polarized exciton-polariton
wave packet with a width of $\sigma_{k}=0.02$ $\mu$m-1 in momentum space with
initial centre-of-mass momentum of $\langle\mathbf{k}\rangle=(0.215,6.73)$
$\mu$m-1 and considered a realistic set of parameters presented in Ref. [6].
The initial condition of the wave packet is chosen so that its size is small
enough so it will stay on the imaginary Fermi arc for a long time, yet not too
small to be realized in experiment. The wave packet has a size of
$\sigma_{r}=50$ $\mu$m in real space, which can be achieved in the
experiments. We found that that the wave packet at $t=5$ ps is still on the
imaginary Fermi arc and covers the pseudospin defect, but is starting to move
away at $t=15$ ps. Therefore, we chose to take the average of the wave packets
every $2$ ps from $t=5$ ps to $t=15$ ps to see if the averaged wave packet
will exhibit the pseudospin defects.

The polarizations, $|\psi^{\pm}|^{2}$, $|\psi^{H,V}|^{2}$ and
$|\psi^{D,A}|^{2}$ are therefore measured every $2$ ps from $t=5$ ps to $t=15$
ps. We then take the average of these intensities
$\langle|\psi^{\pm}|^{2}\rangle_{ave}$,
$\langle|\psi^{H,V}|^{2}\rangle_{ave}$, $\langle|\psi^{D,A}|^{2}\rangle_{ave}$
and use them to compute the pseudopin textures. We observe the wave packet and
its time-averaged pseudospin textures take forms similar to those at $t=5$ ps
in momentum space [see Fig. 10(a-d)]. Therefore the snapshot of the wave
packet at $t=5$ ps will dominate the time-averaged measurement in momentum
space. This is because the wave packets in later times have much lower
intensities due to the continuing decay (negative imaginary part of
eigenenergies) of the exciton polaritons. On the other hand, in the real
space, the averaged wave packet and pseudospin textures do not completely
agree with those at $t=5$ ps as seen in Fig. 10(e-h). However, the pseudospin
anti-meron structure at the centre of the wave packet still persist after the
averaging. Therefore, we conclude that the pseudospin anti-merons generated by
non-Hermitian wave packet dynamics can be measured experimentally in both
momentum space and real space.

##  Appendix F Wave-Packet Trajectories in Other Two-Band Systems

Previous studies [21] have suggested that the wave-packet centre-of-mass
momenta of a non-Hermitian Dirac model follow the gradient of the imaginary
part of the eigenenergies $\nabla\operatorname{Im}E$. Although this seems to
accurately describe the trajectories of the exciton-polariton wave packets,
this is not always the case. In this section, we illustrate this point by
presenting the analysis of the wave packet dynamics in two exemplary models, a
non-Hermitian Dirac model and a non-Hermitian Chern insulator. These two
models have similar band structures near the origin, but their wave-packet
dynamics and asymptotic behaviours are qualitatively different.

Figure 11: (a), (b): The real and imaginary parts of the eigenenergies of the
non-Hermitian Dirac model. (c), (d): The real and imaginary parts of the
eigenenergies of the non-Hermitian Chern insulator in the nodal phase. Also
showing the wave-packet trajectories (red) with $\nabla\operatorname{Im}E$
(black) with the Fermi arcs and exceptional points of the (e) exciton-
polariton model, (f) the non-Hermitian Dirac model and (g) the non-Hermitian
Chern insulator. (h), (i): Wave-packet trajectories with different sets of
initial centre-of-mass momenta (blue arrows) in non-Hermitian Dirac model and
non-Hermitian Chern insulator, respectively, showing that they do not follow
the same paths indicated the red arrows.

###  F.1 Non-Hermitian Dirac Model

We considered the Dirac model, described by the Hamiltonian

$\begin{split}\mathbf{H}&=\hbar ck_{x}\sigma_{x}+\hbar
ck_{y}\sigma_{y}.\end{split}$ (11)

This model has eigenenergies of $E_{\pm}=\pm\hbar c\sqrt{k_{x}^{2}+k_{y}^{2}}$
and has a spectral degeneracy at the origin called the Dirac point. In solid
state physics, this model describes the Dirac semimetal, which has elementary
excitations of massless Dirac fermions and describes the electronic properties
of graphenes [43, 44, 45, 46, 47]. It also plays an important role in optical
physics. When it is written in the basis of circularly-polarized modes, the
Dirac point describes a diabolical point which exhibits conical refraction
[32, 21, 48, 49].

The non-Hermitian Dirac model that we considered is described by the following
Hamiltonian [21]

$\mathbf{H}=\hbar ck_{x}\sigma_{x}+(\hbar
ck_{y}+i\kappa)\sigma_{y}+\Delta\sigma_{z},$ (12)

which has the eigenenergies

$E_{\pm}=\pm\sqrt{\hbar^{2}c^{2}k_{x}^{2}+(\hbar
ck_{y}+i\kappa)^{2}+\Delta^{2}}.$ (13)

The non-Hermitian term $i\kappa\sigma_{y}$ splits the Dirac point at
$\mathbf{k}^{DP}=(0,0)$ into a pair of non-Hermitian degeneracies called the
exceptional points in momentum space at
$\mathbf{k}^{EP}=(\pm\sqrt{\kappa^{2}-\Delta^{2}}/(\hbar c),0)$.

In non-Hermitian Dirac model, $E_{+}$ has the larger imaginary part at
$k_{y}<0$ while $E_{-}$ has the larger imaginary part at $k_{y}>0$ [see Fig.
11(a,b)]. Therefore, as time evolves, the wave packets at $k_{y}<0$ tend to
self-organize into the upper eigenstate, while the wave packets at $k_{y}>0$
tend to self-organize into the lower eigenstate. Furthermore, at $\Delta=0$,
$\max(\operatorname{Im}E)$ lies at the $k_{y}$-axis. Therefore, the wave
packets in the non-Hermitian Dirac model propagate towards to a line of fixed
points, unlike the wave packets in the exciton-polariton model. From Fig.
11(e,f), we can observe that, unlike the trajectories of the exciton-polariton
wave packets, the centre-of-mass momenta of the Dirac wave packets seem to
only follow $\nabla\operatorname{Im}E_{\pm}$ initially. As time evolves, the
trajectories of the wave packets start to deviate from
$\nabla\operatorname{Im}E_{\pm}$. More importantly, we can see that the
trajectories of the wave packets as well as the fixed points they propagate to
highly depend on their initial momenta. From Fig. 11(h), we can see that the
trajectories of the wave packets with different set of initial centre-of-mass
momenta will not follow the same paths towards the same fixed points as the
wave packets in Fig. 11(f).

###  F.2 Non-Hermitian Chern Insulator

The non-Hermitian Chern insulator is described by the Hamiltonian [50]

$\mathbf{H}=(m+\cos k_{x}+\cos k_{y})\sigma_{x}+(i\gamma+\sin
k_{x})\sigma_{y}+\sin k_{y}\sigma_{z}$ (14)

and has eigenenergies

$E_{\pm}=\pm\Big{(}(m+\cos k_{x}+\cos k_{y})^{2}+(i\gamma+\sin
k_{x})^{2}+\sin^{2}k_{y}\Big{)}^{1/2}.$ (15)

The phase diagram and edge states of this model have been extensively studied
in Ref. [50]. At $m=-2$, $\gamma=1$, there is a pair of exceptional points in
the Brillouine zone at $\mathbf{k}=(0,\pm\pi/3)$ and the band structure is
similar to that of the non-Hermitian Dirac model. However, unlike the Dirac
model, there is a maximum of $\operatorname{Im}E_{\pm}$ at the middle of the
Brillouine zone, which is also the middle of the bulk Fermi arc [see Fig.
11(c,d)].

The wave packets with different initial centre-of-mass momenta in this model
also follow different paths [see Fig. 11(g)]. However, unlike the non-
Hermitian Dirac model, where the asymptotic behaviours of the wave-packet
centre-of-mass momenta are sensitive to the initial conditions, the wave
packets in the non-Hermitian Chern insulator seem to propagate to the same
fixed point asymptotically.

The above analysis of the non-Hermitian Dirac model and the non-Hermitian
Chern insulator shows that the centre-of-mass momenta of a wave packet only
follow $\nabla\operatorname{Im}E$ of its corresponding eigenstate on a short
time scale. The trajectories of the wave packets will deviate increasingly
from $\nabla\operatorname{Im}E$ as time evolves. Furthermore, in cases where
there are multiple maxima of $\operatorname{Im}E_{\pm}$, both the paths that
the wave packets take and the fixed points that the wave packets propagate
towards to will be sensitive to the initial conditions of the wave packets.

## References

  * Ghatak and Das [2019] A. Ghatak and T. Das, New topological invariants in non-Hermitian systems, Journal of Physics: Condensed Matter 31, 263001 (2019).
  * Bergholtz  _et al._ [2021] E. J. Bergholtz, J. C. Budich, and F. K. Kunst, Exceptional topology of non-Hermitian systems, Rev. Mod. Phys. 93, 015005 (2021).
  * El-Ganainy  _et al._ [2018] R. El-Ganainy, K. G. Makris, M. Khajavikhan, Z. H. Musslimani, S. Rotter, and D. N. Christodoulides, Non-Hermitian physics and PT symmetry, Nature Physics 14, 11 (2018).
  * Özdemir  _et al._ [2019] c. K. Özdemir, S. Rotter, F. Nori, and L. Yang, Parity–time symmetry and exceptional points in photonics, Nature Materials 18, 783 (2019).
  * Gao  _et al._ [2015] T. Gao, E. Estrecho, K. Y. Bliokh, T. C. H. Liew, M. D. Fraser, S. Brodbeck, M. Kamp, C. Schneider, S. Höfling, Y. Yamamoto, F. Nori, Y. S. Kivshar, A. G. Truscott, R. G. Dall, and E. A. Ostrovskaya, Observation of non-Hermitian degeneracies in a chaotic exciton-polariton billiard, Nature 526, 554 (2015).
  * Su  _et al._ [2021] R. Su, E. Estrecho, D. Biegańska, Y. Huang, M. Wurdack, M. Pieczarka, A. G. Truscott, T. C. H. Liew, E. A. Ostrovskaya, and Q. Xiong, Direct measurement of a non-Hermitian topological invariant in a hybrid light-matter system, Science Advances 7, eabj8905 (2021), https://www.science.org/doi/pdf/10.1126/sciadv.abj8905 .
  * Kunst  _et al._ [2018] F. K. Kunst, E. Edvardsson, J. C. Budich, and E. J. Bergholtz, Biorthogonal bulk-boundary correspondence in non-Hermitian systems, Phys. Rev. Lett. 121, 026808 (2018).
  * Hofmann  _et al._ [2020] T. Hofmann, T. Helbig, F. Schindler, N. Salgo, M. Brzezińska, M. Greiter, T. Kiessling, D. Wolf, A. Vollhardt, A. Kabaši, C. H. Lee, A. Bilušić, R. Thomale, and T. Neupert, Reciprocal skin effect and its realization in a topolectrical circuit, Phys. Rev. Research 2, 023265 (2020).
  * Weidemann  _et al._ [2020] S. Weidemann, M. Kremer, T. Helbig, T. Hofmann, A. Stegmaier, M. Greiter, R. Thomale, and A. Szameit, Topological funneling of light, Science 368, 311 (2020).
  * Jin and Song [2018] L. Jin and Z. Song, Incident direction independent wave propagation and unidirectional lasing, Phys. Rev. Lett. 121, 073901 (2018).
  * Peng  _et al._ [2014] B. Peng, Ş K. Özdemir, S. Rotter, H. Yilmaz, M. Liertzer, F. Monifi, C. M. Bender, F. Nori, and L. Yang, Loss-induced suppression and revival of lasing, Science 346, 328 (2014), https://www.science.org/doi/pdf/10.1126/science.1258004 .
  * Longhi  _et al._ [2015] S. Longhi, D. Gatti, and G. D. Valle, Robust light transport in non-Hermitian photonic lattices, Scientific Reports 5, 13376 (2015).
  * Kawabata  _et al._ [2021] K. Kawabata, K. Shiozaki, and S. Ryu, Topological field theory of non-Hermitian systems, Phys. Rev. Lett. 126, 216405 (2021).
  * Kasprzak  _et al._ [2006] J. Kasprzak, M. Richard, S. Kundermann, A. Baas, P. Jeambrun, J. M. J. Keeling, F. M. Marchetti, M. H. Szymańska, R. André, J. L. Staehli, V. Savona, P. B. Littlewood, B. Deveaud, and L. S. Dang, Bose–Einstein condensation of exciton polaritons, Nature 443, 409 (2006).
  * Carusotto and Ciuti [2013] I. Carusotto and C. Ciuti, Quantum fluids of light, Rev. Mod. Phys. 85, 299 (2013).
  * Deng  _et al._ [2010] H. Deng, H. Haug, and Y. Yamamoto, Exciton-polariton Bose-Einstein condensation, Rev. Mod. Phys. 82, 1489 (2010).
  * Liao  _et al._ [2021] Q. Liao, C. Leblanc, J. Ren, F. Li, Y. Li, D. Solnyshkov, G. Malpuech, J. Yao, and H. Fu, Experimental measurement of the divergent quantum metric of an exceptional point, Phys. Rev. Lett. 127, 107402 (2021).
  * Bleu  _et al._ [2018a] O. Bleu, G. Malpuech, Y. Gao, and D. D. Solnyshkov, Effective theory of nonadiabatic quantum evolution based on the quantum geometric tensor, Phys. Rev. Lett. 121, 020401 (2018a).
  * Culcer  _et al._ [2005] D. Culcer, Y. Yao, and Q. Niu, Coherent wave-packet evolution in coupled bands, Phys. Rev. B 72, 085110 (2005).
  * Leblanc  _et al._ [2021] C. Leblanc, G. Malpuech, and D. D. Solnyshkov, Universal semiclassical equations based on the quantum metric for a two-band system, Phys. Rev. B 104, 134312 (2021).
  * Solnyshkov  _et al._ [2021] D. D. Solnyshkov, C. Leblanc, L. Bessonart, A. Nalitov, J. Ren, Q. Liao, F. Li, and G. Malpuech, Quantum metric and wave packets at exceptional points in non-Hermitian systems, Phys. Rev. B 103, 125302 (2021).
  * Longhi [2022] S. Longhi, Non-Hermitian skin effect and self-acceleration, Phys. Rev. B 105, 245143 (2022).
  * Cilibrizzi  _et al._ [2016] P. Cilibrizzi, H. Sigurdsson, T. C. H. Liew, H. Ohadi, A. Askitopoulos, S. Brodbeck, C. Schneider, I. A. Shelykh, S. Höfling, J. Ruostekoski, and P. Lagoudakis, Half-skyrmion spin textures in polariton microcavities, Phys. Rev. B 94, 045315 (2016).
  * Nagaosa and Tokura [2013] N. Nagaosa and Y. Tokura, Topological properties and dynamics of magnetic skyrmions, Nature Nanotechnology 8, 899 (2013).
  * Guo  _et al._ [2020] C. Guo, M. Xiao, Y. Guo, L. Yuan, and S. Fan, Meron spin textures in momentum space, Phys. Rev. Lett. 124, 106103 (2020).
  * Zhang  _et al._ [2021] Q. Zhang, Z. Xie, L. Du, P. Shi, and X. Yuan, Bloch-type photonic skyrmions in optical chiral multilayers, Phys. Rev. Research 3, 023109 (2021).
  * Göbel  _et al._ [2021] B. Göbel, I. Mertig, and O. A. Tretiakov, Beyond skyrmions: Review and perspectives of alternative magnetic quasiparticles, Physics Reports 895, 1 (2021), beyond skyrmions: Review and perspectives of alternative magnetic quasiparticles.
  * Król  _et al._ [2021] M. Król, H. Sigurdsson, K. Rechcińska, P. Oliwa, K. Tyszka, W. Bardyszewski, A. Opala, M. Matuszewski, P. Morawiak, R. Mazur, W. Piecek, P. Kula, P. G. Lagoudakis, B. Piȩtka, and J. Szczytko, Observation of second-order meron polarization textures in optical microcavities, Optica 8, 255 (2021).
  * Flayac  _et al._ [2013] H. Flayac, D. D. Solnyshkov, I. A. Shelykh, and G. Malpuech, Transmutation of skyrmions to half-solitons driven by the nonlinear optical spin Hall effect, Phys. Rev. Lett. 110, 016404 (2013).
  * Vishnevsky  _et al._ [2013] D. V. Vishnevsky, H. Flayac, A. V. Nalitov, D. D. Solnyshkov, N. A. Gippius, and G. Malpuech, Skyrmion formation and optical spin-Hall effect in an expanding coherent cloud of indirect excitons, Phys. Rev. Lett. 110, 246404 (2013).
  * Mohanta  _et al._ [2017] N. Mohanta, A. P. Kampf, and T. Kopp, Emergent momentum-space skyrmion texture on the surface of topological insulators, Scientific Reports 7, 45664 (2017).
  * Terças  _et al._ [2014] H. Terças, H. Flayac, D. D. Solnyshkov, and G. Malpuech, Non-Abelian gauge fields in photonic cavities and photonic superfluids, Phys. Rev. Lett. 112, 066402 (2014).
  * Klembt  _et al._ [2018] S. Klembt, T. H. Harder, O. A. Egorov, K. Winkler, R. Ge, M. A. Bandres, M. Emmerling, L. Worschech, T. C. H. Liew, M. Segev, C. Schneider, and S. Höfling, Exciton-polariton topological insulator, Nature 562, 552 (2018).
  * Zhou  _et al._ [2018] H. Zhou, C. Peng, Y. Yoon, C. W. Hsu, K. A. Nelson, L. Fu, J. D. Joannopoulos, M. Soljačić, and B. Zhen, Observation of bulk Fermi arc and polarization half charge from paired exceptional points, Science 359, 1009 (2018), https://www.science.org/doi/pdf/10.1126/science.aap9859 .
  * Donati  _et al._ [2016] S. Donati, L. Dominici, G. Dagvadorj, D. Ballarini, M. D. Giorgi, A. Bramati, G. Gigli, Y. G. Rubo, M. H. Szymańska, and D. Sanvitto, Twist of generalized skyrmions and spin vortices in a polariton superfluid, Proceedings of the National Academy of Sciences 113, 14926 (2016), https://www.pnas.org/doi/pdf/10.1073/pnas.1610123114 .
  * Everschor-Sitte  _et al._ [2018] K. Everschor-Sitte, J. Masell, R. M. Reeve, and M. Kläui, Perspective: Magnetic skyrmions—overview of recent progress in an active research field, Journal of Applied Physics 124, 240901 (2018), https://doi.org/10.1063/1.5048972 .
  * Kovalev and Sandhoefner [2018] A. A. Kovalev and S. Sandhoefner, Skyrmions and antiskyrmions in quasi-two-dimensional magnets, Frontiers in Physics 6, 10.3389/fphy.2018.00098 (2018).
  * Tretiakov and Tchernyshyov [2007] O. A. Tretiakov and O. Tchernyshyov, Vortices in thin ferromagnetic films and the skyrmion number, Phys. Rev. B 75, 012408 (2007).
  * Hertel and Schneider [2006] R. Hertel and C. M. Schneider, Exchange explosions: Magnetization dynamics during vortex-antivortex annihilation, Phys. Rev. Lett. 97, 177202 (2006).
  * Richter  _et al._ [2019] S. Richter, H.-G. Zirnstein, J. Zúñiga Pérez, E. Krüger, C. Deparis, L. Trefflich, C. Sturm, B. Rosenow, M. Grundmann, and R. Schmidt-Grund, Voigt exceptional points in an anisotropic ZnO-based planar microcavity: Square-root topology, polarization vortices, and circularity, Phys. Rev. Lett. 123, 227401 (2019).
  * Król  _et al._ [2022] M. Król, I. Septembre, P. Oliwa, M. Kȩdziora, K. Łempicka-Mirek, M. Muszyński, R. Mazur, P. Morawiak, W. Piecek, P. Kula, W. Bardyszewski, P. G. Lagoudakis, D. D. Solnyshkov, G. Malpuech, B. Piȩtka, and J. Szczytko, Annihilation of exceptional points from different Dirac valleys in a 2D photonic system, Nature Communications 13, 5340 (2022).
  * Bleu  _et al._ [2018b] O. Bleu, D. D. Solnyshkov, and G. Malpuech, Measuring the quantum geometric tensor in two-dimensional photonic and exciton-polariton systems, Phys. Rev. B 97, 195422 (2018b).
  * Novoselov  _et al._ [2004] K. S. Novoselov, A. K. Geim, S. V. Morozov, D. Jiang, Y. Zhang, S. V. Dubonos, I. V. Grigorieva, and A. A. Firsov, Electric field effect in atomically thin carbon films, Science 306, 666 (2004), https://www.science.org/doi/pdf/10.1126/science.1102896 .
  * Novoselov  _et al._ [2005] K. S. Novoselov, A. K. Geim, S. V. Morozov, D. Jiang, M. I. Katsnelson, I. V. Grigorieva, S. V. Dubonos, and A. A. Firsov, Two-dimensional gas of massless Dirac fermions in graphene, Nature 438, 197 (2005).
  * Zhang  _et al._ [2005] Y. Zhang, Y.-W. Tan, H. L. Stormer, and P. Kim, Experimental observation of the quantum Hall effect and Berry’s phase in graphene, Nature 438, 201 (2005).
  * Castro Neto  _et al._ [2009] A. H. Castro Neto, F. Guinea, N. M. R. Peres, K. S. Novoselov, and A. K. Geim, The electronic properties of graphene, Rev. Mod. Phys. 81, 109 (2009).
  * Armitage  _et al._ [2018] N. P. Armitage, E. J. Mele, and A. Vishwanath, Weyl and Dirac semimetals in three-dimensional solids, Rev. Mod. Phys. 90, 015001 (2018).
  * Hamilton [1837] W. R. Hamilton, Third supplement to an essay on systems of rays, Transactions of the Royal Irish Academy 17, 1 (1837).
  * Lloyd [1837] H. Lloyd, On the phenomena presented by light in its passage along the axes of biaxial crystals, Transactions of the Royal Irish Academy 17, 145 (1837).
  * Kawabata  _et al._ [2018] K. Kawabata, K. Shiozaki, and M. Ueda, Anomalous helical edge states in a non-Hermitian Chern insulator, Phys. Rev. B 98, 165148 (2018).


# Exploiting modality-invariant feature for robust multimodal emotion
recognition with missing modalities

###### Abstract

Multimodal emotion recognition leverages complementary information across
modalities to gain performance. However, we cannot guarantee that the data of
all modalities are always present in practice. In the studies to predict the
missing data across modalities, the inherent difference between heterogeneous
modalities, namely the modality gap, presents a challenge. To address this, we
propose to use invariant features for a missing modality imagination network
(IF-MMIN) which includes two novel mechanisms: 1) an invariant feature
learning strategy that is based on the central moment discrepancy (CMD)
distance under the full-modality scenario; 2) an invariant feature based
imagination module (IF-IM) to alleviate the modality gap during the missing
modalities prediction, thus improving the robustness of multimodal joint
representation. Comprehensive experiments on the benchmark dataset IEMOCAP
demonstrate that the proposed model outperforms all baselines and invariantly
improves the overall emotion recognition performance under uncertain missing-
modality conditions. We release the code at: https://github.com/ZhuoYulang/IF-
MMIN.

Index Terms—  Multimodal emotion recognition, Missing modality imagination,
Central moment discrepancy (CMD), Invariant feature

##  1 Introduction

The study of multimodal emotion recognition with missing modalities seeks to
perform emotion recognition in realistic environments [1, 2], where some data
could be missing due to obscured cameras, damaged microphones, etc. The
mainstream solution for the missing modality problem can be summarized in two
categories: 1) missing data generation [3, 4, 5], 2) multimodal joint
representation learning [6, 7]. In [3], an encoder-decoder network was
proposed to generate high-quality missing modality images according to the
available modality, In [7], a translation-based method with cycle consistency
loss was studied to learn joint representations between modalities. In [1], a
Missing Modality Imagination Network, or MMIN for short, was studied to learn
joint representations by predicting missing modalities, which combines the
above two methods.

The modality gap between heterogeneous modalities [8, 9, 10] remains an issue,
which adversely affects emotion recognition accuracy. The question is how to
alleviate such a modality gap. While the modalities have their unique
characteristics, they share the same information in the semantic space. The
modality-invariant feature was introduced to multimodal emotion recognition
with full modality data, which shows remarkable performance. Hazarika et al.
[8] proposed the shared subspace to learn potential commonalities between
modalities to reduce the influence of the modality gap. Liu et al. [11]
proposed discrete shared spaces for capturing fine-grained representations to
improve cross-modal retrieval accuracy. All the studies suggest that the
modality-invariant feature effectively bridges the modality gap. We note that
there has been no related work for emotion recognition under the missing-
modality conditions.

In this work, we propose a missing modality imagination network with the
invariant feature (IF-MMIN). Specifically, we first learn the modality-
invariant feature among various modalities by using a central moment
discrepancy (CMD) distance [12] based constraint training strategy. We then
design the IF-MMIN neural architecture to predict the invariant features of
the missing modality from the available modality. In this way, we fully
explore the available modality to alleviate the modality gap problem in cross-
modal imagination, thus, improving the robustness of multimodal joint
representation. The experimental results, on the benchmark dataset IEMOCAP,
show that the proposed method outperforms the state-of-the-art baseline models
under all missing-modality conditions.

The main contributions of this work are, 1) we propose a CMD-based distance
constraint training to learn the modality-invariant feature among full
modalities; 2) we introduce invariant features into the cross-modality
imagination process to reduce the impact of the modality gap and enhance the
robustness of multimodal joint representation; and 3) experimental results on
various missing modalities conditions demonstrate that the proposed IF-MMIN
can perform accurate emotion recognition performance in the scenario of
missing modalities.

##  2 IF-MMIN: Methodology

The proposed IF-MMIN scheme first employs a central moment discrepancy (CMD)
distance based invariant feature learning strategy under full-modality signals
to learn the modality-specific and modality-invariant features. During IF-MMIN
training, the IF-IM reads these two features to learn the robust joint
representation through missing modality imagination.

###  2.1 CMD Distance-based Invariant Feature Learning

As shown in Fig.1, the pipeline of invariant feature learning includes three
modules: specificity encoder, invariance encoder, and classifier. Specificity
encoder aims to extract the high-level features $(h^{a},h^{v},h^{t})$ from the
raw features $(x^{a},x^{v},x^{t})$ of each modality to represent the modality-
specific features. The invariance encoder takes the modality-specific features
as input to extract the modality-invariant features $H$, which is
concatenation by the high-level features $(H^{a},H^{v},H^{t})$ among all
modalities. At last, the full-connected layer-based classifier input the
concatenation of $h$ and $H$ to predict the emotion category. After
pertaining, we will adopt pretrained specificity and invariance encoders along
with the proposed IF-IM module to build the IF-MMIN architecture.

####  2.1.1 Specificity & Invariance Encoders

As the blue blocks in Fig.1, the specificity encoder is composed of three
modules: acoustic, visual, and textual encoder, Enca, Encv, and Enct for short
respectively. Specifically, Enca employs the LSTM [13] and max-pooling layer
to extract the utterance-level acoustic feature $h^{a}$ from the raw feature
$x^{a}$. Encv shares a similar structure with Enca to read the raw features
$x^{v}$ and output the utterance-level visual features $h^{v}$. Enct adopts
the TextCNN [14], which is a power text representation model in NLP filed, to
extract the utterance-level textual features $h^{t}$ from the raw feature
$x^{t}$.

Fig. 1: The pipeline of the central moment discrepancy (CMD) distance-based
invariant feature learning, which includes the specificity and invariance
encoders, and classifier. The red arrows mean the CMD-based distance
constraint to force various modality features to map to the same semantic
subspace.

The invariance encoder, Enc′, is shown as the green blocks in Fig.1, which
consists of the full-connected layer, the activation function, and the dropout
layer. It aims to map modality-specific features $(h^{a},h^{v},h^{t})$ into a
shared subspace with CMD-based distance constraint strategy (as shown by the
red arrow in Fig.1) to obtain high-level features $(H^{a},H^{v},H^{t})$. Then,
we concatenate the three high-level features into modality-invariant features
$H$.

####  2.1.2 CMD-based Distance Constraint

The CMD-based distance constraint aims to reduce the discrepancy between the
high-level features $(H^{a},H^{v},H^{t})$ of modalities. Note that CMD [12] is
a state-of-the-art distance metric that measures the discrepancy between the
distribution of two features by matching their order-wise moment differences.
We ensure that modality-invariant representation can be learned by minimizing
the $\mathcal{L}_{\text{cmd}}$:

$\displaystyle\mathcal{L}_{\operatorname{cmd}}$
$\displaystyle=\frac{1}{3}\\!\\!\sum_{\begin{subarray}{c}\left(m_{1},m_{2}\right)\in\\\
\\{(t,a),(t,v)\\\
(a,v)\\}\end{subarray}}\big{(}\|\mathbf{E}(H^{m_{1}})-\mathbf{E}(H^{m_{2}})\|_{2}$
(1)
$\displaystyle+\sum_{k=2}^{K}\left\|C_{k}(H^{m_{1}})-C_{k}(H^{m_{2}})\right\|_{2}\big{)}$

where $\mathbf{E}(H)$ is the empirical expectation vector of the input sample
$H$, and $C_{k}(H)=\mathbf{E}((H-\mathbf{E}(H))^{k})$ is the vector of all
$k^{th}$ order sample central moments of the coordinates of $H$.

###  2.2 IF-MMIN Training

The overall architecture of IF-MMIN is illustrated in Fig.2(a), which includes
1) Specificity Encoder; 2) Invariance Encoder; 3) Modality-invariant Feature
aware Imagination Module, IF-IM for short; and 4) Classifier.

Assume that the full-modalities input is $x=(x^{a},x^{v},x^{t})$. Specificity
encoder takes $(x^{a},x^{v}_{miss},x^{t})$, where miss indicates the specific
missing modality, as input to extract the modality-specific features
$(h^{a},h^{v},h^{t})$, which are then concatenated as final output $h$.
Invariance encoder reads $(h^{a},h^{v},h^{t})$ to predict the modality-
invariant feature $H^{\prime}$, which is a concatenation of high-level
features $(H^{\prime a},H^{\prime v},H^{\prime t})$. $h$ and $H^{\prime}$ are
then fused to fed in the IF-IM to predict the feature $h^{\prime}$ of the
missing modality. The hidden features of all intermediate layers of IF-IM are
then combined together as the joint representation $C$, as shown in Fig.2(b),
to predict the final emotion category $O$. To ensure the stability of the
modality-invariant feature prediction and missing modality imagination
processes in IF-MMIN, Imagination Loss ($\mathcal{L}_{img}$) and Invariance
Loss ($\mathcal{L}_{inv}$) are added on the basis of Classification Loss
($\mathcal{L}_{cls}$). Note that the parameters of the specificity and
invariance encoders are initialized by the corresponding modules as mentioned
in Sec. 2.1.1.

Fig. 2: The diagrams for the proposed IF-MMIN. (a) shows the overall IF-MMIN
architecture, the blue locks mean their parameters are fixed during the IF-
MMIN training; (b) shows the detailed structure of the modality-invariant
feature aware imagination module.

####  2.2.1 Invariant Feature Aware Imagination Module (IF-IM)

As shown in Fig.2(b), IF-IM is built with the cascaded autoencoder which
includes $M$ autoencoders. Different from [1], IF-IM reads the $h$ and
$H^{\prime}$ simultaneously. In addition, $H^{\prime}$ is a cascaded input
given to each autoencoder to assist the missing modality imagination and
alleviate the modality gap problem.

Each autoencoder denoted as $\omega_{i}$, i = 1,2,…,$M$. Then the calculation
of each autoencoder can be defined as:

$\left\\{\begin{array}[]{l}\Delta z_{i}=\omega_{i}(H^{\prime}+h),\quad i=1\\\
\Delta z_{i}=\omega_{i}\left(H^{\prime}+\Delta z_{i-1}\right),\quad 1<i\leq
M\end{array}\right.\vspace{-2mm}$ (2)

where $\Delta z_{i}$ is the output of the $i^{th}$ autoencoder. The imagined
missing modality $h^{\prime}$ of IF-IM can be defined as: $h^{\prime}=\Delta
z_{M}$.

####  2.2.2 Loss Functions

During IF-MMIN training, classification loss $\mathcal{L}_{\text{cls}}$ is
used to supervise the training with emotion category target $\hat{O}$:
$\mathcal{L}_{\text{cls }}=\text{CrossEntropy}(O,\hat{O})$. More important,
Imagination Loss $\mathcal{L}_{\text{img}}$ is used to minimize the distance
between the IF-IM output $h^{\prime}$ and the modality-specific feature of the
missing modality $x^{v}$: $\mathcal{L}_{\text{img}}=\text{RMSE
}(h_{i}^{\prime\prime},h_{i}^{\prime})$, while Invariance Loss
$\mathcal{L}_{\text{inv}}$ aims to force the predicted modality-invariant
feature $H^{\prime}$ and the target modality-invariant feature $H$ of full-
modality signals are close to each other: $\mathcal{L}_{\text{inv}}=\text{RMSE
}(H_{i},H_{i}^{\prime})$.

The total loss function is the sum of the three functions:
$\mathcal{L}=\mathcal{L}_{\text{cls}}+\lambda_{1}\mathcal{L}_{\text{img}}+\lambda_{2}\mathcal{L}_{\text{inv}}$,
where $\lambda_{1}$ and $\lambda_{2}$ are the balance factors.

System | Testing Conditions
---|---
{a} | {v} | {t} | {a,v} | {a,t} | {v,t} | Average
WA | UA | WA | UA | WA | UA | WA | UA | WA | UA | WA | UA | WA | UA
MCTN [7] | 0.4975 | 0.5162 | 0.4892 | 0.4573 | 0.6242 | 0.6378 | 0.5634 | 0.5584 | 0.6834 | 0.6946 | 0.6784 | 0.6834 | 0.5894 | 0.5913
MMIN [1] | 0.5511 | 0.5726 | 0.5299 | 0.5117 | 0.6649 | 0.6769 | 0.6491 | 0.6584 | 0.7328 | 0.7467 | 0.7244 | 0.7307 | 0.6420 | 0.6495
MMIN w/o cycle [1] | 0.5503 | 0.5821 | 0.5116 | 0.5006 | 0.6577 | 0.6705 | 0.6239 | 0.6454 | 0.7185 | 0.7438 | 0.7202 | 0.7301 | 0.6304 | 0.6454
IF-MMIN (ours) | 0.5620 ${}^{\uparrow}_{\Uparrow}$ | 0.5813 ${}^{*}_{\Uparrow}$ | 0.5197 ${}^{*}_{\Uparrow}$ | 0.5041 ${}^{*}_{\Uparrow}$ | 0.6702 ${}^{\uparrow}_{\Uparrow}$ | 0.6820 ${}^{\uparrow}_{\Uparrow}$ | 0.6533 ${}^{\uparrow}_{+}$ | 0.6652 ${}^{\uparrow}_{+}$ | 0.7405 ${}^{\uparrow}_{\Uparrow}$ | 0.7544 ${}^{\uparrow}_{\Uparrow}$ | 0.7268 ${}^{\uparrow}_{\Uparrow}$ | 0.7362 ${}^{\uparrow}_{\Uparrow}$ | 0.6454 ${}^{\uparrow}_{\Uparrow}$ | 0.6538 ${}^{\uparrow}_{\Uparrow}$
w/o $\mathcal{L}_{\text{inv}}$ | 0.5513 | 0.5767 | 0.5156 | 0.4955 | 0.6631 | 0.6768 | 0.6511 | 0.6665 | 0.7333 | 0.7511 | 0.7163 | 0.7277 | 0.6385 | 0.6490
w/o cascaded input | 0.5552 | 0.5768 | 0.5159 | 0.5038 | 0.6642 | 0.6790 | 0.6536 | 0.6650 | 0.7337 | 0.7530 | 0.7173 | 0.7300 | 0.6400 | 0.6514
Table 1: The experimental results of our IF-MMIN, three baselines, and two
ablation systems under six missing-modality conditions (i.e. testing condition
$\\{t\\}$ indicates that only textual modality is available and both visual
and acoustic modalities are missing.). “Average” means the average performance
in overall conditions. $\uparrow$ and $\Uparrow$ indicate that the current
result outperforms all baselines and ablation systems respectively. * and +
indicate that the results are on par with those of the best baseline or best
ablation system respectively.

##  3 Experiments and Results

We validate the IF-MMIN on the Interactive Emotional Dyadic Motion Capture
(IEMOCAP) dataset [15]. Following [1], we process IEMOCAP emotional labels
into four categories: happy, angry, sad, and neutral. The splitting ratio of
training/validation/testing sets is 8:1:1.

###  3.1 Experimental Setup

Similar to those in [1], the raw features $x^{a}$, $x^{v}$ and $x^{t}$ are
130-dim OpenSMILE [16] features with the configuration of “IS13_ComParE”,
342-dim “Denseface” features extracted by a pretrained DenseNet model [17] and
1024-dim BERT word embeddings, respectively.

The hidden size of specifically encoders Enca and Encv is set to 128, Enct
contains 3 convolution blocks with kernel sizes of 3,4,5 and the output size
of 128. The size of invariance encoder Enc′ output, $\mathcal{H}$, is 128. IF-
IM consists of 5 autoencoders in size 384-256-128-64-128-256-384, where the
hidden-vector size is 64. The classifier includes 3 fully-connected layers of
size {128,128,4}. Since the value of $\mathcal{L}_{\text{inv}}$ is quite
smaller (about 1%) than $\mathcal{L}_{\text{img}}$, we set $\lambda_{1}$ is 1
and $\lambda_{2}$ is 100 to balance the numerical difference and elevate the
importance of $\mathcal{L}_{\text{inv}}$ in the total loss. The batch size is
128 and the dropout rate is 0.5. We adopt the Adam optimizer [18] which with
the dynamic learning rate and the initial rate is 0.0002, and use the Lambda
LR [19] to update the learning rate.

We conduct all experiments, including invariant feature learning and IF-MMIN
training, with 10-fold cross-validation, where each fold contains 40 epochs.
To demonstrate the robustness of our models, we run each model three times to
alleviate the influences of random initialization of parameters. We select the
best model on the validation set and report its performance on the testing
set. All models are implemented with Pytorch deep learning toolkit and run on
a single NVIDIA Tesla P100 graphic card.

###  3.2 Comparative Study

We develop three multimodal emotion recognition systems for a comparative
study. 1) MCTN [7] learns the joint representation via a cyclic translation
between missing and available modalities; 2) MMIN [1] is the state-of-the-art
model for the missing modality problem which learns the joint representation
through cross-modality imagination via the autoencoder and cycle consistency
learning; 3) MMIN w/o cycle [1] removes the cycle consistency learning part of
MMIN and just retains the forward missing modality imagination process, which
is a fair counterpart for our IF-MMIN.

###  3.3 Main Results for Uncertain Missing-Modality

To validate our IF-MMIN under different missing modality testing conditions
[1], We report all results in terms of Weighted Accuracy (WA) [20] and
Unweighted Accuracy (UA) [21].

As shown in rows 2 through 5 of Table 1, our IF-MMIN achieves the highest
average values under all missing-modality testing conditions. For each
condition, IF-MMIN also outperforms all baselines except for conditions {a}
and {v}, where it is comparable to the optimal baseline. The possible reason
is that the textual modality contains more semantic information than the audio
and visual modalities [22]. In a nutshell, all the results show that IF-MMIN
can learn robust multimodal joint representation, alleviate the modality gap
by introducing modality-invariant features, and thus enable remarkable
performance under different missing-modality testing conditions.

###  3.4 Ablation Study

IF-MMIN exploits the invariant feature $H^{\prime}$ and adds the Invariance
Loss $\mathcal{L}_{inv}$ to boost the missing modality imagination and IF-IM
takes the invariant feature $H^{\prime}$ with cascaded input. To verify their
contributions, we conduct the following ablation experiments: 1) IF-MMIN w/o
$\mathcal{L}_{\text{inv}}$ system discards the $\mathcal{L}_{\text{inv}}$
during IF-MMIN training. 2) IF-MMIN w/o cascaded input system just only takes
$H^{\prime}$ as the input of the first autoencoder, instead of each
autoencoder, in IF-IM.

As shown in rows 5 through 7 of Table 1, IF-MMIN also outperforms the IF-MMIN
w/o $\mathcal{L}_{\text{inv}}$ and IF-MMIN w/o cascaded input in most cases,
which confirms that 1) the invariance encoder of IF-MMIN can predict accurate
invariant feature under the constraints of $\mathcal{L}_{\text{inv}}$, so as
to better serve the IF-IM; 2) the cascaded input can provide prior knowledge
when each layer of autoencoder works and indeed strengthen the imagination
ability of IF-IM.

###  3.5 Visualization Analysis

The accuracy of invariant feature learning is the premise for IF-MMIN to work
well. Therefore, to verify the role of invariant feature learning related
modules, including $\mathcal{L}_{cmd}$, $\mathcal{L}_{inv}$, $H^{\prime}$ and
$H$, we conduct the following visualization experiments for IF-MMIN.

We visualize the $H^{\prime}$ under six missing conditions using t-SNE
algorithm in a two dimensional plane [23], as shown in Fig. 3(a). We randomly
select 600 sentences from the testing set, 100 sentences for each condition,
and extract 600 invariant features $H^{\prime}$. Therefore, there are 600
points in Fig. 3(a), 100 points in each color. It’s observed that all
$H^{\prime}$ under various conditions form a clear clustering in terms of
feature distributions, which is encouraging. Fig. 3(b) shows the convergence
the trajectory of $\mathcal{L}_{inv}$ during the IF-MMIN training, where the
x-axis represents the epoch and the y-axis represents the loss value. The
smooth curve in the figure shows that $H^{\prime}$ and $H$ are close to each
other in the training process, thus further proving the effectiveness of
$\mathcal{L}_{inv}$. Since $H$ is learned under the constraint of
$\mathcal{L}_{cmd}$, it also proves the effectiveness of $\mathcal{L}_{cmd}$.

(a) Invariant features visualization

(b) Convergence trajectory of $\mathcal{L}_{\text{inv}}$

Fig. 3: Visualization analysis about the invariant feature and the
$\mathcal{L}_{\text{inv}}$. (a) shows the t-SNE plot of the distribution of
the predicted modality-invariant features $H^{\prime}$ under all six missing-
modality conditions. (b) denotes the convergence trajectory of
$\mathcal{L}_{\text{inv}}$ during the IF-MMIN training. (the x-axis represents
the iteration number and the y-axis represents the loss value).

##  4 Conclusion

This work studied a novel invariant feature aware multimodal emotion
recognition model (IF-MMIN) that includes a CMD-based distance-based invariant
feature learning and the invariant feature aware missing modality imagination
module (IF-IM). By exploiting the invariant feature, our IF-MMIN can alleviate
the modality gap and improve the robustness of the multimodal joint
representation. Experimental results on IEMOCAP and demonstrated that the
proposed IF-MMIN outperforms the start-of-the-art baselines under various
missing modality conditions. In future work, we will explore ways to further
improve invariant feature learning.

## References

  * [1] Jinming Zhao, Ruichen Li, and Qin Jin,  “Missing modality imagination network for emotion recognition with uncertain missing modalities,”  in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, Eds. 2021, pp. 2608–2618, Association for Computational Linguistics.
  * [2] Jiajia Tang, Kang Li, Xuanyu Jin, Andrzej Cichocki, Qibin Zhao, and Wanzeng Kong,  “CTFN: Hierarchical learning for multimodal sentiment analysis using coupled-translation fusion network,”  in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, Aug. 2021, pp. 5301–5311, Association for Computational Linguistics.
  * [3] Lei Cai, Zhengyang Wang, Hongyang Gao, Dinggang Shen, and Shuiwang Ji,  “Deep adversarial learning for multi-modality missing data completion,”  in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018, Yike Guo and Faisal Farooq, Eds. 2018, pp. 1158–1166, ACM.
  * [4] Qiuling Suo, Weida Zhong, Fenglong Ma, Ye Yuan, Jing Gao, and Aidong Zhang,  “Metric learning on healthcare data with incomplete modalities,”  in Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, Sarit Kraus, Ed. 2019, pp. 3534–3540, ijcai.org.
  * [5] Changde Du, Changying Du, Hao Wang, Jinpeng Li, Wei-Long Zheng, Bao-Liang Lu, and Huiguang He,  “Semi-supervised deep generative modelling of incomplete multi-modality emotional data,”  in 2018 ACM Multimedia Conference on Multimedia Conference, MM 2018, Seoul, Republic of Korea, October 22-26, 2018, Susanne Boll, Kyoung Mu Lee, Jiebo Luo, Wenwu Zhu, Hyeran Byun, Chang Wen Chen, Rainer Lienhart, and Tao Mei, Eds. 2018, pp. 108–116, ACM.
  * [6] Jing Han, Zixing Zhang, Zhao Ren, and Björn W. Schuller,  “Implicit fusion by joint audiovisual training for emotion recognition in mono modality,”  in IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019, Brighton, United Kingdom, May 12-17, 2019. 2019, pp. 5861–5865, IEEE.
  * [7] Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, and Barnabás Póczos,  “Found in translation: Learning robust joint representations by cyclic translations between modalities,”  in The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019. 2019, pp. 6892–6899, AAAI Press.
  * [8] Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria,  “Misa: Modality-invariant and -specific representations for multimodal sentiment analysis,”  in Proceedings of the 28th ACM International Conference on Multimedia, New York, NY, USA, 2020, MM ’20, p. 1122–1131, Association for Computing Machinery.
  * [9] Ricardo Guerrero, Hai Xuan Pham, and Vladimir Pavlovic,  “Cross-modal retrieval and synthesis (X-MRS): closing the modality gap in shared subspace learning,”  in MM ’21: ACM Multimedia Conference, Virtual Event, China, October 20 - 24, 2021, Heng Tao Shen, Yueting Zhuang, John R. Smith, Yang Yang, Pablo Cesar, Florian Metze, and Balakrishnan Prabhakaran, Eds. 2021, pp. 3192–3201, ACM.
  * [10] Dan Jia, Alexander Hermans, and Bastian Leibe,  “Domain and modality gaps for lidar-based person detection on mobile robots,”  CoRR, vol. abs/2106.11239, 2021.
  * [11] Alexander Liu, SouYoung Jin, Cheng-I Lai, Andrew Rouditchenko, Aude Oliva, and James Glass,  “Cross-modal discrete representation learning,”  in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, May 2022, pp. 3013–3035, Association for Computational Linguistics.
  * [12] Werner Zellinger, Thomas Grubinger, Edwin Lughofer, Thomas Natschläger, and Susanne Saminger-Platz,  “Central moment discrepancy (CMD) for domain-invariant representation learning,”  in 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. 2017, OpenReview.net.
  * [13] Hasim Sak, Andrew W. Senior, and Françoise Beaufays,  “Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition,”  CoRR, vol. abs/1402.1128, 2014.
  * [14] Yoon Kim,  “Convolutional neural networks for sentence classification,”  in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, Alessandro Moschitti, Bo Pang, and Walter Daelemans, Eds. 2014, pp. 1746–1751, ACL.
  * [15] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N. Chang, Sungbok Lee, and Shrikanth S. Narayanan,  “IEMOCAP: interactive emotional dyadic motion capture database,”  Lang. Resour. Evaluation, vol. 42, no. 4, pp. 335–359, 2008.
  * [16] Florian Eyben, Martin Wöllmer, and Björn W. Schuller,  “Opensmile: the munich versatile and fast open-source audio feature extractor,”  in Proceedings of the 18th International Conference on Multimedia 2010, Firenze, Italy, October 25-29, 2010, Alberto Del Bimbo, Shih-Fu Chang, and Arnold W. M. Smeulders, Eds. 2010, pp. 1459–1462, ACM.
  * [17] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger,  “Densely connected convolutional networks,”  in 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017. 2017, pp. 2261–2269, IEEE Computer Society.
  * [18] Diederik P. Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun, Eds., 2015.
  * [19] Neo Wu, Bradley Green, Xue Ben, and Shawn O’Banion,  “Deep transformer models for time series forecasting: The influenza prevalence case,”  arXiv preprint arXiv:2001.08317, 2020.
  * [20] Ishwar Baidari and Nagaraj Honnikoll,  “Accuracy weighted diversity-based online boosting,”  Expert Syst. Appl., vol. 160, pp. 113723, 2020.
  * [21] Shruti Gupta, Md. Shah Fahad, and Akshay Deepak,  “Pitch-synchronous single frequency filtering spectrogram for speech emotion recognition,”  Multim. Tools Appl., vol. 79, no. 31-32, pp. 23347–23365, 2020\.
  * [22] Ziwang Fu, Feng Liu, Qing Xu, Jiayin Qi, Xiangling Fu, Aimin Zhou, and Zhibin Li,  “NHFNET: A non-homogeneous fusion network for multimodal sentiment analysis,”  in IEEE International Conference on Multimedia and Expo, ICME 2022, Taipei, Taiwan, July 18-22, 2022. 2022, pp. 1–6, IEEE.
  * [23] Laurens Van der Maaten and Geoffrey Hinton,  “Visualizing data using t-sne.,”  Journal of machine learning research, vol. 9, no. 11, 2008.