# Controlling Frequency-Domain Hong–Ou–Mandel Interference via
Electromagnetically Induced Transparency

Zi-Yu Liu1,2, Jiun-Shiuan Shiu1,2, Chin-Yao Cheng1,2, and Yong-Fan Chen1,2,3
<EMAIL_ADDRESS>1Department of Physics, National Cheng Kung University,
Tainan 70101, Taiwan
2Center for Quantum Frontiers of Research $\&$ Technology, Tainan 70101,
Taiwan
3Center for Quantum Technology, Hsinchu 30013, Taiwan

###### Abstract

Hong–Ou–Mandel (HOM) interference is a compelling quantum phenomenon that
demonstrates the nonclassical nature of single photons. Herein, we investigate
an electromagnetically induced transparency-based double-$\Lambda$ four-wave
mixing system from the perspective of quantized light fields. The system can
be used to realize efficient HOM interference in the frequency domain. By
using the reduced density operator theory, we demonstrate that, although the
double-$\Lambda$ medium does not exhibit phase-dependent properties for the
closed-loop case of two incident single photons, frequency-domain HOM two-
photon interference occurs. For experimentally achievable optical depth
conditions, our theory indicates that this double-$\Lambda$ scheme can perform
high-fidelity Hadamard gate operations on frequency-encoded single-photon
qubits, and thereby generate HOM two-photon NOON states with a fidelity
greater than 0.99. Furthermore, we demonstrate that this scheme can be used to
realize arbitrary single-qubit gates and two-qubit SWAP gates by simply
controlling the laser detuning and phase, exhibiting its multifunctional
properties and providing a new route to scalable optical quantum computing.

##  I Introduction

Hong–Ou–Mandel (HOM) interference [1, 2, 3], a well-known quantum effect
without a classical counterpart, has played a key role in optical quantum
information processing [4, 5, 6, 7] and quantum metrology [8, 9, 10, 11].
Instead of passive devices, such as spatial or polarizing beam splitters,
recent studies have used frequency converters to transfer HOM interference
from the spatial to the spectral domain; this method facilitates quantum
frequency multiplexing and engineering [12, 13, 14, 15]. Studies of frequency-
domain HOM interference for single photons have relied on linear optics, which
involve the use of electro-optic phase modulators [16, 17, 18], or the
nonlinear effects of solid materials [19, 20, 21]. Both schemes operate under
far-detuned interaction conditions and thus require high-voltage amplifiers or
strong pump light; however, these methods often generate adjacent sidebands or
parametric noise photons, reducing the visibility of HOM interference [22,
23]. Herein, we propose another promising scheme based on near-resonant
nonlinear optics that enables frequency-domain HOM interference with high
visibility. The proposed scheme leverages the closed-loop four-wave mixing
(FWM) effect in a double-$\Lambda$ electromagnetically induced transparency
(EIT) system [24]. Because of the EIT effect, this near-resonant FWM scheme
can substantially suppress vacuum field noise at low light levels, thereby
effectively preventing photon dissipation and avoiding the generation of noise
photons [25]. Such properties enable this double-$\Lambda$ scheme to generate
frequency-domain HOM two-photon NOON states with extremely high fidelity.

To study frequency-domain HOM interference in a closed-loop double-$\Lambda$
medium, we constructed a quantum model based on the reduced density operator
theory [26]. Unlike semiclassical approaches, the proposed model quantizes two
incident weak light fields. No discussion of such closed-loop FWM systems from
the perspective of quantized light fields was found in a literature review.
Previous semiclassical models have demonstrated that the respective
transmittances of the two incident light fields through a closed-loop
double-$\Lambda$ medium are affected by the relative phases of the applied
light fields; this is known as the phase-dependent property [27, 28, 29].
However, our quantum model revealed that the phase-dependent property of a
closed-loop double-$\Lambda$ medium depends on the quantum states of the two
incident weak light fields. If both incident weak light fields are in coherent
states, the double-$\Lambda$ medium exhibits phase-dependent properties, which
is consistent with the predictions of a semiclassical model and has been
experimentally confirmed [30, 31, 32]. By contrast, if both incident weak
light fields are single photons, this phase-dependent property is no longer
present. Although the double-$\Lambda$ medium does not have phase-dependent
properties in the case of two incident single photons, frequency-domain HOM
interference of the two-photon state occurs. Our theory shows that such a
double-$\Lambda$ scheme can perform high-fidelity Hadamard gate operations on
frequency-encoded single-photon qubits. For experimentally achievable optical
depth (OD) conditions, HOM two-photon NOON states with a fidelity of 0.99 can
be generated. Furthermore, we demonstrate for the first time that this scheme
can be used to implement arbitrary single-qubit gates and high-fidelity two-
qubit SWAP gates, revealing its advantages in fabricating multifunctional
photonic logic gates.

##  II Theoretical model

We consider a medium comprising double-$\Lambda$ four-level atoms with two
metastable ground states and two excited states [Fig. 1(a)]. The weak probe
field $\hat{a}_{p}$ driving the $|1\rangle\leftrightarrow|3\rangle$ transition
and the weak signal field $\hat{a}_{s}$ driving the
$|1\rangle\leftrightarrow|4\rangle$ transition can both be described by
quantum operators obeying the bosonic commutation relation. High-intensity
coherent coupling and driving fields, described by the semiclassical Rabi
frequency $\Omega_{c(d)}=|\Omega_{c(d)}|e^{i\phi_{c(d)}}$ with a well-defined
phase $\phi_{c(d)}$, drive $|2\rangle\leftrightarrow|3\rangle$ and
$|2\rangle\leftrightarrow|4\rangle$ transitions, respectively, forming a
closed-loop double-$\Lambda$ EIT system together with the probe and signal
fields. The parameters $\gamma_{31}$ and $\gamma_{41}$ represent the total
coherence decay rates of the excited states $|3\rangle$ and $|4\rangle$,
respectively. $\gamma_{21}$ is the dephasing rate between the ground states
$|1\rangle$ and $|2\rangle$. We consider the case in which the signal field
and the driving field satisfy the two-photon resonance condition, but a one-
photon detuning, denoted as $\Delta$, exists. By using the collective atomic
operator approach and ignoring interactions between atoms, the interaction
Hamiltonian $\hat{H}$ of the double-$\Lambda$ system can be expressed as
follows:

$\displaystyle\hat{H}$ $\displaystyle=-\frac{\hbar
N}{2L}\int^{L}_{0}\big{[}2g_{p}\hat{\sigma}_{31}(z,t)\hat{a}_{p}(z,t)+2g_{s}\hat{\sigma}_{41}(z,t)\hat{a}_{s}(z,t)$
$\displaystyle+\Omega_{c}\hat{\sigma}_{32}(z,t)+\Omega_{d}\hat{\sigma}_{42}(z,t)+\Delta\hat{\sigma}_{44}(z,t)+\textrm{h.c.}\big{]}dz,$
(1)

where $N$ and $L$ are the total number of atoms and the atomic medium length,
respectively. $g_{p}=\frac{d_{31}\varepsilon_{p}}{\hbar}$ denotes the coupling
constant between the probe field and the single atom, where $d_{31}$ is the
dipole moment of the corresponding $|3\rangle\leftrightarrow|1\rangle$
transition and $\varepsilon_{p}=\sqrt{\frac{\hbar\omega_{p}}{2\epsilon_{0}V}}$
is the electric field of the single probe photon. Similarly, there are
$g_{s}=\frac{d_{41}\varepsilon_{s}}{\hbar}$ and
$\varepsilon_{s}=\sqrt{\frac{\hbar\omega_{s}}{2\epsilon_{0}V}}$ parameters for
the signal field. $\hat{\sigma}_{jk}(z,t)$ represents a collective atomic
operator in the slowly-varying amplitude approximation [33] that obeys the
following Heisenberg–Langevin equation (HLE) between states $|j\rangle$ and
$|k\rangle$

$\displaystyle\frac{\partial}{\partial
t}\hat{\sigma}_{jk}=\frac{i}{\hbar}[\hat{H},\hat{\sigma}_{jk}]-\frac{\gamma_{jk}}{2}\hat{\sigma}_{jk}+\gamma^{sp}_{jk}+\hat{F}_{jk},$
(2)

where $\gamma^{sp}_{jk}$ is the parameter related to the spontaneous decay
process, and $\hat{F}_{jk}$ denotes the associated Langevin noise operator.
For the double-$\Lambda$ EIT system, both the probe and signal fields can be
regarded as perturbation fields. Therefore, we can solve the first-order
atomic operators by substituting the zero-order perturbation results into the
relevant first-order HLEs as follows:

$\displaystyle\frac{\partial}{\partial t}$
$\displaystyle\hat{\sigma}^{(1)}_{21}(z,t)=\hat{F}_{21}(z,t)-\frac{1}{2}\gamma_{21}\hat{\sigma}^{(1)}_{21}(z,t)$
$\displaystyle-i\left[\frac{\Omega_{c}}{2}\hat{\sigma}^{(1)}_{31}(z,t)+\frac{\Omega_{d}}{2}\hat{\sigma}^{(1)}_{41}(z,t)\right],$
(3) $\displaystyle\frac{\partial}{\partial t}$
$\displaystyle\hat{\sigma}^{(1)}_{31}(z,t)=\hat{F}_{31}(z,t)-\frac{1}{2}\gamma_{31}\hat{\sigma}^{(1)}_{31}(z,t)$
$\displaystyle-i\left[g_{p}\hat{a}^{\dagger}_{p}(z,t)+\frac{\Omega^{\ast}_{c}}{2}\hat{\sigma}^{(1)}_{21}(z,t)\right],$
(4) $\displaystyle\frac{\partial}{\partial t}$
$\displaystyle\hat{\sigma}^{(1)}_{41}(z,t)=\hat{F}_{41}(z,t)-\frac{1}{2}\gamma_{41}\hat{\sigma}^{(1)}_{41}(z,t)$
$\displaystyle-i\left[g_{s}\hat{a}^{\dagger}_{s}(z,t)+\frac{\Omega^{\ast}_{d}}{2}\hat{\sigma}^{(1)}_{21}(z,t)+\Delta\hat{\sigma}^{(1)}_{41}(z,t)\right].$
(5)

Figure 1:  Diagram of the four-level double-$\Lambda$ EIT system. (a) The
corresponding transitions for the four applied light fields. All light fields
are assumed to propagate in the same direction to eliminate the Doppler shift
in the atomic medium. (b) Both the input probe and signal fields are in
coherent states. In this case, the two coherent fields $|\beta_{p}\rangle$ and
$|\beta_{s}\rangle$ interfere in the double-$\Lambda$ medium, and the output
of the two coherent fields depends on the relative phase of the applied light
fields. (c) Input probe and signal fields are two single photons (i.e.,
$|1_{p}1_{s}\rangle$). In this case, although the phase-dependent property of
the double-$\Lambda$ medium is no longer present, HOM interference of the two-
photon state occurs. The frequency-domain two-photon NOON state can be
generated simply by optimizing the OD and $\Delta$.

To investigate the behavior of the probe and signal fields propagating in the
double-$\Lambda$ medium, the solutions of $\hat{\sigma}^{(1)}_{31}(z,t)$ and
$\hat{\sigma}^{(1)}_{41}(z,t)$ from the first-order HLEs are incorporated into
the following Maxwell–Schrödinger equations (MSEs):

$\displaystyle\bigg{(}\frac{\partial}{\partial t}+c\frac{\partial}{\partial
z}\bigg{)}\hat{a}^{\dagger}_{p}(z,t)=-ig_{p}N\hat{\sigma}^{(1)}_{31}(z,t),$
(6) $\displaystyle\bigg{(}\frac{\partial}{\partial
t}+c\frac{\partial}{\partial
z}\bigg{)}\hat{a}^{\dagger}_{s}(z,t)=-ig_{s}N\hat{\sigma}^{(1)}_{41}(z,t),$
(7)

where $c$ is the speed of light. Subsequently, through the Fourier transform
from the time domain to the frequency domain, the coupled equations of
$\tilde{a}^{\dagger}_{p}(z,\omega)$ and $\tilde{a}^{\dagger}_{s}(z,\omega)$
can be derived as follows:

$\displaystyle\frac{\partial}{\partial
z}\tilde{a}^{\dagger}_{p}+\Lambda_{p}\tilde{a}^{\dagger}_{p}+\kappa_{p}e^{i(-\phi_{c}+\phi_{d})}\tilde{a}^{\dagger}_{s}=\sum_{jk}\zeta^{p}_{jk}\tilde{f}_{jk},$
(8) $\displaystyle\frac{\partial}{\partial
z}\tilde{a}^{\dagger}_{s}+\Lambda_{s}\tilde{a}^{\dagger}_{s}+\kappa_{s}e^{-i(-\phi_{c}+\phi_{d})}\tilde{a}^{\dagger}_{p}=\sum_{jk}\zeta^{s}_{jk}\tilde{f}_{jk},$
(9)

where $\Lambda_{p(s)}$ and $\kappa_{p(s)}$ are the EIT dispersion coefficient
and coupling constant of the probe (signal) transition, respectively [34].
$\tilde{f}_{jk}\equiv\sqrt{\frac{N}{c}}\tilde{F}_{jk}$ is the renormalized
Langevin noise operator, where $jk\in\\{21,31,41\\}$ denotes the subscript of
the atomic operator $\tilde{\sigma}_{jk}$. $\zeta^{p}_{jk}$ and
$\zeta^{s}_{jk}$ are the coefficients of the Langevin noise operators of
interest. For simplicity, we consider the steady-state case of $\omega$ = 0
and assume that $|\Omega_{c}|=|\Omega_{d}|\equiv|\Omega|$, $g_{p}=g_{s}\equiv
g$, $\gamma_{31}=\gamma_{41}\equiv\Gamma$, and $\gamma_{21}=0$. Under these
conditions, the general solution of the aforementioned coupled equations can
be obtained as follows:

$\displaystyle\begin{bmatrix}\tilde{a}^{\dagger}_{p,L}\\\
\tilde{a}^{\dagger}_{s,L}\\\
\end{bmatrix}=e^{-ML}\begin{bmatrix}\tilde{a}^{\dagger}_{p,0}\\\
\tilde{a}^{\dagger}_{s,0}\\\
\end{bmatrix}+\sum_{jk}\int^{L}_{0}e^{-M(L-z)}\begin{bmatrix}\zeta^{p}_{jk}\\\
\zeta^{s}_{jk}\\\ \end{bmatrix}\tilde{f}_{jk}dz,$ (10)

where $\tilde{a}^{\dagger}_{p(s),L}$ and $\tilde{a}^{\dagger}_{p(s),0}$ are
abbreviations for $\tilde{a}^{\dagger}_{p(s)}(L,0)$ and
$\tilde{a}^{\dagger}_{p(s)}(0,0)$, respectively. All the coefficients of the
general solution can be found in Appendix A. The term $e^{-ML}$ characterizes
the parametric evolution of the double-$\Lambda$ medium. Its matrix form is

$\displaystyle e^{-ML}$
$\displaystyle=\textrm{exp}\left(-\begin{bmatrix}\Lambda_{p}&\kappa_{p}e^{i(-\phi_{c}+\phi_{d})}\\\
\kappa_{s}e^{-i(-\phi_{c}+\phi_{d})}&\Lambda_{s}\\\ \end{bmatrix}L\right)$
$\displaystyle\equiv\begin{bmatrix}A&B\\\ C&D\\\ \end{bmatrix}.$ (11)

The second term on the right-hand side of Eq. (10) describes the quantum
fluctuation effect induced by the vacuum field reservoir. To enable
correspondence with more commonly used parameters, we use the substitution
$\frac{\alpha\Gamma}{4L}=\frac{g^{2}N}{c}$, where $\alpha$ denotes the OD of
the atomic medium. Therefore, the matrix elements in Eq. (11) are obtained as

$\displaystyle A$
$\displaystyle=\frac{1}{2}\left[1+e^{i\alpha\Gamma/2(\Delta-i\Gamma)}\right],$
(12) $\displaystyle B$
$\displaystyle=\frac{1}{2}\left[1-e^{i\alpha\Gamma/2(\Delta-i\Gamma)}\right]e^{-i(\phi_{c}-\phi_{d})},$
(13) $\displaystyle C$
$\displaystyle=\frac{1}{2}\left[1-e^{i\alpha\Gamma/2(\Delta-i\Gamma)}\right]e^{i(\phi_{c}-\phi_{d})},$
(14) $\displaystyle D$
$\displaystyle=\frac{1}{2}\left[1+e^{i\alpha\Gamma/2(\Delta-i\Gamma)}\right].$
(15)

In Eq. (10), $A$ and $C$ ($D$ and $B$) represent the mode-preserved and mode-
converted coefficients, respectively, of the probe (signal) field propagating
in the double-$\Lambda$ medium. These coefficients are similar to the
transmission and reflection coefficients of light passing through a spatial
beam splitter.

To investigate the response of the double-$\Lambda$ medium to different input
photon quantum states, the density matrix of the output state,
$\rho_{f}=\tilde{U}\rho_{i}\tilde{U}^{\dagger}$, is constructed in accordance
with the density matrix of the input state $\rho_{i}$ and the evolution
operator $\tilde{U}$ of the combined system in the frequency domain, where
$\rho_{i}=\rho^{PS}(z=0,\omega)\otimes\rho^{R}$ is in the product form of the
incident probe and signal fields $\rho^{PS}$ and the reservoir $\rho^{R}$. In
accordance with the reduced density operator approach in Ref. [35] and under
the steady-state condition, the matrix element of the output probe and signal
fields in the Fock-state basis is given by

$\displaystyle\rho^{PS}_{m_{p}m_{s}n_{p}n_{s}}(L,0)$
$\displaystyle=\textrm{Tr}\left\\{\tilde{\rho}_{m_{p}m_{s}n_{p}n_{s}}(L,0)\rho_{i}\right\\},$
(16)

where $\tilde{\rho}_{m_{p}m_{s}n_{p}n_{s}}(L,0)$ =
$\tilde{U}^{\dagger}\left(|n_{p}n_{s}\rangle\langle m_{p}m_{s}|\otimes
I_{R}\right)\tilde{U}$ is the operator for the matrix element of the output
probe and signal fields. If Eq. (10) is applied and the vacuum density matrix
with annihilation and creation operators are rewritten as $|0\rangle\langle
0|=\sum^{\infty}_{l=0}\frac{(-1)^{l}}{l!}(\tilde{a}^{\dagger})^{l}(\tilde{a})^{l}$,
the operator $\tilde{\rho}_{m_{p}m_{s}n_{p}n_{s}}$ can be derived as

$\displaystyle\tilde{\rho}_{m_{p}m_{s}n_{p}n_{s}}(L,0)$
$\displaystyle=\sum^{\infty}_{l_{p},l_{s}=0}\chi_{mnl}(A\tilde{a}^{\dagger}_{p,0}+B\tilde{a}^{\dagger}_{s,0})^{l_{p}+n_{p}}(C\tilde{a}^{\dagger}_{p,0}+D\tilde{a}^{\dagger}_{s,0})^{l_{s}+n_{s}}$
$\displaystyle\times(A^{\ast}\tilde{a}_{p,0}+B^{\ast}\tilde{a}_{s,0})^{l_{p}+m_{p}}(C^{\ast}\tilde{a}_{p,0}+D^{\ast}\tilde{a}_{s,0})^{l_{s}+m_{s}},$
(17)

where
$\chi_{mnl}=\frac{1}{\sqrt{m_{p}!m_{s}!n_{p}!n_{s}!}}\frac{(-1)^{l_{p}+l_{s}}}{l_{p}!l_{s}!}$.
In the derivation of Eq. (17), the normal-ordered noise caused by the Langevin
operator $\tilde{f}_{jk}$ is proportional to the atomic population of the
excited state [36]; hence, it can be ignored under the perturbation condition
of the weak probe and signal fields (see Appendix B for more details).

##  III Two coherent fields

We first consider the case of two coherent fields passing through the
double-$\Lambda$ medium. As indicated in Fig. 1(b), the two coherent states
with probe and signal frequency modes, respectively, are the initial states
$|\psi^{PS}_{i}\rangle=|\beta_{p}\rangle\otimes|\beta_{s}\rangle$. If Eqs.
(16) and (17) are calculated, the matrix element of the the output probe and
signal fields can be obtained as

$\displaystyle\rho^{PS}_{m_{p}m_{s}n_{p}n_{s}}(L,0)$
$\displaystyle=\sum^{\infty}_{l_{p},l_{s}=0}\chi_{mnl}(A\beta^{\ast}_{p}+B\beta^{\ast}_{s})^{l_{p}+n_{p}}(C\beta^{\ast}_{p}+D\beta^{\ast}_{s})^{l_{s}+n_{s}}$
$\displaystyle\quad\times(A^{\ast}\beta_{p}+B^{\ast}\beta_{s})^{l_{p}+m_{p}}(C^{\ast}\beta_{p}+D^{\ast}\beta_{s})^{l_{s}+m_{s}}$
$\displaystyle=e^{-|A^{\ast}\beta_{p}+B^{\ast}\beta_{s}|^{2}}\frac{(A^{\ast}\beta_{p}+B^{\ast}\beta_{s})^{m_{p}}(A\beta^{\ast}_{p}+B\beta^{\ast}_{s})^{n_{p}}}{\sqrt{m_{p}!n_{p}!}}$
$\displaystyle\quad\times
e^{-|C^{\ast}\beta_{p}+D^{\ast}\beta_{s}|^{2}}\frac{(C^{\ast}\beta_{p}+D^{\ast}\beta_{s})^{m_{s}}(C\beta^{\ast}_{p}+D\beta^{\ast}_{s})^{n_{s}}}{\sqrt{m_{s}!n_{s}!}},$
(18)

by which the output state of the probe and signal fields becomes

$\displaystyle|\psi^{PS}_{f}\rangle$
$\displaystyle=|A^{\ast}\beta_{p}+B^{\ast}\beta_{s}\rangle\otimes|C^{\ast}\beta_{p}+D^{\ast}\beta_{s}\rangle$
$\displaystyle\equiv|\beta^{\prime}_{p}\rangle\otimes|\beta^{\prime}_{s}\rangle,$
(19)

where both the output probe $|\beta^{\prime}_{p}\rangle$ and signal
$|\beta^{\prime}_{s}\rangle$ fields remain in coherent states, indicating that
the double-$\Lambda$ medium only coherently redistributes the incident probe
and signal amplitudes through the EIT-based FWM process without changing the
photon statistics of these two coherent fields or entangling them [37].
Furthermore, Eq. (19) reveals that the amplitude and phase changes of the
output probe and signal fields depend on the relative phase of the applied
light fields. The transmittances and phase shifts of the probe and signal
fields through the double-$\Lambda$ medium are given by

$\displaystyle T_{p}$
$\displaystyle=\left|\frac{\beta^{\prime}_{p}}{\beta_{p}}\right|^{2}=\left|A^{\ast}+u_{\beta}B^{\prime\ast}e^{-i\phi_{r}}\right|^{2},$
(20) $\displaystyle\Delta\phi_{p}$
$\displaystyle=\textrm{arg}\left[\frac{\beta^{\prime}_{p}}{\beta_{p}}\right]=\textrm{arg}\left[A^{\ast}+u_{\beta}B^{\prime\ast}e^{-i\phi_{r}}\right],$
(21) $\displaystyle T_{s}$
$\displaystyle=\left|\frac{\beta^{\prime}_{s}}{\beta_{s}}\right|^{2}=\left|u^{-1}_{\beta}C^{\prime\ast}e^{i\phi_{r}}+D^{\ast}\right|^{2},$
(22) $\displaystyle\Delta\phi_{s}$
$\displaystyle=\textrm{arg}\left[\frac{\beta^{\prime}_{s}}{\beta_{s}}\right]=\textrm{arg}\left[u^{-1}_{\beta}C^{\prime\ast}e^{i\phi_{r}}+D^{\ast}\right],$
(23)

where $B^{\prime\ast}\equiv B^{\ast}e^{-i(\phi_{c}-\phi_{d})}$,
$C^{\prime\ast}\equiv C^{\ast}e^{i(\phi_{c}-\phi_{d})}$,
$u_{\beta}\equiv|\beta_{s}|/|\beta_{p}|$ is the ratio of the probe and signal
amplitudes, and $\phi_{r}\equiv(\phi_{p}-\phi_{s})-(\phi_{c}-\phi_{d})$ is the
relative phase of the applied light fields. Figure 2 reveals that the
theoretical curves of transmittance and phase shift, as a function of the
relative phase $\phi_{r}$ for the probe and signal fields obtained from the
current quantum model (solid lines), are identical to those of the
semiclassical model (dotted lines) in Ref. [30]. If the input light fields are
all in coherent states, this closed-loop double-$\Lambda$ medium exhibits
phase-dependent properties, which have been experimentally investigated in
atomic ensembles [30, 31, 32].

Figure 2:  The closed-loop case of two incident weak coherent fields. (a)
Transmission and (b) phase shift of the probe (blue) and signal (red) coherent
fields versus $\phi_{r}$ plotted using the quantum (solid) and semiclassical
(dotted) models under conditions of $|\beta_{p}|=|\beta_{s}|$,
$\Delta=13\Gamma$, and $\textrm{OD}=50$.  Figure 3:  Hadamard gates for
single-photon two-color qubits. (a) Probability that the two-color qubit with
$u=1$ and $\phi_{u}=0$ is in the probe frequency mode ($P_{1p0s}$, solid blue)
and the signal frequency mode ($P_{0p1s}$, dashed red) versus OD under the
conditions of $\phi_{r}=\pi/2$ and $\Delta=(200/\pi)\Gamma$. Inset: For
$\Delta=(\textrm{OD}/\pi)\Gamma$, the maximum probability approaches 1 as OD
increases. (b) Theoretical $\Delta$ curves for maximizing the probabilities of
$|1_{p}0_{s}\rangle$ (solid blue) and $|0_{p}1_{s}\rangle$ (solid red) as a
function of $u$ for $\textrm{OD}=200$. The two curves intersect at $u=1$ where
$\Delta=(200/\pi)\Gamma$, which is the condition in (a).

##  IV Hadamard gates

We next consider a single-photon qubit case in which an incident single photon
has entangled probe and signal frequency modes, namely

$\displaystyle|\psi^{PS}_{i}\rangle=\frac{1}{\sqrt{1+u^{2}}}\left(|1_{p}0_{s}\rangle+ue^{-i\phi_{u}}|0_{p}1_{s}\rangle\right),$
(24)

where $u$ is a parameter indicating the amplitude ratio between the probe and
signal frequency modes, and $\phi_{u}$ denotes the relative phase between the
two modes. A single photon with a superposition state of two frequency modes
is also called a frequency-encoded photonic qubit or two-color qubit [38].
Here, $\phi_{u}$ is similar to the role of the relative phase
$(\phi_{p}-\phi_{s})$ between the two coherent fields mentioned in the
previous section. Therefore, in the case of a single-photon two-color qubit,
the double-$\Lambda$ medium can also be expected to exhibit phase-dependent
properties. To confirm this, the probability that the output single photon is
in the probe and signal frequency modes, respectively, can be determined using
Eqs. (16) and (17) as follows:

$\displaystyle
P_{1p0s}=\rho^{PS}_{1010}=\frac{1}{1+u^{2}}\left|A^{\ast}+uB^{\prime\ast}e^{-i\phi_{r}}\right|^{2},$
(25) $\displaystyle
P_{0p1s}=\rho^{PS}_{0101}=\frac{1}{1+u^{2}}\left|C^{\prime\ast}e^{i\phi_{r}}+uD^{\ast}\right|^{2},$
(26)

where $\phi_{r}=\phi_{u}-(\phi_{c}-\phi_{d})$ is equivalent to the relative
phase in the previous case of two coherent fields with
$\phi_{u}=\phi_{p}-\phi_{s}$. As expected, these results verify the dependence
of the output state of the single-photon qubit on the relative phase of the
applied light fields in the double-$\Lambda$ medium. Figure 3(a) plots the
theoretical curves of $P_{1p0s}$ and $P_{0p1s}$ as a function of OD under the
conditions of $\phi_{r}=\pi/2$ and $\Delta=\left(200/\pi\right)\Gamma$ for
single-photon qubits with equal-amplitude and in-phase modes, namely $u=1$ and
$\phi_{u}=0$. The theoretical curves reveal that $P_{1p0s}$ increases as OD
increases and reaches a maximum value close to 1 at OD = 200, whereas
$P_{0p1s}$ decreases to almost zero. In other words, under these conditions,
the two-color single photon entering the double-$\Lambda$ medium is converted
into a single photon carrying only one color, which is equivalent to the
Hadamard gate, and its operation fidelity can reach 0.99 at $\textrm{OD}=200$.

The physics of this frequency-domain Hadamard gate can be described as
follows. Because of the large detuning condition of
$\Delta=\left(\textrm{OD}/\pi\right)\Gamma$, the amplitudes of the mode-
preserved coefficient and mode-converted coefficient of the double-$\Lambda$
medium are almost equal; that is, $|A|\approx|C|\approx 0.5$ and
$|D|\approx|B|\approx 0.5$ [Eqs. (12)–(15)]. In this case, when
$\phi_{r}=\pi/2$, the probe mode-preserved coefficient $A^{\ast}$ and the
signal mode-converted coefficient $B^{\prime\ast}e^{-i\pi/2}$ in Eq. (25) are
almost in phase, resulting in constructive interference, which causes the
output photon to be converted into the probe frequency mode (i.e.,
$P_{1_{p}0_{s}}\approx$ 1). However, if $\phi_{r}$ is $3\pi/2$, the two
aforementioned coefficients have out-of-phase destructive interference,
causing the theoretical curves in Fig. 3(a) to become interchanged, and the
output photon is converted to the signal frequency mode, resulting in
$P_{0_{p}1_{s}}\approx$ 1.

For the input single-photon qubits with arbitrary $u$ values, the output of
$|1_{p}0_{s}\rangle$ can be achieved by using a specific detuning value
$\Delta$ that satisfies the $|A^{\ast}|=|uB^{\prime\ast}|$ condition. In this
case, constructive interference in the probe frequency mode occurs if
$\phi_{r}=\pi/2$. By contrast, the output of $|0_{p}1_{s}\rangle$ can be
achieved by simply setting the relative phase $\phi_{r}$ to $3\pi/2$ and
adjusting $\Delta$ to satisfy the condition $|C^{\prime\ast}|=|uD^{\ast}|$.
This suggests that the double-$\Lambda$ scheme can be used as a single-qubit
gate with x-rotation and y-rotation. Therefore, arbitrary single-qubit gates
can be realized by concatenating the x- and y-rotation gates based on this
scheme [39]. Figure 3(b) presents plots of two theoretical curves of $\Delta$
as a function of $u$ required to achieve the maximum probabilities of
$|1_{p}0_{s}\rangle$ and $|0_{p}1_{s}\rangle$ for $\textrm{OD}=200$. The two
curves intersect at $u=1$ and $\Delta=(200/\pi)\Gamma$, corresponding to the
condition $\textrm{OD}=200$ of Fig. 3(a). For one-color input single photons
($u=0$), the double-$\Lambda$ medium does not have phase-dependent properties
because no interference occurs [Eqs. (25) and (26)]. This system is also known
as an open-loop FWM system and can be used as an efficient frequency converter
[40, 41, 42, 43, 44].

##  V HOM interference

We now consider the case in which the two input single photons carry the probe
and signal frequency modes, respectively. This is a two-photon state with no
correlation between the modes, as depicted by
$|1_{p}\rangle\otimes|1_{s}\rangle$ in Fig. 1(c). As previously stated, under
the ideal conditions of an extremely large OD value, the double-$\Lambda$
medium causes little photon dissipation. Therefore, the output photons only
have the following three possible quantum states with corresponding
probabilities: (1) state $|2_{p}0_{s}\rangle$ , both photons exit under the
probe frequency mode with $P_{2p0s}$; (2) state $|0_{p}2_{s}\rangle$, both
photons exit under the signal frequency mode with $P_{0p2s}$; (3) state
$|1_{p}1_{s}\rangle$, photons exit with disparate colors with $P_{1p1s}$.
These three probabilities are obtained as

$\displaystyle P_{2p0s}=\rho^{PS}_{2020}=2\left|A^{\ast}B^{\ast}\right|^{2},$
(27) $\displaystyle
P_{0p2s}=\rho^{PS}_{0202}=2\left|C^{\ast}D^{\ast}\right|^{2},$ (28)
$\displaystyle
P_{1p1s}=\rho^{PS}_{1111}=\left|A^{\ast}D^{\ast}+B^{\ast}C^{\ast}\right|^{2}.$
(29)

Unlike the previous case of the input two-color qubit, the probabilities of
the output states in this case are not affected by the relative phase of the
applied light fields; they are phase-independent. This phenomenon is ascribed
to the quantum uncertainty of the relative phase between the two input single
photons, which prevents them from interfering with each other in the
double-$\Lambda$ medium. However, quantum interference of the two-photon state
occurs [Eqs. (27)–(29)].

Figure 4:  HOM interference and color swap for two input single photons. (a)
Theoretical curves for the probabilities $P_{2p0s}$ (solid blue), $P_{0p2s}$
(dashed red), and $P_{1p1s}$ (dotted black) versus $\Delta$ at
$\textrm{OD}=200$. HOM interference and two-photon swap occur at
$\Delta=(200/\pi)\Gamma$ and $\Delta=(100/\pi)\Gamma$, respectively. (b) At
$\Delta=(\textrm{OD}/\pi)\Gamma$, the maximum probabilities of $P_{2p0s}$ and
$P_{0p2s}$ tend to approach 1 as OD increases. (c) The fidelity of the HOM
two-photon NOON state versus OD.

Figure 4(a) presents plots of the probabilities $P_{2p0s}$, $P_{0p2s}$, and
$P_{1p1s}$ versus $\Delta$ at $\textrm{OD}=200$. These theoretical curves
reveal that if $\Delta=(\textrm{200}/\pi)\Gamma$, both $P_{2p0s}$ and
$P_{0p2s}$ are close to 0.5, whereas $P_{1p1s}$ approaches 0. That is, the
input two-photon state $|1_{p}1_{s}\rangle$ is almost completely suppressed by
the double-$\Lambda$ medium, and the two single photons tend to have the same
color, exhibiting HOM interference in the frequency domain. This phenomenon of
two-photon state interference is caused by the operation of the Hadamard gate.
If $\Delta=(\textrm{OD}/\pi)\Gamma$, the magnitudes of the coefficients for
mode preservation and mode conversion are almost equal (approximately 0.5),
which is the Hadamard gate operation and causes the transition amplitudes
$A^{\ast}D^{\ast}$ (describing the two photons preserved in their respective
modes) and $B^{\ast}C^{\ast}$ (indicating that the modes of the two photons
mutually swap) to have the same magnitude but be out of phase (i.e.,
$A^{\ast}D^{\ast}\approx-B^{\ast}C^{\ast}$). Therefore, destructive
interference between the two transition amplitudes in Eq. (29) occurs,
resulting in $P_{1p1s}\approx 0$.

When the laser is detuned as $(\textrm{100}/\pi)\Gamma$ in Fig. 4(a), the two
single photons exit with disparate colors; the state is $|1_{p}1_{s}\rangle$.
In contrast with the two-photon HOM interference produced in the Hadamard gate
operation, the mode-converted coefficients $B^{\ast}$ and $C^{\ast}$ are both
approximately 1 in this case, whereas the mode-preserved coefficients
$A^{\ast}$ and $D^{\ast}$ approach 0, resulting in $P_{1p1s}\approx 1$ and
$P_{2p0s}\approx P_{0p2s}\approx 0$. Thus, the two single photons mutually
swap colors in a near-perfect frequency conversion process and end up in the
same state $|1_{p}1_{s}\rangle$, which also shows that whether the output
photons generate HOM interference or swap colors can be controlled in this
double-$\Lambda$ scheme by simply adjusting the laser detuning conditions.

Figure 4(b) plots the theoretical curves of $P_{2p0s}$ and $P_{0p2s}$ versus
OD under the HOM interference condition of $\Delta$ =
$\left(\textrm{OD}/\pi\right)\Gamma$. The theoretical results indicate that
increasing OD can suppress the loss of output photons due to spontaneous
emission. Therefore, if the OD value is large, the loss of output photons can
be neglected, and its quantum state is obtained as

$\displaystyle|\psi^{PS}_{f}\rangle=\frac{1}{\sqrt{2}}\left[|2_{p}0_{s}\rangle+e^{-2i(\phi_{c}-\phi_{d})}|0_{p}2_{s}\rangle\right],$
(30)

which is a two-photon NOON state in the frequency domain that has practical
applications in photonic quantum metrology for its ability to conduct
precision phase measurements if used in optical interferometers [45]. Although
the amplitude of the two-photon NOON state is independent of the phases of the
applied light fields, the relative phase between modes $|2_{p}0_{s}\rangle$
and $|0_{p}2_{s}\rangle$ can be controlled by varying $\phi_{c}$ and
$\phi_{d}$. Furthermore, for actual cases of finite OD, the two input single
photons in the double-$\Lambda$ medium will be slightly dissipated, which not
only causes the output photons to become a mixed state but also reduces the
fidelity of the two-photon NOON state, as displayed in Fig. 4(c). The
theoretical curve reveals that the fidelity approaches 1 as the OD increases
owing to the proportional suppression of photon loss due to spontaneous
emission in the double-$\Lambda$ EIT medium. If $\textrm{OD}=500$, the
fidelity can reach 0.99, which is a condition that can currently be achieved
experimentally [46, 47, 48]. Hence, HOM interference based on the
double-$\Lambda$ scheme is applicable for generating high-purity two-photon
NOON states in the frequency domain.

Figure 5:  Performance of the color SWAP gate. (a) Average (solid line) and
standard deviation (gray area) of the SWAP gate fidelity plotted against the
OD obtained from the density matrices of the output results according to the
four computational input states. The vertical dash lines denote the OD of
around $500$ and $1000$ for fidelities reaching $0.98$ and $0.99$,
respectively. (b) Truth table of the SWAP process according to the output
probabilities using the four inputs when $\textrm{OD}=1000$ in (a). Under this
condition, the average success rate per gate operation is around $0.98$.

##  VI SWAP gates

In the previous section, we have shown that when the condition
$\Delta=\left(\textrm{OD}/2\pi\right)\Gamma$ is satisfied, the colors of two
input single photons are exchanged in the double-$\Lambda$ medium [see Fig.
4(a)]. Next, we demonstrate that this double-$\Lambda$ scheme can be used as a
high-fidelity two-qubit SWAP gate under this condition, which is essential for
scalable quantum networks [49, 50]. Figure 5(a) presents the average (curve)
and standard deviation (gray area) of the SWAP gate fidelity according to the
four computational input states (i.e., $|0_{p}0_{s}\rangle$,
$|1_{p}0_{s}\rangle$, $|0_{p}1_{s}\rangle$, and $|1_{p}1_{s}\rangle$) versus
OD. The theoretical curve shows that the average fidelity approaches 1 as the
OD increases. By contrast, the standard deviation of fidelity decreases with
increasing OD. When $\textrm{OD}=500$, the fidelity of the SWAP gate can reach
$0.98$. For the region of OD exceeding $1000$, the SWAP gate achieves a high
fidelity of $F=0.99$ with a low standard deviation of $0.01$. The truth table
of the SWAP process when $\textrm{OD}=1000$ is shown in Fig. 5(b). Under this
condition, the average success probability per gate operation reaches $0.98$,
corresponding to an error rate of $0.02$ for a single SWAP operation.

##  VII Conclusion

We propose an EIT-based double-$\Lambda$ FWM system for achieving efficient
HOM interference in the frequency domain. By considering quantized light
fields, we theoretically elucidated the quantum properties of the
double-$\Lambda$ medium. Our quantum model confirms that if the incident light
fields are coherent states, the double-$\Lambda$ medium under closed-loop
conditions has phase-dependent properties caused by the mutual interference of
the light fields, consistent with the predictions of the semiclassical model.
However, in the closed-loop case of two incident single photons, the
double-$\Lambda$ medium no longer has this phase-dependence; instead,
frequency-domain HOM interference of the two-photon state occurs. If laser
detuning is controlled, the proposed double-$\Lambda$ scheme can perform high-
fidelity Hadamard gate operations on frequency-encoded single-photon qubits,
and the scheme can generate HOM two-photon NOON states with a fidelity greater
than 0.99 at OD = 500. This near-resonant FWM scheme based on double-$\Lambda$
EIT is a novel and promising scheme for frequency-domain HOM interference with
high visibility, which has practical applications in photonic quantum
metrology. In addition, we demonstrated that the proposed scheme can be used
to realize arbitrary high-fidelity single-qubit gates and two-qubit SWAP gates
by simply controlling the laser detuning and phase. These double-$\Lambda$
FWM-based photonic gates can be easily integrated with various EIT-related
quantum technologies, such as quantum memories [51, 52, 53, 54, 55, 56] and
quantum repeaters [57, 58, 59, 60, 61, 62, 63], facilitating the development
of quantum networks and scalable quantum information processing [64, 65, 66,
67, 68].

## ACKNOWLEDGMENTS

The authors acknowledge Ite A. Yu, Ying-Cheng Chen, and Che-Ming Li for
helpful discussions. This work was supported by the National Science and
Technology Council of Taiwan under Grant No. 111-2112-M-006-027, No.
111-2639-M-007-001-ASP, and No. 111-2119-M-007-007.

##  APPENDIX A COEFFICIENTS OF COUPLED EQUATIONS

In Sec. II, we derive the frequency-domain coupled equations between the probe
field and the signal field, as shown in Eqs. (8) and (9). All coefficients in
Eq. (10) can be obtained as follows:

$\displaystyle\Lambda_{p}$
$\displaystyle=\Lambda_{s}=\frac{\alpha\Gamma}{4L}\left(\frac{1}{\Gamma+i\Delta}\right),$
(31) $\displaystyle\kappa_{p}$
$\displaystyle=\kappa_{s}=-\frac{\alpha\Gamma}{4L}\left(\frac{1}{\Gamma+i\Delta}\right),$
(32) $\displaystyle\zeta^{p}_{21}$
$\displaystyle=i\sqrt{\frac{\alpha\Gamma}{4L}}\left[\frac{i\Gamma-2\Delta}{(\Gamma+i\Delta)|\Omega|}\right]e^{-i\phi_{c}},$
(33) $\displaystyle\zeta^{p}_{31}$
$\displaystyle=-i\sqrt{\frac{\alpha\Gamma}{4L}}\left(\frac{1}{\Gamma+i\Delta}\right),$
(34) $\displaystyle\zeta^{p}_{41}$
$\displaystyle=i\sqrt{\frac{\alpha\Gamma}{4L}}\left(\frac{1}{\Gamma+i\Delta}\right)e^{-i(\phi_{c}-\phi_{d})},$
(35) $\displaystyle\zeta^{s}_{21}$
$\displaystyle=i\sqrt{\frac{\alpha\Gamma}{4L}}\left[\frac{i\Gamma}{(\Gamma+i\Delta)|\Omega|}\right]e^{-i\phi_{d}},$
(36) $\displaystyle\zeta^{s}_{31}$
$\displaystyle=i\sqrt{\frac{\alpha\Gamma}{4L}}\left(\frac{1}{\Gamma+i\Delta}\right)e^{i(\phi_{c}-\phi_{d})},$
(37) $\displaystyle\zeta^{s}_{41}$
$\displaystyle=-i\sqrt{\frac{\alpha\Gamma}{4L}}\left(\frac{1}{\Gamma+i\Delta}\right).$
(38)

##  APPENDIX B NOISE CORRELATIONS AND DIFFUSION COEFFICIENTS

In the derivation of Eq. (17), we assume that the Langevin noise operator
satisfies the delta correlation as follows:

$\displaystyle\langle\tilde{f}_{jk}(z,\omega)\tilde{f}_{k^{\prime}j^{\prime}}(z^{\prime},\omega^{\prime})\rangle=\frac{L}{2\pi
c}D_{jk,k^{\prime}j^{\prime}}\delta(z-z^{\prime})\delta(\omega-\omega^{\prime}),$
(39)

where $D_{jk,k^{\prime}j^{\prime}}$ is the diffusion coefficient, which can be
obtained using the Einstein relation [69] as follows:

$\displaystyle D_{jk,k^{\prime}j^{\prime}}$
$\displaystyle=\frac{\partial}{\partial{t}}\langle\hat{\sigma}_{jk}\hat{\sigma}_{k^{\prime}j^{\prime}}\rangle-\left\langle\left(\frac{\partial}{\partial{t}}\hat{\sigma}_{jk}-\hat{F}_{jk}\right)\hat{\sigma}_{k^{\prime}j^{\prime}}\right\rangle$
$\displaystyle-\left\langle\hat{\sigma}_{jk}\left(\frac{\partial}{\partial{t}}\hat{\sigma}_{k^{\prime}j^{\prime}}-\hat{F}_{k^{\prime}j^{\prime}}\right)\right\rangle.$
(40)

For the case of $\gamma_{21}=0$ considered in the main text, we can obtain the
diffusion coefficient of the double-$\Lambda$ atomic system through Eq. (40).
Its matrix form is

$\displaystyle D_{jk,k^{\prime}j^{\prime}}$
$\displaystyle=\begin{bmatrix}D_{2112}&D_{2113}&D_{2114}\\\
D_{3112}&D_{3113}&D_{3114}\\\ D_{4112}&D_{4113}&D_{4114}\end{bmatrix}$
$\displaystyle=\begin{bmatrix}\frac{\Gamma}{2}(\langle\hat{\sigma}_{33}\rangle+\langle\hat{\sigma}_{44}\rangle)&0&0\\\
0&0&0\\\ 0&0&0\end{bmatrix}.$ (41)

Under the condition of the weak probe and signal fields,
$\langle\hat{\sigma}_{33}\rangle\approx\langle\hat{\sigma}_{44}\rangle\approx
0$, that is, all the normal-ordered noise correlations
$\langle\tilde{f}_{jk}\tilde{f}_{k^{\prime}j^{\prime}}\rangle$ are close to
zero. According to Eq. (10), the normal-ordered noise correlations contribute
additional photons to the probe and signal fields, i.e.
$\langle\tilde{a}^{\dagger}_{p,L}\tilde{a}_{p,L}\rangle$ and
$\langle\tilde{a}^{\dagger}_{s,L}\tilde{a}_{s,L}\rangle$, which can be
regarded as the spontaneous-emitted noise photons induced by the vacuum field
reservoir. Additionally, we calculate the anti-normal-ordered noise terms,
$\langle\tilde{f}_{k^{\prime}j^{\prime}}\tilde{f}_{jk}\rangle$. Under the same
conditions, the corresponding diffusion coefficient matrix is

$\displaystyle D_{k^{\prime}j^{\prime},jk}$
$\displaystyle=\begin{bmatrix}D_{1221}&D_{1231}&D_{1241}\\\
D_{1321}&D_{1331}&D_{1341}\\\ D_{1421}&D_{1431}&D_{1441}\end{bmatrix}$
$\displaystyle\approx\begin{bmatrix}0&0&0\\\
0&\Gamma\langle\hat{\sigma}_{11}\rangle&0\\\
0&0&\Gamma\langle\hat{\sigma}_{11}\rangle\end{bmatrix}.$ (42)

Unlike the normal-ordered noise terms, Eq. (42) indicates that the anti-
normal-ordered noise correlations in this double-$\Lambda$ system do not
disappear even in the case of weak probe and signal fields. In Appendix C, we
will show that the anti-normal-ordered noise terms lead to the dissipation of
the incident light fields.

##  APPENDIX C COMMUTATION RELATION OF THE FIELD OPERATORS

The frequency-domain commutation relation of the creation and annihilation
operators for the probe field can be expressed as

$\displaystyle\left[\tilde{a}_{p}(z,\omega),\tilde{a}^{\dagger}_{p}(z,\omega^{\prime})\right]=\frac{L}{2\pi
c}\delta(\omega-\omega^{\prime}).$ (43)

We use Eq. (10) in Sec. II to calculate the communication relation of the
probe field propagating through the double-$\Lambda$ atomic medium as follows:

$\displaystyle\langle\tilde{a}^{\dagger}_{p,L}\tilde{a}_{p,L}\rangle$
$\displaystyle=|A^{\ast}|^{2}\langle\tilde{a}^{\dagger}_{p,0}\tilde{a}_{p,0}\rangle+|B^{\ast}|^{2}\langle\tilde{a}^{\dagger}_{s,0}\tilde{a}_{s,0}\rangle$
$\displaystyle+\frac{L}{2\pi
c}\sum_{jk}\sum_{j^{\prime}k^{\prime}}\int^{L}_{0}\mathcal{P}_{jk}D_{jk,k^{\prime}j^{\prime}}\mathcal{P}^{\ast}_{j^{\prime}k^{\prime}}dz,$
(44) $\displaystyle\langle\tilde{a}_{p,L}\tilde{a}^{\dagger}_{p,L}\rangle$
$\displaystyle=|A^{\ast}|^{2}\langle\tilde{a}_{p,0}\tilde{a}^{\dagger}_{p,0}\rangle+|B^{\ast}|^{2}\langle\tilde{a}_{s,0}\tilde{a}^{\dagger}_{s,0}\rangle$
$\displaystyle+\frac{L}{2\pi
c}\sum_{jk}\sum_{j^{\prime}k^{\prime}}\int^{L}_{0}\mathcal{P}^{\ast}_{j^{\prime}k^{\prime}}D_{k^{\prime}j^{\prime},jk}\mathcal{P}_{jk}dz,$
(45)

where $\mathcal{P}_{jk}=e^{-M(L-z)}\zeta^{p}_{jk}$. Substituting Eqs. (44) and
(45) into Eq. (43), and combining the results of Eqs. (41) and (42), the
expectation value of the commutator of the output probe field is derived as

$\displaystyle\left\langle[\tilde{a}_{p,L},\tilde{a}^{\dagger}_{p,L}]\right\rangle$
$\displaystyle=\frac{L}{2\pi c}\left(|A^{\ast}|^{2}+|B^{\ast}|^{2}\right)$
$\displaystyle+$ $\displaystyle\frac{L}{2\pi
c}\int^{L}_{0}(\mathcal{P}^{\ast}_{31}D_{1331}\mathcal{P}_{31}+\mathcal{P}^{\ast}_{41}D_{1441}\mathcal{P}_{41})dz.$
(46)

Based on Eqs. (C1) and (C4), we obtain

$\displaystyle\int^{L}_{0}(\mathcal{P}^{\ast}_{31}D_{1331}\mathcal{P}_{31}$
$\displaystyle+\mathcal{P}^{\ast}_{41}D_{1441}\mathcal{P}_{41})dz$
$\displaystyle=1-|A^{\ast}|^{2}-|B^{\ast}|^{2}.$ (47)

The left-hand side of Eq. (47) comes from the contribution of anti-normal-
ordered noise correlations. This indicates that once the output probe field is
dissipated, i.e. $|A^{\ast}|^{2}+|B^{\ast}|^{2}<1$, it must be caused by the
anti-normal-ordered noise correlations. However, for a double-$\Lambda$ medium
with a sufficiently large OD, $|A^{\ast}|^{2}+|B^{\ast}|^{2}\approx 1$, that
is, the contribution of anti-normal-ordered noise terms approaches to zero,
indicating that the vacuum field reservoir is unable to disturb the lossless
system through quantum fluctuations. The formula for the output signal field
can also be obtained as follows:

$\displaystyle\int^{L}_{0}(\mathcal{Q}^{\ast}_{31}D_{1331}\mathcal{P}_{31}$
$\displaystyle+\mathcal{Q}^{\ast}_{41}D_{1441}\mathcal{P}_{41})dz$
$\displaystyle=1-|C^{\ast}|^{2}-|D^{\ast}|^{2},$ (48)

where $\mathcal{Q}_{jk}=e^{-M(L-z)}\zeta^{s}_{jk}$.

## References

  * [1] C. K. Hong, Z. Y. Ou, and L. Mandel, “Measurement of subpicosecond time intervals between two photons by interference,” Phys. Rev. Lett. 59, 2044 (1987).
  * [2] R. Lopes, A. Imanaliev, A. Aspect, M. Cheneau, D. Boiron, and C. I. Westbrook, “Atomic Hong-–Ou–-Mandel experiment,” Nature, 520, 66 (2015).
  * [3] F. Bouchard, Alicia Sit, Y. Zhang, R. Fickler, F. M. Miatto, Y. Yao, F. Sciarrino, and E. Karimi, “Two-photon interference: the Hong–-Ou–-Mande effect,” Rep. Prog. Phys. 84, 012402 (2020).
  * [4] I. Afek, O. Ambar, and Y. Silberberg, “High-NOON states by mixing quantum and classical light,” Science 328, 879 (2010).
  * [5] M. C. Tichy, “Interference of identical particles from entanglement to boson-sampling,” J. Phys. B 47, 103001 (2014).
  * [6] M. Erhard, M. Malik, M. Krenn, and A. Zeilinger, “Experimental Greenberger–Horne–Zeilinger entanglement beyond qubits,” Nat. Photonics 12, 759 (2018).
  * [7] H. Wang et al., “Boson Sampling with 20 Input Photons and a 60-Mode Interferometer in a $10^{14}$-Dimensional Hilbert Space,” Phys. Rev. Lett. 123, 250503 (2019).
  * [8] A. N. Boto, P. Kok, D. S. Abrams, S. L. Braunstein, C. P. Williams, and J. P. Dowling, “Quantum Interferometric Optical Lithography: Exploiting Entanglement to Beat the Diffraction Limit,” Phys. Rev. Lett. 85, 2733 (2000).
  * [9] M. W. Mitchell, J. S. Lundeen, and A. M. Steinberg, “Super-resolving phase measurements with a multiphoton entangled state,” Nature (London) 429, 161 (2004).
  * [10] M. Müller, H. Vural, C. Schneider, A. Rastelli, O. G. Schmidt, S. Höfling, and P. Michler, “Quantum-Dot Single-Photon Sources for Entanglement Enhanced Interferometry,” Phys. Rev. Lett. 118, 257402 (2017).
  * [11] E. Polino, M. Valeri, N. Spagnolo, and F. Sciarrino, “Photonic quantum metrology,” AVS Quantum Sci. 2, 024703 (2020).
  * [12] S. Ramelow, L. Ratschbacher, A. Fedrizzi, N. K. Langford, and A. Zeilinger, “Discrete Tunable Color Entanglement,” Phys. Rev. Lett. 103, 253601 (2009).
  * [13] C. Joshi, A. Farsi, S. Clemmen, S. Ramelow, and A. L. Gaeta, “Frequency multiplexing for quasi-deterministic heralded single-photon sources,” Nat. Commun. 9, 847 (2018).
  * [14] M. Kues, C. Reimer, J. M. Lukens, W. J. Munro, A. M. Weiner, D. J. Moss, and Roberto Morandotti, “Quantum optical microcombs,” Nat. Photonics 13, 170 (2019).
  * [15] G. Maltese, M. I. Amanti, F. Appas, G. Sinnl, A. Lemaître, P. Milman, F. Baboux and S. Ducci, “Generation and symmetry control of quantum frequency combs,” npj Quantum Information 6, 13 (2020).
  * [16] H.-P. Lo and H. Takesue, “Precise tuning of single-photon frequency using an optical single sideband modulator,” Optica 4, 919 (2017).
  * [17] P. Imany, O. D. Odele, M. S. Alshaykh, H.-H. Lu, D. E. Leaird, and A. M. Weiner, “Frequency-domain Hong–Ou–Mandel interference with linear optics,” Opt. Lett. 43 (12), 2760 (2018).
  * [18] H.-H. Lu, J. M. Lukens, N. A. Peters, B. P. Williams, A. M. Weiner, and P. Lougovski, “Quantum interference and correlation control of frequency-bin qubits,” Optica 5, 1455 (2018).
  * [19] T. Kobayashi et al., “Frequency-domain Hong–Ou–Mandel interference,” Nat. Photonics 10, 441 (2016).
  * [20] Y. Chen, M. Fink, F. Steinlechner, J. P. Torres and R. Ursin, “Hong-Ou-Mandel interferometry on a biphoton beat note,” npj Quantum Information 5, 43 (2019).
  * [21] C. Joshi et al., “Frequency-Domain Quantum Interference with Correlated Photons from an Integrated Microresonator,” Phys. Rev. Lett. 124, 143601 (2020).
  * [22] A. K. Kashi and M. Kues, “Spectral Hong–Ou–Mandel Interference between Independently Generated Single Photons for Scalable Frequency-Domain Quantum Processing,” Laser Photon. Rev. 15, 2000464 (2021).
  * [23] J. S. Pelc, C. Langrock, Q. Zhang, and M. M. Fejer, “Influence of domain disorder on parametric noise in quasi-phase-matched quantum frequency converters,” Opt. Lett. 35, 2804 (2010).
  * [24] M. Fleischhauer, A. Imamoglu, and J. P. Marangos, “Electromagnetically induced transparency: Optics in coherent media,” Rev. Mod. Phys. 77, 633 (2005).
  * [25] C.-Y. Cheng, J.-J. Lee, Z.-Y. Liu, J.-S. Shiu, and Y.-F. Chen, “Quantum frequency conversion based on resonant four-wave mixing,” Phys. Rev. A 103, 023711 (2021).
  * [26] M. O. Scully and M. S. Zubairy, Quantum Optics (Cambridge University Press, Cambridge, England, 1997).
  * [27] E. A. Korsunsky and D. V. Kosachiov, “Phase-dependent nonlinear optics with double-$\Lambda$ atoms,” Phys. Rev. A 60, 4996 (1999).
  * [28] Y.-H. Chen, P.-J. Tsai, I. A. Yu, Y.-C. Chen, and Y.-F. Chen “Phase-dependent double-$\Lambda$ electromagnetically induced transparency,” arXiv:1409.4153.
  * [29] M. Artoni and A. Zavatta, “Large Phase-by-Phase Modulations in Atomic Interfaces,” Phys. Rev. Lett. 115, 113005 (2015).
  * [30] Z.-Y. Liu, Y.-H. Chen, Y.-C. Chen, H.-Y. Lo, P.-J. Tsai, I. A. Yu, Y.-C. Chen, and Y.-F. Chen, “Large Cross-Phase Modulations at the Few-Photon Level,” Phys. Rev. Lett. 117, 203601 (2016).
  * [31] S. Sagona-Stophel, R. Shahrokhshahi, B. Jordaan, M. Namazi, and E. Figueroa, “Conditional $\pi$-Phase Shift of Single-Photon-Level Pulses at Room Temperature,” Phys. Rev. Lett. 125, 243601 (2020).
  * [32] B. Kim and H. Kang, “Cross-phase modulation between pulse matched lights in optical loop system,” Opt. Commun. 508, 127753 (2022).
  * [33] M. Fleischhauer and M. D. Lukin, “Quantum memory for photons: Dark-state polaritons,” Phys. Rev. A 65, 022314 (2002).
  * [34] P. Kolchin, “Electromagnetically-induced-transparency-based paired photon generation,” Phys. Rev. A 75, 033814, (2007).
  * [35] H. Hsu, C.-Y. Cheng, J.-S. Shiu, L.-C. Chen, and Y.-F. Chen, “Quantum fidelity of electromagnetically induced transparency: the full quantum theory,” Opt. Express 30, 2097 (2022).
  * [36] N. Lauk, C. O’Brien, and M. Fleischhauer, “Fidelity of photon propagation in electromagnetically induced transparency in the presence of four-wave mixing,” Phys. Rev. A 88, 013823 (2013).
  * [37] K.-F. Chang, T.-P. Wang, C.-Y Chen, Y.-H. Chen, Y.-S. Wang, Y.-F. Chen, Y.-C. Chen, and I. A. Yu, “Low-loss high-fidelity frequency beam splitter with tunable split ratio based on electromagnetically induced transparency,” Phys. Rev. Research 3, 013096 (2021).
  * [38] M.-J. Lee, J. Ruseckas, C.-Y. Lee, V. Kudriašov, K.-F. Chang, H.-W. Cho, G. Juzeliānas, and I. A. Yu “Experimental demonstration of spinor slow light,” Nat. Commun. 5, 5542 (2014).
  * [39] M. A. Nielsen and I. L. Chuang, Quantum Computation and Quantum Information (Cambridge University Press, Cambridge, 2000).
  * [40] C.-K. Chiu, Y.-H. Chen, Y.-C. Chen, I. A. Yu, Y.-C. Chen, and Y.-F. Chen, “Low-light-level four-wave mixing by quantum interference,” Phys. Rev. A 89, 023839 (2014).
  * [41] C.-Y. Lee, B.-H. Wu, G. Wang, Y.-F. Chen, Y.-C. Chen, and I. A. Yu, “High conversion efficiency in resonant four-wave mixing processes,” Optics Express 24, 1008 (2016).
  * [42] Z.-Y. Liu, J.-T. Xiao, J.-K. Lin, J.-J. Wu, J.-Y. Juo, C.-Y. Cheng and Y.-F. Chen, “High-efficiency backward four-wave mixing by quantum interference,” Sci. Rep. 7, 15796, (2017).
  * [43] J.-Y. Juo, J.-K. Lin, C.-Y. Cheng, Z.-Y. Liu, I. A. Yu, and Y.-F. Chen, “Demonstration of spatial-light-modulation-based four-wave mixing in cold atoms,” Phys. Rev. A 97, 053815 (2018).
  * [44] C.-Y. Cheng et al., “Efficient frequency conversion based on resonant four-wave mixing,” Opt. Lett. 46, 681 (2021).
  * [45] S. Slussarenko, M. M. Weston, H. M. Chrzanowski, L. K. Shalm, V. B. Verma, S. W. Nam, and G. J. Pryde, “Unconditional violation of the shot-noise limit in photonic quantum metrology,” Nat. Photonics 11, 700 (2017).
  * [46] F. Blatt, T. Halfmann, and T. Peters, “One-dimensional ultracold medium of extreme optical depth,” Opt. Lett. 39, 446 (2014).
  * [47] Y.-F. Hsiao, H.-S. Chen, P.-J. Tsai, and Y.-C. Chen, “Cold atomic media with ultrahigh optical depths,” Phys. Rev. A 90, 055401 (2014).
  * [48] K. T. Kaczmarek et al., “Ultrahigh and persistent optical depths of cesium in Kagomé-type hollow-core photonic crystal fibers,” Opt. Lett. 40, 5582 (2015).
  * [49] M. Saeedi, R. Wille and Rolf Drechsler, “Synthesis of Quantum Circuits for Linear Nearest Neighbor Architectures,” Quantum Inf. Process 10, 355 (2011).
  * [50] A. G. Fowler, M. Mariantoni, J. M. Martinis, and A. N. Cleland, “Surface codes: Towards practical large-scale quantum computation,” Phys. Rev. A 86, 032324 (2012).
  * [51] M. Fleischhauer and M. D. Lukin, “Dark-state polaritons in electromagnetically induced transparency,” Phys. Rev. Lett. 84, 5094 (2000).
  * [52] A. I. Lvovsky, B. C. Sanders, and W. Tittel, “Optical quantum memory,” Nat. Photonics 3, 706 (2009).
  * [53] Y.-F. Hsiao et al., “Highly Efficient Coherent Optical Memory Based on Electromagnetically Induced Transparency,” Phys. Rev. Lett. 120, 183602 (2018).
  * [54] P. Vernaz-Gris, K. Huang, M. Cao, A. S. Sheremet, and J. Laurat, “Highly-efficient quantum memory for polarization qubits in a spatially-multiplexed cold atomic ensemble,” Nat. Commun. 9, 363 (2018).
  * [55] Y. Wang et al., “Efficient quantum memory for single-photon polarization qubits,” Nat. Photonics 13, 346 (2019).
  * [56] N. Jiang, Y.-F. Pu, W. Chang, C. Li, S. Zhang and L.-M. Duan, “Experimental realization of 105-qubit random access quantum memory,” npj Quantum Information 5, 28 (2019).
  * [57] H.-J. Briegel, W. Dür, J. I. Cirac, and P. Zoller, “Quantum Repeaters: The Role of Imperfect Local Operations in Quantum Communication,” Phys. Rev. Lett. 81, 5932 (1998).
  * [58] L.-M. Duan, M. D. Lukin, J. I. Cirac, and P. Zoller, “Long-distance quantum communication with atomic ensembles and linear optics,” Nature 414, 413 (2001).
  * [59] Z.-S. Yuan, Y.-A. Chen, B. Zhao, S. Chen, J. Schmiedmayer, and J.-W. Pan, “Experimental demonstration of a BDCZ quantum repeater node,” Nature 454, 1098 (2008).
  * [60] K. Hammerer, A. S. Sørensen, and E. S. Polzik, “Quantum interface between light and atomic ensembles,” Rev. Mod. Phys. 82, 1041 (2010).
  * [61] N. Sangouard, C. Simon, H. de Riedmatten, and N. Gisin, “Quantum repeaters based on atomic ensembles and linear optics,” Rev. Mod. Phys. 83, 33 (2011).
  * [62] S. Langenfeld, O. Morin, M. Körber and G. Rempe, “A network-ready random-access qubits memory,” npj Quantum Information 6, 86 (2020).
  * [63] X. Liu et al., “Heralded entanglement distribution between two absorptive quantum memories,” Nature 594, 41 (2021).
  * [64] H. J. Kimble, “The quantum internet,” Nature 453, 1023 (2008).
  * [65] F. Mei, M. Feng, Y.-F. Yu, and Z.-M. Zhang, “Scalable quantum information processing with atomic ensembles and flying photons,” Phys. Rev. A 80, 042319 (2009).
  * [66] S. D. Barrett, P. P. Rohde, and T. M. Stace, “Scalable quantum computing with atomic ensembles,” New J. Phys. 12, 093032 (2010).
  * [67] C. Simon, “Towards a global quantum network,” Nat. Photonics 11, 678 (2017).
  * [68] H. Li et al., “Multipartite entanglement of billions of motional atoms heralded by single photon,” npj Quantum Information 7, 146 (2021).
  * [69] J. C. Garrison and R. Y. Chiao, Quantum Optics (Oxford Graduate Texts, 2014).


# Contextual Emotion Estimation from
Image Captions  ††thanks: This work was supported by NSERC Discovery Grant
06908-2019.

Vera Yang, Archita Srivastava, Yasaman Etesam, Chuxuan Zhang, Angelica Lim
School of Computing Science
Simon Fraser University
Burnaby, Canada
{veray, archita_srivastava, yetesam, chuxuan_zhang<EMAIL_ADDRESS>
###### Abstract

Emotion estimation in images is a challenging task, typically using computer
vision methods to directly estimate people’s emotions using face, body pose
and contextual cues. In this paper, we explore whether Large Language Models
(LLMs) can support the contextual emotion estimation task, by first captioning
images, then using an LLM for inference. First, we must understand: how well
do LLMs perceive human emotions? And which parts of the information enable
them to determine emotions? One initial challenge is to construct a caption
that describes a person within a scene with information relevant for emotion
perception. Towards this goal, we propose a set of natural language
descriptors for faces, bodies, interactions, and environments. We use them to
manually generate captions and emotion annotations for a subset of 331 images
from the EMOTIC dataset. These captions offer an interpretable representation
for emotion estimation, towards understanding how elements of a scene affect
emotion perception in LLMs and beyond. Secondly, we test the capability of a
large language model to infer an emotion from the resulting image captions. We
find that GPT-3.5, specifically the text-davinci-003 model, provides
surprisingly reasonable emotion predictions consistent with human annotations,
but accuracy can depend on the emotion concept. Overall, the results suggest
promise in the image captioning and LLM approach.

###### Index Terms:

Large language model, emotion estimation, image captioning, context, ChatGPT,
GPT-3.5

Figure 1: Manual Annotation for the given image produces the following
caption: Sean is a male adult. Sean is a(n) passenger. Sean is or has raising
eyebrows, side-eyeing. Mia is a child and she is sitting behind Sean and
kicking Sean’s chair. Sean’s physical environment is on an airplane.

##  I Introduction

_“She sat in a hospital hallway, with an empty stare and slumped shoulders.”_
How does this person feel? Writers have long known that describing a scene
with carefully selected words, without specifically naming the emotion, is an
effective way of moving their reader. The ability to place ourselves in the
shoes of another underlies our ability to infer their emotion, towards taking
socially appropriate and empathetic actions. Similarly, a photo can capture
the emotion of a person in a scene. Automatic emotion estimation systems based
on images or videos have the potential to facilitate better human-machine
interaction, yet performance in the wild is still poor [1].

Many emotion recognition studies focus on using facial [2] or body[3]
features. The context in which emotions are expressed can also affect the
perception of emotions [4, 5, 6, 7], whether the face is visible or covered in
the image. As a result, the context-based emotion recognition task was
introduced. It was elaborated with the introduction of the EMOTIC dataset [8],
and there has been an increased focus on improving accuracy on this task [9,
10, 11]. These models utilize a variety of inputs beyond facial data by
including, for examples, body posture and context, which encompass factors
such as the presence of other humans or environmental aspects. Context-based
emotion recognition in audio-visual media has also been studied [12], but how
exactly specific cues contribute to the detected emotion remains a relatively
unexplored area [13].

In recent years, large language models (LLMs) have emerged as a hot topic in
the field of Natural Language Processing (NLP). This growth can be attributed
to the introduction of transformers in 2017 by Vaswani et al. [14], which
provided a more efficient way of processing sequences of data. Subsequently,
other researchers have introduced various methods based on transformer
encoder/decoder structures and different pre-training techniques [15, 16, 17].
These approaches have allowed sophisticated language models to perform a range
of tasks with high accuracy and efficiency. These improvements in LLMs paved
the way not only to the improvements in NLP problems, but also to many multi-
modal problems such as Visual Question Answering [18], and Caption Generation
[19]. The ability of these models to understand human language and store data
in their extensive neural network attributed this success. At the same time,
to what extent these language models have the ability to perceive human
emotion remains an open question.

In this study, we aim to answer the following questions: how well do LLMs
perceive human emotions? And which parts of the information enable them to
determine emotions? We first created an annotation interface that allows for
annotating images with various factors related to emotion, such as physical
signs, social interactions, environmental cues, and demographic information.
Using this information, we created an image caption describing a person’s
facial expressions and body poses, their social contexts with other people in
an image, and their environmental surrounding. We then passed the image
caption to a GPT-3.5111https://platform.openai.com/docs/models/gpt-3-5 model
to predict an emotion from the text description only.

We conducted an emotion prediction experiment using full image captions,
followed by two ablation studies using cropped image captions. For the
ablation studies, we altered our image captions by selectively removing
certain types of contextual information, such as social interactions and
environmental features. Through the ablation studies, we aim to evaluate the
impact of each type of data on the emotion detection output. To summarize, our
contributions are:

  * •

Compiling a set of physical signals for each of our emotion labels using LLMs
and a writer’s thesaurus [20].

  * •

Developing an interface to annotate the image data and collected emotion
labels and description showing the physical signals, human interactions, and
environmental features for each person.

  * •

Providing an initial analysis on GPT-3.5’s ability to predict human emotion
from image captions, and how well it can predict the emotion given information
on physical signals and contextual information.

  * •

Analyzing the importance of context on how large language models perceive
emotions, and how different types of context will affect the prediction.

##  II Methodology

TABLE I: Physical Signals Used in Annotated Image Captions Signal Categories |  Physical Signals
---|---
Eyes Signals |  Closed eyes, Frowning, Staring off into the distance, Furrowed eyebrows, Glaring, Side-eyeing, Averted gaze, Looking up, Squeezing eyes shut, Rolling eyes, Avoiding eye contact, Looking away, Lowered eyebrows, Empty stare, Raising eyebrows, Squinting eyes, Looking sideway, Eyebrows squishing together, Eyes wide open, Looking down, Peeking, Eyes are damp and bright, Staring down at the ground, Unfocused gaze, Downcast eyes, Raising one’s eyebrows, Gaze clouding, Glassy stare, Refusing to look, Glancing as if looking for answers, Eyes are cold, Side look
Mouth Signals |  Gritting teeth, Open mouth, Mouth wide open, Clenched jaw, Downturned mouth, Biting lips, Lips that flatten, Smiling, Sticking tongue out, Curling lip, Poking one’s tongue into the cheek, Yawning, Smirking, Breathing excessively, Biting finger, Pressing lips tight, Grimacing
Facial Signals |  High chin, Flat expression, Crying, Chin dipping down, Resting one side of face on hand, Resting chin on hand, Tilting one’s head to the side, Smelling oneself, Hiding face in arms, Tilting head downward, Leaning head on hand, Using something to hide face, Wrinkling nose, Resting forehead on hand, Puffing out the cheeks, Tilting head upward, Cheek resting on own palm, Expression that appears pained
Body Signals |  Sitting, Bending down, Bent spine, Bending forward, Visible sweating, Leaning back, Hunched shoulders, Leaning forward, Body posture that loosens or collapses, Lying flat, Bracing against a wall, Chest caving, Bending forward and laying head on arms, Shoulders slumping or curling forward, Leaning on an object, Falling onto ground, Tight shoulders, Naked body, Tense posture, Picking fights, Body freezing in place, Turning body away, Curling up body into a ball, Stiff posture, Slouching on an object, Hood over the head, Throwing up, Covering oneself with something, Keeping one’s back to a wall, Sliding down in a chair, Slouching and leaning on objects, Visible tension in the neck, Hands covering face, Pointing fingers, Pulling a hood over the head, Rubbing the affected area of one’s body
Hands Signals |  Hand up in air, Folding arms across the chest, Hand wiping tears, Covering face with hands, Thumbs down, Pressing a fist to the mouth, Throwing things, Pointing middle finger, Palms open, Curling fingers, Rubbing one’s forehead or eyebrows, Hands in pockets, Crossed arms, Squeezing nose, Grabbing onto someone, Hand on own chest, Hand resting on forehead, Hands on both sides of head, Rubbing the eyes, Grabbing own hair, Clenched fists, Pointing finger, Scratching at cheek or temple, Rubbing the foot, Arms reaching out, Gripping something and knuckles going white, Palms covering forehead, Palms up facing outward, Hands on the hip, Rubbing temples, Rubbing the back of the neck, Hands covering ears, Rubbing the back, Taking off eyeglasses, Hand on neck, Throwing hands up in the air, Wiping tears, Rubbing the shoulder, Sweeping hand across the forehead to get rid of sweat, Clapping palms together, Hand covering mouth, Rubbing the chest, Hands curling around their body, Rubbing the nose, Clutching the stomach, Hand touching the lips, Nervous hand gestures
Feet Signals |  Knees pulling together, Bringing the feet together, Squatting

The ultimate goal of our research is to explore whether automatic emotion
estimation of people in images could be implemented by first captioning images
appropriately, then feeding the caption to a large language model for
inference on the text. Our approach is comprised of three steps: a) generating
a large list of physical signals used for writing about emotion, b) annotating
images using these signals along with questions about demographic information,
interaction, and environment, and c) using a large language model to predict
an emotion based on the image caption.

###  II-A Generation of Physical Signals

Because existing algorithms perform relatively poorly on negative labels
compared to positive ones (e.g. 14.5% vs. 40.3% [8]), we focused on the 13
negative emotion labels from the EMOTIC dataset [8]: Anger, Annoyance,
Aversion, Confusion, Disapproval, Disconnection, Disquietment, Embarrassment,
Fatigue, Fear, Pain, Sadness, and Suffering. To mitigate an annotator’s
potential confusion with the labels Pain and Suffering, we merged and replaced
these labels with Pain/Suffering - Emotional and Pain/Suffering - Physical.

As a first step, we generated descriptions of physical signals indicative of
our 13 target emotions. We used an Emotion Thesaurus, “A Writer’s Guide to
Character Expression” by Becca Puglisi and Angela Ackerman [20], which
provided a range of physical signals associated with commonly recognized
emotions including Anger, Annoyance, Confusion, Embarrassment, Fear, and
Sadness. For emotions not listed in the book, we utilized the Large Language
Models ChatGPT and GPT-3.5 to generate a list of physical descriptions or
expressions associated with each emotion label. The prompts used to generate
the physical descriptions were of the form, “List physical cues/physical
expressions that would indicate the emotion of ‘disapproval’ in an image.” and
“Give a list of facial expressions/physical descriptions/physical movements
that might indicate that a person is feeling ‘fatigued’.”

The generated descriptions were then filtered and combined to create a
comprehensive set of physical signals for our set of emotion labels. This
resulted in a total of 222 distinct physical signals that could indicate the
emotion an individual is experiencing in an image. It should be noted that the
remainder of the study did not assume that any specific physical signals were
associated with any particular emotion.

###  II-B Annotations

The interface shown in Fig. 1 was created to facilitate image annotation. To
assess a large language model’s ability to predict human emotions from images,
we annotated a set of images from the EMOTIC dataset[8]. These images
contained bounding boxes of different colours surrounding the people in the
scene. This allowed us to focus on one person (e.g. marked with a “red”
bounding box) at a time within an image.

During the annotation process, both physical signals and contextual components
were considered. To make the annotation process easier, we divided the
physical signals into multiple categories based on body parts, and annotators
could use checkboxes to select relevant descriptions. The annotator could also
tag the person within a bounding box with various attributes, including their
perceived age group, perceived sex, and social identity or occupation.

Finally, contextual information including factors such as their social
interactions (e.g. alone, surrounded by people), social relationships with
others in the image (e.g. mother and daughter, husband and wife), and their
environmental setting could be input into an open text box. In the end, the
annotation interface generated an appropriate image caption based on all the
chosen tags (e.g. creating a sentence with a first name), allowing the human
annotator to double-check the caption before saving their work.

Out of 222 physical signals proposed to the annotators, 153 were ultimately
used to describe the images in the dataset in this study and are reported in
Table I. A full listing of interactions and environmental contexts derived
from annotators is provided in Supplementary Materials.

###  II-C Assessing Prediction Abilities of Large Language Models

Once the annotations were complete, GPT-3.5 was used to predict emotion labels
with the help of a prompt. The prompt was structured to elicit single emotion
prediction when presented with an image annotation.

The prompt was as follows (considering the annotation from Fig. 1): ”Sean is a
male adult. Sean is a(n) passenger. Sean is or has raising eyebrows, side-
eyeing. Mia is a child and she is sitting behind Sean and kicking Sean’s
chair. Sean’s physical environment is on an airplane. Sean is likely feeling a
high level of {placeholder}? Choose one emotion from the list: Anger,
Annoyance, Aversion, Confusion, Disapproval, Disconnection, Disquietment,
Embarrassment, Fatigue, Fear, Pain/Suffering (emotional), Pain/Suffering
(physical), and Sadness.”

To evaluate the performance of the models, we compared the LLM’s predictions
to the ground truth of the images established by the annotators. For instance,
from the manual annotation as shown in Fig. 1, the ground truth for the person
within the green bounding box was determined to be “Annoyance”. It should be
noted that the ground truth labels in our study were different from that of
the EMOTIC dataset [8], which was a multi-label dataset. For this reason, it
is not straightforward to evaluate the existing multilabel baseline algorithms
on our dataset.

##  III Experiments

We conducted three experiments with our manually annotated image captions to
test a large language model’s ability to estimate human emotions. The first
experiment used the full image captions that included all the visually
contextual information in an image. We then performed two ablation studies to
test the contribution of social interactions and environmental contexts in
predicting emotions. Table II shows how every image caption could differ
between experiments.

TABLE II: Different Versions of an Image Caption Used in Experiments Type of Caption | Image Caption
---|---
Full Captiona |  Sean is a male adult. Sean is a(n) passenger. Sean is or has raising eyebrows, side-eyeing. Mia is a child and she is sitting behind Sean and kicking Sean’s chair. Sean’s physical environment is on an airplane.
Minus Interactionsb |  Sean is a male adult. Sean is a(n) passenger. Sean is or has raising eyebrows, side-eyeing. Sean’s physical environment is on an airplane.
Minus Environmentsc |  Sean is a male adult. Sean is a(n) passenger. Sean is or has raising eyebrows, side-eyeing. Mia is a child and she is sitting behind Sean and kicking Sean’s chair.
aFull captions are used in Experiment A.
bCaptions without interactions are used in Experiment B.
cCaptions without environments are used in Experiment C.

###  III-A Dataset and Annotation

The image samples used in this study are from the EMOTIC dataset[8]. An image
could contain one or more bounding boxes and each bounding box enclosed a
person. Every person could depict multiple emotions, but only one emotion
label mutually agreed upon by two annotators was picked as the ground truth
and counted towards a sample for that emotion. If an image contained multiple
people where one person showed Anger while the other person showed Fear, then
the image counted towards a sample for both Anger and Fear. If two people in
an image were showing the same emotion such as Sadness, then the image was
counted twice as a sample for Sadness. Table III shows the sample
distribution. To summarize, our sample
dataset222https://rosielab.github.io/emotion-captions/consisted of:

  * •

331 unique images

  * •

360 samples

  * •

360 captions generated through manual annotation

  * •

Two types of images: One person and Multiple people

All emotion categories had a sample size of 30, except for Confusion (16) and
Embarrassment (14), and ground truth was observed by two annotators upon
mutual agreement.

TABLE III: Number of Samples per Emotion Across Two Types of Images Emotion | One person | Multiple people | Total
---|---|---|---
Anger | 14 | 16 | 30
Annoyance | 16 | 14 | 30
Aversion | 16 | 14 | 30
Confusion | 12 | 4 | 16
Disapproval | 18 | 12 | 30
Disconnection | 18 | 12 | 30
Disquietment | 15 | 15 | 30
Embarrassment | 0 | 14 | 14
Fatigue | 23 | 7 | 30
Fear | 15 | 15 | 30
Pain/Suffering - Emotional | 15 | 15 | 30
Pain/Suffering - Physical | 15 | 15 | 30
Sadness | 15 | 15 | 30

Figure 2: Images (a) and (b) are examples of how Embarrassment images required multiple people and interactions to be recognizable. Images (c) and (d) have limited facial or body signals of a person, hence we have to rely on environmental contexts to estimate an emotion. TABLE IV: Results of GPT-3.5 Emotion Estimation Using Captions Emotions | Full Caption | Minus Interactions | Minus Environments
---|---|---|---
Precision | Recall | F1 Score | Precision | Recall | F1 Score | Precision | Recall | F1 Score
Anger | 0.59 | 0.77 | 0.67 | 0.55 | 0.73 | 0.63 | 0.47 | 0.70 | 0.56
Annoyance | 0.64 | 0.23 | 0.34 | 0.67 | 0.20 | 0.31 | 0.43 | 0.10 | 0.16
Aversion | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
Confusion | 0.75 | 0.19 | 0.30 | 0.75 | 0.19 | 0.30 | 1.00 | 0.13 | 0.22
Disapproval | 0.25 | 0.47 | 0.32 | 0.23 | 0.47 | 0.31 | 0.24 | 0.50 | 0.32
Disconnection | 0.25 | 0.07 | 0.11 | 0 | 0 | 0 | 0.25 | 0.10 | 0.14
Disquietment | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
Embarrassment | 0.28 | 0.79 | 0.41 | 0.24 | 0.50 | 0.33 | 0.26 | 0.71 | 0.38
Fatigue | 0.68 | 0.43 | 0.53 | 0.70 | 0.47 | 0.56 | 0.73 | 0.37 | 0.49
Fear | 0.39 | 0.77 | 0.52 | 0.37 | 0.70 | 0.48 | 0.26 | 0.50 | 0.34
Pain/Suffering - Emotional | 0.25 | 0.03 | 0.06 | 0 | 0 | 0 | 0.13 | 0.03 | 0.05
Pain/Suffering - Physical | 0.86 | 0.63 | 0.73 | 0.75 | 0.40 | 0.52 | 1.00 | 0.30 | 0.46
Sadness | 0.27 | 0.87 | 0.42 | 0.21 | 0.80 | 0.33 | 0.24 | 0.80 | 0.37
Total Accuracy | 0.39 | 0.34 | 0.32
Random chance is 0.07.

###  III-B Model Parameters and Stability

We used OpenAI’s Completions API 333https://platform.openai.com/docs/api-
reference/completions to provide our image captions as prompts to the GPT-3.5
model, and it returned predicted emotions through completions. The model
version was text-davinci-003, which is part of the GPT3.5 family. To ensure
that the results generated from GPT-3.5 were stable and reproducible, we set
the model’s temperature parameter to 0, allowing the model to give a nearly
deterministic answer to every prompt. To further the reproducibility of our
results, we ran the GPT-3.5 model over each caption ten times to generate a
list of ten predicted emotions. We limited the emotions that GPT-3.5 could
output to the 13 negative emotions that we focused on in this study. For each
caption, the emotion with the maximum number of occurrences was selected as
the final prediction. This prediction generation and selection process was
done for all three experiments.

###  III-C Experiment A: Predicting with Full Image Captions

Experiment A studies how accurately a large language model can predict a
person’s emotional state in an image given all the contextual information. A
full image caption includes a person’s perceived age, perceived sex, social
identity if apparent, facial expression and body poses if applicable. It also
includes their interactions with other people and their environmental
surrounding if applicable. Table II shows a full caption for Fig. 1.

###  III-D Experiment B: Ablation Study on Interactions with People

Experiment B studies how describing a person’s interactions and relationships
with other people in an image contributes to determining a person’s emotional
state. It used the same dataset as Experiment A. We removed all the
information about a person’s social interactions and relationships in an image
from the full captions. Thus, the captions used in this experiment contained
only perceived age, perceived sex, applicable social identity, face and body
signals, and environment. Table II shows a caption without social interaction
for Fig. 1. Fig. 2 (a) and (b) show two examples where social interactions may
need to be considered to fully understand Embarrassment depicted in the
images.

###  III-E Experiment C: Ablation Study on Environmental Contexts

Experiment C studies how describing a person’s environmental context in an
image contributes to predicting a person’s emotion. Environmental contexts can
range from location and time to different types of animals and activities, and
this information provides valuable insight into what a person may be feeling.
Especially when facial and body features are missing in an image, we can rely
on scene context to predict an emotion. Fig. 2 (c) and (d) show two examples
where a person’s face is not visible in the images, and therefore, scene
context becomes important to accurately predict their emotion.

This experiment used the same dataset as Experiment A, but all information
about a person’s environment and physical surrounding was removed from the
caption. Therefore, the captions used in the experiment contained only
perceived age, perceived sex, applicable social identity, face and body
signals, and interactions and relationships with others. Table II shows a
caption without environmental context for Fig. 1.

Figure 3: Confusion matrix from the experiment using full captions.

Figure 4: Three emotions - Excitement, Happiness and Joy, that were not on the
list of emotions that we provided to GPT-3.5 to choose from, were predicted.

Figure 5: Two emotions - Happiness and Love, that were not on the list of
emotions that we provided to GPT-3.5 to choose from, were predicted.

Figure 6: Image (a) shows that when medication on the side was removed from
the full caption, the predicted emotion changed from Physical Pain/Suffering
to Disapproval. Image (b) shows that when right in front of an alien hand in
the dark was removed from the full caption, the predicted emotion changed from
Fear to Disapproval.

Figure 7: Examples of images where new positive emotions such as Excitement,
Happiness, and Love were predicted by GPT-3.5 when interactions and
environments were removed from the captions. The first emotion above each
image was generated using full captions.

##  IV Results and Analysis

The results of our GPT-3.5 emotion prediction are shown in Table IV. The table
contains precision, recall, and F1 score for each emotion and the total
accuracy for each experiment. Experiment A with full captions has the highest
accuracy. Experiment C with environments removed has the lowest accuracy. The
confusion matrix results are in Fig. 3, Fig. 4, and Fig. 5. From the confusion
matrix for full captions in Fig. 3, we can notice that Anger and Sadness were
the most picked, but overall, according to F1 scores, Physical Pain/Suffering
was best estimated. GPT-3.5 was not able to predict Aversion. Disconnection
and Disquietment were also not well recognized. Emotional Pain/Suffering was
frequently recognized as Sadness, which may be reasonable. Annoyance and
Confusion were often recognized as Disapproval. Fear appeared to need
environmental cues to be well predicted. Disapproval and Fatigue seem not to
be impacted by social and environmental contexts. Embarrassment was fairly
well predicted with social interactions.

###  IV-A Importance of Interactions in Emotion Estimation

The lack of context about a person’s social interactions with other people
seems to impact how Embarrassment was perceived by GPT-3.5 the most. Its F1
score for Experiment B without interactions is $0.33$, which is lower than the
two experiments with interactions, $0.41$ and $0.38$. A potential reason is
that for all the sample images that were annotated as Embarrassment as shown
in Fig. 2 (a) and (b), multiple people were present, and there were
interactions between the people. In this case, removing social interaction
contexts from the captions removes a critical piece of information that may
indicate a person may be feeling embarrassed.

Physical Pain/Suffering is another emotion that may need social interactions
to be well recognized. Its F1 score dropped from $0.73$ with full captions to
$0.52$ without interaction and was predicted as Fear by GPT-3.5 more times
when compared to full captions.

###  IV-B Importance of Environments in Emotion Estimation

The lack of description about environment in the image captions seemed to
impact Physical Pain/Suffering the most. The F1 score dropped from $0.73$ with
full captions to $0.46$ and it is even lower than the F1 score for captions
without interactions. As a result, we observed that Physical Pain/Suffering
needed both the interaction and environment descriptions to be well recognized
by GPT-3.5, especially environments out of the two. An example is shown in
Fig. 6 (a) where the predicted emotion changed from Physical Pain/Suffering to
Disapproval after removing the environmental description. The full caption for
this image with the removed part in italics is: Jack is a male adult. Jack is
or has frowning, rubbing the back. Jack’s physical environment is on a bed
with medication on the side.

Another emotion that may benefit from environmental context is Fear. Its F1
score dropped from $0.52$ and $0.48$ with environments to $0.34$ without
environments. An example is shown in Fig. 6 (b) where the predicted emotion
changed from Fear to Disapproval. The full caption for this image with the
removed part in italics is: Chloe is a female adult. Chloe is or has frowning,
open mouth. Chloe’s physical environment is right in front of an alien hand in
the dark.

###  IV-C Importance of Facial and Body Signals

Across the three experiments, we noticed Anger was well recognized as Anger by
GPT-3.5 with F1 scores of $0.67$, $0.63$, and $0.56$. A reason for this may be
that our samples had distinct facial expressions and body postures that
differentiated it from the other emotions, like furrowed eyebrows, gritting
teeth, and wrinkling nose.

GPT-3.5’s understanding of Disapproval also seems to not be affected by the
existence of interaction and environment descriptions in the captions. The F1
scores are $0.32$, $0.31$, and $0.32$. This can suggest that Disapproval was
recognized through a person’s facial expressions and body poses more than a
person’s physical surrounding. Body gestures such as crossed arms, pointing
finger, and thumbs down may all be indicators of Disapproval to GPT-3.5.

Sadness was almost always predicted as Sadness, but Emotional Pain/Suffering
was frequently predicted as Sadness in all three experiments as well. In fact,
the F1 scores for Emotional Pain/Suffering are only $0.06$, $0$, and $0.05$.
This may indicate that these two emotions share similar physical signals, such
as crying, downturned mouth, and tilting head downward, and thus GPT-3.5 was
not capable of differentiating the two emotions. Or, it could be that GPT-3.5
tends to select the more commonly known emotion from the list that it was
provided.

###  IV-D New Emotions Predicted by GPT-3.5

Interestingly, four positive emotions were predicted but they were not on the
list of 13 negative emotions that we provided to GPT-3.5 to choose from. The
emotions and their number of occurrences are Excitement (1), Happiness (2),
Joy (1), and Love (1). All four cases happened in the two ablation studies
when interaction or environmental features were missing from the captions.
Fig. 7 shows the image for these cases and the corresponding captions. We also
show the emotions that were originally predicted using the full captions and
the new emotions they were changed to. The removed interactions and
environments are in italics:

  * •

Fig. 7 (a) Fear to Excitement. Terry is a male adult. Karl is a security guard
and he is grabbing onto Terry and carrying him out from the stadium. Terry’s
physical environment is at a sports game.

  * •

Fig. 7 (b) Embarrassment to Happiness. Jack is a male adult. Jack is or has
smiling. Beth is a customer and she is side-eyeing Jack. Zoe is a customer and
she is staring at Jack. Jack’s physical environment is eating in a movie
theatre.

  * •

Fig. 7 (c) Embarrassment to Happiness. Lucas is a male adult. Lucas is a(n)
groom. Lucas is or has lips that flatten, palms open. Mia is Lucas’ bride and
she is smiling. Lucas’ physical environment is cake falling down at wedding.

  * •

Fig. 7 (d) Sadness to Love. Jane is a female adult. Jane is or has taking off
eyeglasses. Mia is Jane’s daughter and Jane is putting her hand on Mia’s
shoulder while Mia has her back turned to Jane. Jane’s physical environment is
on a couch.

There was no new negative emotion.

##  V Discussion and Future Work

We proposed a new approach for emotion estimation which couples text-based
contextual descriptions of people in images with LLMs. Towards this goal, this
study provided a benchmark of GPT-3.5 on a set of image captions depicting
negative emotions. We also investigated the contributions of social cues and
broader contextual information when perceiving human emotions. For example, we
observed that Embarrassment and Sadness contained overlapping physical
signals, such as “covering own face” or “tilting head downwards”, and that
social interactions could help distinguish between these two labels, i.e., a
person covering their face with a tilted down head, along with being pointed
at and laughed at by others, could appear to suggest Embarrassment. Moreover,
our results showed that Aversion was never predicted as an emotion across all
three experiments on 360 captions, while Disquietment was only predicted three
times. Emotional Pain/Suffering and Disconnection were also frequently
predicted as Sadness, even in the presence of scene contexts. A possible
explanation for this may be that GPT-3.5 might not have been sufficiently
trained on language data that contained such emotion words.

The study is not without limitations. Firstly, we only focused on the negative
emotions of the EMOTIC dataset, and the social signals list employed for
annotations was restricted to those associated with our set of negative
emotions. Secondly, the list of social signals was partially generated by LLMs
and subsequently tested on them. The resulting contextual descriptions were
ultimately determined and validated by our team of annotators. Aside from
that, the size of the resulting descriptions list, as well as the number of
annotated images, was relatively small. Finally, our study did not delve into
the individual contribution of each physical description or demographic
information for emotion detection, making it an interesting area to explore
for future work. In the future, an independent perception study of the
captions and ablations could also help provide a comparison to the GPT-3.5
results, as well as addressing the challenge of fully automatic captioning,
and evaluating over all EMOTIC labels.

Overall, our approach may be used to enhance transparency and facilitate an
effective breakdown of scene representation for contextual emotion estimation.
It is hoped that our study can also serve as a catalyst for future research in
interpretability of LLMs, as well as understanding human perception of
emotions, especially if reproduced with other languages and cultures.

##  VI Ethical Impact Statement

_Issues Related to Human Subjects._ In this study, all social signal and
context coding and emotion annotation of the images was performed by two
members of the research team. The photos are from the EMOTIC dataset which
contain images from the internet, of which some belong to the public datasets
MSCOCO and Ade20k. Access to the EMOTIC dataset requires a request to the
database authors. The research team are not related to the pictures. The
images are of people who may be experiencing negative emotions including grief
at a funeral, protesting, or war. It does not contain images stronger than
images that a person might encounter in news media (e.g. no nudity, torture,
etc.)

_Potential Negative Societal Impact._ Software that can detect negative
emotions from images accurately could potentially be used for surveillance by
authorities for intervention and restriction of autonomy. The application of
this research for such use is not condoned by the authors.

_Limits of Generalizability_. The proposed list of physical signals is not
claimed to be exhaustive, not only because we focus on a limited set of
negative emotions, but also that different cultures will express emotions with
different facial and bodily signals. The source Emotion Thesaurus is written
by North American authors, and similar writing guides in other languages may
produce differing results. ChatGPT, used to supplement the Emotion Thesaurus,
also contains its own biases [21]. In addition, GPT-3.5, trained in English,
carries biases in the association of facial, bodily and contextual signals
with the final emotion. In addition, there were also only two annotators, and
we acknowledge that they may also carry their own cultural bias.

_Other Issues_. This work relied on a pre-trained large language model
GPT-3.5. While this work did not perform any additional training, the carbon
cost of training LLMs cannot be underestimated [22].

## References

  * [1] L. F. Barrett, R. Adolphs, S. Marsella, A. M. Martinez, and S. D. Pollak, “Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements,” _Psychological Science in the Public Interest_ , vol. 20, no. 1, pp. 1–68, 2019.
  * [2] M. Pantic and L. J. Rothkrantz, “Expert system for automatic analysis of facial expressions,” _Image and Vision Computing_ , vol. 18, no. 11, pp. 881–905, 2000.
  * [3] K. Schindler, L. Van Gool, and B. De Gelder, “Recognizing emotions expressed by body pose: A biologically inspired neural model,” _Neural Networks_ , vol. 21, no. 9, pp. 1238–1246, 2008.
  * [4] L. F. Barrett, B. Mesquita, and M. Gendron, “Context in emotion perception,” _Current Directions in Psychological Science_ , vol. 20, no. 5, pp. 286–290, 2011.
  * [5] L. F. Barrett, _How emotions are made: The secret life of the brain_. Pan Macmillan, 2017.
  * [6] M. Calbi, K. Heimann, D. Barratt, F. Siri, M. A. Umiltà, and V. Gallese, “How context influences our perception of emotional faces: A behavioral study on the kuleshov effect,” _Frontiers in Psychology_ , vol. 8, p. 1684, 2017.
  * [7] U. Hess and S. Hareli, “The influence of context on emotion recognition in humans,” in _Proceedings of the 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG)_ , vol. 03, 2015, pp. 1–6.
  * [8] R. Kosti, J. M. Alvarez, A. Recasens, and A. Lapedriza, “Context based emotion recognition using EMOTIC dataset,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 42, no. 11, pp. 2755–2766, 2019.
  * [9] N. Le, K. Nguyen, A. Nguyen, and B. Le, “Global-local attention for emotion recognition,” _Neural Computing and Applications_ , vol. 34, no. 24, pp. 21 625–21 639, 2022.
  * [10] T. Mittal, P. Guhan, U. Bhattacharya, R. Chandra, A. Bera, and D. Manocha, “EmotiCon: Context-aware multimodal emotion recognition using frege’s principle,” in _Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 14 234–14 243.
  * [11] Z. Wang, L. Lao, X. Zhang, Y. Li, T. Zhang, and Z. Cui, “Context-dependent emotion recognition,” _Journal of Visual Communication and Image Representation_ , vol. 89, p. 103679, 2022.
  * [12] B. Dudzik, J. Broekens, M. Neerincx, and H. Hung, “Exploring personal memories and video content as context for facial behavior in predictions of video-induced emotions,” in _Proceedings of the 2020 International Conference on Multimodal Interaction_ , 2020, pp. 153–162.
  * [13] S. Shin, D. Kim, and C. Wallraven, “Contextual modulation of affect: Comparing humans and deep neural networks,” in _Companion Publication of the 2022 International Conference on Multimodal Interaction_ , 2022, pp. 127–133.
  * [14] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in Neural Information Processing Systems_ , vol. 30, 2017.
  * [15] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” _OpenAI blog_ , vol. 1, no. 8, p. 9, 2019.
  * [16] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [17] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 1877–1901, 2020.
  * [18] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh, “VQA: Visual question answering,” in _Proceedings of the 2015 IEEE International Conference on Computer Vision_ , 2015, pp. 2425–2433.
  * [19] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” in _Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 3156–3164.
  * [20] B. Puglisi and A. Ackerman, _The emotion thesaurus: A writer’s guide to character expression_. JADD Publishing, 2019, vol. 1.
  * [21] M. Nadeem, A. Bethke, and S. Reddy, “StereoSet: Measuring stereotypical bias in pretrained language models,” _arXiv preprint arXiv:2004.09456_ , 2020\.
  * [22] D. Patterson, J. Gonzalez, Q. Le, C. Liang, L.-M. Munguia, D. Rothchild, D. So, M. Texier, and J. Dean, “Carbon emissions and large neural network training,” _arXiv preprint arXiv:2104.10350_ , 2021.