# Solving Differential-Algebraic Equations in Power Systems Dynamics with
Quantum Computing

Huynh T. T. Tran1, Hieu T. Nguyen1, Long Thanh Vu2, and Samuel T. Ojetola3
1Department of Electrical & Computer Engineering, North Carolina A&T State
University
2Energy & Environment Directorate, Pacific Northwest National Laboratories
(PNNL)
3Electric Power System Research, Sandia National Laboratories
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Power system dynamics are generally modeled by high dimensional nonlinear
differential-algebraic equations due to a large number of generators, loads,
and transmission lines. Thus, its computational complexity grows exponentially
with the system size. In this paper, we aim to evaluate the alternative
computing approach, particularly the use of quantum computing algorithms to
solve the power system dynamics. Leveraging a symbolic programming framework,
we convert the power system dynamics’ DAEs into an equivalent set of ordinary
differential equations (ODEs). Their data can be encoded into quantum
computers via amplitude encoding. The system’s nonlinearity is captured by
Taylor polynomial expansion and the quantum state tensor whereas state
variables can be updated by a quantum linear equation solver. Our results show
that quantum computing can solve the dynamics of the power system with high
accuracy whereas its complexity is polynomial in the logarithm of the system
dimension.

###### Keywords:

Power system dynamics, DAEs, ODEs, quantum computing.

##  I Introduction

Power system dynamics are generally modeled by a large number of differential-
algebraic equations (DAEs) due to a large number of generators, loads, and
transmission lines that form the network [1]. Specifically, the power system
dynamics include a set of ordinary differential equations (ODEs) modeling the
dynamics of synchronous generators along with algebraic nonlinear equations of
power flow balances and Kirchhoff voltage laws for individual buses in the
network. Solving such DAEs is a challenging task as they first need to be
transformed into standard ODE forms, which is not a trivial task [2]. Then,
the obtained ODEs can be solved by numerical discretization such as Euler, the
Runge-Kutta, and the backward differentiation formula methods in classical
computers [3]. However, the implementation on a classical computer scales
exponentially with the size of the problem and the need of linearizing the
nonlinear terms embedded in the complex power system dynamics, which often
leads to high computational complexity [4, 3].

Quantum computer has a different way to perform computation and possesses
algorithmically superior scaling for certain problems, e.g., the Harrow-
Hassidim-Lloyd (HHL) algorithm linear equations [5]. The HHL algorithm is
based on amplitude encoding, where a system with n variables can be
represented as an n-level of quantum state
$|\phi\rangle=\sum^{n}_{j}\phi_{j}|j\rangle$, in which $\phi_{j}$ is the
amplitudes of computational basis states $|j\rangle$. Since the number of
required qubits equals $\log_{2}(n)$, the algorithm provides an exponential
memory advantage over the classical computing method [3]. Extending quantum
linear equation solvers to tackle high-dimensional systems of linear ODEs are
studied in [6, 7, 8]. Basically, they construct the quantum states
proportional to the solution of the block-encoded $n$-dimensional system of
linear equations, thus offering the prospect of rapidly characterizing the
solutions of high-dimensional linear ODEs.

This paper aims to evaluate the potential of quantum computing for solving
large-scale power system dynamics using quantum computing algorithms and
recent advances in scientific computing frameworks. Unlike linear ODEs studied
in [6, 7, 8], the power systems dynamics pose high-dimensional nonlinear DAEs.
To this end, we leverage symbolic programming packages from Julia/SciML [9],
particularly ModelingToolkit [10], to implement and transform power systems’
DAEs into an equivalent set of ODEs using the Pantelides based index
reduction. Then, we employ Leyton’s quantum algorithm for nonlinear cases of
ODEs [11]. Specifically, the obtained nonlinear ODEs are linearized by Taylor
expansion as a set of polynomial functions of state variables. By storing
multiple copies of quantum states, the nonlinearities embedded in polynomial
functions are then captured by amplitudes of the tensor of quantum states.
Then, state variables, which are updated in the classical computer according
to the Euler method, are now equivalently updated using the HHL algorithm. The
complexity, however, is polynomial to the logarithm of the system dimension.

The rest of the paper is organized as follows. The mathematical model of power
systems dynamics are presented in Section II. The classical method for solving
power system dynamics is presented in Section III. Section IV summarizes the
quantum method to solve the power system dynamics. Section V illustrates the
procedure to implement the power system dynamics into Julia, subsequently,
transforming it to ODEs by ModelToolkit, and solving it by interfacing it with
quantum simulator Yao/QuDiffEq. Section VI presents numerical studies on the
SMIB and the three-machine nine-bus test systems. Section VII concludes the
paper.

##  II Mathematical model

The dynamic circuit of a generic electric power network consisting of $m$
synchronous generators and $n$ buses, together with the power transmission
network and the loads can be represented in Figure 1. Buses indexed
$1,\ldots,m$ are generation buses and buses indexed $m+1,\ldots,n$ are load
buses. Thus, the power grid can be considered a dynamics network represented
by a set of Differential Algebraic Equations (DAEs) that consists of (i) a set
of ordinary equations (ODEs) modeling the system dynamics of synchronous
generators and (ii) a set of algebraic equations modeling generators’ stator
voltage equations and power flow balances in the network [12].

###  II-A Dynamics of Generators

Figure 1: Interconnection of $m$ synchronous machine dynamic circuits and $n$
buses with network admittance matrix $\bar{Y}_{N}$
($Y_{ik}\angle{\alpha_{ik}}$ represents the $\text{ik}^{\text{th}}$ element of
$\bar{Y}_{N}$), $P_{Li}$ and $Q_{Li}$ as active and reactive power demands at
buses $i$.

In Figure 1, the dynamic circuit of the generator in the two-axis $d$-$q$
coordinates [1] is considered as a constant voltage source behind impedance
($R_{si}+jX^{\prime}_{di}$) with an IEEE-Type I exciter. Its dynamics are
represented by the following ODEs:

$\displaystyle T^{\prime}_{qoi}\frac{\mathrm{d}E^{\prime}_{di}}{\mathrm{d}t}$
$\displaystyle=-E^{\prime}_{di}+(X_{qi}-X^{\prime}_{qi})I_{qi}$ (1a)
$\displaystyle T^{\prime}_{doi}\frac{\mathrm{d}E^{\prime}_{qi}}{\mathrm{d}t}$
$\displaystyle=-E^{\prime}_{qi}-(X_{di}-X^{\prime}_{di})I_{di}+E_{fdi}$ (1b)
$\displaystyle\frac{\mathrm{d}\delta_{i}}{\mathrm{d}t}$
$\displaystyle=\omega_{i}-\omega_{s}$ (1c)
$\displaystyle\frac{2H_{i}}{\omega_{s}}\frac{\mathrm{d}\omega_{i}}{\mathrm{d}t}$
$\displaystyle=T_{Mi}-E^{\prime}_{di}I_{di}-E^{\prime}_{qi}I_{qi}$
$\displaystyle-(X^{\prime}_{qi}-X^{\prime}_{di})I_{di}I_{qi}-D_{i}(\omega_{i}-\omega_{s})$
(1d) $\displaystyle T_{Ei}\frac{\mathrm{d}E_{fdi}}{\mathrm{d}t}$
$\displaystyle=-(K_{Ei}+S_{Ei}(E_{fdi}))E_{fdi}+V_{Ri}$ (1e) $\displaystyle
T_{Fi}\frac{\mathrm{d}R_{fi}}{\mathrm{d}t}$
$\displaystyle=-R_{fi}+\frac{K_{Fi}}{T_{Fi}}E_{fdi}$ (1f) $\displaystyle
T_{Ai}\frac{\mathrm{d}V_{Ri}}{\mathrm{d}t}$
$\displaystyle=-V_{Ri}+K_{Ai}R_{fi}$
$\displaystyle-\frac{K_{Ai}K_{Fi}}{T_{Fi}}E_{fdi}+K_{Ai}(V_{refi}-V_{i})$ (1g)
$\displaystyle\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\forall
i=1,\ldots,m,$

where $i$ is the generator index. The state variables of a generator $i$
include $d$-axis and $q$-axis internal voltages, $E^{\prime}_{di}$ and
$E^{\prime}_{qi}$, the rotor angle, $\delta_{i}$, and the angular velocity,
$\omega_{i}$. Their dynamics are captured in equations (1a)-(1d) respectively.
The state variables of the exciter of the generator $i$ include the scaled
field voltage $E_{fdi}$, the stabilizer rate feedback, $R_{fi}$, and the
scaled input to the main, $V_{Ri}$, [13]. Their dynamics are captured in
equations (1e)-(1g) respectively. The generator parameters used in (1) include
$d$-axis and $q$-axis transient open-circuit time constants, $T^{\prime}_{do}$
and $T^{\prime}_{qo}$, the synchronous and transient reactances in $d$\- and
$q$-axis, $X_{d}$, $X^{\prime}_{d}$, $X_{q}$ and $X^{\prime}_{q}$, the
synchronous angular speed, $\omega_{s}$, the inertia constant, $H$, the
mechanical torque, $T_{M}$, and the damping coefficient, $D$. The parameters
of the exciter and voltage stabilizer of the generator $i$ used in (1) include
the exciter time constant, $T_{Ei}$, the exciter gain, $K_{Ei}$, exciter
saturation, $S_{Ei}$111Note the exciter saturation $S_{Ei}$ is a function of
the scaled field voltage $E_{fdi}$, i.e., typically
$S_{Ei}(E_{fdi})=A_{x}e^{B_{x}E_{fdi}}$ , the rate feedback time constant,
$T_{Fi}$, the rate feedback gain, $K_{Fi}$, the amplifier time constant,
$T_{Ai}$, and the amplifier gain, $K_{Ai}$.

###  II-B Kirshoff voltage and network power flow equations

The stator currents $I_{di}$ $I_{qi}$ in $d$-$q$ coordinates of $m$ generators
in buses $1,\ldots,m$, together with the magnitudes $V_{i}$ and phase angle
$\theta_{i}$ of $n$ nodal voltages ($V_{i}\angle{\theta_{i}}$) in buses
$1,\ldots,n$ are algebraic variables. For buses that have generators
($i=1,\ldots,m$), $I_{di}$, $I_{qi}$, and $V_{i}\angle{\theta_{i}}$ are
coupled with the state variables of internal generators’ voltages,
$E^{\prime}_{di}$ $E^{\prime}_{qi}$, by Kirchhoff voltage law as:

$\displaystyle
E^{\prime}_{di}-V_{i}\sin(\delta_{i}-\theta_{i})-R_{si}I_{di}+X^{\prime}_{qi}I_{qi}=0,$
(2a) $\displaystyle
E^{\prime}_{qi}-V_{i}\cos(\delta_{i}-\theta_{i})-R_{si}I_{qi}-X^{\prime}_{di}I_{di}=0,$
(2b) $\displaystyle\forall i=1,\ldots,m,$

where $R_{si}$ denotes the stator resistance and recall that $X^{\prime}_{di}$
denotes the transient reactance of generator $i$. Also, the active/reactive
power balances for these generation buses are:

$\displaystyle
I_{di}V_{i}\sin(\delta_{i}-\theta_{i})+I_{qi}V_{i}\cos(\delta_{i}-\theta_{i})-P_{Li}$
$\displaystyle=\sum^{n}_{k=1}V_{i}V_{k}Y_{ik}\cos(\theta_{i}-\theta_{k}-\alpha_{ik})$
(3a) $\displaystyle
I_{di}V_{i}\cos(\delta_{i}-\theta_{i})-I_{qi}V_{i}\sin(\delta_{i}-\theta_{i})-Q_{Li}$
$\displaystyle=\sum^{n}_{k=1}V_{i}V_{k}Y_{ik}\sin(\theta_{i}-\theta_{k}-\alpha_{ik})$
(3b) $\displaystyle\quad\quad\quad\quad i=1,\ldots,m,$ where
$I_{di}V_{i}\sin(\delta_{i}-\theta_{i})+I_{qi}V_{i}\cos(\delta_{i}-\theta_{i})$
and
$I_{di}V_{i}\cos(\delta_{i}-\theta_{i})-I_{qi}V_{i}\sin(\delta_{i}-\theta_{i})$
respectively are the active and reactive power generated by the generator at
bus $i$. Similarly, the active and reactive power balance equations for $n-m$
remaining load buses are: $\displaystyle-
P_{Li}=\sum^{n}_{k=1}V_{i}V_{k}Y_{ik}\cos(\theta_{i}-\theta_{k}-\alpha_{ik})$
(3c) $\displaystyle-
Q_{Li}=\sum^{n}_{k=1}V_{i}V_{k}Y_{ik}\sin(\theta_{i}-\theta_{k}-\alpha_{ik})$
(3d) $\displaystyle i=m+1,\ldots,n.$

###  II-C DAE compact form

In short, the set of DAEs of power system dynamics can be written in the
compact form [12, 13, 1].

$\displaystyle\dfrac{\mathrm{d}x}{\mathrm{d}t}=f(t,x,y)$ (4a) $\displaystyle
0=g(t,x,y),$ (4b) $\displaystyle
x=[E^{\prime}_{di},E^{\prime}_{qi},\delta_{i},\omega_{i},E_{fi},R_{fi},V_{Ri}~{}~{}~{}(i=1,\ldots,n)]^{\top},$
$\displaystyle y=[I_{di},I_{qi},V_{i},\theta_{i}~{}~{}(i=1,\ldots,n)]^{\top},$

where $f$ and $g$ are nonlinear vector-valued functions of state variables
$x$, algebraic variables $y$, and time $t$. The dynamics of $m$ generators are
composed in (4a) whereas the Kirchhoff voltage laws and power flow balance are
encapsulated in (4b).

##  III  Classical Methods for Power System Dynamics

Given a large number of algebra equations and variables in (4b), the power
system dynamics in the form (4b) can be a DAE of index greater than two, which
is very difficult to solve [14]. It needs to be transformed into an equivalent
set of ODEs [2]:

$\displaystyle\dfrac{\mathrm{d}z}{\mathrm{d}t}=\hat{f}(t,z)$ (5)

where $z$ is a new set of state variables so it can be solved by existing ODE
numerical methods [2] Such transformation, unfortunately, is a challenging
task due to the complexity of nonlinear power flow equations embedded in (4b).
If the Jacobian $\dfrac{\partial g}{\partial y}$ is nonsingular, i.e., the
power flow and the Kirshoff voltage equations have a (unique) solution, which
generally holds for normal operations of power systems. Based on the implicit
function theorem, there exists a function $\hat{g}$ to equivalently convert
(4b) into $y=\hat{g}(t,x)$. Thus, the formulation (4b) can be written in the
standard form (5) by substituting $y$ by $\hat{g}(t,x)$, i.e.,
$\dfrac{\mathrm{d}x}{\mathrm{d}t}=f(t,x,\hat{g}(t,x))=\hat{f}(t,x)$.
Unfortunately, we cannot explicitly model the nodal voltages (magnitudes and
phases) as explicit functions of generators’ state variables, particularly,
injected currents and internal voltages. Also, in many cases, the Jacobian
$\dfrac{\partial g}{\partial y}$ can be singular depending on the operational
points of the grid. To overcome these issues, this paper employs the
Pantelides algorithm, a systematic method for reducing high-indexed DAEs to
lower-indexed ones by selectively adding differentiated forms of the equations
already in the
system222https://ptolemy.berkeley.edu/projects/embedded/eecsx44/lectures/Spring2013/modelica-
dae-part-2.pdf. This enables us to convert the power system dynamics (4b) into
(5) where $z$ composes of original state variables $x$ and parts of variables
$y$.

The formulation (5) is now in the standard form of ODEs and can be solved by
numerical discretization methods [2] in classical computers such as the
Forward Euler method:

$\displaystyle\dfrac{z(t+\Delta)-z(t)}{\Delta}\approx\hat{f}(z(t)\iff
z(t+\Delta)=z(t)+\Delta\hat{f}(z(t)),$ (6)

which is indeed a linear equation.

##  IV Quantum Computing Method for Power System Dynamics

The linear equation (6) implies the potential application of advances in
quantum computing for solving linear equations [5], for tackling the power
system dynamics of a large-scale network. This includes the following steps
[11].

###  IV-A Data encoding:

We encode the state vector $z$ in (6) of length $N$ into amplitudes of an
n-qubit quantum state with $n=\log_{2}{N}+1$. Since the classical information
$z$ now forms the amplitudes of a quantum state, the input $z$ needs to be
normalized such that $\sum_{j=1}^{N}z_{j}=1$. We can encode the original state
vector $z$ as the state of $n$-qubits as follows:

$\displaystyle|{z}\rangle=\frac{1}{\sqrt{2}}|0\rangle+\frac{1}{\sqrt{2}}\sum_{j=1}^{n}\bar{z}_{j}|j\rangle,$
(7)

where $|j\rangle$ is the computational basis for the Hilbert space. Thus, in
principle, the forward Euler equation (6) can be equivalently implemented in a
quantum computer as follows:

$\displaystyle|z(t+\Delta)\rangle=|z(t)\rangle+\Delta|\hat{f}(\bar{z}(t)\rangle.$
(8)

The remaining issue in (8) is how to construct the quantum state
$|\hat{f}(z(t))\rangle$ with $\log_{2}(N)+1$ qubits from $|z(t)\rangle$. This
can be achieved in quantum computers by combining Taylor expansion and
mathematical tensor, which are described next.

###  IV-B Reformulate the state function using Taylor Expansion

The second-order Taylor polynomial approximation of the $j$-th element in
$\hat{f}(z)$ is:

$\displaystyle\hat{f}_{j}(z(t))\approx\hat{f}_{j}(z(t-\Delta))+\frac{\partial\hat{f}(z(t-\Delta))}{\partial
z}(z(t)-z(t-\Delta))$
$\displaystyle+\frac{1}{2}(z(t)-z(t-\Delta))^{\top}Hf_{j}(z(t-\Delta))(z(t)-z(t-\Delta))$
$\displaystyle+O(\Delta^{2})$ (9)

which can be rewritten in the form:

$\displaystyle\hat{f}_{j}(z(t))=\underbrace{\hat{f}_{j}(z({t}{-}{\Delta}))}_{a^{j}_{0,0}}+\sum\limits_{k=1}^{N}a^{j}_{0,k}z_{k}(t)+\sum\limits_{v,k=1}^{N}a^{j}_{v,k}z_{v}(t)z_{k}(t)$

where $a^{j}_{v,k},v=0,\ldots,N,k=0,\ldots N$ are coefficients resulted from
Taylor expansion. Also, the first term $\hat{f}_{j}(z(t-\Delta))$ is known
since it is computed at time step $t-\Delta$ and can be considered as
$a^{j}_{0,0}$. Thus, by adding extra variable $z_{0}=1$, we have the following
compact form of $\hat{f}_{j}$:

$\displaystyle\hat{f}_{j}(z(t))=\sum\limits_{v,k=0}^{N}a^{j}_{v,k}z_{v}(t)z_{k}(t)$
(10)

Computing $\hat{f}_{j}(z(t))$ requires encoding all monominals
$z_{v}(t)z_{k}(t),\forall v,k$.

###  IV-C Nonlinear transformation of state function

A Quantum computer captures all values $z_{v}(t)z_{k}(t),\forall v,k$ in the
probability amplitudes of the tensor product:

$\displaystyle|z\rangle|z\rangle=\frac{1}{2}\sum^{N}_{v,k=0}z_{v}z_{k}|v\rangle|k\rangle,$
(11)

where we need to make a copy of the quantum state $|z\rangle$. We now need an
operator $A$ to assign corresponding coefficients, $a^{j}_{v,k}$, to
individual monomial $z_{v}z_{k}$:

$\displaystyle A=\sum\limits_{j,v,k=0}^{N}a^{j}_{v,k}|j0\rangle\langle vk|$
(12)

Acting on $|z\rangle|z\rangle$ gives us the information of $\hat{f}(z)$:

$\displaystyle
A|z\rangle|z\rangle=\frac{1}{2}\sum\limits_{j,v,k=0}^{N}a^{j}_{v,k}z_{v}z_{k}|j\rangle|0\rangle=\frac{1}{\sqrt{2}}|\hat{f}(z)\rangle|0\rangle$
(13)

The remaining question is how to simulate $A|z\rangle|z\rangle$ in a quantum
computer, which is described next.

###  IV-D Simulation in a quantum computer

In order to simulate $A|z\rangle|z\rangle$, we can simply set up the following
Hamiltonian using the well-known trick of von-Neumann measurement prescription
[15]:

$\displaystyle H=-iA\otimes|1\rangle_{P}\langle
0|+iA^{\dagger}\otimes|0\rangle_{P}\langle 1|$ (14)

where $A^{\dagger}$ is the adjoin of $A$ and P is the adjoined qubit
”pointer”. If we simulate $H$ in the quantum computer with the initial state
$|z\rangle|z\rangle|0\rangle_{P}$, the quantum system will evolve according to
$H$ for a time $\epsilon$ and reach the steady state:

$\displaystyle|\Psi\rangle$ $\displaystyle=e^{i\epsilon
H}|z\rangle|z\rangle|0\rangle=\sum\limits_{j=0}^{\infty}\frac{(i\epsilon
H)^{j}}{j!}|z\rangle|z\rangle|0\rangle$
$\displaystyle=|z\rangle|z\rangle|0\rangle+\epsilon
A|z\rangle|z\rangle|1\rangle-\ldots.$ (15)

where the second term contains $A|z\rangle|z\rangle$. In other words, we
simply need to measure the state quantum state $\Psi$ of the $n$-qubits and
post-select on $|1\rangle$ to produce $|\hat{f}(z)\rangle$. As
$|\hat{f}(z)\rangle$ can be simulated, the Forward Euler linear equation (6)
can be solved by the HHL algorithm [5].

##  V Implementation with Scientific Machine Learning and Quantum Simulation
Tools

We leverage recent advances in scientific machine learning to implement the
aforementioned scientific computing algorithms as shown in Figure 2. This
enables scalable/re-usable implementation of mathematical concepts such as
Taylor expansion, ODEs/DAEs, and quantum algorithms, especially for different
test cases of the power grid. The DAEs of power system dynamics to Julia, then
use the ModelToolkit package in Julia, a symbolic equation-based model system,
to convert DAEs to ODEs by index reduction method [10]. It transforms the DAEs
to MTK symbolic representation, then uses dae_index_lowering to generate the
index-1 form, particularly formulation (4b), which can be solved by ODE
numerical methods in a classical computer. The ODEs obtained by MTK can also
be solved by the solvers in the QuDiffEq package, which employs Yao as a
quantum simulator. We employ the QuNLDE solver to linearize the ODE systems,
and perform amplitude encoding, and Hamiltonian simulation.

Figure 2: Implementation’s workflow.

##  VI Numerical Results

We solve the power system dynamics by quantum computing through two case
studies including (i) a single-machine infinite bus system and (ii) Western
System Coordinating Council (WSCC) three-machine nine-bus system. The time
step for linearization is set as 0.01s. The difference between the quantum and
classical methods is measured by the root means square error.

###  VI-A Single machine infinite bus system

Figure 3: Single machine infinite bus system [13]

Figure 3 depicts a single-machine infinite bus (SIMB) system whose dynamics
are in the simple ODE form [13]:

$\displaystyle\frac{d\delta}{dt}=\omega-\omega_{s}$ (16a)
$\displaystyle\frac{2H}{\omega_{s}}\frac{d\omega}{dt}=P_{M}-\frac{E^{\prime}_{q}V}{X_{d\Sigma}}\sin(\delta)-\frac{D(\omega-\omega_{s})}{\omega_{s}}$
(16b)

The parameters used in (16b) include the initial constant, $H$ = 15 kWs/kVA,
synchronous angular speed, $\omega_{s}$ = 376.9911 rad/s, the mechanical power
input of the synchronous machine, $P_{M}$ = 1 p.u., the voltage magnitude on
the machine by $q$-axis, $E^{\prime}_{q}$ = 1.0566 p.u., the voltage of
infinite bus, $V$ = 1 p.u., the total reactance by $d$-axis of the machine,
$X_{d\Sigma}$ = 0.8805 p.u., and the damping constant, $D$ = 1.2. The SIMB
system is equilibrium at $\delta_{0}$ = 0.9851 rad and
$\omega_{0}=\omega_{s}$. We consider two cases when the initial generator’s
angle departs from its equilibrium a) small disturbance, which
$\Delta\delta=0.02$, the new $\delta_{0}=1.0051$ and b) large disturbance,
which $\Delta\delta=0.4$, the new $\delta_{0}=1.3851$, while keeping
$\omega_{0}=\omega_{s}$. The system oscillations derived from the initial
values are shown in Figure 5. The solid line in Figure 5 represents the values
of quantum computing, and the dashed line is the values of the classical
method. It can be seen that for both situations quantum computing has the same
value as the classical method with an error almost zero.

(a) Small disturbance

(b) Large disturbance

Figure 5: Results of SIMB system

###  VI-B WSCC three-machine nine-bus system

The WSCC system is shown in Figure 6 whose parameters and its equilibrium can
be found in [1]. Its DAE system has 21 state variables and 24 algebraic
variables, which are converted into ODEs with 39 state variables using
Modelingtoolkit. At $t=0$, we consider an initial overvoltage occurs in the
generator G1, i.e., its internal $q$-axis voltage, ${E^{\prime}}_{q,1}(0)$, is
0.45pu larger than the normal value, and its angle, $\delta_{1}(0)$, is 0.01pu
increased from the equilibrium. Then, at $t=20s$, we consider the disturbances
of load with two cases:

  * •

Small disturbance: demands at B5, B6, and B7 increase by $0.2+j0.05$pu,
$0.15$pu, and $0.2$pu respectively.

  * •

Large disturbance: demands at B5, B6, and B7 increase by $0.5+j0.25$pu,
${0.3}{+}{j}{0.05}$pu, and ${0.4}{+}{j}$0.05pu respectively.

Figure 6: WSCC three-machine nine-bus system [13, 1]

Figures 8-10-12 show the changes of variables (frequency,
$f_{i}=\dfrac{\omega_{i}}{2\pi}$, the scaled field voltage, $E_{fd}$, and the
stabilizer rate feedback $R_{f}$ of the exciter system, and active power
output, $P_{e}$ corresponding to generators G1, G2, and G3 ($i=1,2,3$)
respectively). It is observed that $t=0$s to $t=20$s, all generators’
variables oscillate for a few seconds from the initial values before reaching
and returning to the equilibrium values. The variation of $E_{fd}$ and $R_{f}$
illustrates the operations of generators’ exciters to stabilize over-voltage
issues occurring at $t=0$. When the load changes at $t=20s$, the active powers
generated by generators all increase to compensate for the increased loads
whereas their frequencies reduce. A larger change in loads leads to a higher
change of generators’ variables when we compare the results in Figures
8a-10a-12a and the results in in Figures 8b-10b-12b.

These results are used to validate the potential of quantum computing for
modeling system dynamics. The solid line in Figures 8-10-12 represents the
values obtained by quantum computing whereas the dashed line is the values
obtained by the classical Forward Euler method. In both periods ( $0\leq t<20$
and $t\geq 20$), quantum solvers simulate well the system dynamics induced by
the initial setting values of system variables as well as the changes of
loads. The differences between results obtained by quantum-based and classical
methods are measured by the RMSE metric. Compared to the SMIB case, the errors
are larger due to the higher complexity of the WSCC test cases along with the
DAEs modeling the system dynamics instead of the purely ODEs used in the SMIB.
However, these errors are very small and acceptable.

##  VII Conclusion

This paper presents the method of quantum computing to solve the power system
dynamics, which is typically represented as a set of DAEs and can be converted
into a set of ODEs using the index reduction method. Their data can be encoded
into quantum computers by using amplitude encoding, which requires a number of
qubits as a logarithm of the number of state variables. To this end, the
nonlinear ODEs need to be linearized as a set of polynomial functions of state
variables to be simulated in quantum computers by amplitudes of tensors and
tailored Hamiltonian tricks. The linear update in traditional ODE solvers can
be replaced by quantum-based linear solvers, such as the HHL algorithm with
proven quantum advantages. Our numerical results demonstrate the potential of
quantum computing in modeling power system dynamics with high accuracy.
Additionally, we also illustrate the use of scientific machine learning tools,
particularly symbolic programming, to facilitate the use of scientific
computing concepts, e.g., Taylor expansion, DAEs/ODEs transform, and quantum
solver, in the field of power engineering.

(a) Small change of loads

(b) Large change of loads

Figure 8: Generator 1

(a) Small change of loads

(b) Large change of loads

Figure 10: Generator 2

(a) Small change of loads

(b) Large change of loads

Figure 12: Generator 3

## References

  * [1] P. W. Sauer, M. A. Pai, and J. H. Chow, _Power system dynamics and stability: with synchrophasor measurement and power system toolbox_. John Wiley & Sons, 2017.
  * [2] U. M. Ascher and L. R. Petzold, _Computer methods for ordinary differential equations and differential-algebraic equations_. Siam, 1998, vol. 61.
  * [3] O. Kyriienko, A. E. Paine, and V. E. Elfving, “Solving nonlinear differential equations with differentiable quantum circuits,” _Physical Review A_ , vol. 103, no. 5, p. 052416, 2021.
  * [4] J. Stiasny, S. Chatzivasileiadis, and B. Zhang, “Solving differential-algebraic equations in power systems dynamics with neural networks and spatial decomposition,” _arXiv preprint arXiv:2303.10256_ , 2023\.
  * [5] A. W. Harrow, A. Hassidim, and S. Lloyd, “Quantum algorithm for linear systems of equations,” _Physical review letters_ , vol. 103, no. 15, p. 150502, 2009\.
  * [6] D. W. Berry, “High-order quantum algorithm for solving linear differential equations,” _Journal of Physics A: Mathematical and Theoretical_ , vol. 47, no. 10, p. 105301, 2014.
  * [7] A. M. Childs, R. Kothari, and R. D. Somma, “Quantum algorithm for systems of linear equations with exponentially improved dependence on precision,” _SIAM Journal on Computing_ , vol. 46, no. 6, pp. 1920–1950, 2017.
  * [8] T. Xin, S. Wei, J. Cui, J. Xiao, I. Arrazola, L. Lamata, X. Kong, D. Lu, E. Solano, and G. Long, “Quantum algorithm for solving linear differential equations: Theory and experiment,” _Physical Review A_ , vol. 101, no. 3, p. 032307, 2020.
  * [9] C. Rackauckas and Q. Nie, “Differentialequations.jl–a performant and feature-rich ecosystem for solving differential equations in julia,” _Journal of Open Research Software_ , vol. 5, no. 1, p. 15, 2017.
  * [10] Y. Ma, S. Gowda, R. Anantharaman, C. Laughman, V. Shah, and C. Rackauckas, “Modelingtoolkit: A composable graph transformation system for equation-based modeling,” 2021.
  * [11] S. K. Leyton and T. J. Osborne, “A quantum algorithm to solve nonlinear differential equations,” _arXiv preprint arXiv:0812.4423_ , 2008.
  * [12] D. J. Hill and G. Chen, “Power systems as dynamic networks,” in _2006 IEEE International Symposium on Circuits and Systems (ISCAS)_. IEEE, 2006, pp. 4–pp.
  * [13] S. Wang, W. Gao, and A. S. Meliopoulos, “An alternative method for power system dynamic state estimation based on unscented transform,” _IEEE transactions on power systems_ , vol. 27, no. 2, pp. 942–950, 2011.
  * [14] L. Petzold, “Differential/algebraic equations are not ode’s,” _SIAM Journal on Scientific and Statistical Computing_ , vol. 3, no. 3, pp. 367–384, 1982.
  * [15] A. Peres, _Quantum theory: concepts and methods_. Springer, 1997.


# Using Emotion Embeddings to transfer knowledge
between emotions, languages, and annotation formats

###### Abstract

The need for emotional inference from text continues to diversify as more and
more disciplines integrate emotions into their theories and applications.
These needs include inferring different emotion types, handling multiple
languages, and different annotation formats. A shared model between different
configurations would enable the sharing of knowledge and a decrease in
training costs, and would simplify the process of deploying emotion
recognition models in novel environments. In this work, we study how we can
build a single model that can transition between these different
configurations by leveraging multilingual models and Demux, a transformer-
based model whose input includes the emotions of interest, enabling us to
dynamically change the emotions predicted by the model. Demux also produces
emotion embeddings, and performing operations on them allows us to transition
to clusters of emotions by pooling the embeddings of each cluster. We show
that Demux can simultaneously transfer knowledge in a zero-shot manner to a
new language, to a novel annotation format and to unseen emotions. Code is
available at https://github.com/gchochla/Demux-MEmo.

Index Terms—  Multilingual emotion recognition, Zero-shot, Emotion clusters

##  1 Introduction

Human experience is permeated by emotions. They can guide our attention and
influence our information consumption, beliefs, and our interactions [10, 26].
Deep learning has enabled us to extract affective constructs from natural
language [7, 5], allowing emotion recognition from text at scale [13].
Nevertheless, the need for better performance across various metrics of
interest still exists.

When inferring emotions from text, earlier approaches have utilized emotion
lexicons [21]. These struggle in more realistic settings, because, for
example, they do not handle context, like negation. On the other hand, while
modern efforts relying on deep learning achieve better performance [7, 3],
these data-driven models have to contend with a multitude of biases, such as
annotation biases in the data used to train the models.

The needs for emotional inference from text have also diversified. First, the
domain of interest can vary greatly between applications, ranging from
everyday dialogues [17] to tweets [19]. Secondly, it is desirable for the
models to be able to handle multiple languages [19], such as when studying
perceptions and reactions to international news stories. Finally, the emotions
of interest can differ between applications, and perhaps even the annotation
scheme might not be similar. For example, in this work, we analyzed tweets
annotated for clusters of emotions, where emotions that co-occur frequently
were grouped, in contrast to single emotions in other settings. Hence,
transfer learning is hindered by the mismatch.

In this work, our main goal is to achieve transfer of emotion recognition
between annotation formats, emotions and languages. Our experimental design
examines this step-by-step. First, we leverage pretrained multilingual
language models [2, 9] to enable knowledge transfer between languages. We also
use and extend `Demux` [7], a model that incorporates the labels in its input
space to achieve the final classification. Emotions are then embedded in the
same space as the language tokens. Emotion word embeddings can facilitate
transfer between emotions, as shown in [7], so we study whether this can also
be achieved in a zero-shot manner. Lastly, we examine how an extension of
`Demux` to clusters can transfer knowledge between different annotation
formats by directly performing operations on label embeddings [18].

Our contributions include the following:

  * •

We show that multilingual emotion recognition models can be competitive with
or even outperform monolingual baselines, and that knowledge can be
transferred to new languages.

  * •

We demonstrate that `Demux` can inherently transfer knowledge to emotions it
has not been trained with.

  * •

We illustrate that operations on the contextual emotion embeddings of `Demux`
can successfully achieve transfer to novel annotation formats in a zero-shot
manner. To the best of our knowledge, we are the first to study this setting.

  * •

We show that `Demux` can be critical for flexible emotion recognition in a
dynamic environment with ever-changing inference needs, such as the addition
and subtraction of emotion types, changes in language, and alterations in the
annotation format, e.g., the clustering of different emotions.

##  2 Related Work

###  2.1 Emotion Recognition

Earlier works utilized Bag-of-Words algorithms driven by emotion lexicons. For
instance, LIWC [21] is a lexicon that is widely used to perform word counting,
while DDR [12] extends lexicon-based methods from word counting to computing
similarities between words. More recently, deep learning has enabled more
accurate extraction of emotion signals from text. Initial efforts have treated
the task as single-label, and use a threshold to transform into the desired
multi-label output [15]. LSTMs have been widely used for the task [11, 3]
e.g., for SemEval 2018 Task 1 [19], where some also used features from
affective lexicons. More recently, Transformers [25] have dominated the field.
`Demux` and `MEmo` [7], state of the art models in SemEval 2018 Task 1 E-c,
prompt BERT-based [9] models in different ways, by including all emotions in
the input or `[MASK]` tokens in language prompts, respectively. They also
employ an intra-group correlation loss to further improve performance.
Transformers have also been used with other architectures, like Graph Neural
Networks [27].

###  2.2 Multilingual Models & Emotion Recognition

Multilingual transformers attempt to model many languages simultaneously.
Normalized sampling from each language is used so that low-resource languages
are not severely hindered, which we also adopt. This is achieved, given
$\alpha\in[0,1]$, by transforming the frequency $p_{l}$ of each language as
$p_{l}^{\prime}\leftarrow p_{l}^{\alpha}$ and renormalizing to create the new
sampling distribution [16]. Note that as $\alpha$ decreases, the distribution
becomes more balanced, achieving parity at $\alpha=0$.

BERT [9] and XLM [16] require preprocessing of languages that do not use
spaces to delimit words. Both are trained on 100 languages on Wikipedia, and
use $\alpha=0.7,\ 0.5$ respectively. XLM also uses language embeddings, and
incorporates Translation Language Modeling (TLM) as a pretraining technique,
which requires parallel data. XLM-R [8] handles all languages without
preprocessing. It decreases $\alpha$ to $0.3$ and switches to the CommonCrawl
dataset, which has a more balanced language distribution. It also disposes of
language embeddings and TLM. XLM-T [2] extends XLM-R by finetuning it on
tweets to perform multilingual sentiment analysis, as does XLM-EMO [4] for
emotion recognition on four emotions.

###  2.3 Zero-shot Emotion Recognition

Very few works explicitly study zero-shot emotion recognition in text with
transformer-based models [28, 22]. They do so by formulating the problem as a
Natural Language Inference problem, i.e., by creating a different prompt per
emotion of interest, and classifying whether each prompt follows from the
input sequence (entailment) or not (contradiction). This requires running the
model once per emotion, creating a bottleneck for classification. Most similar
to ours are earlier approaches that used semantic similarity with emotion word
embeddings to classify in a zero-shot manner [24].

##  3 Methodology

We present the technical details of interest for `Demux` and our simple
extension for it to handle clusters of emotions. Let $E=\\{e_{i}:i\in[n]\\}$
be the set of emotions and $C=\\{C_{i}:i\in[m]\\}$ be some clustering of $E$
s.t. $n\geq m$, $\cup_{i\in[m]}C_{i}=[n]$ and $\cap_{i\in[m]}C_{i}=\emptyset.$

###  3.1 Demux

Let $x$ be an input text sequence. We construct
$x^{\prime}=\text{``}e_{1},e_{2},\dots,\text{or}\ e_{n}?\text{''}$ and use a
LM $L$ with its corresponding tokenizer $T$:

$\begin{split}\tilde{x}=T(x^{\prime},x)=(&[CLS],t_{1,1},\dots,t_{1,N_{1}},\dots,\\\
&t_{n,1},\dots,t_{n,N_{n}},[SEP],x_{1},\dots,x_{l}),\end{split}$ (1)

where $x_{i}$ are the tokens from $x$, $t_{i,j}$ the $j$-th subtoken of
$e_{i}$, and `[SEP]` and `[CLS]` are special tokens of $T$.

We pass $\tilde{x}$ through $L$ to finally get $\hat{x}=L(\tilde{x})$, where
$\hat{x}$ contains one output embedding corresponding to each input token. We
denote the output embedding corresponding to $t_{i,j}$ as
$\hat{t}_{i,j}\in\mathbb{R}^{d}$, where $d$ is the feature dimension of $L$.
Thereafter, we average-pool the outputs of the subtokens of each emotion, and
get the final predictions using a 2-layer neural network,
$\text{NN}:\mathbb{R}^{d}\rightarrow\mathbb{R}$, followed by sigmoid $\sigma$:

$\forall i\in[n],\quad
p(e_{i}|x)=\sigma(\text{NN}(\frac{\sum_{j=1}^{N_{i}}\hat{t}_{i,j}}{N_{i}}))).$
(2)

In the case of emotion clusters, $x^{\prime}$ contains all emotions from every
cluster. After the forward pass through $L$, we instead aggregate across all
emotions of a cluster and predict, for each cluster:

$\forall i\in[m],\quad p(C_{i}|x)=\sigma(\text{NN}(\frac{\sum_{j\in
C_{i}}\sum_{k=1}^{N_{j}}\hat{t}_{j,k}}{\sum_{j\in C_{i}}N_{j}}))).$ (3)

Moreover, when using multilingual models, we keep all emotions in English to
retain the same prompt, $x^{\prime}$, across all languages.

###  3.2 Correlation-aware Regularization

To provide extra supervision to the model and enhance its correlation
awareness between emotions, we include a label-correlation regularization
loss. This loss takes into account the ground-truth labels for each example in
its formulation. Therefore, the emotions are split into two groups, the
present and the absent emotions based on annotations $y$, $\mathcal{P}$ and
$\mathcal{N}$ respectively. We regularize intra-group relationships, meaning
we only pick pairs of emotions when they are both in $\mathcal{P}$ or both in
$\mathcal{N}$. The formulation is:

$\begin{split}\mathcal{L}_{L,\text{intra}}(y,\hat{y})=\frac{1}{2}&\left[\frac{1}{|\mathcal{N}|^{2}-|\mathcal{N}|}\sum_{(i,j)\in\mathcal{N}^{2}}^{i>j}e^{\hat{y}_{i}+\hat{y}_{j}}\right.+\\\
&\left.\frac{1}{|\mathcal{P}|^{2}-|\mathcal{P}|}\sum_{(i,j)\in\mathcal{P}^{2}}^{i>j}e^{-\hat{y}_{i}-\hat{y}_{j}}\right],\\\
\end{split}$ (4)

where $\hat{y}$ is the prediction of the model and subscripts indicate
indexing. In this manner, we decrease the distance of pairs of emotions when
they have the same gold labels. The denominators simply average the terms. The
final loss is a convex combination of the classification and the
regularization loss, dictated by hyperparameter $c$:

$\mathcal{L}=(1-c)\mathcal{L}_{BCE}+c\mathcal{L}_{L,\text{intra}}.$ (5)

##  4 Experiments

###  4.1 Datasets

We use the publicly available SemEval 2018 Task 1 E-c [19], and private data
containing tweets from the French elections of 2017.

SemEval 2018 Task 1 E-c (SemEval E-c) contains tweets annotated for 11
emotions in a multilabel setting, namely anger, anticipation, disgust, fear,
joy, love, optimism, pessimism, sadness, surprise, and trust, in English,
Arabic, and Spanish. The cardinalities are 6838 training, 886 development and
3259 testing for English, 2278 training, 585 development and 1518 testing for
Arabic, and 3561 training, 679 development and 2854 testing for Spanish. We
have also machine-translated the English subset of the tweets into French.

Emotion Cluster | Support in FrE-A | Support in FrE-B
---|---|---
Admiration | 589 (14.4%) | 51 (1.1%)
Sarcasm | 395 (9.6%) | 233 (5.1%)
Anger | 595 (14.5%) | 279 (6.1%)
Embarrassment | 182 (4.4%) | 56 (1.2%)
Fear | 271 (6.6%) | 116 (2.5%)
Joy | 169 (4.1%) | 40 (0.9%)
Optimism | 501 (12.2%) | 350 (7.7%)
Pride | 539 (13.1%) | 92 (2%)
Positive-other | 632 (15.4%) | 1268 (27.7%)
Negative-other | 542 (13.2%) | 1426 (31.2%)
All tweets | 4102 (100%) | 4574 (100%)
Table 1: Support per emotion cluster for our French election datasets.

The French election dataset contains mostly French tweets annotated for 10
clusters of emotions in a multilabel setting, namely admiration-love,
amusement-sarcasm, anger-hate-contempt-disgust, embarrassment-guilt-shame-
sadness, fear-pessimism, joy-happiness, optimism-hope, pride (including
national pride), any other positive, and any other negative emotions. The
formulation dictates that if one emotion in the cluster is present, the
cluster is considered present. Only cluster-wise annotations are requested. We
have two separate subsets, collected by third parties using keywords related
to prominent politicians, agendas and concerns of the French elections, and
annotated independently also by third parties. The first (“FrE-A”) contains
4102 tweets that we randomly split into training, development and test sets
with ratios 8:1:1. The second subset (“FrE-B”) contains 4574 tweet that we
also randomly split with ratios 8:1:1. In Table 1, we present the support for
each class. Annotations for FrE-B are a lot sparser.

###  4.2 Implementation Details

We use Python (v3.7.4), PyTorch (v1.11.0) and the transformers library
(v4.19.2). We use NVIDIA GeForce GTX 1080 Ti. Given the difference in writing
formality on Twitter, we use Twitter-based in addition to general-purpose
models. For the former, XLM-T [2] is our multilingual model, BERTweet [20] for
English, RoBERTuito [23] for Spanish, AraBERT [1] for Arabic, and BERTweetFR
[14] for French. For the latter, we use multilingual BERT [9] as our
multilingual model, BERT for English and Arabic, and BETO [6] for Spanish. We
retain the hyperparameters used in [7], such as the learning rate and its
warmup, early stopping, batch size, the convex combination coefficient $c$
($\alpha$ in [7]), and the text preprocessor. During multilingual finetuning
on SemEval E-c, we sample from different languages with equal probability
($\alpha=0$ in aforementioned normalization in Section 2.2). We evaluate using
the F1 score for individual emotions, and micro F1, macro F1 and Jaccard score
(JS) otherwise. We also used Micro F1 for early stopping in FrE-A because we
found JS to be unreliable due to the label distribution.

###  4.3 Knowledge Transfer between Languages

Multilingual Training and Evaluation First, we examine how feasible and
competitive it is to use multilingual emotion recognition models trained and
evaluated on a mixture of languages. To establish this, we first consider
training monolingual and multilingual models, pretrained either on tweets or
general-purpose text. For multilingual models, we also consider how training
and/or evaluating on multiple languages fares with training and/or evaluating
on a single language. For example, we compare performance on the Arabic subset
when training and performing early stopping solely on Arabic, with the
performance when training on all languages simultaneously, but still
performing early stopping on Arabic, and other combinations.

We find that training and evaluating on all languages in SemEval E-c has
either only slightly negative or strongly positive influence on accuracy for
higher-resource languages. Our dev set results are presented in Table 2. It
becomes immediately obvious that models trained on Twitter comfortably
outperform their general-purpose alternatives (first and the second row, and
third and final row). We also observe that using a dev set of multiple
languages not only does not hurt performance, but actually achieves positive
knowledge transfer for Spanish, achieving the best JS overall. The monolingual
alternatives perform favorably in English and Arabic, with the increase in the
former being relatively minor. Overall, since our ultimate goal is to transfer
to French tweets, and given the competent or superior performance for Latin-
based languages, we do adopt the multilingual model trained and evaluated on
multilingual data for pretraining.

|  |  |  | JS
---|---|---|---|---
Setting | En | Es | Ar
| Monolingual models |  |  |  $61.0{\scriptstyle\pm 0.3}$ |  $53.2{\scriptstyle\pm 0.5}$ |  $55.1{\scriptstyle\pm 0.3}$
Twitter-based | Monolingual models |  |  |  $\mathbf{62.0}{\scriptstyle\pm 0.4}$ |  $55.6{\scriptstyle\pm 0.3}$ |  $\mathbf{61.6}{\scriptstyle\pm 0.5}$
| Multilingual models | w/ Multilingual Training | & Evaluation |  $58.4{\scriptstyle\pm 1.0}$ |  $50.3{\scriptstyle\pm 0.4}$ |  $49.2{\scriptstyle\pm 0.0}$
Twitter-based | Multilingual models |  |  |  $61.6{\scriptstyle\pm 0.1}$ |  $56.8{\scriptstyle\pm 1.0}$ |  $56.5{\scriptstyle\pm 0.6}$
Twitter-based | Multilingual models | w/ Multilingual Training |  |  $61.3{\scriptstyle\pm 0.4}$ |  $56.5{\scriptstyle\pm 0.6}$ |  $57.9{\scriptstyle\pm 1.0}$
Twitter-based | Multilingual models | w/ Multilingual Training | & Evaluation |  $61.3{\scriptstyle\pm 0.2}$ |  $\mathbf{58.0}{\scriptstyle\pm 0.7}$ |  $57.7{\scriptstyle\pm 0.4}$
Table 2: Comparing Jaccard scores in SemEval 2018 Task 1 E-c. The variables we
consider are: monolingual or multilingual models, Twitter-based or general-
purpose models, monolingual or multilingual training, where the training set
is comprised of one or a mixture of all languages, and monolingual or
multilingual evaluation, where the evaluation set is comprised of one or a
mixture of all languages.

Transfer to New Languages In trying to establish how emotion recognition
knowledge is transferred to new languages, we conducted experiments with one
SemEval E-c language left out during training. Results are shown in Table 3.
We notice a drop in performance, with all models performing roughly
equivalently across all metrics on the new language notwithstanding original
performance. In detail, all metrics drop by around $25\%$ in Arabic, $<22\%$
for Spanish, while the drop in English ranges from $18\%$ to $32\%$.
Nonetheless, emotion recognition in the new language occurs at a competent
level, picking up clear signals despite the noise from the language switch,
rendering multilingual emotion recognition models capable of being used with
new languages.

|  |  |  | SemEval 2018 Task 1 E-c
---|---|---|---|---
Train langs | Eval langs | Mic-F1 | Mac-F1 | JS
En | Es |  | Ar |  $55.8{\scriptstyle\pm 0.9}$ |  $42.4{\scriptstyle\pm 2.7}$ |  $43.3{\scriptstyle\pm 1.3}$
En | Es | Ar | Ar |  $70.2{\scriptstyle\pm 0.8}$ |  $57.4{\scriptstyle\pm 1.4}$ |  $57.9{\scriptstyle\pm 1.0}$
En |  | Ar | Es |  $55.9{\scriptstyle\pm 0.7}$ |  $45.1{\scriptstyle\pm 1.6}$ |  $44.0{\scriptstyle\pm 0.9}$
En | Es | Ar | Es |  $65.1{\scriptstyle\pm 0.6}$ |  $54.7{\scriptstyle\pm 0.8}$ |  $56.5{\scriptstyle\pm 0.6}$
| Es | Ar | En |  $54.6{\scriptstyle\pm 0.6}$ |  $47.0{\scriptstyle\pm 1.3}$ |  $41.9{\scriptstyle\pm 0.7}$
En | Es | Ar | En |  $72.3{\scriptstyle\pm 0.3}$ |  $57.6{\scriptstyle\pm 2.0}$ |  $61.3{\scriptstyle\pm 0.4}$
Table 3: Leave-one-language-out experiments.

###  4.4 Knowledge Transfer to New Emotions

We also assess `Demux`’s ability to perform zero-shot emotion recognition by
excluding emotions from SemEval E-c, one at a time. We choose anger, joy,
pessimism, and trust to capture the change in accuracy across a wide spectrum
of performance levels. In particular, joy and trust are the highest and lowest
performing emotions, whereas anger and pessimism have relatively high and low
scores, respectively. Results are presented in Table 4. Again, we do notice a
decrease in performance. However, the model can still predict these unseen
emotions, especially those with which the original model was competent,
whereas pessimism deteriorates significantly.

To remedy the decrease in performance, we experimented with freezing word
embeddings in an effort to retain the relationships between emotions. We
examine two alternatives, freezing the word embedding layer altogether, and
freezing only the emotion word embeddings (including the novel emotions). Both
techniques are also presented in Table 4. In general, we find this decreases
performance for the model, indicating `Demux` already competently captures
relationships across emotions in its input.

|  | SemEval 2018 Task 1 E-c F1
---|---|---
ZS | Frozen | Anger | Joy | Pessimism | Trust
✗ | - |  $78.6{\scriptstyle\pm 0.3}$ |  $86.0{\scriptstyle\pm 0.3}$ |  $40.5{\scriptstyle\pm 1.5}$ |  $9.9{\scriptstyle\pm 3.9}$
✓ | - |  $42.5{\scriptstyle\pm 11.8}$ |  $57.8{\scriptstyle\pm 3.2}$ |  $11.7{\scriptstyle\pm 0.9}$ |  $3.6{\scriptstyle\pm 3.5}$
✓ | Words |  $25.1{\scriptstyle\pm 18.2}$ |  $51.5{\scriptstyle\pm 7.3}$ |  $15.1{\scriptstyle\pm 12.9}$ |  $1.7{\scriptstyle\pm 2.0}$
✓ | Emos |  $21.5{\scriptstyle\pm 20.8}$ |  $36.2{\scriptstyle\pm 14.3}$ |  $4.3{\scriptstyle\pm 4.3}$ |  $3.7{\scriptstyle\pm 4.6}$
Table 4: Zero-shot performance of unseen emotions. |  |  | Mic-F1 | Mac-F1 | JS
---|---|---|---|---|---
| Pretrained | Finetuned | FrE-A
| Most frequent | $0$ | $0$ | $24.9$
| Uni. Random |  $17.6{\scriptstyle\pm 0.4}$ |  $18.0{\scriptstyle\pm 0.4}$ |  $10.0{\scriptstyle\pm 0.2}$
XLM-T | ✓ | ✗ |  $21.0{\scriptstyle\pm 1.2}$ |  $15.8{\scriptstyle\pm 2.0}$ |  $26.6{\scriptstyle\pm 2.2}$
✗ | ✓ |  $28.9{\scriptstyle\pm 1.4}$ |  $25.7{\scriptstyle\pm 0.7}$ |  $\mathbf{29.8}{\scriptstyle\pm 1.7}$
✓ | ✓ |  $\mathbf{30.5}{\scriptstyle\pm 2.5}$ |  $\mathbf{26.5}{\scriptstyle\pm 5.2}$ |  $29.4{\scriptstyle\pm 2.3}$
BERT TweetFr | ✓ | ✗ |  $24.8{\scriptstyle\pm 0.7}$ |  $18.5{\scriptstyle\pm 1.4}$ |  $15.2{\scriptstyle\pm 0.6}$
✗ | ✓ |  $27.0{\scriptstyle\pm 12.5}$ |  $24.9{\scriptstyle\pm 12.8}$ |  $29.6{\scriptstyle\pm 2.6}$
✓ | ✓ |  $\mathbf{34.3}{\scriptstyle\pm 1.1}$ |  $\mathbf{31.4}{\scriptstyle\pm 1.7}$ |  $\mathbf{31.8}{\scriptstyle\pm 1.0}$
|  |  | FrE-B
| Most frequent | $0$ | $0$ | $35.0$
| Uni. Random |  $15.0{\scriptstyle\pm 0.7}$ |  $12.3{\scriptstyle\pm 0.7}$ |  $8.3{\scriptstyle\pm 0.4}$
XLM-T | ✓ | ✗ |  $18.3{\scriptstyle\pm 3.7}$ |  $11.5{\scriptstyle\pm 1.9}$ |  $29.4{\scriptstyle\pm 4.6}$
✗ | ✓ |  $55.3{\scriptstyle\pm 6.5}$ |  $19.3{\scriptstyle\pm 9.9}$ |  $55.3{\scriptstyle\pm 6.5}$
✓ | ✓ |  $\mathbf{62.7}{\scriptstyle\pm 0.9}$ |  $\mathbf{44.4}{\scriptstyle\pm 2.8}$ |  $\mathbf{59.3}{\scriptstyle\pm 0.8}$
* 11 ✓ | ✗ |  $15.9{\scriptstyle\pm 2.6}$ |  $11.3{\scriptstyle\pm 1.0}$ |  $14.8{\scriptstyle\pm 3.3}$
* 11 ✓ | ✓ |  $61.9{\scriptstyle\pm 1.2}$ |  $39.6{\scriptstyle\pm 9.6}$ |  $58.4{\scriptstyle\pm 0.8}$
BERT TweetFr | ✓ | ✗ |  $27.9{\scriptstyle\pm 1.9}$ |  $15.9{\scriptstyle\pm 1.1}$ |  $15.3{\scriptstyle\pm 1.4}$
✗ | ✓ |  $63.8{\scriptstyle\pm 1.4}$ |  $29.7{\scriptstyle\pm 7.6}$ |  $60.2{\scriptstyle\pm 1.7}$
✓ | ✓ |  $\mathbf{66.9}{\scriptstyle\pm 0.5}$ |  $\mathbf{40.3}{\scriptstyle\pm 6.0}$ |  $\mathbf{63.2}{\scriptstyle\pm 1.3}$
Table 5: Performance in French election data when the models pretrain on
SemEval E-c and/or finetune on the corresponding dataset. *: maximum
probability instead of pooling embeddings in Demux.

###  4.5 Knowledge Transfer to New Annotation Format

Finally, we evaluate if the model can successfully transfer knowledge to a new
annotation format. In particular, we transfer from SemEval E-c to FrE-a and
FrE-B. The former contains tweets in English, Spanish and Arabic, and
annotations for emotions. The latter contain French tweets annotated for
clusters. Additionally, the emotions of the latter are not a subset of the
former. Therefore, we expect to observe compounding effects from the multiple
changes in the setting. To address the language switch, we also use the French
translations to train BERTweetFr as a monolingual parallel to XLM-T.

We also show two simple baselines, one that predicts the most frequent label
for each emotion, and another that predicts uniformly randomly. Moreover, our
experiments include an alternative to averaging emotion embeddings with
`Demux`, where we instead select the maximum predicted probability across
emotions in a cluster.

Results are presented in Table 5. We consider three different settings for
each model and each dataset in order to study how training on SemEval E-c
(Pretrained) and training on the French election datasets (Finetuned) affects
performance. In the first row per model and dataset, we see the zero-shot
performance on the dataset. The second shows the performance of a model only
trained on the dataset. The last row shows performance from a model first
trained on SemEval E-c and then on the corresponding dataset.

For FrE-A, zero-shot performance of XLM-T is close to either finetuned
performance, indicating the model can successfully transfer despite the
complete change of environment. Performance increases for both when training
on French data, and when SemEval E-c is also included. The monolingual model
performs favorably only when fully supervised, which indicates that
translations are not ideal for zero-shot transfer, and overall proves the
ability of multilingual models to transfer knowledge to a new language.

Performance is evidently better in FrE-B. Again, XLM-T performs better than
random and favorably to BERTweetFr in the zero-shot setting. Finetuning
improvements are significant for both models. Lastly, the alternative pooling
performs worse than our proposed one. This speaks to the subjectivity of the
annotations, allowing operations on emotion embeddings but not the predictions
themselves.

##  5 Conclusion

In this work, we study how to transfer emotion recognition knowledge to
different languages, different emotions, and different annotation formats. We
find that multilingual models have the capacity to transfer that kind of
knowledge sufficiently well. In order to transfer knowledge between emotions,
we leverage `Demux`’s transferability between emotions through word-level
associations. We see that the model also inherently performs zero-shot emotion
recognition without the need for further changes. Finally, we modify `Demux`
to perform aggregation operations on its label embeddings, and show this can
transfer knowledge to novel annotation formats, such as clusters of emotions,
even in conjunction with the presence of novel emotions and in a different
language. We show that multilingual models pretrained on other languages
perform favorably in the zero-shot setting to native models pretrained on
machine translations.

##  6 Acknowledgements

This project was funded in part by DARPA under contract HR001121C0168.

## References

  * [1] Antoun, W., Baly, F., and Hajj, H. Arabert: Transformer-based model for arabic language understanding.  In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020 (2020), p. 9.
  * [2] Barbieri, F., Espinosa-Anke, L., and Camacho-Collados, J. Xlm-t: Multilingual language models in twitter for sentiment analysis and beyond.  Proceedings of the LREC, Marseille, France (2022), 20–25.
  * [3] Baziotis, C., Athanasiou, N., Chronopoulou, A., Kolovou, A., Paraskevopoulos, G., Ellinas, N., Narayanan, S., and Potamianos, A. Ntua-slp at semeval-2018 task 1: Predicting affective content in tweets with deep attentive rnns and transfer learning.  arXiv preprint arXiv:1804.06658 (2018).
  * [4] Bianchi, F., Nozza, D., and Hovy, D. Xlm-emo: Multilingual emotion prediction in social media text.  In Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis (2022), pp. 195–203.
  * [5] Calvo, R. A., D’Mello, S., Gratch, J. M., and Kappas, A. The Oxford handbook of affective computing.  Oxford Library of Psychology, 2015.
  * [6] Cañete, J., Chaperon, G., Fuentes, R., Ho, J.-H., Kang, H., and Pérez, J. Spanish pre-trained bert model and evaluation data.  In PML4DC at ICLR 2020 (2020).
  * [7] Chochlakis, G., Mahajan, G., Baruah, S., Burghardt, K., Lerman, K., and Narayanan, S. Leveraging label correlations in a multi-label setting: A case study in emotion, 2022.
  * [8] Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. Unsupervised cross-lingual representation learning at scale.  arXiv preprint arXiv:1911.02116 (2019).
  * [9] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805 (2018).
  * [10] Dukes, D., Abrams, K., Adolphs, R., Ahmed, M. E., Beatty, A., Berridge, K. C., Broomhall, S., Brosch, T., Campos, J. J., Clay, Z., et al. The rise of affectivism.  Nature human behaviour 5, 7 (2021), 816–820.
  * [11] Felbo, B., Mislove, A., Søgaard, A., Rahwan, I., and Lehmann, S. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm.  arXiv preprint arXiv:1708.00524 (2017).
  * [12] Garten, J., Hoover, J., Johnson, K. M., Boghrati, R., Iskiwitch, C., and Dehghani, M. Dictionaries and distributions: Combining expert knowledge and large scale textual data content analysis.  Behavior research methods 50, 1 (2018), 344–361.
  * [13] Guo, S., Burghardt, K., Rao, A., and Lerman, K. Emotion regulation and dynamics of moral concerns during the early covid-19 pandemic.  arXiv preprint arXiv:2203.03608 (2022).
  * [14] Guo, Y., Rennard, V., Xypolopoulos, C., and Vazirgiannis, M. Bertweetfr: Domain adaptation of pre-trained language models for french tweets.  arXiv preprint arXiv:2109.10234 (2021).
  * [15] He, H., and Xia, R. Joint binary neural network for multi-label learning with applications to emotion classification.  In CCF International Conference on Natural Language Processing and Chinese Computing (2018), Springer, pp. 250–259.
  * [16] Lample, G., and Conneau, A. Cross-lingual language model pretraining.  arXiv preprint arXiv:1901.07291 (2019).
  * [17] Li, Y., Su, H., Shen, X., Li, W., Cao, Z., and Niu, S. Dailydialog: A manually labelled multi-turn dialogue dataset.  arXiv preprint arXiv:1710.03957 (2017).
  * [18] Mikolov, T., Yih, W.-t., and Zweig, G. Linguistic regularities in continuous space word representations.  In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies (2013), pp. 746–751.
  * [19] Mohammad, S., Bravo-Marquez, F., Salameh, M., and Kiritchenko, S. Semeval-2018 task 1: Affect in tweets.  In Proceedings of the 12th international workshop on semantic evaluation (2018), pp. 1–17.
  * [20] Nguyen, D. Q., Vu, T., and Nguyen, A. T. Bertweet: A pre-trained language model for english tweets.  arXiv preprint arXiv:2005.10200 (2020).
  * [21] Pennebaker, J. W., Francis, M. E., and Booth, R. J. Linguistic inquiry and word count: Liwc 2001.  Mahway: Lawrence Erlbaum Associates 71, 2001 (2001), 2001.
  * [22] Plaza-del Arco, F. M., Martín-Valdivia, M.-T., and Klinger, R. Natural language inference prompts for zero-shot emotion classification in text across corpora.  arXiv preprint arXiv:2209.06701 (2022).
  * [23] Pérez, J. M., Furman, D. A., Alemany, L. A., and Luque, F. Robertuito: a pre-trained language model for social media text in spanish, 2021.
  * [24] Sappadla, P. V., Nam, J., Mencía, E. L., and Fürnkranz, J. Using semantic similarity for multi-label zero-shot classification of text documents.  In ESANN (2016).
  * [25] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need.  Advances in neural information processing systems 30 (2017).
  * [26] Wahl-Jorgensen, K. Emotions, media and politics.  John Wiley & Sons, 2019.
  * [27] Xu, P., Liu, Z., Winata, G. I., Lin, Z., and Fung, P. Emograph: Capturing emotion correlations using graph networks.  arXiv preprint arXiv:2008.09378 (2020).
  * [28] Yin, W., Hay, J., and Roth, D. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (Hong Kong, China, Nov. 2019), Association for Computational Linguistics, pp. 3914–3923.