# Multilogue-Net: A Context Aware RNN for Multi-modal Emotion Detection and
Sentiment Analysis in Conversation

Aman Shenoy
Birla Inst. of Technology and Science, Pilani
Pilani, RA, India
<EMAIL_ADDRESS>
&Ashish Sardana
NVIDIA Graphics
Bengaluru, KA, India
<EMAIL_ADDRESS>
* The following work was pursued when author was an intern at NVIDIA Graphics, Bengaluru

###### Abstract

Sentiment Analysis and Emotion Detection in conversation is key in several
real-world applications, with an increase in modalities available aiding a
better understanding of the underlying emotions. Multi-modal Emotion Detection
and Sentiment Analysis can be particularly useful, as applications will be
able to use specific subsets of available modalities, as per the available
data. Current systems dealing with Multi-modal functionality fail to leverage
and capture - the context of the conversation through all modalities, the
dependency between the listener(s) and speaker emotional states, and the
relevance and relationship between the available modalities. In this paper, we
propose an end to end RNN architecture that attempts to take into account all
the mentioned drawbacks. Our proposed model, at the time of writing, out-
performs the state of the art on a benchmark dataset on a variety of accuracy
and regression metrics.

##  1 Introduction

Multi-modal Emotion Detection and Sentiment Analysis in conversation is
gathering a lot of attention recently considering its potential use cases
owing to the rapid growth of online social media platforms such as YouTube,
Facebook, Instagram, Twitter etc. (Chen et al., 2017, Poria et al., 2016,
Poria et al., 2017, Zadeh et al., 2016b, Zadeh et al., 2017), especially
knowing that information obtained from any combination of more than one of the
available modalities (e.g. text, audio, video) can be used to produce
meaningful results.

The current state of the art systems on multi-modal emotion detection and
sentiment analysis do not treat the modalities in accordance to the
information they are capable of holding (e.g. textual information is
significantly more likely to hold contextual information then audio or video
features are), lack an adequate fusion mechanism, and fail to effectively
capture the context of a conversation in a multi-modal setting. In addition to
the lack of proper usage of the available modalities, models also fail to
effectively capture the flow of a conversation, the separation between speaker
and listener states, and the emotional effect a speaker’s utterance has on the
listener (s) in dyadic conversations.

Our proposed model Multilogue-Net, attempts to embed basic domain knowledge
and takes insight from Poria et al. (2019), assuming that the sentiment or
emotion governing a particular utterance predominantly depends on 4 factors –
interlocutor state, interlocutor intent, the preceding and future emotions,
and the context of the conversation. Interlocutor intent amongst the mentioned
is particularly difficult to model due to its dependency of prior knowledge
about the speaker, but modelling the other 3 separately, yet in an
interrelated manner was theorized to produce meaningful results if managed to
be captured effectively. The key intention was to attempt to simulate the
setting in which an utterance is said, and use the actual utterance at that
point to be able to gain better insights regarding emotion and sentiment of
that utterance. The model uses information from all modalities learning
multiple state vectors (representing interlocutor state) for a given
utterance, followed by a pairwise attention mechanism inspired by Ghosal et
al. (2018), attempting to better capture the relationship between all pairs of
the available modalities.

The model uses two gated recurrent units (GRU) Chung et al. (2014) for each
modality for modelling interlocutor state and emotion. Along with these GRU’s,
the model also uses an inter-connected context network, consisting of the same
number of GRU’s as the number of available modalities, to model a different
learned context representation for each modality. The incoming utterance
representations and the historical GRU outputs are used at every timestamp to
be able to arrive at a prediction for that timestamp.

The model produces $m$ different representations at every timestamp (Where $m$
is the number of modalities), where each representation is the emotional state
at that timestamp as conveyed by each of the modalities. These $m$
representations are used by the fusion mechanism to incorporate information
from each of the $m$ representations to be able to arrive at the final
prediction for that timestamp. We understand that the usage of the pairwise
attention mechanism, along with the Emotion GRU are what make the model
flexible across tasks.

The usage of only the text representation as input to the context GRU’s has
been observed to be key to the results, as the context of the conversation
would be better captured by textual information then it would have with audio
or video information. We believe that Multilogue-net performs better than the
current state of the art Ghosal et al. (2018) on multi-modal datasets because
of better context representation leveraging all available modalities.111A
basic model and training implementation of Multilogue-Net can be found at
https://github.com/amanshenoy/multilogue-net.

The remaining sections of the paper are arranged as follows: Section 2 –
discusses related work; Section 3 – discusses the model in detail; Section 4 –
provides experimental results, dataset details, and analysis; Section 5
contains our ablation studies and its implications; and finally Section 6 –
speaks on potential future work, and concludes our paper.

##  2 Related Work

Multi-modal Emotion recognition and Sentiment Analysis has always attracted
attention in multiple fields such as natural language processing, psychology,
cognitive science, and so on Picard (2010). Previous works have been done
studying factors of variation that have a more direct correlation with
emotion, such as Ekman et al. (1992), who found correlation between emotion
and facial cues, and a lot of studies extensively focus on emotions and their
relationship with one another such as Plutchik’s wheel of emotions, which
defines eight primary emotion types, each of which has a multitude of emotions
as sub-types.

Early work done to leverage multi-modal information for emotion recognition
includes works such as Datcu and Rothkrantz (2012), who fused acoustic
information with visual cues for emotion recognition and Eyben et al. (2010),
who used contextual information for emotion recognition in multi-modal
settings. More recently, deep recurrent neural networks have been used to be
able make the best of the learned representations of the modalities available
to be able to give very effective and accurate emotion and sentiment
predictions. Poria et al. (2017) successfully used RNN-based deep networks for
multi-modal emotion recognition, which was followed by multiple other works
(Chen et al., 2017; Zadeh et al., 2018a; Zadeh et al., 2018c) giving results
far better than what was seen before. Recent works also include works such as
Hazarika et al. (2018), who used memory networks for emotion recognition in
dyadic conversations, where two distinct memory networks enabled inter-speaker
interaction.

Some works such as DialogueRNN Majumder et al. (2018), though focused on
emotion recognition and sentiment analysis using a single modality (text),
works very well in a multi-modal setting by just replacing the text
representation with a concatenated vector of all the modality representations.
DialogueRNN effectively leveraged the separation between the speakers by
maintaining two independent gated recurrent units to keep track of the
interlocutor states, also effectively capturing context in the conversation,
yielding state-of-the-art performance on uni-modal data. Even though
DialogueRNN was able to give reasonably good results on multi-modal data, the
lack of an adequate fusion mechanism and the lack of focus on a multi-modal
representation held its multi-modal performance back.

Apart from the kind of works shown before, where a methodology or a model was
proposed, works such as Poria et al. (2019) spoke extensively about the
research challenges and advancements in emotion detection in conversation and
gave a comprehensive overview of the problem. Most recently Ghosal et al.
(2018) introduced the idea of learning the relationship between pairs of all
available modalities using pairwise attention, in a multi-modal setting, where
similar attributes learned by multiple modalities are emphasized and
differences between the modality representations are diminished. Pairwise
attention proved to be incredibly effective yielding state-of-the-art
performance on multi-modal data with just simple representations for each
modality.

##  3 Proposed Methodology

###  3.1 Problem Formulation

Let there be a $P$ number of participants $p_{1},p_{2},...,p_{P}$ in the
conversation. The problem is defined such that for every utterance
$u_{1},u_{2},...,u_{N}$ uttered by any participant(s), a sentiment score is
allotted along with a predicted emotion label (one of happy, sad, angry,
surprise, disgust, and fear). Each utterance corresponds to a particular
participant of the conversation, allowing this formulation of the problem to
also capture the average sentiment of a participant in the conversation.
Predictions over utterances also avoid problems such as classification during
long moments of silence when predictions are made for a fixed time interval,
and is also mostly common practice.

For every utterance $u_{t}(p)$, where $p$ is the party who uttered the
utterance, there exist three independent representations ,
$t_{t}\in\mathbb{R}^{D_{t}}$ , $a_{t}\in\mathbb{R}^{D_{a}}$, and
$v_{t}\in\mathbb{R}^{D_{v}}$, and are obtained using the feature extractors
further explained in section 4.2.

This gives us our overall formulation of the problem, which is to be able to
learn a function which would take as input three independent representations
of a particular utterance, information regarding the previous emotional state
of the participant, and a representation of the current context of the
conversation - to be able to map to an output prediction of a sentiment score
and emotion label.

Details regarding how these representations are updated and how the output is
generated using these inputs are described in detail below.

###  3.2 Model Details

Modelling was done under the underlying assumption that the sentiment or
emotion of an utterance predominantly depends on four factors as mentioned
before:

  * •

Interlocutor State

  * •

Interlocutor Intent

  * •

Context of the conversation until that point

  * •

Previous interlocutor states and emotions of a particular participant in the
conversation

The proposed model attempts to model three out of the mentioned four
explicitly, and assume that interlocutor intent will be modelled implicitly
during model training. Interlocutor state is modelled using a state GRU (will
be referred to as $sGRU$), A context GRU is used to keep track of the context
of the conversation ($cGRU$), and an emotion GRU ($eGRU$) is used to keep
track of the emotional state of that particular participant. Finally, a
pairwise attention mechanism, which uses the emotion representation of all
modalities at a particular timestamp is used to leverage the important
modalities and relevant combination of the modalities for emotion or sentiment
prediction at that timestamp.

Figure 1: Description of all the state updates at timestamp $t$ for a single
participant $p_{1}$

Every utterance has three independent feature representations (text, audio,
and video features), $t_{t}\in\mathbb{R}^{D_{t}}$ ,
$a_{t}\in\mathbb{R}^{D_{a}}$, and $v_{t}\in\mathbb{R}^{D_{v}}$. Each of these
feature representations are treated and operated on independently until the
pairwise attention mechanism. The model consists of two GRU’s (state GRU, and
emotion GRU) for every modality and participant, and a context GRU for each
modality common to all participants in the conversation (If $p$ is the number
of participants and $m$ is the number of modalities, the model would have a
total of $2mp+m$ GRU’s). The inputs at the current timestamp and the previous
state, context, and emotion representations are operated on to be able to
arrive at the prediction at that timestamp. Figure 1 describes the updates at
a particular timestamp and the role of each GRU is further explained below.

Figure 2: State updates and final prediction output in a conversation between
two participants $p_{1}$ and $p_{2}$, where the updates of each participant at
a timestamp is as given in figure 1

####  3.2.1 Context GRU ($cGRU$)

The Context GRU ($cGRU$) for each modality aims to capture the context of the
conversation by jointly encoding the utterance representation of that modality
(at timestamp $t$ in the given diagram) ($t_{t}\in\mathbb{R}^{D_{t}}$ ,
$a_{t}\in\mathbb{R}^{D_{a}}$, or $v_{t}\in\mathbb{R}^{D_{v}}$) and the
previous timestamp speaker state GRU output of that modality. This accounts
for inter-speaker and inter-utterance dependencies to produce an effective
context representation. The current utterance $t_{t}$, $a_{t}$, or $v_{t}$,
changes the state of that speaker from ($s_{t}^{t}$, $s_{t}^{a}$, $s_{t}^{v}$)
to ($s_{t+1}^{t}$, $s_{t+1}^{a}$, $s_{t+1}^{v}$). To capture this change in
context we use GRU cell $cGRU$ having output size $D_{c}$, using $t_{t}$,
$a_{t}$, or $v_{t}$ and ($s_{t}^{t}$, $s_{t}^{a}$, $s_{t}^{v}$) as:

$c_{t+1}^{t}=cGRU(c_{t}^{t},(t_{t}\oplus s_{t}^{t}))$ (1)

$c_{t+1}^{a}=cGRU(c_{t}^{a},(a_{t}\oplus s_{t}^{a}))$ (2)

$c_{t+1}^{v}=cGRU(c_{t}^{v},(v_{t}\oplus s_{t}^{v}))$ (3)

Where $D_{c}$ is the size of the context vectors $c_{t+1}^{t}$, $c_{t+1}^{a}$,
and $c_{t+1}^{v}$.$D_{t}$, $D_{a}$, and $D_{v}$ are the sizes of utterance
representations of text, audio, and video respectively.$\oplus$ represents the
concatenation operation, $D_{s}$ is the size of all the state vectors
$s_{t+1}^{t}$, $s_{t+1}^{a}$, and $s_{t+1}^{v}$; and all GRU weight and biases
shapes are such that they produce the expected shape of outputs taking the
given shape of inputs.

####  3.2.2 State GRU ($sGRU$)

The network keeps track of the participants involved in a conversation by
employing a $p*m$ number of ($sGRU$)’s, where $p$ is the number participants
in the conversation and $m$ is the number of available modalities.The $sGRU$
associated with a participant outputs fixed size vectors which serve as an
encoding to represent the interlocutor state, and are directly used for both
emotion and sentiment prediction, and updating the context vectors.

All the state vectors are initialized to null at the first timestamp. For a
timestamp $t$, the state vector of participant $p$ and modality
$m\in\\{t,a,v\\}$ is updated using the input feature representation of that
modality and simple attention over all the context vectors until that
timestamp. The simple attention mechanism over all the context vectors is
described by the following equations:

$\alpha=softmax({m_{t}^{T}}{W_{\alpha}}{[c_{1}^{m},c_{2}^{m},...,c_{t}^{m}]})$
(4) $att_{t}={\alpha}{[c_{1}^{m},c_{2}^{m},...,c_{t}^{m}]^{T}}$ (5)

Where $m_{t}^{T}\in\\{t_{t}^{T},a_{t}^{T},v_{t}^{T}\\}$,
$W_{\alpha}\in\mathbb{R}^{D_{t,a,v}\times D_{c}}$,
${\alpha}^{T}\in\mathbb{R}^{(t-1)}$, and $att_{t}\in\mathbb{R}^{D_{c}}$. In
equation 4, we calculate attention scores over all previous context
representations of all previous utterances, highlighting the relative
importance of all the previous context vectors to $m_{t}$. A softmax layer is
applied to amplify this relative importance, and finally equation 5 the final
output of attention over context $att_{t}$ is calculated by pooling the
previous context vectors with $\alpha$.

We then employ $sGRU^{t,a,v}$ to update $s_{t}^{t,a,v}$ to $s_{t+1}^{t,a,v}$
on the basis of incoming utterance representations for each modality
$m_{t}^{T}\in\\{t_{t}^{T},a_{t}^{T},v_{t}^{T}\\}$ and the context
representations $att_{t}^{t}$, $att_{t}^{a}$, and $att_{t}^{v}$ using GRU
cells $sGRU_{t}^{t}$, $sGRU_{t}^{a}$, and $sGRU_{t}^{v}$, each of output size
$D_{s}$.

$s_{t+1}^{t}=sGRU(s_{t}^{t},(t_{t}\oplus att_{t+1}^{t}))$ (6)

$s_{t+1}^{a}=sGRU(s_{t}^{a},(a_{t}\oplus att_{t+1}^{a}))$ (7)

$s_{t+1}^{v}=sGRU(s_{t}^{v},(v_{t}\oplus att_{t+1}^{v}))$ (8)

Where $D_{s}$ is the size of all the state vectors $s_{t+1}^{t}$,
$s_{t+1}^{a}$, and $s_{t+1}^{v}$.$D_{t},D_{a},D_{v}$ are the sizes of
utterance representations of text, audio, and video respectively.$\oplus$
represents concatenation operation, and all GRU weights shapes are such that
they produce the expected shape of outputs taking the given shape of inputs.

The intended purpose of using this as the input to $sGRU^{t,a,v}$ is to model
the dependency of the speaker state on the context of the conversation as
understood by the utterances until that point, along with the utterance
representation at that point. The output of the $sGRU$ for modality $m$ and
timestamp $t$ serves as an encoding of the speaker state as conveyed by
modality $m$, at time $t$.

####  3.2.3 Emotion GRU ($eGRU$)

The emotion GRU serves as the decoder for the encoding produced by the state
GRU. The emotion GRU uses the previous timestamp $eGRU$ output, and the
encoding provided by $sGRU$ to produce an emotion or sentiment representation
which is further used by the pairwise attention mechanism to be able to
produce the relevant output for prediction. At timestamp $(t+1)$ the emotion
vectors are updated as:

$e_{t+1}^{t}=eGRU(e_{t}^{t},s_{t+1}^{t})$ (9)

$e_{t+1}^{a}=eGRU(e_{t}^{a},s_{t+1}^{a})$ (10)

$e_{t+1}^{v}=eGRU(e_{t}^{v},s_{t+1}^{v})$ (11)

Where $D_{e}$ is the size of all the emotion vectors $e_{t+1}^{t}$,
$e_{t+1}^{a}$, and $e_{t+1}^{v}$.$D_{t},D_{a},andD_{v}$ are the sizes of
utterance representations of text, audio, and video respectively.$D_{e}$ is
the size of the state vectors $s_{t+1}^{t}$, $s_{t+1}^{a}$, and $s_{t+1}^{v}$;
and all GRU weights shapes are such that they produce the expected shape of
outputs taking the given shape of inputs.

The emotion GRU acts as a decoder to the encoding produced by the associated
state GRU, producing a vector which can be used for both sentiment and emotion
prediction.

####  3.2.4 Pairwise Attention Mechanism

The emotion GRU for each timestamp will produce an $m$ number of vectors
(where $m$ is the number of modalities available). Pairwise attention is then
used over these $m$ vectors to produce the final prediction output. In
particular pairwise attention is calculated over the following pairs in our
case – $(e^{v},e^{t}),(e^{t},e^{a})$, and $(e^{a},e^{v})$. Pairwise attention
for pair $(e^{v},e^{t})$ would be calculated as follows:

Figure 3: Pairwise attention mechanism used as the fusion mechanism followed
by the final prediction layer

$B_{1}=e^{v}.(e^{t})^{T},B_{2}=e^{t}.(e^{v})^{T}$ (12)
$N_{1}=softmax(B_{1}),N_{2}=softmax(B_{2})$ (13)
$O_{1}=N_{1}.e^{t},O_{2}=N_{2}.e^{v}$ (14) $A_{1}=O_{1}\odot
e^{v},A_{2}=O_{2}\odot e^{t}$ (15) $pairwise(e^{v},e^{t})=A_{1}\oplus A_{2}$
(16)

Where $B_{1},B_{2}\in\mathbb{R}^{D_{e}\times D_{e}}$;
$N_{1},N_{2}\in\mathbb{R}^{D_{e}\times D_{e}}$;
$A_{1},A_{2}\in\mathbb{R}^{D_{e}\times D_{e}}$; and
$pairwise(e^{v},e^{t})\in\mathbb{R}^{D_{e}\times 2D_{e}}$; $\odot$ represents
element-wise product; and $\oplus$ represents concatenation.

A complete analysis on the pairwise attention mechanism has been done by
Ghosal et al. (2018), where the role of each one of the intermediate variables
has been described. These equations (12, 13, 14, 15, 16) calculate
$\prescript{m\mkern-0.5mu}{}{C}_{2}$ pairwise fusion representations, which
are further concatenated to make the final prediction as described below.

####  3.2.5 Final Predictions

The prediction layer varies based on whether a sentiment or emotion prediction
is expected. For sentiment prediction first all three pairs of pairwise
attention i.e. $pairwise(e^{v},e^{t})$, $pairwise(e^{a},e^{t})$, and
$pairwise(e^{v},e^{a})$ at that timestamp are concatenated along with the
emotion GRU outputs at that timestamp ($e_{t}^{t}$, $e_{t}^{a}$, and
$e_{t}^{v}$) and the concatenated layer is passed through a fully connected
layer followed by a $softmax$ or $tanh$ layer based on the nature of the
expected prediction. For sentiment prediction between -1 and +1 at timestamp t
the output layer would equate as follows:

$pw=pw(e^{v},e^{t})\oplus pw(e^{a},e^{t})\oplus pw(e^{v},e^{a})$ (17)
$L_{t}=pw\oplus e_{t}^{t}\oplus e_{t}^{a}\oplus e_{t}^{v}$ (18)
$pred_{sentiment(t)}=tanh({W_{L}}{L_{t}})$ (19)

Where $pairwise(e^{v},e^{t})$ has been represented as $pw(e^{v},e^{t})$; and
$W_{L}\in\mathbb{R}^{9D_{e}\times 1}$.

For emotion prediction we use a fully connected layer along with a final
$softmax$ layer to calculate 6 emotion class probabilities from $L_{t}$.

$l_{t}=ReLU({W_{l}}{L_{t}}+{b_{l}})$ (20)
$P_{t}=softmax({W_{smax}}{l_{t}}+b_{smax})$ (21)
$pred_{emotion(t)}=argmax({P_{t}})$ (22)

Where $W_{l}\in\mathbb{R}^{D_{l}\times 9D_{e}}$;
$b_{l}=\in\mathbb{R}^{D_{l}};{W_{smax}}\in\mathbb{R}^{c\times
D_{l}};b_{smax}\in\mathbb{R}^{c}$ and $P_{t}\in\mathbb{R}^{c}$

####  3.2.6 Training

Fairly standard practices have been employed for the training of the model.
Categorical cross-entropy has been used along with L2-regularization as the
loss function during training for emotion prediction, to maximize likelihood
over each of the classes.

Mean Square Error (MSE) along with L2 regularization has been employed as loss
function during training for sentiment regression. The usage of a saturating
output layer and a loss function that does not undo the saturation, leads to
the model to stop training when it makes extreme predictions (close to -1 or
+1) due to very small gradients. Using initialization strategies that start at
smaller model weights, mini-batch gradient descent-based Adam Kingma and Ba
(2014) optimizer, and using L2 regularization is used to avoid this failure
mode.

##  4 Experiments, Datasets, and Results

Metric | A2 | F1
---|---|---
Text + Audio |  |
BC-LSTM | 79.30 | -
MMMU-BA | 80.58 | -
DialogueRNN | 78.81 | 79.12
Multilogue-net | 80.12 | 78.84
Video + Audio |  |
BC-LSTM | 62.10 | -
MMMU-BA | 65.16 | -
DialogueRNN | 63.22 | 60.14
Multilogue-net | 69.55 | 63.40
Text + Video |  |
BC-LSTM | 80.20 | -
MMMU-BA | 81.51 | -
DialogueRNN | 79.88 | 79.10
Multilogue-net | 80.66 | 79.62
Text + Audio + Video |  |
BC-LSTM | 80.30 | -
MMMU-BA | 82.31 | -
DialogueRNN | 79.80 | 79.48
Multilogue-net | 81.19 | 80.10
Table 1:  Multilogue-Net performance on CMU-MOSI in comparison with the
current and previous state-of-the-art on the dataset. A2 indicating accuracy
with 2 classes, and F1 indicating F1 score .

###  4.1 Datasets

We evaluate our model using two benchmark datasets - CMU Multi-modal Opinion-
level Sentiment Intensity (CMU-MOSI) Zadeh et al. (2016a) and the recently
published CMU Multi-modal Opinion Sentiment and Emotion Intensity (CMU-MOSEI)
dataset Zadeh et al. (2018b).

####  4.1.1 CMU-MOSI

CMU-MOSI dataset consists of 93 videos spanning over 2199 utterances. Each
utterance has a sentiment label associated with it. It has 52, 10 & 31 videos
in training, validation & test set accounting for 1151, 296 & 752 utterances.
CMU-MOSEI has 3229 videos with 22676 utterances from more than 1000 online
YouTube speakers. The training, validation & test set consist of 16216, 1835 &
4625 utterances, respectively. Each utterance in CMU-MOSI dataset has been
annotated as either positive or negative.

####  4.1.2 CMU-MOSEI

In CMU-MOSEI dataset labels are in a continuous range of -3 to +3 and are
accompanied by an emotion label being one of six emotions. However, in this
work we also project the instances of CMU-MOSEI in a two-class classification
setup with values $\geq$ 0 signifies positive sentiments and values $<$ 0
signify negative sentiments. We have called this A2 accuracy (accuracy with 2
classes). Along with this we have also shown results for continuous range
prediction between -3 and +3, and emotion prediction with the 6 emotion labels
for each utterance in CMU- MOSEI. We have used A2 as a metric to be consistent
with the previous published works on CMU-MOSEI dataset (Ghosal et al., 2018;
Zadeh et al., 2018b). CMU-MOSEI has further been used for other comprehensive
experiments due to its large sizer and easier feature extraction

###  4.2 Uni-modal Feature Extraction

Metric | A2 | F1 | MAE | r
---|---|---|---|---
T + A |  |  |  |
MMMU-BA | 79.74 | - | - | -
DialogueRNN | 79.80 | 78.32 | - | -
Multilogue-net | 80.18 | 79.88 | - | -
V + A |  |  |  |
MMMU-BA | 76.66 | - | - | -
DialogueRNN | 73.90 | 73.92 | - | -
Multilogue-net | 75.16 | 74.04 | - | -
V + T |  |  |  |
MMMU-BA | 79.40 | - | - | -
DialogueRNN | 78.90 | 78.12 | - | -
Multilogue-net | 80.06 | 79.84 | - | -
T + A + V |  |  |  |
Graph-MFN | 76.90 | 77.00 | 0.71 | 0.54
MMMU-BA | 79.80 | - | - | -
DialogueRNN | 79.98 | 79.82 | 0.69 | 0.42
Multilogue-net | 82.10 | 80.01 | 0.59 | 0.50
Table 2:  Multilogue-Net performance on CMU-MOSEI Sentiment Labels compared to
previous state-of-the-art models on regression and accuracy Metrics. All
metrics apart from MAE represents higher values for better results, MAE
represents lower values for better results.

####  4.2.1 CMU-MOSEI

We use the CMU-Multi-modal Data SDK Zadeh et al. (2018b) for feature
extraction. For MOSEI dataset, sentiment label-level features were provided
where text features used were GloVe embeddings Pennington et al. (2014),
visual features extracted by Facet Stöckli et al. (2017) & acoustic features
by OpenSMILE Eyben et al. (2010). Thereafter, we compute the average of
sentiment label-level features in an utterance to obtain the utterance-level
features. For each sentiment label-level feature, the dimension of the feature
vector is set to 300 (text), 35 (visual) & 384 (acoustic).

####  4.2.2 CMU-MOSI

In contrast, for MOSI dataset we use utterance level features provided in
Poria et al. (2017). These utterance-level features represent the outputs of a
convolutional neural network Karpathy et al. (2014), 3D convolutional neural
network Ji et al. (2010) & openSMILE Eyben et al. (2010) for text, visual &
acoustic modalities, respectively. Dimensions of utterance-level features are
100, 100 & 73 for text, visual & acoustic, respectively.

###  4.3 Experiments

We evaluate our proposed approach on CMU-MOSI (test-set) on accuracy and F1
score, and CMU-MOSEI (dev-set) on accuracy, F1 score, mean absolute error
($MAE$), pearson score ($r$), and accuracy’s on the emotion labels. Due to the
lack of speaker information in CMU-MOSI we were not able to use the CMU-Multi-
modal Data SDK for sentiment label extraction, to be able to evaluate our
approach on CMU-MOSI on mean absolute error and Pearson score.

MOSEI Emotions (Text + Video + Audio)
---
Emotion | Anger | Disgust | Fear | Happy | Sad | Surprise
Metric | WA | F1 | WA | F1 | WA | F1 | WA | F1 | WA | F1 | WA | F1
Graph-MFN | 62.6 | 72.8 | 69.1 | 76.6 | 62.0 | 89.9 | 66.3 | 66.3 | 60.4 | 66.9 | 53.7 | 85.5
Multilogue-Net | 83.1 | 80.9 | 90.3 | 87.3 | 89.7 | 87.0 | 70.0 | 68.4 | 76.1 | 74.5 | 87.4 | 84.0
Table 3:  Multilogue-Net performance on MOSEI Emotion Labels compared with
that of Graph-MFN on weighted accuracy and F1 score. MOSEI Emotion label
results were presented by only one model, and comprehensive results have not
been published for the same.

Results have also been reported for usage of two of the three available
modalities. Uni-modal performance has not been reported as the focus of the
paper is the effective usage of multi-modal data. In a uni-modal setting the
model would not be using the fusion mechanism and the output would be
equivalent to having a few dense layers after the emotion GRU to directly
output the final prediction. F1 scores have not been mentioned by most
previous models being used for comparison, but have been reported for
Multilogue-Net for additional comparison to any future models using CMU-MOSI
dataset.

Table 1 shows the performance of Multilogue-Net on CMU-MOSI dataset, comparing
to the current state of the art Ghosal et al. (2018), previous state-of-the-
art Poria et al. (2017), and DialogueRNN Majumder et al. (2018) (Multi-modal
performance of DialogueRNN has not been reported by Majumder et al. (2018),
and we have run these experiments additionally for a better comparative study,
where concatenating the input representations has been used as a fusion
mechanism). Our model consistently outperforms the previous state-of-the-art
but performs better only on one of the subsets of the modalities when compared
to the current state-of-the-art.

In comparison to MMMU-BA our model also lacks in Multi-modal performance. We
theorize that the model performance is lacking because of the low number of
training examples (CMU-MOSI consists only of 93 conversations out of which 62
were used for training), in contrast to our model which has a high capacity
(Relative to models being compared with). Since Multilogue-Net learns a lot of
intermediate representations in order to make a prediction, it would need a
larger dataset with more variability to be able to learn meaningful
representations. The proposition that performance lacks due to a lack of
training examples is backed by the results on CMU-MOSEI (demonstrated in a
comparative setting in Table 2 and 3) where the model consistently outperforms
the current state-of-the-art on most metrics.

On CMU-MOSEI, our model seems to perform very consistently on both sentiment
and emotion labels. The model outperforms the current state of the art on all
but one metric (both classification and accuracy) on sentiment labels in the
tri-modal setting. Multilogue-Net also outperforms the current state of the
art on the emotion labels by a considerable margin (This is also attributed to
the fact that not a lot of models have presented results on these labels).

Similar observations are made in both datasets, where the tri-modal metrics
show the best performance, and audio + video show the worst relative
performance (suggesting the importance of text in a multi-modal setting).
Textual information seems to be the guiding factor for multi-modal
performance, with video and audio features simply acting as a push to the uni-
modal performance on text.

We theorize that the performance of Multilogue-Net is majorly attributed to
its increased capacity as compared to previous models. Effective usage of this
increased capacity, using representations inspired from a basic understanding
of conversation, along with a larger dataset for training have been key in
achieving the improved results.

##  5 Ablation Studies and Analysis

Until now, some architectural considerations, such as the use of $eGRU$ and
the fusion mechanism, have been briefly explained but not empirically
justified. This section aims to get empirical evidence regarding the
effectiveness of these modules. Since our model completely hinges around the
usage of the context and state GRU’s, our ablation studies and analysis have
focused on the fusion mechanism and emotion GRU ($eGRU$) only.

###  5.1 Fusion Mechanism

The effectiveness of the fusion mechanism can be very easily examined by
observing the results of the model on both tasks $-$ Sentiment Regression and
Emotion Recognition, with and without the fusion mechanism. Table 4 shows
these results on CMU-MOSEI modality subsets.

Fusion Mechanism | A2 | MAE
---|---|---
Text + Audio |  |
without | 75.78 | -
with | 80.18 | -
Video + Audio |  |
without | 75.66 | -
with | 75.16 | -
Text + Video |  |
without | 76.80 | -
with | 80.06 | -
Text + Audio + Video |  |
without | 79.80 | 0.66
with | 82.10 | 0.59
Table 4:  Multilogue-Net performance on CMU-MOSEI with and without the fusion
mechanism - for ’without’ fusion we have concatenated all the representations
and directly passed them to the prediction layer.

The bi-modal results in table 4 involve evaluating the pairwise attention
module only once (Since there is only one pair available), directly followed
by the prediction layer. The tri-modal case on the other hand involves
evaluating the pairwise attention module thrice (Once for each pair). In
general, the number of times this module will have to be evaluated for $m$
modalities is $\prescript{m\mkern-0.5mu}{}{C}_{2}$, which raises a fair
concern regarding the trade-off between the additional computational cost and
performance.

We empirically observe that the additional computational cost can be
considered negligible in context of the increased performance, largely
attributing to the non-parametric nature of the fusion mechanism and the
relatively small number of additional parameters in the prediction layer
($6D_{e}$ for the sentiment regression; $36D_{e}$ for emotion recognition).

The fusion mechanism seems to clearly be beneficial in all of the reported
cases apart from video + audio, implying that the fusion mechanism is useful
only in the cases the text representation is used. This further strengthens
our claim that the text representation guides tri-modal performance.

###  5.2 Emotion GRU ($eGRU$)

Unlike as done with the fusion mechanism, the effectiveness of the $eGRU$
cannot be examined by evaluating metrics with and without it. Removing the
Emotion GRU would clearly be detrimental to the results, and would not convey
the intention of having it.

The primary intention of having the $eGRU$ can be considered to be maintaining
consistency between tasks. To better understand what this means table 5
quantitatively demonstrates this effect. The model was trained separately for
Emotion Detection and Sentiment Regression tasks. After both the models were
trained satisfactorily, a particular sample from the test set (test sample 6)
was inferred on. We then retrieved the intermediate text representations
($e_{4}^{t}$, $c_{4}^{t}$, and $s_{4}^{t}$; superscript $t$ indicating text
modality) at a particular timestamp ($t=4$) for both models on that sample.
The Euclidean Distance between these two sets of representations (one for each
task) was evaluated and have been shown in table 5, where we can clearly
observe that the euclidean distance between the emotion representations is
much larger as compared to the state and context representations.

This shows that for both tasks, interlocutor state and context representations
are relatively similar to each other, whereas the emotion state representation
is more varied and task dependant. This not only allows us to use the same
$cGRU$ and $sGRU$ weights across tasks, but would also allow us to train for
multiple tasks in parallel using a different $eGRU$ for each task - giving us
consistent and accurate predictions across multiple tasks. Analysis of such a
network, and whether training for multiple tasks in parallel aids one another,
has not been covered in this paper and is left to our future work.

Representation | Euclidean Distance
---|---
Sample 6 with $t=4$ |
$s_{4}^{t}$ | 4.6 units
$c_{4}^{t}$ | 6.1 units
$e_{4}^{t}$ | 26.4 units
Table 5:  Euclidean Distance between the same representations for Sentiment
Regression as compared to Emotion Detection. (Distances have been converted to
units for convenience and easier comparison)

##  6 Conclusion

In this paper, we have presented an RNN architecture for multi-modal sentiment
analysis and emotion detection in conversation. In contrast to the current
state-of-the-art models, our model focuses on effectively capturing the
context of a conversation and treats each modality independently, taking into
account the information a particular modality is capable of holding. Our model
consistently performs well on benchmark datasets such as CMU-MOSI and CMU-
MOSEI in any multi-modal setting.

The model can be further extended to have better feature extractors, and
increase both the number of modalities and the number of participants in the
conversation. Due to the lack of availability of datasets consisting of these
extensions with emotion or sentiment labels, we have left this to our future
work.

## References

  * Chen et al. (2017) Minghai Chen, Sen Wang, Paul Pu Liang, Tadas Baltrušaitis, Amir Zadeh, and Louis-Philippe Morency. 2017.  Multimodal sentiment analysis with word-level fusion and reinforcement learning.  In _Proceedings of the 19th ACM International Conference on Multimodal Interaction_ , pages 163–171.
  * Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Y. Bengio. 2014.  Empirical evaluation of gated recurrent neural networks on sequence modeling.
  * Datcu and Rothkrantz (2012) Dragos Datcu and Léon Rothkrantz. 2012.  Semantic audiovisual data fusion for automatic emotion recognition.
  * Ekman et al. (1992) P. Ekman, Edmund Rolls, David Perrett, and H. Ellis. 1992.  Facial expressions of emotion: An old controversy and new findings: Discussion.  _Royal Society of London Philosophical Transactions Series B_ , 335:69–.
  * Eyben et al. (2010) Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010.  opensmile – the munich versatile and fast open-source audio feature extractor.  pages 1459–1462.
  * Ghosal et al. (2018) Deepanway Ghosal, Md Shad Akhtar, Dushyant Chauhan, Soujanya Poria, and Pushpak Bhattacharyya Asif Ekbal. 2018.  Contextual inter-modal attention for multi-modal sentiment analysis.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3454–3466.
  * Hazarika et al. (2018) Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philippe Morency, and Roger Zimmermann. 2018.  Conversational memory network for emotion recognition in dyadic dialogue videos.  volume 2018, pages 2122–2132.
  * Ji et al. (2010) Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 2010.  3d convolutional neural networks for human action recognition.  volume 35, pages 495–502.
  * Karpathy et al. (2014) Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. 2014.  Large-scale video classification with convolutional neural networks.  pages 1725–1732.
  * Kingma and Ba (2014) Diederik Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _International Conference on Learning Representations_.
  * Majumder et al. (2018) Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. 2018.  Dialoguernn: An attentive rnn for emotion detection in conversations.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christoper Manning. 2014.  Glove: Global vectors for word representation.  volume 14, pages 1532–1543.
  * Picard (2010) Rosalind Picard. 2010.  Affective computing: From laughter to ieee.  _IEEE Transactions on Affective Computing_ , 1:11–17.
  * Poria et al. (2017) Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir Zadeh, and Louis-Philippe Morency. 2017.  Context-dependent sentiment analysis in user-generated videos.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_ , pages 873–883.
  * Poria et al. (2016) Soujanya Poria, Iti Chaturvedi, Erik Cambria, and Amir Hussain. 2016.  Convolutional mkl based multimodal emotion recognition and sentiment analysis.  In _IEEE 16th International Conference on Data Mining (ICDM)_ , pages 439–448.
  * Poria et al. (2019) Soujanya Poria, Navonil Majumder, Rada Mihalcea, and Eduard Hovy. 2019.  Emotion recognition in conversation: Research challenges, datasets, and recent advances.  _arXiv:1905.02947_.
  * Stöckli et al. (2017) Sabrina Stöckli, Michael Schulte-Mecklenbeck, Stefan Borer, and Andrea Samson. 2017\.  Facial expression analysis with affdex and facet: A validation study.  _Behavior Research Methods_ , 50.
  * Zadeh et al. (2017) Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017.  Tensor fusion network for multimodal sentiment analysis.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1103–1114.
  * Zadeh et al. (2018a) Amir Zadeh, Paul Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018a.  Memory fusion network for multi-view sequential learning.
  * Zadeh et al. (2018b) Amir Zadeh, Paul Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018b.  Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph.  pages 2236–2246.
  * Zadeh et al. (2018c) Amir Zadeh, Paul Liang, Soujanya Poria, Prateek Vij, Erik Cambria, and Louis-Philippe Morency. 2018c.  Multi-attention recurrent network for human communication comprehension.  _Proceedings of the 2018 AAAI Conference on Artificial Intelligence_ , 2018.
  * Zadeh et al. (2016a) Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016a.  Mosi: Multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos.
  * Zadeh et al. (2016b) Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016b.  Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages.  _Intelligent Systems, IEEE_ , pages 82–88.


# Constraints on charged black holes from merger-ringdown signals in GWTC-3
and prospects for the Einstein Telescope

Hua-Peng Gu  Hai-Tian Wang<EMAIL_ADDRESS>Department of Astronomy, School
of Physics, Peking University, Beijing 100871, China  Kavli Institute for
Astronomy and Astrophysics, Peking University, Beijing 100871, China  Lijing
Shao<EMAIL_ADDRESS>Kavli Institute for Astronomy and Astrophysics, Peking
University, Beijing 100871, China  National Astronomical Observatories,
Chinese Academy of Sciences, Beijing 100012, China

###### Abstract

Whether astrophysical black holes (BHs) can have charge is a question to be
addressed by observations. In the era of gravitational wave (GW) astronomy,
one can constrain the charge of a merged BH remnant using the merger-ringdown
signal of the GW data. Extending earlier studies, we analyze five GW events in
GWTC-3, assuming Kerr-Newman BHs. Our results show no strong evidence for a
charged BH, and give a limit on the charge-to-mass-ratio $Q<0.37$ at $90\%$
credible level (CL). Due to the charge-spin degeneracy in the waveform and the
limited signal-to-noise ratios (SNRs), it is challenging for LIGO/Virgo/KAGRA
observations to provide better constraints. We further simulate data for the
Einstein Telescope (ET), where SNRs can be as large as $\sim 270$ in the
ringdown signal. These simulated events allow us to consider the 220, 221, and
330 ringdown modes altogether, which can help break the charge-spin
degeneracy. The analysis of our simulation shows that ET can improve the
constraints on the charge-to-mass-ratio to $Q\lesssim 0.2$ at $90\%$ CL with
ringdown signals.

##  I Introduction

The existence of black holes (BHs), as predicted by Einstein’s General
Relativity (GR), has been substantiated through a series of observations
(Remillard and McClintock, 2006; Akiyama  _et al._ , 2019). These astronomical
observations not only empirically elucidate the physical attributes of BHs but
also serve as a robust platform for validating different BH models and
theories of gravity. With the groundbreaking detection of the gravitational
wave (GW) event GW150914 by the LIGO/Virgo Collaboration Abbott  _et al._
(2016a), GW astronomy has emerged as a potent tool for probing the properties
of BHs. Recent observations by the LIGO/Virgo/KAGRA (LVK) Collaboration have
achieved signal-to-noise ratios (SNRs) as high as $26$ for binary BH (BBH)
merger events Abbott  _et al._ (2016b). Looking ahead, next-generation GW
observatories like the Einstein Telescope (ET) Punturo  _et al._ (2010) are
anticipated to reach SNRs exceeding $200$ for BBH mergers, thereby enabling
even more precise measurements of BH properties Maggiore  _et al._ (2020);
Sathyaprakash  _et al._ (2019); Kalogera  _et al._ (2021).

In the context of GR, the most general solution for a stationary,
asymptotically flat BH is the Kerr-Newman (KN) BH, characterized by mass,
spin, and charge Kerr (1963); Newman  _et al._ (1965). However, astrophysical
BHs are generally considered to be Kerr-like, as they are expected to be
electrically neutral due to de-charging effects from their surrounding
environment. To illustrate this, consider a toy model of a BH with mass $M$
and charge $+Q_{M}$, surrounded by particles of mass $m$ and charge $\pm
q_{m}$. If the BH charge is sufficiently large such that, with $G=1$ in
Gaussian units,

$Q_{M}q_{m}>Mm,$ (1)

the electromagnetic force will dominate over gravity, causing the BH to
attract only negatively charged particles. Consequently, the BH will become
neutralized on a timescale far shorter than the timescale associated with GWs
(Gibbons, 1975; Zajaček and Tursunov, 2019; Gong  _et al._ , 2019). Moreover,
even in the vacuum, processes like vacuum polarization and pair production
contribute to the neutralization of the BH Blandford and Znajek (1977). The
upper limit for the charge-to-mass ratio of a BH has been estimated to be

${Q_{M}}/{M}\lesssim 10^{-5}\frac{M}{M_{\odot}},$ (2)

thereby reinforcing the Kerr hypothesis for BHs Cardoso  _et al._ (2016).

Empirical evidence lends credence to the theoretical premises discussed above.
For instance, observations of the bremsstrahlung surface brightness decay in
the Galactic central BH, Sgr A*, suggest that its charge-to-mass ratio is less
than $10^{-18}$ (Zajaček  _et al._ , 2018, 2019). Since the discharge
mechanisms mentioned above apply for all BHs regardless of their mass, all the
astrophysical BHs should have negligible charge. It is worth noting that the
term “charge” in this context refers exclusively to the electromagnetic
charge. Before we only talked about the electric charge, while the BHs may
carry magnetic charge by primordial magnetic monopoles (Preskill, 1984). The
electric and magnetic charges are indistinguishable for a perturbed BH in
vacuum (Bozzola and Paschalidis, 2021a; Pereñiguez, 2023; Dyson and
Pereñiguez, 2023). However, theories extending beyond the Standard Model or GR
permit BHs to carry different kinds of charges (see e.g. Ref. Xu  _et al._
(2023)). The foregoing analysis, for example, relies on the high charge-to-
mass ratio of electrons. In contrast, the minicharged dark matter model,
predicated on an additional hidden $U(1)$ gauge field, could give rise to
particles with lower charge-to-mass ratios, thereby making charged BHs less
constrained (Cardoso  _et al._ , 2016; Stebbins and Krnjaic, 2019). Moreover,
certain modified gravity theories also admit the possibility of BHs carrying
specific kinds of charges, which would influence the metric and yield
observable effects. As such, direct observations of the metric offer a model-
independent avenue for testing these theories.

Nonetheless, conventional methods for constraining the BH charge via metric
measurements in its vicinity tend to yield rather weak limits. For instance,
the Event Horizon Telescope’s recent imaging of Sgr A* has established an
upper bound of $0.72$ for the charge-to-mass ratio at a $68\%$ credible level
(CL) Ghosh and Afrin (2023); Akiyama  _et al._ (2022a, b). More stringent
constraints can be garnered from GW detectors, particularly in the context of
BBH mergers. In such events, the GW signal during both the inspiral and
ringdown phases serves as a probe of BH properties pre- and post-merger,
respectively. Our analysis is primarily concerned with the latter. At the
ringdown stage, the GW signal manifests as a composite of damped sinusoids,
known as quasi-normal modes (QNMs) Ferrari and Gualtieri (2008). Both the
oscillation frequency and damping time of these QNMs are dependent on the
attributes of the resultant merged BH, including its charge. By extracting QNM
frequencies from the ringdown data, one can deduce the parameters
characterizing the QNM spectrum—a technique commonly referred to as BH
spectroscopy (Dreyer  _et al._ , 2004; Berti  _et al._ , 2016).

There are two primary challenges associated with constraining the charge of
the resultant BH using GW detectors. The first challenge pertains to the
computation of QNMs for KN BHs. The interplay between electromagnetic and
gravitational perturbations complicates the analytical derivation of QNM
frequencies for KN BHs (Kokkotas, 1993; Berti and Kokkotas, 2005). Initial
perturbative solutions for KN BH QNMs were first obtained by Pani  _et al._
(2013a, b) in the slow-rotation limit, by Mark  _et al._ (2015) under the
weakly-charged condition, and by Zimmerman and Mark (2016) for extremal KN
BHs. Wang  _et al._ (2021a) later employed geodesic correspondence to
approximate KN QNMs in the eikonal limit. This approximation constrains the
charge-to-mass ratio of the BH remnant in GW150914 to be less than $0.38$ at
$90\%$ CL. While this result made an ansatz for QNMs at high values of the
charge. Moreover, numerical solutions, especially for certain dominant QNMs
have been calculated by Dias  _et al._ (2015, 2022a, 2022b). For the inspiral
stage, however, a complete KN solution remains elusive. Existing studies have
primarily focused on numerical simulations at some certain limits, such as for
low charge-to-mass ratios (Wang  _et al._ , 2021b; Bozzola and Paschalidis,
2021a, b; Gupta  _et al._ , 2021; Benavides-Gallego and Han, 2023).
Consequently, a comprehensive inspiral-merger-ringdown (IMR) signal analysis
for KN BHs is currently unfeasible García-Quirós  _et al._ (2020). The second
challenge arises from the limited sensitivity of existing GW detectors. Even
for high-confidence events like GW150914, the SNRs for postmerger data barely
exceed $10$ Carullo  _et al._ (2022). This limitation hampers our ability to
extract higher-order QNMs from the ringdown data. When only dominant modes are
considered, the charge and spin parameters become strongly degenerate,
rendering precise constraints on the BH remnant’s charge impracticable.

The GW Transient Catalog (GWTC) provides a compendium of GW events observed by
the LVK Collaboration (Abbott  _et al._ , 2019a, 2021a, 2021b). This dataset
has been extensively employed in various BH analyses, including tests of GR
and its modifications. Prior analyses have rigorously measured parameters such
as the mass and spin of the remnant BHs, providing a basis for validating
subsequent model-dependent studies (Abbott  _et al._ , 2019b, 2021c, 2021d).
In testing GR, the working model is typically Kerr-like, and to date, no
significant deviations from GR have been observed. Carullo  _et al._ (2022)
leveraged the merger-ringdown signals from the GWTC-2 dataset Abbott  _et al._
(2021a) to constrain the charge of remnant BHs. Utilizing numerical solutions
for dominant QNMs from Dias  _et al._ (2022a, b), they performed a KN BH
analysis and derived the charge-spin distribution for several high-confidence
GW events. A prior based on previous IMR analyses of Kerr BHs was also applied
to further constrain the remnant BH charge. Their most stringent constraint on
the charge-to-mass ratio is less than $0.33$ at $90\%$ CL for GW150914.
Building upon this work, we employ GWTC-3 data Abbott  _et al._ (2021b) to
scrutinize the charge of remnant BHs in five high-credibility GW events. Our
results show negative Bayes factors between the KN and Kerr model.
Consequently, our analysis does not strongly support the existence of charged
BH remnants. Due to the low SNRs in ringdown data, coupled with charge-spin
degeneracy, the error margins in our analysis are substantial even with the
improved noise levels in current LVK observations. Although our charge
constraints are substantially weaker than those derived from electromagnetic
observations, which can be on the order of $Q\sim 10^{-18}$, our approach is
complementary, and offers the advantage of being model-independent.

Moreover, to assess the potential for detecting BH charges with the future GW
detector ET, we conduct analyses using simulated GW data. Designed to achieve
a tenfold increase in sensitivity Punturo  _et al._ (2010), ET has not yet
been constructed, but simulated data analyses can offer valuable insights into
future constraints on remnant BH charges using merger-ringdown signals.
Enhanced detector sensitivity not only reduces the uncertainty in our results
but also enables us to include higher-order QNMs in the template, thereby
mitigating the charge-spin degeneracy. In this study, we generate merger-
ringdown waveforms based on the Kerr BH model and inject them into noise
simulated from ET’s designed power spectral density (PSD). Subsequent analyses
indicate that the constraint on BH charge could be tightened to $Q<0.2$ at the
$90\%$ CL.

The remainder of this paper is organized as follows. In Sec. II, we introduce
the Bayesian framework for constraining BH charge. Section III presents our
findings derived from GWTC-3 data, confirming the charge-spin degeneracy of
the remnant BH. In Sec. IV, we discuss results based on simulated ET ringdown
data, specifically addressing how the inclusion of higher multipole modes can
alleviate charge-spin degeneracy and yield more stringent constraints on BH
charge. Conclusions and discussions are provided in Sec. V.

##  II Method

In this section, we outline the waveform model and the GW data employed for
Bayesian inference.

During the ringdown phase, the GW signal emanates from the oscillations of the
remnant BH. In the context of a Kerr BH, the metric tensor field can be
decomposed into a background field and a perturbation term. This perturbation
term is a superposition of a set of eigenfunctions, leading to the solutions
known as QNMs (Teukolsky, 1973; Berti  _et al._ , 2009). Each QNM is
characterized by a damped sinusoid with a complex frequency,

$\tilde{\omega}_{\ell mn}\equiv\omega_{\ell mn}-i\gamma_{\ell
mn},\quad\gamma_{\ell mn}=\frac{1}{\tau_{\ell mn}},$ (3)

where $\omega_{\ell mn}$ represents the oscillating frequency, $\gamma_{\ell
mn}$ denotes the damping frequency, and $\tau_{\ell mn}$ is the damping time.
Each QNM is distinguished by three indices, including $\ell$ and $m$, which
arise from the angular eigenfunctions, and $n$, referred to as the overtone
number that associated with the damping time. Generally, higher overtone modes
exhibit more rapid damping Isi and Farr (2021).

In the case of a KN BH, electromagnetic perturbations come into play,
intertwining with gravitational perturbations. The specific calculations for
this are intricate and fall beyond the scope of this paper. As posited by the
no-hair theorem Misner  _et al._ (1973), the QNM frequencies are fully
determined by the mass $M$, dimensionless spin $a$, and charge-to-mass ratio
$Q$ of the remnant BH, expressed as

$\tilde{\omega}_{\ell mn}=\tilde{\omega}_{\ell mn}(M,a,Q).$ (4)

Here, we adopt the convention where $a$ represents the dimensionless spin and
$Q$ signifies the charge-to-mass ratio $q/M$. Without losing generality, we
assume that $a$ and $Q$ are non-negative. Both $a$ and $Q$ are constrained to
lie between $0$ and $1$, subject to a GR-based constraint

$a^{2}+Q^{2}\leq 1.$ (5)

For the precise form of Eq. (4), we rely on the numerical solutions provided
by Dias  _et al._ (2022a, b). To elaborate further, Carullo  _et al._ (2022)
used an analytical fit for the numerical solutions (Dias  _et al._ , 2022a, b)
and listed the fitting coefficients in their appendix. In this work we use
their analytical fitting results.

GW signal during the ringdown phase constitutes a superposition of various
possible QNMs. Typically, the $\ell=2,m=2,n=0$ mode (hereafter referred to as
the 220 mode) serves as the dominant contributor to the ringdown signal
(Abbott  _et al._ , 2016b; Cotesta  _et al._ , 2022; Carullo  _et al._ ,
2019). The 221 mode, characterized by a similar oscillating frequency but a
shorter damping time, has also been considered in some previous QNM fittings
for Kerr BHs (Isi  _et al._ , 2019; Cotesta  _et al._ , 2022). Additionally,
Capano  _et al._ (2021) reported evidence supporting the existence of the 330
mode. While this may be debatable since Siegel  _et al._ (2023) found no
evidence for that mode, but the 210 and 320 modes when including precessing
degrees of freedom. Since Dias  _et al._ (2022a, b) only provided the
numerical solutions for the 220, 221 and 330 modes, we remain considering 220,
221 and 330 modes in our KN analysis. The higher modes, despite their
theoretical underpinning Shi  _et al._ (2019), remain elusive in current LVK
observations due to limited detector sensitivity. Consequently, the template
we employ for QNM fitting is,

$\displaystyle h^{+}(t)-ih^{\times}(t)=$
$\displaystyle~{}A_{220}e^{-i(\tilde{\omega}_{220}(t-t_{0})+\varphi_{220})}{}_{-2}S_{220}(\iota,\varphi)$
(6)
$\displaystyle+A_{221}e^{-i(\tilde{\omega}_{221}(t-t_{0})+\varphi_{221})}{}_{-2}S_{221}(\iota,\varphi)$
$\displaystyle+A_{330}e^{-i(\tilde{\omega}_{330}(t-t_{0})+\varphi_{330})}{}_{-2}S_{330}(\iota,\varphi),$

where ${}_{-2}S_{\ell mn}$ represents the spheroidal harmonics Berti  _et al._
(2006), which depend on the spin polar angle $\iota$ and the spin azimuthal
angle $\varphi$, and $t_{0}$ is the reference start time for the postmerger.
The real amplitude $A_{\ell mn}$ and phase $\varphi_{\ell mn}$ of each mode
are treated as free parameters. In Eq. (6) we do not consider the contribution
of the counter-rotating modes with the negative-$m$ (see Refs. (Carullo  _et
al._ , 2022; Lim  _et al._ , 2019; Dhani, 2021; Dhani and Sathyaprakash,
2021)). In addition we assume a non-precessing symmetry. Moreover, the
orientation of the GW event must also be specified to convert the observed GW
data into $h^{+}$ and $h^{\times}$ components Isi and Farr (2021). In summary,
the parameters utilized for QNM fitting include,

$\\{M,a,Q,A_{\ell mn},\varphi_{\ell mn},\iota,\psi\\}.$ (7)

In this formulation, the spin azimuthal angle $\varphi$ is subsumed into the
phase $\varphi_{\ell mn}$. For our work based on the GWTC-3 data, we adopt the
values of $\iota$ and $t_{0}$ from Ref. (The LIGO Scientific Collaboration,
the Virgo Collaboration, the KAGRA Collaboration, LIGO-P2100456 (), 2021).

The prevailing model for GW data analysis is based on Kerr-like BHs, premised
on the expectation that BHs are electromagnetically neutral. Nonetheless, even
when focusing solely on ringdown data, introducing charge as an additional
parameter yields results comparable to those from the Kerr BH model. This
arises from the strong degeneracy between the spin and charge of the remnant
BH, allowing the effects of spin to be partially offset by an appropriate
charge component (as elaborated in Sec. III). Consequently, spin estimates
from prior analyses employing the Kerr BH model can be interpreted as a form
of “effective spin” if there is a nonvanishing charge. Employing a KN BH model
for data fitting allows us to derive charge distributions for the remnant,
thereby establishing upper limits on its charge. While the emergence of charge
in these KN fits is most likely an artifact of charge-spin degeneracy, the
constraints obtained are model-independent and serve to rule out scenarios
involving highly charged remnants.

Building upon the GWTC-2 dataset, which includes credible GW events up to the
O3a observing run of the LVK Collaboration, Carullo  _et al._ (2022) examined
the potential influence of charge in the merger-ringdown signals. Their
analysis revealed the presence of charge-spin degeneracy and also included a
“null test.” In this test, they assumed the validity of the Kerr hypothesis
and incorporated a Kerr BH prior for the final spin and mass, based on
previous IMR results. Utilizing the derived posterior distributions of charge,
an upper limit of $Q<0.33$ for GW150914 at the $90\%$ CL was obtained.

In the present study, we focus on five events cataloged in GWTC-3: GW191109,
GW191222, GW200129, GW200224, and GW200311. These events have been
specifically discussed in Ref. (Abbott  _et al._ , 2021d) due to their
credible ringdown signals. The parameters of their remnant BHs are well-
constrained relative to the prior, and the Bayesian evidence supports the
existence of a signal over mere Gaussian noise. Furthermore, the sky-frame
orientations of these events have been accurately determined. For these five
events, the LVK Collaboration (Abbott  _et al._ , 2021d) provided the outcomes
of Kerr BH analyses, which serve as the foundation for our subsequent
analyses.

Our analysis of KN BHs employs the pyRing software package, a time-domain
Bayesian inference tool designed for analyzing ringdown signals (Carullo  _et
al._ , 2023; Abbott  _et al._ , 2016b; Cotesta  _et al._ , 2022; Carullo  _et
al._ , 2019). The sampling method utilized is the CPNest algorithm Pozzo and
Veitch (2015). After multiple iterations, the random walk through the
parameter space converges to yield the posterior distributions of parameters.
For our KN BH analysis, we initially set uniform priors for $M$, $A_{\ell
mn}$, $\varphi_{\ell mn}$, and $\psi$ as indicated in Eq. (7). Specifically,
$M$ is restricted in the range $[0,200]\,M_{\odot}$, $A_{\ell mn}$ in the
range $[0,10]\times 10^{-21}$, and both $\varphi_{\ell mn}$ and $\psi$ in the
range $[0,2\pi]$. The parameters $a$ and $Q$ are uniformly distributed within
the quarter disc defined by $a^{2}+Q^{2}\leq 1$, $a\geq 0$, and $Q\geq 0$.

The posterior distribution of these parameters is then computed using Bayes’
theorem

$p(\vec{\theta}\mid d,\mathcal{H})=\frac{p(\vec{\theta}\mid\mathcal{H})\cdot
p(d\mid\vec{\theta},\mathcal{H})}{p(d\mid\mathcal{H})},$ (8)

where $\vec{\theta}$ represents the model parameters, $d$ is the observed
data, and $\mathcal{H}$ denotes the model. Here, $p(d\mid\mathcal{H})$ is the
evidence, calculated iteratively via CPNest Skilling (2006). This evidence is
instrumental for model comparison, expressed through the Bayes factor when
comparing model 1 and model 2,

$\mathcal{B}_{1}^{2}=\frac{p(d\mid\mathcal{H}_{2})}{p(d\mid\mathcal{H}_{1})}=\frac{\int
p(d\mid\vec{\theta_{2}},\mathcal{H}_{2})p(\vec{\theta_{2}}\mid\mathcal{H}_{2})\mathrm{d}\vec{\theta_{2}}}{\int
p(d\mid\vec{\theta_{1}},\mathcal{H}_{1})p(\vec{\theta_{1}}\mid\mathcal{H}_{1})\mathrm{d}\vec{\theta_{1}}}.$
(9)

For the GWTC-3 ringdown data, we incorporate both the 220 and 221 modes into
our model. Given that the 330 mode is considerably weaker and usually falls
below the noise level of LVK detectors, it is excluded from our Bayesian
inference. This agrees with the results in Ref. (Abbott  _et al._ , 2021d),
where they did not find preference for the 330 mode in the five GWTC-3 events
that are used in this work. Our analysis employs ringdown data sampled at a
rate of $16$ kHz, which surpasses the commonly used $4$ kHz rate. We utilize
$0.4$ s of data following the merger for autocorrelation function (ACF)
estimation to extract the signal, and the full $32$ s of data to estimate the
PSD.

##  III Results from GWTC-3

In this section, we delineate the outcomes of our KN BH analysis for the five
GW events selected from GWTC-3. A central aspect of our discussion will
revolve around the phenomenon of charge-spin degeneracy.

Differently from the approach taken by Carullo  _et al._ (2022), our analysis
employs a sampling rate of $16$ kHz. Figure 1 displays the charge-spin
distribution we obtained for the remnant BH of GW150914. The $90\%$ CL region
in our findings closely aligns with early work (Carullo  _et al._ , 2022).
Employing the same methodology, we establish an upper limit on the remnant BH
charge of $Q<0.35$ at $90\%$ CL. Further, in the $Q\rightarrow 0$ limit, the
final spin approximates $0.67$, aligning with IMR results based on the Kerr BH
model. The final mass, although not depicted in the figure, also concurs with
these Kerr-based IMR outcomes. Subsequent analyses are performed on the
selected events from GWTC-3.

Figure 1:  Charge-spin posterior distribution for GW150914. Different colors
signify varying CLs within the distribution. The $90\%$ CL and $50\%$ CL
regions are demarcated by white and yellow lines, respectively. The gray
region violates the constraint $a^{2}+Q^{2}\leq 1$. Note that the mass
parameter exhibits minimal correlation with the charge and is thus not
displayed.

Figure 2:  Same as Fig. 1, but for GW191222, GW200129, GW200224, GW200311 and
GW191109.

.

Figure 2 presents the charge-spin distributions of the remnant BHs for the
five events featured in GWTC-3. Among them, GW200129 and GW200224 yield more
stringent constraints on both charge and spin, while GW191222 and GW200311
offer weaker constraints due to their relatively low ringdown SNRs.
Intriguingly, the most stringent yet puzzling constraints emanate from
GW191109. Although there exists a strong correlation between charge and spin
for this event, the constraint on charge is surprisingly lax. No discernible
improvement in SNR accounts for this anomaly (see Table 1). The ringdown SNR
for GW191109 is $12.6$, comparable to that of GW150914. We will delve into
this particular case in further detail later.

To assess the validity of our KN BH model, we concurrently perform analyses
under the assumption of the Kerr hypothesis, setting the charge to zero.
Subsequently, we compute the Bayes factor between the KN and Kerr BH models as
outlined in Eq. (9). The results indicate a slight preference for the Kerr BH
model, as most of the log Bayes factors are negative (see Table 1). This
outcome aligns with similar findings from GWTC-2 events Carullo  _et al._
(2022), lending support to the prevailing Kerr hypothesis that remnant BHs are
uncharged.

The charge-spin distribution for GW191109 exhibits a “belt-like” shape, a
stark contrast to the more diffuse distributions typically observed. This
unique formation is a direct consequence of the charge-spin degeneracy,
wherein various points in the charge-spin parameter space yield
indistinguishable QNM frequencies, thus forming a belt-like distribution in
the results. While this phenomenon is also manifest in the other four events,
it remains less conspicuous due to the larger associated uncertainties.

Notably, our ringdown analysis for GW191109 reveals a discrepancy when
compared to previous IMR results in the Kerr BH model. The IMR result for
GW191109 gives $M_{f}=132.7^{+21.9}_{-13.8}M_{\odot}$ and
$a_{f}=0.60^{+0.22}_{-0.19}$. However, in the $Q\rightarrow 0$ limit, our
analysis yields $M_{f}=184^{+12}_{-12}M_{\odot}$ and
$a_{f}=0.85^{+0.05}_{-0.06}$. These findings align with the ringdown analysis
by the LVK Collaboration (Abbott  _et al._ , 2021d), where the Kerr BH model
was employed to fit the ringdown signal, resulting in
$M_{f}=179.0^{+23.7}_{-21.7}$ and $a_{f}=0.81^{+0.08}_{-0.14}$ at $90\%$ CL.
This discordance between the ringdown and inspiral signals points to an
inconsistency, an issue also noted in Ref. (Abbott  _et al._ , 2021d), where
they attributed it to the non-Gaussian noise. For the remaining four events,
our ringdown-derived results are consistent with earlier IMR analyses.

Figure 3:  QNM frequencies for the 220, 221, and 330 modes. The left panels
display the oscillating frequencies, while the right panels show the damping
frequencies. Different colors signify different frequency values. The gray
region violates the constraint $a^{2}+Q^{2}\leq 1$.

We now delve into the charge-spin degeneracy alluded to earlier, extending the
preliminary discussion by Carullo  _et al._ (2022). Figure 3 vividly
illustrates how the QNM frequencies vary with both the final spin $a$ and
charge $Q$, based on the analytical fit results from Carullo  _et al._ (2022).
Specifically, the dimensionless frequencies $M\omega$ and $M\tau^{-1}$ serve
as functions of the dimensionless spin $a$ and charge-to-mass ratio $Q$. As
depicted in Fig. 3, the oscillating and damping frequencies of the 220 mode
exhibit a strikingly similar dependence on $a$ and $Q$, giving rise to the
belt-like distribution in the $Q\mbox{-}a$ plane.

Table 1:  Constraints on BH charges, denoted by $Q_{\text{max}}$, for five GWTC-3 events and two GWTC-2 events, given at the $90\%$ CL for both Gaussian IMR and flat bounded IMR priors. Also listed are the logarithmic Bayes factors between the KN BH and Kerr BH models, denoted by $\ln\mathcal{B}_{\text{Kerr}}^{\text{KN}}$, for both the uniform-prior and IMR prior cases (null tests), as well as the SNRs for the ringdown data. Event |  $Q_{\rm max}$ (Gaussian prior) |  $Q_{\rm max}$(flat bounded prior) | $\ln\mathcal{B}_{\rm Kerr}^{\rm KN}$ |  $\ln\mathcal{B}_{\rm Kerr}^{\rm KN}$(null) | Ringdown SNR
---|---|---|---|---|---
GW191109 | $0.77$ | - | $-1.4$ | $-0.5$ | $12.6$
GW191222 | $0.50$ | $0.59$ | $0.3$ | $-0.4$ | $6.4$
GW200129 | $0.37$ | $0.45$ | $0.3$ | $-0.6$ | $13.0$
GW200224 | $0.37$ | $0.53$ | $-1.1$ | $-0.7$ | $10.7$
GW200311 | $0.47$ | $0.57$ | $-0.8$ | $-0.5$ | $7.9$
GW150914 | $0.37$ | $0.35$ | $-0.6$ | $-0.7$ | $12.6$
GW190521_07 | $0.40$ | $0.41$ | $-0.2$ | $-0.8$ | $9.6$

Introducing additional QNM modes into the fitting—such as the 221 and 330
modes—can alleviate this degeneracy. As Fig. 3 reveals, different damping
frequencies manifest distinct dependencies on $a$ and $Q$. The inclusion of
higher modes enriches the information available for $a$ and $Q$ estimation,
thereby yielding more precise constraints. However, due to the limited
sensitivity of current LVK observations, accurate extraction of these higher
modes remains challenging. In our analysis based on GWTC-3 data, we confined
ourselves to the 220 and 221 modes, resulting in pronounced degeneracy. Next-
generation GW detectors like the ET are expected to facilitate the inclusion
of higher modes in the models, thereby breaking the degeneracy more
effectively. Additionally, it is worth noting that this degeneracy is
intrinsic for small $Q$ values, as the frequencies have weak dependence on $Q$
when $Q$ is near zero (Dias  _et al._ , 2022a, b). This implies that BHs with
minimal charge may evade detection through this method, setting a lower bound
on the charge constraints.

The charge and spin are correlated in the above results. To obtain more
stringent constraints on the charge of the remnant BH, we can constrain the
spin first. Since our method is based on the Bayesian inference, we could
apply a more stringent prior on the spin. Since the electromagnetic
observations (Zajaček  _et al._ , 2018, 2019) support the Kerr hypothesis, a
tentative choice is to use the final spin result from IMR analysis, which is
based on the Kerr BH model. This procedure, i.e. using the Kerr prior to
obtain the posterior distribution of charge, is called the “null test” in
Carullo  _et al._ (2022), as this prior is based on the hypothesis that the
final BH does not possess charge. Among the obtained posterior distributions
of charge, all the events except GW191109 show a decreasing probability
density from $Q=0$, signifying that they are very likely to have negligible
charge. GW191109, however, has a very broad charge distribution. This is due
to the strong correlation of charge and spin, where they cannot be well
constrained at the same time. These results are shown in Fig. 4. The results
of GW150914 and GW190521_07 (i.e. GW190521_074359) are also presented for
comparison. Here we choose the form of the priors as Gaussian, instead of the
flat bounded prior used by Carullo  _et al._ (2022). We chose a Gaussian prior
because it matches the distribution of the parameters from IMR analysis more
accurately. Furthermore, the final spin distribution of GW191109 from Kerr
ringdown analysis falls outside the range of its IMR result, which makes it
unsuitable to adopt a flat bounded prior. We choose the value of the priors
according to the results of the LVK Collaboration (Abbott  _et al._ , 2021d).

We use the $90\%$ CL limit in the posterior charge distributions as the upper
bound on charge, for the selected events separately. The upper limits on
charge are listed in Table 1, where we also show the limits given by flat
bounded prior for comparison. For results of GWTC-3, GW200129 and GW200224
give the strongest constraint $Q<0.37$. This constraint is at the same level
as the GWTC-2 results from Carullo  _et al._ (2022). GW191222 and GW200311
have lower ringdown SNRs and can only give limits at about $Q<0.5$. GW191109
gives the worst constraint in our selected GWTC-3 events. One may argue that
this is due to the discrepancy between the IMR Kerr prior and KN results for
GW191109, i.e. we choose a prior smaller than its Kerr spin from ringdown
analysis, and the charge-spin degeneracy causes the nonzero value of charge
(the charge is fitted to compensate for the spin), thus leading to this broad
distribution. However, as will be shown in the following, this is not related
to the choice of prior.

Figure 4: Constraints on the final BH charge, according to the Gaussian prior of spin from Kerr IMR analysis. Different lines represent the posterior charge distributions of different events. Table 2: Fitting results for simulated ringdown signals with varying remnant charges. The columns represent the injected remnant charge $Q$, the logarithm of the Bayes factor between the KN and Kerr BH models $\ln\mathcal{B}_{\rm Kerr}^{\rm KN}$, the prior for the final mass and spin utilized in the null test at the $90\%$ confidence level,222In actual scenarios, the priors for mass and spin would be determined by IMR analysis. However, as we have not simulated the inspiral phase, we employ the Kerr BH fitting results for this ringdown signal as a surrogate prior. Though this approach lacks strict Bayesian rigor, it serves as a reasonable approximation for the $Q=0.2$ and $Q=0.4$ cases. and the posterior distribution of charge at the $1\mbox{-}\sigma$ level in the null test. Injected $Q$ | $\ln\mathcal{B}_{\rm Kerr}^{\rm KN}$ | Kerr mass ($M_{\odot}$) | Kerr spin | Posterior $Q$ (null)
---|---|---|---|---
0.0 | $-0.6$ | $68.7^{+0.5}_{-0.6}$ | $0.670^{+0.015}_{-0.015}$ | $0.09^{+0.05}_{-0.09}$
0.2 | 0.2 | $68.3^{+0.7}_{-0.7}$ | $0.682^{+0.017}_{-0.019}$ | $0.17^{+0.10}_{-0.12}$
0.4 | 0.7 | $67.3^{+0.5}_{-0.5}$ | $0.719^{+0.012}_{-0.012}$ | $0.24^{+0.13}_{-0.09}$

Constraints here are based on the Kerr BH hypothesis. To assess the validity
of this assumption, we perform a companion analysis using the Kerr BH model
and compare the results with those obtained using the KN BH model. In both
situations, we adopt the Kerr prior. Figure 5 presents the posterior
distributions of the remnant mass and spin for the five GW events under
consideration. For all events, except GW191109, the obtained remnant mass and
spin are visually identical for the two models. The spin in the Kerr BH model
may be slightly larger than that in the KN BH model, which can be attributed
to the charge-spin degeneracy. We also compute the evidence for the Kerr BH
model and calculate the Bayes factors for the null test. The log Bayes factors
are all negative in our results, suggesting that the Kerr model provides a
better fit to the data. These results are also summarized in Table. 1.

For GW191109, the spin obtained from the Kerr BH model is notably larger than
the spin obtained from the KN BH model. In both cases, we adopt the IMR Kerr
prior, implying that the observed difference is not a result of prior
selection but rather originates from the intrinsic differences between the
Kerr BH and KN BH models. This may simply be an effect of charge-spin
degeneracy. Furthermore, GW191109 is a unique event, whose results are very
likely influenced by its intrinsic noises. The logarithmic Bayes factor for
this event is $-0.5$, suggesting that the KN result is not convincing.
Consequently, we are inclined to attribute the discrepancy in the final spin
between the Kerr BH and KN BH models to the non-Gaussian noise, rather than
the presence of charge.

Figure 5: Distributions of final mass $M$ (in the unit of $M_{\odot}$) and
final spin $a$ for five GWTC-3 events, under the Kerr IMR prior. The blue and
red shadows represent the probability density distributions in the Kerr BH
model and in the KN BH model, respectively. The top and right panels display
the projected distributions of final mass and final spin.

##  IV ET simulation

The charge-spin degeneracy prevents us from giving stronger constraints on the
BH charge. To break the degeneracy, as illustrated in Sec. III, incorporating
higher QNMs might be a solution. However, the requirement to detect higher
modes places higher demands on the SNR of the ringdown signal. Therefore, in
this section, we analyze the simulated GW ringdown data, using the noise of
ET. ET is one of the next-generation GW observatories planned for construction
in the 2030s. Designed as an equilateral triangle with arms extending $10$ km,
ET aims for a tenfold enhancement relative to LVK detectors in sensitivity,
thereby paving the way for more precise QNM analyses Freise  _et al._ (2009).

To begin, we simulate the ringdown waveform using parameters that mirror those
of the GW150914 event, with $M_{f}=68.0\,M_{\odot}$, $a_{f}=0.67$, and the
polarization angle $\psi=0.0$. We incorporate three QNMs—220, 221, and
330—with amplitude levels consistent with the GW150914 signal. In the Kerr BH
framework, we set the amplitude ratios $A_{220}=2.0$, $A_{221}=3.0$, and
$A_{330}=0.2$, guided by theoretical values that relate the 22 ($l=2,m=2$) and
33 modes Kamaretsos  _et al._ (2012).333Although the relative amplitude for
the 221 mode is not theoretically determined in Ref. (Kamaretsos  _et al._ ,
2012), we adopt our own fitting results from GW150914. Specifically, the
maximum likelihood set of values yielded a ratio of $A_{220}:A_{221}=2:2.9$.
This approximate ratio was further corroborated by our GWTC-3 analysis for all
events. Subsequently, we embed this simulated waveform into the Gaussian noise
spectrum of ET. The SNR for our simulated event approximates $270$.

Figure 6:  Same as Fig. 1, but for the simulated ET ringdown data, where the
injected final spin and the final charge of the remnant are $a=0.67$ and
$Q=0$, respectively.

.

Figure 7:  Same as Fig. 1, but for posterior comparisons of only considering
the 220 and 221 modes (left) versus incorporating the 220, 221, and 330 modes
collectively (right). The injected parameters are the same as in Fig. 6.

We proceed with Bayesian inference using the KN BH model on the simulated ET
ringdown data, employing a methodology consistent with our earlier approach.
The only modification is the inclusion of an additional higher mode,
specifically the 330 mode, in the fitting procedure. The resulting charge-spin
distribution is depicted in Fig. 6. The posterior distribution for charge
predominantly converges toward zero. When juxtaposed with the GWTC-3 results,
the error margins are markedly reduced, and the correlation between $Q$ and
$a$ is more discernible. While the belt-like distribution persists, indicating
that the charge-spin degeneracy is not entirely eliminated, the degeneracy is
noticeably lessened.

We proceed to replicate the foregoing analysis using the Kerr BH model and
also compute the Bayes factor $\mathcal{B}_{\text{Kerr}}^{\text{KN}}$. The
logarithm of the Bayes factor amounts to $-0.6$, signaling a preference for
the Kerr BH model over the KN BH model. Given that the signal was simulated
under the assumption of the Kerr BH model, this outcome aligns with our
expectation. However, the presence of charge-spin degeneracy raises the
question of whether the Kerr BH model would still yield a reasonable fit even
if the actual signal had included a nonzero charge. To scrutinize this, we
inject ringdown signals with charges $Q=0.2$ and $Q=0.4$. The log Bayes
factors are positive in both cases, validating the robustness of our analysis.
The negative log Bayes factor emerges only when the BH is not substantially
charged, underscoring the utility of the Bayes factor in evaluating the Kerr
hypothesis in future ET observations. These findings are summarized in Table.
2.

We further investigate the impact of incorporating higher modes into our
analysis. As delineated in Sec. III, the inclusion of higher modes can
mitigate the charge-spin degeneracy. Figure 7 contrasts the outcome when
considering only the 220 and 221 modes against that obtained when the 220,
221, and 330 modes are all included in the fitting, under the KN BH model for
the simulated ringdown signal. The latter scenario exhibits slightly weaker
degeneracy effects. Although we anticipate that even more precise results
could be achieved by including still higher modes like the 440 mode, the
requisite QNMs for the KN BH model remain to be computed.

Figure 8: Posterior distributions of the final BH charge for the simulated ET
ringdown data using the Kerr prior. The blue, red and gray shadows represent
the results with an injected remnant charge $Q=0$, $Q=0.2$ and $Q=0.4$,
respectively.

We next turn our attention to the so-called “null test”, in which we employ
Kerr-based priors to constrain the final spin and evaluate the upper limits on
the remnant charge. The priors used for the remnant mass and spin are outlined
in Table 2, along with the derived posterior distributions for the remnant
charge. In addition to the Kerr BH scenario, we also examine cases with
injected charges $Q=0.2$ and $Q=0.4$ for comparative analysis. As depicted in
Fig. 8, these cases yield noticeably distinct charge distributions, even when
the priors are rooted in the Kerr hypothesis. This outcome signals that the
charge-spin degeneracy is only partially alleviated. For the $Q=0$ case, we
constrain the remnant charge to $Q<0.2$ at the $90\%$ CL

Our constraint of $Q<0.2$ on the remnant charge may appear somewhat modest,
especially when contrasted with existing LVK constraints, $Q\lesssim 0.3$,
despite the tenfold enhancement in ET’s sensitivity. Cardoso  _et al._ (2016)
theorized that more stringent constraints could be achieved for high-spin
events. Confirming this, our simulation with $a=0.9$ yielded an upper limit of
$Q<0.15$. However, this tighter constraint may in part be attributed to the
$a^{2}+Q^{2}\leq 1$ restriction. Indeed, our simulation results fall short of
the predictions made by Cardoso  _et al._ (2016), who posited an upper limit
on $Q$ of less than $0.1$. We attribute this discrepancy to the differing
methodologies. While Cardoso  _et al._ (2016) employed the Fisher information
matrix approach with a fixed spin, our Bayesian analysis contends with the
inherent degeneracy between spin and charge, making the results more
realistic. Simulations with higher SNRs did not significantly alter these
charge constraints, suggesting that including additional QNMs in the ringdown
model is crucial to obtaining more robust limits.

##  V Conclusions

In this study, our primary focus has been on constraining the charge of
remnant BHs using merger-ringdown GW data from GWTC-3 and simulated data from
the ET. We computed the posterior distributions for both charge and spin of
the remnant BHs. While our constraints from GWTC-3 data echoed previous
findings, the simulated ET data offered enhanced constraints. However, the
overarching challenge remains the degeneracy between charge and spin.

From the GWTC-3 dataset, we zeroed in on five events that exhibited
trustworthy ringdown signals. In our QNM analysis, we took into account both
the 220 and 221 modes, deriving distributions for charge and spin of the
resultant BHs. Notably, these distributions are heavily influenced by the
charge-spin degeneracy. This is particularly evident in the case of GW191109,
which displayed a pronounced correlation between remnant charge and spin,
resulting in a distinctive belt-like distribution in the charge-spin plane.
Upon employing a Kerr-based prior informed by preceding IMR analyses,
constraints such as $Q<0.37$ emerged from single events like GW200129 and
GW200224, aligning closely with findings from GWTC-2. Our subsequent
evaluations of the Kerr hypothesis suggest that most events exhibit charges
consistent with zero, though the case for GW191109 remains somewhat ambiguous.
Given the prevailing charge-spin degeneracy and the current limitations in the
sensitivity of detectors, all computed Bayes factors between the KN BH and
Kerr BH models are negative. This underscores the greater reliability of the
Kerr BH model in describing the five selected GW events.

In our ET simulation, we generated synthetic ET data for BHs with varying
remnant charges. Given the elevated SNRs achievable with ET, we were able to
include 220, 221, and 330 modes in our analysis, thereby ameliorating the
charge-spin degeneracy issue to some extent. Consequently, the posterior
distributions for charge and spin improved substantially, manifesting
significantly reduced errors. However, complete elimination of charge-spin
degeneracy remains elusive, as evidenced by the persistence of belt-like
distributions in our results. Our simulation validated the efficacy of Bayes
factors in scrutinizing the Kerr hypothesis in the context of ET data.
Furthermore, the upper limit on charge obtained from the “null test” improved
to $Q<0.2$ for $a\approx 0.67$ and $Q<0.15$ for $a\approx 0.9$ with ET.

The crux of our investigation has been the persistent appearance of the
charge-spin degeneracy. This phenomenon stems from the similar dependencies of
currently utilized QNMs on both remnant charge and spin. The inclusion of
additional QNMs in future analyses could potentially alleviate this
degeneracy, thereby sharpening constraints on BH charge and facilitating more
accurate identification of charged remnant BHs. Given that ET provides the
capability to probe higher QNMs, future work will require expanded
calculations on higher KN QNMs. Finally, if there are some mechanism resulting
in similar charge-to-mass ratios for remnant BHs, one may also consider to
perform event stacking in order to obtain tighter constraints on the BH
charge. Nevertheless, such stacking relies on extra assumptions and is less
general than the constraints from single events as obtained in this work.

###### Acknowledgements.

We thank Gregorio Carullo and Harrison Siegel for helpful comments. This work
was supported by the National Natural Science Foundation of China (11975027,
12247152, 11991053), the China Postdoctoral Science Foundation (2022TQ0011),
the National SKA Program of China (2020SKA0120300), the Max Planck Partner
Group Program funded by the Max Planck Society, and the High-performance
Computing Platform of Peking University. HTW is supported by the Opening
Foundation of TianQin Research Center. LIGO Laboratory and Advanced LIGO are
funded by the United States National Science Foundation (NSF) as well as the
Science and Technology Facilities Council (STFC) of the United Kingdom, the
Max-Planck-Society (MPS), and the State of Niedersachsen/Germany for support
of the construction of Advanced LIGO and construction and operation of the
GEO600 detector. Additional support for Advanced LIGO was provided by the
Australian Research Council. Virgo is funded, through the European
Gravitational Observatory (EGO), by the French Centre National de Recherche
Scientifique (CNRS), the Italian Istituto Nazionale di Fisica Nucleare (INFN)
and the Dutch Nikhef, with contributions by institutions from Belgium,
Germany, Greece, Hungary, Ireland, Japan, Monaco, Poland, Portugal, Spain.
KAGRA is supported by Ministry of Education, Culture, Sports, Science and
Technology (MEXT), Japan Society for the Promotion of Science (JSPS) in Japan;
National Research Foundation (NRF) and Ministry of Science and ICT (MSIT) in
Korea; Academia Sinica (AS) and National Science and Technology Council (NSTC)
in Taiwan of China.

## References

  * Remillard and McClintock (2006) R. A. Remillard and J. E. McClintock, Ann. Rev. Astron. Astrophys. 44, 49 (2006).
  * Akiyama  _et al._ (2019) K. Akiyama _et al._ (Event Horizon Telescope), Astrophys. J. Lett. 875, L1 (2019).
  * Abbott  _et al._ (2016a) B. P. Abbott _et al._ (LIGO Scientific, Virgo), Phys. Rev. Lett. 116, 061102 (2016a).
  * Abbott  _et al._ (2016b) B. P. Abbott _et al._ (LIGO Scientific, Virgo), Phys. Rev. Lett. 116, 221101 (2016b), [Erratum: Phys.Rev.Lett. 121, 129902 (2018)].
  * Punturo  _et al._ (2010) M. Punturo _et al._ , Class. Quant. Grav. 27, 194002 (2010).
  * Maggiore  _et al._ (2020) M. Maggiore _et al._ , JCAP 03, 050 (2020).
  * Sathyaprakash  _et al._ (2019) B. S. Sathyaprakash _et al._ , Bull. Am. Astron. Soc. 51, 251 (2019).
  * Kalogera  _et al._ (2021) V. Kalogera _et al._ , (2021), arXiv:2111.06990 [gr-qc] .
  * Kerr (1963) R. P. Kerr, Phys. Rev. Lett. 11, 237 (1963).
  * Newman  _et al._ (1965) E. T. Newman, R. Couch, K. Chinnapared, A. Exton, A. Prakash, and R. Torrence, J. Math. Phys. 6, 918 (1965).
  * Gibbons (1975) G. W. Gibbons, Commun. Math. Phys. 44, 245 (1975).
  * Zajaček and Tursunov (2019) M. Zajaček and A. Tursunov, (2019), arXiv:1904.04654 [astro-ph.GA] .
  * Gong  _et al._ (2019) Y. Gong, Z. Cao, H. Gao, and B. Zhang, Mon. Not. Roy. Astron. Soc. 488, 2722 (2019).
  * Blandford and Znajek (1977) R. D. Blandford and R. L. Znajek, Mon. Not. Roy. Astron. Soc. 179, 433 (1977).
  * Cardoso  _et al._ (2016) V. Cardoso, C. F. B. Macedo, P. Pani, and V. Ferrari, JCAP 05, 054 (2016), [Erratum: JCAP 04, E01 (2020)].
  * Zajaček  _et al._ (2018) M. Zajaček, A. Tursunov, A. Eckart, and S. Britzen, Mon. Not. Roy. Astron. Soc. 480, 4408 (2018).
  * Zajaček  _et al._ (2019) M. Zajaček, A. Tursunov, A. Eckart, S. Britzen, E. Hackmann, V. Karas, Z. Stuchlík, B. Czerny, and J. A. Zensus, J. Phys. Conf. Ser. 1258, 012031 (2019).
  * Preskill (1984) J. Preskill, Ann. Rev. Nucl. Part. Sci. 34, 461 (1984).
  * Bozzola and Paschalidis (2021a) G. Bozzola and V. Paschalidis, Phys. Rev. Lett. 126, 041103 (2021a).
  * Pereñiguez (2023) D. Pereñiguez, (2023), arXiv:2302.10942 [gr-qc] .
  * Dyson and Pereñiguez (2023) C. Dyson and D. Pereñiguez, (2023), arXiv:2306.15751 [gr-qc] .
  * Xu  _et al._ (2023) R. Xu, D. Liang, and L. Shao, Phys. Rev. D 107, 024011 (2023).
  * Stebbins and Krnjaic (2019) A. Stebbins and G. Krnjaic, JCAP 12, 003 (2019).
  * Ghosh and Afrin (2023) S. G. Ghosh and M. Afrin, Astrophys. J. 944, 174 (2023).
  * Akiyama  _et al._ (2022a) K. Akiyama _et al._ (Event Horizon Telescope), Astrophys. J. Lett. 930, L12 (2022a).
  * Akiyama  _et al._ (2022b) K. Akiyama _et al._ (Event Horizon Telescope), Astrophys. J. Lett. 930, L17 (2022b).
  * Ferrari and Gualtieri (2008) V. Ferrari and L. Gualtieri, Gen. Rel. Grav. 40, 945 (2008).
  * Dreyer  _et al._ (2004) O. Dreyer, B. J. Kelly, B. Krishnan, L. S. Finn, D. Garrison, and R. Lopez-Aleman, Class. Quant. Grav. 21, 787 (2004).
  * Berti  _et al._ (2016) E. Berti, A. Sesana, E. Barausse, V. Cardoso, and K. Belczynski, Phys. Rev. Lett. 117, 101102 (2016).
  * Kokkotas (1993) K. D. Kokkotas, Nuovo Cim. B 108, 991 (1993).
  * Berti and Kokkotas (2005) E. Berti and K. D. Kokkotas, Phys. Rev. D 71, 124008 (2005).
  * Pani  _et al._ (2013a) P. Pani, E. Berti, and L. Gualtieri, Phys. Rev. Lett. 110, 241103 (2013a).
  * Pani  _et al._ (2013b) P. Pani, E. Berti, and L. Gualtieri, Phys. Rev. D 88, 064048 (2013b).
  * Mark  _et al._ (2015) Z. Mark, H. Yang, A. Zimmerman, and Y. Chen, Phys. Rev. D 91, 044025 (2015).
  * Zimmerman and Mark (2016) A. Zimmerman and Z. Mark, Phys. Rev. D 93, 044033 (2016), [Erratum: Phys.Rev.D 93, 089905 (2016)].
  * Wang  _et al._ (2021a) H.-T. Wang, S.-P. Tang, P.-C. Li, and Y.-Z. Fan, Phys. Rev. D 104, 104063 (2021a).
  * Dias  _et al._ (2015) O. J. C. Dias, M. Godazgar, and J. E. Santos, Phys. Rev. Lett. 114, 151101 (2015).
  * Dias  _et al._ (2022a) O. J. C. Dias, M. Godazgar, J. E. Santos, G. Carullo, W. Del Pozzo, and D. Laghi, Phys. Rev. D 105, 084044 (2022a).
  * Dias  _et al._ (2022b) O. J. C. Dias, M. Godazgar, and J. E. Santos, JHEP 07, 076 (2022b).
  * Wang  _et al._ (2021b) H.-T. Wang, P.-C. Li, J.-L. Jiang, G.-W. Yuan, Y.-M. Hu, and Y.-Z. Fan, Eur. Phys. J. C 81, 769 (2021b).
  * Bozzola and Paschalidis (2021b) G. Bozzola and V. Paschalidis, Phys. Rev. D 104, 044004 (2021b).
  * Gupta  _et al._ (2021) P. K. Gupta, T. F. M. Spieksma, P. T. H. Pang, G. Koekoek, and C. V. D. Broeck, Phys. Rev. D 104, 063041 (2021).
  * Benavides-Gallego and Han (2023) C. A. Benavides-Gallego and W.-B. Han, Symmetry 15, 537 (2023).
  * García-Quirós  _et al._ (2020) C. García-Quirós, M. Colleoni, S. Husa, H. Estellés, G. Pratten, A. Ramos-Buades, M. Mateu-Lucena, and R. Jaume, Phys. Rev. D 102, 064002 (2020).
  * Carullo  _et al._ (2022) G. Carullo, D. Laghi, N. K. Johnson-McDaniel, W. Del Pozzo, O. J. C. Dias, M. Godazgar, and J. E. Santos, Phys. Rev. D 105, 062009 (2022).
  * Abbott  _et al._ (2019a) B. P. Abbott _et al._ (LIGO Scientific, Virgo), Phys. Rev. X 9, 031040 (2019a).
  * Abbott  _et al._ (2021a) R. Abbott _et al._ (LIGO Scientific, Virgo), Phys. Rev. X 11, 021053 (2021a).
  * Abbott  _et al._ (2021b) R. Abbott _et al._ (LIGO Scientific, VIRGO, KAGRA), (2021b), arXiv:2111.03606 [gr-qc] .
  * Abbott  _et al._ (2019b) B. P. Abbott _et al._ (LIGO Scientific, Virgo), Phys. Rev. D 100, 104036 (2019b).
  * Abbott  _et al._ (2021c) R. Abbott _et al._ (LIGO Scientific, Virgo), Phys. Rev. D 103, 122002 (2021c).
  * Abbott  _et al._ (2021d) R. Abbott _et al._ (LIGO Scientific, VIRGO, KAGRA), (2021d), arXiv:2112.06861 [gr-qc] .
  * Teukolsky (1973) S. A. Teukolsky, Astrophys. J. 185, 635 (1973).
  * Berti  _et al._ (2009) E. Berti, V. Cardoso, and A. O. Starinets, Class. Quant. Grav. 26, 163001 (2009).
  * Isi and Farr (2021) M. Isi and W. M. Farr, (2021), arXiv:2107.05609 [gr-qc] .
  * Misner  _et al._ (1973) C. W. Misner, K. S. Thorne, and J. A. Wheeler,  _Gravitation_ (W. H. Freeman, San Francisco, 1973).
  * Cotesta  _et al._ (2022) R. Cotesta, G. Carullo, E. Berti, and V. Cardoso, Phys. Rev. Lett. 129, 111102 (2022).
  * Carullo  _et al._ (2019) G. Carullo, W. Del Pozzo, and J. Veitch, Phys. Rev. D 99, 123029 (2019), [Erratum: Phys.Rev.D 100, 089903 (2019)].
  * Isi  _et al._ (2019) M. Isi, M. Giesler, W. M. Farr, M. A. Scheel, and S. A. Teukolsky, Phys. Rev. Lett. 123, 111102 (2019).
  * Capano  _et al._ (2021) C. D. Capano, M. Cabero, J. Westerweck, J. Abedi, S. Kastha, A. H. Nitz, Y.-F. Wang, A. B. Nielsen, and B. Krishnan, (2021), arXiv:2105.05238 [gr-qc] .
  * Siegel  _et al._ (2023) H. Siegel, M. Isi, and W. M. Farr, Phys. Rev. D 108, 064008 (2023).
  * Shi  _et al._ (2019) C. Shi, J. Bao, H. Wang, J.-d. Zhang, Y. Hu, A. Sesana, E. Barausse, J. Mei, and J. Luo, Phys. Rev. D 100, 044036 (2019).
  * Berti  _et al._ (2006) E. Berti, V. Cardoso, and M. Casals, Phys. Rev. D 73, 024013 (2006), [Erratum: Phys.Rev.D 73, 109902 (2006)].
  * Lim  _et al._ (2019) H. Lim, G. Khanna, A. Apte, and S. A. Hughes, Phys. Rev. D 100, 084032 (2019).
  * Dhani (2021) A. Dhani, Phys. Rev. D 103, 104048 (2021).
  * Dhani and Sathyaprakash (2021) A. Dhani and B. S. Sathyaprakash, (2021), arXiv:2107.14195 [gr-qc] .
  * The LIGO Scientific Collaboration, the Virgo Collaboration, the KAGRA Collaboration, LIGO-P2100456 () (2021) The LIGO Scientific Collaboration, the Virgo Collaboration, the KAGRA Collaboration, LIGO-P2100456 (2021), https://dcc.ligo.org/P2100456 .
  * Carullo  _et al._ (2023) G. Carullo, W. Del Pozzo, and J. Veitch, “pyring,”  (2023).
  * Pozzo and Veitch (2015) W. D. Pozzo and J. Veitch, https://github.com/johnveitch/cpnest (2015).
  * Skilling (2006) J. Skilling, Bayesian Analysis 1, 833 (2006).
  * Freise  _et al._ (2009) A. Freise, S. Chelkowski, S. Hild, W. Del Pozzo, A. Perreca, and A. Vecchio, Class. Quant. Grav. 26, 085012 (2009).
  * Kamaretsos  _et al._ (2012) I. Kamaretsos, M. Hannam, and B. Sathyaprakash, Phys. Rev. Lett. 109, 141102 (2012).