11institutetext: Security, Crime, and Intelligence Innovation Institute,
Cardiff University, 11email<EMAIL_ADDRESS>
# Detecting Anti-vaccine Content on Twitter using Multiple Message-Based
Network Representations

James R. Ashford 11 0000-0002-2678-577X

###### Abstract

Social media platforms such as Twitter have a fundamental role in facilitating
the spread and discussion of ideas online through the concept of retweeting
and replying. However, these features also contribute to the spread of
mis/disinformation during the vaccine rollout of the COVID-19 pandemic. Using
COVID-19 vaccines as a case study, we analyse multiple social network
representation derived from three message-based interactions on Twitter (quote
retweets, mentions and replies) based upon a set of known anti-vax hashtags
and keywords. Each network represents a certain hashtag or keyword which were
labelled as “controversial” and “non-controversial” according to a small group
of participants. For each network, we extract a combination of global and
local network-based metrics which are used as feature vectors for binary
classification. Our results suggest that it is possible to detect
controversial from non-controversial terms with high accuracy using simple
network-based metrics. Furthermore, these results demonstrate the potential of
network representations as language-agnostic models for detecting
mis/disinformation at scale, irrespective of content and across multiple
social media platforms.

###### Keywords:

Social Network Analysis Social Media Disinformation Misinformation

##  1 Introduction

Social networks have a fundamental role in the way in which users communicate
with one another online. As a result of this, issues such as misinformation
begin to emerge due to the size and heavily connected nature of social media
platforms [41, 38]. More specifically, since the COVID-19 pandemic,
discussions surrounding vaccine usage has attracted highly emotive, positive
and negatives view-points which, consequently, increases the potential for
misinformation to emerge [18, 28]. This can have far-reaching consequences in
an offline setting as ill-informed decisions, as of a result of
misinformation, can be a threat to public health [37, 38].

Furthermore, microblogging platforms such as X (known as Twitter in the
context of this paper) have further contributed to this issue due to the
spread of misinformation surrounding vaccines through the use of message-led
interactions (e.g. replies, retweets etc) [35, 39]. What makes Twitter a
particularly interesting platform to study is how user’s use the platform to
communicate with others using three distinct types of message-based
interactions. These interactions include mentions (a tweet which contain
“mentions” another person’s username), replies (a user responding to another
user’s tweet) and quote retweets (tweeting another person’s tweet with a
comment). These pairwise user interactions support different modes of
communication and user engagement.

The three interaction mentioned (mention, reply and quote retweet) are the
focus of this paper and are represented in the form of a social network for
modelling user activity. An example is shown in Figure 1.

(a) Mention

(b) Quote retweet

(c) Reply

Figure 1: Examples of three message-based interaction networks in the form of
mentions (left), quote retweets (centre) and replies (right) on Twitter based
on tweets which mention #NoNewNormal.

In this paper, “anti-vaccine content” is defined using a combination of known,
crowdsourced and custom COVID-19 anti-vax hashtags and keywords (collectively
referred to as “terms”) used on Twitter which are ranked by a small group of
participants ($N=5$). Each term is represented by three individual networks
from each interaction type. A term is labelled either “controversial” or “non-
controversial” depending on how it scores on a Likert scale.

We hypothesise that network metrics and substructures can be used to
differentiate between controversial and non-controversial terms using either
or a combination of networks constructed from quote retweets, mentions, and
replies.

In doing so, the aim of this paper is two-fold. Firstly, to understand how
network-based features can be used to observe nuances between each network
term and interaction type. Little research has been performed in an attempt to
study the utility of these three types of interaction (both combined and in
isolation) by cross comparison. And secondly, to extract latent signals within
each network to differentiate between controversial and non-controversial
terms.

###  1.1 Background and Related Work

The ability to detect misinformation and fake news on social media is by no
means a novel idea and there has been a wealth of research invested in
computational solutions that can identify and detect user accounts responsible
for spreading such information in a semi autonomous fashion [34, 13]. More
specifically, misinformation surrounding vaccine usage has been a historical
issue which has been exacerbated in recent years due to the COVID-19 pandemic
[14].

Within the literature, a significant component for predicting anti-vax content
relies on the use of natural language processing (NLP) as a solution for
analysing textual information. Research has demonstrated that the use of NLP
and other methods have the potential to identify tweets containing
misinformation [16, 3], conspiracies [29, 23] and hate speech [42, 30]. Within
the NLP-based literature, a subset of research focuses specifically on the
role of sentiment analysis for identifying both positive and negative view
points. Sentiment analysis has been used to understand the various themes and
trends surrounding support and opposition towards vaccines [18, 28, 19]. The
research shows that users promoting anti-vax related content frequently
engaged in replies and, overall, were more negative, showing emotions such as
rage and sorrow as one of the key themes [26, 10]. Alternatively, a network
science-based approach has been used to demonstrate how anti-vax users form
echo chambers of polarised communities with other like-minded users by
retweeting each other’s content [25].

As of this writing, there is a gap in the literature for using network-based
methods exclusively for detecting anti-vax related content on Twitter. The use
of social network analysis on Twitter has important implications to this study
by considering the utility of the platform’s three message-based interactions
- retweets, mentions and replies.

To begin, the role of retweet networks have been used to identify communities
of like-minded individuals [9] and understand the spread of information [22]
which, consequently, also includes misinformation [41]. Retweet networks
provide predictive signals for predicting retweeting behaviour [40] and can be
used as a reliable proxy for gauging popularity, social capital and friendship
formation [31, 1].

In addition to this, the use of mention interactions are primarily used to
direct a tweet to a user (or group of users) by mentioning their username in
their tweet. This type of interaction has been observed in multiple settings,
demonstrating that mentions provide utility for predicting links between users
and [6, 17] and shares similar structural properties to networks produced
using the “favourite” button [20]. Finally, replies can also be treated as a
network representation for modelling conversational dynamics. Reply networks
have been used to study patterns in multiple Q&A discussions [33], monitor
audience approval or disapproval [27] and to gain followers [32] in a
language-agnostic capacity [8]. Research has demonstrated how replies can be
used to validate Dunbar number revealing that much like offline interactions,
online interactions can reach a limit to the number of possible interactions
that can be preserved [15].

Overall, in this paper, we expand upon existing literature by using Twitter as
part of a case study to assess the ways in which network-based approaches can
be used to model multiple message-based interactions. The literature
demonstrates how these interactions have potential to provide predictive
utility for detecting controversial and non-controversial networks among
various anti-vax topics.

##  2 Dataset

All tweets that are used in this study are a part of the Coronavirus
(COVID-19) Tweets Dataset [21]. This dataset serves as a baseline for our
analysis as it uses a set of broad, predetermined, generic keywords which are
of relevance to the COVID-19 pandemic. In particular, we focus our analysis on
a specific window between 9th November and 8th December 2020. This date range
refers to the period when the initial vaccines were first approved for use in
the United Kingdom thus starting the conversation and reactions around
vaccines on Twitter [12].

In addition to this, this paper introduces a set of potentially controversial
hashtags and keywords (collectively referred to as “terms”). There are two
papers within the supporting literature which provide useful examples of terms
which align with disruptive activity.

The first paper “#Scamdemic, #Plandemic, or #Scaredemic: What Parler Social
Media Platform Tells Us about COVID-19 Vaccine” uses data originating from
Parler [4]. While this paper does not focus on Twitter itself, Parler as a
platform is well known for its issues regarding echo chambers, filter bubbles
and lack of fact-checking [7, 2].

The second paper of interest (“COVID-19 Vaccine Hesitancy on Social Media:
Building a Public Twitter Data Set of Antivaccine Content, Vaccine
Misinformation, and Conspiracies”) focuses on attempts to build a set of anti-
vax related content by manually collecting keywords which co-occur with
previously known / observed anti-vax terms such as #vaccineskill or
#vaccinedamage [29]. Furthermore, their results indicate that 229,041 (12.5%)
of their tweets originated from the UK. This means that the dataset is likely
to feature these hashtags, especially within our specified timeframe. As a
result, the combination of an applied date range and relevant terms ensures
that the tweets used as part of the study are in the scope of vaccine-related
content

As well as these two papers, an additional set of terms are used featuring the
original keywords / hashtags used as part of the IEEE Coronavirus (COVID-19)
Tweets Dataset along with a few terms that were manually selected using the
Twitter search tool.

##  3 Methodology

###  3.1 Ranking of Terms

To begin, a total of ($N=5$) participants were recruited and were asked to
rate each unique term on a Likert scale where each term is scored according to
a weight $w\in[0..4]$ based upon the following scale: “Neutral” ($w=0$),
“Somewhat Controversial” ($w=1$), “Controversial” ($w=2$), “Very
Controversial” ($w=3$), “Highly Controversial” ($w=4$). The results from each
participant are then aggregated to include the total (sum of scores), mean and
standard deviation of each score. Each of the terms were then ranked according
to mean score.

###  3.2 Hydrating tweets

In order to retrieve the specific content and metadata (such as the timestamp,
reply to, retweet and mention fields) from the Coronavirus (COVID-19) Tweets
Dataset, the tweets need to be “hydrated” from the original ID where the
Twitter API is required to lookup and retrieve the original tweet. In doing
so, the process of “hydrating” takes the list of IDs provide by the IEEE
dataset and transforms them into a set of tweets complete with the information
need for subsequent analysis.

###  3.3 Network generation

As mentioned previously, this study focuses on three different types of
interaction of interest: quote retweets, mentions and replies. A single
network $G_{i}=(V,E)$ is generated for each interaction type. A node $v_{i}\in
V$ represents a user and the presence of a directed edge $(v_{i},v_{j})\in E$
indicates an interaction towards another user. For example, $v_{i}\rightarrow
v_{j}$, can be interpreted as “$v_{i}$ mentions/replies to/quote retweets from
$v_{j}$”.

For each term used in this paper, the three interaction networks generated are
conditioned on the presence of the term appearing in the body of a tweet. For
example, a subset $t$ of all tweets $t\subset T$ are determined by only
focusing on tweets which contain the term “#covid19”. For all tweets in this
subset, three distinct networks are extracted according to the presence of one
of the interactions of interest. Overall, a total of $N=199$ terms are
considered focusing on $M=3$ interactions of interest producing a total of
$N\times M=597$ unique networks.

###  3.4 Network features

To evaluate the utility of these network representations, a classification
task is used to evaluate how well they perform in predicting controversial
terms from non-controversial terms. This is achieved using two sets of network
features - global and local network features.

####  3.4.1 Global network features

This paper considers the network-based metrics at a global-level by observing
how users in each interaction network behave collectively. These include
density (the capacity of how many interaction edges occupy the network),
reciprocity (the ratio of bidirectional ties in the network), transitivity
(the extent to which nodes for transitive edges in a triad), in degree
(mean,max,min), out degree (mean,max,min).

These metrics provide genetic structural properties and also serve as a
baseline to determine the predictive utility in comparison to the local
network features. Properties such as density, reciprocity and transitivity are
fundamental for capturing social traits such as trust, friendships and
communities within social networks [24, 5, 36]. These, in turn, can be used to
provide predictive signals for differentiating between controversial and non-
controversial terms.

####  3.4.2 Local network features

In addition to global network features, local network features are derived by
counting the frequency of all induced subgraphs of a certain size. We
described these as local features as they are used to understand the
fundamental structure or “building blocks” of the network based upon
interactions which take place between users. Due to the subgraph isomorphism
problem [11], subgraph counting does not scale well with time and is therefore
resource intensive. This means that counting subgraphs grows exponentially
with time and that larger subgraphs take much longer to compute. For this
reason, we focused on subgraphs with 3 and 4 nodes producing $N=13$ and
$N=199$ possible combinations respectively, with a total of $N=212$ subgraphs
overall. Each interaction network for a given term produces a vector
$V_{G_{i}}$ where:

$V_{G_{i}}=(v_{1},v_{2},\dots,v_{212})$ (1)

and where $v_{i}$ represent the frequency of the $i$th subgraph in the set. In
addition to this, each of these vectors $V_{G_{i}}$ are normalised making it
possible to compare to networks of different sizes using the following:

$V_{G_{i}}=\frac{1}{\Sigma_{j=1}^{212}v_{j}}(v_{1},v_{2},\dots,v_{212})$ (2)

As a result, $V_{G_{i}}$ is used to represent the ratio of subgraph
frequencies and provides the basis for discovering under and over-
representations of induced subgraphs with respect to other networks.
Furthermore, this approach can be used to determine the extent to which
interactions and terms share similar structural features.

##  4 Results

###  4.1 Discovery of anti-vax terms

A crowdsourced ranking task was used to discover controversial Twitter terms
from non-controversial terms. The combined results from all participants
reveal that most terms were labelled as “Neutral” which would appear, on
average, 37.9% of the time. The most uncommon label is “Highly Controversial”
at 8.74%.

Label | % of Appearing
---|---
Neutral | 37.89%
Somewhat Controversial | 20.8%
Controversial | 19.9%
Very Controversial | 12.6%
Highly Controversial | 8.74%
Table 1: Full list of labels used within the Likert scale and the probability
of appearing

The distribution of mean term score is shown in Figure 2 is used to determine
the position of a threshold $t$ as a cut-off point for separating non-
controversial and controversial terms.

Figure 2: Distribution of mean score for all terms in the set with the
threshold $t=0.95$ marked in red

Based upon the distribution of mean scores (as shown in Figure 2) terms are
partitioned into the two groups according to a set threshold of $t=0.95$ where
values are momentarily reduced before increasing again.

Terms with an average score exceeding the threshold are considered
“controversial” and terms lower than the threshold are considered “non-
controversial”. Using this classification technique, a total of ($N=115$)
controversial and ($N=84$) non-controversial terms were discovered producing a
58/42 split.

###  4.2 Data Overview

Using the labels provided as part of the classification task, the data was
split into the two sets according to the appropriate label. As a result of
computing the global and local metrics, a few additional observations of
interest emerged when principal component analysis is performed to determine
the spatial relevance of each set of features. These are outlined as follows:

####  4.2.1 Global network features

Principal component analysis is used to determine the spatial relevance of all
global network features, where all three interaction types are combined and
cross-compared. These results can be observed in Figure 3 with supplemented
with the corresponding eigenvector values in Figure 4.

Figure 3: Two-dimensional principal component analysis for all global network
features combining reply, mention and quote retweet interactions Figure 4: The
corresponding eigenvector values for each principal component as shown in
Figure 3

The PCA scatter plot in Figure 3 reveals how there are no obvious spacial
clustering or patterns which emerge between the two types and provides little
spatial utility. The PCA eigenvectors demonstrate how “In Degree (Max)
(Reply)” and “In Degree (Max) (Mention)” appear as the strongest features used
for each principal component.

####  4.2.2 Local network features

Due to the size of each of the feature vectors ($N=212$ for each type of
interaction), PCA is performed to reduce the size of the feature space making
it possible visualise the data in two dimensions for each type of interaction.
Furthermore, the PCA eigenvectors are used to determine inflectional subgraphs
which contribute to the spacial positioning of each feature vector. These
results are presented in Figures 5 and 6 for the PCA scatter plots and
eigenvector values respectively.

Figure 5: Two-dimensional principal component analysis for all local network
features for each interaction type Figure 6: Eigenvector values for each
principal component and interaction type as shown in Figure 5

The results in Figures 5 and 6 suggest that there are no obvious distinction
between each of the three interaction types and little indication of
clustering potential. Additionally, the coefficients in Figure 6 reveal that
the same set of subgraphs are dominant throughout each of the interaction
types. By setting a threshold of $t=0.1$ (determined by observing the
distribution of values), across all the eigenvector values, a total of five
subgraphs emerged which exceeded this threshold. These include subgraphs `S2`,
`S15`, `S18`, `S25` and `S143` which appear consistently across all three
interactions and are shown in Figure 7 for reference.

(a)

(b)

(c)

(d)

(e)

Figure 7: Subset of subgraphs selected from the PCA eigenvectors for local
features used to provide spacial relivance

###  4.3 Classification of controversial terms

Using the data produced in earlier tasks, we use the feature vectors produced
from global and local features (see Sections 3.4.2 and 3.4.1) with the
intention of understanding the predictive utility of both global and local
network features.

To archive this, we apply binary logistic regression (BLR), support vector
machine (SVM) and a random forest classifier (RFC) applied with 10-fold cross-
validation. To assess the classification performance, we report the Accuracy,
F1 Score, Precision, Recall, Sensitivity, Specificity, Positive predictive
value (+PV) and Negative predictive value (-PV).

####  4.3.1 Global network features

We train each of the three classification models using the raw metrics defined
in Section 3.4.1. Our classification results are reported for each type of
interaction and combined. The results can be found in Table 2 and 3 for each
interaction in isolation and combined respectively.

| Mention | Quote retweet | Reply
---|---|---|---
Classifier | BLR | SVM | RFC | BLR | SVM | RFC | BLR | SVM | RFC
Accuracy | 0.744 | 0.619 | 0.81 | 0.68 | 0.609 | 0.746 | 0.698 | 0.598 | 0.716
F1 Score | 0.792 | 0.733 | 0.83 | 0.748 | 0.718 | 0.782 | 0.758 | 0.724 | 0.742
Precision | 0.695 | 0.583 | 0.788 | 0.635 | 0.575 | 0.706 | 0.656 | 0.567 | 0.711
Recall | 0.921 | 0.989 | 0.876 | 0.909 | 0.955 | 0.875 | 0.899 | 1.0 | 0.775
Sensitivity | 0.544 | 0.203 | 0.734 | 0.432 | 0.235 | 0.605 | 0.475 | 0.15 | 0.65
Specificity | 0.921 | 0.989 | 0.876 | 0.909 | 0.955 | 0.875 | 0.899 | 1.0 | 0.775
+PV | 0.695 | 0.583 | 0.788 | 0.635 | 0.575 | 0.706 | 0.656 | 0.567 | 0.711
-PV | 0.86 | 0.941 | 0.841 | 0.814 | 0.826 | 0.817 | 0.809 | 1.0 | 0.722
Table 2: Complete classification results for all three interaction types using
global features reporting the performance for each classifier. The best
performing classifier is highlighted in bold

The results in Table 2 show the RFC outperforms both SVM and BLR consistently
across each type of interaction using global features. As highlighted in Table
2, mention interactions combined with an RFC produces the best performing
classifier with an accuracy of $p=0.81$. This is then followed by quote
retweets with an accuracy of $p=0.746$ and finally, reply with an accuracy of
$p=0.719$

In addition to this, we perform a separate classification task whereby all
global network features for each interaction are combined into a single
feature vector. All interaction types are combined to assess whether this has
an impact on classification performance. These results can be found in Table
3.

Classifier | BLR | SVM | RFC
---|---|---|---
Accuracy | 0.847 | 0.852 | 0.886
F1 Score | 0.914 | 0.918 | 0.937
Precision | 0.873 | 0.869 | 0.887
Recall | 0.96 | 0.973 | 0.993
Sensitivity | 0.192 | 0.154 | 0.269
Specificity | 0.96 | 0.973 | 0.993
\+ Predict Value | 0.873 | 0.869 | 0.887
\- Predict Value | 0.455 | 0.5 | 0.875
Table 3: Classification results combining all three interaction types using
global features. The best performing classifier is highlighted in bold

The results in Table 3 indicate that by combining all interaction types, the
accuracy of the RFC increases to around $p=0.886$. This produces a performance
gain of approximately 9.3% compared with the mention RFC accuracy - the best
performing classifier of all interaction types.

####  4.3.2 Local network features

Following the same format as Section 4.3.1, we train the three same
classifiers (RFC, BLR and SVM) using local network features exclusively. The
classification results for each interaction type can be found in Table 4.

| Mention | Quote retweet | Reply
---|---|---|---
Classifier | BLR | SVM | RFC | BLR | SVM | RFC | BLR | SVM | RFC
Accuracy | 0.72 | 0.732 | 0.786 | 0.663 | 0.663 | 0.704 | 0.663 | 0.692 | 0.615
F1 Score | 0.715 | 0.731 | 0.8 | 0.667 | 0.692 | 0.747 | 0.674 | 0.711 | 0.677
Precision | 0.776 | 0.782 | 0.791 | 0.687 | 0.66 | 0.673 | 0.686 | 0.703 | 0.607
Recall | 0.663 | 0.685 | 0.809 | 0.648 | 0.727 | 0.841 | 0.663 | 0.719 | 0.764
Sensitivity | 0.785 | 0.785 | 0.759 | 0.679 | 0.593 | 0.556 | 0.662 | 0.662 | 0.45
Specificity | 0.663 | 0.685 | 0.809 | 0.648 | 0.727 | 0.841 | 0.663 | 0.719 | 0.764
+PV | 0.776 | 0.782 | 0.791 | 0.687 | 0.66 | 0.673 | 0.686 | 0.703 | 0.607
-PV | 0.674 | 0.689 | 0.779 | 0.64 | 0.667 | 0.763 | 0.639 | 0.679 | 0.632
Table 4: Complete classification results for all three interaction types using
local features reporting the performance for each classifier. The best
performing classifier is highlighted in bold

Much like Section 4.3.1, RFC is the highest performing classifier for each
interaction type except for reply interactions where SVM performs the best.
Similarly, mention interactions outperform quote retweets and replies with
respect to predictive utility and performance. Each of the feature vectors
used as part of the classification task in Table 4 are combined and used in a
separate classification task with the results shown in Table 5.

Classifier | BLR | SVM | RFC
---|---|---|---
Accuracy | 0.852 | 0.852 | 0.858
F1 Score | 0.92 | 0.92 | 0.922
Precision | 0.852 | 0.852 | 0.865
Recall | 1.0 | 1.0 | 0.987
Sensitivity | 0.0 | 0.0 | 0.115
Specificity | 1.0 | 1.0 | 0.987
\+ Predict Value | 0.852 | 0.852 | 0.865
\- Predict Value | - | - | 0.6
Table 5: Classification results combining all three interaction types using
local features. The best performing classifier is highlighted in bold

The results in Table 5 show that RFC is the best performing classifier when
all interactions are combined using local features. The accuracy of the RFC
mode increased to $p=0.858$ which, in turn, produces a performance gain of
9.16% compared with the best performing result in Table 4.

##  5 Discussion

Both the data overview (see Section 4.2) and classification results (see
Section 4.3) provide meaningful insights on the utility of using network
representations for predicting controversial and non-controversial terms. As a
result, a number of key observations are explored and disused further in this
section which relate to the utility of both global and local network
representations and their prediction performance. These are discussed as
follows:

As described in Section 4.2.2, the PCA eigenvectors shown in Figure 6 reveals
five subgraphs (see Figure 7 for subgraphs `S2`, `S15`, `S18`, `S25` and
`S143`) which exceed an arbitrary threshold of $t=0.1$. These subgraphs
reflect those that resemble a tree-like structure where all interactions are
centred around one node (similar to the previous study). This potentially
correlates with features such as the maximum in degree of a network where many
nodes are directed towards a single “central” node. This is evident based upon
the features which emerged in the PCA eigenvectors for global features in
Figure 4 where the maximum in degree is dominant.

The use of global features for predicting controversial and non-controversial
networks reveals rather promising results. The use of a RFC (Random Forest
Classifier) consistently outperforms alternative classifiers across each of
the three interaction types. The results in Table 2 show that mention
interactions can best differentiate between the two types with a
classification accuracy of $p=0.81$. Based open the eigenvector values in
Figure 4, it is possible to speculate that the maximum in degree for mention
interactions (one of the most dominant features) provide the best spatial
distribution of networks in order to separate the two groups. As a result of
combining all three interactions, it is also reasonable to imply that improved
accuracy of $p=0.886$ is due to the maximum in degree for replies also being a
dominant feature in the PCA eigenvectors in Figure 4. By using local features
for differentiating between controversial and non-controversial networks a
similar trend can be observed compared with global features. The results are
fairly consistent across each of the interactions which is unsurprising
considering that the PCA eigenvectors in Figure 6 for each interaction are
almost all identical. Similarly, mention interactions provide the best result
with respect to accuracy ($p=0.786$) which is improved to $p=0.858$ when all
interactions are combined.

##  6 Conclusion

Using COVID-19 anti-vaccine content as a case study, the insights gained from
this investigation provide meaningful insights towards understanding the
utility of using social network representations for differentiating between
controversial and non-controversial. In particular, this paper satisfies the
hypothesis by using a prediction task to show how three message-based user
interactions (quote retweets, mentions and replies) provide network-based
representations to understand how users behave collectively based upon their
underlying network substructures and metrics.

The results of this paper provide evidence that simple graph-based metrics
such as in/out degree, density, reciprocity and transitivity are sufficient
for differentiating between controversial and non-controversial networks. By
combining all three interactions into one, it is clear that global features
adequately capture the nuances between each of the networks using relatively
few features - contrary to subgraph “local” approach. As well as providing a
performance gain, the set of global features used in this study are relatively
easy to calculate and can almost be done in near real-time.

The implications of this paper impact how we are to consider using quote
retweets, mentions, and replies (or a combination of the three) in future
work. The research featured in this paper clearly demonstrate that it is
possible to differentiate between different networks using interactions
derived from human behaviour exclusively. The clear advantage of this is that
little textual analysis (e.g NLP) is needed, making it possible to replicate
results using a non-English speaking corpus. Furthermore, these results have
applications in content moderation, whereby moderators can use similar
techniques to identify the presence of controversial content using simple
network-based metrics. Finally, we emphasise that these techniques are
transferable and can be used for a wide range of scenarios for detecting
disruptive activity more broadly due to widespread adoption of features such
as replying and sharing - fundamental user-to-user interactions relevant to
almost all social media platforms.

## References

  * [1] Hyerim Ahn and Ji-Hong Park.  The structural effects of sharing function on twitter networks: Focusing on the retweet function.  Journal of Information Science, 41(3):354–365, 2015.
  * [2] Faisal Alatawi, Lu Cheng, Anique Tahir, Mansooreh Karami, Bohan Jiang, Tyler Black, and Huan Liu.  A survey on echo chambers on social media: Description, detection and mitigation.  arXiv preprint arXiv:2112.05084, 2021.
  * [3] Dimosthenis Antypas, Jose Camacho-Collados, Alun Preece, and David Rogers.  Covid-19 and misinformation: A large-scale lexical analysis on twitter.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop, pages 119–126, 2021.
  * [4] Annalise Baines, Muhammad Ittefaq, and Mauryne Abwao.  # scamdemic,# plandemic, or# scaredemic: what parler social media platform tells us about covid-19 vaccine.  Vaccines, 9(5):421, 2021.
  * [5] Per Block.  Reciprocity, transitivity, and the mysterious three-cycle.  Social Networks, 40:163–173, 2015.
  * [6] Rosa Borge Bravo and Marc Esteve Del Valle.  Opinion leadership in parliamentary twitter networks: A matter of layers of interaction?  Journal of Information Technology & Politics, 14(3):263–276, 2017.
  * [7] Jonathan Bright, Nahema Marchal, Bharath Ganesh, and Stevan Rudinac.  How do individuals in a radical echo chamber react to opposing views? evidence from a content analysis of stormfront.  Human Communication Research, 48(1):116–145, 2022.
  * [8] Axel Bruns, Tim Highfield, and Jean Burgess.  The arab spring and its social media audiences: English and arabic twitter users and their networks.  In Cyberactivism on the participatory web, pages 96–128. Routledge, 2014.
  * [9] Darko Cherepnalkoski and Igor Mozetič.  Retweet networks of the european parliament: Evaluation of the community structure.  Applied network science, 1(1):1–20, 2016.
  * [10] Harshita Chopra, Aniket Vashishtha, Ridam Pal, Ananya Tyagi, Tavpritesh Sethi, et al.  Mining trends of covid-19 vaccine beliefs on twitter with lexical embeddings.  arXiv preprint arXiv:2104.01131, 2021.
  * [11] Stephen A Cook.  The complexity of theorem-proving procedures.  In Proceedings of the third annual ACM symposium on Theory of computing, pages 151–158, 1971.
  * [12] Liviu-Adrian Cotfas, Camelia Delcea, Ioan Roxin, Corina Ioanăş, Dana Simona Gherai, and Federico Tajariol.  The longest month: analyzing covid-19 vaccination opinions dynamics from tweets in the month following the first vaccine announcement.  Ieee Access, 9:33203–33223, 2021.
  * [13] Matthew R DeVerna, Francesco Pierri, Bao Tran Truong, John Bollenbacher, David Axelrod, Niklas Loynes, Christopher Torres-Lugo, Kai-Cheng Yang, Filippo Menczer, and John Bryden.  Covaxxy: A collection of english-language twitter posts about covid-19 vaccines.  In ICWSM, pages 992–999, 2021.
  * [14] Miguel Gallegos, Viviane de Castro Pecanha, and Tomás Caycho-Rodríguez.  Anti-vax: the history of a scientific problem.  Journal of Public Health, 2022.
  * [15] Bruno Gonçalves, Nicola Perra, and Alessandro Vespignani.  Modeling users’ activity on twitter networks: Validation of dunbar’s number.  PloS one, 6(8):e22656, 2011.
  * [16] Tamanna Hossain, Robert L Logan IV, Arjuna Ugarte, Yoshitomo Matsubara, Sean Young, and Sameer Singh.  Covidlies: Detecting covid-19 misinformation on social media.  2020\.
  * [17] Hadrien Hours, Eric Fleury, and Márton Karsai.  Link prediction in the twitter mention network: impacts of local structure and similarity of interest.  In 2016 IEEE 16th International Conference on Data Mining Workshops (ICDMW), pages 454–461. IEEE, 2016.
  * [18] Man Hung, Evelyn Lauren, Eric S Hon, Wendy C Birmingham, Julie Xu, Sharon Su, Shirley D Hon, Jungweon Park, Peter Dang, Martin S Lipsky, et al.  Social network analysis of covid-19 sentiments: Application of artificial intelligence.  Journal of medical Internet research, 22(8):e22590, 2020.
  * [19] Gloria J Kang, Sinclair R Ewing-Nelson, Lauren Mackey, James T Schlitt, Achla Marathe, Kaja M Abbas, and Samarth Swarup.  Semantic network analysis of vaccine sentiment in online social media.  Vaccine, 35(29):3621–3638, 2017.
  * [20] Shoko Kato, Akihiro Koide, Takayasu Fushimi, Kazumi Saito, and Hiroshi Motoda.  Network analysis of three twitter functions: favorite, follow and mention.  In Pacific Rim Knowledge Acquisition Workshop, pages 298–312. Springer, 2012.
  * [21] Rabindra Lamsal.  Design and analysis of a large-scale covid-19 tweets dataset.  Applied Intelligence, 51(5):2790–2804, 2021.
  * [22] Kristina Lerman and Rumi Ghosh.  Information contagion: An empirical study of the spread of news on digg and twitter social networks.  In Fourth international AAAI conference on weblogs and social media, 2010.
  * [23] Yachao Li, Sylvia Twersky, Kelsey Ignace, Mei Zhao, Radhika Purandare, Breeda Bennett-Jones, and Scott R Weaver.  Constructing and communicating covid-19 stigma on twitter: a content analysis of tweets during the early stage of the covid-19 outbreak.  International Journal of Environmental Research and Public Health, 17(18):6847, 2020.
  * [24] Guanfeng Liu, Yan Wang, and Mehmet A Orgun.  Trust transitivity in complex social networks.  In twenty-fifth AAAI conference on artificial intelligence, 2011.
  * [25] Elena Milani, Emma Weitkamp, and Peter Webb.  The visual vaccine debate on twitter: A social network analysis.  Media and Communication, 8(2):364–375, 2020.
  * [26] Kunihiro Miyazaki, Takayuki Uchiba, Kenji Tanaka, and Kazutoshi Sasahara.  Characterizing the anti-vaxxers’ reply behavior on social media.  In IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, pages 83–89, 2021.
  * [27] Logan Molyneux and Rachel R Mourão.  Political journalists’ normalization of twitter: Interaction and new affordances.  Journalism Studies, 20(2):248–266, 2019.
  * [28] Michal Monselise, Chia-Hsuan Chang, Gustavo Ferreira, Rita Yang, Christopher C Yang, et al.  Topics and sentiments of public concerns regarding covid-19 vaccines: social media trend analysis.  Journal of Medical Internet Research, 23(10):e30765, 2021.
  * [29] Goran Muric, Yusong Wu, Emilio Ferrara, et al.  Covid-19 vaccine hesitancy on social media: building a public twitter data set of antivaccine content, vaccine misinformation, and conspiracies.  JMIR public health and surveillance, 7(11):e30642, 2021.
  * [30] Xin Pei and Deval Mehta.  # coronavirus or# chinesevirus?!: Understanding the negative sentiment reflected in tweets with racist hashtags across the development of covid-19.  arXiv preprint arXiv:2005.08224, 2020.
  * [31] Raquel Recuero, Ricardo Araujo, and Gabriela Zago.  How does social capital affect retweets?  In Fifth International AAAI Conference on Weblogs and Social Media, 2011.
  * [32] Luca Rossi and Matteo Magnani.  Conversation practices and network structure in twitter.  In Proceedings of the International AAAI Conference on Web and Social Media, volume 6, pages 563–566, 2012.
  * [33] Jeffrey M Rzeszotarski, Emma S Spiro, Jorge Nathan Matias, Andrés Monroy-Hernández, and Meredith Ringel Morris.  Is anyone out there? unpacking q&a hashtags on twitter.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 2755–2758, 2014.
  * [34] Chengcheng Shao, Giovanni Luca Ciampaglia, Alessandro Flammini, and Filippo Menczer.  Hoaxy: A platform for tracking online misinformation.  In Proceedings of the 25th international conference companion on world wide web, pages 745–750, 2016.
  * [35] Karishma Sharma, Yizhou Zhang, and Yan Liu.  Covid-19 vaccine misinformation campaigns and social media narratives.  In Proceedings of the International AAAI Conference on Web and Social Media, volume 16, pages 920–931, 2022.
  * [36] Kumar Subramani, Alexander Velkov, Irene Ntoutsi, Peer Kroger, and Hans-Peter Kriegel.  Density-based community detection in social networks.  In 2011 IEEE 5th International Conference on Internet Multimedia Systems Architecture and Application, pages 1–8. IEEE, 2011.
  * [37] Lauren Vogel.  Viral misinformation threatens public health, 2017.
  * [38] Yuxi Wang, Martin McKee, Aleksandra Torbica, and David Stuckler.  Systematic literature review on the spread of health-related misinformation on social media.  Social science & medicine, 240:112552, 2019.
  * [39] Echo L Warner, Juliana L Barbati, Kaylin L Duncan, Kun Yan, and Stephen A Rains.  Vaccine misinformation types and properties in russian troll tweets.  Vaccine, 40(6):953–960, 2022.
  * [40] Zi Yang, Jingyi Guo, Keke Cai, Jie Tang, Juanzi Li, Li Zhang, and Zhong Su.  Understanding retweeting behaviors in social networks.  In Proceedings of the 19th ACM international conference on Information and knowledge management, pages 1633–1636, 2010.
  * [41] Sixie Yu, Yevgeniy Vorobeychik, and Scott Alfeld.  Adversarial classification on social networks.  arXiv preprint arXiv:1801.08159, 2018.
  * [42] Caleb Ziems, Bing He, Sandeep Soni, and Srijan Kumar.  Racism is a virus: Anti-asian hate and counterhate in social media during the covid-19 crisis.  arXiv preprint arXiv:2005.12423, 2020.


# Enhanced Neural Beamformer with Spatial Information for Target Speech
Extraction

Aoqi Guo1, Junnan Wu2, Peng Gao2, Wenbo Zhu2, Qinwen Guo1, Dazhi Gao1,* and
Yujun Wang2 1 Ocean University of China, China 2 Xiaomi Inc., Beijing, China
E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>Work conducted when the first
author was intern at Xiaomi Inc.* Corresponding author.

###### Abstract

Recently, deep learning-based beamforming algorithms have shown promising
performance in target speech extraction tasks. However, most systems do not
fully utilize spatial information. In this paper, we propose a target speech
extraction network that utilizes spatial information to enhance the
performance of neural beamformer. To achieve this, we first use the UNet-TCN
structure to model input features and improve the estimation accuracy of the
speech pre-separation module by avoiding information loss caused by direct
dimensionality reduction in other models. Furthermore, we introduce a multi-
head cross-attention mechanism that enhances the neural beamformer’s
perception of spatial information by making full use of the spatial
information received by the array. Experimental results demonstrate that our
approach, which incorporates a more reasonable target mask estimation network
and a spatial information-based cross-attention mechanism into the neural
beamformer, effectively improves speech separation performance.

##  1 Introduction

The target speech extraction task is derived from the speech separation task,
which aims to extract the speech of a specific target speaker from a mixed
signal. Unlike conventional speech separation, which deals with the
permutation order of signals, target speech extraction only focuses on
extracting the target signal, thus avoiding the permutation problem [1]. In
recent years, multi-channel beamforming algorithms have shown great potential
in the task of target speech extraction. Beamforming algorithms achieve
spatial domain filtering of the observed signal by enhancing the signal in the
desired direction and suppressing the signal in other directions, which
naturally meets the task goal of target speech extraction. Classical
beamforming algorithms require accurate estimation of the wave direction and
determination of the speech or noise part in the mixed signal [2], which is
difficult to obtain in noisy and reverberant environments using traditional
algorithms. Therefore, obtaining accurate target orientation information in
real scenes, improving the estimation accuracy of the covariance matrix, and
solving the numerical instability problem in matrix operations are the key to
improving beamforming performance.

In recent years, the development of deep learning technology has provided new
research ideas for solving the above problems. First, using neural networks
for wave direction estimation has achieved good experimental results [3],
providing more accurate target orientation information for beamforming
algorithms. At the same time, combining neural networks with beamforming
algorithms improves the estimation accuracy of the covariance matrix and
effectively improves the performance upper limit of beamforming algorithms.
Among them, Mask Based Beamforming, as a classic representative of the
combination of deep learning and beamforming algorithms, uses a neural network
to predict the mask of speech and noise to calculate the spatial covariance
matrix of speech and noise, respectively, and then inputs it into traditional
beamforming algorithms such as GEV or MVDR for numerical calculation [4], [5].
For the numerical instability problem that traditional beamforming algorithms
still have difficulty in solving even after diagonal loading [6] in matrix
numerical operations, ADL-MVDR [7] used an RNN to replace the matrix inversion
and eigendecomposition processes in the MVDR beamforming algorithm, avoided
the numerical instability caused by singular matrices in matrix operations,
and achieved significant performance improvement for the neural beamformer. In
recent experiments, the combination of deep learning and beamforming
algorithms has been further deepened. GRNNBF [8] used an RNN to model the
covariance matrices of speech and noise, replaced all matrix numerical
operations in traditional beamforming algorithms, and directly predicted
beamforming weights using neural networks, achieved good separation results.
SARNN [9] added self-attention mechanism to the beamforming module, enhanced
the neural network’s modeling ability for the spatial and temporal information
contained in the covariance matrix. EABNet [10] adopted a more aggressive
approach, directly modelled the array observation signals to obtain higher-
dimensional information representations than the covariance matrix. The neural
network implicitly includes all the calculation steps of beamforming and
directly outputs beamforming weights. The above methods combined with deep
learning technology have all achieved good performance improvements for
beamforming algorithms.

In scenarios such as in-car and remote conferences, the position of the target
speaker relative to the microphone array is often fixed, and its angle
information is easily obtained through the device [1]. To leverage the angle
information of the target sound source, Chen et al. proposed the wave
direction feature calculated using the angle information of the target sound
source, which enhanced the directionality of the speech separation system
[11]. ADL-MVDR [7], GRNNBF [8], SARNN [9], UFE [12], etc. inputted the angle
feature into the neural network, improved the separation module’s perception
of spatial information, thereby improved the separation performance of the
system. Gu et al. extended the angle feature from a two-dimensional plane to a
three-dimensional space, improved the separation performance of the model when
the azimuth angle between the target sound source and the interference source
was small [13].

In this paper, we designed a model that uses target spatial orientation
information to enhance the performance of neural beamforming. The system
consists of a front-end pre-separation module and a back-end beamforming
module. Compared with other models, our improvements mainly focus on the
following two points:

$\bullet$ To improve the estimation accuracy of the spatial covariance
matrices of speech and noise, we stack the input features in the channel
dimension and use the UNet-TCN structure to model the input features. This
avoids the loss of feature information caused by directly reducing the input
features after connecting them in the frequency dimension.

$\bullet$ To better utilize the spatial information obtained by the array, we
transform the spatial features of the input signal and the covariance matrix
calculated by the pre-separation module into a feature space with the same
dimensionality. We then use a cross-attention mechanism to enhance the
beamforming network’s perception of spatial orientation, thereby improving the
prediction accuracy of the beamforming weights.

Experimental results show that after the improvements to the pre-separation
module and beamforming module mentioned above, the target speech separation
performance of the neural beamformer could be effectively improved.

The remainder of this paper is organized as follows. Section II presents the
signal model for the target speech extraction task and the beamforming
algorithm. Section III details our proposed neural beamforming algorithm.
Section IV provides an overview of the experimental setup and presents an
analysis of the experimental results. Finally, Section V concludes the paper.

##  2 Signal Modeling and Beamforming

Figure 1: General structure of a neural beamformer.

Consider the far-field time-domain signal model of the real scene, described
as:

$y(t)=x(t)*a_{1}(t)+n(t)*a_{2}(t)+s(t)$ (1)
$=x^{{}^{\prime}}(t)+n^{{}^{\prime}}(t)+s(t)$ (2)

where $y(t)=[y^{(0)}(t),y^{(1)}(t),...,y^{(M-1)}(t)]^{T}$ indicating the time-
domain signal received by the M-channel microphone array. $x(t),n(t)$
represent the reverberation-free speech signals from the target speaker and
the interfering speaker, respectively. $a_{1}(t),a_{2}(t)$ are the room
impulse response(RIR) between the speaker and the array elements of the
microphone. $*$ denotes the convolution operation.
$x^{{}^{\prime}}(t),n^{{}^{\prime}}(t)$ represent the speech signals of the
target speaker and the interfering speaker with reverberation, respectively.
$s(t)$ is the background noise. When not focusing on the dereverberation task,
the task goal of target speech extraction is to extract the target speaker’s
speech $x^{{}^{\prime}}(t)$ from the noisy signal $y(t)$.

After the signal model is transferred to the time-frequency domain by short-
time fourier transform(STFT), it is expressed as:

$Y(t,f)=X(t,f)+N(t,f)+S(t,f)$ (3)

The purpose of the beamforming algorithm is to obtain the filter weight $w$
for the array observation signal, and extract the desired signal from the
array observation signal by performing spatial filtering on the observation
signal, that is:

$X(t,f)=w^{H}Y(t,f)$ (4)

where $(\cdot)^{H}$ denotes the conjugate transpose operation.

Taking the traditional MVDR beamforming algorithm as an example, the purpose
is to minimize the output noise power of the beamforming algorithm without
distorting the signal, that is:

$min(w^{H}\Phi_{NN}w^{H}),\quad s.t\quad w^{H}\alpha(\theta)=1$ (5)

For the above formula, the desired filter weight $w$ is solved by the lagrange
multiplier method:

$w=\frac{\Phi_{NN}^{-1}\alpha(\theta)}{\alpha^{H}(\theta)\Phi_{NN}^{-1}\alpha(\theta)}$
(6)

where $\Phi_{NN}=N(t,f)N^{H}(t,f),\Phi_{NN}\in M\times M$ is the covariance
matrix of the noise signal,
$\alpha(\theta)=[e^{-j\omega\tau_{m}}],m=[0,1,...,M-1]$ is the steering vector
that calculated using target sound source’s DOA. When the steering vector is
difficult to obtain due to the interference of environmental factors, PCA can
be performed on the covariance matrix of the speech signal to obtain the
estimated value of the steering vector [14].

Figure 2: The overall structure of our proposed model.

Both the input features and the covariance matrix are stacked along the
channel dimension when inputting to the network.

Figure 3: The combination of input features. Figure 4: Pre-separator based
UNet-TCN structure

Therefore, it can be seen that the key to the beamforming algorithm lies in
the estimation of the covariance matrix. Traditional beamforming methods often
rely on certain algorithms to determine the time periods of speech and noise
in the signal, but these approach is not optimal. With the development of deep
learning technology, using deep neural networks to determine the speech and
noise components in the signal has achieved good experimental results.

Fig. 1 shows the general framework of the combination of neural network and
beamforming. It first predicts a set of time-frequency masks representing the
corresponding relationship between the desired signal and the original mixed
signal through the neural network, such as IBM [4], IRM [5], CRM [15], CRF
[16], etc. Taking the IRM as an example, it defines the energy ratio
relationship between the desired signal and the mixed signal:

$IRM_{x}=\frac{|X(t,f)|}{|Y(t,f)|}$ (7)

Figure 5: A cross attention module with spatial information as Query and
covariance matrix information as Key and Value. All features are on the time
frame level.

Then the speech and noise components in the mixed signal can be recovered
through mask, so as to calculate the speech and noise space covariance matrix
respectively [4]:

$\Phi=\frac{\sum_{t=1}^{T}M^{2}(t,f)Y(t,f)Y^{H}(t,f)}{\sum_{t=1}^{T}M^{2}(t,f)}$
(8)

Once the covariance matrix of the target signal and noise is calculated, it
can be substituted into traditional beamforming algorithms such as GEV and
MVDR to obtain the beamforming weight $w$. Recent experiments have
demonstrated that utilizing neural networks to model covariance matrix
information and predict beam weights can lead to improved target speech
separation compared to traditional algorithms.

##  3 Enhanced Neural Beamformer with Spatial Information

Fig. 2 shows the overall architecture of our proposed model. First, the
observation signal of the array is transformed into the time-frequency domain
by STFT to extract the spatial features of the signal. The pre-separation
network then estimates the mask of the target speech and noise based on the
input features and calculates the covariance matrix. Following this, the
beamforming network leverages the spatial features to predict the beamforming
weight, which is subsequently used to beamform the observed signal. Finally,
the original time domain signal is restored through ISTFT. Specifically as
follows:

###  3.1 Feature Extraction

The first microphone from the multi-channel mixed signal is selected as the
reference microphone, and its received signal’s amplitude spectrum is
calculated as the input feature for the pre-separation network.

$Y^{mag}_{1}(t,f)=|Y_{1}(t,f)|$ (9)

To obtain the spatial information of the microphone array, we select P pairs
of microphones and calculate the phase difference between each pair as the
spatial feature for the input of our model. This approach has been widely used
in speech enhancement and source separation tasks, as it provides a concise
and effective way to capture the spatial information of the microphone array.

$IPD_{i,j}(t,f)=\angle Y_{i}(t,f)-\angle Y_{j}(t,f)$ (10)
$i,j\in(0,1,...,M-1),i\neq j$

In order to enhance the spatial information acquired by the model, the cosine
distance between the phase delay $\varphi_{1}$ of the signal in the desired
direction on the array and the phase difference $\varphi_{2}$ observed by a
certain pair of microphones is calculated. Then sum over all P pairs of
microphones. That is, the angular characteristics of the target signal serve
as supplementary input.

$AF(\theta,f)=\sum_{p=1}^{P}cos(IPD_{i,j}(t,f)-\frac{2\pi fdcos\theta}{c})$
(11)

When the phase angle of the mixed signal and the desired signal is closer, the
value of the angle characteristic tends to 1 approximately [12]. All these
features are stacked along the channel dimension to serve as the network’s
input features.

$Feature=[Y^{mag}_{1}(t,f),cosIPD_{i,j}(t,f),AF(\theta,f)]$

###  3.2 Pre-separation Module

The pre-separation network adopts an Encoder-TCN-Decoder structure, which
includes 2D convolutional layers, linear layers, and a variant of TCN [17]. To
process the input features, it’s a common practice to concatenate them on the
frequency dimension, followed by 1D convolution to reduce dimensionality
before feeding them into the TCN or RNN module [7], [8], [9]. However, to
avoid the loss of feature information due to direct dimensionality reduction
of the input features, we employ the UNet-TCN structure to model them. By
stacking the input features along the channel dimension, we obtain
$Y_{input}\in R^{N\times F\times T}(N=P+1+1)$, which is then fed into the
UNet-TCN layer to model the nonlinear mapping of the time dimension from the
source signal domain to the separated signal domain and noise domain. Finally,
the predicted cRM is restored through the linear layer to recover the target
speech.

$X_{cRM}(t,f)=(cRM_{r}+jcRM_{i})(Y_{r}+jY_{i})$ (12)

The noise signal is also calculated in the above way. All activation functions
in the pre-separation network use PReLU to accelerate the convergence speed of
the network. Compared with ReLU, it allows negative values to appear, which is
synergistic with the negative values between the input phase difference and
the output mask.

###  3.3 Neural Beamforming Module

The beamforming network is composed of a recurrent neural network and a cross-
attention module. First, the speech and noise signals generated by the pre-
separation of the front-end network are used to calculate the spatial
covariance matrix $\Phi_{SS}(t,f)\in C^{M\times M},\Phi_{NN}(t,f)\in
C^{M\times M}$ at the time frame level respectively.

$\Phi_{SS}(t,f)=X_{cRM}(t,f)X_{cRM}^{H}(t,f)$ (13)
$\Phi_{NN}(t,f)=N_{cRM}(t,f)N_{cRM}^{H}(t,f)$ (14)

Since the spatial covariance matrix is a complex-valued Hermitian matrix, and
the defined neural network is a real network, the real and imaginary parts of
the speech and noise signal covariance matrices are concatenated on the
channel dimension, and LayerNorm is used to the covariance matrix of is
standardized. In order to enhance the spatial constraints on the beamformer,
we use the spatial information received by the array, that is, the phase
difference between channels and the angle characteristics of the target sound
source, to perform a cross attention mechanism on the covariance matrix of
speech and noise. First, the linear layer and GRU are used to transform the
spliced covariance matrix and cosIPD, Angle-Feature(AF) to model the inter-
channel information at the time frame level and transform it into embedding of
the same dimension. Subsequently, the modeled spatial information is used as
the query of multi-head attention, and the covariance matrix information is
used as the key and value, and the spatial orientation information is used to
enhance the ability of attention to model inter-channel information at the
time frame level.

$Query=RNN-DNN(cosIPD,AF)$ (15) $Key,Value=RNN-DNN(\Phi_{SS},\Phi_{NN})$ (16)
$Attention(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V$ (17)

Then, the observed signals from the microphones are beamformed using the
beamforming weights output by the linear layer to obtain the spectrogram of
the model prediction output. The time-domain signal is then recovered through
ISTFT, resulting in the desired time-domain speech signal.

##  4 Experiment setup and result analysis

###  4.1 Dataset settings

The source speech signals for the target and interfering speakers were
obtained from the Aishell2 [18] dataset, while the background noise data was
sourced from DNS2023 [19]. For an indoor reverberation scene, the room size
was set to a random size between [3, 3, 1.5] and [8, 8, 2.5] meters for
length, width, and height, and the reverberation time(rt60) was set to 0.1-0.6
seconds. The simulation was carried out using the image source method,
generating room impulse responses for the target and interference signals
[20]. The microphone array consisted of a four-element linear array with an
element spacing of 3 cm. To ensure a certain degree of spatial independence in
the signals, we set the minimum angle between the two sources relative to the
microphone array to 5°. The signal-to-noise ratio between the target and
interference signals was set to [-6, 6] db. To enhance the model’s robustness,
a background noise of [-5, 20] db was added to the simulated data. In total,
the simulation generated 120,000 pieces of training set data, 14,000 pieces of
validation set data, and 7,000 pieces of test set data. All audio was down-
sampled to 16kHz and each piece of data is 4 seconds long.

###  4.2 Experiment settings

During the training process, the window length of STFT and the number of FFT
points are set to 32ms, and the frame shift is set to 16ms. For the array, we
select (0,1), (0,2), (0,3) three microphone pairs to calculate cosIPD.
Therefore, the channel dimension of the input feature is 5, that is
$Feature\in R^{5\times F\times T}$. For the pre-separation module, the number
of channels of Conv2D in the UNet structure is (5, 32, 64, 128). The TCN
structure is a 3x8 stacked TCN Block with 128 channels. For the beamforming
module, the number of hidden layer units of the double-layer GRU network is
256, the dimension of the cross attention module is 128, and the last linear
layer outputs complex-valued beamforming weights, so the output dimension is
set to 8 (channels*2), the proposed The model is trained end-to-end.

The experiment uses two A100 graphics cards to train 60 epochs on the
simulation data set, and the Batch Size is set to 24. The Adam optimizer is
used for optimization, the initial learning rate is set to 2e-3 and decays
continuously with the number of epochs, and the decay coefficient is 0.98. The
maximum gradient clipping is set to 10 to speed up the network convergence
process.

We use the MVDR method based on the IRM and the GRNNBF [8] and SARNN[9] models
trained according to the MISO model as the baseline for experimental
comparison. For IRM-MVDR, we use the TCN network to predict the IRM, which is
then substituted into the MVDR algorithm for beamforming. For GRNNBF and
SARNN, the 4x8 TCN Block is used to predict the CRF mask, which is input to
the GRU or self-attention module with 500 hidden units after LayerNorm to
output beamforming weights. For all models, we use the joint loss function of
SI-SDR and MSE for training, and the weights of the two are equal.

###  4.3 Results

Table 1: SI-SDR(DB), PESQ AND STOI RESULTS OF IRM-MVDR, GRNNBF, SARNN AND OUR PROPOSED MODELS. Model |  Para.(M) |  PESQ↑ |  STOI↑ |  SISDR↑
---|---|---|---|---
No processing |  - | 1.148 | 0.563 | -1.76
IRM MVDR[5] | 5.33 | 1.586 | 0.757 | 5.25
GRNNBF[8] | 15.73 | 2.176 | 0.845 | 8.42
SARNN[9] | 16.99 | 2.324 | 0.859 | 9.48
Proposed | 8.64 | 2.499 | 0.88 | 10.51
Table 2: SI-SDR(DB), PESQ AND STOI RESULTS OF ABLATION EXPERIMENTS WITH OUR PROPOSED MODEL Model |  Para.(M) |  PESQ↑ |  STOI↑ |  SISDR↑
---|---|---|---|---
TCN IRM (cat) | 5.33 | 1.586 | 0.755 | 5.95
\+ UNet IRM (stack) | 6.76 | 1.846 | 0.799 | 6.82
TCN cRM (cat) | 5.4 | 1.591 | 0.759 | 5.97
\+ UNet cRM (stack) | 7.09 | 1.851 | 0.801 | 6.86
\+ BF(MHSA 128) | 8.05 | 2.466 | 0.876 | 10.11
\+ BF(MHSA 256) | 8.89 | 2.481 | 0.878 | 10.41
\+ BF(MHCA) | 8.64 | 2.499 | 0.88 | 10.51

(a) Noisy, SI-SDR: -3.4dB

(b) IRM-MVDR, SI-SDR: 5.4dB

(c) GRNNBF, SI-SDR: 6.91dB

(d) SARNN, SI-SDR: 7.81dB

(e) Proposed, SI-SDR: 9.97dB

(f) Target

Figure 6: Spectrum of a piece of data in the test set after model processing.
SIR=-2.2dB, SNR=-4.8dB. The angle between the speaker is 9°.

Table 1 shows the experimental results of our proposed model and baseline
model on the test set. In terms of model parameters, our model has increased
compared with IRM-MVDR, which is mainly due to the difference in the
prediction mask and the replacement of the MVDR algorithm with a neural
beamforming module. Compared with GRNNBF and SARNN, our parameter reduction is
more obvious. This is because after adding the cross-attention mechanism based
on spatial information, the model does not require a large number of hidden
layer units to achieve a satisfactory separation effect. Therefore, the
experimental results show that our proposed model can significantly improve
the separation performance of the target speech compared with the baseline
while reducing the number of parameters.

In Table 2, we present the results of our ablation experiments on the proposed
model. The experimental results show that for the pre-separation module, the
pre-separation effect obtained by first encoding the input features through
the Unet structure and then inputting them to the TCN block to output the mask
is better than directly inputting the input features to the TCN through
Conv1D. For the neural beamformer, the UNet-TCN structure makes the model more
fully utilize the input feature information, reduces the information loss in
the dimension reduction process, and makes the estimation of the covariance
matrix more accurate. In terms of mask prediction, whether it is TCN or UNet-
TCN structure, the effect of network prediction cRM on recovering speech
signals is slightly better than that of IRM, but the gap is not large. This is
because for frequency-domain speech separation, when the frame length is 32ms,
the effect of phase estimation on the separation effect is limited [21].

For the neural beamforming module, we take multi-head self-attention(MHSA)
module with 128 and 256 hidden layer units as an example. It is evident that
incorporating the embedding modeled by the covariance matrix and spatial
information into the multi-head cross-attention(MHCA) module leads to better
speech separation performance than the MHSA module with an equivalent number
of hidden layers. Experimental results demonstrate the effectiveness of
introducing spatial information into the beamforming module via the cross-
attention mechanism.

###  4.4 Spectrogram analysis

We analyze and compare the spectrograms of the target speech generated using
different separation methods, and discuss the results further. In order to
verify the robustness of the algorithm, we selected a piece of data with a low
SNR and a small angle between the target speaker and the interference speaker
from the test set for visualization, as shown in Fig. 6. Due to numerical
instability in the training process and the influence of the MVDR algorithm
itself, the IRM-MVDR model has a poor suppression effect on interference
noise, and the separated spectrum is severely damaged by noise. In GRNNBF and
SARNN using neural beamers, the separation performance of the system has been
improved to a certain extent, and both have a good suppression effect on
background noise, but there may still be some residual interference noise
present in the separated signal. After the proposed model uses the UNet-TCN
structure to improve the pre-separation network and adds spatial information
to the beamforming module, the background and interference noise are further
suppressed, and the degree of spectral distortion is lower than that of the
baseline model, achieved better experimental results.

###  4.5 Beam pattern

To further analyze the spatial filtering performance of the model, Fig. 7
visualizes the beam patterns of different neural beamformers, including
GRNNBF, SARNN and our proposed model. The speech is divided into four
segments, segment (a) has only background noise, segment (b) has two speakers
speaking at the same time, segment (c) has only the interfering speaker
active, segment (d) has only the target speaker active. All beam patterns are
averaged over the frequency dimension.

All three models have a strong suppression near 90° when there is no sound
source activity, which is a little strange, we don’t know if this is due to
the influence of background noise. When two sound sources speak at the same
time, all three models produce a strong suppression effect on the interference
direction. Our proposed model suppresses the disturber to a greater extent
when only the disturber is active. When only the target sound source is
active, the directivity of our proposed model is better than that of GRNNBF
and SARNN, which reduces the degree of distortion of the model-predicted
spectrum.

##  5 Conclusion

In this study, we propose a novel UNet-TCN structure to model input features,
resulting in improved estimation accuracy of the covariance matrix.
Furthermore, we introduce spatial information through the cross-attention
module in the neural beamforming module to enhance beamformer performance,
enabling better noise rejection and reduced spectral distortion. Objective
analysis and subjective evaluation show significant improvements in separation
effect. In the future, we plan to achieve better target speech extraction
while reducing model complexity and output spectrum distortion.

Figure 7: An example of beam patterns derived by GRNNBF, SARNN and our
proposed module. The DOA of the target speaker and the interference speaker
are 79 and 164, and marked with red and green respectively.(a) is a silent
segment, (b) is two speakers are active at the same time, (c) is only the
interference speaker, (d) is only the target speaker.

## References

  * [1] Katerina Zmolikova et al.  “Neural Target Speech Extraction: An overview”  In _IEEE Signal Processing Magazine_ 40.3 IEEE, 2023, pp. 8–29
  * [2] Sharon Gannot, Emmanuel Vincent, Shmulik Markovich-Golan and Alexey Ozerov  “A consolidated perspective on multimicrophone speech enhancement and source separation”  In _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 25.4 IEEE, 2017, pp. 692–730
  * [3] Pierre-Amaury Grumiaux, Srđan Kitić, Laurent Girin and Alexandre Guérin  “A survey of sound source localization with deep learning methods”  In _The Journal of the Acoustical Society of America_ 152.1 AIP Publishing, 2022, pp. 107–151
  * [4] Hakan Erdogan et al.  “Improved mvdr beamforming using single-channel mask prediction networks.”  In _Interspeech_ , 2016, pp. 1981–1985
  * [5] Takuya Higuchi, Nobutaka Ito, Takuya Yoshioka and Tomohiro Nakatani  “Robust MVDR beamforming using time-frequency masks for online/offline ASR in noise”  In _2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2016, pp. 5210–5214  IEEE
  * [6] Jian Li, Petre Stoica and Zhisong Wang  “On robust Capon beamforming and diagonal loading”  In _IEEE transactions on signal processing_ 51.7 IEEE, 2003, pp. 1702–1715
  * [7] Zhuohuang Zhang et al.  “ADL-MVDR: All deep learning MVDR beamformer for target speech separation”  In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021, pp. 6089–6093  IEEE
  * [8] Yong Xu et al.  “Generalized spatio-temporal RNN beamformer for target speech separation”  In _arXiv preprint arXiv:2101.01280_ , 2021
  * [9] Xiyun Li et al.  “Mimo self-attentive rnn beamformer for multi-speaker speech separation”  In _arXiv preprint arXiv:2104.08450_ , 2021
  * [10] Andong Li, Wenzhe Liu, Chengshi Zheng and Xiaodong Li  “Embedding and beamforming: All-neural causal beamformer for multichannel speech enhancement”  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 6487–6491  IEEE
  * [11] Zhuo Chen et al.  “Multi-channel overlapped speech recognition with location guided speech extraction network”  In _2018 IEEE Spoken Language Technology Workshop (SLT)_ , 2018, pp. 558–565  IEEE
  * [12] Jian Wu et al.  “An end-to-end architecture of online multi-channel speech separation”  In _arXiv preprint arXiv:2009.03141_ , 2020
  * [13] Rongzhi Gu, Shi-Xiong Zhang, Meng Yu and Dong Yu  “3D Spatial Features for Multi-Channel Target Speech Separation”  In _2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , 2021, pp. 996–1002  IEEE
  * [14] Mehrez Souden, Jacob Benesty and Sofiene Affes  “On optimal frequency-domain multichannel linear filtering for noise reduction”  In _IEEE Transactions on audio, speech, and language processing_ 18.2 IEEE, 2009, pp. 260–276
  * [15] Donald S Williamson, Yuxuan Wang and DeLiang Wang  “Complex ratio masking for monaural speech separation”  In _IEEE/ACM transactions on audio, speech, and language processing_ 24.3 IEEE, 2015, pp. 483–492
  * [16] Hendrik Schroter, Alberto N Escalante-B, Tobias Rosenkranz and Andreas Maier  “DeepFilterNet: A low complexity speech enhancement framework for full-band audio based on deep filtering”  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 7407–7411  IEEE
  * [17] Scott Wisdom et al.  “Unsupervised sound separation using mixture invariant training”  In _Advances in Neural Information Processing Systems_ 33, 2020, pp. 3846–3857
  * [18] Jiayu Du, Xingyu Na, Xuechen Liu and Hui Bu  “Aishell-2: Transforming mandarin asr research into industrial scale”  In _arXiv preprint arXiv:1808.10583_ , 2018
  * [19] Harishchandra Dubey et al.  “Icassp 2023 deep speech enhancement challenge”  In _arXiv preprint arXiv:2303.11510_ , 2023
  * [20] Robin Scheibler, Eric Bezzam and Ivan Dokmanić  “Pyroomacoustics: A python package for audio room simulation and array processing algorithms”  In _2018 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , 2018, pp. 351–355  IEEE
  * [21] Tal Peer and Timo Gerkmann  “Phase-aware deep speech enhancement: It’s all about the frame length”  In _JASA Express Letters_ 2.10 Acoustical Society of America, 2022, pp. 104802