# Graph-based Joint Pandemic Concern and Relation Extraction on Twitter

Jingli Shi
Engineering, Computer and Mathematical Sciences
Auckland University of Technology
Auckland, New Zealand
<EMAIL_ADDRESS>
&Weihua Li
Engineering, Computer and Mathematical Sciences
Auckland University of Technology
Auckland, New Zealand
<EMAIL_ADDRESS>
&Sira Yongchareon
Engineering, Computer and Mathematical Sciences
Auckland University of Technology
Auckland, New Zealand
<EMAIL_ADDRESS>
&Yi Yang
Applied Artificial Intelligence Institute
Deakin University
Victoria, Australia
<EMAIL_ADDRESS>
&Quan Bai
Information and Communication Technology
University of Tasmania
Hobart, Australia
<EMAIL_ADDRESS>

###### Abstract

Public concern detection provides potential guidance to the authorities for
crisis management before or during a pandemic outbreak. Detecting people’s
concerns and attentions from online social media platforms has been widely
acknowledged as an effective approach to relieve public panic and prevent a
social crisis. However, detecting concerns in-time from massive information in
social media turns out to be a big challenge, especially when sufficient
manually labelled data is in the absence of public health emergencies, e.g.,
COVID-19. In this paper, we propose a novel end-to-end deep learning model to
identify people’s concerns and the corresponding relations based on Graph
Convolutional Network and Bi-directional Long Short Term Memory integrated
with Concern Graph. Except for the sequential features from BERT embeddings,
the regional features of tweets can be extracted by the Concern Graph module,
which not only benefits the concern detection but also enables our model to be
high noise-tolerant. Thus, our model can address the issue of insufficient
manually labelled data. We conduct extensive experiments to evaluate the
proposed model by using both manually labelled tweets and automatically
labelled tweets. The experimental results show that our model can outperform
the state-of-art models on real-world datasets.

_Keywords_ Concern detection $\cdot$ COVID-19 $\cdot$ Auto Concern Extraction
$\cdot$ Concern Graph $\cdot$ Concern Relation Extraction $\cdot$ Graph
Convolutional Network $\cdot$ Bi-directional Long Short Term Memory

##  1 Introduction

The outbreak of coronavirus (COVID-19) in 2019 has been causing a rapid
increase in both infection and death rates around the world. Especially when
the pandemic moved into the second wave, it caused devastating loss of human
lives, impacted the global economy, transformed our daily lives, and posed a
threat to our society (Killgore et al. (2020)). According to the studies on
the past pandemic outbreaks, e.g., Zika, Ebola, and H1N1, social media
platforms, e.g., Twitter, have proven to be a popular channel for spreading
information, especially related to public opinions and concerns (Damiano and
Catellier JR (2020)). This is because people intend to perceive more details
regarding the pandemic by reading the newsfeeds and interpreting the comments
from others through social networks (Li et al. (2018); Hu et al. (2019)).
Twitter, a popular and informative social network platform, allows people to
post and interact with messages known as “tweets". They can also communicate
and express opinions about the latest events (Killgore et al. (2020)). User-
generated tweets from Twitter turn out to be “prophetic", namely, valuable
indicators of what issues will likely happen in the pandemic. It is important
to make use of tweets and investigate what various people are discussing
during the pandemic. The attitudes and behaviours of our society are affected
directly by public concerns. Thus, how to effectively extract public concerns
and analyse the corresponding relationships will assist people in
understanding the anxiety and fears of the society in this pandemic situation.
Furthermore, the potential social crisis also can be revealed by analysing
public concerns, which significantly contributes to social management control.

Motivated by this background, a great many efforts have been dedicated to
mining social media data and exploring the opinions towards pandemic outbreaks
(da Silva et al. (2020)). Most existing research works can be categorised into
traditional survey methods, i.e., survey and questionnaire (Nelson et al.
(2020) and machine learning model-based methods, i.e., topic modelling (van
der Vegt and Kleinberg (2020); Kassab et al. (2020)). The existing studies are
capable of extracting fundamental public concerns, e.g., “social distancing",
“hand sanitizer" and “face masks", which require intensive human-effort in
labelling large datasets, turning out to be inefficient. Moreover, in any
epidemic emergence situation, e.g., COVID-19, traditional approaches, such as
questionnaires and clinical tests, neither collect enough data for deep
learning model training nor rapidly generate a model for concern detection.
Therefore, it is vital to design an end-to-end model that is capable of
automatically analysing social media data and detecting public concerns
without requiring a large-scale of data to be labelled manually.

Deep learning methods are increasingly applied to valuable information
extraction. However, most methods highly rely on data labelled by the
annotators, requiring much time and financial resources (Kipf and Welling
(2017)). Moreover, the noisy and imbalanced social media data prevent deep
learning-based methods from generalisation (Rathan et al. (2018)). In many
existing studies, the proposed models are not able to track real-time
statistics of public concerns related to pandemics due to the required
labelled dataset (Li et al. (2020); Jahanbin and Rahmanian (2020); Hou et al.
(2020); Lazard et al. (2015)). To mitigate this issue, we have conducted
preliminary research to mine public concerns by proposing an Automated Concern
Exploration (ACE) framework (Shi et al. (2020)). The proposed framework can
detect concerns from tweets automatically and construct a concern knowledge
graph to present the interconnections of the extracted concern entity set.
However, several advent limitations are still to be addressed. (1) only BERT
embedding of tweets is used, which cannot capture regional dependency word
features from tweets to improve the performance of concern extraction. (2) the
relation between concerns in one tweet posted by a user is not detected, which
is critical to reveal meaningful information about public concerns. (3) The
framework employs a rule-based method, having poor generalisability and
appearing difficult to transfer to future occurring pandemics.

In this paper, we propose and develop an end-to-end model with concern graph
and concern states to identify public concerns and corresponding relations
simultaneously. We formally define “public concern" with a consideration of
its type and degree, and construct concern graph to represent the regional
features, improving the concern identification effectiveness. Furthermore, by
integrating concern states with Graph Convolutional Network (GCN) (Kipf and
Welling (2017), the proposed method can extract concern relations. Extensive
experiments are conducted to evaluate the proposed method by using both
manual-labelled and auto-labelled datasets. The experimental results
explicitly demonstrate that our method outperforms state-of-the-art models.

Our contributions in this research work are summarized below:

  * •

We define a concern graph data structure to capture the inherent structural
information of concerns more efficiently.

  * •

We present a novel end-to-end model to jointly extract concerns and relations
consisting of CG and shared state of concerns.

  * •

We evaluate our model on manual-labelled data and auto-labelled data, and the
results indicate the proposed method is effective for auto-labelled data.

##  2 Related Work

In this section, we first review the existing studies related to public
concern mining and detection. Then, modern approaches of Named Entity
Recognition (NER) and Relation Extraction (RE) are inspected and compared
since the concern detection, defined in this paper, tends to explore the
concern entities and the corresponding relations. Finally, we review the GCN
and its variants since GCN has been widely adopted in NER and RE based on
recent studies.

###  2.1 Concern Detection

Social media has become a prevalent platform for people to communicate and
express their opinions. With the outbreaks of the corona-virus pandemic, i.e.,
COVID-19, how to effectively extract people’s opinions and address the public
concern in pandemic situations has attracted great attention to researchers.
Thus, great efforts have been dedicated to the analysis of public response to
pandemics in social media platforms, e.g., Twitter. The current approaches are
mainly categorized into two types of methods: probabilistic model-based and
deep learning-based. In probabilistic-based models, Latent Dirichlet
Allocation (LDA) is commonly used for public concern extractions. For example,
Chandrasekaran et al. conduct a temporal assessment on COVID-19-related
tweets, aiming to uncover public concern trends through extracting topics and
predicting sentiment scores (Chandrasekaran et al. (2020)). Xue et al. utilise
LDA to analyse public response towards COVID-19 pandemic on the social media
platform, which aims to identify popular uni-grams and bi-grams topics from
tweets (Xue et al. (2020)). Wahbeh et al. adopt qualitative analysis tool to
detect recommendations, topics, and opinions related to COVID-19 pandemic from
Twitter (Wahbeh et al. (2020)). Whereas, probabilistic model-based methods
perform poorly on public concern identification since contextual information
is ignored. By contrast, deep learning-based methods are able to retain
contextual features of sentences. Nowadays, deep learning is widely adopted as
a popular approach for many NLP tasks, e.g., sentiment analysis. By employing
such an approach, many studies aim to extract insightful information for
assisting the authorities in making appropriate responses and reactions (Wang
et al. (2020); Yin et al. (2020); Chen et al. (2020)). However, most existing
research works only identify a few pre-defined public concerns but neglect the
relations between the concerns. Without concern relations, it is difficult to
identify the cause of public concerns or reveal people’s thoughts behind the
expressed concerns. Different from the above two types of approaches, our
proposed method is able to capture regional and sequential features of a
sentence and assist the extraction of public concerns with the corresponding
relations.

###  2.2 Named Entity Recognition

NER, also named entity extraction, is one of the classic tasks of Natural
Language Processing (NLP), which aims to identify the classify entities from
unstructured text into pre-defined categories (Mohit (2014)). Recent studies
have shown two typical NER approaches, i.e., traditional statistical models
and deep learning-based methods. Zhou et al. propose an entity extraction
model with a chunk tagger method based on Hidden Markov Model (HMM), and the
model outperforms the hand-crafted rules-based models (Zhou and Su (2002)).
Lafferty et al. present the Conditional Random Fields (CRF), to segment and
label sequence data by building a probabilistic model (Lafferty et al.
(2001)). However, traditional statistical models perform poorly on complex
sentences because they fail to discover hidden features from data. Compared
with traditional methods, deep learning-based approaches are able to learn
latent representations from raw data and achieve promising performance. By
relying on character and word representations, a novel neural architecture is
introduced, which combines bidirectional Long Short-Term Memory (BiLSTM) and
CRF (Lample et al. (2016)). Ma et al. propose a novel deep learning-based
model by combining biLSTM, Convolutional Neural Network (CNN), and CRF (Ma and
Hovy (2016)). Recently, the current SoTA models adopt context-dependent
embeddings, e.g., ELMo ((Peters et al. (2018)), Flair ((Akbik et al. (2018)),
and BERT ((Devlin et al. (2018)), to encode the input. Although deep learning-
based models are capable of capturing contextual features of data, interaction
information between entities is neglected. Different from the above models,
apart from contextual information, we also propose a designated Concern Graph
(CG) to capture specific features of entities, enabling our method to perform
better on Twitter data.

###  2.3 Relation Extraction

As a fundamental task in the NLP field, Relation extraction (RE) aims to
detect and classify the semantic relationship between entity mentions
(Chinchor (1998)). Early research works mainly focus on rule-based models, in
which proper rules are difficult to define without domain knowledge. To
address such an issue, many efforts have been dedicated to kernel-based models
with manual-labelled data (Culotta and Sorensen (2004); Zhou and Zhu (2011);
Seewald and Kleedorfer (2007)). The key weakness of kernel-based methods is
that contextual features are not captured, leading to wrong relation
extraction on data with a long sentence. Recently, deep neural networks are
applied to relation extraction due to their supremacy in terms of accuracy.
Therefore, some popular deep learning models, e.g., CNN, LSTM, GCN, are
utilised to learn contextual features of data and achieve better performance
than kernel-based models (Zeng et al. (2014); Miwa and Bansal (2016); Fu et
al. (2019)).

Apart from the models extracting entity and relation separately, many other
research works exploring joint methods to extract both simultaneously. Arzoo
et al. propose an attention-based RNN model for joint entity mentions and
relations extraction (Katiyar and Cardie (2017)). Zheng et al. present a novel
tagging strategy to covert sequence labelling and classification tasks to a
tagging problem and extract entities and relations directly using the joint
model (Zheng et al. (2017)). Miwa et al. use Tree-LSTM with bidirectional
sequential LSTM to extract entity and relation simultaneously (Miwa and Bansal
(2016)). Zeng et al. propose a sequence-to-sequence model with a copy
mechanism to extract entity and relation (Zeng et al. (2018)). However,
existing NER, RE, and joint entity and relation extraction models suffer from
two issues. First, existing models only discover contextual features of a
sentence and neglect entity features, which is vital for entity extraction.
Second, relation extraction mainly rely on contextual features of a sentence
and the information of entities corresponding to a relation is ignored. This
can be a severe problem for social media data, where numerous grammatical
mistakes exist in sentences. To address these two issues, we combine
contextual and concern features for concern identification and integrate
learned concern features with the module of relation extraction.

###  2.4 Graph Convolutional Network

Graph Convolutional Network (GCN) has demonstrated advent advantages in
capturing the dependency structure of sentences, and it has been widely
adopted in many NLP tasks (Battaglia et al. (2016); Defferrard et al. (2016);
Hamilton et al. (2017)). Hong et al. present a joint model based on GCN to
perform entity and relation extraction by considering context and syntactic
information of sentences (Hong et al. (2020)). Zhang et al. utilise GCN over a
pruned dependency tree to tackle the relation extraction task (Zhang et al.
(2018)). Inspired by the existing studies, we incorporate GCN into the
proposed model to effectively preserve the dependency information of
sentences. Furthermore, concern states are integrated with GCN to improve the
accuracy of relation extraction.

In this paper, we proposed an end-to-end model with a concern graph module to
perform joint extraction of concerns and relations. Meanwhile, we integrate
the concern states from BiLSTM with the input features of BiGCN to enhance the
influences from concerns to improve relation extraction performance.

Figure 1: Bidirectional Graph Convolutional Network (BiGCN) Overview

##  3 Preliminaries

In this section, the relevant definitions are presented, including public
concerns, concern relations, and graph. In addition, the concern detection
problem is formally formulated.

###  3.1 Formal Definition

Definition 1: Concern refers to people worry a real or imagined issue. Public
concern represents a word or a phrase in a tweet towards which most people
express strong opinions about a particular aspect of the pandemic. Given a
concern set $C=\\{c_{1},...,c_{n}\\}$, the $i$th potential concern detected in
tweet $t_{j}$ can be defined as $c_{i}^{j}=(ce_{i}^{j},cs_{i}^{j})$, where
$ce_{i}^{j}\in CE$ is the concern entity identified in tweet $t_{j}$ and it
can be words or phrase, e.g., _“China"_ , _“corona emergency relief"_ and
_“florida medical examiner"_ , and $cs_{i}^{j}$ is concern score of concern
$c_{i}^{j}$ and the value is calculated by use of Equation 1, in which
$rt_{i}^{j}$ represents the retweet count of tweet $t_{j}$.
$sp_{i}^{j}\in[-1,1]$ denotes the sentiment polarity of tweet $t_{j}$, where
-1 indicates an extremely negative attitude, 0 means a neutral attitude and +1
implies an extremely positive attitude. The range of $cs_{i}^{j}$ is [0,1],
where the greater the value is, the more likely it becomes a concern.
$C=\\{c_{i}^{t}|i\in[1,N],t\in T\\}$ denotes the set of public concerns
detected in Twitter dataset $T$. For each concern $c_{i}$, there is one
attribute named type $ct_{i}$, where $ct_{i}\in CT$ and
$CT=\\{ct_{1},...,ct_{n}\\}$ is the set of concern types.

$cs^{j}=(1-\theta)*|sp^{j}|+\theta*{\tilde{rt}}^{j},$ (1)

where $\theta\in[0,1]$ refers to the weight parameter and $\tilde{rt}^{j}$
describes the normalized value of $rt^{j}$.

Definition 2: Concern Relation describes the relationship between public
concern pair. We use $r_{m,n}^{j}\in R$ to present the relation between
concern $c_{m}^{j}$ and $c_{n}^{j}$ in tweet $t_{j}$, where $r_{m,n}^{j}$ is
unidirectional relation, i.e., the same as $r_{n,m}^{j}$, and $R$ is the set
of relation extracted from Twitter dataset $T$.

Definition 3: Concern Triple is the fundamental element of the public concern
graph which is extracted from a tweet. Because some short words or phrases are
very limited in context information to present the real meaning of concern.
Whereas, the concern triple is capable of semantically representing what a
concern is about. A public concern triple in the tweet $t_{j}$,
$ct_{m,n}^{j}=(s_{m}^{j},r_{m,n}^{j},o_{n}^{j})$, has three components, i.e.,
$s_{m}^{j}$, $r_{m,n}^{j}$ and $o_{n}^{j}$, referring to as the subject,
relation, and object of the concern triple, respectively. The $s_{m}^{j}$ and
$o_{n}^{j}$ are extracted entities and $r_{m,n}^{j}$ is the extracted relation
based on dependency parser analysis of the tweet $t_{j}$.

Definition 4: Concern Graph (CG) aims to explore discriminative public
concerns and what kind of relations are existing between concerns. In order to
present the relation of public concerns, Concern Graph (CG) is proposed as the
control signal for capturing public concerns. CG of the tweet $t_{j}$ can be
denoted as $G=(\nu,\varepsilon)$, where $\nu$ is the set of nodes, and
$\varepsilon$ is the edges set. As shown in Figure 2, nodes of CG are
classified into four categories: (1) object $o^{j}$, subject node $s^{j}$; (2)
relation note $r^{j}$; (3) attribute node $a^{j}$ including concern type
$ct^{j}$; (4) concern score $cs^{j}$.

The CG $G$ is constructed via the following steps:

  1. 1.

Detect public concern $c_{i}^{j}$ and add it to $G$, where $c_{i}^{j}$ is
grounded in the tweet $t_{j}$.

  2. 2.

Extract the descriptive details of concern $c_{i}^{j}$ as the attribute
$a_{i,l}^{j}$ including type $ct_{i}^{j}$ and score $cs_{i}^{j}$, then add
them to $G$ and assign an un-directed edge from $c_{i}^{j}$ to $a_{il}^{j}$,
where $|l|$ is the number of attributes towards concern $c_{i}^{j}$.

  3. 3.

Identify the relation $r_{ik}$ between concerns $c_{i}^{j}$ (subject in
concern triple) and $c_{k}^{j}$ (object in concern triple), which is a
unidirectional type of relation. Adding relation node $r_{ik}$ to $G$ and
assigning edges from $c_{i}^{j}$ to $r_{ik}$ and from $r_{ik}$ to $c_{k}^{j}$.

Figure 2: Concern Graph (CG): each concern has two attributes, i.e., type and
score, along with relation to another concern to form the concern graph.

###  3.2 Problem Formulation

In the previous section, we describe the definitions related to our proposed
method. Based on the definitions, our model aims to jointly extract typical
concerns $\\{c_{i}^{j}|c_{i}^{j}\in C\wedge p_{i}^{j}<0\\}$ from tweet $t_{j}$
and concern relations $r_{mn}$ between concern $c_{m}$ and $c_{n}$ from
Twitter dataset $T$ by constructing Concern Graph (CG).

##  4 Graph-based Concern and Relation Extraction

For a tweet $T$, the goal of our method is to identify public concerns
$C=\\{c_{1},...,c_{n}\\}$ and concern relations $R=\\{r_{1},...,r_{n}\\}$. In
this section, the joint extraction of concerns and relations model with
concern graph is illustrated in Figure 3. The proposed method consists of four
main components, i.e., Concern and Relation Extraction (CRE) embedding layer,
Concern and Relation Extraction (CRE) encoding layer, concern decoding layer,
and concern relation extraction layer. We describe each component in detail
below. Concern and Relation Extraction (CRE) Embedding layer is introduced in
Section 4.1, followed by Concern and Relation Extraction (CRE) encoding in
Section 4.2. Concern decoding and concern relation extraction layer are
presented in Sections 4.3 and 4.4, respectively. The model objective function
is explained in Section 4.5.

Input : $T$, $\upsilon_{w}$, $\upsilon_{cg}$, $E$, $B$, $lr$, $d$

$T$ indicates labelled Twitter corpus for model training

$\upsilon_{w}$ represents BERT embedding of Twitter corpus

$\upsilon_{cg}$ is embedding of CG

$E$ is epoch number

$B$ is batch size

$lr$ indicates learning rate

$d$ is embedding dimension

Output : $L$

$L$ is the loss function value

initialization: wight vector $W$;

while  _$ep$ in $EP$_ do

while  _$b$ in $B$_ do

generate embedding $\grave{X^{(b)}}$, $\acute{X}^{(b)}$;

compute concern hidden state ${h_{t}}^{\prime(b)}$ as Equation 8 ;

$L_{(c)}^{(b)}=max(\sum log(P_{(c)}^{(b)}=S_{(c)}^{(b)}))$ ;

$L_{(r)}^{(b)}=max(\sum log(P_{(r)}^{(b)}=S_{(r)}^{(b)}))$ ;

$L^{(b)}=L_{(c)}^{(b)}+\alpha*L_{(r)}^{(b)}$ ;

end while

end while

Algorithm 1 Training Process of CG-CRE Model  Figure 3: The overview of CG-CRE
model.

###  4.1 Concern and Relation Extraction (CRE) Embedding Layer

Since deep learning models are integrated into our method, word tokens and
proposed CG need to be transformed into low-dimensional vectors by embedding
layer. For a tweet $t=\\{w_{1},...,w_{i},...,w_{n}\\}$, where $w_{i}$ denotes
the $i$th word in the tweet. Given the tweet $t$, pre-trained BERT model is
used to generate word embedding set
$\grave{X}=\\{\tilde{e_{1}},...\,\tilde{e_{i}},\,...,\tilde{e_{n}}\,|\,\tilde{e_{i}}\in\mathbb{R}^{d}\\}$,
where $\tilde{e_{i}}$ represents the embedding of word $w_{i}$ and $d$ means
the embedding dimension.

In order to enhance model input features, we further encode proposed Concern
Graph (CG) $G$ to obtain CG node embedding $\acute{x}_{i}^{(0)}$ as below:

$\centering\acute{x}_{i}^{(0)}=\left\\{\begin{matrix}(v_{i}^{(dep)}+v_{i}^{(pos)})\,\odot\,W_{cr}[0],&if\,i\in
C;\\\ v_{i}^{a}\,\odot\,W_{cr}[1],&if\,i\in A;\\\
v_{i}^{r}\,\odot\,W_{cr}[2],&if\,i\in R;\end{matrix}\right.\@add@centering$
(2)

where $v_{i}^{(dep)}$ and $v_{i}^{(pos)}$ denote the syntactic dependency
relation and POS tag feature, respectively. Both $v_{i}^{(dep)}$ and
$v_{i}^{(pos)}$ are used to capture the meaning of tweet and words syntactic
dependency. $C$ represents the concern set. $v_{i}^{a}$ represents the
attribute features including concern type and score. $A$ means attribute set.
$v_{i}^{r}$ indicates relation feature, and $R$ is relation set.
$W(\cdot)\in\mathbb{R}^{3\times d}$ refers to parameters, where $d$ means the
feature dimension.

###  4.2 Concern and Relation Extraction (CRE) Encoding Layer

In order to capture long-distance dependencies and forward and backward
features between tokens in tweets, bidirectional LSTM is used in this paper.
The Bi-LSTM contains forward and backward layers, and a concatenation layer of
backward and forward state information. The embeddings in Section 4.1 are
concatenated as the input of the concern encoder layer. The BiLSTM encoding
layer is defined by using Equations 3 \- 8:

$i_{t}=\sigma(W_{ex}^{(i)}*[\tilde{e_{i}};\acute{x}_{j}^{(i)}]+W_{h}^{(i)}*h_{t-1}+b^{(i)})$
(3)
$f_{t}=\sigma(W_{ex}^{(f)}*[\tilde{e_{i}};\acute{x}_{j}^{(f)}]+W_{h}^{(f)}*h_{t-1}+b^{(f)})$
(4)
$o_{t}=\sigma(W_{ex}^{(o)}*[\tilde{e_{i}};\acute{x}_{j}^{(o)}]+W_{h}^{(o)}*h_{t-1}+b^{(o)})$
(5)
$u_{t}=\sigma(W_{ex}^{(u)}*[\tilde{e_{i}};\acute{x}_{j}^{(u)}]+W_{h}^{(u)}*h_{t-1}+b^{(u)})$
(6) $c_{t}=i_{t}\odot u_{t}+f_{t}\odot c_{t-1}$ (7) $h_{t}=o_{t}\odot
tanh(c_{t}),$ (8)

where $\sigma$ is sigmoid activation function, $W(\cdot)$ refers to weight
parameters. and $[;]$ is a vector concatenation operation. $\tilde{e_{i}}$ and
$\acute{x}_{j}(\cdot)$ denote word embedding and embedding of CG $G$ defined
in Section 4.1. In Equations 3 \- 6, $b(\cdot)$ refers to the bias vector, and
$\odot$ represents element-wise multiplication. $c$ and $h$ denote cell state
and hidden state, respectively, carrying information from the previous layer
to the next layer. Because Bi-LSTM is applied in our method, the hidden state
is obtained by concatenating both direction hidden state, namely, forward
direction $\overrightarrow{{h_{t}}^{\prime}}$ and backward direction
$\overleftarrow{{h_{t}}^{\prime}}$, therefore, the final hidden state can be
denoted as
${h_{t}}^{\prime}=[\overrightarrow{{h_{t}}^{\prime}},\overleftarrow{{h_{t}}^{\prime}}]$.
By passing hidden state to a fully connected neural network, the final output
of Bi-LSTM can be defined as:

$\centering O=W^{o}*{h_{t}}^{\prime}+b^{o},\@add@centering$ (9)

where $W^{o}$ is the output weight parameters and $b^{o}$ is the bias vector.

###  4.3 Concern Decoding Layer

In our model, the CRF is employed to produce a tag sequence since it can
produce a higher tagging accuracy than that of the existing models (Hong et
al. (2020)). For one tweet $t=\\{w_{1},...,w_{n}\\}$, we aim to predict the
concern tag sequence $Y^{(c)}=\\{y^{(c)}_{1},y^{(c)}_{2},...,y^{(c)}_{n}\\}$
where $n$ denotes the number of words and superscript $(c)$ means the notation
of concern. Thus, the CRF score can be defined as in Equation 10:

$\centering
S^{(c)}(t,Y^{(c)})=\sum_{i=1}^{n}O_{i,y^{(c)}_{i}}+\sum_{i=1}^{n}T_{y^{(c)}_{i},y^{(c)}_{i+1}},\@add@centering$
(10)

where $O\in\mathbb{R}^{n\times k}$ indicates the matrix of scores output from
the previous encoding layer with $k$ as the number of distinct tags, and
$O_{i,j}$ denotes the score of the $jth$ tag of the $ith$ word in tweet $t$.
$T$ represents a matrix of transition scores as being introduced in (Huang et
al. (2015), and $T_{i,j}$ means the score of a transition from tag $i$ to tag
$j$. Then, for input tweet $t$, the probability of a given sequence of tags
over the sequence of predicted tags $Y^{(c)}$ is defined by applying the
softmax layer as in Equation 11.

$\centering P^{(c)}=\frac{e^{S^{(c)}(t,Y^{(c)})}}{\sum_{\widetilde{y}^{(c)}\in
Y^{(c)}_{X}}e^{S^{(c)}(t,\widetilde{y}^{(c)})}},\@add@centering$ (11)

In Equation 11, $Y^{(c)}_{X}$ denotes all possible concern tag sequences for
tweet $t$.

###  4.4 Concern Relation Extraction Layer

Given concern set $C=\\{c_{1},...,c_{m}\\}$ in tweet
$t=\\{t_{1},...,t_{n}\\}$, we aim to extract corresponding relation $r_{i}\in
R$. Except for sequential features, BiGCN is utilised to capture regional
features from the tweets. Both forward and backward directions are considered
and the hidden state of BiGCN is defined using Equations - ).

$\overrightarrow{{h_{t}}^{\prime\prime}}=\varsigma(\sum_{v\in\overrightarrow{N(w)}}(\overrightarrow{W_{h}}*[\overrightarrow{{h^{v}_{t-1}}^{\prime\prime}};\overrightarrow{{h_{t-1}}^{\prime}}]+\overrightarrow{b}))$
(12)
$\overleftarrow{{h_{t}}^{\prime\prime}}=\varsigma(\sum_{v\in\overleftarrow{N(w)}}(\overleftarrow{W_{h}}*[\overleftarrow{{h^{v}_{t-1}}^{\prime\prime}};\overleftarrow{{h_{t-1}}^{\prime}}]+\overleftarrow{b}))$
(13)
${h_{t}}^{\prime\prime}=[\overrightarrow{{h_{t}}}^{\prime\prime}\,;\,\overleftarrow{{h_{t}}}^{\prime\prime}],$
(14)

where $\varsigma$ represents ReLU activation function,
${h_{t}}^{\prime\prime}$ refers to the hidden state at $t$th layer and
$\overrightarrow{{h_{t-1}}^{\prime}}$ indicates the shared hidden state from
concern detection module. $\overrightarrow{N(w)}$ describes the neighbours of
word $w$ in the forward direction and $\overleftarrow{N(w)}$ means the
neighbours of word $w$ in the backward direction. $\overrightarrow{W_{h}}$ and
$\overleftarrow{W_{h}}$ represent weight parameters in the forward and
backward direction, respectively. $\overrightarrow{b}$ and $\overleftarrow{b}$
are the bias of the model. $h^{t}$ refers to the final hidden state of word
$w$, concatenating hidden states in both directions.

By using hidden states of BiGCN, the relation tendency score
$S^{(r)}_{(r_{ij}|c_{i},c_{j})}$ is defined in Equation 15.

$S^{(r)}_{(r_{ij}|c_{i},c_{j})}=W^{(r)}*\varsigma(W^{(r)}_{c_{i}}*h^{\prime\prime}_{c_{i}}+W^{(r)}_{c_{j}}*h^{\prime\prime}_{c_{j}}+b^{(r)}),$
(15)

where, superscript $(r)$ means the notation of concern relation.
$S^{(r)}_{(r_{ij}|c_{i},c_{j})}$ represents the tendency score of concern
relation on concerns pair $(c_{i},c_{j})$. $W^{(r)}$, $W^{(r)}_{c_{i}}$ and
$W^{(r)}_{c_{j}}$ are weight parameters. $b^{(r)}$ denotes the bias term. We
apply the activation function (softmax) to the tendency score
$S^{(r)}_{(r_{ij}|c_{i},c_{j})}$ to obtain the probability of relation
$r_{i,j}$ in Equation 16).

$P^{(r)}=\sigma(S^{(r)}_{(r_{ij}|c_{i},c_{j})})$ (16)

###  4.5 Model Objective Function

In this subsection, the final objective function for model training is
described. To train proposed model, we use maximum log-likelihood as the loss
function and maximize combined loss functions of concern and relation by using
Equations 17, 18 and 19.

$L_{(c)}=max(\sum_{i=1}^{|\mathbb{R}_{T}|}\sum_{w=1}^{|W_{i}|}log(P_{w}^{(c)}=S_{w}^{(c)}|t_{i},\Theta))$
(17)

$L_{(r)}=max(\sum_{j=1}^{|\mathbb{R}_{T}|}log(P_{w}^{(r)}=S_{w}^{(r)}|t_{j},\Theta))$
(18)

$L=L_{(c)}+\alpha*L_{(r)}$ (19)

where $|\mathbb{R}_{T}|$ is the size of the training dataset, $t_{i}$ and
$t_{j}$ is the $i$th and $j$th tweet in the training dataset, respectively.
$|W_{i}|$ is the sentence length. $\alpha\in[0,1]$ is a trade-off coefficient
between loss of concern and concern relation, and the larger value means the
greater influence of concern relation on the proposed method.

##  5 Experiments

In this section, extensive experiments are conducted to evaluate the proposed
approach by using COVID-19 Twitter datasets. First, COVID-19 dataset
collection and pre-processing are described. Second, we compare the proposed
approach against six state-of-the-art baselines in terms of accuracy, recall,
and F1 score. Third, we present quantitative analytical results and conduct
ablation studies. Finally, a case study is given to illustrate the
effectiveness of our approach.

###  5.1 Dataset and Experiment Setting

The experiments are conducted by using a public large-scale Twitter dataset
about COVID-19, which contains English language-specific tweets being posted
from 204 different countries and territories (Lamsal (2020)). The dataset is
proposed in the scientific literature for research with topics related to
COVID-19. The statistics of datasets are listed in Table 1.

Table 1: Statistics of manual-labelled and auto-labelled dataset | Train | Test
---|---|---
Manual-labelled | 1418 | 355
Auto-labelled | 32264 | 8066

The dataset has been pre-processed in two ways, i.e., manual annotated and
auto-annotated. In the former, the annotators label the tweets according to
the concern definitions and formulations. While, in the latter, tweets are
annotated by using the approach proposed in our past research work (Shi et al.
(2020)).

Many existing research works have explored people’s reactions and attempt to
discovered wide-spreading topics about COVID-19 (Li et al. (2020); Killeen et
al. (2020); Hou et al. (2020); Kaveh-Yazdy and Zarifzadeh (2020); Li et al.
(2020)). Based on the findings and conclusion of these works, we extract the
most popular topics and define eight types of concerns, i.e., Finance (FIN),
Government (GOV), Disease (DIS), Medicine (MED), Person (PER), Location (LOC),
Food (FOD), and Date and Time (DAT). On top of that, two types of relations
among the concerns, i.e., co-occurrence and cause-effect, are investigated.
This is because both types of relations are capable of capturing the implicit
information about public concerns, demonstrating their associations and
potential causes. For instance, by analysing the tweet “… due to the locked
transportation…, farmers forced to dump green chilli …", it is important to
know the concern “green chilli" is dumped due to the concern “locked
transportation" in the time of COVID-19 pandemic. The statistics of concerns
and the relations are listed in Table 2 and Table 3, respectively.

The dataset is divided into 3 sub-datasets: train dataset and test dataset,
occupying 80% and 20%, respectively.

Table 2: Statistics of the Concern Categories

Type | Tweets | Concern Category
---|---|---
FIN | GOV | DIS | MED | PER | LOC | FOD | DAT
Manual-labelled | 1761 | 315 | 457 | 1239 | 471 | 289 | 341 | 204 | 206
Auto-labelled | 40068 | 4341 | 19941 | 23853 | 6944 | 10977 | 1519 | 1498 | 11063

Table 3: Statistics of Concern Relation Categories Type | Tweets | Concern Relation
---|---|---
CO_OCC | CA_EFF
Manual-labelled | 1761 | 932 | 829
Auto-labelled | 40068 | 19485 | 20583

####  5.1.1 Evaluation Metrics

In this paper, three standard evaluation metrics, i.e., precision, recall and
F1 score, are employed to evaluate our model.

The outcome of predicted concerns is considered correct only when both of the
concerns in one tweet are predicted correctly. In other words, $(c1,c2)$ is
recognised as a correct concern pair if $c1$ and $c2$ are correctly predicted
at same time. Correspondingly, the relation prediction is considered as valid
only when the associated concern pair is correctly predicted.

####  5.1.2 Hyper-parameters

In the experiments, BERT is utilised to obtain word embedding of tweet corpus
and the word embeddings dimension is set as $d=300$. Our network is
regularised by using dropout at the embedding layer, with a dropout ratio of
0.2. Bi-LSTM and GCN are adopted as the encoding layer, with 300 LSTM units.
We employ the full dependency tree of sentences as the adjacency matrix of
GCN.

###  5.2 Baselines

The proposed approach is evaluated by comparing against the following
baselines.

  * •

_Joint Model_(Zheng et al. (2017) is a joint extraction method to detect both
entity and relation in one tweet by using a novel tagging scheme. It is an
end-to-end model consisting of a bi-directional Long Short Term Memory (Bi-
LSTM) encoder layer and a Long Short Term Memory (LSTM) decoder layer.

  * •

_Copy Mechanism Model_(Zeng et al. (2018) is a state-of-the-art model for
jointly extracting relation triplets from a sentence. It is also an end-to-end
model based on seq-to-seq learning with a decoder layer, having two different
decoding methods, i.e., one-decoder and multi-decoder. We use both different
strategies as counterparts in the experiments.

  * •

_SPTree_(Miwa and Bansal (2016) is a novel end-to-end recurrent neural network
model, aiming at extracting entities and relations by capturing word sequence
and dependency tree substructure feature. The stacked bidirectional tree-
structured LSTM-RNN models are applied on sequential bi-LSTM-RNN models to
detect both entities and relations with shared parameters jointly.

  * •

_JointER_(Yu et al. (2020) is a joint entity and relation extraction model
which can address the limitations, including redundant entity pairs and
ignoring the important inner structure of entities. The model decomposes a
joint extraction task into Head-Entity (HE) extraction and Tail-Entity-
Relation (TER) extraction to detect head-entity, tail-entity, and relations.

  * •

_SPERT_(Eberts and Ulges (2019) is introduced as a span-based model, which can
jointly extract entity and relation by conducting light-weight reasoning on
BERT embedding and relation classification based on localised and marker-free
context features.

###  5.3 Experimental Results and Model Analysis

In this section, we present and analyse the strengths and weaknesses of the
proposed method by comparing against the state-of-the-art models as mentioned
previously. The ensure the fairness and rationality of the experiments, we
select all the counterparts, which incorporate a Bi-LSTM encoder layer.

The experimental results are demonstrated in Table 4, which presents the
predicted outcome, i.e., Precision, Recall, and F1, of both the proposed
approach and the state-of-the-art methods, using manual-labelled and auto-
labelled datasets. As can be observed from the table that the proposed
approach outperforms the others in terms of F1 score, which proves its
effectiveness. Specifically, in Figures 4 \- 6, the CG-based model outperforms
One-decoder, Multi-decoder, NovelTagging, and SPTree models on both manual-
labelled and auto-labelled Twitter datasets. Although JointER and SPERT yield
better performance than that of ours in terms of precision and recall on the
manual-labelled dataset, SPERT leverages the pre-trained BERT model to obtain
contextual features of sentences but the inner structure of entities is
neglected, which inevitably hinders the performance of entity and relation
extraction. The embeddings in JointER are initialized using the shallow
representatives model, i.e., Glove (Pennington et al. (2014), without context-
specific information, which is critical for entity and relation extraction
models.

The outstanding performance of the proposed approach mainly attributes to its
structural design. First, the interaction of the CG structure captures the
inner dependency between concerns. Second, the shared state passing from
concern extraction module to relation extraction module, provides important
concern features for relation extraction. It is worth noting that baselines
can achieve state-of-the-art results on high-quality datasets, e.g., NYT and
WebNLG, but the performance significantly degrades on the noisy and imbalanced
social media data. The grammatical mistakes of tweets make it difficult to
capture relations between concerns. NovelTagging and SPTree utilise novel
tagging, but cannot carry out promising results. Other baselines, including
One-Decoder, Multi-Decoder, JointER, apply BiLSTM to capture sequential
features of concerns but they fail to detect the relation features and
concerns due to the unstructured sentences in the tweet dataset.

Table 4: Evaluation results of different models on COVID-19 Tweets Dataset

Model | Manual-labelled Tweets | Auto-labelled Tweets
---|---|---
Precision | Recall | F1 | Precision | Recall | F1
One-Decoder | 0.160 | 0.160 | 0.160 | 0.316 | 0.316 | 0.316
Multi-Decoder | 0.150 | 0.150 | 0.150 | 0.340 | 0.340 | 0.340
NovelTagging | 0.273 | 0.336 | 0.302 | 0.570 | 0.593 | 0.582
SPTree | 0.424 | 0.349 | 0.383 | 0.434 | 0.366 | 0.397
JointER | 0.644 | 0.369 | 0.469 | 0.405 | 0.314 | 0.354
SPERT | 0.239 | 0.675 | 0.339 | 0.31 | 0.839 | 0.421
Proposed Model | 0.545 | 0.630 | 0.567 | 0.638 | 0.642 | 0.592

Figure 4: Experiment results (Precision) on manual-labelled dataset. Figure 5:
Experiment results (Recall) on manual-labelled dataset. Figure 6: Experiment
results (F1 score) on manual-labelled dataset..

###  5.4 Ablation Study

The ablation study in this section aims to investigate the impact of Concern
Graph and shared state components in the proposed approach.

Recall that manually labelling a large-scale dataset turns out to be a tedious
and non-trivial task. To conduct public concern extraction and analysis for an
emergency event, sufficient manual-labelled training data sets are usually not
available. Furthermore, the public concern coverage in datasets also appears
imbalanced, which prevents the existing models from generalisation,
subsequently impacting the performance to a large extent. The proposed
approach can mitigate this issue, giving outstanding performance on both
manual-labelled and auto-labelled datasets.

Table 5 lists the results of the ablation study. The approach has been re-
evaluated by comparing the performance against that without CG component and
shared state components. It can be seen from the table that, in manual-
labelled dataset, CG-CRE with CG and shared state outperforms the models
without CG and shared state by 11% and 7%, respectively. While in auto-
labelled datasets, it surpasses 6% and 1%, respectively. The results
explicitly reveal that Concern Graph (CG) and shared state components play a
significant role in jointly identifying concerns and relations.

Table 5: Ablation study of CG-CRE model on manual-labelled and auto-labelled
Tweets Dataset

Dataset | Method | Precision | Recall | F1
---|---|---|---|---
Manual-labelled Tweets | CG-CRE (without CG) | 0.416 | 0.482 | 0.457
CG-CRE (without shared state) | 0.463 | 0.516 | 0.494
CG-CRE (with all components) | 0.545 | 0.630 | 0.567
Auto-labelled Tweets | CG-CRE (without CG) | 0.551 | 0.583 | 0.536
CG-CRE (without shared state) | 0.615 | 0.624 | 0.586
CG-CRE (with all components) | 0.638 | 0.642 | 0.592

###  5.5 Case Study

In this section, we conduct case studies, presenting some representative
public concern extraction examples, to further prove the effectiveness and
validity of the proposed approach.

Table 6 shows the outputs from three models, including NovelTagging, JointER,
and the proposed CG-CRE. In the first case, both concerns and concern relation
are identified incorrectly by NovelTagging, and JointER predicts nothing. By
contrast, CG-CRE can extract both concerns correctly. Similar outputs is
presented in the fifth and the sixth case. As for the second and third case,
NovelTagging only detects one concern correctly and cannot extract the second
one and relation. However, JointER and CG-CRE can accurately identify concerns
and concern relation. JointER is not able to carry out the prediction results.
In the fourth case, NovelTagging can identify only one concern correctly.
JointER is able to yield correct predictions but it still remains to be
improved in eliminating null prediction. NovelTagging is weak at extracting
relations from Twitter datasets.

Based on the experimental results and case studies, we can conclude that the
proposed CG-CRE model can yield better performance on both entity recognition
and relation extraction than the state-of-the-art models.

Table 6: Outputs from different models on tweets. “pred:[]” means the model
predicts null for this tweet. NovelTagging only predicts “c1" and “c2" without
concern types.

Models | Tweet
---|---
NovelTagging |  |  $[\textbf{seeing everyone}]_{{c1},{r:co\\_{occ}}}$ suffer $[\textbf{in pakistan}]_{{c2},{r:co\\_{occ}}}$ with food shortage
---
due to the corona virus i have made bag which contain rice
JointER |  | seeing everyone suffer in pakistan with food shortage
---
due to the corona virus i have made bag which contain rice
$[\textbf{pred:[]}]$
CG-CRE |  | seeing everyone suffer in pakistan with $[\textbf{food shortage}]_{{c1:FOD},{r:ca\\_{eff}}}$
---
due to the $[\textbf{corona virus}]_{{c2:DIS},{r:ca\\_{eff}}}$ i have made bag
which contain rice
NovelTagging |  | a greeting from the heart to $[\textbf{doctors}]_{{c1},{r:co\\_{occ}}}$ , nurses, $[\textbf{paramedics}]_{{c2},{r:co\\_{occ}}}$, …
---
who stand together to tackle the corona epidemic.
JointER |  | a greeting from the heart to $[\textbf{doctors}]_{{c1:MED},{r:co\\_{occ}}}$ , $[\textbf{nurses}]_{{c2:MED},{r:co\\_{occ}}}$, paramedics, …
---
who stand together to tackle the corona epidemic.
CG-CRE |  | a greeting from the heart to $[\textbf{doctors}]_{{c1:MED},{r:co\\_{occ}}}$ ,$[\textbf{nurses}]_{{c2:MED},{r:co\\_{occ}}}$, paramedics, …
---
who stand together to tackle the corona epidemic.
NovelTagging |  |  $[\textbf{coronavirus}]_{{c1},{r:co\\_{occ}}}$ could double number of people going hungry.
---
the risk of major interruptions to $[\textbf{food
supplies}]_{{c21},{r:co\\_{occ}}}$ over the coming months is growing.
JointER |  |  $[\textbf{coronavirus}]_{{c1:DIS},{r:ca\\_{eff}}}$ could double number of people $[\textbf{going hungry}]_{{c2:FOD},{r:ca\\_{eff}}}$.
---
the risk of major interruptions to food supplies over the coming months is
growing.
CG-CRE |  |  $[\textbf{coronavirus}]_{{c1:DIS},{r:ca\\_{eff}}}$ could double number of people $[\textbf{going hungry}]_{{c2:FOD},{r:ca\\_{eff}}}$.
---
the risk of major interruptions to food supplies over the coming months is
growing.
NovelTagging |  | breaking one of somalia ’s greatest artist ha $[\textbf{died}]_{{c1},{r:co\\_{occ}}}$ in london
---
after contracting $[\textbf{corona virus}]_{{c2},{r:co\\_{occ}}}$ …
JointER |  | breaking one of somalia ’s greatest artist ha died in london
---
after contracting corona virus … $[\textbf{pred:[]}]$
CG-CRE |  | breaking one of somalia ’s greatest $[\textbf{artist}]_{{c1:PER},{r:ca\\_{eff}}}$ ha died in london
---
after contracting $[\textbf{corona virus}]_{{c2:DIS},{r:ca\\_{eff}}}$ …
NovelTagging |  | what are the $[\textbf{common}]_{{c1},{r:co\\_{occ}}}$ $[\textbf{symptom}]_{{c2},{r:co\\_{occ}}}$ . fever , sore throat …
---
JointER |  | what are the common symptom . fever , sore throat … $[\textbf{pred:[]}]$
---
CG-CRE |  | what are the common symptom . $[\textbf{fever}]_{{c1:DIS},{r:co\\_{occ}}}$ , $[\textbf{sore throat}]_{{c2:DIS},{r:co\\_{occ}}}$ …
---
NovelTagging |  | social distancing, stay home, $[\textbf{naija people}]_{{c1},{r:co\\_{occ}}}$ will not hear.
---
this corona thing has just started with us in this
$[\textbf{country}]_{{c2},{r:co\\_{occ}}}$, we …
JointER |  | social distancing, stay home, naija people will not hear.
---
this corona thing has just started with us in this country, we …
$[\textbf{pred:[]}]$
CG-CRE |  |  $[\textbf{social distancing}]_{{c1:GOV},{r:co\\_{occ}}}$, $[\textbf{stay home}]_{{c2:GOV},{r:co\\_{occ}}}$, naija people will not hear.
---
this corona thing has just started with us in this country, we …

##  6 Conclusion and Future Work

In this paper, we present an end-to-end model to simultaneously extract
concern and concern relation from the social media dataset of COVID-19. We
jointly combine GCN and Bi-LSTM to learn sequential and regional dependency
features from tweets. In order to capture more features of model input, the
influence of graph structure for concern and relation extraction is explored.
The sequential and regional features from the dataset are concatenated,
enabling the embedding vectors to represent rich contextual information of
both concerns and relations. The proposed model is evaluated on manual-
labelled and auto-labelled datasets. The experimental results show that the
proposed model can outperform the existing entity and relation extraction
models, which demonstrate the effectiveness of our method. Furthermore,
different from previous works, our model turns out to be applicable for both
manual-labelled and auto-labelled datasets rather than only works with
handcrafted datasets. Therefore, our method can be easily transferred and
applied to other pandemics situations, e.g., Zika, Dengue Fever, and Yellow
Fever.

In the future, we plan to improve the proposed model from two aspects. First,
more concern types and concern relation types can be predicted to understand
what people’s attention and the relation between them. In addition, time
factor can be used to track the trend of one specific concern over time.

## References

  * Killgore et al. [2020] William DS Killgore, Sara A Cloonen, Emily C Taylor, and Natalie S Dailey.  Loneliness: A signature mental health concern in the era of covid-19.  _Psychiatry Research_ , page 113117, 2020.
  * Damiano and Catellier JR [2020] AD Damiano and Allen Catellier JR.  A content analysis of coronavirus tweets in the united states just prior to the pandemic declaration.  _Cyberpsychology, Behavior and Social Networking_ , 2020.
  * Li et al. [2018] Weihua Li, Quan Bai, Minjie Zhang, and Tung Doan Nguyen.  Automated influence maintenance in social networks: an agent-based approach.  _IEEE Transactions on Knowledge and Data Engineering_ , 31(10):1884–1897, 2018.
  * Hu et al. [2019] Yuxuan Hu, Quan Bai, and Weihua Li.  Context-aware influence diffusion in online social networks.  In _Pacific Rim Knowledge Acquisition Workshop_ , pages 153–162. Springer, 2019.
  * da Silva et al. [2020] Jaime A Teixeira da Silva, Panagiotis Tsigaris, and Mohammadamin Erfanmanesh.  Publishing volumes in major databases related to covid-19.  _Scientometrics_ , pages 1–12, 2020.
  * Nelson et al. [2020] Benjamin W Nelson, Adam Pettitt, Jessica E Flannery, and Nicholas B Allen.  Rapid assessment of psychological and epidemiological correlates of covid-19 concern, financial strain, and health-related behavior change in a large online sample.  _PloS one_ , 15(11):e0241990, 2020.
  * van der Vegt and Kleinberg [2020] Isabelle van der Vegt and Bennett Kleinberg.  Women worry about family, men about the economy: Gender differences in emotional responses to covid-19.  In _Social informatics 2020: Proceedings International Conference on Social Informatics_ , pages 397–409. Springer, 2020.
  * Kassab et al. [2020] Lara Kassab, Alona Kryshchenko, Hanbaek Lyu, Denali Molitor, Deanna Needell, and Elizaveta Rebrova.  On nonnegative matrix and tensor decompositions for covid-19 twitter dynamics.  _arXiv preprint arXiv:2010.01600_ , 2020.
  * Kipf and Welling [2017] Thomas N. Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  In _International Conference on Learning Representations (ICLR)_ , 2017.
  * Rathan et al. [2018] M Rathan, Vishwanath R Hulipalled, KR Venugopal, and LM Patnaik.  Consumer insight mining: aspect based twitter opinion mining of mobile phone reviews.  _Applied Soft Computing_ , 68:765–773, 2018.
  * Li et al. [2020] Lifang Li, Qingpeng Zhang, Xiao Wang, Jun Zhang, Tao Wang, Tian-Lu Gao, Wei Duan, Kelvin Kam-fai Tsoi, and Fei-Yue Wang.  Characterizing the propagation of situational information in social media during covid-19 epidemic: A case study on weibo.  _IEEE Transactions on Computational Social Systems_ , 2020.
  * Jahanbin and Rahmanian [2020] Kia Jahanbin and Vahid Rahmanian.  Using twitter and web news mining to predict covid-19 outbreak, 2020.
  * Hou et al. [2020] Zhiyuan Hou, Fanxing Du, Hao Jiang, Xinyu Zhou, and Leesa Lin.  Assessment of public attention, risk perception, emotional and behavioural responses to the covid-19 outbreak: social media surveillance in china.  _Risk Perception, Emotional and Behavioural Responses to the COVID-19 Outbreak: Social Media Surveillance in China (3/6/2020)_ , 2020.
  * Lazard et al. [2015] Allison J Lazard, Emily Scheinfeld, Jay M Bernhardt, Gary B Wilcox, and Melissa Suran.  Detecting themes of public concern: a text mining analysis of the centers for disease control and prevention’s ebola live twitter chat.  _American journal of infection control_ , 43(10):1109–1111, 2015.
  * Shi et al. [2020] Jingli Shi, Weihua Li, Yi Yang, Naimeng Yao, Quan Bai, Yongchareon Sira, and Jian Yu.  Automated concern exploration in pandemic situations - covid-19 as a use case.  In _2020 Principle and Practice of Data and Knowledge Acquisition Workshop (PKAW2020)_ , page Accepted, 2020.
  * Chandrasekaran et al. [2020] Ranganathan Chandrasekaran, Vikalp Mehta, Tejali Valkunde, and Evangelos Moustakas.  Topics, trends, and sentiments of tweets about the covid-19 pandemic: Temporal infoveillance study.  _Journal of medical Internet research_ , 22(10):e22624, 2020.
  * Xue et al. [2020] Jia Xue, Junxiang Chen, Ran Hu, Chen Chen, Chengda Zheng, Yue Su, and Tingshao Zhu.  Twitter discussions and emotions about the covid-19 pandemic: Machine learning approach.  _Journal of Medical Internet Research_ , 22(11):e20550, 2020.
  * Wahbeh et al. [2020] Abdullah Wahbeh, Tareq Nasralah, Mohammad Al-Ramahi, and Omar El-Gayar.  Mining physicians’ opinions on social media to obtain insights into covid-19: mixed methods analysis.  _JMIR public health and surveillance_ , 6(2):e19276, 2020.
  * Wang et al. [2020] Tianyi Wang, Ke Lu, Kam Pui Chow, and Qing Zhu.  Covid-19 sensing: Negative sentiment analysis on social media in china via bert model.  _Ieee Access_ , 8:138162–138169, 2020.
  * Yin et al. [2020] Hui Yin, Shuiqiao Yang, and Jianxin Li.  Detecting topic and sentiment dynamics due to covid-19 pandemic using social media.  In _International Conference on Advanced Data Mining and Applications_ , pages 610–623. Springer, 2020.
  * Chen et al. [2020] Long Chen, Hanjia Lyu, Tongyu Yang, Yu Wang, and Jiebo Luo.  In the eyes of the beholder: Sentiment and topic analyses on social media use of neutral and controversial terms for covid-19.  _arXiv preprint arXiv:2004.10225_ , 2020.
  * Mohit [2014] Behrang Mohit.  Named entity recognition.  In _Natural language processing of semitic languages_ , pages 221–245. Springer, 2014.
  * Zhou and Su [2002] GuoDong Zhou and Jian Su.  Named entity recognition using an hmm-based chunk tagger.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 473–480, 2002.
  * Lafferty et al. [2001] John Lafferty, Andrew McCallum, and Fernando CN Pereira.  Conditional random fields: Probabilistic models for segmenting and labeling sequence data.  2001\.
  * Lample et al. [2016] Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer.  Neural architectures for named entity recognition.  pages 260–270, 2016.
  * Ma and Hovy [2016] Xuezhe Ma and Eduard Hovy.  End-to-end sequence labeling via bi-directional lstm-cnns-crf.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_ , pages 1064–1074, 2016.
  * Peters et al. [2018] Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2227–2237, 2018.
  * Akbik et al. [2018] Alan Akbik, Duncan Blythe, and Roland Vollgraf.  Contextual string embeddings for sequence labeling.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 1638–1649, 2018.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Chinchor [1998] Nancy A Chinchor.  Overview of muc-7/met-2.  Technical report, SCIENCE APPLICATIONS INTERNATIONAL CORP SAN DIEGO CA, 1998.
  * Culotta and Sorensen [2004] Aron Culotta and Jeffrey Sorensen.  Dependency tree kernels for relation extraction.  In _Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04)_ , pages 423–429, 2004.
  * Zhou and Zhu [2011] Guo-Dong Zhou and Qiao-Ming Zhu.  Kernel-based semantic relation detection and classification via enriched parse tree structure.  _Journal of Computer Science and Technology_ , 26(1):45–56, 2011.
  * Seewald and Kleedorfer [2007] Alexander K Seewald and Florian Kleedorfer.  Lambda pruning: an approximation of the string subsequence kernel for practical svm classification and redundancy clustering.  _Advances in Data Analysis and Classification_ , 1(3):221–239, 2007.
  * Zeng et al. [2014] Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao.  Relation classification via convolutional deep neural network.  In _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_ , pages 2335–2344, 2014\.
  * Miwa and Bansal [2016] Makoto Miwa and Mohit Bansal.  End-to-end relation extraction using lstms on sequences and tree structures.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_ , pages 1105–1116, 2016.
  * Fu et al. [2019] Tsu-Jui Fu, Peng-Hsuan Li, and Wei-Yun Ma.  Graphrel: Modeling text as relational graphs for joint entity and relation extraction.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1409–1418, 2019.
  * Katiyar and Cardie [2017] Arzoo Katiyar and Claire Cardie.  Going out on a limb: Joint extraction of entity mentions and relations without dependency trees.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_ , pages 917–928, 2017.
  * Zheng et al. [2017] Suncong Zheng, Feng Wang, Hongyun Bao, Yuexing Hao, Peng Zhou, and Bo Xu.  Joint extraction of entities and relations based on a novel tagging scheme.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_ , pages 1227–1236, 2017.
  * Zeng et al. [2018] Xiangrong Zeng, Daojian Zeng, Shizhu He, Kang Liu, and Jun Zhao.  Extracting relational facts by an end-to-end neural model with copy mechanism.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_ , pages 506–514, 2018.
  * Battaglia et al. [2016] Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and Koray kavukcuoglu.  Interaction networks for learning about objects, relations and physics.  In _Proceedings of the 30th International Conference on Neural Information Processing Systems_ , pages 4509–4517, 2016.
  * Defferrard et al. [2016] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst.  Convolutional neural networks on graphs with fast localized spectral filtering.  In _Advances in neural information processing systems_ , pages 3844–3852, 2016.
  * Hamilton et al. [2017] Will Hamilton, Zhitao Ying, and Jure Leskovec.  Inductive representation learning on large graphs.  In _Advances in neural information processing systems_ , pages 1024–1034, 2017.
  * Hong et al. [2020] Yin Hong, Yanxia Liu, Suizhu Yang, Kaiwen Zhang, and Jianjun Hu.  Joint extraction of entities and relations using graph convolution over pruned dependency trees.  _Neurocomputing_ , 411:302–312, 2020.
  * Zhang et al. [2018] Yuhao Zhang, Peng Qi, and Christopher D Manning.  Graph convolution over pruned dependency trees improves relation extraction.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2205–2215, 2018.
  * Huang et al. [2015] Zhiheng Huang, Wei Xu, and Kai Yu.  Bidirectional lstm-crf models for sequence tagging.  _arXiv preprint arXiv:1508.01991_ , 2015.
  * Lamsal [2020] Rabindra Lamsal.  Design and analysis of a large-scale covid-19 tweets dataset.  _Applied Intelligence_ , pages 1–15, 2020.
  * Killeen et al. [2020] Benjamin D Killeen, Jie Ying Wu, Kinjal Shah, Anna Zapaishchykova, Philipp Nikutta, Aniruddha Tamhane, Shreya Chakraborty, Jinchi Wei, Tiger Gao, Mareike Thies, et al.  A county-level dataset for informing the united states’ response to covid-19.  _arXiv preprint arXiv:2004.00756_ , 2020.
  * Kaveh-Yazdy and Zarifzadeh [2020] Fatemeh Kaveh-Yazdy and Sajjad Zarifzadeh.  Track iran’s national covid-19 response committee’s major concerns using two-stage unsupervised topic modeling.  _International Journal of Medical Informatics_ , page 104309, 2020\.
  * Yu et al. [2020] Bowen Yu, Zhenyu Zhang, Xiaobo Shu, Yubin Wang, Tingwen Liu, Bin Wang, and Sujian Li.  Joint extraction of entities and relations based on a novel decomposition strategy.  In _Proc. of ECAI_ , 2020.
  * Eberts and Ulges [2019] Markus Eberts and Adrian Ulges.  Span-based joint entity and relation extraction with transformer pre-training.  _arXiv preprint arXiv:1909.07755_ , 2019.
  * Pennington et al. [2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 1532–1543, 2014.