# A Multi-task Model for Sentiment Aided Stance Detection of
Climate Change Tweets

Apoorva Upadhyaya, Marco Fisichella, Wolfgang Nejdl

###### Abstract

Climate change has become one of the biggest challenges of our time. Social
media platforms such as Twitter play an important role in raising public
awareness and spreading knowledge about the dangers of the current climate
crisis. With the increasing number of campaigns and communication about
climate change through social media, the information could create more
awareness and reach the general public and policy makers. However, these
Twitter communications lead to polarization of beliefs, opinion-dominated
ideologies, and often a split into two communities of climate change deniers
and believers. In this paper, we propose a framework that helps identify
denier statements on Twitter and thus classifies the stance of the tweet into
one of the two attitudes towards climate change (denier/believer). The
sentimental aspects of Twitter data on climate change are deeply rooted in
general public attitudes toward climate change. Therefore, our work focuses on
learning two closely related tasks: Stance Detection and Sentiment Analysis of
climate change tweets. We propose a multi-task framework that performs stance
detection (primary task) and sentiment analysis (auxiliary task)
simultaneously. The proposed model incorporates the feature-specific and
shared-specific attention frameworks to fuse multiple features and learn the
generalized features for both tasks. The experimental results show that the
proposed framework increases the performance of the primary task, i.e., stance
detection by benefiting from the auxiliary task, i.e., sentiment analysis
compared to its uni-modal and single-task variants.

##  1 Introduction

Climate change is the burning crisis of our time, and it is happening even
faster than we thought. A recent article on the BBC News
website111https://www.bbc.com/news/science-environment-60525591 states that
many of the effects of global warming are now simply “irreversible” according
to the latest assessment and that more than $40$% of the world’s population is
“at high risk” from climate. In fact, according to a report by the
Intergovernmental Panel on Climate Change (IPCC), it is very likely that
climate change is caused by man-made activities (Myhre et al. 2013). Despite
the scientific consensus on the causes and main impacts of climate change, it
remains a controversial topic in public discourse. Therefore, understanding
public perceptions plays a critical role in addressing climate change by
increasing the public’s willingness to accept appropriate action on climate
change (Shi, Visschers, and Siegrist 2015).

Recently, social media platforms such as Twitter have played an important role
in raising public awareness of the current climate crisis and influencing
public attitudes toward climate change (Lineman et al. 2015). Twitter
conversations, however, are often influenced by the polarization of beliefs
and, in the case of climate change, are divided into two competing groups, one
that believes in climate change (Believers) and a second that is skeptical or
denies that climate change is occurring (Disbelievers) (Jang and Hart 2015;
Cann, Weaver, and Williams 2021). The article published on the Euronews
website222https://www.euronews.com/green/2021/11/18/climate-misinformation-is-
getting-more-sophisticated-and-experts-say-cop26-progress-could- after the
COP26 conference revealed that scientists have found that climate change
deniers are not only skeptical about climate change, but have also led to the
problem of delaying climate change by either shifting responsibility or
eventually capitulating - the idea that it is not possible to prevent climate
change, which often leads to the spread of misinformation (Zhou and Shen
2021). Therefore, it is important for government agencies, researchers, and
technology companies to monitor such content on social media to identify and
intervene in tweets from climate change deniers, which will help combat
climate misinformation. This has motivated us to identify such content and
understand public attitudes toward climate change by performing the important
task of stance detection.

Stance detection is the task of automatically identifying the author’s point
of view in relation to the target object (for/against/neutral). It has been
used to identify social attitudes toward pressing issues (e.g., covid19
vaccination (Argyris et al. 2021), climate change, abortion, feminism
(Mohammad et al. 2016)). In our study, we focus on climate change as a target
and perform statement-level stance detection. The goal is to predict the
attitude expressed in a single tweet, where the stance is for (believers) or
against climate change (deniers).

Numerous works have classified tweets into favor or against the target using
the SemEval 2016 benchmark dataset. The dataset contains $5$ target topics,
including climate change, with $29$ denier and $335$ believer tweets (Li and
Caragea 2019; Wang et al. 2020). However, due to the small number of climate-
specific tweets, these techniques do not focus on understanding the specific
characteristics of climate change deniers and believers. This is because
climate change deniers not only deny climate change, but also disagree with
the solutions to combat climate change, which often leads to public negligence
(Zhou and Shen 2021). Other works that deal with the classification of tweets
on climate change lack a suitable architecture that efficiently performs the
classification task (Kabaghe and Qin 2019). Therefore, in our work, we use a
multi-tasking approach that uses different attention frameworks to classify
the attitude of the climate change tweet into one of the two polarized classes
(deniers/believers) to identify the denier statements on Twitter.

Sentiment analysis has helped in many other tasks, such as detecting hate
speech and sarcasm in a multi-task architecture (Majumder et al. 2019).
Studies on climate change have also justified the role of sentiment in
conversations about climate change, either by assessing sentiment in tweets
from climate change deniers or by examining the emotional impact of climate
change data on Twitter (Dahal, Kumar, and Li 2019; El Barachi et al. 2021).
Therefore, in our study, we leverage the sentiment analysis task to decipher
the attitude of the tweets.

In addition, we use multiple inputs, i.e., combining tweet text and topic
representations, to build a reliable classification model that helps identify
the sentiment of the tweeter and determine the correct tweet attitude. Topic
representations have helped in detecting fake news by providing more
discriminatory power to the model used (Gautam, V, and Masud 2021). In our
study, topic embedding provides a global context to a single tweet and thus
more information and can circumvent the drawback of the short length of the
tweet text to efficiently train the proposed system.

Our work focuses on learning two closely related tasks, stance detection and
sentiment analysis of tweets on climate change. Stance detection is our main
task, which is supported by sentiment analysis as an auxiliary task. We
propose a multi-task framework that incorporates feature-specific and shared-
specific attention frameworks to fuse multiple features and learn the features
for both tasks. Our proposed approach is useful for government agencies and
technology companies to detect the attitude of posts (deniers/believers) and
curb the spread of such content that denies climate change and is false or
misleading to combat climate misinformation.

We summarise the contributions of our work as follows: (i) We create a new
dataset333The dataset and code are available at the repository:
https://github.com/apoorva-upadhyaya/Climate-Change-Tweets. for the climate
change domain consisting of tweets with the stance and sentiment labels which
is beneficial for the research community. (ii) We illustrate the importance to
consider the sentiment associated with the tweet while categorising the stance
of tweet into favor(believers) or against(deniers) climate change. (iii) We
propose a multi-task framework that jointly performs the stance detection
(primary) and sentiment analysis (auxiliary) tasks. We integrate feature-
specific and shared-specific attention frameworks to integrate information
across multiple features and shared tasks to learn features that optimise task
performance. Experimental results indicate that the proposed framework
increases performance of the primary task, i.e., stance detection by
benefiting from the auxiliary task, i.e., sentiment analysis compared to its
uni-modal and single-task variants.

##  2 Related Works

Climate change has become one of the greatest challenges of our time. Several
works explore the role of psychology in climate change by examining the impact
of climate change on people’s well-being and perceptions (Clayton 2020).
Engaging the public is an important part of addressing climate change.
Therefore, social media platforms like Twitter allow anyone to explore and
report public viewpoints on the complex issue of climate change (Lineman et
al. 2015; Dahal, Kumar, and Li 2019). However, debates and discussions on
Twitter about climate change are widely associated with increasing
polarization and are often divided into climate change believers and deniers
(Jang and Hart 2015). In order to identify and understand public attitudes
toward climate change, the task of identifying attitudes plays an important
role.

###  2.1 Stance Detection

Stance detection is about classifying the attitude that the author expresses
towards a target object. The author may support the target object, reject it,
or have a neutral stance. In our work, we focus on the climate change target,
where opinions are either for climate change (believers) or against climate
change (deniers). There are several climate change specific studies where the
goal is to predict a user’s attitude (Chen, Zou, and Zhao 2019; Tyagi et al.
2020a). It has been found that multiple tweets from the same user can have
different stance classes. In order not to miss the denier attitude of a single
tweet that could interfere with the implementation of climate change policies,
we focus on detecting statement-level stance detection, where the goal is to
predict the stance described in a single tweet.

The stance detection of tweets has been studied in a variety of work on the
popular SemEVAL 2016 dataset, which includes the $5$ targets including climate
change (with $364$ climate change tweets) (Vychegzhanin and Kotelnikov 2021;
Wang et al. 2020). However, in these previous studies, little attention was
paid to understanding the characteristics of climate change denier and
believer tweets in particular (only $29$ climate change denier tweets in the
SemEVAL 2016 dataset). Recently, (Luo, Card, and Jurafsky 2020) published a
stance-annotated dataset on global warming and proposed an opinion framing
task to explore the discourse used in the global warming debate. One of the
papers (Kabaghe and Qin 2019) classified tweets into three classes based on
their attitude toward climate change: $-1$ (negative belief), $0$ (neutral
belief), and $1$ (positive belief). However, the lack of an architectural
framework led us to propose an efficient model that can efficiently classify a
tweet’s attitude toward climate change into one of two categories
(deniers/believers) in real-time, while taking advantage of the other task.

###  2.2 Sentiment Analysis

Some of the works on stance recognition have emphasized the importance of
sentiment (Wang et al. 2020), while some have cited the orthogonal
relationship between stance and sentiment of the statement (Sen, Flöck, and
Wagner 2020). However, several works have focused on the sentimental aspects
of climate change conversations and justified their role in climate change
(Cody et al. 2015; Jiang et al. 2017). One recent work (El Barachi et al.
2021) proposes a real-time framework that uses sentiment and emotion analysis
to provide meaningful insights into public opinion, and tested the model with
tweets posted by Greta Thunberg and her followers on climate change. These
studies motivated us to investigate the role of sentiment in classifying
tweets on climate change.

###  2.3 Multi-Task Learning

Multi-task learning (MTL) (Caruana 1997) is a learning paradigm that aims to
learn multiple related tasks together in the hope of improving generalization
performance for all tasks at hand. In our work, we focus on learning stance
detection task with the help of sentiment analysis in a multi-tasking system.
The study by (Li and Caragea 2019) uses sentiment to predict the stance
through a multi-task learning model. Another work by (Chauhan, Kumar, and
Ekbal 2019) uses sentiment as an auxiliary task to predict attitude. However,
in our work, multiple features in the form of text and topic words are used to
separate the task-dependent and independent feature spaces and perform both
tasks simultaneously by using attention frameworks to focus on the most
important feature representations and discarding the useless shared features
that may affect the performance of both tasks.

##  3 Dataset

In this section, we describe the Twitter datasets we use for our experiments.
We consider both publicly available tweets about climate change and live
tweets about climate change that we collect using hashtags from the previous
literature. We first discuss data collection strategies, followed by data
labeling and data pre-processing techniques. The qualitative aspect and the
temporal analysis of the dataset is described in the Supplementary.

###  3.1 Data Collection Method

Previous works have used hashtags to identify the stances of different groups
on social media, creating a method of tagging content by topic (Misra et al.
2016), we also use this explicit annotation quality of hashtags to collect a
larger dataset. First, we select the hashtags for deniers and believers (Tyagi
et al. 2020a) from the previous literature (see row $1$ of the table 1). We
then consider the two publicly available Twitter datasets on climate change,
(i) tweet IDs collected from September 21, 2017 to May 17, 2019 using a set of
climate change keywords available in the Harvard Dataverse (Littman and Wrubel
2019), and (ii) tweet IDs used in the work (Samantray and Pin 2019) collected
from 2007 to 2019. We start by retrieving the tweet objects from these
publicly available tweet IDs using the Tweepy
API444https://docs.tweepy.org/en/stable/ . We analyzed the hashtags used in
the collected tweets and found that the hashtags mentioned in row $2$ of the
table 1 are the most frequently used and co-occur with the denier and believer
seed hashtags. We draw a random sample of $1000$ tweets containing these most
frequently used hashtags separately for both categories and identify $98\%$
tweets as deniers and $99\%$ tweets as believers. We conclude that the final
set of seed hashtags (see table 1) can be used to identify tweets from deniers
and believers. Because the publicly available datasets contain tweets with a
large time span that reflect Twitter trends and topics related to climate
change and cover a wide range of audiences, they helped us identify additional
relevant hashtags associated with climate change deniers and believers and
enriched us with a final set of seed hashtags that can be used for further
data collection related to climate change deniers and believers. We then
select the unique tweets (excluding retweets) from the collected data that
contain either the denier or believer hashtags. We obtain a total of $5,682$
denier and $32,111$ believer tweets after the filtering process based on seed
hashtags. The collected dataset appears to be relatively small, with a smaller
number of denier tweets, so we collect the real-time tweets from July 28 to
December 26, 2021 using the live-streaming Tweepy API with the final set of
seed hashtags. The number of tweets filtered out as deniers and believers
according to the seed hashtags from various sources are listed in the table 2.

Sources | Denier Hashtags | Believer Hashtags
---|---|---
Tyagi et al. 2020a |  | ClimateHoax, YellowVests,
---
Qanon
| ClimateChangeIsReal,
---
ClimateActionNow,
FactsMatter,
ScienceMatters,
ScienceIsReal
| Most Used & Co-occur
---
from public data
| GlobalWarmingHoax,
---
ClimateChangeHoax,
ClimateDenial,ClimateHoax
| SaveClimate,
---
ActOnClimate

Table 1: Denier & Believer Seed Hashtags

Dataset | Total Tweets | As per seed Hashtag |  | As per label prop.
---
algo (out of
seed hashtag)
- | - | Denier | Believer | Denier | Believer
| Harvard Data
---
(Littman and
Wrubel 2019)
1322969 | 1595 | 20886 | 1042 | 18421
| Credibility Data
---
(Samantray
and Pin 2019)
9672907 | 4087 | 11225 | 2883 | 7735
Live tweets | 5711743 | 10594 | 39787 | 9200 | 34274
Total tweets | - | 16276 | 71898 | 13125 | 60430

Table 2: Dataset Statistics for Stance Detection Task

input : Graph $G$ with nodes $n$ and edges $e$ with $e_{ij}$ as edge weight
between $i\in n$ and $j\in n$

initialize $\gamma$=$50/100$ and $i$=0 ;

for  _$each$ $n$_ do

define $l$ = integer(i/$\gamma$); $i$+=$1$ ;

for  _$each$ $n$_ do

if  _$n$ $not$ $labeled$_ then

compute $t$ = neighbors of $n$ ;

compute $t_{l}$ = labeled neighbors of $n$ ;

if  _$t_{l}+l$ $\geq$ $t$_ then

initialize $score,c$ ;

for  _$each$ $t_{i}\in t$_ do

$score+=$ label $t_{i}*e_{nt_{i}}$; $c$+=$e_{nt_{i}}$;

Algorithm 1 Label Propagation Algorithm

###  3.2 Data Annotation

#### Stance Detection

To validate the stances of the collected tweets provided by the hashtag self-
annotation technique, we run a variant of label propagation algorithm (Tyagi
et al. 2020a, b), which transfers the labels from the seed hashtags to other
hashtags (refer Algorithm 1). The authors who provided the label propagation
algorithm claim that their approach is similar to various other works (Weber,
Garimella, and Batayneh 2013; Garimella et al. 2018). First, we weight the
seed hashtags of believers by $+1$ and those of deniers by $-1$ (as in table
1). We create a weighted hashtag$*$hashtag co-occurrence graph with all
hashtags present in the data, where each node represents a hashtag and an edge
is created between the hashtags that occur in the same tweet, where the weight
of the edge is proportional to the frequency of their co-occurrence. The
weights of the seed hashtags are then transferred to other hashtags as
specified in the algorithm 1. The hashtag scores (believer hashtag = $+1$,
denier hashtag = $-1$) are then arithmetically summed for all hashtags that
occur in each tweet and then averaged. The final score is then used to
classify tweets into deniers ( score $<0$) or believers ( score $>0$). In
total, we found a set of $13,125$ denier and $60,430$ believer tweets (refer
table 2). Three trained annotators drew a random sample of $1000$ tweets from
both categories and manually annotated them. To determine the consistency
between the ratings of the annotators, we use the Fleiss-Kappa (Spitzer et al.
1967) measure and achieve an agreement score of $0.84$, indicating that the
annotations are of good quality. We consider the manually annotated tweets as
the ground truth and compared them with the annotations found after the label
propagation algorithm. We found that $98.40\%$ of denier tweets belong to the
denier category and $99.6\%$ of believer tweets belong to the believer
category. Therefore, to save time and cost, we consider the labels generated
by the label propagation algorithm as the final labels for denier and believer
tweets. The steps for data collection and data annotation for the stance
detection task are briefly described in Fig. 1 in the Supplementary.

#### Sentiment Analysis

We leverage weak supervision approach to annotate tweets for sentiment
analysis. Similar to previous work (Singh et al. 2021), we use three sentiment
classifiers to generate sentiment labels for each preprocessed text of the
tweet, namely (i) VADER (Hutto and Gilbert 2014): a popular lexicon and rule-
based sentiment analysis tool that relies on a dictionary to generate
sentiment scores, (ii) TextBlob555https://textblob.readthedocs.io/en/dev/: a
Python-based library that provides an API for handling common NLP tasks such
as sentiment analysis, POS tagging, etc., and (iii) NLTK666
https://www.nltk.org/index.html : a Python bundle that provides a collection
of NLP algorithms such as sentiment analysis, NER, etc. This results in $3$
labels (positive, negative, neutral) per tweet, from which a single label is
finally selected as the sentiment expressed by the tweet based on the majority
voting based ensemble method. The data statistics is mentioned in table 3.
Three trained annotators manually evaluated the labels for $1000$ randomly
selected tweets and obtained an inter-annotator agreement score of $0.81$
using the Fleiss-Kappa measure. We consider the final annotations generated
after inter-annotator agreement as the ground truth and compared them with the
annotations generated using the weak supervision approach and found an
accuracy of $97.6\%$. To save time and cost, we consider the annotations
generated using the weak supervision approach for the sentiment analysis task.

Tweets | Negative | Positive | Neutral
---|---|---|---
Denier | 60.2% | 18.2% | 21.6%
Believer | 24.7% | 46.2% | 29.1%

Table 3: Data Statistics for Sentiment Task

###  3.3 Data pre-processing

Data pre-processing is important because raw tweets without pre-processing are
very unstructured and contain redundant and often problematic information that
affects the performance of the model training and classification task.

#### Text

We remove mentions, URLs, punctuation, spaces, and unwanted characters such as
RT (retweet), CC (carbon copy), and stopwords from the tweet text. We use
ekphrasis (Baziotis, Pelekis, and Doulkeridis 2017) to extract hashtags by
segmenting long strings into their individual words. For further text
processing, we use the Python toolkit NLTK. The NLTK-based tokenizer is used
to tokenize tweets. All words are converted to lowercase letters. Then, we
reduce the inflected words by applying the NLTK Wordnet lemmatizer, and then
apply PorterStemmer for stemming.

#### Topic

We first remove the seed hashtags used for data collection, otherwise topics
created can be biased towards the hashtags. The tweet text is pre-processed
using the procedure described above. In this study, we use BERTopic modeling,
which uses transformer-based embeddings to create easily interpretable topics
and their distributions (Grootendorst 2020). This modeling technique has
recently gained popularity and provided promising results in previous studies
(Anwar et al. 2021), therefore we focus on using BERTopic that detect semantic
similarity and integrate topics with pre-trained contextual representations.
The tweet text is then fed into the BERTopic library with the
calculate_probabilities=True, which creates topics from the data and assigns a
probability score to each created topic for each tweet sample in the data. We
select the m-most similar topics for each tweet sample, where each topic is
represented by the top ‘p’ topic words.

Category | Tweet | Topic Words
---|---|---
Denier |  | CO2 is greening the planet
---
and restoring the rainforest.
Its almost like the planet is
able to self regulate #ClimateHoax .
| nonsense, hoax, science,
---
denial, destroy, ridiculous,
myth, planet, hypocrisy…
Believer |  | Great format and read #climate
---
#ClimateActionNow
| warm, hot, earth, real,
---
emergency, possible, crisis,
urgent, sun, warming,…

Table 4: Examples with stance, tweet text and topic words

Role of Topic words as Feature In table 4, we present two samples from the
dataset that illustrate the importance of considering topic words along with
the tweet text for the analysis tasks. It can be observed that the tweet text
alone is not helpful in identifying the stance of the tweet in the given
samples. However, the addition of topic words gives the tweet more context and
information that helps in correctly predicting the denier or believer stance
of the tweet. These examples show that the presence of complementary
information in the form of topic words aids the process of stance detection.

##  4 Methodology

In this section, we outline the working of our proposed multi-task model for
the stance detection (primary task) and the sentiment analysis (auxiliary
task). The proposed model consists of the following components : Feature
Extraction, Attention Framework, and Classification Layer. The key factor of
multi-task learning is the sharing of features across the tasks. Therefore, we
first describe the two variants of the model depending on the framework of
multi-task learning and then explain the model components in detail.

###  4.1 Variants of the Proposed Model

We depict the following $2$ variants of the multi-task model depending on the
sharing of features across the tasks: (i)Shared-Only Multi-Task Model (SO-MT)
: In the SO-MT model, we use single shared layers for the feature extractor
and the attention framework to extract features for all tasks, as shown in
Fig. 1. The single shared output of the attention framework is then used as an
input to the classification layer, which produces separate outputs for the
stance and sentiment tasks. This model focuses on the task-invariant features
and ignores the fact that some features are task-dependent.

(ii)Shared-Private Multi-Task Model (SP-MT): In the model SP-MT, we have two
separate feature spaces for the two tasks that capture task-specific features,
and a single shared feature space that captures the task-invariant features
(refer Fig. 3). The final features are the concatenation of the features from
the private space and the shared space, which are then fed into the
classification layer to generate the output for both tasks.

The input and output of each model component for both variants (Figures 1 and
3) are mentioned in the following subsection 4.2. We now describe each of the
model components in detail.

Figure 1: Architectural diagram of the Shared-Only Multi-Task (SO-MT)
Framework.

Figure 2: Architecture of the Shared-Private Multi-Task
(SP-MT) Framework

Figure 3: Attention Framework : (Top) Shared-Specific Attention; (Bottom)
Feature-Specific Attention

###  4.2 Components of the Model

#### Feature Extraction

Text : Each tweet text ‘T’ contains $n_{t}$ number of words, where the
embedding of each word $w_{1},...,w_{n_{t}}$ is acquired from BERT (Devlin et
al. 2019) with dimension(d) = 768. We obtain final embedding for each tweet
text as $T\in\mathbb{R}^{n_{t}\times d}$. Since Bi-LSTM has shown excellent
performance in text classification due to its ability to learn long-term
dependencies and incorporate past and future context information without
retaining duplicate information (Zhou et al. 2016), we use Bi-LSTM to
sequentially encode the embedded input text representations. The embedded text
is fed to Bi-LSTM with dimension $d_{l}$, which learns the long-term context
dependent semantic features into hidden states. The final hidden matrix of
text is $H_{t}\in\mathbb{R}^{n_{t}\times 2d_{l}}$.

Topic : Each tweet is tagged with the number of m-most similar topics based on
the probability score assigned to each of the topics, where each topic is
represented by the top ‘p’ topic words created using BERTopic (described in
section 3.3). Here we represent the topic feature as ‘U’ containing a set of
$n_{u}$ words ($n_{u}$ $\leq m\times p$), where the representation of each
word $w_{1},...,w_{n_{u}}$ are obtained from BERT with $d=768$. We obtain
final embedding for each tweet topic as $U\in\mathbb{R}^{n_{u}\times d}$. This
representation of topic is then fed to the Bi-LSTM layer with $d_{l}$ that
sequentially encodes these representation, and gives the final hidden matrix
of topic $H_{u}\in\mathbb{R}^{n_{u}\times 2d_{l}}$.

#### Attention Framework

Attention mechanism has been used as an important component across a wide
range of NLP models (Bahdanau, Cho, and Bengio 2014). Typically, the attention
layer concentrates on the relevant part of the input and extracts the most
important information from the input. We apply the attention framework similar
to (Vaswani et al. 2017), in which the authors consider an attention function
as a mapping to a set of queries, keys, and values. To obtain queries, keys,
and values for the final feature representations, we pass the hidden matrix
output from the Bi-LSTM layer of text ($H_{t}$) and topic ($H_{u}$),
respectively, through three fully connected layers of dimension $d_{a}$. There
are two triplets of query, key, and value for text ($Q_{t},K_{t},V_{t}$) and
topic ($Q_{u},K_{u},V_{u}$) in the model SO-MT, while we have a total of four
triplets for the model SP-MT, forming two pairs of two triplets each for text
and topic, which are used for stance detection
(($Q_{td},K_{td},V_{td}$),($Q_{ud},K_{ud},V_{ud}$)) and sentiment
(($Q_{ts},K_{ts},V_{ts}$),($Q_{us},K_{us},V_{us}$)) tasks respectively. Fig. 3
visually shows the two attention frameworks used in our model: feature-
specific attention and shared-specific attention. The lower part of the figure
shows how different queries, keys, and values are encoded to obtain self-
attention and inter-attention, which form the two sub-modules of feature-
specific attention. The upper part of the figure shows the connections between
queries, keys, and values to achieve shared-specific attention. In the
following, we describe in detail the attention mechanisms used in our study.

Feature-Specific Attention We apply two types of attention to the features to
capture the most informative parts of them. Fig. 3 (bottom) shows the visual
representation and connections of feature-specific attention. Feature-specific
attention is further divided into Self Attention (SA) and Inter Attention
(IA).

Self Attention (SA) We use Self Attention (SA) to relate different positions
of a single sequence of say tweet text or topic to quantify the most important
part of that sequence (Vaswani et al. 2017).

$\displaystyle SA_{j}=softmax(Q_{j}K^{T}_{j})V_{j}$ (1)

SA scores are calculated using the equation 1, where
$SA_{t}\in\mathbb{R}^{n_{t}\times d_{a}}$, and
$SA_{u}\in\mathbb{R}^{n_{u}\times d_{a}}$. Here, two SA scores are computed
for SO-MT, while four such SA scores are required for SP-MT model
($SA_{td},SA_{ud},SA_{ts},SA_{us}$) (as shown with black dotted arrow
connections in Fig. 3 (bottom).

Inter Attention (IA) We find out the Inter Attention (IA) scores to learn the
interdependence between different features. IA scores are determined using
below equations where query of one feature is intervened with key and value of
the other. IA scores help to reveal the significant contributions amongst
different inputs to learn optimal features for both tasks. The equations that
represent the IA scores for text and topic ($IA_{tu}$, $IA_{ut}$) are:

$\displaystyle IA_{tu}=softmax(Q_{t}K^{T}_{u})V_{u},$ (2) $\displaystyle
IA_{ut}=softmax(Q_{u}K^{T}_{t})V_{t},$ (3)

where $IA_{tu}\in\mathbb{R}^{n_{t}\times d_{a}}$, and
$IA_{ut}\in\mathbb{R}^{n_{u}\times d_{a}}$. IA equations are represented
graphically with orange and royal red dotted arrows in Fig. 3 (bottom) part.
The SA and IA scores are then concatenated finally, where A is directly used
for shared-only (SO) variant (refer Fig. 1) while average of attention vector
($A_{shared}$) specific to stance and sentiment tasks ($A_{d}$,$A_{s}$) is
used for Shared-private (SP) variant of model (mentioned in Fig. 3 (bottom)).

$\displaystyle A=concat(SA_{t},SA_{u},IA_{tu},IA_{ut}),$ (4) $\displaystyle
A_{d}=concat(SA_{td},SA_{ud},IA_{tud},IA_{utd})),$ (5) $\displaystyle
A_{s}=concat(SA_{ts},SA_{us},IA_{tus},IA_{uts})),$ (6) $\displaystyle
A_{shared}=Average(A_{d},A_{s})$ (7)

Shared-Specific Attention Some of the works mentioned in section 2.2 focused
on the orthogonal relationship between sentiment and stance detection.
Although climate change deniers and proponents dominate in sentiment, there
are a few examples where sentiment does not match attitude/stance (see Table
3). (Wu et al. 2019) also mentioned the disadvantage of the shared-private
model of multi-task learning, explaining that the shared space usually mixes
some task-relevant features, which makes learning different tasks difficult.
To solve the above problems, we use the (Wu et al. 2019) inspired shared-
specific attention, which filters out the useless features that interfere with
the model prediction and only pay attention to the selected features from the
shared layer that lead to the correct predictions of the SP-MT model (see Fig.
3). Next, we describe the sub-modules to achieve the desired result (as shown
in Fig. 3 (top))

Gate Sharing Cell We use similar approach used by authors (Wu et al. 2019)
where a single gate mechanism removes the useless shared features from shared
layer. We first express the cell with reference to stance detection task. The
stance specific, and shared attention scores ($A_{d},A_{shared})$ from
equations (5,7) are passed through dense layers with $d_{s}$ units. The
weights and biases are captured when passing $A_{d}$ through dense layer which
are used for $A_{shared}$, and is expressed as gated sharing cell.

$\displaystyle g_{d}=\sigma(W_{d}.A_{shared}+b_{d})$ (8)

where, $W_{d}\in\mathbb{R}^{n_{t}(d_{s})\times n_{t}(d_{s})}$ and
$b_{d}\in\mathbb{R}^{1\times n_{t}(d_{s})}$. Similar equations are followed
for sentiment task also, hence, we do not iterate here. The final output of
the shared features for both the task after filtering will be represented as :

$\displaystyle G_{d}=g_{d}\odot A_{shared},$ (9) $\displaystyle
G_{s}=g_{s}\odot A_{shared}$ (10)

where $\odot$ denotes element-wise multiplication. Fig. 3 (top) shows the
connections of gate sharing cell for stance detection and sentiment tasks.

Shared-Private specific Inter Attention (SPIA) We use similar concept of the
inter attention of feature-specific attention (equations 2, 3). We capture the
important shared features relevant to the specific task, by using query matrix
of the particular task (stance/sentiment) and keys and values of the shared
task. The attention vectors ($A_{d},A_{s}$) are passed through fully connected
layers with $d_{s}$ units to create $Q_{d},Q_{s}$ for stance and sentiment
tasks, while $A_{shared}$ is passed through dense layer to generate
$K_{shared}$, and $V_{shared}$. The equations are represented visually with
green dotted arrows in Fig. 3 (top) part.

$\displaystyle SPIA_{d}=softmax(Q_{d}K^{T}_{shared})V_{shared},$ (11)
$\displaystyle SPIA_{s}=softmax(Q_{s}K^{T}_{shared})V_{shared}$ (12)

Fusion The final output of the shared layer is the fusion of the output of the
gated cell and shared-private specific inter attention. Recently, fusion
technique with absolute difference and element-wise product is found to be
effective in (Mou et al. 2015).

$\displaystyle C_{1}=[G_{d};SPIA_{d};G_{d}-SPIA_{d};G_{d}\odot SSIA_{d}];$
(13) $\displaystyle C_{2}=[G_{s};SPIA_{s};G_{s}-SPIA_{s};G_{s}\odot
SSIA_{s}];$ (14) $\displaystyle F_{dshared}=\tanh(W_{fd}.C_{1}+b_{fd}),$ (15)
$\displaystyle F_{sshared}=\tanh(W_{fs}.C_{2}+b_{fs})$ (16)

#### Classification Layer

The final representation of the tweet obtained($A_{d}$,$A_{s}$) is passed
through separate outputs for stance and sentiment tasks (for SO-MT model (Fig.
1) ), however individual task specific tweet representations along with the
shared layer representations are passed through two output channels, subjected
to polarisation ($A_{d}$,$F_{dshared}$) and sentiment ($A_{s}$,$F_{sshared}$)
tasks for SP-MT Model (Fig. 3). The task specific and shared loss are used as

$\displaystyle L_{total}=L_{task}+\lambda L_{shared}$ (17)

where $\lambda$ = $0.5$ is a hyper-parameter (Liu, Qiu, and Huang 2017).

##  5 Experiment

###  5.1 Datasets

We evaluate our model performance and compare with the other baselines on the
two datasets:

Climate Change Data The details of the data collection and statistics are
covered in section 3 and table 2.

SemEval is provided in the SemEval-2016 shared task 6.A on tweet stance
detection (Mohammad et al. 2016). Each tweet is in favor, against or neutral
corresponding to one of the five targets: Atheism, Climate Change is a Real
Concern, Feminist Movement, Hillary Clinton, and Legalization of
Abortion(Abortion). There has been several works that use this benchmark
dataset for stance classification.

Model | Single-Task Stance Detection | Single-Task Sentiment
---|---|---
Text | Text+Topic | Text | Text+Topic
| Acc. | F1 | Acc. | F1 | Acc. | F1 | Acc. | F1
Shared-only (SO) | 76.73$\pm$2.48 | 70.75$\pm$2.11 | 79.56$\pm$1.10 | 75.72 $\pm$1.08 | 71.61$\pm$0.45 | 70.11$\pm$0.79 | 75.82$\pm$2.03 | 74.63$\pm$2.41
SO + Self Attn. (SA) | 80.29$\pm$0.55 | 74.24$\pm$0.31 | 82.88$\pm$0.78 | 77.56 $\pm$0.87 | 73.49$\pm$0.92 | 72.57$\pm$1.27 | 77.68$\pm$1.27 | 76.59$\pm$1.66
SO + Inter Attn. (IA) | 81.33$\pm$1.04 | 73.35$\pm$1.41 | 83.13$\pm$1.42 | 76.04 $\pm$1.11 | 73.68$\pm$3.20 | 72.23$\pm$3.14 | 77.93$\pm$2.01 | 74.11$\pm$1.98
| SO + SA + IA
---
(Feature-Specific Attn.)
82.15$\pm$0.05 | 76.78$\pm$0.81 | 85.01$\pm$1.05 | 80.03 $\pm$2.42 | 76.21$\pm$0.84 | 74.95$\pm$1.02 | 80.81$\pm$1.29 | 79.28$\pm$1.44

Table 5: Results of the Single-Task Stance Detection and Sentiment tasks for
various combinations

Model | Multi-Task Stance Detection + Sentiment
---|---
Stance Detection | Sentiment
Text | Text+Topic | Text | Text+Topic
Acc. | F1 | Acc. | F1 | Acc. | F1 | Acc. | F1
Shared-only (SO) | 81.16$\pm$1.89 | 77.01$\pm$2.61 | 84.86$\pm$1.13 | 81.66 $\pm$3.28 | 75.44$\pm$1.23 | 74.40$\pm$1.75 | 80.49$\pm$2.05 | 79.19$\pm$2.8
Shared-Private (SP) | 84.64$\pm$1.95 | 81.53$\pm$2.55 | 87.47$\pm$2.01 | 84.24 $\pm$2.04 | 77.93$\pm$2.09 | 74.11$\pm$2.19 | 85.28$\pm$0.91 | 84.58$\pm$1.02
| Shared-Private (SP) +
---
Feature-Specific Attn.
86.49$\pm$2.29 | 81.67$\pm$3.21 | 91.31$\pm$1.06 | 85.93 $\pm$1.22 | 81.46$\pm$0.71 | 80.71$\pm$0.39 | 85.46$\pm$0.9 | 84.73$\pm$1.10
| Shared-Private (SP) +
---
Shared-Specific Attn.
88.10$\pm$1.04 | 84.09$\pm$1.39 | 92.29$\pm$1.32 | 86.12 $\pm$1.4 | 81.67$\pm$0.61 | 80.89$\pm$0.32 | 87.59$\pm$0.45 | 87.07$\pm$0.39
| Shared-Private +
---
Feature-Sp. Attn. +
Shared-Sp. Attn.
(SP-MT)
89.99$\pm$2.67 | 86$\pm$2.02 | 93.95$\pm$1.27 | 90.24 $\pm$1.16 | 84.60$\pm$0.44 | 83.98$\pm$0.81 | 89.08$\pm$1.01 | 88.48$\pm$1.60

Table 6: Results of the Multi-Task Stance Detection and Sentiment tasks for
various combinations

###  5.2 Set-up

We use the python-based library Keras777https://keras.io/ at various stages of
our implementations. For the experiments, we perform stratified k-fold cross-
validation on our dataset, oversample the minority class (deniers) in the k-1
training data using the sklearn resampling technique, and report the averaged
scores and standard deviation (over 5 folds) for the accuracy and F1 scores.
We select $m=5$ and $p=10$, which fits our dataset well, where $m$ denotes the
number of most similar topics and each topic contains $p$ number of words for
the topic feature of each tweet. In the feature extraction sub-module, Bi-LSTM
( $d_{l}$ ) with $100$ memory cells is used. The dimensions $d_{a}$ and
$d_{s}$ of the fully connected layers used in the attention framework to
extract queries, keys and values for feature-specific and shared-specific
attention (refer section 4.2) are used with $100$ units each. The stance and
sentiment output channels contain $2$ and $3$ output neurons, respectively.
The loss functions binary cross-entropy and categorical cross-entropy are used
for the stance and sentiment output channels, respectively. The experiments
are run on an NVIDIA GeForce GTX 1080Ti GPU and the models are optimized using
Adam optimizer with a learning rate of $0.0001$. All these values are selected
using TPE in the hyperopt python library (Bergstra et al. 2013) and after a
thorough sensitivity analysis of the parameters that minimise the loss
functions.

###  5.3 Baseline Techniques

We compare our proposed approach with the following baselines on our climate
change dataset :

Logistic regression (Argyris et al. 2021): This study use logistic regression
with Count Vectorizer feature extraction method to classify vaccine-related
tweets into provaccine, antivaccine, and neutral stances.

ESD (Vychegzhanin and Kotelnikov 2021): The authors form a relevant feature
set using an ensemble of feature selection methods and propose the model ESD
by selecting an optimal ensemble of classifiers. They evaluate the performance
of the model using the UKP Sentential Argument Mining Corpus and the
SemEval-2016 dataset.

HAN (Wang et al. 2020): In this article, researchers proposed a hierarchical
attention neural model, focusing on different features such as document,
sentiment, dependency, and argument representations. They evaluated the model
performance on SemEval-2016 and the H&N14 dataset.

AT-JSS-LEX (Li and Caragea 2019): is a multi-task framework for stance
detection with sentiment analysis as auxiliary task. The attention mechanism
of the model is guided by target-specific attention along with sentiment and
stance lexicons.

MNB (Kabaghe and Qin 2019): Multinomial naive bayes performed better with
respect to other models proposed in the study, to classify tweets into
positive, negative or neutral beliefs towards climate change.

DNN (Chen, Zou, and Zhao 2019): Deep Neural Network (DNN) is used as a
classifier to identify users who either believe or deny climate change based
on the content of tweets. The model’s performance is assessed on the real-time
collection of climate change twitter data.

SVM-ngram (Sobhani, Mohammad, and Kiritchenko 2016): is trained on word and
character n-grams features for stance detection task on SemEval 2016 dataset.
The model surpassed the best model in SemEval-2016 competition.

We evaluate our model performance on SemEval 2016 dataset and contrast with
the these state-of-the art models : ESD (Vychegzhanin and Kotelnikov 2021),
HAN (Wang et al. 2020), AT-JSS-LEX (Li and Caragea 2019), and SVM-ngram
(Sobhani, Mohammad, and Kiritchenko 2016) as described above.

##  6 Result and Analysis

In this section, we investigate the performance of the proposed approach. We
first compare different single-task and multi-task variants and then compare
them with the state-of-the-art methods mentioned in section 5.3. We also
analyze the importance of each feature and the different variants of the
attention framework. We report all the results of the five-fold cross-
validation (mean and standard deviation of accuracy and F1 score) for the
different combinations of the proposed system.

No. | Tweet | Sentiment | True |  | Predicted
---
Stance
| Predicted
---
Stance +
Sentiment
1. | My family support Oil and Gas! ClimateHoax | positive | denier | believer | denier
2. |  | Once again brainwashing kids to push the green tax agenda,
---
under the globalwarminghoax umbrella! Stop
negative | denier | denier | denier
3. | His Green BS policies will send us back to the Dark Ages. ClimateChangeHoax | negative | denier | believer | denier
4. |  | ClimateHoax The climate has fluctuated since the time of creation,
---
and nothing those people will do can change that one way or the other
neutral | denier | believer | denier
5. |  | And yet, there are those who deny climate change?? Ice-shelves breaking off,
---
heat waves, etc. Sad ScienceIsReal
negative | believer | denier | believer
6. |  | Have you seen this? Its very moving. We definitely need more ClimateAction
---
positive | believer | believer | believer
7. |  | For those adamant that global warming is real, THIS is Today in Alaska.
---
Four inches of snow overnight, and still coming down! ClimateChange
neutral | denier | believer | believer
8. |  | I am glad you went by plane. Way better for our climate instead of zoommeetings…
---
positive | believer | denier | denier
9. |  | Over 60° today. Over 6” of snow tomorrow. But yeah, climatechange is total bullshit right?
---
negative | believer | denier | denier

Table 7: Few example tweets with ground truth and predicted labels for single
and multi-task models

The Tables 5 and 6 illustrate the results of the single-task and the various
combinations of the proposed multi-task models for both the stance detection
and sentiment tasks. It is evident that the addition of topic words
consistently improves the performance of the models. This improvement means
that the proposed architecture makes very effective use of the interaction
between input features. This shows the importance of incorporating multiple
features for various analysis tasks.

###  6.1 Comparison amongst Single-Task and Multi-task Framework

From the tables 5 and 6, the multi-task variants perform better than the
single-task variants by achieving an average macro F1 score of $90.24$ and
$88.48$ for the stance detection (primary) and sentiment analysis (auxiliary)
tasks respectively. The results show that the sentiment and stance tasks
improve each other’s performance when learned together. The single stance
detection task is able to correctly label some tweets from deniers and
believer that contain predominantly negative and positive sentiments,
respectively (examples $2$ and $6$ from Table 7). However, examples $1$, $4$,
and $5$ from Table 7 clearly show that the stance task, together with the
sentiment analysis task, is able to unambiguously identify denier and
believers tweets with the corresponding less dominant positive, neutral, and
negative sentiment polarities. As stated earlier, we consider sentiment
analysis as an auxiliary task that supports the main task, i.e., stance
detection. However, we report the performance of the sentiment task for the
proposed model for both single-task and multi-task frameworks in the tables 5
and 6 to illustrate the impact of the main task on the auxiliary task and to
show that the multiple features in the form of tweet text and topic words, as
well as the attention framework, also benefit the sentiment classification
task. However, we do not make explicit efforts to improve the model
performance on the auxiliary task.

Model | Training Time (secs) | Mean Accuracy
---|---|---
| Single Task Best
---
(SO + SA + IA) [Table 5]
870 | 85.01
Multi Task Variants [Refer Table 6]
Shared-only (SO) | 918 | 84.86
Shared-Private (SP) | 1218 | 87.47
| Shared-Private (SP) +
---
Feature-Specific Attn.
1419 | 91.31
| Shared-Private (SP) +
---
Shared-Specific Attn.
1506 | 92.29
| Shared-Private +
---
Feature-Sp. Attn. +
Shared-Sp. Attn. (SP-MT)
1791 | 93.95

Table 8: Training time of Different Text + Topic Models

###  6.2 Comparison amongst Different Multi-task Frameworks

Table 6 shows the improvement of the multi-task framework from the shared-only
variant (Fig. 1) to the shared-private multi-task model(SP-MT) (Fig. 3). The
inclusion of feature-specific and shared-specific attention frameworks helped
the multi-task models focus on the important parts of the features and
effectively discard the useless shared features, resulting in a $7.40\%$
increase in accuracy and a $12.46\%$ increase in F1 score. Furthermore, in
Table 8 we give the training times of the best performing single-task model
and different variants of the multi-task model for 20 epochs to analyze the
additional time required by the best performing multi-task model with text and
topic as input features (SP-MT) compared to other variants. As can be seen
from the table 8, SP-MT requires about $15$ more minutes (approximately twice
the time) to achieve a $10.51$% improvement in accuracy compared to the best
performing single-task model, while SP-MT requires $9.5$ more minutes to
achieve a $7.4$% improvement in performance compared to the shared-private
multi-task variant.

All results reported here are statistically significant as we performed a
t-test at the $5\%$ significance level (Welch 1947) against the null
hypothesis, which states that the mean accuracy/F1 score of all the multi-task
variants is more when compared to the the best performing proposed model SP-MT
(Shared-Private + Feature-Specific Attention +Shared-Specific Attention)
[refer table 6]. If the p-value is significant ($p<0.05$), we reject the null
hypothesis. Our best performing proposed model outperforms all the other
multi-task variants while meeting statistical significance under t-tests
($p<0.05$). For the confidence analysis, we also report the p-values and
t-test statistics of all the multi-task variant models compared to the best
performing model in tables $1$ and $2$ in Supplementary.

###  6.3 Comparison with the Baseline Methods

In Table 9, we report the results for the baseline methods by re-implementing
them on the Climate Change dataset (section 3). It is observed that our
proposed multi-task approach SP-MT outperforms the SOTA approaches in terms of
accuracy and F1 score. Our best performing model achieved better results
compared to ESD (Vychegzhanin and Kotelnikov 2021) and HAN (Wang et al. 2020).
This highlights that the shared-private multi-tasking approach takes advantage
of task-specific and invariant features to improve classification task
performance. Although the AT-JSS-LEX (Li and Caragea 2019) model was
implemented with a multi-tasking approach, our model performs better because
it keeps the task-dependent and task-independent feature spaces separate and
removes the useless shared features that hinder task performance of the stance
detection, demonstrating the importance of the shared-specific attention
framework. It is also observed that the methods that use sentiment features
(ESD, HAN, and AT-JSS-LEX) perform better than the other baselines. This
proves the importance of the proposed sentiment analysis approach for climate
change. Our best performing single-task polarization framework (Table 5) also
outperforms MNB, DNN, and SVM-ngram approaches. This justifies the benefits of
using topic words in addition to tweet text and feature-specific attention
framework to improve the performance of the model.
Consequently, we also performed a comparative analysis of the proposed multi-
tasking approach SP-MT with the state-of-the-art models (SOTA) on the SemEval
2016 dataset. The model is trained with three polarized classes (Favour,
Against, None) and the metrics ($F_{avg}$, $MacF_{avg}$) are evaluated
according to the procedure defined in (Li and Caragea 2019). Table 10 shows
that our approach outperforms other methods with an overall $MacF_{avg}$ value
of $66.84$. Our proposed framework performs better in the climate, feminism,
and abortion domains, while the $F_{avg}$ values are comparable in the atheism
and Hillary domains, showing that our framework generalises well in different
domains.

###  6.4 Error Analysis

We perform an in-depth error analysis to understand where the proposed model
has faltered. These are the following scenarios: (i) The climate change
dataset is an imbalanced dataset with a high proportion of believer tweets,
resulting in low F1 values compared to accuracy. Although we applied
oversampling to partially counter this problem, even finer categories of
believers can be identified and labeled, which can be beneficial for the model
to learn different classes with a clear separation of distribution in tweets,
such as “tweet conveys causes of climate change” or “tweet believes in human-
caused climate change”. (ii) We determine the frequency of unigrams and
bigrams extracted using TF-IDF and find that some of the denier’s tweets
containing either rarely used keywords or keywords frequently used in
believers’ tweets were misclassified. For example, the denier’s tweet in
example $7$ (table 7) contains words such as real, snow overnight, which are
most commonly found in believers’ tweets and confuse the model and lead to an
incorrect prediction. (iii) We investigated that of the total misclassified
denier tweets, $35.7\%$ of the tweets contained sarcasm to express their
denial. Of the sarcastic denier tweets, $50.16\%$ of the tweets have positive
sentiment, $31.70\%$ have neutral sentiment, and the rest have negative
sentiment, while $25.78\%$ of the misclassified believer tweets have sarcastic
labels (examples $8$ and $9$ of table 7). The labeling of sarcasm is based on
the majority vote of three trained annotators with an inter-rater agreement of
$0.78$, calculated with the Fleiss-Kappa measure. This motivated us to
investigate the presence of sarcasm in climate change tweets to further
improve the performance of the model as a part of our future work.

Model | Accuracy | F1-score
---|---|---
Proposed SP-MT (Fig. 3) | 93.95 | 90.24
LR (Argyris et al. 2021) | 81.48 | 81.00
ESD (Vychegzhanin and Kotelnikov 2021) | 89.65 | 85.11
HAN (Wang et al. 2020) | 89.47 | 86.00
AT-JSS-LEX (Li and Caragea 2019) | 88.02 | 84.01
MNB (Kabaghe and Qin 2019) | 85.44 | 78.08
DNN (Chen, Zou, and Zhao 2019) | 84.61 | 76.23
SVM-ngram (Sobhani, Mohammad, and Kiritchenko 2016) | 85.55 | 66.33

Table 9: Results of Proposed Framework SP-MT with baselines on our Climate
Change Dataset

Model |  | Atheism
---
$F_{avg}$
| Climate
---
$F_{avg}$
| Feminism
---
$F_{avg}$
| Hillary
---
$F_{avg}$
| Abortion
---
$F_{avg}$
Mac $F_{avg}$
Proposed SP-MT(Fig.3) | 69.5 | 63.5 | 63.2 | 67.5 | 70.5 | 66.84
| ESD
---
(Vychegzhanin and
Kotelnikov 2021)
66.64 | 43.82 | 62.85 | 67.79 | 64.94 | 61.20
HAN (Wang et al. 2020) | 70.53 | 49.56 | 57.50 | 61.23 | 66.16 | 61.00
| AT-JSS-LEX
---
(Li and Caragea 2019)
69.22 | 59.18 | 61.49 | 68.33 | 68.41 | 65.33
| SVM-ngram
---
(Sobhani, Mohammad,
and Kiritchenko2016)
65.19 | 42.35 | 57.46 | 58.63 | 66.42 | 58.01

Table 10: Results of Proposed Framework SP-MT with baselines on SemEval 2016
Dataset

##  7 Privacy and Ethics

Although social media offers innovative ways to raise awareness about climate
change, phenomena such as climate denial and climate delay have become a
serious problem for scientists and the government to convince people of the
importance of understanding the current climate crisis. Climate change deniers
are not only skeptical about climate change but also emphasize the
disadvantages of all measures proposed to combat climate change and abandon
the idea that it is not possible to prevent climate change. This often leads
to the spread of misinformation, resulting in a delay in the implementation of
effective climate change mitigation measures (Zhou and Shen 2021). Since our
work is dedicated to classifying Twitter content into climate change deniers
or believers, the proposed approach can be useful for government agencies,
researchers, and tech companies that monitor such content on social media to
identify and intervene tweets from climate change deniers. The proposed
approach is useful in combating climate misinformation by identifying posts by
climate change deniers and reducing the spread of such content that is deemed
false or misleading.

The input feature of the proposed model, such as the tweet text, is available
as soon as the user posts something. However, the topic feature can be
extracted by performing topic modeling for a collection of tweets after a
fixed interval, e.g., after every 5, 10, 15, or t minutes of duration.
Therefore, our proposed approach can be used in a real-time environment by
interested agencies and authorities to classify social media content into one
of the two polarized classes.

Although we conduct our work with public data from social media, however, we
are committed to protecting the privacy of individuals and therefore avoid
providing personally identifiable content. The dataset that is made publicly
available consists only of tweet IDs and comments.

##  8 Conclusion and Future Work

In this paper, we focus on the importance of classifying Twitter content into
climate change deniers and believers, because climate change deniers emphasize
the downsides of any action to address climate change, which often leads to
the spread of misinformation and delays the implementation of effective action
to mitigate climate change. Our proposed approach, implemented in real-time,
can be useful for government agencies, researchers, and tech companies in
combating climate misinformation by identifying content that denies climate
change and reducing the spread of such posts that are considered misleading.
In this work, we investigated the role of sentiment in classifying stance of
the tweets related to climate change. We curate a novel dataset that includes
annotations for both stance detection and sentiment analysis tasks, which will
be useful to the research community in exploring other needed classification
tasks. We propose a shared-private multi-task framework for the optimization
of stance detection task benefiting from the sentiment analysis (auxiliary
task). The proposed module uses feature-specific and shared-specific attention
to fuse multiple features and learn useful and relevant private and shared
features for both tasks. The results show that multi-tasking increased the
performance of the stance detection task compared to its uni-modal and single-
task variants. Although we examined the performance of the proposed approach
in detecting attitudes in the domain of climate change, the performance of the
model on the SemEval dataset shows that it is much more broadly applicable
beyond the domain of climate change, suggesting that our framework can
generalize well in different domains. Future work will attempt to analyse what
other aspects of natural language, such as sarcasm, aspect-based sentiment,
and emotion recognition, might help to more accurately classify attitudes
toward climate change. The inclusion of other modality encodings such as
images, emoji, and advanced architectures will also be the subject of our
future work.

## References

  * Anwar et al. (2021) Anwar, A.; Ilyas, H.; Yaqub, U.; and Zaman, S. 2021.  Analyzing QAnon on Twitter in Context of US Elections 2020: Analysis of User Messages and Profiles Using VADER and BERT Topic modeling.  In _DG. O2021: The 22nd Annual International Conference on Digital Government Research_ , 82–88.
  * Argyris et al. (2021) Argyris, Y. A.; Monu, K.; Tan, P.-N.; Aarts, C.; Jiang, F.; and Wiseley, K. A. 2021\.  Using Machine Learning to Compare Provaccine and Antivaccine Discourse Among the Public on Social Media: Algorithm Development Study.  _JMIR public health and surveillance_ 7(6): e23105.
  * Bahdanau, Cho, and Bengio (2014) Bahdanau, D.; Cho, K.; and Bengio, Y. 2014.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_ .
  * Baziotis, Pelekis, and Doulkeridis (2017) Baziotis, C.; Pelekis, N.; and Doulkeridis, C. 2017.  Datastories at semeval-2017 task 4: Deep lstm with attention for message-level and topic-based sentiment analysis.  In _Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017)_ , 747–754.
  * Bergstra et al. (2013) Bergstra, J.; Yamins, D.; Cox, D. D.; et al. 2013.  Hyperopt: A python library for optimizing the hyperparameters of machine learning algorithms.  In _Proceedings of the 12th Python in science conference_ , volume 13, 20. Citeseer.
  * Cann, Weaver, and Williams (2021) Cann, T. J.; Weaver, I. S.; and Williams, H. T. 2021.  Ideological biases in social sharing of online information about climate change.  _Plos one_ 16(4): e0250656.
  * Caruana (1997) Caruana, R. 1997.  Multitask learning.  _Machine learning_ 28(1): 41–75.
  * Chauhan, Kumar, and Ekbal (2019) Chauhan, D. S.; Kumar, R.; and Ekbal, A. 2019.  Attention based shared representation for multi-task stance detection and sentiment analysis.  In _International Conference on Neural Information Processing_ , 661–669. Springer.
  * Chen, Zou, and Zhao (2019) Chen, X.; Zou, L.; and Zhao, B. 2019.  Detecting climate change deniers on twitter using a deep neural network.  In _Proceedings of the 2019 11th International Conference on Machine Learning and Computing_ , 204–210.
  * Clayton (2020) Clayton, S. 2020.  Climate anxiety: Psychological responses to climate change.  _Journal of Anxiety Disorders_ 74: 102263.
  * Cody et al. (2015) Cody, E. M.; Reagan, A. J.; Mitchell, L.; Dodds, P. S.; and Danforth, C. M. 2015\.  Climate change sentiment on Twitter: An unsolicited public opinion poll.  _PloS one_ 10(8): e0136092.
  * Dahal, Kumar, and Li (2019) Dahal, B.; Kumar, S. A.; and Li, Z. 2019.  Topic modeling and sentiment analysis of global climate change tweets.  _Social Network Analysis and Mining_ 9(1): 1–20.
  * Devlin et al. (2019) Devlin, J.; Chang, M.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , 4171–4186.
  * El Barachi et al. (2021) El Barachi, M.; AlKhatib, M.; Mathew, S.; and Oroumchian, F. 2021.  A novel sentiment analysis framework for monitoring the evolving public opinion in real-time: Case study on climate change.  _Journal of Cleaner Production_ 127820\.
  * Garimella et al. (2018) Garimella, K.; Morales, G. D. F.; Gionis, A.; and Mathioudakis, M. 2018.  Quantifying controversy on social media.  _ACM Transactions on Social Computing_ 1(1): 1–27.
  * Gautam, V, and Masud (2021) Gautam, A.; V, V.; and Masud, S. 2021.  Fake News Detection System Using XLNet Model with Topic Distributions: CONSTRAINT@AAAI2021 Shared Task.  In Chakraborty, T.; Shu, K.; Bernard, H. R.; Liu, H.; and Akhtar, M. S., eds., _Combating Online Hostile Posts in Regional Languages during Emergency Situation - First International Workshop, CONSTRAINT 2021, Collocated with AAAI 2021_ , volume 1402 of _Communications in Computer and Information Science_ , 189–200. Springer.
  * Grootendorst (2020) Grootendorst, M. 2020.  BERTopic: Leveraging BERT and c-TF-IDF to create easily interpretable topics.  doi:10.5281/zenodo.4381785.  URL https://doi.org/10.5281/zenodo.4381785.
  * Hutto and Gilbert (2014) Hutto, C.; and Gilbert, E. 2014.  Vader: A parsimonious rule-based model for sentiment analysis of social media text.  In _Proceedings of the International AAAI Conference on Web and Social Media_ , volume 8.
  * Jang and Hart (2015) Jang, S. M.; and Hart, P. S. 2015.  Polarized frames on “climate change” and “global warming” across countries and states: Evidence from Twitter big data.  _Global Environmental Change_ 32: 11–17.
  * Jiang et al. (2017) Jiang, Y.; Song, X.; Harrison, J.; Quegan, S.; and Maynard, D. 2017.  Comparing Attitudes to Climate Change in the Media using sentiment analysis based on Latent Dirichlet Allocation.  In _Proceedings of the 2017 EMNLP Workshop: Natural Language Processing meets Journalism_ , 25–30.
  * Kabaghe and Qin (2019) Kabaghe, C.; and Qin, J. 2019.  Classifying Tweets Based on Climate Change Stance.  _Training_ 66(60.9): 61–6.
  * Li and Caragea (2019) Li, Y.; and Caragea, C. 2019.  Multi-task stance detection with sentiment and stance lexicons.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 6299–6305.
  * Lineman et al. (2015) Lineman, M.; Do, Y.; Kim, J. Y.; and Joo, G.-J. 2015.  Talking about climate change and global warming.  _PloS one_ 10(9): e0138996.
  * Littman and Wrubel (2019) Littman, J.; and Wrubel, L. 2019.  Climate Change Tweets Ids.  doi:10.7910/DVN/5QCCUU.  URL https://doi.org/10.7910/DVN/5QCCUU.
  * Liu, Qiu, and Huang (2017) Liu, P.; Qiu, X.; and Huang, X. 2017.  Adversarial Multi-task Learning for Text Classification.  In Barzilay, R.; and Kan, M., eds., _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017_ , 1–10.  URL https://doi.org/10.18653/v1/P17-1001.
  * Luo, Card, and Jurafsky (2020) Luo, Y.; Card, D.; and Jurafsky, D. 2020.  Detecting Stance in Media on Global Warming.  _arXiv preprint arXiv:2010.15149_ .
  * Majumder et al. (2019) Majumder, N.; Poria, S.; Peng, H.; Chhaya, N.; Cambria, E.; and Gelbukh, A. 2019\.  Sentiment and sarcasm classification with multitask learning.  _IEEE Intelligent Systems_ 34(3): 38–43.
  * Misra et al. (2016) Misra, A.; Ecker, B.; Handleman, T.; Hahn, N.; and Walker, M. 2016.  NLDS-UCSC at SemEval-2016 Task 6: A Semi-Supervised Approach to Detecting Stance in Tweets.  In _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)_. Association for Computational Linguistics.
  * Mohammad et al. (2016) Mohammad, S.; Kiritchenko, S.; Sobhani, P.; Zhu, X.; and Cherry, C. 2016.  Semeval-2016 task 6: Detecting stance in tweets.  In _Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016)_ , 31–41.
  * Mou et al. (2015) Mou, L.; Men, R.; Li, G.; Xu, Y.; Zhang, L.; Yan, R.; and Jin, Z. 2015.  Natural language inference by tree-based convolution and heuristic matching.  _arXiv preprint arXiv:1512.08422_ .
  * Myhre et al. (2013) Myhre, G.; Shindell, D.; Bréon, F.-M.; Collins, W.; Fuglestvedt, J.; Huang, J.; Koch, D.; Lamarque, J.-F.; Lee, D.; Mendoza, B.; et al. 2013.  Climate change 2013: the physical science basis. Contribution of Working Group I to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change.  _K., Tignor, M., Allen, SK, Boschung, J., Nauels, A., Xia, Y., Bex, V., and Midgley, PM, Cambridge University Press Cambridge, United Kingdom and New York, NY, USA_ .
  * Samantray and Pin (2019) Samantray, A.; and Pin, P. 2019.  Credibility of climate change denial in social media.  _Palgrave Communications_ 5(1): 1–8.
  * Sen, Flöck, and Wagner (2020) Sen, I.; Flöck, F.; and Wagner, C. 2020.  On the reliability and validity of detecting approval of political actors in tweets.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 1413–1426.
  * Shi, Visschers, and Siegrist (2015) Shi, J.; Visschers, V. H.; and Siegrist, M. 2015.  Public perception of climate change: The importance of knowledge and cultural worldviews.  _Risk Analysis_ 35(12): 2183–2201.
  * Singh et al. (2021) Singh, A.; Saha, S.; Hasanuzzaman, M.; and Dey, K. 2021.  Multitask learning for complaint identification and sentiment analysis.  _Cognitive Computation_ 1–16.
  * Sobhani, Mohammad, and Kiritchenko (2016) Sobhani, P.; Mohammad, S.; and Kiritchenko, S. 2016.  Detecting stance in tweets and analyzing its interaction with sentiment.  In _Proceedings of the fifth joint conference on lexical and computational semantics_ , 159–169.
  * Spitzer et al. (1967) Spitzer, R. L.; Cohen, J.; Fleiss, J. L.; and Endicott, J. 1967.  Quantification of agreement in psychiatric diagnosis: A new approach.  _Archives of General Psychiatry_ 17(1): 83–87.
  * Tyagi et al. (2020a) Tyagi, A.; Babcock, M.; Carley, K. M.; and Sicker, D. C. 2020a.  Polarizing tweets on climate change.  In _International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation_ , 107–117. Springer.
  * Tyagi et al. (2020b) Tyagi, A.; Field, A.; Lathwal, P.; Tsvetkov, Y.; and Carley, K. M. 2020b.  A computational analysis of polarization on Indian and Pakistani social media.  In _International Conference on Social Informatics_ , 364–379. Springer.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , 5998–6008.
  * Vychegzhanin and Kotelnikov (2021) Vychegzhanin, S.; and Kotelnikov, E. 2021.  A New Method for Stance Detection Based on Feature Selection Techniques and Ensembles of Classifiers.  _IEEE Access_ 9: 134899–134915.
  * Wang et al. (2020) Wang, Z.; Sun, Q.; Li, S.; Zhu, Q.; and Zhou, G. 2020.  Neural Stance Detection With Hierarchical Linguistic Representations.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 28: 635–645.
  * Weber, Garimella, and Batayneh (2013) Weber, I.; Garimella, V. R. K.; and Batayneh, A. 2013.  Secular vs. islamist polarization in egypt on twitter.  In _Proceedings of the 2013 IEEE/ACM international conference on advances in social networks analysis and mining_ , 290–297.
  * Welch (1947) Welch, B. L. 1947.  The generalization of ‘STUDENT’S’problem when several different population varlances are involved.  _Biometrika_ 34(1-2): 28–35.
  * Wu et al. (2019) Wu, L.; Rao, Y.; Jin, H.; Nazir, A.; and Sun, L. 2019.  Different absorption from the same sharing: Sifted multi-task learning for fake news detection.  In _Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 4636–4645.
  * Zhou et al. (2016) Zhou, P.; Qi, Z.; Zheng, S.; Xu, J.; Bao, H.; and Xu, B. 2016.  Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling.  In Calzolari, N.; Matsumoto, Y.; and Prasad, R., eds., _COLING 2016, 26th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, December 11-16, 2016, Osaka, Japan_ , 3485–3495. ACL.  URL https://aclanthology.org/C16-1329/.
  * Zhou and Shen (2021) Zhou, Y.; and Shen, L. 2021.  Confirmation Bias and the Persistence of Misinformation on Climate Change.  _Communication Research_ 00936502211028049\.