# Detecting Political Opinions in Tweets through Bipartite Graph Analysis: A
Skip Aggregation Graph Convolution Approach

Xingyu Peng  Zhenkun Zhou<EMAIL_ADDRESS>Chong Zhang  Ke Xu

###### Abstract

Public opinion is a crucial factor in shaping political decision-making.
Nowadays, social media has become an essential platform for individuals to
engage in political discussions and express their political views, presenting
researchers with an invaluable resource for analyzing public opinion. In this
paper, we focus on the 2020 US presidential election and create a large-scale
dataset from Twitter. To detect political opinions in tweets, we build a user-
tweet bipartite graph based on users’ posting and retweeting behaviors and
convert the task into a Graph Neural Network (GNN)-based node classification
problem. Then, we introduce a novel skip aggregation mechanism that makes
tweet nodes aggregate information from second-order neighbors, which are also
tweet nodes due to the graph’s bipartite nature, effectively leveraging user
behavioral information. The experimental results show that our proposed model
significantly outperforms several competitive baselines. Further analyses
demonstrate the significance of user behavioral information and the
effectiveness of skip aggregation.

###### keywords:

political opinion mining , graph neural networks , social media

††journal: Pattern Recognition

[label1] organization=State Key Lab of Software Development Environment,
Beihang University, city=Beijing, postcode=100191, country=China

[label2] organization=School of Statistics, Capital University of Economics
and Business, city=Beijing, postcode=100070, country=China

##  1 Introduction

In today’s digital age, social media has emerged as an indispensable tool for
the general public to engage in discussions on political issues, such as
elections, taxes, education, and regulations. As was seen in the 2016 US
presidential election [1] and the 2019 Argentina presidential election [2],
social media provides a platform for candidates to engage with potential
voters, share their policies, and build their brand during the pre-election
phase. Similarly, voters can use social media to learn more about the
candidates, their policies, and their positions on important issues, as well
as express their opinions on the election. With the occurrence of such
political events, significant numbers of reviews and responses surface online,
making it possible to capture public opinion and social trends. However,
manually analyzing such a massive amount of textual data is extremely time-
consuming and costly, so it is imperative to automate the analysis procedure.

Early studies have primarily concentrated on people’s opinions about hotels,
restaurants, electronics, and other consumer products. In contrast,
understanding opinions in a political context [3, 4, 5, 6, 7, 8, 9, 10, 11,
12, 13] has received less attention, and is generally more difficult due to
nuanced language, ambiguous sentiment, and complex context. In this study, our
focus is on the 2020 US presidential election, and we aim to ascertain the
political polarities of election-related tweets that were posted during the
pre-election period. Compared to news articles [7, 8, 9, 10], debate
transcripts [11, 12], or tweets from official sources like legislators, news
agencies, and politicians [13], which typically follow strict grammatical
rules, analyzing tweets posted by ordinary users presents unique challenges,
such as the brevity of the text, absence of contextual information, and
frequent use of emoticons, abbreviations, and hashtags.

Figure 1: Hashtag clouds. The words or multi-word phrases in the clouds are
the most frequently occurring hashtags related to the 2020 US presidential
election. The hashtags in blue express support for Biden or opposition to
Trump (pro-Biden), while the hashtags in red express support for Trump or
opposition to Biden (pro-Trump). The dimension of the hashtags is proportional
to their frequency.

Additionally, labels for such social data are difficult to obtain. Because of
the vast amount of data, manual annotation methods such as crowdsourcing and
editorial review [7] are impractical. Some studies [13, 11, 12] annotate texts
based on authors’ or speakers’ ideological positions, whereas the political
ideologies of personal Twitter accounts are generally unknown. Actually,
hashtags are frequently used by Twitter users to provide context and convey
their sentiments or opinions on particular topics or issues. For example, a
user might include a hashtag like #ActOnClimate to advocate for action on
climate change, or #BlackLivesMatter to express support for the movement to
end police brutality and racism against Black people. In general, hashtags
tend to align with the overall opinion of the tweet they are included in.
Therefore, we manually select a set of hashtags expressing support or
opposition to one of the candidates and categorize them as either pro-Biden or
pro-Trump. In cases where a hashtag opposes one candidate, we consider it as
supporting the other candidate (e.g., treating #traitortrump as pro-Biden). We
then annotate the tweets according to the polarities of the hashtags within
them. The hashtags used for tweet annotation are presented in Figure 1.

So far, we have acquired a sizable set of labeled tweets, which can help the
model overcome the challenge of informal language and comprehend the rich
context of the election. Instead of relying solely on text analysis to infer
the opinions expressed in these tweets, we propose leveraging the abundance of
behavioral data available on social media platforms. Specifically, we first
build a user-tweet bipartite graph, where users and tweets are represented as
nodes, and edges denote users’ posting and retweeting behaviors towards
tweets. Then, we conduct the task of political opinion mining as a GNN-based
node classification problem. We initialize tweet node features using a pre-
trained language model (e.g., BERT [14]). However, the features of user nodes
cannot be initialized in the same way due to the lack of textual information,
making it difficult to assign them appropriate initial features aligned with
tweet nodes. To address this issue, we introduce a novel skip aggregation
mechanism that leverages the bipartite nature of the graph. At each iteration,
every tweet node aggregates information from its second-order neighbors, which
comprise other tweets posted or retweeted by its author or retweeters and tend
to express similar opinions. User nodes only play a bridging role during
neighborhood sampling and do not require any characterization. In addition, we
incorporate edge type information into the model to make more effective use of
behavioral data, which leads to further improvements in performance.

We compare our model with several existing competitive text classification and
node classification models. The experimental results show that our model
significantly outperforms these models, achieving the best performance.
Further analyses demonstrate the significance of incorporating user behavioral
information in political opinion mining and the effectiveness of skip
aggregation in leveraging such information. Moreover, we provide evidence of
our model’s robustness in handling short texts through performance analysis
and showcase the high-quality tweet representations learned by our model
through embedding visualization. Our contributions can be summarized as
follows:

  * 1.

We collect a large amount of data from Twitter related to the 2020 US
presidential election, including tweets and associated metadata such as
authors and retweeters, and automatically annotate the tweets using manually
labeled hashtags.

  * 2.

We propose a GNN-based framework to detect political opinions in tweets. The
framework adopts a novel skip aggregation mechanism to effectively learn
meaningful tweet representations from a bipartite graph consisting of user-
tweet interactions.

  * 3.

We conduct extensive experiments and analyses to demonstrate the significance
of user behavioral information and the effectiveness of skip aggregation, both
of which contribute to significantly better performance of our model over
baselines.

##  2 Related Work

###  2.1 Political Opinion Mining

Political opinion mining is a Natural Language Processing (NLP) task that
involves analyzing political texts to identify the author’s stance or opinion
on a certain political event or issue. This is an important area of research
that can provide insights into public opinion and help inform policy
decisions. Researchers have conducted numerous studies in this area, including
but not limited to determining the political leanings of tweets [6, 11, 12,
13], inferring the political alignments of Twitter users [3, 4, 5], detecting
the amount of polarisation in the electorate [15], and predicting voting
intentions or election results [16, 2]. Maynard and Funk [6] employed advanced
NLP techniques and incorporated extra-linguistic contextual information to
extract more meaningful and higher quality opinions from a collection of pre-
election tweets. Rather than relying on traditional bag-of-words methods that
overlook the syntactic structure, Iyyer et al. [11] applied Recursive Neural
Network (RNN) to accurately identify the political position evinced by a
sentence. Chen et al. [12] suggested first building an opinion-aware knowledge
graph by integrating extracted opinions and targeted entities into an existing
structured knowledge base, and then performing ideology inference by
propagating information across the graph. Xiao et al. [13] proposed to
quantify the political polarity of tweets by explicitly assigning polarity
scores to the entities and hashtags within them. Manickam et al. [3]
introduced a framework for jointly estimating the ideology of social media
users and news websites, and tracing changes in user ideology over time. Xiao
et al. [4] presented a framework for ideology detection on Twitter, which
utilizes a multi-relational encoder and a multi-task decoder to assess the
significance of each relation. Fagni and Cresci [5] introduced an unsupervised
method for extracting fine-grained political leanings from social media posts.
The method projects users into a low-dimensional ideology space, where they
are clustered and their political leanings are automatically derived from the
assigned cluster. Conover et al. [15] employed a two-phase approach to tackle
the problem of predicting the political alignments of users. Lampos et al.
[16] analyzed tweets from the UK and Austria and demonstrated successful
prediction of voting intention in over 300 polls across both countries. Zhou
et al. [2] proposed an opinion tracking method that utilizes machine learning
models and social network big data analysis to overcome the limitations of
traditional polls and achieve accurate results in the 2019 Argentina
elections.

In addition to analyzing social media data, researchers have also investigated
methods for detecting political perspectives in news media. Li and Goldwasser
[7] suggested that the political perspectives expressed in news articles can
be inferred from the way the documents are disseminated and the
characteristics of the users who endorse them. They then applied GCN to
capture the social information embedded in the spread of news articles. Li and
Goldwasser [8] introduced a framework for pre-training models that leverages
the rich social and linguistic context readily available in data sources. Li
and Goldwasser [17] proposed an entity-centric framework that incorporates
entity and relation representations learned from external knowledge sources
and text corpus and utilizes word- and sentence-level attention mechanisms to
measure the importance of different aspects of the article for prediction.
Feng et al. [9] framed the task as a graph classification problem. They
constructed heterogeneous information networks to jointly model news content
and external knowledge and adopted a relational GNN to learn graph
representations. Zhang et al. [10] proposed a novel approach that employs
textual cues as paragraph-level labels and integrates multi-hop knowledge
reasoning for inference.

###  2.2 Graph Neural Networks

GNNs have emerged as a powerful tool for modeling complex and structured data
that can be represented as graphs. Graph Convolutional Network (GCN) [18]
utilizes the Laplacian matrix to represent the structure of the graph and
capture spatial features between the nodes. GCN can be considered as an
approximation of spectral domain convolution of graph signals. Graph Attention
Network (GAT) [19] combines the power of GCN with the attention mechanism,
which allows nodes to selectively attend to their most informative neighbors.
The graph convolutional operation used in models such as GraphSAGE [20] and
FastGCN [21] can be viewed as a process of sampling and aggregating of
neighborhood information, which is computationally efficient and scalable to
large graphs. Graph Isomorphism Network (GIN) [22] was developed to support
more complex forms of aggregation and has been proven to be theoretically the
most powerful GNN under the neighborhood aggregation scheme. GIN’s learnable
aggregation function allows it to capture more fine-grained information about
the graph structure and learn more expressive representations of the nodes in
the graph. Although GCN frameworks have shown great success in modeling
single-relation graphs, they are limited to single-relation graphs and ignore
the various types of relations that exist in realistic scenarios. Relational
Graph Convolutional Network (R-GCN) [23] extends GCN to handle multi-
relational graphs by introducing relation-specific transformations that
consider the type and direction of each edge. R-GCN have achieved state-of-
the-art results on a number of node classification and link prediction tasks
for multi-relational graphs. The success of R-GCN highlights the importance of
considering the different types of relations in complex graphs.

In recent years, GNNs have also been successfully applied in various NLP tasks
including text classification, sentiment analysis, machine translation, and
question answering. One of the key benefits of using GNNs in NLP is the
ability to capture the structural information in text data, such as syntax and
semantics, which can be represented as graphs. Yao et al. [24] proposed
TextGCN, which is a pioneering method that employs GCN for text
classification. TextGCN involves creating a heterogeneous graph consist of
both word nodes and document nodes and converting the task into a node
classification problem. The results of TextGCN showed significant improvements
over a variety of state-of-the-art methods and attracted much attention in the
NLP community. Chen et al. [25] introduced a GNN-based model for the
conversational Machine Reading Comprehension (MRC) task. The model utilizes an
attention mechanism to dynamically build a passage graph that incorporates
both the question and the conversation history. Each passage word is
represented as a node in the graph at each turn of the conversation. Xu et al.
[26] proposed a novel hybrid graph model for document-level Neural Machine
Translation (NMT) that considers both intra- and inter-sentential relations,
which addresses the severe long-dependency issue that arises in document-level
NMT by allowing the model to capture dependencies not only within sentences
but also across sentence boundaries. Hu et al. [27] developed an end-to-end
graph neural model for fake news detection, which compares the news to the
knowledge base through entities and incorporates topics to enrich the news
representation. Peng et al. [28] applied GNNs to the task of document-level
Relation Extraction (RE). They extract a subgraph from the document graph by
tracing reasoning paths between the target entity pair and use R-GCN to
capture the relations.

##  3 Methodology

###  3.1 Data Preparation, Annotation, and Preprocessing

In this study, we focus on the 2020 United States presidential election held
on November 3, 2020. We crawl tweets posted between October 1, 2020, and
November 2, 2020, using the Twitter Streaming API, filtering by the following
query: biden $OR$ trump, which corresponds to the two main candidates from the
Democratic Party (Joe Biden) and the Republican Party (Donald Trump). In
total, we gather 138.9 million tweets in English, along with the associated
metadata for each tweet.

We leverage the fact that users frequently include hashtags in their tweets,
which tend to reflect the opinion expressed in the entire tweet, using
hashtags to annotate the tweets we previously collected. In particular, we
first identify the most commonly used hashtags and manually categorize those
explicitly expressing support or opposition to one of the candidates as either
pro-Biden or pro-Trump. For hashtags that oppose one candidate, we treat them
as supporting the other candidate (e.g., #traitortrump is considered pro-
Biden). Next, we expand the hashtag set by discovering new hashtags that are
significantly related to the initial set, based on co-occurrence. As a result,
the expanded hashtag set contains 245 labeled hashtags. We select a subset of
these hashtags based on their frequency of occurrence and present them in
Figure 1. Then, we filter tweets that contain at least one hashtag from the
hashtag set. In cases where a tweet contains multiple hashtags, we keep it
only if all the hashtags support the same candidate. Finally, we annotate each
tweet according to the polarity of the hashtags within it.

To avoid duplicates, we remove retweets (tweets that start with “RT
@username”) and record the retweet information as metadata in the
corresponding original tweets. Additionally, we remove the hashtags used for
annotation from tweets to avoid target leakage, while retaining other hashtags
as they may contain crucial information about the opinion. The corpus
ultimately consists of $1123749$ labeled tweets, which is sufficient to train
an effective neural classifier.

Figure 2:  An example of the user-tweet bipartite graph. The links represent
posting and retweeting behaviors.

###  3.2 User-Tweet Bipartite Graph Construction

In addition to the content of a tweet, its social context also helps to
determine its opinion. Therefore, we build a bipartite graph to model user-
tweet interactions, with each edge representing a user’s behavior towards a
tweet (as illustrated in Figure 2). We consider both posting and retweeting
behaviors because users generally retweet messages they ideologically agree
with (retweeting is sharing a tweet without adding any comment). The resulting
graph contains $1123749$ tweet nodes, $700507$ user nodes, $1123749$ post
links, and $1473818$ retweet links.

Recently, GNNs have been widely used to process such graph-structure data and
have shown remarkable performance, so we intend to apply a GNN to learn the
representations of tweets from the graph to further determine their opinions.
To this end, we first need to assign initial features to the nodes in the
graph. Tweet node features can be readily initialized using a pre-trained
language model. However, user node features cannot be initialized in the same
way due to the lack of textual information. Thus, it is difficult to assign
appropriate initial features aligned with tweet nodes to user nodes.

In fact, in a bipartite graph, each node’s first-order neighbors are of a
different type than itself, while its second-order neighbors are of the same
type as itself. In other words, the second-order neighbors of a tweet node in
the user-tweet bipartite graph are also tweet nodes. Furthermore, these tweets
tend to share the same opinions as the original tweet. As shown in Figure 2,
using users A and B as bridges, tweet A’s second-order neighbors are tweets B,
C, and D, which express similar opinions as it. To leverage this insight, we
introduce a novel skip aggregation mechanism that enables each tweet node to
aggregate information from its second-order neighbors. User nodes only serve
as bridges during neighborhood sampling, bypassing the initialization problem.

###  3.3 Skip Aggregation Graph Convolution Layer

Most modern GNNs follow a two-phase scheme involving aggregation and
combination to update node features in a graph [22]. Formally, the $l$-th
layer of a GNN can be defined as:

$\displaystyle
a_{v}^{(l)}=\operatorname{AGGREGATE}^{(l)}\left(\left\\{h_{u}^{(l-1)}:u\in\mathcal{N}(v)\right\\}\right)$
(1) $\displaystyle
h_{v}^{(l)}=\operatorname{COMBINE}^{(l)}\left(h_{v}^{(l-1)},a_{v}^{(l)}\right)$

where $a_{v}^{(l)}$ is the aggregated node feature of node $v$’s neighborhood,
$\mathcal{N}(v)$ is a set of neighbor nodes of node $v$, and $h_{v}^{(l)}$ is
the feature of node $v$ at the $l$-th iteration. $h_{v}^{(0)}=x_{v}$, where
$x_{v}$ is the initial feature of node $v$.

Figure 3:  An overview of our proposed framework. A user-tweet bipartite graph
is constructed and fed into SA-GNN to infer the opinions of tweets. We
illustrate how the SA layer works with an example of a target node aggregating
information from two sampled second-order neighbors. The output of the SA
layer is a new representation of the target node that combines information
from both the target node and its second-order neighbors in the graph.

The choice of $\mathcal{N}(\cdot)$, $\operatorname{AGGREGATE}^{(l)}(\cdot)$,
and $\operatorname{COMBINE}^{(l)}(\cdot)$ is critical. To achieve efficient
training on the large-scale graph, we use a random walk-based strategy for
neighborhood sampling. Specifically, for each tweet node in the graph, we
simulate $n$ random walks of length 2, recording the visited tweet nodes and
their corresponding visit times. The neighborhood of a tweet node is then
defined as the top $k$ most visited tweet nodes in the random walks starting
at it.

By doing so, we can obtain an informative and representative set of neighbor
nodes for each tweet node. The tweet node $v$ and its $k$ neighbor nodes form
$k$ unique center-neighbor pairs $\\{(v,u):u\in\mathcal{N}(v)\\}$. Unlike
typical scenarios where only a single edge connects each pair, in this case,
each pair is connected by a user node and two edges: one edge $e_{v,u}^{v}$
links the center node $v$ with the user node, and the other edge $e_{v,u}^{u}$
links the neighbor node $u$ with the user node. Moreover, due to the
heterogeneity of the user-tweet bipartite graph, different center-neighbor
pairs can have different types of $e_{v,u}^{v}$ and $e_{v,u}^{u}$.

To effectively leverage the information from these two edges, our proposed
Skip Aggregation (SA) graph convolution layer takes a different approach than
typically aggregating information from neighbor nodes and combining it with
the center node’s original feature. Instead, the SA layer aggregates features
for center-neighbor pairs, considering the types of the two edges between each
pair. Formally,

$\displaystyle
a_{v}^{(l)}=\operatorname{AGGREGATE}^{(l)}\big{(}\big{\\{}\sigma\big{(}\big{[}\textbf{W}_{cen,\phi(e_{v,u}^{v})}^{(l)}\cdot
h_{v}^{(l-1)}\|\textbf{W}_{nei,\phi(e_{v,u}^{u})}^{(l)}\cdot
h_{u}^{(l-1)}\big{]}\big{)}:u\in\mathcal{N}(v)\big{\\}}\big{)}$ (2)
$\displaystyle h_{v}^{(l)}=\sigma\big{(}\textbf{W}_{c}^{(l)}\cdot
a_{v}^{(l)}\big{)}$

Algorithm 1 SA-GNN embedding generation (i.e., forward propagation) algorithm

Input: Bipartite graph $\mathcal{G}(\mathcal{U},\mathcal{V},\mathcal{E})$;
input features $\left\\{\textbf{x}_{v}:v\in\mathcal{V}\right\\}$; depth $L$

Output: Vector representations $\textbf{z}_{v}$ for all $v\in\mathcal{V}$

1: $h_{v}^{(0)}\leftarrow\textbf{x}_{v},\forall v\in\mathcal{V}$;

2: for $l=1\ldots L$ do

3: for $v\in\mathcal{V}$ do

4: for $u\in\mathcal{N}(v)$ do

5:
$a_{v,u}^{(l)}\leftarrow\sigma\big{(}\big{[}\textbf{W}_{cen,\phi(e_{v,u}^{v})}^{(l)}\cdot
h_{v}^{(l-1)}\|\textbf{W}_{nei,\phi(e_{v,u}^{u})}^{(l)}\cdot
h_{u}^{(l-1)}\big{]}\big{)}$;

6: end for

7:
$a_{v}^{(l)}\leftarrow\operatorname{AGGREGATE}^{(l)}\big{(}\big{\\{}a_{v,u}^{(l)}:u\in\mathcal{N}(v)\big{\\}}\big{)}$

8: $h_{v}^{(l)}\leftarrow\sigma\big{(}\textbf{W}_{c}^{(l)}\cdot
a_{v}^{(l)}\big{)}$;

9: end for

10: $h_{v}^{(l)}\leftarrow h_{v}^{(l)}/\|h_{v}^{(l)}\|_{2},\forall
v\in\mathcal{V}$;

11: end for

12: $\textbf{z}_{v}\leftarrow h_{v}^{(L)},\forall v\in\mathcal{V}$;

where $\|$ represents concatenation, $\sigma(\cdot)$ refers to a nonlinear
activation function, and $\phi(\cdot)$ is an edge type mapping function.
$\textbf{W}_{cen,\phi(e_{v,u}^{v})}^{(l)},\textbf{W}_{nei,\phi(e_{v,u}^{u})}^{(l)}\in\mathbb{R}^{d\times
d}$ are trainable parameters that transform the features of the center node
$v$ and the neighbor node $u$ according to the types of $e_{v,u}^{v}$ and
$e_{v,u}^{u}$, respectively. $\textbf{W}_{c}^{(l)}\in\mathbb{R}^{2d\times d}$
combines the concatenated features for use in the next layer. The
$\operatorname{AGGREGATE}$ operator denotes a versatile aggregator of center-
neighbor pair representations that offers several aggregation options,
including mean, max, sum, and weighted sum [29].

We refer to the GNN model formed by stacking multiple SA layers as SA-GNN.
Algorithm 1 describes the embedding generation process of $L$-Layer SA-GNN
when the bipartite graph $\mathcal{G}=(\mathcal{U},\mathcal{V},\mathcal{E})$,
where $\mathcal{U}$ and $\mathcal{V}$ are user node and tweet node sets,
respectively, and the initial features $\textbf{x}_{v}$ for tweet nodes are
provided as input. The normalization in Line 10 makes training more stable.

###  3.4 Model Training

Figure 3 shows an overview of our proposed framework. We first construct a
user-tweet bipartite graph based on users’ posting and retweeting behaviors
and initialize the features of tweet nodes using BERT. Then, we sample
neighborhood and apply SA-GNN to generate behavior-aware representations for
each tweet node. A fully connected layer and a sigmoid function are finally
employed to compute the predicted label probabilities.

$\hat{y}_{v}=\operatorname{Sigmoid}\left(\textbf{W}_{o}\cdot\textbf{z}_{v}\right)$
(3)

where $\textbf{z}_{v}$ is the tweet node $v$’s new representation generated by
SA-GNN.

We use binary cross-entropy loss to train our model.

$\mathcal{L}=-\sum_{v\in\mathcal{V}}y_{v}\log\hat{y_{v}}+\left(1-y_{v}\right)\log\left(1-\hat{y_{v}}\right)$
(4)

where $\mathcal{V}$ represents the set of tweet nodes in the graph.

##  4 Experiments

###  4.1 Experimental Settings

####  4.1.1 Baselines

To verify the significance of user behavioral information for detecting
political opinions and the effectiveness of skip aggregation in leveraging
such information, we compare our model with the following baseline models:

  * 1.

BERT [14] is a powerful language representation model based on the Transformer
architecture. It was pre-trained on a large unlabeled corpus using two self-
supervised tasks: Masked Language Modeling (MLM) and Next Sentence Prediction
(NSP) and can be fine-tuned for various downstream tasks.

  * 2.

GCN [18] is a type of neural network designed to perform semi-supervised
learning on graph-structured data. It utilizes the symmetrically normalized
graph Laplacian to generate node embeddings, which can be viewed as
aggregating information from neighbor nodes.

  * 3.

GAT [19] improves the neighborhood aggregation scheme of GCN by incorporating
a self-attention mechanism to assign weights to each node in the neighborhood
based on its importance to the target node.

  * 4.

GraphSAGE [20] is a framework for inductive representation learning on large
graphs that can predict the embeddings of new nodes without the need for re-
training the entire model.

  * 5.

GIN [22] adopts a message passing scheme to update node representations in a
way that incorporates information about the entire graph structure, which
allows it to capture important structural information that other GNNs may
miss.

  * 6.

R-GCN [23] is an extension of GCN that operates on multi-relational graphs,
assigning different transformations to edges of different types and
directions.

####  4.1.2 Implementation Details

For BERT baseline, we fine-tune the pre-trained BERTbase model for 5 epochs
with a learning rate of 5e-5. The BERTbase model has 12 layers and 768 hidden
units for each layer.

For graph-based models, we use a pre-trained BERTbase model to encode the
content of tweets as the initial features of the corresponding tweet nodes in
the graph. Subsequently, we initialize the user node features for
GCN/GAT/GraphSAGE/GIN/R-GCN in three different ways, one random and two
heuristic:

  * 1.

Random Initialization: Randomly initialize user node features by sampling from
a normal distribution.

  * 2.

Centroid-based Initialization: Inspired by the cluster representation [30], we
regard the neighborhood of a user node as a cluster, and take the centroid of
the cluster as the user node’s initial feature. In other words, each user
node’s feature is initialized with the average of the features of tweet nodes
in its neighborhood.

  * 3.

Medoid-based Initialization: Initialize the user node’s feature with the
medoid of the cluster, which is the feature of the tweet node with the
smallest sum of squared distances in terms of features from other tweet nodes
in the neighborhood.

Due to the massive graph, it is impossible to fit the features of all nodes
into the GPU. Therefore, we perform stochastic mini-batch training by virtue
of the neighborhood sampling technique. For graph-based baselines, we randomly
select 10 neighbors per layer. For our own model, we sample neighbors using
the random walk-based strategy outlined in Section 3.3, simulating 20 random
walks and selecting the top 10 most visited tweet nodes per layer.

Table 1: The performance of different models. We conduct five trials with
different random seeds and report the mean and standard deviation of the
results on the test set. SA-GNNw/o edge type information is a variant of our
model that does not utilize edge type information. Bold denotes the best
result, and underline denotes the second-best result.

Model |  | Accuracy | F1 score | AUC
---|---|---|---|---
BERT baseline | BERT | 90.70 $\pm$ 0.04 | 86.67 $\pm$ 0.07 | 89.43 $\pm$ 0.08
Graph-based baselines | Random Initialization | GCN | 89.08 $\pm$ 0.05 | 84.26 $\pm$ 0.07 | 87.54 $\pm$ 0.05
GAT | 91.41 $\pm$ 0.04 | 87.69 $\pm$ 0.08 | 90.21 $\pm$ 0.10
GraphSAGE | 91.42 $\pm$ 0.05 | 87.80 $\pm$ 0.07 | 90.38 $\pm$ 0.05
GIN | 90.12 $\pm$ 0.07 | 85.78 $\pm$ 0.11 | 88.70 $\pm$ 0.10
| R-GCN | 91.06 $\pm$ 0.05 | 87.26 $\pm$ 0.08 | 89.95 $\pm$ 0.06
Centroid-based Initialization | GCN | 91.32 $\pm$ 0.04 | 87.48 $\pm$ 0.09 | 89.95 $\pm$ 0.10
GAT | 91.71 $\pm$ 0.03 | 88.09 $\pm$ 0.05 | 90.47 $\pm$ 0.05
GraphSAGE | 91.61 $\pm$ 0.06 | 87.94 $\pm$ 0.10 | 90.35 $\pm$ 0.10
GIN | 91.40 $\pm$ 0.07 | 87.56 $\pm$ 0.10 | 89.98 $\pm$ 0.09
| R-GCN | 91.63 $\pm$ 0.07 | 87.95 $\pm$ 0.09 | 90.34 $\pm$ 0.07
Medoid-based Initialization | GCN | 91.36 $\pm$ 0.03 | 87.51 $\pm$ 0.04 | 89.96 $\pm$ 0.04
GAT | 91.93 $\pm$ 0.05 | 88.42 $\pm$ 0.08 | 90.73 $\pm$ 0.07
| GraphSAGE | 91.75 $\pm$ 0.05 | 88.14 $\pm$ 0.07 | 90.49 $\pm$ 0.06
| GIN | 91.54 $\pm$ 0.07 | 87.78 $\pm$ 0.10 | 90.17 $\pm$ 0.08
|  | R-GCN | 91.77 $\pm$ 0.03 | 88.16 $\pm$ 0.06 | 90.51 $\pm$ 0.07
Ours | SA-GNN | 92.73 $\pm$ 0.04 | 89.62 $\pm$ 0.06 | 91.73 $\pm$ 0.05
SA-GNNw/o edge type information | 92.35 $\pm$ 0.03 | 89.03 $\pm$ 0.05 | 91.20 $\pm$ 0.06

All graph-based models have a hidden size of 768 and are trained for 5 epochs
with a learning rate of 1e-3. The AdamW optimizer and a linear learning rate
scheduler with 6% warm-up are adopted to train all models.

We consider Accuracy, F1 score, and AUC as evaluation metrics to
quantitatively evaluate model performance. We train the model on the training
set and obtain the best model on the validation set, using 80%, 10%, and 10%
instances for training, validation, and testing, respectively. To ensure the
reliability of our results, we repeat the experiment five times and report the
average performance.

Table 2: The performance comparison when using different aggregators to aggregate center-neighbor pair representations. Model | Accuracy | F1 score | AUC
---|---|---|---
SA-GNN$\operatorname{mean}$ | 92.32 $\pm$ 0.06 | 89.02 $\pm$ 0.08 | 91.25 $\pm$ 0.05
SA-GNN$\max$ | 92.73 $\pm$ 0.04 | 89.62 $\pm$ 0.06 | 91.73 $\pm$ 0.05
SA-GNN$\operatorname{sum}$ | 92.30 $\pm$ 0.05 | 88.97 $\pm$ 0.07 | 91.18 $\pm$ 0.06
SA-GNN$\operatorname{weighted\ sum}$ | 92.16 $\pm$ 0.04 | 88.78 $\pm$ 0.05 | 91.05 $\pm$ 0.03

###  4.2 Main Results

As shown in Table 1, our proposed model SA-GNN demonstrates remarkable
performance on the test set. Specifically, SA-GNN achieves 92.73% Accuracy,
89.62% F1 score, and 91.73% AUC, consistently outperforming all baseline
models. Notably, SA-GNN outperforms BERT, which is fine-tuned for text
classification without introducing any additional information, by 2.03%,
2.95%, and 2.30% in terms of Accuracy, F1 score, and AUC, respectively.
Moreover, compared with graph-based baselines that incorporate user behavioral
information but encounter the user node initialization problem, SA-GNN yields
better results across all three random and heuristic initialization
strategies. We also consider a variant of our model that does not utilize edge
type information, simply aggregating information from second-order neighbors
without considering the two edges between each center-neighbor pair, similar
to PinSAGE [29]. The result shows that the ablated model SA-GNNw/o edge type
information achieves the second-best performance, but still has a significant
gap with the complete model in all aspects. Table 2 compares the performance
of our model when using different aggregators. Among the four aggregators we
considered, the $\max$ aggregator proves to be the most effective, further
highlighting the importance of selecting the right aggregator.

###  4.3 Model Analysis

####  4.3.1 Effect of User Behavioral Information

To evaluate the significance of user behavioral information for detecting
political opinions, we compare graph-based models that incorporate such
information with the BERT baseline. The results show that all graph-based
models outperform the BERT baseline with the help of user behavioral
information, with GCN exhibiting the smallest improvement. This may be due to
GCN’s inability to selectively aggregate information from neighbor nodes
through an attention mechanism or relation type discrimination, which
restricts its capacity to represent graph-structured data. In summary, our
observations confirm that incorporating user behavioral information is
advantageous for political opinion mining.

####  4.3.2 Effect of Skip Aggregation

To investigate the effectiveness of skip aggregation in leveraging user
behavioral information, we compare SA-GNN and its variant that both aggregate
information from second-order neighbors with graph-based baselines. Among the
three initialization strategies, medoid-based initialization is found to be
the most effective for graph-based baselines, with GAT achieving the highest
performance of 91.93% Accuracy, 88.42% F1 score, and 90.73% AUC. Nevertheless,
SA-GNN still demonstrates superior performance over GAT by 0.80%, 1.20%, and
1.00% in terms of Accuracy, F1 score, and AUC, respectively. Furthermore, SA-
GNN shows better performance than graph-based baselines even without the use
of edge type information. These findings suggest that skip aggregation can not
only bypass the user node initialization problem but also make more effective
use of user behavioral information.

####  4.3.3 Effect of Edge Type Information

To assess the importance of edge type information and our model’s capacity for
utilizing such information, we conduct a comparison between SA-GNN and its
variant, as well as R-GCN. As presented in Table 1, there is still a
significant gap between SA-GNN and its variant, which confirms our intuition
that the information contained in the two edges between the center-neighbor
pair is indeed valuable. Similar trends can also be observed when comparing
R-GCN with GCN, which further demonstrates that incorporating edge type
information can lead to certain improvements. Moreover, SA-GNN outperforms
R-GCN by 0.96, 1.46, and 1.22 points in terms of Accuracy, F1, and AUC,
respectively. These results indicate that SA-GNN is more effective in
utilizing edge type information and can thus make better use of user
behavioral information.

Figure 4: The results of performance analysis in terms of tweet length. Figure
5: The impact of layer number on model performance.

####  4.3.4 Performance on Short Texts

In this part, we aim to conduct a performance analysis to explore our model’s
performance on short texts. To accomplish this, we first examine the
distribution of tweet lengths in the dataset. The length of a tweet is defined
as the number of sub-word tokens split by the WordPiece tokenizer used in
BERT. Statistics show that 30% of the tweets in the dataset are no longer than
32 in length, which we consider to be short texts. Then, we compare the
performance of SA-GNN and BERT on short texts. As shown in Figure 4, BERT’s
performance degrades significantly when dealing with short texts, while SA-GNN
is only slightly affected. This can be attributed to the use of user
behavioral information in SA-GNN. In general, tweets do not follow strict
grammatical rules, and when they are short in length, they contain limited
linguistic features and lack context. As a result, it is difficult to
determine a tweet’s opinion solely based on its content, resulting in BERT’s
poor performance. However, SA-GNN overcomes this challenge by aggregating
information from other relevant tweets through graph convolution, which
enriches the representation of short texts and enables the classifier to
capture opinions based on sufficient information.

####  4.3.5 Impact of SA Layer Number

We investigate the impact of layer number on the performance of our proposed
model SA-GNN and the best-performing graph-based baseline GAT. The layer
number is varied from 1 to 6, and the accuracy curves of the two models on the
test set are presented in Figure 5. Overall, both models’ performance improves
as the layer number increases and gradually degrades after reaching a certain
depth. The 3-layer SA-GNN achieves the best performance, while GAT performs
optimally with 4 layers. Notably, the performance of 1-layer GAT is
significantly worse than stacking multiple layers. This observation can be
explained by the fact that 1-layer GAT only aggregates information from first-
order neighbors, which are all user nodes that contain limited information,
resulting in even worse performance than the BERT baseline. Additionally,
since SA-GNN aggregates information from second-order neighbors, GAT requires
stacking twice as many layers as SA-GNN to capture the same amount of relevant
tweet information. However, as the number of layers increases, the vanishing
gradient and over-smoothing problems gradually emerge, leading to a drop in
performance instead.

Figure 6: The t-SNE projections of the tweet representations learned by SA-GNN
and BERT. The blue and red dots respectively indicate pro-Biden and pro-Trump
tweets.

###  4.4 Embedding Visualization

To provide a more intuitive verification of our model’s effectiveness, we
randomly select 1% of tweets from the test set in a stratified fashion and
visualize the t-SNE projections of their representations learned by SA-GNN and
BERT in Figure 6. It can be clearly seen that tweets with the same opinion
tend to be more tightly clustered and the boundary between tweets with
different opinions is more distinct in the latent space of SA-GNN. This can be
attributed to the graph structure used in SA-GNN, which captures the inter-
dependencies between different tweets and allows the model to utilize the
context and opinions of relevant tweets to generate more distinguishable and
higher quality tweet representations than BERT.

###  4.5 Misclassification Analysis

In this section, we investigate and compare the misclassifications made by SA-
GNN and BERT on the test set. Specifically, we isolate the samples that are
misclassified and analyze the output of the logits layer, which is typically
the neural network’s final layer for classification tasks and produces raw
prediction values.

Figure 7:  The distributions of logit values output by SA-GNN and BERT for
misclassified samples.

As illustrated in Figure 7, the logit values produced by SA-GNN for
misclassified samples are mostly around 0, forming a probability distribution
similar to the normal distribution. This may be due to the opinions in most of
the tweets misclassified by SA-GNN are implicit or close to neutral, causing
the model to make slightly incorrect judgments. In contrast, BERT’s logit
values for misclassified samples show a bimodal distribution with two peaks at
both ends, indicating high confidence on these samples. This could be
attributed to BERT determining opinions solely based on content, overfitting
to certain common words. Thus, BERT may misinterpret implicit opposing
opinions as support or fail to grasp nuances like sarcasm and irony, which are
heavily context dependent. Therefore, in addition to superior performance, SA-
GNN is less overconfident and more robust than BERT [31].

(a)

(b)

Figure 8: The results of performance analysis in terms of tweets’ (a) first-
order neighbor number and (b) second-order neighbor number in the graph.

###  4.6 Limitations

Our proposed model employs GNNs to learn behavior-aware tweet representations
from the user-tweet bipartite graph, resulting in significant improvements.
However, we acknowledge that the model has some limitations. For example, when
dealing with tweets that are not popular or whose authors are not active, the
model can only aggregate information from a limited number of relevant tweets
during the graph convolution phase. As a result, the performance of the model
may be degraded. Similarly, for new tweets posted after data collection, their
authors or retweeters may not be included in the earlier constructed bipartite
graph, in which case the model cannot make inferences properly.

To confirm our intuition, we conduct a performance analysis. Specifically, we
compare the performance of SA-GNN and BERT for tweets with different numbers
of first-order and second-order neighbors in the graph. As shown in Figures 8a
and 8b, the performance of BERT, which is unaffected by any graph-related
factors, increases with tweets’ first-order neighbor number and second-order
neighbor number. This could be because tweets with more neighbors in the graph
have more explicit opinions or more formal language, leading the model to be
more accurate about them. The performance of SA-GNN also increases with
tweets’ first-order neighbor number and exhibits the same trend as BERT,
indicating that SA-GNN’s performance is unaffected by the number of first-
order neighbors, as expected because SA-GNN aggregates information from
second-order neighbors.

However, the number of second-order neighbors has an impact on SA-GNN’s
performance. From Figure 8b, we can observe that the performance of SA-GNN
degrades dramatically for tweets with no more than 5 second-order neighbors,
even far below that of BERT, which is a common problem of GNNs that are biased
against low-degree nodes [32]. In fact, although BERT’s performance is
inferior to graph-based models as it only utilizes the linguistic information
of the tweet, it is more flexible and efficient. Hence, we intend to address
the limitation in the future by incorporating user behavioral information into
BERT, which could be accomplished, for example, by designing new pre-training
objectives or contrastive learning, resulting in a more flexible and efficient
political opinion mining model while maintaining high performance.

##  5 Conclusion

In this paper, we create a large-scale dataset from Twitter related to the
2020 US presidential election and propose a GNN-based framework that leverages
user behavioral information to improve the accuracy of political opinion
mining. The framework adopts a novel skip aggregation mechanism to learn
behavior-aware tweet representations from a user-tweet bipartite graph, which
is shown to be effective in capturing the complex and nuanced nature of
political opinions on social media. With further analyses and visualization,
we demonstrate the robustness of the framework and showcase the high quality
of tweet representations it generates. Finally, we discuss the limitations of
this work and suggest new directions for future research.

## References

  * [1] A. Bovet, F. Morone, H. A. Makse, Validation of twitter opinion trends with national polling aggregates: Hillary clinton vs donald trump, Scientific reports 8 (1) (2018) 1–16.
  * [2] Z. Zhou, M. Serafino, L. Cohan, G. Caldarelli, H. A. Makse, Why polls fail to predict elections, Journal of Big Data 8 (1) (2021) 1–28.
  * [3] I. Manickam, A. S. Lan, G. Dasarathy, R. G. Baraniuk, Ideotrace: a framework for ideology tracing with a case study on the 2016 U.S. presidential election, in: ASONAM ’19: International Conference on Advances in Social Networks Analysis and Mining, Vancouver, British Columbia, Canada, 27-30 August, 2019, ACM, 2019, pp. 274–281.  doi:10.1145/3341161.3342887.
  * [4] Z. Xiao, W. Song, H. Xu, Z. Ren, Y. Sun, TIMME: twitter ideology-detection via multi-task multi-relational embedding, in: KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, ACM, 2020, pp. 2258–2268.
  * [5] T. Fagni, S. Cresci, Fine-grained prediction of political leaning on social media with unsupervised deep learning, Journal of Artificial Intelligence Research 73 (2022) 633–672.
  * [6] D. Maynard, A. Funk, Automatic detection of political opinions in tweets, in: Extended semantic web conference, Springer, 2011, pp. 88–99.
  * [7] C. Li, D. Goldwasser, Encoding social information with graph convolutional networks forPolitical perspective detection in news media, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Florence, Italy, 2019, pp. 2594–2604.  doi:10.18653/v1/P19-1247.
  * [8] C. Li, D. Goldwasser, Using social and linguistic information to adapt pretrained representations for political perspective identification, in: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, Association for Computational Linguistics, Online, 2021, pp. 4569–4579.  doi:10.18653/v1/2021.findings-acl.401.
  * [9] S. Feng, M. Luo, Z. Chen, Q. Li, X. Chang, Q. Zheng, Knowledge graph augmented political perspective detection in news media, ArXiv preprint abs/2108.03861 (2021).
  * [10] W. Zhang, S. Feng, Z. Chen, Z. Lei, J. Li, M. Luo, KCD: Knowledge walks and textual cues enhanced political perspective detection in news media, in: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, Seattle, United States, 2022, pp. 4129–4140.  doi:10.18653/v1/2022.naacl-main.304.
  * [11] M. Iyyer, P. Enns, J. Boyd-Graber, P. Resnik, Political ideology detection using recursive neural networks, in: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Baltimore, Maryland, 2014, pp. 1113–1122.  doi:10.3115/v1/P14-1105.
  * [12] W. Chen, X. Zhang, T. Wang, B. Yang, Y. Li, Opinion-aware knowledge graph for political ideology detection, in: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, ijcai.org, 2017, pp. 3647–3653.  doi:10.24963/ijcai.2017/510.
  * [13] Z. Xiao, J. Zhu, Y. Wang, P. Zhou, W. H. Lam, M. A. Porter, Y. Sun, Detecting political biases of named entities and hashtags on twitter, ArXiv preprint abs/2209.08110 (2022).
  * [14] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp. 4171–4186.  doi:10.18653/v1/N19-1423.
  * [15] M. Conover, J. Ratkiewicz, M. Francisco, B. Gonçalves, F. Menczer, A. Flammini, Political polarization on twitter, in: Proceedings of the international aaai conference on web and social media, Vol. 5, 2011, pp. 89–96.
  * [16] V. Lampos, D. Preoţiuc-Pietro, T. Cohn, A user-centric model of voting intention from social media, in: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Sofia, Bulgaria, 2013, pp. 993–1003.
  * [17] C. Li, D. Goldwasser, MEAN: Multi-head entity aware attention networkfor political perspective detection in news media, in: Proceedings of the Fourth Workshop on NLP for Internet Freedom: Censorship, Disinformation, and Propaganda, Association for Computational Linguistics, Online, 2021, pp. 66–75.  doi:10.18653/v1/2021.nlp4if-1.10.
  * [18] T. N. Kipf, M. Welling, Semi-supervised classification with graph convolutional networks, in: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, OpenReview.net, 2017.
  * [19] P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Liò, Y. Bengio, Graph attention networks, in: 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings, OpenReview.net, 2018.
  * [20] W. L. Hamilton, Z. Ying, J. Leskovec, Inductive representation learning on large graphs, in: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, 2017, pp. 1024–1034.
  * [21] J. Chen, T. Ma, C. Xiao, Fastgcn: Fast learning with graph convolutional networks via importance sampling, in: 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings, OpenReview.net, 2018.
  * [22] K. Xu, W. Hu, J. Leskovec, S. Jegelka, How powerful are graph neural networks?, in: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net, 2019.
  * [23] M. Schlichtkrull, T. N. Kipf, P. Bloem, R. v. d. Berg, I. Titov, M. Welling, Modeling relational data with graph convolutional networks, in: European semantic web conference, Springer, 2018, pp. 593–607.
  * [24] L. Yao, C. Mao, Y. Luo, Graph convolutional networks for text classification, in: The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, AAAI Press, pp. 7370–7377.  doi:10.1609/aaai.v33i01.33017370.
  * [25] Y. Chen, L. Wu, M. J. Zaki, Graphflow: Exploiting conversation flow with graph neural networks for conversational machine comprehension, in: Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, ijcai.org, 2020, pp. 1230–1236.  doi:10.24963/ijcai.2020/171.
  * [26] M. Xu, L. Li, D. F. Wong, Q. Liu, L. S. Chao, Document graph for neural machine translation, in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 2021, pp. 8435–8448.  doi:10.18653/v1/2021.emnlp-main.663.
  * [27] L. Hu, T. Yang, L. Zhang, W. Zhong, D. Tang, C. Shi, N. Duan, M. Zhou, Compare to the knowledge: Graph neural fake news detection with external knowledge, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021, pp. 754–763.
  * [28] X. Peng, C. Zhang, K. Xu, Document-level relation extraction via subgraph reasoning, in: Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, International Joint Conferences on Artificial Intelligence Organization, 2022, pp. 4331–4337.  doi:10.24963/ijcai.2022/601.
  * [29] R. Ying, R. He, K. Chen, P. Eksombatchai, W. L. Hamilton, J. Leskovec, Graph convolutional neural networks for web-scale recommender systems, in: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018, ACM, 2018, pp. 974–983.  doi:10.1145/3219819.3219890.
  * [30] A. Pal, C. Eksombatchai, Y. Zhou, B. Zhao, C. Rosenberg, J. Leskovec, Pinnersage: Multi-modal user embedding framework for recommendations at pinterest, in: KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, ACM, 2020, pp. 2311–2320.
  * [31] J. Grabinski, P. Gavrikov, J. Keuper, M. Keuper, Robust models are less over-confident, arXiv preprint arXiv:2210.05938 (2022).
  * [32] X. Tang, H. Yao, Y. Sun, Y. Wang, J. Tang, C. Aggarwal, P. Mitra, S. Wang, Investigating and mitigating degree-related biases in graph convoltuional networks, in: Proceedings of the 29th ACM International Conference on Information & Knowledge Management, 2020, pp. 1435–1444.