#  LMBot: Distilling Graph Knowledge into Language Model for Graph-less
Deployment in Twitter Bot Detection

Zijian Cai<EMAIL_ADDRESS>Xi’an Jiaotong
UniversityXi’anShaanxiChina ,  Zhaoxuan Tan<EMAIL_ADDRESS>Xi’an
Jiaotong UniversityXi’anShaanxiChina ,  Zhenyu Lei<EMAIL_ADDRESS>Xi’an Jiaotong UniversityXi’anShaanxiChina ,  Zifeng Zhu
<EMAIL_ADDRESS>Xi’an Jiaotong UniversityXi’anShaanxiChina ,  Hongrui
Wang<EMAIL_ADDRESS>Xi’an Jiaotong UniversityXi’anShaanxiChina ,
Qinghua Zheng<EMAIL_ADDRESS>Xi’an Jiaotong
UniversityXi’anShaanxiChina and  Minnan Luo<EMAIL_ADDRESS>Xi’an
Jiaotong UniversityXi’anShaanxiChina

(2018)

###### Abstract.

As malicious actors employ increasingly advanced and widespread bots to
disseminate misinformation and manipulate public opinion, the detection of
Twitter bots has become a crucial task. Though graph-based Twitter bot
detection methods achieve state-of-the-art performance, we find that their
inference depends on the neighbor users multi-hop away from the targets, and
fetching neighbors is time-consuming and may introduce bias. At the same time,
we find that after finetuning on Twitter bot detection, pretrained language
models achieve competitive performance and do not require a graph structure
during deployment. Inspired by this finding, we propose a novel bot detection
framework LMBot that distills the knowledge of graph neural networks (GNNs)
into language models (LMs) for graph-less deployment in Twitter bot detection
to combat the challenge of data dependency. Moreover, LMBot is compatible with
graph-based and graph-less datasets. Specifically, we first represent each
user as a textual sequence and feed them into the LM for domain adaptation.
For graph-based datasets, the output of LMs provides input features for the
GNN, enabling it to optimize for bot detection and distill knowledge back to
the LM in an iterative, mutually enhancing process. Armed with the LM, we can
perform graph-less inference, which resolves the graph data dependency and
sampling bias issues. For datasets without graph structure, we simply replace
the GNN with an MLP, which has also shown strong performance. Our experiments
demonstrate that LMBot achieves state-of-the-art performance on four Twitter
bot detection benchmarks. Extensive studies also show that LMBot is more
robust, versatile, and efficient compared to graph-based Twitter bot detection
methods.

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock,
NY††booktitle: Woodstock ’18: ACM Symposium on Neural Gaze Detection, June
03–05, 2018, Woodstock, NY††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06 Figure
1. A typical Twittersphere in TwiBot-20 dataset (left) and the number of
fetched users when doing inference for 20 users using language model and graph
neural networks with the increasing number of layers (right)

##  1\. Introduction

Twitter bots are automated accounts that perform various functionalities. They
are manipulated by malignant actors and pose significant risks, particularly
in terms of disseminating false information and facilitating fraudulent
activities. They have been continuously involving spreading misinformation
(Howard et al., 2017), manipulating opinion (Ferrara et al., 2016), and
invading privacy by inferring sensitive details from tweets (Volkova et al.,
2015). With the increasing prevalence and advancing techniques, their
influence could be greatly amplified if left unregulated. Therefore, it is of
great significance and urgency to develop effective Twitter bot detection
methods.

Researchers have devoted significant efforts to detecting Twitter bots, and
the existing methods can be categorized into three types: feature-based, text-
based, and graph-based. Early methods that focus on designing efficient
features are primarily feature-based and text-based methods. Feature-based
methods extract features from users’ metadata and tweets, which are then fed
into random forests for bot identification (Echeverrï£¡ a et al., 2018; Beskow
and Carley, 2018; Kouvela et al., 2020). However, bot manipulators began to be
aware of the selected features and bots can disguise themselves from feature-
based methods by tampering with bot metadata. Text-based methods apply NLP
techniques to encode users’ descriptions and tweet information for bot
detection (Hayawi et al., 2022; Wu et al., 2023). However, text-based methods
can be easily deceived when bots steal tweets posted by genuine users. Later,
researchers proposed graph-based methods that leverage the intrinsic graph
structure in Twittersphere and apply network science and geometric deep
learning techniques for bot detection (Feng et al., 2021a, 2022a). The state-
of-the-art performance and extensive studies on graph-based methods indicate
the necessity of graph structure for Twitter bot detection. Despite the
impressive accuracy achieved by graph-based methods, we argue that making
inferences with graph structure is time-consuming, that being said, for each
target user, we need to fetch its multi-hop neighbor’s information. Given
Twitter API’s slow response rate and strict rate limits, querying neighborhood
information is expensive and burdens the deployment of real-world Twitter bot
detection applications. As shown in Figure 1, in the TwiBot-20 dataset (Feng
et al., 2021c), the number of neighboring users explodes exponentially as the
number of GNN layers increases. Moreover, restricted by the Twitter API, we
are only able to obtain 20 neighboring users through random sampling, which
also introduces sampling bias. Hence, developing a Twitter bot detection
framework that can make inferences without graph structure is essential for
real-world applications.

With the goal of developing a Twitter bot detection framework that can perform
inference without graph structure, we first examine existing non-graph-based
methods and find that text-based methods perform the best (Feng et al.,
2022b). However, we observe that these methods mostly process user text
features using frozen pretrained language models, which greatly limits the
expressive power of language models and their adaptation to the Twitter bot
detection domain. To address this problem, we first finetune the language
models for Twitter bot detection and find that they can perform surprisingly
well. As demonstrated in Figure 2, simply finetuning language models can
achieve competitive performance that already outperforms the state-of-the-art
non-graph-based methods and is close to the state-of-the-art performance
achieved by graph-based methods. These results indicate that the ability of
language models for bot detection is largely underestimated, and LMs do not
require neighboring information for inference (Figure 1), making them a good
proxy for deploying graph-less Twitter bot detection.

However, merely finetuning language models is yet to be perfect, as there
remains a gap between the performance of vanilla finetuned language models and
state-of-the-art methods. Given the state-of-the-art performance of graph-
based methods, and previous studies showing that the graph structure in
Twittersphere is highly effective in tackling various challenges, such as bot
communities (Liu et al., 2023), and bot disguise (Feng et al., 2021a). As a
result, we are convinced that the graph structure in Twittersphere is
essential for Twitter bot detection. Nevertheless, the question of how to
incorporate graph information into language models for bot detection remains
underexplored.

Figure 2. The performance of finetuned language models on TwiBot-20 dataset.
Simply finetuning language models already outperforms the state-of-the-art
non-graph-based methods and is close to the state-of-the-art performance.

To efficiently incorporate graph information into language models for Twitter
bot detection, we propose LMBot, a novel framework that iteratively distills
knowledge from graph neural networks into language models. Specifically, we
first extract each user’s metadata, descriptions, and tweets into a textual
sequence for unified user representation. We then feed these text sequences
into the LM and conduct finetuning for domain adaption, allowing the LM to
acquire domain knowledge of the Twittersphere. Next, we iteratively distill
knowledge and inductive bias from GNN into LM. We first update GNN’s
parameters, where the output of LM is leveraged as the initial feature for
GNN, providing GNN with better input _w.r.t_ Twitter bot detection task. After
updating GNN’s parameters, we then update LM’s learnable parameters by using
GNN’s prediction logits to guide LM’s training through knowledge distillation,
therefore incorporating graph information into LM. LMBot performs these two
steps iteratively, allowing the GNN and LM to mutually enhance and iteratively
distill graph information into the language model. In the inference stage,
either LM or GNN can be used for Twitter bot detection. Armed with LM, we
don’t need to fetch neighboring information for deployment, therefore saving
lots of time waiting for API response and avoiding graph sampling bias. For
datasets without graph structure, we are inspired by the success of feature-
based methods that only leverage MLP (Hayawi et al., 2022) for bot
identification and simply replace GNN with MLP, and find LMBot still achieves
state-of-the-art performance.

Our experiments demonstrate that LMBot achieves state-of-the-art performance
on four widely-acknowledged Twitter bot detection benchmarks, outperforming
ten competitive Twitter bot detection baselines. In addition, we study
different combinations of LMs and GNNs, and find that the selection of LMs is
more robust, while GNNs is sensitive, where heterogeneous GNNs perform much
better than homogeneous ones. We also analyze the convergence process of the
LM and GNN, validate the mutually enhancing process between them, and confirm
the necessity of domain adaptation for the language model’s fast convergence
speed and stability. Moreover, We study the prediction consistency between LM
and GNN, confirming that LM can give similar predictions as GNN even without
graph structure in the inference stage, proving that LM can effectively learn
the inductive bias in GNN. Further studies also reveal the high data
efficiency of LMBot, that is, LMBot consistently achieves high performance and
exhibits lower volatility when the ground truth label and edge in the Twitter
network decrease.

Figure 3. Overview of our proposed framework LMBot. LMBot first represents
user as a textual sequence, which is the input of LM. LMBot iteratively
distills graph structure knowledge in GNN into LM, where LM provides GNN with
user embeddings and GNN provides LM with soft labels.

Our main contributions are summarized as follows:

  * •

We are the first to discover that language model-based Twitter bot detection
methods are significantly underestimated. Specifically, we finetune language
models for the Twitter bot detection task and find them to outperform all non-
graph-based methods and achieve performance close to the state-of-the-art
graph-based methods.

  * •

We propose LMBot, the first graph-less bot detection framework that
incorporates graph structure in LM by iteratively distilling graph information
extracted by GNN into LM. Armed with LM, neither neighboring information nor
graph structure is required during inference, resolving the challenge of data
dependency and sampling bias in graph-based Twitter bot detection deployment.

  * •

Our experiments show that LMBot achieves state-of-the-art on four widely used
Twitter bot benchmarks, even without graph structure. Extensive studies
further confirm the effectiveness, efficiency, and robustness of LMBot.

##  2\. Related Work

###  2.1. Twitter Bot Detection

Twitter bot detection refers to detecting and identifying bot accounts on the
Twitter platform. Bot accounts are usually automatically controlled by the
software and imitate genuine users to post tweets. Identifying these bot
accounts can preserve the integrity of online discourse, and combat
misinformation and conspiracy theories. Existing methods are mostly feature-
based, text-based, and graph-based (Feng et al., 2022b).

Feature-Based Methods. These methods extract features from the user’s metadata
and tweets and leverage a classifier for bot detection. Echeverrï£¡ a et al.
(2018) extract 10 features from users’ metadata and 19 feature from users’
tweets and adopts a random forest for classification. Beskow and Carley (2018)
extracts features from user attributes, network attributes, contents, and
timing information and exploits random forest as the classifier. Kouvela et
al. (2020) extract 36 features from each user’s metadata and the latest 20
tweets and classifies users with random forest. Kudugunta and Ferrara (2018)
extract features from users’ metadata and combines synthetic minority
oversampling (SMOTE) with undersampling techniques. Miller et al. (2014)
cluster human accounts using DBSCAN and K-MEANS to obtain human clusters in
training. In testing, users not belonging to any human clusters are considered
bots. Abreu et al. (2020) choose five essential Twitter user features and
classifies users with four machine learning algorithms.

Text-Based Methods. These methods leverage NLP techniques to encode the users’
textual information for bot detection. Hayawi et al. (2022) encode users’
textual information with LSTM and fuse it with features extracted from
metadata using a dense layer. Wu et al. (2023) produce word, sentence, and
account embeddings with a triplet network that refines raw embeddings using
metric learning techniques. Guo et al. (2021) construct a heterogeneous graph
of the corpus, where words and user descriptions embedded by pretrained BERT
are graph nodes. The final prediction is the combination of BERT and GCN
output. Wei and Nguyen (2019) use pretrained GloVe word vectors on Twitter as
word embedding and multiple layers of bidirectional LSTMs for bot detection.

Graph-Based Methods. These methods model Twitter as graphs and apply network
science and geometric deep learning techniques for Twitter bot detection. Feng
et al. (2021b) utilize user semantics, attributes, and neighborhood
information, pretraining on abundant self-supervised data and finetuning for
different bot detection tasks. Feng et al. (2021a) first apply heterogeneous
graph neural networks RGCN (Schlichtkrull et al., 2018) to the bot detection
task, achieving breakthrough performance. Feng et al. (2022a) propose
relational graph transformers to model heterogeneous user influence and use
semantic attention networks to aggregate cross-relation messages. Ali
Alhosseini et al. (2019) extract 6 numerical features for each user, and the
extracted features are fed into GCN layers for classification. Yang et al.
(2022) use reinforcement learning and self-supervision to search GNN
architectures, learns user subgraph embeddings from heterogeneous networks for
social bot detection. Yang et al. (2013) design features from profile,
content, graph, neighbor, timing, and automation and classifies with random
forest.

Multi-Modal Methods. Recent studies have used multi-modal methods that fuse
multiple information sources to detect Twitter bots. Instead of relying on a
single source of information such as user attributes, behavioral features, or
tweet content alone, multiple sources of information are fused together for
analysis. Ng and Carley (2022) use a mixture of experts approach where each
expert analyzes a specific type of account information and their results are
combined to determine if the account is a bot. Lei et al. (2022) integrate
text and graph modality with a text-graph interaction module, with additional
functionality to detect advanced bots with a semantic consistency model. Liu
et al. (2023) use multiple user information types in community-aware mixture-
of-experts layers to improve the detection of deceptive bots and adapt to
different Twitter communities. Tan et al. (2023) leverage the largest ensemble
to probe the bot percentage on Twitter.

Upon analyzing existing methods, we found that graph-based methods generally
outperform non-graph-based methods, indicating the importance of social
network structural information in the Twitter bot detection task. Although
graph-based methods achieve state-of-the-art performance, they struggle to be
deployed in real-world social networks because they require fetching
exponential neighboring information when detecting target users, which is
heavily time-consuming. In this work, we proposed LMBot to leverage language
models as a proxy for graph-less Twitter bot detection inference and integrate
Twitter’s graph context information in the training stage by iteratively
distilling the knowledge of GNN into LM. With the LM module in LMBot, we can
efficiently perform Twitter bot detection without needing to fetch neighboring
information, making it a practical method for real-world deployment.

###  2.2. Knowledge Distillation

Knowledge Distillation (KD) (Hinton et al., 2015) is a model compression
technique where a small student model learns from a large trained teacher
model. The student model mimics the teacher to achieve similar performance but
with fewer parameters and computations. Among the existing knowledge
distillation methods, there are distillations between the same type of models
and different types of models

Distillations between the same type of models. This is conducted in order to
reduce the number of parameters. For example, Jiao et al. (2019) propose a
two-stage framework that performs distillation during pretraining and task
training, enabling TinyBERT to capture BERT’s general and task knowledge. Sanh
et al. (2019) introduce a triple loss combining language modeling,
distillation, and cosine-distance losses to distill BERT to a smaller general-
purpose language model DistilBERT during pretraining. Yan et al. (2020)
propose a small GNN with explicit peer neighborhood modeling and implicit
knowledge distillation from a deeper GNN to obtain high-performance and fast
node representations.

Distillations between different types of models. Naturally, knowledge
distillation can also be performed between different types of models. When
there are differences in the structure or mechanisms of the two models, the
knowledge background and global understanding provided by the teacher model
are difficult for the student model to acquire on its own, so this type of
knowledge distillation is also significant. For example, GNN can be distilled
into MLP. Zhang et al. (2021) propose graph-less neural networks (GLNNs) by
distilling GNN knowledge into MLPs, which have no graph dependency and infer
faster, to address GNN scalability while maintaining accuracy. Guo et al.
(2022) propose a relational KD framework, Linkless Link Prediction (LLP) that
distills relational knowledge which is centered around each node to the
student MLP.

Despite much research progress has been achieved in GNN distillation,
distilling text-rich heterogeneous information network information into
language models remains under-explored. In this work, we treat LM as the
student model and GNN as the teacher model and distill the graph structure
knowledge in GNN into LM. At the same time, LM provides embeddings for GNN.
After iteratively learning the knowledge from GNN, a graph-free LM can achieve
higher performance compared to graph-based Twitter bot detection methods.

##  3\. Methodology

Figure 3 presents an overview of our proposed framework LMBot. Specifically,
LMBot first presents user as a textual sequence, which is fed into LM to get
user embedding. We first conduct LM domain adaptation finetuning on the target
task. LMBot further iteratively distills graph structure knowledge in GNN into
LM, where LM provides GNN with user embeddings and GNN provides LM with soft
labels. Enhanced by graph structure knowledge, LM can outperform graph-based
methods.

###  3.1. User as a Textual Sequence

For every Twitter account, we have access to its metadata, tweets, and
description. To obtain a universal representation for each user that is
compatible with language models’ input, we propose a new raw representation
method that encodes users as textual sequences, without relying on any complex
feature engineering. Specifically, for users’ metadata, we directly extract
the corresponding information from the raw text, such as the username,
location, and number of followers, and organize it into a sequence {metadata}
as follows:

$\rm[M]\\{metadata_{1}\\}[SEP]\\{metadata_{2}\\}[SEP]\ldots,$

where [M] is a special token denoting the beginning of metadata, and [SEP]
denotes separation. We apply a similar processing procedure to tweets and
descriptions, obtaining a sequence that represents a user:

$\rm[M]\\{metadata\\}[D]\\{description\\}[T]\\{tweet\\},$

where [M], [D], and [T] are other special tokens to be added to the tokenizer
of the corresponding language model.

In the original tweets and descriptions, there are many noises (_e.g._ ,
hashtags, mentions, and URLs) that may harm the quality of representation LM
generates. We adopted a similar processing method in (Nguyen et al., 2020),
that is, replacing all hashtags, mentions, and URLs with #HASHTAG, @USER, and
HTTPURL tokens, and using the `TweetTokenizer` from `nltk` package to further
denoise users’ descriptions and tweets. The above operations minimize the
noise in the users’ textual information as much as possible, which is more
conducive to representation learning. By encoding diverse user information
into a textual sequence, we have a universal user representation containing
all available information for Twitter bot detection.

###  3.2. Iterative Distillation

To integrate the domain knowledge in Twitter bot detection, we first apply LM
domain adaptation finetuning to pretrained language models with our raw user
sequences and corresponding user labels. Subsequently, LMBot jointly train LM
and GNN/MLP, which could optionally insert the knowledge of graph structure
into the language model. In this framework, LM and GNN/MLP do not update
parameters simultaneously. Instead, they update iteratively in LM-step and
GNN/MLP-step. Furthermore, we combine these two steps organically using the
method of knowledge distillation.

####  3.2.1. LM domain adaptation finetuning

Considering that there are some differences between the pretraining corpus of
mainstream language models and the Twitter corpus, we first conduct domain
adaptation fine-tuning of the language model on the target task. This step is
to make the model performance more stable and accelerate the convergence of
the model. Firstly we use LM to encode the user’s textual sequence. It can be
formulated as:

(1)
$\boldsymbol{z}_{i}=\frac{1}{M_{i}}\sum_{j=1}^{M_{i}}\mathrm{LM}(\boldsymbol{t}_{i})_{j},$

where $\boldsymbol{t}_{i}$ is the textual sequence of $i$-th user, LM($\cdot$)
denotes the language model to be utilized as the feature extractor, $M_{i}$ is
the number of tokens in $\boldsymbol{t}_{i}$. We apply mean pooling to LM
output and obtain a 768-dimension embedding for user representation.

To make predictions, we apply an $L$-layer MLP to reduce the dimension of
$\boldsymbol{z}_{i}$ and project it into a two-way classification space and
obtain the predicted logit, which is formulated as:

(2)
$\boldsymbol{z}^{(l)}_{i}=\mathrm{LeakyReLU}(\boldsymbol{W}^{(l)}\cdot\boldsymbol{z}^{(l-1)}_{i}+\boldsymbol{b}^{(l)}),$

where $\boldsymbol{z}^{(0)}_{i}$ is the output of LM in Equation (1). Then we
apply softmax to the output logit $\boldsymbol{z}^{(L)}_{i}$ to get the
prediction $\boldsymbol{\hat{y}}_{i}$. We then optimize the language model for
domain adaptation with the following training objective,

(3) $Loss_{\text{LM
finetuning}}=-\sum_{i\in\mathcal{H}}\sum_{c=1}^{C}y_{ic}\log(\hat{y}_{ic})+\lambda_{1}\sum_{\theta\in{\Theta_{\text{LM}}}}\theta^{2},$

where $\mathcal{H}$ is the set of hard labels, $C=2$ is the number of classes
in the Twitter bot detection task, $y_{ic}\in\\{0,1\\}$ is the ground truth
hard label, and $\lambda_{1}$ is the coefficient of L2 regularization for LM.

####  3.2.2. GNN/MLP step

LMBot can optionally insert the knowledge of graph structure into LM, where we
use GNN for datasets with graph structure and MLP for datasets without graph
structure. In the training stage for GNN or MLP updating, GNN/MLP utilizes the
embedding $\boldsymbol{z}$ generated by the language model and optimizes
directly on the Twitter bot detection.

When the dataset has a graph structure, the GNN module can be universally
expressed as:

(4)
$\displaystyle\boldsymbol{a}_{i}^{(l)}=\underset{j\in\mathcal{N}(i)}{\mathrm{AGGERGATE}}$
$\displaystyle\left[\mathrm{MSG}^{(l)}\left(\boldsymbol{h}_{i}^{(l-1)},\boldsymbol{h}_{j}^{(l-1)},\boldsymbol{e}_{ji}\right)\right],$
(5) $\displaystyle\boldsymbol{h}_{i}^{(l)}=\mathrm{UPDATE}$
$\displaystyle\left(\boldsymbol{h}_{i}^{(l-1)},\boldsymbol{a}_{i}^{(l)}\right),$

where the $\mathrm{MSG}(\cdot,\cdot,\cdot)$ denotes messages passing from user
$i$ to user $j$ through relation, depending on the specific implementation of
the GNN layer. $\mathrm{AGGERGATE}(\cdot)$ aggregates messages from all
neighbors of node $i$ to get $\boldsymbol{a}_{i}^{(l)}$. Finally,
$\boldsymbol{h}_{i}^{(l-1)}$ and $\boldsymbol{a}_{i}^{(l)}$ are used together
to update the representation of node $i$ to get $\boldsymbol{h}_{i}^{(l)}$
through $\mathrm{UPDATE}(\cdot,\cdot)$.

After passing $L$ layer of GNN layer, we obtain GNN-based user representations
and apply a linear layer to obtain the corresponding user’s prediction logit:

(6)
$\boldsymbol{h}^{o}_{i}=\boldsymbol{W}_{o}\cdot\mathrm{LeakyReLU}(\boldsymbol{h}^{(L)}_{i})+\boldsymbol{b}_{o}.$

For datasets without graph structure, we are inspired by the success of the
MLP-based method (Hayawi et al., 2022) and replace GNN with an MLP. The MLP
module can be expressed as:

(7) $\displaystyle\boldsymbol{h}^{(l)}_{i}$
$\displaystyle=\boldsymbol{W}^{(l)}\cdot\mathrm{LeakyReLU}(\boldsymbol{h}^{(l-1)}_{i})+\boldsymbol{b}^{(l)},$
(8) $\displaystyle\boldsymbol{h}^{o}_{i}$
$\displaystyle=\boldsymbol{W}_{o}\cdot\mathrm{LeakyReLU}(\boldsymbol{h}^{(L)}_{i})+\boldsymbol{b}_{o}.$

The loss function to optimized MLP is similar to Equation (3):

(9)
$Loss_{\text{GNN/MLP}}=-\sum_{i\in\mathcal{H}}\sum_{c=1}^{C}y_{ic}\log(\hat{y}_{ic})+\lambda_{2}\sum_{\theta\in{\Theta_{\text{GNN/MLP}}}}\theta^{2}.$

After completing GNN/MLP training in this step, we select the GNN/MLP with the
best performance on the validation set to generate soft labels for the LM step
and distill the knowledge of GNN/MLP into the training of LM in the next step.
Soft labels are generated as follows:

(10)
$\displaystyle\boldsymbol{\overline{y}}_{i}=\mathrm{softmax}(\boldsymbol{h}^{o}_{i}/T),$

where $T$ is the temperature of knowledge distillation, which could control
the difficulty of the distillation task (Hinton et al., 2015).

####  3.2.3. LM step

In LM training, GNN acts as the teacher model while LM acts as the student
model to train on the target task. We first get users’ embeddings and
corresponding classification logits as described in LM domain adaptation
finetuning. Then the training objective in this step can be written as:

(11)
$\begin{split}Loss=&-(1-\alpha)\sum_{i\in\mathcal{H}}\sum_{c=1}^{C}y_{ic}\log(\hat{y}_{ic})\\\
&-\alpha\sum_{i\in\mathcal{S}}\sum_{c=1}^{C}\overline{y}_{ic}\left[\log(\hat{y}_{ic})-\log(\overline{y}_{ic})\right]\\\
&+\lambda_{1}\sum_{\theta\in{\Theta_{\text{LM}}}}\theta^{2},\end{split}$

where $\mathcal{S}$ is the set with soft labels, and $\alpha$ is the
coefficient to balance the soft label loss and the hard label loss. Note that
the first term is cross entropy loss on hard labels and the second term is
Kullback-Leibler divergence (Kullback and Leibler, 1951) loss on soft labels.

After completion of LM training in this step, we select the LM parameter with
the best performance on the validation set to generate embedding for the next
step of GNN training.

Algorithm 1 LMBot Optimization Algotithm

0: Twitter bot detection dataset $S$

0: Optimized LM parameters $\Theta_{\text{LM}}$, Optimized GNN/MLP parameters
$\Theta_{\text{GNN/MLP}}$

1: initialize $\Theta_{\text{LM}}$, $\Theta_{\text{GNN/MLP}}$

2: for each user $i\in S$ do

3: preprocess user textual sequence $\boldsymbol{t}_{i}$

4: obtain user embedding $\boldsymbol{z}_{i}\leftarrow$ Equation (1)

5: $Loss_{\text{LM finetuning}}\leftarrow$ Equation (3)

6: $\Theta_{\text{LM}}\leftarrow$ BackPropagate ($Loss_{\text{LM
finetuning}}$)

7: while $\Theta_{\text{LM}}$, $\Theta_{\text{GNN/MLP}}$ have not converged do

8: $Loss_{\text{GNN/MLP}}\leftarrow$ Equation (9)

9: $\Theta_{\text{GNN/MLP}}\leftarrow$ BackPropagate ($Loss_{\text{GNN/MLP}}$)

10: Update soft label $\boldsymbol{\overline{y}}_{i}\leftarrow$ Equation (10)

11: $Loss_{\text{LM}}\leftarrow$ Equation (11)

12: $\Theta_{\text{LM}}\leftarrow$ BackPropagate ($Loss_{\text{LM}}$)

13: Update user embedding $\boldsymbol{z}_{i}\leftarrow$ Equation (1)

14: end while

15: end for

16: return $\Theta_{\text{LM}}$, $\Theta_{\text{GNN/MLP}}$

###  3.3. Training and Inference

The entire training process is presented in Algorithm 1, which starts from LM
domain adaption, where training objective is shown in Equation (3). After
that, LMBot iteration process begins. Each iteration starts from the GNN/MLP
step and updates the parameters of LM and GNN/MLP iteratively. The loss for LM
and GNN/MLP training is shown in Equation (11) and Equation (9), respectively.

When the performance of LM or GNN/MLP does not improve on the validation set
in a curtain iteration compared with the previous one, we consider that the
model has converged and the LMBot iteration process ends. Further analysis of
convergence is presented in Section 4.4. After this, these two models can both
be used for inference, which are respectively named LMBot-LM and LMBot-
GNN/MLP.

Table 1. Statistics of the 4 selected datasets

Dataset Cresci-2015 Cresci-2017 Midterm-2018 TwiBot-20 # User 5,301 14,368
50,538 229,580 # Human 1,950 3,474 8,092 5,237 # Bot 3,351 10,894 42,446 6,589
# Tweet 2,827,757 6,637,615 0 33,488,192 Graph ✔ ✗ ✗ ✔

##  4\. Experiments

###  4.1. Experimental Settings

##### Dataset

We selected 4 widely-acknowledged Twitter bot detection datasets: Cresci-2015
(Cresci et al., 2015), Cresci-2017 (Cresci et al., 2017), Midterm-2018 (Yang
et al., 2020), and TwiBot-20 (Feng et al., 2021c). Cresci-2015 includes
account information (i.e., metadata and description) for 5,301 users and their
tweets, as well as a graph structure composed of two types of user
relationships. The Cresci-2017 dataset contains account information and tweets
for 14,368 users, without graph structure. Midterm-2018 is also a graph-less
dataset that contains account information for 50,538 users. TwiBot-20 includes
account information and tweets for 229,580 users, as well as a graph structure
composed of two types of user relationships, but only 11,826 annotated users
were actually used. For all datasets, we randomly split users into 1:1:8 ratio
for train, valid, and test respectively. We run experiments 5 times and report
the average results with standard deviations. Detailed information about the
datasets is shown in Table 1.

##### Baselines

We compare our LMBot with the following methods:

  * •

SGBot (Yang et al., 2020) leverages user metadata and engineered features from
tweets and adopts random forest for scalable and generalizable bot detection.

  * •

BotHunter (Beskow and Carley, 2018) extracts features from user’s metadata and
exploit random forest as classifier.

  * •

Kudugunta et al. (Kudugunta and Ferrara, 2018) combines synthetic minority
oversampling (SMOTE) with undersampling techniques. Then the best result is
achieved by AdaBoost Classifier.

  * •

LOBO (Echeverrï£¡ a et al., 2018) extracts 19 features from each users’
metadata and tweets and adopts random forest for classification.

  * •

BotRGCN (Feng et al., 2021a) encodes the users’ tweets and description with
RoBERTa and extract numerical and categorical features and employs RGCN for
classification.

  * •

RGT (Feng et al., 2022a) leverages graph transformers under each type of
relationship and semantic attention network across all relationships to learn
users’ representation.

  * •

SimpleHGN (Lv et al., 2021) is an improvement over GAT, making it suitable for
heterogeneous graphs with different types of edges.

  * •

GLNN (Zhang et al., 2021) distills the knowledge of GNN into MLP to achieve
fast inference without graph structure.

  * •

BIC (Lei et al., 2022) integrates text and graph modality with a text-graph
interaction module, with additional functionality to detect advanced bots with
a semantic consistency model.

  * •

BotBuster (Ng and Carley, 2022) uses mixture-of-experts approach where each
expert analyzes a specific type of information to facilitate cross-platform
bot detection.

Table 2. Hyperparameter settings of LMBot Hyperparameter | Value
---|---
optimizer | AdamW
LM model | RoBERTa
GNN model | RGCN
learning rate for LM | $10^{-5}$
learning rate for GNN | $5\times 10^{-4}$
dropout for LM | $0.1$
dropout for GNN | $0.4$
L2 regularization for LM $\lambda_{1}$ | $10^{-2}$
L2 regularization for GNN $\lambda_{2}$ | $10^{-5}$
number of layers for GNN $L$ | $2$
hidden size for GNN | $128$
temperature of knowledge distillation $T$ | 3
weight on soft label loss $\alpha$ | $0.5$
finetuning epochs for LM | $5$
Table 3. Average accuracy and F1-score over five runs on four Twitter bot detection datasets. The values in parentheses are standard deviations. The best results are in bold and the second best are underlined. ’-’ indicates missing information (such as tweets or graph structure) in the dataset to implement the method. Method | Cresci-2015 | TwiBot-20 | Cresci-2017 | Midterm-2018
---|---|---|---|---
Accuracy | F1-score | Accuracy | F1-score | Accuracy | F1-score | Accuracy | F1-score
SGBot (Yang et al., 2020) | 97.76 (0.3) | 98.23 (0.2) | 79.58 (0.4) | 83.51 (0.4) | 95.49 (0.3) | 97.04 (0.2) | 98.61 (0.1) | 99.18 (0.1)
BotHunter (Beskow and Carley, 2018) | 96.95 (0.6) | 97.58 (0.5) | 73.43 (0.7) | 78.01 (0.9) | 95.76 (0.3) | 97.21 (0.2) | 98.81 (0.0) | 99.29 (0.0)
Kudugunta et al. (Kudugunta and Ferrara, 2018) | 96.11 (0.9) | 96.88 (0.7) | 58.08 (1.7) | 48.07 (4.3) | 91.26 (0.5) | 93.98 (0.4) | 91.86 (1.2) | 94.96 (0.8)
LOBO (Echeverrï£¡ a et al., 2018) | 97.76 (0.4) | 98.23 (0.3) | 76.17 (0.5) | 80.64 (0.4) | 97.67 (0.3) | 98.46 (0.2) | - | -
BotRGCN (Feng et al., 2021a) | 98.60 (0.1) | 98.90 (0.1) | 84.42 (0.6) | 86.90 (0.1) | - | - | - | -
RGT (Feng et al., 2022a) | 98.69 (0.1) | 98.97 (0.1) | 84.70 (0.3) | 87.19 (0.2) | - | - | - | -
SimpleHGN (Lv et al., 2021) | 98.63 (0.1) | 98.92 (0.1) | 84.65 (0.6) | 87.13 (0.7) | - | - | - | -
BIC (Lei et al., 2022) | 96.13 (0.9) | 96.94 (0.8) | 75.75 (1.3) | 79.24 (1.8) | - | - | - | -
BotBuster (Ng and Carley, 2022) | 96.68 (0.2) | 96.42 (0.2) | 79.34 (0.8) | 82.47 (1.2) | - | - | - | -
GLNN-RGT (Zhang et al., 2021) | 97.72 (0.7) | 98.22 (0.5) | 83.02 (0.5) | 85.60 (0.5) | - | - | - | -
GLNN-BotRGCN (Zhang et al., 2021) | 97.71 (0.3) | 98.21 (0.2) | 82.56 (0.6) | 85.17 (0.4) | - | - | - | -
RoBERTa-finetune | 98.13 (0.3) | 97.43 (0.3) | 84.05 (0.2) | 86.13 (0.2) | 96.32 (0.2) | 92.51 (0.5) | 97.61 (0.4) | 98.58 (0.2)
LMBot-GNN/MLP (Ours) |  99.06 (0.3) |  99.26 (0.2) |  85.25 (0.2) |  87.38 (0.2) |  98.34 (0.3) |  98.91 (0.2) |  99.21 (0.1) |  99.53 (0.1)
LMBot-LM (Ours) |  98.74 (0.4) |  99.01 (0.3) |  85.63 (0.2) |  87.61 (0.3) |  98.28 (0.3) |  98.87 (0.2) |  99.21 (0.1) |  99.53 (0.1)

##### Implementation Details

We use PyTorch (Paszke et al., 2019), PyTorch Geometirc (Fey and Lenssen,
2019), scikit-learn (Pedregosa et al., 2011), and Transformers (Wolf et al.,
2020) library to implement our proposed LMBot. To facilitate reproducibility,
hyperparameters of LMBot are shown in Table 2. Our experiments are run on one
Tesla V100 GPU with 32 GB memory.

###  4.2. Main Results

We evaluate our proposed LMBot and 10 other baselines on four Twitter bot
detection benchmarks. Our results, presented in Table 3, demonstrate that:

  * •

LMBot consistently outperforms all baseline methods on the four datasets with
or without graph structure. Specifically, LMBot outperforms state-of-the-art
0.37%, 0.97%, 0.67%, 0.40% on Cresci-2015, TwiBot-20, Cresci-2015,
Midterm-2018, respectively, even without graph structure in inference stage.
Moreover, the performance of RoBERTa-finetune also confirms the effectiveness
of LMBot.

  * •

LM shows its core role in text-rich tasks. For datasets with graph structure
(Cresci-2015, and TwiBot-20), graph-based methods generally outperform non-
graph-based methods. However, LMBot outperforms traditional graph-based
methods. Moreover, no graph structure is needed during inference,
demonstrating the efficiency of LMs. At the same time, we found that GNN
enhanced by LM outperforms the original graph-based method (BotRGCN),
indicating that LM can provide GNN with better initial representations. For
datasets without graph structure, LM also shows better performance than
traditional feature-based and text-based methods, demonstrating the important
role of language models in text-rich tasks.

  * •

LMBot significantly outperforms GNN distillation baselines (GLNN (Zhang et
al., 2021)). The difference is that GLNN distills GNN into MLP, while LMBot
designs an iterative strategy to distill GNN into LM. Our LMBot’s superiority
indicates that in text-rich heterogeneous information networks, distilling GNN
into LM is more effective for graph-less deployment. Moreover, the iterative
distillation process enables GNN and LM mutually enhance each other and
therefore boost the performance of LMBot.

Figure 4. Accuracy with different combinations of LM and GNN. LMs are robust
under different combinations while the performance of GNNs is highly dependent
on the specific combination.

(a) Convergence curve when LM is RoBERTa. When LM is finetuned, it takes fewer
iterations to achieve optimal performance.

(b) Convergence curve when LM is T5. If LM is untuned, it cannot achieve
optimal performance.

Figure 5. Convergence curve when LM is finetuned and untuned. We confirm the
necessity of domain adaptation for the language model’s fast convergence speed
and stability. Figure 6. Predictions of LM and GNN/MLP on TwiBot-20 and
Cresci-2017, where $x$ axis indicates the probability of LM predicting bot and
$y$ axis indicates the probability of GNN/MLP predicting bot.

###  4.3. Combination of Different GNNs and LMs

Since various language models contain different initial inserted knowledge and
inductive bias, while diverse GNNs could be applied to distill knowledge with
different focuses, we want to study the effect of different combinations of
GNN and LM on performance. Specifically, we selected four of each, resulting
in $4\times 4=16$ combinations. For LM, we selected T5 (Raffel et al., 2020),
BART (Lewis et al., 2019), RoBERTa (Liu et al., 2019), and DeBERTa (He et al.,
2020); for GNN, we selected GCN (Kipf and Welling, 2016), GAT (Veličković et
al., 2017), SimpleHGN (Lv et al., 2021), and RGCN (Schlichtkrull et al.,
2018). The experimental results on TwiBot-20 dataset are presented in Figure
4.

  * •

For LM, regardless of the GNN combined with it, it consistently achieves
relatively stable and good performance, indicating that LM is not sensitive to
GNN combined with it and can be compatible with various GNNs and maintains
performance. Among them, the average performance (84.94) of T5 is slightly
worse than the other three models, which are close to 85.5, indicating its
slight deficiency of capacity to be adaptive to the current field.

  * •

For GNNs, performance is sensitive to the choice of both LM and GNN. On the
one hand, given LM, heterogeneous graph neural networks (SimpleHGN, RGCN)
perform significantly better than homogeneous graph neural networks (GCN,
GAT), indicating that graph structural information is very important for GNNs.
On the other hand, given GNN, choosing a high-performance LM can improve the
performance of GNN, which once again illustrates the core role of language
models. The most obvious improvement can be seen in GAT, which achieves 84.46
when combined with DeBERTa, but only 76.47 when combined with T5.

###  4.4. Convergence Analysis

To examine the iterative training process and its impact to LM and GNN,
respectively, we study how LM and GNN converge in LMBot. Specifically, we
tried two settings on the TwiBot-20 dataset: LMBot under full settings and
LMBot without LM domain adaptation to study their convergence process and the
impact of LM domain adaptation. The convergence curves are shown in Figure
5(a).

Figure 7. The performance of LM and GNN in LMBot with partial training set and
graph structure. The left half is the label-level data efficiency experiment
and the right half is the edge-level data efficiency experiment. It is
demonstrated that LMBot remains highly robust even in the partial absence of
graph structure and training labels.

Two conclusions can be drawn from the curves: First, in the LMBot iteration,
LM and GNN can promote each other’s performance. In each iteration, both
obtain improvements, indicating that LM generates better embeddings for GNN,
and GNN also passes knowledge of graph structure to LM. LM can outperform GNN
by fusing the inductive biases of GNN and its own inductive biases. Second,
regardless of whether LM starts from a low or high starting point, it can
converge to similar optimal performance, but the finetuned LM requires
significantly fewer iterations to reach convergence than the untuned LM, which
validates the effectiveness of LM’s domain adaptation.

When we replaces LM with the lower-performing LM (_e.g._ T5), the results are
shown in Figure 5(b). It can be seen that only the finetuned LM can converge
to the optimal value after more iterations, while the untuned LM cannot
converge to the optimal value at all. Here we can again see the necessity of
LM finetuning.

###  4.5. Prediction Consistency

To explore the differences and similarities between the predictions of LM and
GNN/MLP, we plot the prediction of LM and GNN/MLP on TwiBot-20 and Cresci-2017
datasets in Figure 6. Specifically, when the data points are in the first and
third quadrants (colored in blue), it indicates that GNN/MLP and LM produce
the same prediction; and the closer the data points are to $y=x$, the closer
the prediction probabilities of GNN and LM/MLP are. The percentage of blue
points among all points is 99.59% on Cresci-2017 and 95.77% on TwiBot-20,
which suggests that LM and GNN/MLP produce consistent prediction on datasets
with and without graph structure. The small differences in probability
predictions can be regarded as the knowledge under the model’s own inductive
bias, which can be learned by another model through LMBot iteration.

###  4.6. Data Efficiency Study

Since manually annotating and collecting labels and graph structure
information in the Twitter bot detection dataset requires massive human effort
and time costs, a model that achieves high performance with fewer annotations
and graph context is regarded as better. As a result, we study the influence
of randomly removing some labels and local graph structures on model
performance, in order to illustrate the capacity of our model in combating
such data efficiency challenges.

First, we trim the training set from 10% to 100% to conduct the label-level
efficiency experiment. Then we randomly retained 10% to 100% of the edges in
the original graph structure to conduct the edge-level efficiency experiment.
The results are shown in Figure 7. It can be seen that LMBot has a relatively
stable performance even with limited training labels. At the same time, it can
be seen that LMBot is still very robust in the partial absence of graph
structure, indicating the strong robustness of LMBot against limited
annotations and contextual social network information.

###  4.7. Ablation Study

To study the impact of each module on the LMBot’s performance, we conduct
ablation studies to validate our design choices. Specifically, we tried three
different settings on the TwiBot-20 dataset: (1) remove the GNN module and
LMBot iteration process, only LM finetuning; (2) remove LM finetuning and
LMBot iteration process, only GNN training; (3) replace the GNN module with
MLP and perform LMBot iteration, the results are shown in Table 4.

  * •

When we included LM training in the training process, i.e., setting (1) and
(3), it is only 1.58% and 0.35% worse than the full model, respectively,
indicating that using LM alone can achieve competitive on this task and
demonstrate LM’s strong potential for Twitter bot detection.

  * •

When we only used GNN for training, i.e., setting (2), the embeddings
generated by LM could not be updated, and the performance was significantly
worse than the full model by 11.28%, indicating that GNN relies heavily on the
representations generated by LM. At the same time, the results of setting (3)
show that the improvement of the GNN module on performance is relatively
limited. Therefore, when the graph structure is too complex or difficult to
obtain, we can only use LM to perform this task while ignoring the graph
structure information for efficient bot detection.

Table 4. Ablation study of LMBot.

Category Ablation Settings Accuracy Dif. archtecture w/o GNN 84.05(0.2) -1.58
w/o LM 73.97(0.5) -11.28 GNN replaced by MLP 85.28(0.3) -0.35 input w/o
metadata 71.34(0.7) -14.29 w/o tweets 82.80(0.3) -2.83

To explore which part of the textual information is the most important to
users, we remove the original three parts of textual information (metadata,
description, tweets) respectively. We found that users’ metadata has an
extremely important influence, which is consistent with the findings in (Feng
et al., 2021a). Since the proportion of tweets exceeds that of the
description, the performance decreased more obviously after the tweets are
removed compared with the description.

##  5\. Discussion

##### Limitations

We identify two limitations in LMBot:

  * •

Since LM contains a vast number of parameters, training LMBot is more
computationally intensive compared to graph-based methods. Consequently, LMBot
requires more time to train, which makes it more challenging to apply on
large-scale datasets such as TwiBot-22 (Feng et al., 2022b).

  * •

Due to the limitation of sequence length in language models (LMs), such as the
512 maximum input length of RoBERTa (Liu et al., 2019), LMBot cannot take into
account all the information of users, resulting in a limited proportion of
tweets being included in the input textual sequence.

##### Future Work

Moving forward, we plan to answer the following questions: how to extend LMBot
to large-scale datasets, how to make more comprehensive use of user
information under limited input sequence length, and how to extract textual
information more effectively so that LM can learn more effective
representation.

##### Ethical Statement

LMBot is designed as an assistive tool rather than the sole decision maker in
the Twitter bot detection application. Human judgment is essential for
reaching final decisions when deployed on real social networks. First, LMBot
can falsely identify legitimate accounts as bots, which may result in serious
implications for individuals or organizations spreading crucial information.
Second, both LM and GNN in LMBot may inherit the biases built into these
models. For instance, LM can perpetuate stereotypes and social prejudices in
pretraining data (Liang et al., 2021; Nadeem et al., 2020) and GNN can
discriminate against certain demographic groups (Dong et al., 2022).

##  6\. Conclusion

Though graph-based methods achieve state-of-the-art Twitter bot detection
performance, their inference depends on multi-hop neighbors which is time-
consuming to fetch through API and may introduce sampling bias. Meanwhile,
pretrained language models can achieve competitive performance without graph
structure. Inspired by this finding and the necessity of social network
structure information, we propose LMBot, a Twitter bot detection method that
can iteratively distill social network contextual information in GNN into LM
and subsequently do graph-free inference with LM to address data dependency
problem. Our experiments demonstrate that LMBot achieves state-of-the-art
performance on four widely-used Twitter bot detection benchmarks. Extensive
studies also show that LMBot is more robust, versatile, and efficient compared
to previous state-of-the-art graph-based Twitter bot detection methods.

## References

  * (1)
  * Abreu et al. (2020) Jefferson Viana Fonseca Abreu, Célia Ghedini Ralha, and João José Costa Gondim. 2020.  Twitter bot detection with reduced feature set. In _2020 IEEE International Conference on Intelligence and Security Informatics (ISI)_. IEEE, 1–6.
  * Ali Alhosseini et al. (2019) Seyed Ali Alhosseini, Raad Bin Tareaf, Pejman Najafi, and Christoph Meinel. 2019.  Detect me if you can: Spam bot detection using inductive representation learning. In _Companion Proceedings of The 2019 World Wide Web Conference_. 148–153.
  * Beskow and Carley (2018) David M Beskow and Kathleen M Carley. 2018.  Bot-hunter: a tiered approach to detecting & characterizing automated activity on twitter. In _Conference paper. SBP-BRiMS: International conference on social computing, behavioral-cultural modeling and prediction and behavior representation in modeling and simulation_ , Vol. 3.
  * Cresci et al. (2015) Stefano Cresci, Roberto Di Pietro, Marinella Petrocchi, Angelo Spognardi, and Maurizio Tesconi. 2015\.  Fame for sale: Efficient detection of fake Twitter followers.  _Decision Support Systems_ 80 (2015), 56–71.
  * Cresci et al. (2017) Stefano Cresci, Roberto Di Pietro, Marinella Petrocchi, Angelo Spognardi, and Maurizio Tesconi. 2017\.  The paradigm-shift of social spambots: Evidence, theories, and tools for the arms race. In _Proceedings of the 26th international conference on world wide web companion_. 963–972.
  * Dong et al. (2022) Yushun Dong, Ninghao Liu, Brian Jalaian, and Jundong Li. 2022\.  Edits: Modeling and mitigating data bias for graph neural networks. In _Proceedings of the ACM Web Conference 2022_. 1259–1269.
  * Echeverrï£¡ a et al. (2018) Juan Echeverrï£¡ a, Emiliano De Cristofaro, Nicolas Kourtellis, Ilias Leontiadis, Gianluca Stringhini, and Shi Zhou. 2018\.  LOBO: Evaluation of generalization deficiencies in Twitter bot classifiers. In _Proceedings of the 34th annual computer security applications conference_. 137–146.
  * Feng et al. (2022a) Shangbin Feng, Zhaoxuan Tan, Rui Li, and Minnan Luo. 2022a.  Heterogeneity-aware twitter bot detection with relational graph transformers. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 36. 3977–3985.
  * Feng et al. (2022b) Shangbin Feng, Zhaoxuan Tan, Herun Wan, Ningnan Wang, Zilong Chen, Binchi Zhang, Qinghua Zheng, Wenqian Zhang, Zhenyu Lei, Shujie Yang, et al. 2022b.  TwiBot-22: Towards graph-based Twitter bot detection.  _arXiv preprint arXiv:2206.04564_ (2022).
  * Feng et al. (2021b) Shangbin Feng, Herun Wan, Ningnan Wang, Jundong Li, and Minnan Luo. 2021b.  Satar: A self-supervised approach to twitter account representation learning and its application in bot detection. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. 3808–3817.
  * Feng et al. (2021c) Shangbin Feng, Herun Wan, Ningnan Wang, Jundong Li, and Minnan Luo. 2021c.  Twibot-20: A comprehensive twitter bot detection benchmark. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. 4485–4494.
  * Feng et al. (2021a) Shangbin Feng, Herun Wan, Ningnan Wang, and Minnan Luo. 2021a.  BotRGCN: Twitter bot detection with relational graph convolutional networks. In _Proceedings of the 2021 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining_. 236–239.
  * Ferrara et al. (2016) Emilio Ferrara, Onur Varol, Clayton Davis, Filippo Menczer, and Alessandro Flammini. 2016\.  The rise of social bots.  _Commun. ACM_ 59, 7 (2016), 96–104.
  * Fey and Lenssen (2019) Matthias Fey and Jan Eric Lenssen. 2019.  Fast graph representation learning with PyTorch Geometric.  _arXiv preprint arXiv:1903.02428_ (2019).
  * Guo et al. (2021) Qinglang Guo, Haiyong Xie, Yangyang Li, Wen Ma, and Chao Zhang. 2021.  Social bots detection via fusing bert and graph convolutional networks.  _Symmetry_ 14, 1 (2021), 30.
  * Guo et al. (2022) Zhichun Guo, William Shiao, Shichang Zhang, Yozen Liu, Nitesh Chawla, Neil Shah, and Tong Zhao. 2022.  Linkless Link Prediction via Relational Distillation.  _arXiv preprint arXiv:2210.05801_ (2022).
  * Hayawi et al. (2022) Kadhim Hayawi, Sujith Mathew, Neethu Venugopal, Mohammad M Masud, and Pin-Han Ho. 2022.  DeeProBot: a hybrid deep neural network model for social bot detection based on user profile data.  _Social Network Analysis and Mining_ 12, 1 (2022), 43\.
  * He et al. (2020) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020\.  Deberta: Decoding-enhanced bert with disentangled attention.  _arXiv preprint arXiv:2006.03654_ (2020).
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015\.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ (2015).
  * Howard et al. (2017) Philip N Howard, Gillian Bolsover, Bence Kollanyi, Samantha Bradshaw, and Lisa-Maria Neudert. 2017\.  Junk news and bots during the US election: What were Michigan voters sharing over Twitter.  _CompProp, OII, Data Memo_ 1 (2017).
  * Jiao et al. (2019) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019\.  Tinybert: Distilling bert for natural language understanding.  _arXiv preprint arXiv:1909.10351_ (2019).
  * Kipf and Welling (2016) Thomas N Kipf and Max Welling. 2016.  Semi-supervised classification with graph convolutional networks.  _arXiv preprint arXiv:1609.02907_ (2016).
  * Kouvela et al. (2020) Maria Kouvela, Ilias Dimitriadis, and Athena Vakali. 2020\.  Bot-Detective: An explainable Twitter bot detection service with crowdsourcing functionalities. In _Proceedings of the 12th International Conference on Management of Digital EcoSystems_. 55–63.
  * Kudugunta and Ferrara (2018) Sneha Kudugunta and Emilio Ferrara. 2018.  Deep neural networks for bot detection.  _Information Sciences_ 467 (2018), 312–322.
  * Kullback and Leibler (1951) Solomon Kullback and Richard A Leibler. 1951.  On information and sufficiency.  _The annals of mathematical statistics_ 22, 1 (1951), 79–86.
  * Lei et al. (2022) Zhenyu Lei, Herun Wan, Wenqian Zhang, Shangbin Feng, Zilong Chen, Qinghua Zheng, and Minnan Luo. 2022.  BIC: Twitter Bot Detection with Text-Graph Interaction and Semantic Consistency.  _arXiv preprint arXiv:2208.08320_ (2022).
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019\.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_ (2019).
  * Liang et al. (2021) Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2021.  Towards understanding and mitigating social biases in language models. In _International Conference on Machine Learning_. PMLR, 6565–6576.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * Liu et al. (2023) Yuhan Liu, Zhaoxuan Tan, Heng Wang, Shangbin Feng, Qinghua Zheng, and Minnan Luo. 2023\.  BotMoE: Twitter Bot Detection with Community-Aware Mixtures of Modal-Specific Experts.  _arXiv preprint arXiv:2304.06280_ (2023).
  * Lv et al. (2021) Qingsong Lv, Ming Ding, Qiang Liu, Yuxiang Chen, Wenzheng Feng, Siming He, Chang Zhou, Jianguo Jiang, Yuxiao Dong, and Jie Tang. 2021\.  Are we really making much progress? revisiting, benchmarking and refining heterogeneous graph neural networks. In _Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining_. 1150–1160.
  * Miller et al. (2014) Zachary Miller, Brian Dickinson, William Deitrick, Wei Hu, and Alex Hai Wang. 2014.  Twitter spammer detection using data stream clustering.  _Information Sciences_ 260 (2014), 64–73.
  * Nadeem et al. (2020) Moin Nadeem, Anna Bethke, and Siva Reddy. 2020.  StereoSet: Measuring stereotypical bias in pretrained language models.  _arXiv preprint arXiv:2004.09456_ (2020).
  * Ng and Carley (2022) Lynnette Hui Xian Ng and Kathleen M Carley. 2022.  BotBuster: Multi-platform Bot Detection Using A Mixture of Experts.  _arXiv preprint arXiv:2207.13658_ (2022).
  * Nguyen et al. (2020) Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020\.  BERTweet: A pre-trained language model for English Tweets.  _arXiv preprint arXiv:2005.10200_ (2020).
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019\.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ 32 (2019).
  * Pedregosa et al. (2011) Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011\.  Scikit-learn: Machine learning in Python.  _the Journal of machine Learning research_ 12 (2011), 2825–2830.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ 21, 1 (2020), 5485–5551.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_ (2019).
  * Schlichtkrull et al. (2018) Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018.  Modeling relational data with graph convolutional networks. In _The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3–7, 2018, Proceedings 15_. Springer, 593–607.
  * Tan et al. (2023) Zhaoxuan Tan, Shangbin Feng, Melanie Sclar, Herun Wan, Minnan Luo, Yejin Choi, and Yulia Tsvetkov. 2023.  BotPercent: Estimating Twitter bot populations from groups to crowds.  _arXiv preprint arXiv:2302.00381_ (2023).
  * Veličković et al. (2017) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017.  Graph attention networks.  _arXiv preprint arXiv:1710.10903_ (2017).
  * Volkova et al. (2015) Svitlana Volkova, Yoram Bachrach, Michael Armstrong, and Vijay Sharma. 2015.  Inferring latent user properties from texts published in social media. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 29.
  * Wei and Nguyen (2019) Feng Wei and Uyen Trang Nguyen. 2019.  Twitter bot detection using bidirectional long short-term memory neural networks and word embeddings. In _2019 First IEEE International conference on trust, privacy and security in intelligent systems and applications (TPS-ISA)_. IEEE, 101–109.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020\.  Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations_. 38–45.
  * Wu et al. (2023) Jun Wu, Xuesong Ye, and Yanyuet Man. 2023.  Bottrinet: A unified and efficient embedding for social bots detection via metric learning. In _2023 11th International Symposium on Digital Forensics and Security (ISDFS)_. IEEE, 1–6.
  * Yan et al. (2020) Bencheng Yan, Chaokun Wang, Gaoyang Guo, and Yunkai Lou. 2020\.  Tinygnn: Learning efficient graph neural networks. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 1848–1856.
  * Yang et al. (2013) Chao Yang, Robert Harkreader, and Guofei Gu. 2013\.  Empirical evaluation and new design for fighting evolving twitter spammers.  _IEEE Transactions on Information Forensics and Security_ 8, 8 (2013), 1280–1293.
  * Yang et al. (2020) Kai-Cheng Yang, Onur Varol, Pik-Mai Hui, and Filippo Menczer. 2020.  Scalable and generalizable social bot detection through data selection. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 34. 1096–1103.
  * Yang et al. (2022) Yingguang Yang, Renyu Yang, Yangyang Li, Kai Cui, Zhiqin Yang, Yue Wang, Jie Xu, and Haiyong Xie. 2022\.  RoSGAS: Adaptive Social Bot Detection with Reinforced Self-Supervised GNN Architecture Search.  _ACM Transactions on the Web_ (2022).
  * Zhang et al. (2021) Shichang Zhang, Yozen Liu, Yizhou Sun, and Neil Shah. 2021\.  Graph-less neural networks: Teaching old mlps new tricks via distillation.  _arXiv preprint arXiv:2110.08727_ (2021).