[e,a,b]Balázs<EMAIL_ADDRESS>
# Social bots sour activist sentiment without eroding engagement

Linda Li  Oxford Internet Institute, University of Oxford, Oxford, UK  Orsolya
Vásárhelyi  Center for Collective Learning, Corvinus Institute for Advanced
Studies, Corvinus University, Budapest, Hungary  Institute of Data Analytics
and Information Systems, Corvinus University, Budapest, Hungary  CEU Democracy
Institute, Budapest, Hungary  Balázs Vedres  Department of Network and Data
Science, Central European University, Vienna, Austria  Oxford Internet
Institute, University of Oxford, Oxford, UK  Center for Collective Learning,
Corvinus Institute for Advanced Studies, Corvinus University, Budapest,
Hungary

###### Abstract

Social media platforms have witnessed a substantial increase in social bot
activity, significantly affecting online discourse. Our study explores the
dynamic nature of bot engagement related to Extinction Rebellion climate
change protests from 18 November 2019 to 10 December 2019. We find that bots
exert a greater influence on human behavior than vice versa during heated
online periods. To assess the causal impact of human-bot communication, we
compared communication histories between human users who directly interacted
with bots and matched human users who did not. Our findings demonstrate a
consistent negative impact of bot interactions on subsequent human sentiment,
with exposed users displaying significantly more negative sentiment than their
counterparts. Furthermore, the nature of bot interaction influences human
tweeting activity and the sentiment towards protests. Political astroturfing
bots increase activity, whereas other bots decrease it. Sentiment changes
towards protests depend on the user’s original support level, indicating
targeted manipulation. However, bot interactions do not change activists’
engagement towards protests. Despite the seemingly minor impact of individual
bot encounters, the cumulative effect is profound due to the large volume of
bot communication. Our findings underscore the importance of unrestricted
access to social media data for studying the prevalence and influence of
social bots, as with new technological advancements distinguishing between
bots and humans becomes nearly impossible.

###### keywords:

social bots; human-bot interaction; information cascades; political
communication; protests

##  1 Introduction

Social media has become the primary channel to learn about and spread news,
engage in political discussions, and participate in collective actions over
the past decade [1, 2, 3]. Automated agents, or so-called ”social bots,” are
increasingly active in our social and political publics online [4], creating
content and interacting with humans [5]. The resulting hybrid ecosystem where
algorithmic agents and humans co-exist can fundamentally alter the nature of
democracy, political accountability, transparency, and civic participation
[6]. Automated accounts can propagate a large volume of messages at minimal
expense, and engage with users promptly and excessively, far exceeding human
capacities [7]. As a consequence, the algorithmic share of social media
content is now on par with human participation: Automated users were estimated
to be responsible for generating 10% to 40% of tweets in recent political
events, such as the 2016 US presidential election, the Brexit referendum, the
yellow vests movement, the Catalan referendum, or the 2019 United Nations
Climate Change Conference [8, 9, 10].

Recent studies have found that social bots – especially those designed to
mimic human behavior –, disrupt online political discussions [11], and there
is overwhelming evidence that they significantly influence political debates
and activism[11, 12, 13]. Such bots are frequently designed to pass as human
accounts, and occasionally explicitly mimic known political figures and
government accounts to gain the attention and trust of human users [14]. Bots
are often highly active during the flare-up of discussions around new
political events [15], and disseminate targeted messages ranging from
fabricated news to contentious, divisive, and negative content[16, 17, 18],
blending legitimate messages and misinformation. Furthermore, bots are also
often deployed in orchestrated efforts to generate the facade of a seemingly
vibrant discussion conforming with hidden agendas [19, 20] by retweeting each
other (known as ”astroturfing”), targeting susceptible users[8].

Research on bot-human interaction found that bots can often hijack the
collective attention of human users [10, 16], can increase the visibility of
extreme views [8, 16, 21], influence communication sentiment [22], and can
even alter human communication networks [23, 24]. Although social bot presence
has been studied before at the macro scale, less is known about the micro-
level impact of human-bot encounters on subsequent human activism. Small-scale
simulations and experiments indicate that bots can alter expressed human
values [24] and behaviors [25], particularly driving users towards more
extreme viewpoints [26] and potentially silencing them [27]. However, there is
little empirical evidence regarding the capacity of bots to modify human
behavior in real-life political communication[17, 19, 23, 28].

Today, online activism constitutes an essential part of democracy, but most of
the existing research focuses on institutionalized political processes such as
elections [22, 29, 30, 21], while few studies examined the role of bots within
activism related to online protests [10, 23, 31]. Furthermore, these studies
have often collapsed acts of communication, analyzing them in aggregate. The
dynamics of social media use in online activism differs markedly from
communication around other political events [32, 33], and is also a critical
component of online political discourse [34, 35]. Since social bots are
designed to be more responsive than humans [36], it is common to observe an
increase in bot activity during the peak of heated online debates, followed by
a decrease in their presence[8]. Although the bursty nature of social media
communication of protests is known [37, 38], the impact of social bots on
human activity during and after bursty periods has not been investigated.

How does interacting with social bots impact human behavior in online
activism? We address this question with data on Twitter discourse on climate-
change-related social movements. We analyze the dynamics of bots and humans
engaging with each other, and we also compare the difference in impact of bot
encounters to activists who did not engage with bots. Our analysis focuses on
protest-related discourse during a series of protest events that erupted from
November to December 2019. We decided to concentrate on online activism
related to climate change as our case study, as algorithmic threats to
engagement in climate change activism can have profound consequences on
societal agreement on the public good in a critical issue [39]. We analyze
communication around the Extinction Rebellion (XR), as the highest profile
activist group online.

The topic of climate change has been shown to attract highly engaged, active,
and committed participants [40], while there is also substantial bot activity
[41, 42]. This enables us to analyze the impact of human-bot interactions on
humans during information cascades [19], and measure the effect of bot
encounters on tweeting activity[27] and sentiment [26]. Our findings
contribute to the growing body of research on machine behaviour[6],
particularly in terms of understanding how rapidly developing hybrid human-
machine systems could potentially modify human opinion over an extended
period.

Early work related to human-bot interactions on social media found that user
sociability and network size predict who will be interacted with by a bot [43,
15]. Our results show that bots have become so widespread that it is unlikely
that an active user of X will not interact with a bot online. Our research
extends previous work by focusing on quantifying the impact of direct human-
bot communication. We found that bot type matters for the impact on users’
tweeting activity, and the initial level of support a user has toward the
climate change movement determines how a bot encounter impacts their sentiment
on climate change. Our results have important policy implications for
increasing platform transparency in how they handle automated profiles.

## Results

### Proportions of bot and human communication

We found $48\%$ of all accounts to fall into the bot category within our
sample from Twitter ($44,121$ of the total $93,499$ users). We identified
automation through a combined approach [44, 45] that integrates the results of
commonly used bot identification methods, Botometer[46] and our self-trained
machine learning-based algorithms. Botometer is a widely adopted open source
tool to identify bots on Twitter [47]. Our self-trained models used various
data sets tailored to identify users with automated behavior that attempts to
mimic humans on social networks, especially in the context of political
behavior [46]. Bots reported in the main text are the combined results of
these two models, with a fixed threshold for the Botometer ($CAP>=0.65$).
Since the concept of ”bot” encompasses varying degrees of automation, using
one fixed bot classification threshold is always a simplification. Therefore,
we repeated all analyzes reported in the main text at various bot thresholds,
and we report these results in the SI.(See Materials & Methods, Bot
Identification and SI Bot Identification for more details).

81% of tweets were replies or retweets in our database, and 51% of these
retweets originated from bots. In general, bot activity is mainly the posting
of original messages or the retweeting of each other (35% of all retweets and
71% of bot retweets were retweeting other bots), and only a small portion of
bot retweets were retweets originating from humans (29%). At the same time,
humans spread roughly the same number of messages produced by bots (54%) and
humans (46% ). If the bot threshold is increased to $CAP>=0.75$, still 48% of
the retweets originate from bots, and 45% of the human retweets originate from
bots (see SI Information cascades and SI Tables S5 to S7 for information
flows).

Figure 1: Information flow of humans and bots. The directions and amount of
tweets retweeted between bots and humans.

### Temporal co-dependence of bot and human tweeting

We found that the quantity, intensity, direction and the sentiment of the
information flow between humans and bots are highly topic-dependent. We
identified seven topics related to XR protests associated with news events,
political campaigns, or outbursts of sentiments (for example, climate change
denial) by applying bi-term topic models [48]. (See Materials & Methods, Topic
Modeling for more details and SI Table S4 for a brief summary of the topics,
including their content, top hashtags, and sample tweets). Out of these seven
topics, four were highly bursty: Topic-related activity increased sharply
periodically and then decreased suddenly[49], resulting in information
cascades. Information cascades occur when users follow the behavior of other
individuals on social networks (e.g. retweeting the same message) [50]. (See
Table S8 in Supplementary Information on Burstiness Scores).

Figure 2 shows the temporal distribution of human and bot tweets at a 5-minute
interval within an illustrative bursty period of climate change discussions on
Twitter. Although human users (red line) generated a higher volume of tweets
during the cascade’s peak, the tweeting frequency trend is jointly influenced
by both bots and humans. Table 1 shows the results of Granger Causality tests,
which examine whether one time series could be used to predict another after
introducing a time lag [51]. Our results indicate that in all identified
cascades, the number of bot tweets during bursty periods could be used to
predict human activity. Also, in $3$ cases, the number of tweets by humans
could be used to predict bots’ activities, suggesting that the effect was
mutually driven, indicting that bots and humans drove the cascade together.
The effect was stronger on bots’ side in two of those three mutually-driven
cascades, and in $3$ out of $4$ cascades humans retweeted bots much more than
bots retweeted humans. Furthermore, in $3$ out of $4$ cascades, the sentiment
of bot tweets could be used to predict human sentiment after introducing a 30-
or 35-minute time lag. The effect was mutual only for ’Anti XR protests’, but
the effect was stronger on bots’ side. This suggests that bots not only
capture human attention, but also have an immediate impact on the sentiment of
humans exposed to them. The effect remains consistent across bot CAP
thresholds ranging from $0.50$ to $0.75$ for $3$ of the identified $4$
cascades. (See more details on Burstiness in Materials & Methods, Identifying
cascades, SI Cascades, SI Table S8-S10 for Granger Causality Results across
different CAP thresholds and for Topic Burstiness Scores).

Figure 2: Panel A: Number of tweets posted by bots (blue line) and humans (red line) in a cascade mutually-driven by bots and humans within the ”Anti-XR protests” topic. Panel B: Ratio of bot and human generated tweets throughout the time period of the cascade. The number of tweets is the rolling average aggregated on 5-minute intervals. Topic | Bots$\rightarrow$Humans | Humans$\rightarrow$Bots
---|---|---
| Amount
---
| Sentiment
---
| Amount
---
| Sentiment
---
| ”Football game
---
protests”
2.27 | * | 0.38 |  | 3.97 | *** | 0.92 |
| ”Disruptive
---
engagement”
15.60 | *** | 2.17 | * | 2.68 | ** | 1.91 |
| ”Anti-XR
---
protests”
2.77 | ** | 3.96 | *** | 2.54 | ** | 3.08 | **
| ”Politicized
---
activism”
6.06 | *** | 11.40 | *** | 1.50 |  | 2.24 |
Table 1: Granger causality test results between bot and human communication
amount and sentiment. *: $p<0.05$, **: $p<0.01$, ***:$p<0.001$. Granger
causality tests whether bot activities can predict human activities (bot-
human) or human activities can predict bot activities (human-bot) in all four
identified cascades. These tests were performed on the first-order differences
for both the number of tweets (aggregated in 5-minute intervals) and the
average sentiment (calculated over 5-minute intervals). The time lags used in
the tests were set at $30$ or $35$ minutes.

### Predicting the impact of direct bot interactions

To test whether bot encounters have any influence on human users - beyond
their above shown immediate collective effect in bursty periods - we developed
two models focusing on 1) Amount and 2) Sentiment. Specifically, we analyzed
how human communication evolves over a span of $30$ days after the first
interactions with a bot in the discussion related to climate change on
Twitter. The first model captures the inclination to ”speak out” quantified by
the average number of tweets posted related to the XR protest. The second
model predicts the change in sentiment about the climate change protest
(measured on a scale ranging from $-1$ to $1$, with $-1$ representing the most
negative sentiment and 1 the most extreme sentiment).

To quantify the impact of bot interaction on human communication change, we
used a three-step process. This includes: 1) we sampled $N=303$ users who
directly interacted with bot accounts (bot-exposed) - replied or commented to
a tweet/comment originated from a bot; 2) we collected a matched sample of
$N=184$ users, who were active in the XR related protest discussion on Twitter
but had no direct interaction with bots. Matching users were selected on the
basis of the similarity score calculated pairwise to our original sample
considering publicly available metrics on human users’ profiles. (See
Materials and Methods, Matching Sample for more details.) 3) Figure 3 shows
two example timelines, one human user who directly interacted with a bot (the
interaction is shown at time 0), and a matched user that did not encounter a
bot.

3) Finally, we applied Difference-in-Difference (DiD) [52] regression models
to quantify the casual effect of meeting a bot on outcomes by comparing a set
of humans who directly interacted with bots (human replying or commenting bot
tweets) with those who did not. Our observation units are the daily activity
of human users relative to the time of interaction with bots - $30$ days
before and $30$ days after. This setting allows us to quantify the prolonged
impact of bot interaction on the frequency of tweeting and sentiment of tweets
compared to users who did not meet a bot.

Figure 3: Two examples of Twitter timelines. Top timeline shows daily tweet
counts and daily mean sentiment for a human account that directly interacted
with a bot on day 0; bottom timeline shows a matched account without a direct
bot interaction. Bar heights are proportional to tweet counts, color indicates
sentiment from $-1$ to $1$.  Figure 4: Predicted change resulting from a bot
interaction. Panel A, C: Predicted change in Amount (number of tweets). Panel
B,D: Predicted change in Sentiment. Panel A-B separate predictions (by color)
are shown for astroturf bots (light blue), and other bots (dark blue), while
Panel C-D separate predictions by users support level towards XR - supporters
(green), neutral users (blue), anti-XR (purple)

Botometer provides prediction values for belonging to seven subbot categories
ranging from $0$ to $5$. We classify a bot into a subcategory if its bot-
specific probability is greater than $2.5$. Our biggest group is
’miscellaneous other bots’ which are bots that are similar to various type of
manually annotated bots (23%), followed by manually labeled political bots,
so-called ’astroturfs’ (14%),’fake followers’ bots purchased to increase
follower counts (7%), ’self declared’ bots from botwiki.org (5%), ’spammers’
(2%) and ’financial bots’ (0.4%). Although ’miscellaneous other bots’ are
quite well represented within the XR discourse online, only 17% of human users
had direct interaction with them. As implied by the topic of our analysis, the
individuals in our sample primarily engaged with ’astrotufs’ (38%). These
automated accounts are specifically designed to participate in political
discussions [2], leading us to analyze users interacting with astroturfs
separately from those who interacted with non-astroturfs.

Figure 4. visualizes the predicted daily change in the average number of daily
tweet counts (Panel A), and the sentiment of the tweets (Panel B) grouped by
different bot probabilities (65,70,75) and subbot category (astroturf or other
type of bots). We found that encounters with astroturf bots result in an
increase in the number of tweets, while encounters with other kind of bots
result in a decrease in the number of tweets. This suggests that the most
politically relevant bot category - astroturf bots - drive the conversation by
provoking engagement from human users, while other kinds of bots have a rather
negative, silencing effect. Regardless of the bot type, direct interaction
with a bot decreases the average sentiment of human users. (See Table S12-S17
for DID models)

Astroturfs tend to be mobilized in a targeted way against users with a
specific opinion[15, 53]. Therefore, we classified bot-exposed and matched
human users by their support of XR Protests: Supporters (Pro - 52%), Neutral
(27%), and Anti-XR (Con - 21%) using ChatGPT. (See Materials and Methods on
Support Categorization). Figure 4, panel C-D indicates that human users who
support or have a neutral opinion about XR are significantly affected by
interacting with bots, while anti-XR users are not affected. Bot interaction
has the strongest negative impact on the sentiment of bot-exposed users with
neutral opinion, indicating that bots might target those users whose opinion
can be changed [15, 54]. The change in the number of tweets was significantly
increased by bot interaction for XR supporters within the least selective bot
probability category (65), although the trends are similar in the more
selective categories (.70,.75). In these models, we control for the astroturf
score of the interacted bot, which has a strong positive significant
relationship with the change in amount. Our results indicate that the type of
bot matters more for the activity change than the original support towards
protests. However, the level of sentiment change depends on the original
support level of users exposed to bots. (See SI Table S18-S23 for DID models)

We also investigated if users affected by bots alter their levels of support
as a result of bot interaction. There were no significant differences between
the distribution of bot-exposed users’ support level before, and after
interacting with a bot (Mann-Whitney U=45536.00, p=0.86).There was no
significant difference either between users exposed to bots and those matched
in terms of the change in support level.(See Model Tables 23-24 in SI on
Opinion Change). Out of the users who interacted with bots, only 9%
experienced a shift towards neutrality, while 10% of the opinions of the users
in the matched group became more neutral towards XR protests. Additionally, 7%
of the users exposed to bots and 6% of the matched users changed their opinion
completely, shifting from negative to positive or the other way around.

## Limitations

There are two potential limitations to our current research design. The first
is the relatively short time range and sample size of our dataset on XR
protests. Our dataset covers several waves of XR protests within a month, but
XR is a global phenomenon that has been going on for several years. However,
comparing our sample size to previous studies on online activism and political
communications on Twitter[34, 32], we believe that it is a valid and
representative case to illustrate the studied aspects of bot activities on
online activism. The Twitter academic product track API (available at the time
of data collection) provided the full archive of tweets based on specific
search queries; therefore, our dataset is a comprehensive sample of the online
record of bot and human activities during the protest period.

The second limitation concerns our binary bot detection method. It has three
main drawbacks, each of which we address through additional efforts and
measures. First, similar to other bot identification approaches [46, 55, 10,
56], and due to the nature of unsupervised learning, our method cannot be a
100% sure whether a user classified as a bot is genuinely a bot. Second, the
concept of ”bot” encompasses varying degrees of automation among Twitter
users, and using a fixed bot classification threshold overlooks these nuances.
We are aware that being a bot, similar to the concept of gender [57], is not a
binary classification problem. Many human users apply automation to increase
their efficiency [58], which does not turn them into bots, but they are no
longer non-automated human users. Third, a fixed threshold-based approach can
still lead to false positives and false negatives, potentially undermining the
validity of our causal inferences regarding bot activities. To address those
concerns, we ran our models with varied CAP scores. We also performed a series
of robustness checks involving various thresholds for all of our analysis and
reported them in the main text. (See Discussions, SI Information flow,
Cascades, and Difference-in-Difference regression results).

## Discussion

Bot presence is considerable and possibly increasing in the public sphere.
Even with stricter thresholds, bot activity is higher in our sample than in
previous related work published on the 2017 Catalan referendum by Stella et
al. [16], where only 19% of all interactions were from bots to humans. It was
shown that tweets from conservative bots are more retweeted by humans, which
can indicate that this difference is not only due to the continuously
increasing presence of bots [59], but could be due to the highly politicized
and international nature of our context.

We have adopted a dynamic approach to political communication, and we have
found that bots should not be thought of a constant presence, but rather bots
react in a dynamic fashion, and are driving the amount and sentiment of heated
and bursty discussions. By our analysis of Granger causality, bots impact
human behavior more, than the other way around.

To quantify the causal impact of human-bot communication, we have compared the
communication histories of human users who have directly interacted with a bot
with those matched human users who have not. This allowed us to see a
consistently negative impact of any kind of bot interaction on the sentiment
of subsequent human communications: Humans who have interacted with a bot
displayed considerably more negative sentiment than matched users.

The change in tweeting activity after a bot interaction depends on the nature
of the bot: On one hand, decidedly political astroturfing bots (aiming to
influence public opinion behind an impression of grassroots opinion) result in
an increased activity. On the other hand, interacting with other kinds of bots
(spammers, fake followers) results in a decrease in activity. However, change
in sentiment towards the protests depends on the original support level of the
user, supporting, neutral, or against XR. Our results indicate that bots might
target users whose opinions are easier to change, since the sentiment of bot-
exposed users with neutral opinion decreased the most. However, bots do not
make human users switch their support level. In sum, bot interaction is not
without impact, even if one encounter itself has only a small effect (it takes
about two bot encoutners to induce one additional tweet). Nevertheless, since
there is an exceeding amount of bot communication, these small impacts add up
to influence the public sphere in a profound way.

Although our analysis covers a period of time prior to the launch of ChatGPT
(2022.11.30), it is becoming increasingly difficult to identify bots due to
the rapid advancement mimicking human behavior[60]. As large language models
are becoming widespread and easily accessible through APIs, new social bots
can act extremely human, making it currently almost impossible to distinguish
between bots and humans, even for experts [61, 62].

Therefore, it is crucial to have unrestricted access to social media data to
assess the influence and prevalence of these new types of social bots,
although recent trends show that social media platforms are less willing to
share free data for research purposes [63, 64]. Since financial evaluations
are highly correlated with the size of the (human) user base [65], platforms
have no interest in quantifying the ratio of non-human accounts and their
impact on misinformation [15]. Most users still underestimate the effect of
bots on themselves, but as they are exposed to increased bot presence, they
tend to prefer stricter bot-regulation policies [66]. That is why we welcome
the news that the European Union requires larger platforms to provide
researchers with access to data to study systemic risks arising from the use
of their services, such as disinformation [67]. Such legislative actions can
help the scientific community continue its work to understand the consequences
of this abrupt change in technology that will alter the nature of human-bot
interactions [61, 68, 69, 70].

## Data and Methods

### Data

Our data is made up of Twitter activity around several waves of Extinction
Rebellion climate change protests from 18 November 2019 to 10 December 2019.
The dataset was collected from November to December 2022 via the Academic
Research product track API provided by Twitter[71], which enabled users to
collect a full archive sample of historical tweets filtered based on keywords
and conditions. We collected all tweets posted during this period of time that
contained the keyword ’Extinction Rebellion’, ’climate change protest’,
’XRebellion’, ’XR’ and multiple variants of keywords with slightly different
spelling. (The complete list of keywords used can be found in SI Table S1) In
total, the final data set contained 201,010 tweets and 122,130 users.

### Bot Identification

To identify social bots on Twitter, we used a combination of two sets of bot
identification methods. The first is a popular Twitter bot identification tool
known as the ’botometer’ (formerly BotorNot), which was primarly used as a
benchmark to compare other methods. The second is a set of our self-trained
bot identification model trained with open source data of bots and humans to
train supervised machine learning models.

Botometer is a publicly available tool that relies on machine learning. It is
designed to calculate a score where low scores indicate likely human accounts,
while high scores suggest likely bot accounts [72]. The algorithm considers
more than 1,000 features related to user profiles, friends, network structure,
and activity patterns, among others. Another part of our bot identification
pipeline comes from self-trained models. Training sets were derived from
existing open-source data from Twitter accounts identified as ’bots’ and
’humans’.

We trained bot identification models with 70% training and 30% of testing set
with five types of algorithms: random forest(RF), support vector machine(SVM),
logistic regression(LOG), XGboost classification (XGB) and deep learning (DL).
We developed two versions of our mdoels with ten and twenty features that were
proved to be most effective for bot identification by previous studies [73,
10]. The evaluation of the models demonstrated that the RF, DL, and XGB models
with 20 traits surpassed other models in terms of sensitivity (true positive
rate), balanced precision, precision, and F1 score. Additionally, these models
exhibited strong performance in an independent test carried out on a dataset
consisting of influential bots and human mimics that were active during the
2018 midterm election of the United States.

In our final bot identification approach, we combined the results derived from
both sets of bot identification methods. Due to the potential false positive
issues inherent in both methods, they yielded somewhat divergent results [47].
To reconcile this disparity, we classified users based on the overlapping
results of botometer and our proprietary algorithms (DL, RF, or XGB). If both
the botometer and at least one of our algorithms identified a user as a bot,
the user was classified as such; conversely, the same principle was applied to
the classification of humans. To account for the potential error in bot
identification, we performed all our analyzes with a varying baseline CAP
score ranging between $0.60-0.75$ (See more details in Supplementary
Information, Bot Identification).

### Topic Modeling

We first identified the themes emerging in the protest-related discourse with
the increasingly popular bi-term topic models [48, 74] that learn topics by
modeling word-word co-occurrence patterns [75]. After removing all retweets,
to preprocess the data we used the nltk python package [76] to remove
stopwords, usernames, emojis and links from the tweets, and lemmtize and stemm
every word.

We then trained bi-term topic models on our preprocessed data with the
bitermplus package[77]. We set up a biterm topic model of all the tweets
related to XR, which classified all tweets into 8 different topics. Based on
the u_mass coherence scores [78], we determined that 8 topics fits our dataset
the best. After evaluating the meaning of topics we dropped one topic whose
keywords and content were too diverse to extract a meaningful media agenda
from it. (See more details in Supplementary Information, Topic Modeling).

### Granger Causality

We tested whether bots attracted people’s attention to given topics, through
’granger causality’ tests. Granger causality is a time-series-based method
that tests whether one time series sequence could be used to predict another.
If $A$ is proved to be effective in predicting $B$, then $A$ is called to
granger cause $B$ [79]. We created an aggregated time series of five minutes
of the number of posts by bots and humans, then introduced a time lag (6 time
lags, or 5*6 = 30 minutes) between the number of tweets by bots and by humans.
The augmented Dickey-Fuller (ADF) test and the Kwiatkowski-Phillips-Schmidt-
Shin (KPSS) tests suggested that all time series we tested were non-stationary
(e.g., the mean or variant was not constant or there were seasonal
fluctuations in the time series trend), which can impact the accuracy of
estimation. Therefore, we used the first-order difference for all Granger
causality tests. If the number of posts by bots granger cause that of humans,
then we would infer that bots were directing humans’ attention, and vice
versa. If no granger casuality was observed on both sides, or if the effect
was mutual, following existing research practice[79, 80, 81], we conclude that
bots and humans were driving the cascade together. The formula for the test is
as follows, in which $X_{1}$ and $X_{2}$ are items in time-series vectors:

$\displaystyle
X_{1}(t)=\sum_{j=1}^{p}{A_{11,j}X_{1}(t-j)}+\sum_{j=1}^{p}{A_{12,j}X_{2}(t-j)+E_{1}(t)}$
$\displaystyle
X_{2}(t)=\sum_{j=1}^{p}{A_{21,j}X_{1}(t-j)}+\sum_{j=1}^{p}{A_{22,j}X_{2}(t-j)+E_{2}(t)}$

In the model, $p$ stands for the time gap between the two vectors for
prediction, $A$ is the coefficient or how much lagged observation can
contribute to the prediction of the model, and $\epsilon$ as residuals.
$X_{1}$ is said to granger cause $X_{2}$ if the inclusion of $X_{1}$ reduces
the variance of the residuals. (See SI Cascades, Table 1 for detailed
information of the granger causality results on all identified cascades).

### Sentiment Analysis

For sentiment analysis, the VADER package was used, which is an open source
rule-based model and has been proven particularly effective for the
classification of feelings in text from social media [82]. We used the raw
text of tweets for sentiment analysis, as suggested by the package
documentation. For each tweet, the algorithm assigns a sentiment score from
$-1$ to $+1$, with $-1$ being the most negative, $+1$ the most positive, and
$0$ neutral. (See SI Figure S6 and S7 for the distribution of sentiment of all
tweets by exposed and matched users.)

### Matched Sample

In order to understand how bots shape in the longer term (30 days after bot
interaction) human tweeting activity and tweet sentiment related to XR, we
created a sample of matched human users who did not interact with bots in our
dataset to compare with ’bot-exposed’ human users. We first identified a group
of $506$ users in our dataset as our exposed sample, the ones who directly
interacted with bots by quoting or replying to bots. Then we calculated a
similarity metric between all the ’non-exposed’ human users and exposed users.
Specifically, we calculated Eucledian distance based on the following metrics:
statuses count, followers count, friends count, favorites count, listed count,
followers growth (average number of followers increased on a daily basis),
friends growth, favorites growth, listed growth, follower friend ratio. These
traits were selected and/or calculated based on the user profile collected via
Twitter API V1.2. The formula for the Euclidian distance is as follows, in
which $p_{n}$ and $q_{n}$ means the $N_{t}h$ trait for the sample and the
matching:

$d(p,q)=\sqrt[]{(p_{1}-q_{1})^{2}+(p_{2}-q_{2})^{2}+...+(p_{n}-q_{n})^{2}}$

It is worth mentioning that not all exposed users have matched users, and not
all matched users were active during the time window in discussions related to
XR. This is because the distribution of activity levels in online political
communication is right skewed [32]; half of the users posted less than five
times about XR in our data set. After dropping non-active potentially matched
users, in total we had $184$ matched users for $303$ exposed users. If a user
has more than one matched users, we include up to the top five matched users
in our dataset.

### Identifying cascades

Cascades were identified by calculating the temporal density (the percentage
of tweets that belong to a given topic in a given time period) of each topic.
Based on previous studies on the life cycle of information cascades online
[83, 84], the time window was specified as $1$ hour. To identify cascades, we
identified bursty periods by calculating the Z scores of the average topic
density per hour for each topic. Then filtered out all time units that had a Z
score larger than 2 ($>$ 95% percentile) in any two or more consecutively two
one hour time windows. Because the z score and the topic density could be high
in time windows with only a few tweets, we also dropped those topics with no
more than 50 tweets in at least one hourly time window. (See SI cascades for
detailed information on all identified cascades.)

### Support group categorization using ChatGPT

For our research objectives, we also categorize users’ opinions regarding the
protests they engage in discussions about. To achieve this, we seek to
determine whether users are in favor of or against the protests they discuss.
This is measured using a scoring system ranging from -1 to 1, where -1
indicates complete opposition to the protest, 1 denotes strong support for the
protest, and 0 signifies a neutral stance or unrelated discussion in their
tweets. Subsequently, we employ a tri-category classification scheme for
further analysis: scores between -1 and -0.1 are classified as ”Anti-XR (Con
protest)”, those from 0.1 to 1 as ”Supporters (Pro protest),” and scores
between -0.1 and 0.1 as ”Neutral.”

The data used for this classification consists of users’ tweets from our
dataset. We evaluate the opinion expressed in all interactions (human replies
to bots) between our sample users and bots. Additionally, for each bot-exposed
user and their matched counterparts, we classify their opinions based on all
tweets from their timeline before bot interaction.

We employed OpenAI’s large language model (LLM), ChatGPT 3.5 [85], to classify
users’ standpoints. This method has been raised and adopted in various studies
[86, 87]. For each user, we provide the model with an instruction prompt on
how to classify their opinion toward climate change protests in general, along
with the text to classify (users’ tweets), and the model outputs the
aforementioned score. To generate the three opinion scores mentioned above, we
used the full timeline for each user before bot interaction (for opinion
before), the full timeline after bot interaction (for opinion after), and
human replies to bots’ tweets during bot interaction (for opinion during
interaction). (See SI Support group categorization for detailed information on
prompt engineering and verification.)

### Difference-in-Difference Models

We apply difference-in-differences (DiD) analysis to assess the effects of the
two-way treatment on human users who directly engaged with bots. DiD is a
statistical model design that incorporates both a treatment and a control
group. In this approach, we estimate the causal effect of treatment by
analyzing time series data from both treatment and control groups. We compare
the treatment effect of users who had direct interaction with bots and those
who did not, 30 days after direct interaction.

The estimator and formula of a DiD model is as followed [52]:

$\displaystyle
Y_{it}=\alpha+\beta_{1}\text{Treat}_{i}+\beta_{2}\text{Post}_{t}+\beta_{3}(\text{Treat}_{i}\times\text{Post}_{t})+\epsilon_{it}$

In the estimator, $\text{Treat}_{i}$ is the key explanatory variable of
differences in the treatment state, and $\text{Post}_{t}$ is the dummy
temporal variable that says if it is before or after treatment.

For models estimating impacts on tweeting ’amount’, we included days without
any records of tweets (zero-tweet days) into our dataset for estimation.
Because of the excessive number of zeros in the dependent variable in this
case, we used zero-inflated negative binomial models to calculate the effect
of bot interaction on the number of tweets. Since average daily sentiment is
normally distributed, we used Linear Models to estimate bot impact.

We also control for variables that can provide alternative explanations for
our findings. Our topic analysis revealed that bot activity is topic dependent
and bots generate cascades, consequently influencing human communication.
Throughout this process, bots might interact with humans multiple times.
Therefore, we control for burstiness, the topic of the interaction, and the
total number of bot interactions for each user. The sentiment of the
interaction and the popularity of the original tweets could impact the level
of activity in a thread. Longer threads may attract more bots, and we take
these factors into account as well. In our models that compare the impact of
bot interaction by support categories, we also control the support level of
the interacted bot and their astroturf score. (See SI Table S12-S23 for full
model tables).

## Author contributions

L.L., O.V., and B.V. designed research; L.L., O.V., and B.V. performed
research; L.L. O.V. and B.V. analyzed data; and L.L., O.V., and B.V. wrote the
paper. L.L and O.V contributed equally to this work.

## License

For the purpose of open access, the author has applied a CC BY public
copyright licence to any Author Accepted Manuscript version arising from this
submission.

## Conflicts of interest

The authors declare no conflict of interest.

## References

  * [1] H. Farrell, “The Consequences of the Internet for Politics,” _Annual Review of Political Science_ , vol. 15, no. 1, pp. 35–52, 2012. [Online]. Available: http://www.annualreviews.org/doi/10.1146/annurev-polisci-030810-110815
  * [2] F. B. Keller, D. Schoch, S. Stier, and J. H. Yang, “Political Astroturfing on Twitter: How to Coordinate a Disinformation Campaign,” _Political Communication_ , vol. 37, no. 2, pp. 256–280, 2020. [Online]. Available: https://doi.org/10.1080/10584609.2019.1661888
  * [3] N. Caren, K. T. Andrews, and T. Lu, “Contemporary social movements in a hybrid media environment,” _Annual Review of Sociology_ , vol. 46, pp. 443–465, 2020.
  * [4] A. Hepp, “Artificial companions, social bots and work bots: communicative robots as research objects of media and communication studies,” _Media, Culture and Society_ , vol. 42, no. 7-8, pp. 1410–1426, 2020.
  * [5] J. Seering, J. P. Flores, S. Savage, and J. Hammer, “The Social Roles of Bots,” _Proceedings of the ACM on Human-Computer Interaction_ , vol. 2, no. CSCW, pp. 1–29, 2018.
  * [6] I. Rahwan, M. Cebrian, N. Obradovich, J. Bongard, J. F. Bonnefon, C. Breazeal, J. W. Crandall, N. A. Christakis, I. D. Couzin, M. O. Jackson, N. R. Jennings, E. Kamar, I. M. Kloumann, H. Larochelle, D. Lazer, R. McElreath, A. Mislove, D. C. Parkes, A. S. Pentland, M. E. Roberts, A. Shariff, J. B. Tenenbaum, and M. Wellman, “Machine behaviour,” _Nature_ , vol. 568, no. 7753, pp. 477–486, 2019. [Online]. Available: http://dx.doi.org/10.1038/s41586-019-1138-y
  * [7] D. Assenmacher, L. Clever, L. Frischlich, T. Quandt, H. Trautmann, and C. Grimme, “Demystifying Social Bots: On the Intelligence of Automated Social Media Actors,” _Social Media and Society_ , vol. 6, no. 3, 2020.
  * [8] M. T. Bastos and D. Mercea, “The Brexit Botnet and User-Generated Hyperpartisan News,” _Social Science Computer Review_ , vol. 37, no. 1, pp. 38–54, 2019.
  * [9] T. R. Keller and U. Klinger, “Social Bots in Election Campaigns: Theoretical, Empirical, and Methodological Implications,” _Political Communication_ , vol. 36, no. 1, pp. 171–189, 2019.
  * [10] S. González-Bailón and M. De Domenico, “Bots are less central than verified accounts during contentious political events,” _Proceedings of the National Academy of Sciences of the United States of America_ , vol. 118, no. 11, 2021.
  * [11] S. C. Woolley and P. N. Howard, “Political communication, Computational Propaganda, and autonomous agents: Introduction,” _International Journal of Communication_ , vol. 10, pp. 4882–4890, 2016.
  * [12] E. Ferrara, O. Varol, C. Davis, F. Menczer, and A. Flammin, “The rise of social robots,” _Communi- cations of the ACM_ , vol. 59, no. 7, 2016. [Online]. Available: https://www.ted.com/talks/cynthia{\\_}breazeal{\\_}the{\\_}rise{\\_}of{\\_}personal{\\_}robots
  * [13] E. Ferrara, “Measuring Social Spam and the Effect of Bots on Information Diffusion in Social Media,” _Complex spreading phenomena in social systems: Influence and contagion in real-world social networks_ , pp. 229–255, 2018.
  * [14] M. C. Forelle, P. N. Howard, A. Monroy-Hernandez, and S. Savage, “Political Bots and the Manipulation of Public Opinion in Venezuela,” _SSRN Electronic Journal_ , pp. 1–8, 2015.
  * [15] C. Shao, G. L. Ciampaglia, O. Varol, K. C. Yang, A. Flammini, and F. Menczer, “The spread of low-credibility content by social bots,” _Nature Communications_ , vol. 9, no. 1, 2018.
  * [16] M. Stella, E. Ferrara, and M. De Domenico, “Bots increase exposure to negative and inflammatory content in online social systems,” _Proceedings of the National Academy of Sciences of the United States of America_ , vol. 115, no. 49, pp. 12 435–12 440, 2018.
  * [17] N. Grinberg, K. Joseph, L. Friedland, B. Swire-Thompson, and D. Lazer, “Political science: Fake news on Twitter during the 2016 U.S. presidential election,” _Science_ , vol. 363, no. 6425, pp. 374–378, 2019.
  * [18] S. Vosoughi, D. Roy, and S. Aral, “The spread of true and false news online,” _Science_ , vol. 359, no. 6380, pp. 1146–1151, 2018.
  * [19] M. Stella, M. Cristoforetti, and M. De Domenico, “Influence of augmented humans in online interactions during voting events,” _PLoS ONE_ , vol. 14, no. 5, pp. 1–8, 2019.
  * [20] R. W. Gehl and M. Bakardjieva, _Socialbots and Their Friends: Digital Media and the Automation of Sociality_ , 1st ed. USA: Routledge, 2016.
  * [21] F. Schäfer, S. Evert, and P. Heinrich, “Japan’s 2014 General Election: Political Bots, Right-Wing Internet Activism, and Prime Minister Shinzō Abe’s Hidden Nationalist Agenda,” _Big Data_ , vol. 5, no. 4, pp. 294–309, 2017.
  * [22] L. Hagen, S. Neely, T. E. Keller, R. Scharf, and F. E. Vasquez, “Rise of the Machines? Examining the Influence of Social Bots on a Political Discussion Network,” _Social Science Computer Review_ , 2020.
  * [23] E. T. C. D. Oliveira, F. O. D. Franca, D. H. Goya, and C. L. D. C. Penteado, “The influence of retweeting robots during brazilian protests,” _Proceedings of the Annual Hawaii International Conference on System Sciences_ , vol. 2016-March, pp. 2068–2076, 2016.
  * [24] N. Köbis, J. F. Bonnefon, and I. Rahwan, “Bad machines corrupt good morals,” _Nature Human Behaviour_ , vol. 5, no. 6, pp. 679–685, 2021. [Online]. Available: http://dx.doi.org/10.1038/s41562-021-01128-2
  * [25] M. Mosleh, C. Martel, D. Eckles, and D. G. Rand, “Shared partisanship dramatically increases social tie formation in a Twitter field experiment,” _Proceedings of the National Academy of Sciences of the United States of America_ , vol. 118, no. 7, p. e2022761118, feb 2021. [Online]. Available: https://www.pnas.org/doi/abs/10.1073/pnas.2022761118
  * [26] C. A. Bail, L. P. Argyle, T. W. Brown, J. P. Bumpus, H. Chen, M. B. F. Hunzaker, J. Lee, M. Mann, F. Merhout, and A. Volfovsky, “Exposure to opposing views on social media can increase political polarization,” _Proceedings of the National Academy of Sciences_ , vol. 115, no. 37, pp. 9216–9221, sep 2018. [Online]. Available: http://www.pnas.org/lookup/doi/10.1073/pnas.1804840115
  * [27] B. Ross, L. Pilz, B. Cabrera, F. Brachten, G. Neubaum, and S. Stieglitz, “Are social bots a real threat? An agent-based model of the spiral of silence to analyse the impact of manipulative actors in social networks,” _European Journal of Information Systems_ , vol. 28, no. 4, pp. 394–412, 2019.
  * [28] R. Schuchard, A. T. Crooks, A. Stefanidis, and A. Croitoru, “Bot stamina: examining the influence and staying power of bots in online social networks,” _Applied Network Science_ , vol. 4, no. 1, 2019.
  * [29] D. Stukal, S. Sanovich, J. A. Tucker, and R. Bonneau, “For Whom the Bot Tolls: A Neural Networks Approach to Measuring Political Orientation of Twitter Bots in Russia,” _SAGE Open_ , vol. 9, no. 2, 2019.
  * [30] D. Murthy, A. B. Powell, R. Tinati, N. Anstead, L. Carr, S. J. Halford, and M. Weal, “Bots and political influence: A sociotechnical investigation of social network capital,” _International Journal of Communication_ , vol. 10, no. June, pp. 4952–4971, 2016.
  * [31] C. A. D. L. Salge and E. Karahanna, “Protesting Corruption on Twitter: Is It a Bot or Is It a Person?” _Academy of Management Discoveries_ , vol. 4, no. 1, pp. 32–49, 2018.
  * [32] S. González-Bailón, J. Borge-Holthoefer, and Y. Moreno, “Broadcasters and Hidden Influentials in Online Protest Diffusion,” _American Behavioral Scientist_ , vol. 57, no. 7, pp. 943–965, 2013.
  * [33] J. Earl, T. V. Maher, and J. Pan, “The digital repression of social movements, protest, and activism: A synthetic review,” pp. 1–16, 2022.
  * [34] D. Freelon, C. McIlwain, and M. Clark, “Quantifying the power and consequences of social media protest,” _New Media and Society_ , vol. 20, no. 3, pp. 990–1011, 2018.
  * [35] W. Jennings and C. Saunders, “Street Demonstrations and the Media Agenda: An Analysis of the Dynamics of Protest Agenda Setting,” _Comparative Political Studies_ , vol. 52, no. 13-14, pp. 2283–2313, 2019.
  * [36] Z. Gilani, R. Farahbakhsh, G. Tyson, and J. Crowcroft, “A large-scale behavioural analysis of bots and humans on twitter,” _ACM Transactions on the Web (TWEB)_ , vol. 13, no. 1, pp. 1–23, 2019.
  * [37] M. Shirakawa, T. Hara, and T. Maekawa, “Never abandon minorities: Exhaustive extraction of bursty phrases on microblogs using set cover problem,” in _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_. Copenhagen, Denmark: Association for Computational Linguistics, Sep. 2017, pp. 2358–2367. [Online]. Available: https://aclanthology.org/D17-1251
  * [38] C. Comito, A. Forestiero, and C. Pizzuti, “Bursty event detection in twitter streams,” _ACM Transactions on Knowledge Discovery from Data (TKDD)_ , vol. 13, no. 4, pp. 1–28, 2019.
  * [39] N. Carter, _The Politics of the Environment: Ideas, Activism, Policy_ , 3rd ed. Cambridge University Press, 2018.
  * [40] W. L. Bennett and A. Segerberg, “The logic of connective action: Digital media and the personalization of contentious politics,” _Information Communication and Society_ , vol. 15, no. 5, pp. 739–768, 2012.
  * [41] T. Marlow, S. Miller, and J. T. Roberts, “Bots and online climate discourses: Twitter discourse on president trump’s announcement of u.s. withdrawal from the paris agreement,” _Climate Policy_ , vol. 21, no. 6, pp. 765–777, 2021. [Online]. Available: https://doi.org/10.1080/14693062.2020.1870098
  * [42] C. F. Chen, W. Shi, J. Yang, and H. H. Fu, “Social bots’ role in climate change discussion on Twitter: Measuring standpoints, topics, and interaction strategies,” _Advances in Climate Change Research_ , vol. 12, no. 6, pp. 913–923, 2021. [Online]. Available: https://doi.org/10.1016/j.accre.2021.09.011
  * [43] C. Wagner, S. Mitter, C. Körner, M. Strohmaier _et al._ , “When social bots attack: Modeling susceptibility of users in online social networks.” in _# MSM_ , 2012, pp. 41–48.
  * [44] E. Alothali, N. Zaki, E. A. Mohamed, and H. Alashwal, “Detecting social bots on twitter: a literature review,” in _2018 International conference on innovations in information technology (IIT)_. IEEE, 2018, pp. 175–180.
  * [45] F. Martini, P. Samula, T. R. Keller, and U. Klinger, “Bot, or not? comparing three methods for detecting social bots in five political discourses,” _Big data & society_, vol. 8, no. 2, p. 20539517211033566, 2021.
  * [46] K. Yang, E. Ferrara, and F. Menczer, “Botometer 101: Social bot practicum for computational social scientists,” _CoRR_ , vol. abs/2201.01608, 2022. [Online]. Available: https://arxiv.org/abs/2201.01608
  * [47] A. Rauchfleisch and J. Kaiser, “The false positive problem of automatic bot detection in social science research,” _PLOS ONE_ , vol. 15, no. 10, pp. 1–20, 10 2020. [Online]. Available: https://doi.org/10.1371/journal.pone.0241045
  * [48] X. Yan, J. Guo, Y. Lan, and X. Cheng, “2013-a Bitem Topic Model for Short Texts.Pdf,” _Www_ , pp. 1445–1455, 2013. [Online]. Available: http://delivery.acm.org/10.1145/2490000/2488514/p1445-yan.pdf?ip=137.189.241.20{\&}id=2488514{\&}acc=ACTIVESERVICE{\&}key=CDD1E79C27AC4E65.63D3CA449C1BD759.4D4702B0C3E38B35.4D4702B0C3E38B35{\&}{\\_}{\\_}acm{\\_}{\\_}=1548658783{\\_}4fb56b68e11530cddd7103a94a10778c
  * [49] K.-I. Goh and A.-L. Barabasi, “Burstiness and memory in complex systems,” 2006.
  * [50] B. D. Easley, “Chapter 16 Information Cascades,” _Networks, Crowds, and Markets_ , pp. 483–508, 2010.
  * [51] C. Hsiao, “Causality tests in econometrics,” _Journal of Economic Dynamics and Control_ , vol. 1, no. 4, pp. 321–346, nov 1979. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0165188979901945
  * [52] S. Athey and G. W. Imbens, “Identification and inference in nonlinear difference-in-differences models,” _Econometrica_ , vol. 74, no. 2, pp. 431–497, 2006.
  * [53] O. Varol, E. Ferrara, C. Davis, F. Menczer, and A. Flammini, “Online human-bot interactions: Detection, estimation, and characterization,” in _Proceedings of the international AAAI conference on web and social media_ , vol. 11, no. 1, 2017, pp. 280–289.
  * [54] O. Varol, E. Ferrara, C. A. Davis, F. Menczer, and A. Flammini, “Online human-bot interactions: Detection, estimation, and characterization,” _Proceedings of the 11th International Conference on Web and Social Media, ICWSM 2017_ , no. Icwsm, pp. 280–289, 2017.
  * [55] K. Yang, O. Varol, C. A. Davis, E. Ferrara, A. Flammini, and F. Menczer, “Arming the public with AI to counter social bots,” _CoRR_ , vol. abs/1901.00912, 2019. [Online]. Available: http://arxiv.org/abs/1901.00912
  * [56] S. Feng, Z. Tan, H. Wan, N. Wang, Z. Chen, B. Zhang, Q. Zheng, W. Zhang, Z. Lei, S. Yang, X. Feng, Q. Zhang, H. Wang, Y. Liu, Y. Bai, H. Wang, Z. Cai, Y. Wang, L. Zheng, Z. Ma, J. Li, and M. Luo, “TwiBot-22: Towards Graph-Based Twitter Bot Detection,” _Advances in Neural Information Processing Systems_ , pp. 1–25, 2022. [Online]. Available: http://arxiv.org/abs/2206.04564
  * [57] B. Vedres and O. Vasarhelyi, “Gendered behavior as a disadvantage in open source software development,” _EPJ Data Science_ , vol. 8, no. 1, p. 25, 2019.
  * [58] Z. Chu, S. Gianvecchio, H. Wang, and S. Jajodia, “Detecting automation of Twitter accounts: Are you a human, bot, or cyborg?” _IEEE Transactions on Dependable and Secure Computing_ , vol. 9, no. 6, pp. 811–824, 2012.
  * [59] L. Luceri, A. Deb, A. Badawy, and E. Ferrara, “Red bots do it better: Comparative analysis of social bot partisan behavior,” in _Companion proceedings of the 2019 world wide web conference_ , 2019, pp. 1007–1012.
  * [60] Q. Mei, Y. Xie, W. Yuan, and M. O. Jackson, “A turing test of whether ai chatbots are behaviorally similar to humans,” _Proceedings of the National Academy of Sciences_ , vol. 121, no. 9, p. e2313925121, 2024.
  * [61] A. Jo, “The promise and peril of generative ai,” _Nature_ , vol. 614, no. 1, pp. 214–216, 2023.
  * [62] K.-C. Yang and F. Menczer, “Anatomy of an ai-powered malicious social botnet,” _arXiv preprint arXiv:2307.16336_ , 2023.
  * [63] The Verge, “Twitter’s new Elon Musk API policy is already chilling academic research,” https://www.theverge.com/2023/5/31/23739084/twitter-elon-musk-api-policy-chilling-academic-research, May 2023.
  * [64] ——, “Reddit’s API changes are already starting to shut down some apps,” https://www.theverge.com/2023/6/9/23755640/reddit-api-changes-apps-apollo-shut-down-ama-spez-steve-huffman, June 2023.
  * [65] TechCrunch, “IRL shuts down, citing issues with fake users,” https://techcrunch.com/2023/06/26/irl-shut-down-fake-users/, June 2023.
  * [66] H. Y. Yan, K.-C. Yang, J. Shanahan, and F. Menczer, “Exposure to social bots amplifies perceptual biases and regulation propensity,” _Scientific Reports_ , vol. 13, no. 1, p. 20707, 2023.
  * [67] European Commission, “Digital Services Act (DSA) overview,” https://commission.europa.eu/strategy-and-policy/priorities-2019-2024/europe-fit-digital-age/digital-services-act_en#:~:text=Digital%20Services%20Act%20(DSA)%20overview,online%20travel%20and%20accommodation%20platforms., August 2023.
  * [68] I. Solaiman, “The gradient of generative ai release: Methods and considerations,” in _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_ , ser. FAccT ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 111–122. [Online]. Available: https://doi.org/10.1145/3593013.3593981
  * [69] S. Li, J. Yang, and K. Zhao, “Are you in a masquerade? exploring the behavior and impact of large language model driven social bots in online social networks,” _arXiv preprint arXiv:2307.10337_ , 2023.
  * [70] M. del Rio-Chanona, N. Laurentsyeva, and J. Wachs, “Are large language models a threat to digital public goods? evidence from activity on stack overflow,” _arXiv preprint arXiv:2307.07367_ , 2023.
  * [71] Twitter, “Twitter API for Academic Research — Products — Twitter Developer Platform,” 2022. [Online]. Available: https://developer.twitter.com/en/products/twitter-api/academic-research
  * [72] M. Sayyadiharikandeh, O. Varol, K. Yang, A. Flammini, and F. Menczer, “Detection of novel social bots by ensembles of specialized classifiers,” _CoRR_ , vol. abs/2006.06867, 2020. [Online]. Available: https://arxiv.org/abs/2006.06867
  * [73] K.-C. Yang, O. Varol, P.-M. Hui, and F. Menczer, “Scalable and Generalizable Social Bot Detection through Data Selection,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 01, pp. 1096–1103, apr 2020. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/5460
  * [74] L. Shi, G. Cheng, S. R. Xie, and G. Xie, “A word embedding topic model for topic detection and summary in social networks,” _Measurement and Control (United Kingdom)_ , vol. 52, no. 9-10, pp. 1289–1298, 2019.
  * [75] J. Wijffels, _BTM: Biterm Topic Models for Short Text_ , 2023, r package version 0.3.7. [Online]. Available: https://CRAN.R-project.org/package=BTM
  * [76] S. Bird, E. Klein, and E. Loper, _Natural language processing with Python: analyzing text with the natural language toolkit_. ” O’Reilly Media, Inc.”, 2009.
  * [77] M. Terpilovskii and C. Kälin, “Bitermplus package,” 2022. [Online]. Available: https://bitermplus.readthedocs.io/en/latest/index.htmlhttps://bitermplus.readthedocs.io/en/latest/install.html
  * [78] M. Röder, A. Both, and A. Hinneburg, “Exploring the space of topic coherence measures,” in _Proceedings of the Eighth ACM International Conference on Web Search and Data Mining_ , ser. WSDM ’15. New York, NY, USA: Association for Computing Machinery, 2015, p. 399–408. [Online]. Available: https://doi.org/10.1145/2684822.2685324
  * [79] M. T. Bahadori and Y. Liu, “An examination of practical granger causality inference,” _Proceedings of the 2013 SIAM International Conference on Data Mining, SDM 2013_ , pp. 467–475, 2013.
  * [80] A. Ceron, L. Curini, and S. M. Iacus, “First- and second-level agenda setting in the Twittersphere: An application to the Italian political debate,” _Journal of Information Technology and Politics_ , vol. 13, no. 2, pp. 159–174, 2016. [Online]. Available: http://dx.doi.org/10.1080/19331681.2016.1160266https://doi.org/10.1080/19331681.2016.1160266
  * [81] M. T. Bastos, D. Mercea, and A. Charpentier, “Tents, tweets, and events: The interplay between ongoing protests and social media,” _Journal of Communication_ , vol. 65, no. 2, pp. 320–350, 2015.
  * [82] C. J. Hutto and E. Gilbert, “Vader: A parsimonious rule-based model for sentiment analysis of social media text.” in _ICWSM_ , E. Adar, P. Resnick, M. D. Choudhury, B. Hogan, and A. H. Oh, Eds. The AAAI Press, 2014. [Online]. Available: http://dblp.uni-trier.de/db/conf/icwsm/icwsm2014.html
  * [83] P. Xu, Y. Wu, E. Wei, T. Q. Peng, S. Liu, J. J. H. Zhu, and H. Qu, “Visual analysis of topic competition on social media,” _IEEE Transactions on Visualization and Computer Graphics_ , vol. 19, no. 12, pp. 2012–2021, 2013.
  * [84] M. D. Vicario, A. Bessi, F. Zollo, F. Petroni, A. Scala, G. Caldarelli, H. E. Stanley, and W. Quattrociocchi, “The spreading of misinformation online,” _Proceedings of the National Academy of Sciences of the United States of America_ , vol. 113, no. 3, pp. 554–559, 2016.
  * [85] OpenAI, “ChatGPT: A large language model trained by openai,” https://openai.com/chatgpt, 2024, accessed on March 13, 2024.
  * [86] Y. Zhu, P. Zhang, E.-U. Haq, P. Hui, and G. Tyson, “Can chatgpt reproduce human-generated labels? a study of social computing tasks,” _arXiv preprint arXiv:2304.10145_ , 2023.
  * [87] L. Wu, Y. Chen, J. Yang, G. Shi, X. Qi, and S. Deng, “Event-centric opinion mining via in-context learning with chatgpt,” in _China Conference on Knowledge Graph and Semantic Computing_. Springer, 2023, pp. 83–94.

See pages - of BOT_SI.pdf