# Engagement, User Satisfaction, and the
Amplification of Divisive Content on Social Media

Smitha Milli  Cornell Tech  Corresponding author<EMAIL_ADDRESS>Micah
Carroll  University of California, Berkeley  Yike Wang  University of
California, Berkeley
Sashrika Pandey  University of California, Berkeley  Sebastian Zhao
University of California, Berkeley  Anca D. Dragan  University of California,
Berkeley

> In a pre-registered randomized experiment, we found that, relative to a
> reverse-chronological baseline, Twitter’s engagement-based ranking algorithm
> amplifies emotionally charged, out-group hostile content that users say
> makes them feel worse about their political out-group. Furthermore, we find
> that users do _not_ prefer the political tweets selected by the algorithm,
> suggesting that the engagement-based algorithm underperforms in satisfying
> users’ stated preferences. Finally, we explore the implications of an
> alternative approach that ranks content based on users’ stated preferences
> and find a reduction in angry, partisan, and out-group hostile content but
> also a potential reinforcement of echo chambers. The evidence underscores
> the necessity for a more nuanced approach to content ranking that balances
> engagement, users’ stated preferences, and sociopolitical outcomes.

Table 1: Prior and Current Research on Algorithmic Ranking Outcomes |  | Ranking by engagement | Ranking by stated pref.
---|---|---|---
No. | Outcome | Obs. | Experiment | Experiment
Sock puppets | Users | Users
1 | Emotions in content |  (?, ?, ?, ?, ?, ?) |  (?) |  Ours, (?) | Ours
2 | Partisanship of content |  (?, ?, ?, ?, ?, ?) |  (?, ?) |  Ours, (?, ?, ?, ?) | Ours
3 | Out-group animosity in content |  (?, ?) |  | Ours | Ours
4 | Reader’s emotions |  (?, ?) |  | Ours | Ours
5 | Reader’s in-group perception |  (?) |  |  Ours, (?) | Ours
6 | Reader’s out-group perception |  (?) |  |  Ours, (?, ?) | Ours
7 | Reader’s stated pref. for content |  |  |  Ours, (?) |  Ours, (?, ?)
The outcomes analyzed in our paper and prior work. Prior research mainly
studied engagement-based ranking, with both observational studies and
controlled experiments that used either sock puppets or real users. To our
knowledge, we are the first to study the implications of ranking by users’
stated preferences on these outcomes (and thus, for brevity, we omit the empty
columns for sock puppet experiments and observational studies). We classified
a study as providing observational evidence for the effects of engagement-
based ranking if it investigated an association between the amount of
engagement content received and any of the listed outcomes. Black cells
indicate instances where we could not find any relevant prior work.

Social media ranking algorithms personalize content to individual user
preferences. However, many hypothesize that these algorithms could
inadvertently facilitate the propagation of angry content, stoke hostile
discourse, and escalate polarization (?, ?, ?, ?). Although numerous studies
have delved into the effects of social media ranking algorithms, most are
observational and have often yielded inconsistent findings (?). The few
randomized experiments (?, ?, ?, ?) that do exist have been primarily
conducted by the companies that design these algorithms. While these in-house
studies provide valuable insights, they are typically infeasible for external
researchers to replicate. This accentuates the need for more randomized
experimentation, especially from outside of the companies.

Moreover, while it is essential to know _whether_ these algorithms are causing
adverse outcomes, to design better alternatives, we must also understand _why_
they do so. Even existing randomized experiments that isolate the effect of
the ranking algorithm seldom delve into identifying which specific elements of
the algorithm contribute to such outcomes. In this paper, we aim to delve
deeper, not only examining if current algorithms are exacerbating divisiveness
but also exploring the extent to which this is driven by their prioritization
of engagement (users’ revealed preferences) over a more reflective notion of
users’ preferences.

To do so, we conducted a randomized controlled experiment measuring the
impacts of Twitter’s111Twitter rebranded to “X” in July 2023. Our study took
place before the rebranding, so we refer to the platform as “Twitter”.
engagement-based ranking algorithm. Our study was pre-registered, required no
collaboration with the platform, and our data and code are available to the
research community at http://github.com/smilli/twitter. Over two weeks in
February 2023, we recruited a group of Twitter users (N=806) and collected the
first ten tweets that the personalized algorithm would have shown them and the
first ten tweets they would have seen under a reverse-chronological baseline.
Using a within-subjects design, we showed users the tweets from both rankings
in a randomized order and surveyed them about each one, allowing us to measure
the causal impact of the engagement-based ranking on readers’ perceptions,
relative to the reverse-chronological baseline.

To investigate the role of the engagement-based algorithm in shaping discourse
and attitudes, we studied its effect on six key outcomes (Outcomes 1-6 in
Table 1) related to the curation of emotional and political content and the
emotions and political perceptions of readers. In particular, we measured how
the algorithm affected the emotions, partisanship, and out-group animosity
expressed in the content shown, as well as the emotions experienced by the
reader, and the reader’s perception of their political in-group and out-group.
Throughout our paper, we refer to these six outcomes as “sociopolitical
outcomes”. As summarized in Table 1, there is little experimental evidence for
many of these outcomes. Thus, our first line of inquiry can be summarized by
the following research question:

  1. Question 1.

How does the engagement-based ranking algorithm impact the sociopolitical
outcomes under consideration (Outcomes 1-6)?

Our second research question asked whether Twitter’s ranking algorithm, which
optimizes primarily for engagement (users’ revealed preferences) also aligns
with users’ explicitly stated preferences. Many hypothesize that an
engagement-centric approach may inadvertently promote polarizing content by
focusing on users’ immediate reactions rather than their reflective
evaluations of the content (?, ?, ?, ?). Therefore, if this engagement-focused
characteristic is a key factor in the amplification of divisiveness, it would
be reasonable to expect that the algorithm might underperform in aligning with
users’ stated preferences. However, we found almost no prior studies that
provided evidence for how well ranking algorithms satisfied users’ stated
preferences222Jia et al. (2023) (?) was an exception. However, they did not
evaluate a real-world recommender system’s impact on user satisfaction, like
the Twitter or Facebook algorithm, but rather their own custom _non-
personalized_ ranking of Facebook posts based on public reaction counts.
(Table 1), relative to simple baselines like a chronological or random feed.

  1. Question 2.

Does the engagement-based ranking algorithm show more content that users say
they want to see (Outcome 7)?

Finally, in exploratory analysis, we measured the impact of ranking based on
users’ explicitly stated preferences rather than the traditional engagement-
based approach. Aligning with a more reflective notion of value—focusing on
users’ stated preferences—could potentially counter the negative effects
commonly associated with engagement-based algorithms (?, ?, ?, ?). However, it
is crucial to verify these theoretical benefits as it could be that users
still prefer content that is divisive or hostile, even after reflection (?).
We were unable to find any prior work measuring the effect of ranking by
stated preferences on the sociopolitical outcomes we consider (Table 1). Thus,
our last, exploratory inquiry aims to fill this gap and is summarized in the
following question.

  1. Question 3.

What impact does ranking by users’ stated preferences have on the
sociopolitical outcomes considered (Outcomes 1-6)?

###  1 Results

From February 11 to February 27, 2023, we conducted our experiment with active
Twitter users from the United States. We used CloudResearch Connect, an online
crowd-working platform, to recruit participants. The study period was divided
into four waves, and participants were allowed to take part in the study once
during each wave. In total, 806 unique users participated in our study a total
of 1730 times. Each time a user participated in our study, we collected two
sets of tweets for them: (a) the first ten tweets they would have been shown
by Twitter’s engagement-based algorithm and (b) the first ten tweets they
would have seen if they were shown the latest tweets from the people they
follow.

Using a within-subjects design, we then surveyed users about tweets from both
of their timelines in a randomized order, querying them about Outcomes 1-7 in
Table 1.333If a tweet was a quote tweet or a reply to another tweet, then we
surveyed the users about both the quote/quoted tweet and the reply/replied
tweet. If a tweet was common to both the engagmeent-based and chronological
timeline, we only surveyed the user about it once. For each tweet, we asked
the user to assess whether it was about a political or social issue. If they
marked the tweet as political, we asked them about the tweet’s ideological
leaning, their own perception of their political in-group and out-group after
reading the tweet, and whether the tweet author was expressing animosity
towards their out-group. For all tweets (including non-political tweets), we
asked users to assess the author’s emotions as well as the emotions the tweet
made them feel, along four dimensions: happiness, anger, sadness, and anxiety.
Finally, we also asked the reader whether they wanted to see tweets like it
when they used Twitter.

Note that, for our main analysis, we rely on the reader’s judgments even for
outcomes that are about the tweet itself (the tweet’s ideological leaning, the
emotions expressed by the author, and whether the tweet expresses out-group
animosity). In Section S4.6, we use GPT-4 (?), a large language model, to
label these outcomes instead and find that all results are qualitatively the
same.

A full description of our study procedure can be found in SM section S1. In
Section S4.1, we provide descriptive statistics of the metadata in both
timelines, e.g., the number of likes, retweets, links, photos, etc in each
tweet, as well as analysis on the amplification of individual user accounts.

####  1.1 Effects of engagement-based ranking

First, we state our findings on how Twitter’s engagement-based algorithm
affects political outcomes, emotional outcomes, and the reader’s stated
preference for the tweets shown. All tested outcomes and our analysis plan
were pre-registered at https://osf.io/upw9a. Figure 1 shows a summary of the
average treatment effect for each outcome. As specified in our pre-analysis
plan, the average treatment effect is estimated through a difference in means
(see Section S1.3), and two-sided $p$-values are estimated by paired
permutation tests. In total, we tested 26 outcomes, and all results that are
significant (at a $p$-value threshold of $0.05$) remain significant at a false
discovery rate (FDR) of $0.01$. Thus, in expectation, none of our discoveries
are false discoveries. The full table of standardized and unstandardized
effect sizes, $p$-values, and FDR-adjusted $p$-values can be found in SM
section S3.

Figure 1: Average treatment effects for all outcomes. ATEs are shown with 95%
Bootstrap confidence intervals (unadjusted for multiple testing). The effects
of two different timelines are shown: (1) Twitter’s own engagement-based
timeline, (2) our exploratory timeline that ranks based on users’ stated
preferences. The effect sizes for both timelines are relative to the reverse-
chronological timeline (the zero line). Average treatment effects are
standardized using the standard deviation of outcomes in the chronological
timeline (see Section S1.3 for details).

Political outcomes (Outcomes 2, 3, 5, and 6). We defined the _partisanship_ of
a tweet as the absolute value of the reader-provided ideological leaning of
the tweet (ranging from -2 = “Far left” to +2 = “Far right”). Relative to the
reverse-chronological baseline, we found that the engagement-based algorithm
amplified tweets that exhibited greater partisanship ($0.24$ SD, $p<0.001$)
and expressed more out-group animosity ($0.24$ SD, $p<0.001$). Furthermore,
tweets from the engagement-based algorithm made users feel significantly worse
about their political out-group ($-0.17$ SD, $p<0.001$) and better about their
in-group ($0.08$ SD, $p=0.0014$). These effects remained significant when
considering left and right-leaning users specifically, except for in-group
perception in left-leaning users, where we found no significant effect.

Amplified emotionality (Outcomes 1 and 4). The algorithm significantly
amplified tweets that expressed negative emotions—anger ($0.47$ SD,
$p<0.001$), sadness ($0.22$ SD, $p<0.001$), and anxiety ($0.23$ SD,
$p<0.001$). It also leads readers to feel more of all four emotions—anger
($0.27$ SD, $p<0.001$), sadness ($0.09$ SD, $p=0.003$), anxiety ($0.12$ SD,
$p<0.001$), and happiness ($0.12$ SD, $p<0.001$). When considering only
political tweets, we found that anger was by far the predominant emotion
amplified by the algorithm, both in terms of the emotions expressed by authors
($0.75$ SD, $p<0.001$) and the emotions felt by readers ($0.37$ SD,
$p<0.001$).

User’s stated preference (Outcome 7). For each tweet, we also asked users
whether they wanted to see tweets like it when they used Twitter. We found
that overall, tweets shown by the algorithm are rated slightly higher ($0.06$
SD, $p=0.022$). Interestingly, however, the political tweets recommended by
the personalized algorithm lead to significantly lower user value than the
political tweets in the chronological timeline ($-0.18$ SD, $p=0.005$).

Figure 2: The distribution of political tweets and out-group animosity. The
graph on the left shows the distribution of political tweets in each timeline,
categorized by whether they align with the reader’s in-group, out-group, or
are moderate. Meanwhile, the graph on the right delineates the proportion of
political tweets that express out-group animosity, broken down by whether they
target the reader’s in-group or out-group. The stated preference timeline has
a lower percentage of tweets with animosity than the engagement timeline.
However, this decrease is mainly due to a decrease in animosity towards the
reader’s _in-group_ ; the percentage of tweets with animosity towards the
reader’s _out-group_ stays roughly the same.

####  1.2 Ranking by stated preferences

Next, we conduct an exploratory analysis in which we simulate an alternative
ranking of tweets that is based on users’ stated preferences. In particular,
first, we take the approximately twenty unique tweets that each user was
surveyed about and re-rank these tweets by the users’ stated preference for
the tweet. Each tweet receives a score of $1$ = “Yes”, $0$ = “Indifferent”, or
$-1$ = “No”, depending on the users’ stated preference for the tweet.444If a
tweet was a quote tweet or a reply to another tweet, then we consider the two
tweets to be one combined “tweet object” for the purposes of ranking and score
them as the average of the users’ stated preferences for both tweets. The
“Stated Preference” (SP) timeline consists of the ten tweets (out of the
approximately twenty unique tweets) that scored highest according to the
users’ stated preferences (ties are broken at random).

As shown in Figure 2, relative to the engagement timeline, the SP timeline
reduces negativity (anger, sadness, anxiety) and increases happiness, both in
terms of the emotions expressed by authors and the emotions felt by readers.
Moreover, the content shown in the SP timeline is less partisan and less
likely to contain expressions of out-group animosity. However, further
analysis shown in Figure 2 reveals that the reduction in partisanship and
animosity in the SP timeline is due almost entirely to reducing the number of
tweets from the reader’s out-group, and by reducing the animosity directed
toward the reader’s in-group (but not animosity towards the reader’s out-
group).

###  2 Discussion

####  2.1 The effects of Twitter’s engagement-based algorithm

###### Amplification of emotions and out-group animosity.

Our study reveals that the engagement-based algorithm tends to amplify
emotionally charged content, particularly that which expresses anger and out-
group animosity. The differences were sizable. Of the political tweets chosen
by Twitter’s algorithm, 62% expressed anger and 46% contained out-group
animosity. In comparison, of the political tweets in the chronological
timeline, 52% expressed anger and 38% contained out-group animosity. This
finding helps clarify inconsistencies in previous research. Earlier studies
relied on observational methods that were limited to specific subsets of
content on the platform (?, ?) and replications produced mixed results (?, ?).
Our randomized experiment provides insight into this debate by providing
causal evidence that, indeed, the algorithm amplifies emotionally charged
content. In order to ensure greater external validity, we also examined the
effect of the algorithm on tweets that users themselves would actually
encounter on the platform in their own timelines.

###### Partisan tweets, but no filter bubble.

Like prior observational and experimental work (?, ?, ?), we found that the
content that the engagement-based algorithm amplified tended to be more
partisan. However, we did not find evidence that the engagement-based
algorithm contributed to a “filter bubble” (?). On the chronological timeline,
48% of political tweets came from the users’ in-group, and 11% came from the
users’ out-group. On the other hand, on the engagement-based timeline, 50% of
sociopolitical tweets were in-group tweets and 16% were out-group tweets (see
Figure 2). Thus, the algorithm actually showed users _more_ content from their
out-group, and this increase was larger than the corresponding increase in in-
group content. This is partially consistent with Guess et al. (2023) who found
that the Facebook news feed ranking algorithm increased the amount of in-group
_and_ out-group content (?). However, they also found that the increase in in-
group content was greater than the increase in out-group content—a reversal of
our finding.

###### Amplification of polarizing content.

We asked users how they felt about their political in-group and out-group
(from “Much worse” to “Much better”) after each individual piece of content
they saw from either the engagement-based or chronological timeline. We found
that, after reading tweets selected by the engagement-based algorithm, users
tended to have more positive perceptions of their in-group and more negative
perceptions of their out-group, compared to the chronological baseline. Our
results suggest that the engagement-based ranking algorithm is more polarizing
than a chronological one.

Nevertheless, it is unclear whether ranking algorithms have lasting effects on
users’ attitudes and affective polarization. In contrast to our content-
specific surveying, Guess et al. (2023) measured the effect of the Facebook
and Instagram ranking algorithms on users’ affective polarization, in general,
between two weeks to three months after the start of their experiment (?).
They found no significant impact on users’ affective polarization. Crucially,
however, the scope of Guess et al. (2023)’s research does not encompass
“general equilibrium” effects, such as the possibility for ranking algorithms
to indirectly shape user attitudes by incentivizing and changing the type of
content that users produce in the first place.

###### Stated preference.

We found that users had only a slight preference for tweets in their
engagement-based timeline compared to tweets in their reverse chronological
timeline. This small preference is notable because Twitter’s algorithm has a
large effect on how much time users spend on the platform (?, ?). Moreover,
users were _less_ likely to prefer the political tweets that the algorithms
selected. This suggests that the engagement-based algorithm may be catering to
users’ _revealed preferences_ but not to their _stated preferences_ ,
especially when it comes to political content.

####  2.2 Ranking by stated preferences

The engagement-based algorithm amplified divisive content and failed to meet
users’ stated preferences. This observation led us to question: what would the
implications be of ranking content by users’ stated preferences instead?

###### The effects of ranking by stated preference.

In exploratory analysis, we evaluated the effects of creating an alternative
ranking based on users’ stated preference (SP). We found that the SP timeline
significantly reduced the prevalence of angry content, partisan content, and
content that expressed out-group animosity, relative to the engagement-based
timeline. This suggests that part of the reason current algorithms amplify
divisiveness is their prioritization of engagement over more reflective
notions of preference.

However, delving deeper, we also found that the decrease in partisan tweets
predominantly came from a reduction in out-group tweets. Furthermore, the drop
in out-group animosity was primarily due to reducing hostility towards the
readers’ in-group (but not towards their out-group). This suggests that while
explicitly stated preferences might lower overall animosity, they could
inadvertently heighten users’ exposure to content that reinforces their pre-
existing beliefs. This is in line with prior research suggesting that users’
preference for pro-attitudinal content and sources may be a primary driver of
echo chambers online (?, ?, ?).

An important caveat in interpreting our results is that we only had a pool of
approximately twenty tweets—the top ten in the users’ chronological and
engagement timelines—to choose from when ranking by stated preferences. The
platform has access to a much larger pool of tweets, and more research is
needed to understand the impact of ranking by stated preferences in such a
context. For example, users might be open to seeing respectful and civil
content from their political out-group, but those tweets may not have been
common in our limited pool of tweets—especially since about half were
explicitly selected using engagement metrics.

###### Trade-offs between ranking approaches.

Given the vast amount of content available for ranking that algorithms can
choose from, relative to the amount of content that an individual user can
view, we hypothesize that it may be possible to do well _enough_ with respect
to all three of these desiderata: to achieve high user engagement, satisfy
users’ stated preferences, and avoid amplifying hostile and polarizing
content. In Section S4.9, we show that a slight preference to the stated
preference timeline, one which involves tie-breaking content ranking based on
the presence of out-group animosity, satisfies users’ stated preferences as
much as the SP timeline, but greatly reduces the amplification of divisive
content and avoids reinforcement of echo chambers. However, more research is
needed to understand the trade-offs inherent between all three of these
desiderata, especially with respect to engagement.

Finally, in terms of the practicalities of content ranking, it is important to
note that we do not advocate for scrapping the use of engagement. Compared to
user surveys, which are sparse and potentially costly to collect, engagement
data is readily available to platforms and still provides _some_ signal about
what users value. Engagement will likely be an important part of any content
ranking algorithm (?). But there may be smarter ways to combine dense
engagement signals with sparser survey data. For example, Milli et al. (2021)
created a latent variable model that learned to interpret different engagement
signals in a way that remained consistent with users’ stated preferences (?).

####  2.3 Limitations

Our study was designed to be carefully controlled. The only difference between
the two conditions in our experiment was whether the content was ranked
reverse-chronologically or through the personalized algorithm. This allowed us
to obtain precise estimates of the (local) effect of the algorithmic ranking.
However, such an approach does not capture the algorithm’s long-term effects.
For example, by incentivizing some types of content over others, the algorithm
changes what type of content is produced in the first place, which, in turn,
affects what is available in the chronological timeline. If users have learned
to produce more of the content that the algorithm incentivizes (?), this
implies that the algorithm’s long-term effects on the content-based outcomes
we measure (emotions, partisanship, and out-group animosity in tweets) may be
even greater. Nevertheless, rigorous experimentation needs to be done to
establish how all the effects found in our study change over time in response
to feedback loops (?).

Another concern pertains to the ecological validity of our study. We only
surveyed participants about the first ten tweets in each timeline, a method
that might not perfectly mimic real-world Twitter usage where users can scroll
past various tweets. Looking at the first ten tweets might be a reasonable
approximation though—Bouchaud et al. (2023) found that users viewed an average
of 30 tweets in one session of using Twitter (?). In Section S4.5, we show
that our results are robust to using a range of thresholds, spanning from the
first five tweets to the first ten tweets.

Finally, we required participants to install a Chrome extension and relied on
an online crowd-working platform for participant recruitment. This may have
impacted the types of participants we could attract. In particular, compared
to Twitter users in the 2020 ANES study (?), our population tended to be
younger (53% of our study are aged 18-34 years old, compared to 45% in the
ANES study) and more likely to affiliate with the Democratic Party (56%
Democrat in our study versus 41% in the ANES study) (see SM section S2 for
full demographic statistics). We report on heterogenous effects by different
demographic groups in Section S4.7, however, more research may be warranted
before generalizing to the full Twitter population.

### References

  * 1. W. J. Brady, J. A. Wills, J. T. Jost, J. A. Tucker, J. J. Van Bavel, Emotion shapes the diffusion of moralized content in social networks, Proceedings of the National Academy of Sciences 114, 7313–7318 (2017).
  * 2. J. W. Burton, N. Cruz, U. Hahn, Reconsidering evidence of moral contagion in online social networks, Nature Human Behaviour 5, 1629–1635 (2021).
  * 3. W. J. Brady, J. J. Van Bavel, Estimating the effect size of moral contagion in online networks: A pre-registered replication and meta-analysis [Preprint] (2021). https://doi.org/10.31219/osf.io/s4w2x.
  * 4. W. J. Brady, J. A. Wills, D. Burkart, J. T. Jost, J. J. Van Bavel, An ideological asymmetry in the diffusion of moralized content on social media among political leaders., Journal of Experimental Psychology: General 148, 1802 (2019).
  * 5. S. Vosoughi, D. Roy, S. Aral, The spread of true and false news online, Science 359, 1146–1151 (2018).
  * 6. A. Hasell, Shared Emotion: The Social Amplification of Partisan News on Twitter, Digital Journalism 9, 1085–1102 (2021).
  * 7. The Economist, Twitter’s algorithm does not seem to silence conservatives, https://www.economist.com/graphic-detail/2020/08/01/twitters-algorithm-does-not-seem-to-silence-conservatives (August 1, 2020). Accessed on December 21, 2023.
  * 8. P. Bouchaud, D. Chavalarias, M. Panahi, Crowdsourced audit of Twitter’s recommender systems, Scientific Reports 13, 16815 (2023).
  * 9. M. Garz, J. Sörensen, D. F. Stone, Partisan Selective Engagement: Evidence from Facebook, Journal of Economic Behavior & Organization 177, 91–108 (2020).
  * 10. F. Giglietto, A. Valeriani, N. Righetti, G. Marino, Diverging patterns of interaction around news on social media: insularity and partisanship during the 2018 Italian election campaign, Information, Communication & Society 22, 1610–1629 (2019).
  * 11. J. C. Medina Serrano, O. Papakyriakopoulos, S. Hegelich, Dancing to the Partisan Beat: A First Analysis of Political Communication on Tiktok, Proceedings of the 12th ACM Conference on Web Science (2020), pp. 257–266.
  * 12. J. Maitra, R. Hänggli, Who speaks and who is heard on Facebook? Political mobilization and engagement patterns of partisanship and gender in Switzerland’s direct democracy, Journal of Quantitative Description: Digital Media 3 (2023).
  * 13. M. Wojcieszak, A. Casas, X. Yu, J. Nagler, J. A. Tucker, Most users do not follow political elites on Twitter; those who do show overwhelming preferences for ideological congruity, Science Advances 8, eabn9418 (2022).
  * 14. J. Bandy, N. Diakopoulos, More Accounts, Fewer Links: How Algorithmic Curation Impacts Media Exposure in Twitter Timelines, Proceedings of the ACM on Human-Computer Interaction 5, 1–28 (2021).
  * 15. F. Huszár, S. I. Ktena, C. O’Brien, L. Belli, A. Schlaikjer, M. Hardt, Algorithmic amplification of politics on Twitter, Proceedings of the National Academy of Sciences 119, e2025334119 (2022).
  * 16. E. Bakshy, S. Messing, L. A. Adamic, Exposure to ideologically diverse news and opinion on Facebook, Science 348, 1130–1132 (2015).
  * 17. A. M. Guess, N. Malhotra, J. Pan, P. Barberá, H. Allcott, T. Brown, A. Crespo-Tenorio, D. Dimmery, D. Freelon, M. Gentzkow, et al., How do social media feed algorithms affect attitudes and behavior in an election campaign?, Science 381, 398–404 (2023).
  * 18. S. Rathje, J. J. Van Bavel, S. Van Der Linden, Out-group animosity drives engagement on social media, Proceedings of the National Academy of Sciences 118, e2024292118 (2021).
  * 19. X. Yu, M. Wojcieszak, A. Casas, Partisanship on Social Media: In-Party Love Among American Politicians, Greater Engagement with Out-Party Hate Among Ordinary Users, Political Behavior pp. 1–26 (2023).
  * 20. L. Doroshenko, F. Tu, Like, Share, Comment, and Repeat: Far-right messages, Emotions, and Amplification in Social Media, Journal of Information Technology & Politics 20, 286–302 (2023).
  * 21. J. Berger, Arousal Increases Social Transmission of Information, Psychological science 22, 891–893 (2011).
  * 22. J. B. Walther, Z. Lew, A. L. Edwards, J. Quick, The effect of social approval on perceptions following social media message sharing applied to fake news, Journal of Communication 72, 661–674 (2022).
  * 23. C. Jia, M. S. Lam, M. C. Mai, J. Hancock, M. S. Bernstein, Embedding Democratic Values into Social Media AIs via Societal Objective Functions, arXiv:2307.13912 (2023).
  * 24. P. Khambatta, S. Mariadassou, J. Morris, S. C. Wheeler, Tailoring recommendation algorithms to ideal preferences makes users better off, Scientific Reports 13, 9325 (2023).
  * 25. L. Yang, M. Sobolev, Y. Wang, J. Chen, D. Dunne, C. Tsangouri, N. Dell, M. Naaman, D. Estrin, How Intention Informed Recommendations Modulate Choices: A Field Study of Spoken Word Content, The World Wide Web Conference (2019), pp. 2169–2180.
  * 26. R. Levy, Social Media, News Consumption, and Polarization: Evidence from a Field Experiment, American Economic Review 111, 831–870 (2021).
  * 27. J. J. Van Bavel, S. Rathje, E. Harris, C. Robertson, A. Sternisko, How social media shapes polarization, Trends in Cognitive Sciences 25, 913–916 (2021).
  * 28. J. Haidt, C. Bail, Social Media and Political Dysfunction: A Collaborative Review, https://tinyurl.com/PoliticalDysfunctionReview (August 28, 2023). Accessed on December 21, 2023.
  * 29. M. D. Ekstrand, M. C. Willemsen, Behaviorism is Not Enough: Better Recommendations through Listening to Users, Proceedings of the 10th ACM Conference on Recommender Systems (2016), pp. 221–224.
  * 30. J. Stray, A. Halevy, P. Assar, D. Hadfield-Menell, C. Boutilier, A. Ashar, L. Beattie, M. Ekstrand, C. Leibowicz, C. M. Sehat, et al., Building Human Values into Recommender Systems: An Interdisciplinary Synthesis, arXiv:2207.10192 (2022).
  * 31. J. Stray, R. Iyer, H. P. Larrauri, The Algorithmic Management of Polarization and Violence on Social Media, https://knightcolumbia.org/content/the-algorithmic-management-of-polarization-and-violence-on-social-media (August 22, 2023). Accessed on December 21, 2023.
  * 32. J. Kleinberg, S. Mullainathan, M. Raghavan, The Challenge of Understanding What Users Want: Inconsistent Preferences and Engagement Optimization, Proceedings of the 23rd ACM Conference on Economics and Computation (2022), pp. 29–29.
  * 33. T. Cunningham, Ranking by engagement, https://tecunningham.github.io/posts/2023-04-28-ranking-by-engagement.html (May 8, 2023). Accessed on December 21, 2023.
  * 34. OpenAI, GPT-4 Technical Report, arXiv:2303.08774 (2023).
  * 35. E. Pariser, The Filter Bubble: How The New Personalized Web is Changing What We Read and How We Think (Penguin, 2011).
  * 36. S. Milli, L. Belli, M. Hardt, Causal Inference Struggles with Agency on Online Platforms, Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (2022), pp. 357–365.
  * 37. J. Bandy, T. Lazovich, Exposure to Marginally Abusive Content on Twitter, Proceedings of the International AAAI Conference on Web and Social Media (2023), vol. 17, pp. 24–33.
  * 38. S. González-Bailón, D. Lazer, P. Barberá, M. Zhang, H. Allcott, T. Brown, A. Crespo-Tenorio, D. Freelon, M. Gentzkow, A. M. Guess, et al., Asymmetric ideological segregation in exposure to political news on Facebook, Science 381, 392–398 (2023).
  * 39. P. Bengani, J. Stray, L. Thorburn, What’s right and what’s wrong with optimizing for engagement, https://medium.com/understanding-recommenders/whats-right-and-what-s-wrong-with-optimizing-for-engagement-5abaac021851 (April 27, 2022). Accessed on December 21, 2023.
  * 40. S. Milli, L. Belli, M. Hardt, From Optimizing Engagement to Measuring Value, Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (2021), pp. 714–722.
  * 41. W. J. Brady, J. C. Jackson, B. Lindström, M. Crockett, Algorithm-mediated social learning in online social networks [Preprint] (2023). https://doi.org/10.31219/osf.io/yw5ah.
  * 42. L. Thorburn, J. Stray, P. Bengani, From “Filter Bubbles”, “Echo Chambers”, and “Rabbit Holes” to “Feedback Loops”, https://techpolicy.press/from-filter-bubbles-echo-chambers-and-rabbit-holes-to-feedback-loops/ (April 17, 2023). Accessed on December 21, 2023.
  * 43. American National Election Studies, ANES 2020 Time Series Study Full Release, www.electionstudies.org (2021). Accessed on December 21, 2023.
  * 44. Y. Benjamini, A. M. Krieger, D. Yekutieli, Adaptive linear step-up procedures that control the false discovery rate, Biometrika 93, 491–507 (2006).
  * 45. M. L. Anderson, Multiple Inference and Gender Differences in the Effects of Early Intervention: A Reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects, Journal of the American Statistical Association 103, 1481–1495 (2008).
  * 46. A. Ovadya, Can Algorithmic Recommendation Systems Be Good For Democracy? (Yes! & Chronological Feeds May Be Bad), https://www.belfercenter.org/publication/can-algorithmic-recommendation-systems-be-good-democracy-yes-chronological-feeds-may-be (February 4, 2022). Accessed on December 21, 2023.
  * 47. Z. Schiffer, C. Newton, Yes, Elon Musk created a special system for showing you all his tweets first, https://www.platformer.news/p/yes-elon-musk-created-a-special-system (February 14, 2023). Accessed on December 21, 2023.
  * 48. @elonmusk, Your article is false and obviously so. Do you really do no research at all? I mean, like reading a few tweets, for example., https://twitter.com/elonmusk/status/1626556852816470024 (2023). Tweeted on February 17, 2023, 4:19 AM [PT]. Accessed on September 8, 2023.

#### Acknowledgements

We thank Katy Glenn Bass, Fabian Baumann, William Brady, Molly Crockett, Tom
Cunningham, Ravi Iyer, Philipp Lorenz-Spreen, Solomon Messing, Aviv Ovadya,
Sara Johansen, Arvind Narayanan, Jonathan Stray, Luke Thorburn, and Nina Vasan
for their discussion and feedback.

Funding. This study was financially supported by the UC Berkeley Center for
Human-Compatible AI. MC was supported by the National Science Foundation
Graduate Research Fellowship Program.

Author contributions.

Conceptualization: SM

Methodology: SM, MC

Software: SM, MC, YW, SP, SZ

Formal analysis: SM, MC, YW, SP, SZ

Investigation: SM, MC, YW, SP

Data curation: SM, MC, YW, SP, SZ

Writing - original draft: SM, MC, YW, SP, SZ

Writing - review & editing: SM, MC, YW, SP, SZ, AD

Supervision: SM, AD

Project administration: SM, AD

Funding acquisition: SM, AD

Competing interests. The authors declare that they have no competing
interests.

Data and materials availability. The data and code needed to reproduce our
results are available to the research community at
http://github.com/smilli/twitter.

##

##  Supplementary Materials

###  Supplementary Materials S1 Materials and methods

####  S1.1 Study procedure

We conducted our study between February 11 to February 27, 2023 on
CloudResearch Connect, a crowd-working platform. Informed consent was obtained
from all participants. The study was approved by UC Berkeley’s IRB under the
CPHS protocol ID number 2021-09-14618 and complies with all ethical
regulations. Moreover, at the end of the study period, our rating (given by
study participants) on CloudResearch was 4.9 stars, higher than 99 percent of
other researchers on the platform.

The study period was broken into four waves and participants could complete
the study once during each wave. The time periods for the waves (inclusive)
were 02/11-02/14, 02/16-02/19, 02/21-02/23, and 02/25-02/27. Every day, we
recruited up to 150 eligible participants who lived in the United States, were
at least 18 years old, and used Google Chrome. Furthermore, participants were
required to use Twitter at least a few times a week and follow at least 50
people on Twitter (both gauged through self-reports). To collect data,
participants were directed to download a Chrome extension that we developed
which scraped their Twitter homepage to collect the top tweets from their
personalized timeline. While scraping, the Chrome extension added an overlay
to the homepage that prevented the user from seeing the tweets during
collection. At the same time that the personalized tweets were collected, we
queried the Twitter API to get the top tweets from the chronological timeline.
Only public tweets were collected and no promoted tweets (advertisements) were
collected.

After collecting both sets of tweets, participants were directed to complete a
survey on Qualtrics that asked questions about each of the top ten tweets from
their personalized and chronological timeline. All tweets are displayed in a
randomized order (thus, tweets from both timelines are typically interwoven
rather than, for example, first showing all the personalized and then all
chronological tweets). If the same tweet was present in both the personalized
and chronological timeline, then participants were only shown it once. Out of
the 10 tweets in both timelines, on average, 2.35 of these tweets were common
to both of the timelines. On each question, the tweet is displayed for
reference as an embedded tweet (see Figure 3), so it looks as similar as
possible to the way it would on Twitter. If a tweet is a reply to another
tweet, the user is shown both the replied tweet and the main tweet, and asked
to answer the questions for both tweets. Similarly, if a tweet is a quote
tweet, then the user is asked to answer the questions for both the quoted
tweet and the main tweet.

The outcomes we measured concerned the emotions expressed by the author (on
four axes: anger, sadness, anxiety, and happiness), the reader emotions (on
four axes: anger, sadness, anxiety, and happiness), the author’s expression of
out-group animosity, the partisan leaning of the tweet, the readers’ in-group
and out-group perception after reading the tweet, and readers’ explicit
preference for the tweet. The full survey is provided in SM section S5, but
the question phrasings and their possible answers are provided below for
quicker reference:

  * •

Author emotions questions. Participants are asked: “How is [@author-handle]
feeling in their tweet?” For each emotion (angry, anxious, happy, sad), the
participant responds on a Likert scale of “Not at all,” “Slightly,”
“Somewhat,” “Moderately,” or “Extremely.”

  * •

Reader emotions. Participants are asked: “How did [@author-handle]’s tweet
make you feel?” For each emotion (angry, anxious, happy, sad), the participant
responds on a Likert scale of “Not at all,” “Slightly,” “Somewhat,”
“Moderately,” or “Extremely.”

  * •

Is tweet political. Participants are asked: “Is [@author-handle]’s tweet about
a political or social issue?” with a binary response of “Yes” or “No”

  * •

Political leaning. Participants are asked: “How does [@author-handle]’s tweet
lean politically?” Participants respond on a Likert scale of “Far left,”
“Left,” “Moderate,” “Right,” or “Far right.”

  * •

Political affect Left. Participants are asked: “How does [@author-handle]’s
tweet make you feel about people or groups on the Left?” Participants respond
on a Likert scale of “Much worse,” “Worse,” “The same as before,” “Better,”,
or “Much better.”

  * •

Political affect Right. Participants are asked: “How does [@author-handle]’s
tweet make you feel about people or groups on the Right?” Participants respond
on a Likert scale of “Much worse,” “Worse,” “The same as before,”, “Better,”
or “Much better,”

  * •

Outgroup animosity towards the left. Participants are asked: “Is [@author-
handle]’s tweet expressing anger, frustration, or hostility towards a person
or group on the Left?” Participants choose between a response of “Yes” or “No”
This question is only asked if political leaning is “Right” or “Far right.”

  * •

Outgroup animosity towards the right. Participants are asked: “Is [@author-
handle]’s tweet expressing anger, frustration, or hostility towards a person
or group on the Right?” Participants choose between a response of “Yes” or
“No.” This question is only asked if political leaning is “Left” or “Far
left.”

  * •

Value. Participants are asked: “When you use Twitter, do you want to be shown
tweets like [@author-handle]’s tweet?” Participants choose between “No,”
“Indifferent,” “Yes.”

Data exclusions. Those who passed the pre-screen but did not complete data
collection with the Chrome extension were paid $0.75, while those who
completed data collection and the full survey (which takes approximately 30
minutes) were paid $10 (a rate of $20/hr). Overall, 18 percent of participants
who consented to the experiment did not complete data collection with the
Chrome extension. This was either because they chose to not use the Chrome
extension or ran into an error during data collection. Since the Chrome
extension relies on scraping Twitter’s homepage to retrieve tweets it may not
work if, for example, the user has conflicting Chrome extensions or is in a
Twitter A/B test that changes the UI of the Twitter homepage. Furthermore, we
also surveyed participants about two attention-check tweets (that changed
during each of the four waves) that were unambiguously either left-leaning or
right-leaning. If a participant’s survey response did not pass the attention
checks, it was excluded from the final data set used for analysis.
Furthermore, if a participant did not successfully complete the survey (e.g.
did not complete data collection or pass the attention checks) in one wave,
then they were excluded from participating in future waves. Our final data set
consisted of 1730 responses from 806 unique participants. Table 3 shows
statistics on the attrition of participants from consenting to passing the
attention checks.

Figure 3: Sample view of the survey: users saw tweets embedded alongside each
question for reference.

####  S1.2 Pre-registration and deviations

All our hypotheses, outcome measures, and statistical analyses were pre-
registered on Open Science Framework (OSF) at https://osf.io/upw9a. However,
we made the following deviations from the pre-registration. First, we opted to
use CloudResearch Connect instead of CloudResearch’s Mechanical Turk toolbelt
for participant recruitment because CloudResearch Connect had features better
suited for our study, such as the ability to flag our study as requiring a
software download. Second, due to limitations in the number of unique Twitter
users we could recruit on CloudResearch Connect, we divided our data
collection period into four intervals and allowed participants to complete the
study once in each interval. Third, we underestimated the study’s costs,
resulting in us stopping data collection when we reached our budget (at around
1700 pairs of timelines) instead of our initial target of 2000 pairs of
timelines. And finally, we asked participants to rate out-group animosity on a
five-point Likert scale from “Not at all” to “Extremely” for main tweets but
accidentally left the response options for quoted and replied tweets as binary
“Yes” or “No” options. Thus, for analysis, we collapse the Likert responses to
a binary scale where “Not at all” corresponds to “No” and anything higher
corresponds to “Yes.”

Times Participant Completed Survey | Number of Unique Participants
---|---
1 | 359
2 | 139
3 | 139
4 | 169
Total | 806
Table 2: The table displays the distribution of the number of times that each participant took part in the study. Most participants only participated once, but many participated two or more times. Wave | Consented | Collected Data | Completed Survey | Passed Attention Check
---|---|---|---|---
1 | 816 | 599 | 586 | 508
2 | 688 | 585 | 574 | 454
3 | 514 | 440 | 434 | 405
4 | 486 | 441 | 432 | 363
Total | 2504 | 2065 | 2026 | 1730
Table 3: The table displays, for each study period, the number of participants
who consented to the study, successfully completed the data collection through
the Chrome extension, completed the survey, and passed the attention check.

####  S1.3 Estimation of average treatment effects

As specified in our pre-registered analysis plan, our main estimation of the
average treatment effect is through a simple difference in mean outcomes
between the personalized and chronological timelines. Let
$\mathcal{U}=\\{1,2,\dots,n\\}$ be the set of participants and
$\mathcal{W}_{u}\subseteq\\{1,2,3,4\\}$ be the set of study waves that a user
$u\in\mathcal{U}$ participates in. During study wave $w\in\mathcal{W}_{u}$,
participant $u$ rates both a set of tweets $\mathcal{T}_{w,u}(1)$ from their
engagement-based timeline (denoted as timeline $1$) and a set of tweets
$\mathcal{T}_{w,u}(0)$ from their chronological timeline (denoted as timeline
$0$). The number of tweets that each participant rates is approximately ten
from each timeline, i.e., $|\mathcal{T}_{w,u}(1)|\approx 10$ and
$|\mathcal{T}_{w,u}(0)|\approx 10$. However, this number can be higher if a
timeline contains quote tweets or replies. For both quote tweets and replies,
participants rate two tweets: the quote tweet and the tweet being quoted or
the reply and the tweet being replied to.

Let $Y_{w,u,t,q}(x)$ be the response of participant $u$ to question $q$ on
tweet $t\in\mathcal{T}_{w,u}(x)$ from timeline $x\in\\{0,1\\}$ in study wave
$w$. Certain questions (partisanship, out-group animosity, in-group and out-
group perception) are only asked if a participant labeled the tweet as being a
political tweet. If the participant did not label the tweet as being
political, then we let $Y_{w,u,t,q}(x)=0$ for those questions. This
corresponds to assuming that the tweet does not have a partisan leaning, does
not contain out-group animosity, and did not impact the readers’ perception of
their in-group or out-group.

Then, let

$\displaystyle\widebar{Y}_{u,q}(x)=\sum_{w\in\mathcal{W}_{u}}\frac{1}{|\mathcal{W}_{u}|}\sum_{t\in\mathcal{T}_{w,u}(x)}\frac{Y_{w,u,t,q}(x)}{|\mathcal{T}_{w,u}(x)|}$
(1)

be the mean outcome on question $q$ across the tweets in timeline $x$ for
individual $u$. The individual effect for participant $u$ on question $q$ is
estimated as $\widebar{Y}_{u,q}(1)-\widebar{Y}_{u,q}(0)$. Finally, the average
treatment effect on question $q$ across participants is estimated as

$\displaystyle\widehat{ATE}(q)=\frac{\sum_{u}\widebar{Y}_{u,q}(1)-\widebar{Y}_{u,q}(0)}{|\mathcal{U}|}\,.$
(2)

In addition to considering tweets overall, we also look at the effect on
authors’ emotions, readers’ emotions, and the users’ stated preference for
political tweets only. Let $\mathcal{T}^{\text{pol}}_{w,u}(x)$ be the tweets
that participant $u$ labeled as being political in study wave $w$ and timeline
$x$. Furthermore, let $\mathcal{W}_{u}^{\text{pol}}$ be the set of waves in
which participant $u$ had at least one political tweet in both the
chronological and engagement timeline. Then, let

$\displaystyle\widebar{Y}^{\text{pol}}_{u,q}(1)=\sum_{w\in\mathcal{W}_{u}^{\text{pol}}}\frac{1}{|\mathcal{W}_{u}^{\text{pol}}|}\sum_{t\in\mathcal{T}^{\text{pol}}_{w,u}(x)}\frac{Y_{w,u,t,q}(x)}{|\mathcal{T}^{\text{pol}}_{w,u}(x)|}$
(3)

be the participant’s mean outcome for question $q$ when considering only
political tweets. Then, the average treatment effect for political tweets only
is

$\displaystyle\widehat{ATE}_{\text{pol}}(q)=\frac{\sum_{u\in\mathcal{U}_{\text{pol}}}\widebar{Y}^{\text{pol}}_{u,q}(1)-\widebar{Y}^{\text{pol}}_{u,q}(0)}{|\mathcal{U}_{\text{pol}}|}\,,$
(4)

where $\mathcal{U}_{\text{pol}}$ is the set of participants with at least one
political tweet in each timeline.

Significance and multiple testing. To compute $p$-values, we conducted two-
tailed paired permutation tests. A participant’s responses to their
personalized tweets and responses to their chronological tweets are considered
paired data, i.e., $\widebar{Y}_{u,q}(1)$ and $\widebar{Y}_{u,q}(0)$, or
$\widebar{Y}^{\text{pol}}_{u,q}(1)$ and $\widebar{Y}^{\text{pol}}_{u,q}(0)$
are considered paired data. We used bootstrap sampling to calculate the 95%
confidence intervals displayed in Figure 1. To adjust for the possibility of
chance findings when conducting multiple tests, we used the Benjamini-Krieger-
Yekutieli two-stage method (?), as described by (?), to compute false
discovery rate (FDR) adjusted $p$-values. The FDR-adjusted $p$-value
(sometimes called a _$q$ -value_) for each test is the lowest false discovery
rate such that the test would still be rejected at a significance level of
$0.05$. We test 26 outcomes in total, and all significant results (significant
at a $p$-value threshold of $0.05$) remain significant at a false discovery
rate of $0.01$. Thus, on expectation, none of our discoveries are false
discoveries. The full table of standardized and unstandardized effect sizes,
$p$-values, and FDR-adjusted $p$-values can be found in SM section S3.

Effect size standardization. When displaying the effect sizes for all outcomes
in Figure 1, we standardize using the standard deviation of the outcome in the
chronological timeline, i.e., $\sigma(\widebar{Y}_{u,q}(0))$. Note that we do
not use the pooled standard deviation between the chronological and
engagement-based timeline as would be done in Cohen’s $d$. The standardized
effect size we present is also known as Glass’s $\Delta$ and uses the standard
deviation of the control group in order to allow comparison between multiple
treatments, i.e., not just Twitter’s engagement-based algorithm but also our
alternative ranking algorithm based on stated preferences. One exception is
that when calculating the standardized effect sizes for author emotion, reader
emotion, and explicit value for _political_ tweets only, we standardize the
effect using the standard deviation of that outcome across _all_ tweets. This
was done to ensure that the effect sizes found when restricting to political
tweets could be compared with the effect sizes observed for all tweets, i.e.,
we use $\sigma(\widebar{Y}_{u,q}(0))$ instead of
$\sigma(\widebar{Y}^{\text{pol}}_{u,q}(0))$.

###  Supplementary Materials S2 User-level survey questions and demographics

In this section, we report on the demographic distribution of our participants
and compare our distribution to that of Twitter users from the 2020 ANES study
(?). The primary differences are that our population is younger (53 percent of
our study are aged 18-34 years old, compared to 45 percent in the ANES study)
and more likely to affiliate with the Democratic Party (56 percent Democrat in
our study versus 41 percent in the ANES study).

####  S2.1 Demographics of our study

Users could participate in our study up to four different times, and we
collected demographic information from them each time. In total, 806 unique
users participated in our study 1730 times. The following tables report
demographics across all users, with each user’s demographic responses being
given a weight of $1/m$, where $m$ represents the number of survey waves they
participated in. In addition to demographic questions, we also asked two user-
level survey questions about the primary reason participants use Twitter and
the primary type of content they saw in the tweets we showed them (Table 9).

Race (Our Study) | Weight | Percent
---|---|---
White | 538.23 | 66.78
Black or African American | 88.23 | 10.95
Hispanic | 80.24 | 9.96
Asian or Native Hawaiian/other Pacific Islander | 62.51 | 7.76
American Indian/Alaska Native or Other | 8.84 | 1.10
Multiple races, non-Hispanic | 27.94 | 3.47
Total | 806 | 100
Table 4: We ask two separate questions about race and ethnicity to
participants, and for comparison, we combine them together in the same way
that the 2020 ANES study does.

Gender (Our Study) | Weight | Percent
---|---|---
Man | 410.26 | 50.90
Woman | 371.00 | 46.03
Non-binary | 20.78 | 2.58
Other | 3.97 | 0.49
Total | 806 | 100

Ideological Leaning (Our Study) | Weight | Percent
---|---|---
Far left | 168.87 | 20.95
Left | 325.56 | 40.39
Moderate | 195.31 | 24.23
Right | 85.86 | 10.65
Far right | 25.29 | 3.14
Other | 5.12 | 0.63
Total | 806 | 100

Political Leaning Further (Our Study) | Weight | Percent
---|---|---
Left-leaning | 126.14 | 62.94
Right-leaning | 74.28 | 37.06
Total | 200.42 | 100
Table 5: Those who identified as being moderate were asked whether they leaned more toward the Left/Right. Summary Leaning (Our Study) | Weight | Percent
---|---|---
Left-leaning | 620.57 | 76.99
Right-leaning | 185.43 | 23.01
Total | 806 | 100
Table 6: Participants who selected “Moderate” or “Other” in response to the
ideological leaning question were asked a follow-up question about whether
they lean towards left or right more as of today. The table shows aggregate
counts of those who identified as being on the Left/Right or that they leaned
more towards the Left/Right.

Political Party (Our Study) | Weight | Percent
---|---|---
Democrat | 452.51 | 56.14
Republican | 111.08 | 13.78
Independent | 204.42 | 25.36
Something else | 37.99 | 4.71
Total | 806 | 100

Political Party Further (Our Study) | Weight | Percent
---|---|---
Republican | 69.94 | 28.85
Democrat | 172.46 | 71.15
Total | 242.40 | 100
Table 7: Participants who selected “Independent” or “Something else” in response to the political party question were asked a follow-up question about whether they lean towards the Democrat or Republican party more as of today. Summary Party (Our Study) | Weight | Percent
---|---|---
Republican | 181.03 | 22.46
Democrat | 624.97 | 77.54
Total | 806 | 100
Table 8: Participants who selected “Independent” or “Something else” in
response to the political party question were asked a follow-up question about
whether they lean towards the Democrat or Republican party more as of today.
The table shows aggregate counts of those who said they were
Democrat/Republican or lean Democrat/Republican.

Education Levels (Our Study) | Count | Percent
---|---|---
Some high school | 5.82 | 0.72
High school graduate | 241.91 | 30.01
Associate degree | 103.44 | 12.83
Bachelor’s degree | 311.18 | 38.61
Master’s degree or above | 127.67 | 15.84
Other | 14.86 | 1.84
Prefer not to answer | 1.12 | 0.14
Total | 806 | 100

Age Group (Our Study) | Weight | Percent
---|---|---
18-24 years old | 119.33 | 14.81
25-34 years old | 307.24 | 38.12
35-44 years old | 211.74 | 26.27
45-54 years old | 92.68 | 11.50
55-64 years old | 50.33 | 6.24
65-74 years old | 23.67 | 2.94
75 years or older | 1.00 | 0.12
Total | 806 | 100

Annual Household Income (Our Study) | Weight | Percent
---|---|---
Less than $25,000 | 117.40 | 14.57
$25,000-$50,000 | 189.92 | 23.56
$50,000-$100,000 | 301.26 | 37.38
$100,000-$200,000 | 147.06 | 18.25
More than $200,000 | 34.40 | 4.27
Prefer not to say | 15.95 | 1.98
Total | 806 | 100

Primary Category of Content (Our Study) | Weight | Percent
---|---|---
Entertainment | 424.54 | 29.24
Other | 103.86 | 7.15
Politics | 493.07 | 33.96
News | 268.71 | 18.51
Hobbies | 148.81 | 10.25
Work | 13.02 | 0.90
Total | 1452 | 100
Table 9: At the end of the study, we ask participants about the content shown to them: “What were the tweets we showed you today predominantly about? Select a maximum of two.” Primary Reason for Using Twitter (Our Study) | Weight | Percent
---|---|---
A way to stay informed | 293.91 | 36.46
Entertainment | 397.41 | 49.31
Keeping me connected to other people | 48.16 | 5.98
It’s useful for my job or school | 28.18 | 3.50
Lets me see different points of view | 16.52 | 2.05
A way to express my opinions | 21.82 | 2.71
Total | 806 | 100
Table 10: We ask participants about the primary reason they use Twitter: “What
would you say is the main reason you use Twitter?”

####  S2.2 Demographics of Twitter users in ANES 2020 study

In this subsection, we present demographic distributions for the 1030
participants in the 2020 ANES study (?) who reported using Twitter at least a
few times a week (only those who reported using Twitter at least a few times a
week were allowed to participate in our study). We weighed each participant
using the relevant post-stratified weights provided by ANES. For certain
demographic questions, participants were not given the option to opt out in
our study, but they were in the ANES study. Thus, for those cases, we do not
include the ANES participants who opted out of the question.

Race (ANES Study) | Weight | Percent
---|---|---
White | 712.31 | 63.33
Black or African American | 98.15 | 8.73
Hispanic | 173.54 | 15.43
Asian or Native Hawaiian/other Pacific Islander | 69.51 | 6.18
American Indian/Alaska Native or Other | 21.92 | 1.95
Multiple races, non-Hispanic | 49.33 | 4.39
Total | 1125 | 100

Sex (ANES Study) | Weight | Percent
---|---|---
Male | 628.24 | 55.85
Female | 496.60 | 44.15
Total | 1125 | 100

Political Party (ANES Study) | Weight | Percent
---|---|---
Democrat | 468.69 | 41.64
Republican | 235.41 | 20.91
Independent | 393.78 | 34.99
Other | 27.67 | 2.46
Total | 1125 | 100

Education Levels (ANES Study) | Weight | Percent
---|---|---
Some high school | 38.93 | 3.45
High school graduate | 438.65 | 38.85
Associate degree | 90.81 | 8.04
Bachelor’s degree | 358.36 | 31.74
Master’s degree or above | 187.77 | 16.63
Other | 14.67 | 1.30
Prefer not to answer | 0.00 | 0.00
Total | 1129 | 100
Table 11: Distribution of educational attainment in the 2020 ANES Twitter population. For comparison purposes, we bin the ANES education categories, which are more fine-grained than ours, into our coarser categories. Age Group (ANES Study) | Weight | Percent
---|---|---
18-24 years old | 276.06 | 25.12
25-34 years old | 215.85 | 19.64
35-44 years old | 216.23 | 19.68
45-54 years old | 156.59 | 14.25
55-64 years old | 142.85 | 13.00
65-74 years old | 66.98 | 6.10
75 years or older | 24.41 | 2.22
Total | 1099 | 100
Table 12: Distribution of ages in the 2020 ANES Twitter population. For
comparison purposes, we group ages to match the age groups that we use in our
study.

####  S2.3 Comparison to ANES 2020 demographic data

Finally, we present figures comparing our user demographics to Twitter users
in the ANES 2020 who used Twitter at least a few times a week. The primary
differences are that our population is younger (53 percent of our study are
aged 18-34 years old, compared to 45 percent in the ANES study) and more
likely to affiliate with the Democratic Party (56 percent Democrat in our
study versus 41 percent in the ANES study).

Figure 4: Race of our study participants compared to Twitter users in the ANES
2020 study. We combine the race and ethnicity questions in our study into the
summary categories used by ANES, shown in the figure. Figure 5: Gender of our
study participants compared to the sex of Twitter users in the ANES 2020
study. Note that we collected gender but not sex and ANES collected sex but
not gender. As such, we can only compare Male/Female against Men/Women. Figure
6: Self-reported political party affiliations of our study participants
compared to that of Twitter users in the ANES 2020 study. Figure 7: The
highest education level achieved by our study participants compared to Twitter
users in the ANES 2020 study. We bin the ANES education categories, which are
more fine-grained than ours, into our coarser categories. Figure 8: Ages of
our study participants compared to Twitter users in the ANES 2020 study. We
bin ages from the ANES study to match our categories, shown in the figure.

###  Supplementary Materials S3 Pre-registered analysis

Outcome | Standardized | Unstandardized | Chron. | Eng. | $p$-value | Adjusted
---|---|---|---|---|---|---
Effect | Effect | Mean | Mean |  $p$-value
Emotional effects (all tweets)
Author Angry | 0.473 | 0.188 | 0.352 | 0.545 | 0.0002 | 0.0005
Author Sad | 0.220 | 0.077 | 0.293 | 0.378 | 0.0002 | 0.0005
Author Anxious | 0.232 | 0.100 | 0.391 | 0.492 | 0.0002 | 0.0005
Author Happy | 0.016 | 0.013 | 1.307 | 1.330 | 0.5125 | 0.1340
Reader Angry | 0.266 | 0.107 | 0.306 | 0.412 | 0.0002 | 0.0005
Reader Sad | 0.086 | 0.032 | 0.316 | 0.356 | 0.0032 | 0.0025
Reader Anxious | 0.119 | 0.051 | 0.346 | 0.398 | 0.0002 | 0.0005
Reader Happy | 0.119 | 0.085 | 0.941 | 1.030 | 0.0002 | 0.0005
Emotional effects (political tweets only)
Author Angry | 0.754 | 0.299 | 1.128 | 1.438 | 0.0002 | 0.0005
Author Sad | 0.309 | 0.108 | 0.688 | 0.782 | 0.0056 | 0.0035
Author Anxious | 0.175 | 0.075 | 0.824 | 0.873 | 0.0316 | 0.0129
Author Happy | -0.061 | -0.048 | 0.513 | 0.532 | 0.2138 | 0.0620
Reader Angry | 0.377 | 0.152 | 1.042 | 1.192 | 0.0008 | 0.0009
Reader Sad | 0.007 | 0.003 | 0.835 | 0.840 | 0.9363 | 0.2288
Reader Anxious | -0.002 | -0.001 | 0.867 | 0.863 | 0.9679 | 0.2288
Reader Happy | -0.015 | -0.011 | 0.421 | 0.471 | 0.7761 | 0.1929
Political effects
Partisanship | 0.244 | 0.051 | 0.151 | 0.202 | 0.0002 | 0.0005
Out-group Animosity | 0.236 | 0.031 | 0.085 | 0.116 | 0.0002 | 0.0005
In-group Perc. | 0.081 | 0.015 | 0.060 | 0.074 | 0.0014 | 0.0014
(all users) |  |  |  |  |  |
Out-group Perc. | -0.171 | -0.037 | -0.108 | -0.147 | 0.0002 | 0.0005
(all users) |  |  |  |  |  |
In-group Perc. | 0.052 | 0.009 | 0.056 | 0.066 | 0.0954 | 0.0329
(left users) |  |  |  |  |  |
Out-group Perc. | -0.157 | -0.031 | -0.097 | -0.132 | 0.0002 | 0.0005
(left users) |  |  |  |  |  |
In-group Perc. | 0.151 | 0.031 | 0.072 | 0.102 | 0.0012 | 0.0013
(right users) |  |  |  |  |  |
Out-group Perc. | -0.204 | -0.052 | -0.145 | -0.200 | 0.0002 | 0.0005
(right users) |  |  |  |  |  |
Reader Preference
Reader Pref | 0.065 | 0.023 | 0.507 | 0.526 | 0.0226 | 0.0097
(all tweets) |  |  |  |  |  |
Reader Pref | -0.180 | -0.065 | 0.519 | 0.465 | 0.0054 | 0.0035
(political tweets) |  |  |  |  |  |
Table 13: Our pre-registered analysis measuring the effects of Twitter’s
engagement-based timeline. The table shows the average treatment effects
(standardized and unstandardized), $p$-values, and FDR-adjusted $p$-values for
all 26 pre-registered outcomes. The way that all statistics are calculated is
described in Section S1.3.

###  Supplementary Materials S4 Exploratory analysis

Next, we describe additional, exploratory analyses that were not pre-
registered.

####  S4.1 Descriptive statistics: tweet metadata

Table 14 compares the metadata of the top 10 tweets in the engagement-based
and chronological timelines, pooled across all participants and waves. We
include both the mean and median for each property. The median may be a more
appropriate summary for attributes with extreme outliers, i.e., the tweet age
(the amount of time since the tweet was created), the author’s number of
followers, the number of likes, and the number of retweets. Histograms for
these properties are provided in Figure 9.

Unsurprisingly, the number of likes and retweets that each tweet has is much
higher in the engagement-based timeline. And as expected, the reader is less
likely to follow the authors of tweets shown by the engagement-based algorithm
compared to the chronological ranking. Interestingly, the average number of
links is almost halved, which is consistent with the results of a prior study
that used eight sock puppets to audit properties of Twitter’s algorithm (?).

Finally, the algorithm does not necessarily favor the most “popular” accounts:
authors shown by the algorithm have a lower median number of followers and are
less likely to be verified. It is important to clarify that our research was
conducted before Twitter’s changed its verification policy on April 1, 2023.
The new policy limits verification (indicated by a blue checkmark on Twitter
profiles) to Twitter Blue subscribers who pay for the service. Our data was
collected under the previous policy which required users to be “active”,
“notable”, and “authentic” to get verified status. Consequently, the verified
users in our data set tend to be official organizational accounts,
celebrities, journalists, or other well-known individuals.

Table 14: Summary of metadata in users’ engagement and chronological timelines

| Timeline | p5 | p50 | p95 | Avg. | SD | Diff
---|---|---|---|---|---|---|---
Tweet age (minutes) | Chronological | 3.8 | 92 | 2,937 | 7,812 | 126,667 |
| Engagement | 18.4 | 899 | 2,722 | 3,846 | 94,308 | p < 0.001
Number of likes | Chronological | 0 | 39 | 7,557 | 2,500 | 18,565 |
| Engagement | 2 | 744 | 83,658 | 16,912 | 62,158 | p < 0.001
Number of retweets | Chronological | 0 | 6 | 1,081 | 389 | 3,752 |
| Engagement | 0 | 68 | 8,875 | 1,787 | 13,415 | p < 0.001
Number of links in tweet | Chronological | 0 | 0 | 1 | 0.397 | 0.552 |
| Engagement | 0 | 0 | 1 | 0.105 | 0.326 | p < 0.001
Number of photos in tweet | Chronological | 0 | 0 | 2 | 0.457 | 0.755 |
| Engagement | 0 | 0 | 2 | 0.496 | 0.752 | p < 0.001
Number of videos in tweet | Chronological | 0 | 0 | 1 | 0.130 | 0.336 |
| Engagement | 0 | 0 | 1 | 0.184 | 0.387 | p < 0.001
Author’s # followers | Chronological | 500 | 139,874 | 25,745,780 | 3,905,313 | 12,059,120 |
| Engagement | 487 | 92,959 | 18,788,850 | 4,877,343 | 20,868,580 | p < 0.001
Is the author verified? | Chronological | 0 | 1 | 1 | 0.504 | 0.500 |
| Engagement | 0 | 0 | 1 | 0.391 | 0.488 | p < 0.001
Does the reader follow the author? | Chronological | 0 | 1 | 1 | 0.694 | 0.461 |
| Engagement | 0 | 1 | 1 | 0.569 | 0.495 | p < 0.001

The table compares the top ten tweets in the chronological and engagement-
based timelines. Both the mean and median are shown for each attribute. The
median is a more representative summary when the mean of the attribute is
dominated by extreme outliers. In gray, we have highlighted which summary
statistic, the mean or median, is more representative based on how long-tailed
the distribution is. Figure 9: The graphs compare the histogram of tweets in
the engagement-based and chronological timelines along four properties: the
author’s number of followers, the tweet’s number of likes, the tweet’s number
of retweets, and the tweet’s age. All graphs are plotted on a log-scale
$x$-axis because all properties have a long-tail of extreme outliers. The
dashed lines show the median value for each timeline.

####  S4.2 Amplification of individual accounts

In this section, we investigate which individual user accounts are the most
amplified or de-amplified by Twitter’s engagement-based algorithm.

First, in Table 15, we show the accounts that appear most frequently in our
study participants’ chronological and engagement timelines. The most common
accounts are quite different in the two timelines, with news outlets having
much greater dominance in the chronological timeline.

Next, in Table 16, we show which accounts were most and least amplified by
Twitter’s algorithm, where the amplification of an individual account is
simply calculated as the number of tweets by that author in users’ engagement-
based timelines minus the number of tweets by that author in users’
chronological timelines. Consistent with prior findings (?), the accounts that
were most _de-amplified_ were news outlets. This is likely because news
outlets post more frequently than ordinary accounts. Though the chronological
timeline is susceptible to this kind of spam, the engagement-based timeline is
not (?).

Elon Musk also stands out as a significant outlier, receiving higher
amplification than any other account by a wide margin. Notably, during our
study period, Platformer alleged that Twitter deployed code to artificially
boost the reach of Musk’s tweets (?). Though Musk denied such claims (?), our
data seems to suggest that he did receive an abnormally high amount of
amplification, relative to other accounts.

Chronological | Engagement
---|---
Top Accounts |  Count |  Top Accounts |  Count
@Reuters |  532 |  @elonmusk |  1387
@nytimes |  529 |  @POTUS |  295
@FatKidDeals |  407 |  @RonFilipkowski |  210
@Wario64 |  365 |  @JackPosobiec |  171
@CNN |  365 |  @fasc1nate |  152
@NBA |  280 |  @nytimes |  135
@Forbes |  280 |  @barstoolsports |  134
@FoxNews |  266 |  @DailyLoud |  133
@AP |  251 |  @hodgetwins |  126
@washingtonpost |  250 |  @CNN |  119
@POTUS |  206 |  @NBA |  115
@IGN |  204 |  @stillgray |  107
@videogamedeals |  185 |  @vidsthatgohard |  103
@TMZ |  179 |  @ClownWorld_ |  102
@elonmusk |  151 |  @catturd2 |  97
@business |  151 |  @BornAKang |  96
@TheOnion |  148 |  @krassenstein |  95
@barstoolsports |  128 |  @Wario64 |  93
@PlayStation |  127 |  @HumansNoContext |  90
@NFL |  126 |  @Dexerto |  84
@JackPosobiec |  123 |  @CalltoActivism |  84
@guardian |  122 |  @mmpadellan |  82
@Independent |  119 |  @NFL |  82
@ABC |  114 |  @joncoopertweets |  81
@nypost |  108 |  @jordanbpeterson |  79
Table 15: The most common accounts seen in our study participants’ chronological and engagement timelines. In both timelines, only the first ten tweets are considered. Most Amplified |  Eng-Chron Diff |  Least Amplified |  Eng-Chron Diff
---|---|---|---
@elonmusk |  1236.0 |  @Reuters |  -518.0
@RonFilipkowski |  144.0 |  @nytimes |  -394.0
@fasc1nate |  119.0 |  @Wario64 |  -272.0
@DailyLoud |  114.0 |  @Forbes |  -265.0
@hodgetwins |  102.0 |  @CNN |  -246.0
@vidsthatgohard |  97.0 |  @AP |  -214.0
@stillgray |  89.0 |  @FoxNews |  -198.0
@POTUS |  89.0 |  @washingtonpost |  -188.0
@BornAKang |  87.0 |  @NBA |  -165.0
@ClownWorld_ |  83.0 |  @videogamedeals |  -156.0
@krassenstein |  83.0 |  @IGN |  -148.0
@HumansNoContext |  80.0 |  @TMZ |  -148.0
@Dexerto |  73.0 |  @business |  -139.0
@rawsalerts |  68.0 |  @TheOnion |  -136.0
@joncoopertweets |  67.0 |  @guardian |  -117.0
@buitengebieden |  60.0 |  @Independent |  -108.0
@Acyn |  59.0 |  @PlayStation |  -107.0
@mmpadellan |  58.0 |  @TheEconomist |  -85.0
@EndWokeness |  58.0 |  @people |  -82.0
@CollinRugg |  58.0 |  @ABC |  -79.0
@historyinmemes |  57.0 |  @WSJ |  -79.0
@CalltoActivism |  53.0 |  @BBCWorld |  -69.0
@libsoftiktok |  52.0 |  @netflix |  -68.0
@FightHaven |  49.0 |  @HuffPost |  -65.0
@JackPosobiec |  48.0 |  @thehill |  -63.0
Table 16: The accounts that were most and least amplified in the engagement
timeline, relative to the chronological timeline. In both timelines, only the
first 10 tweets are considered.

####  S4.3 Distribution of responses to survey questions

In SM section S3, we reported our pre-registered results on the effects of the
engagement-based timeline, relative to the chronological timeline. Here, we
also additionally show how the full distribution of responses to the Likert
survey questions differs between users’ chronological and engagement-based
timelines.

#####  S4.3.1 Emotions

First, we show the distribution of survey responses for questions gauging the
author and reader’s emotions (sad, happy, anxious, and angry), considering
both the distribution for tweets overall and for only political tweets, in
Figure 10 and Tables 17-20. For further ease of interpretation, in Figure 11,
we also show the mean level of author and reader emotions in the chronological
and engagement-based timelines after binarizing responses. In particular, if a
reader responds “Not at all” to an emotion, that is coded as zero, while any
response between “Slightly” and “Extremely” is coded as one. Notably, there
was a large increase in tweets expressing anger between the engagement and
chronological timeline. In particular, 62 percent of political tweets in the
engagement timeline expressed anger, compared to 52 percent in the
chronological timeline.

Figure 10: Distribution of author and reader emotions in the chronological (C) and engagement-based (E) timelines Answer | Angry | Anxious | Happy | Sad
---|---|---|---|---
| % Chron | % Eng | % Chron | % Eng | % Chron | % Eng | % Chron | % Eng
Not at all (0) | 81.57 | 73.99 | 76.81 | 72.10 | 47.55 | 47.38 | 82.96 | 79.21
Slightly (1) | 7.80 | 9.28 | 12.20 | 13.94 | 10.85 | 11.04 | 8.50 | 10.01
Somewhat (2) | 4.79 | 7.09 | 6.61 | 7.99 | 14.18 | 13.72 | 4.67 | 5.78
Moderately (3) | 3.89 | 6.36 | 3.12 | 4.01 | 15.33 | 15.40 | 2.43 | 3.06
Extremely (4) | 1.95 | 3.28 | 1.26 | 1.96 | 12.08 | 12.47 | 1.44 | 1.95
Table 17: Author Emotions (Overall) Answer | Angry | Anxious | Happy | Sad
---|---|---|---|---
| % Chron | % Eng | % Chron | % Eng | % Chron | % Eng | % Chron | % Eng
Not at all (0) | 84.75 | 80.44 | 79.76 | 77.73 | 56.06 | 54.42 | 82.58 | 80.88
Slightly (1) | 5.94 | 7.17 | 10.30 | 10.87 | 14.71 | 13.66 | 8.28 | 9.42
Somewhat (2) | 4.25 | 5.13 | 5.63 | 6.04 | 11.88 | 12.16 | 4.77 | 4.77
Moderately (3) | 3.01 | 4.09 | 2.86 | 3.34 | 10.04 | 10.81 | 2.67 | 2.89
Extremely (4) | 2.05 | 3.17 | 1.45 | 2.02 | 7.31 | 8.95 | 1.70 | 2.04
Table 18: Reader Emotions (Overall) Answer | Angry | Anxious | Happy | Sad
---|---|---|---|---
| % Chron | % Eng | % Chron | % Eng | % Chron | % Eng | % Chron | % Eng
Not at all (0) | 47.93 | 37.65 | 56.43 | 53.38 | 75.66 | 76.24 | 61.82 | 58.64
Slightly (1) | 18.15 | 17.13 | 20.26 | 21.45 | 8.44 | 7.90 | 18.94 | 18.55
Somewhat (2) | 15.23 | 18.21 | 13.76 | 13.61 | 6.40 | 6.33 | 10.43 | 12.12
Moderately (3) | 12.09 | 16.70 | 7.00 | 7.80 | 4.99 | 5.79 | 5.59 | 6.17
Extremely (4) | 6.60 | 10.31 | 2.55 | 3.76 | 4.52 | 3.74 | 3.23 | 4.52
Table 19: Author Emotions (Political) Answer | Angry | Anxious | Happy | Sad
---|---|---|---|---
| % Chron | % Eng | % Chron | % Eng | % Chron | % Eng | % Chron | % Eng
Not at all (0) | 53.24 | 48.51 | 54.30 | 57.09 | 78.17 | 79.70 | 58.10 | 58.75
Slightly (1) | 17.00 | 16.68 | 21.99 | 18.22 | 8.65 | 6.61 | 18.52 | 18.09
Somewhat (2) | 13.16 | 13.51 | 12.35 | 12.76 | 5.55 | 6.51 | 11.15 | 11.17
Moderately (3) | 8.96 | 11.68 | 7.32 | 6.91 | 4.61 | 3.77 | 7.77 | 6.54
Extremely (4) | 7.63 | 9.62 | 4.04 | 5.03 | 3.02 | 3.40 | 4.45 | 5.45
Table 20: Reader Emotions (Political) Figure 11: The percentage of each (binarized) emotion in the engagement and chronological timelines. Timeline | Author | Reader
---|---|---
| Angry | Anxious | Happy | Sad | Angry | Anxious | Happy | Sad
Chronological | 18.396 | 23.186 | 52.448 | 17.020 | 15.243 | 20.225 | 43.973 | 17.414
Engagement | 25.963 | 27.897 | 52.687 | 20.786 | 19.540 | 22.272 | 45.642 | 19.119
Table 21: Binarized Emotions (Overall). The mean percent of each binarized emotion for readers and authors across each timeline. Timeline | Author | Reader
---|---|---
| Angry | Anxious | Happy | Sad | Angry | Anxious | Happy | Sad
Chronological | 52.151 | 43.752 | 24.341 | 38.281 | 46.893 | 45.729 | 21.663 | 42.065
Engagement | 62.312 | 46.123 | 23.944 | 41.538 | 51.279 | 42.929 | 20.536 | 41.383
Table 22: Binarized Emotions (Political). The mean percent of each binarized
emotion for readers and authors across each timeline, limited to just
political tweets.

#####  S4.3.2 Political outcomes

Next, we show the distribution of survey responses for political questions
(political leaning, out-group animosity, in-group perception, and out-group
perception). We show the distributions for users overall (Figure 12, Tables
23-25) as well as separately for left-leaning (Figure 13, Tables 26-28) and
right-leaning users (Figure 14, Tables 29-31).

Interestingly, in the chronological timeline, right-leaning users are exposed
to much more cross-cutting content than left-leaning users; 21 percent of the
political tweets shown to right-leaning users are left-leaning while only 8
percent of the political tweets shown to left-leaning users are right-leaning.
This may reflect the fact the population of Twitter users tends to be
predominantly left-leaning (see SM section S2). The engagement-based algorithm
doubles the amount of cross-cutting content that left-leaning users see (16
percent compared to 8 percent in the chronological timeline). For right-
leaning users, the proportion of cross-cutting content remains the same, i.e.,
21 percent in both timelines.

Figure 12: Distribution of political outcomes outcomes in the chronological (C) and engagement-based (E) timelines for all users Answer |  % Chronological |  % Engagement
---|---|---
Far left (-2) |  8.48 |  10.64
Left (-1) |  34.20 |  32.70
Moderate (0) |  41.27 |  33.94
Right (1) |  12.20 |  15.28
Far right (2) |  3.85 |  7.44
Table 23: Political leaning (all users) Answer | In-group perc. | Out-group perc.
---|---|---
|  % Chron |  % Eng |  % Chron |  % Eng
Much worse (-2) |  0.87 |  0.64 |  11.97 |  15.63
Worse (-1) |  5.27 |  5.91 |  21.90 |  22.51
The same as before (0) |  72.56 |  69.91 |  63.58 |  59.21
Better (1) |  15.00 |  16.07 |  2.33 |  2.07
Much better (2) |  6.30 |  7.47 |  0.22 |  0.58
Table 24: In-group and out-group perception (all users) Answer |  % Chronological |  % Engagement
---|---|---
No (0) |  63.26 |  55.92
Yes (1) |  36.74 |  44.08
Table 25: Out-group animosity (all users) Figure 13: Distribution of political outcomes in the chronological (C) and engagement-based (E) timelines for left-leaning users only Answer |  % Chronological |  % Engagement
---|---|---
Far left (-2) |  9.77 |  11.42
Left (-1) |  39.09 |  39.05
Moderate (0) |  42.76 |  33.85
Right (1) |  6.12 |  9.54
Far right (2) |  2.26 |  6.15
Table 26: Political leaning for left-leaning users only Answer | In-group perc. (left users) | Out-group perc. (left users)
---|---|---
|  % Chron |  % Eng |  % Chron |  % Eng
Much worse (-2) |  0.64 |  0.58 |  11.30 |  15.84
Worse (-1) |  4.74 |  4.91 |  22.59 |  22.78
The same as before (0) |  72.76 |  70.76 |  64.19 |  59.49
Better (1) |  15.19 |  16.16 |  1.82 |  1.48
Much better (2) |  6.68 |  7.59 |  0.10 |  0.41
Table 27: In-group and out-group perception for left-leaning users only Answer |  % Chronological |  % Engagement
---|---|---
No (0) |  65.72 |  57.04
Yes (1) |  34.28 |  42.96
Table 28: Out-group animosity for left-leaning users only Figure 14: Distribution of political outcomes in the chronological (C) and engagement-based (E) timelines for right-leaning users only Answer |  % Chronological |  % Engagement
---|---|---
Far left (-2) |  4.11 |  8.19
Left (-1) |  17.37 |  12.65
Moderate (0) |  36.96 |  34.74
Right (1) |  32.46 |  33.17
Far right (2) |  9.10 |  11.24
Table 29: Political leaning for right-leaning users only Answer | In-group | Out-group
---|---|---
|  % Chron |  % Eng |  % Chron |  % Eng
Much worse (-2) |  2.01 |  1.21 |  14.35 |  14.94
Worse (-1) |  7.02 |  9.06 |  19.44 |  21.29
The same as before (0) |  71.97 |  67.17 |  61.32 |  58.40
Better (1) |  14.07 |  15.47 |  3.97 |  3.91
Much better (2) |  4.92 |  7.09 |  0.93 |  1.46
Table 30: In-group and out-group perception for right-leaning users only Answer |  % Chronological |  % Engagement
---|---|---
No (0) |  55.33 |  52.44
Yes (1) |  44.67 |  47.56
Table 31: Out-group animosity for right-leaning users only

#####  S4.3.3 Reader’s stated preference

Next, we show how the distribution of users’ stated preference for tweets in
their chronological and engagement-based timelines differs. Here we examine
stated preference across all tweets and just political tweets. Notably, when
considering all tweets, readers see similar percentages of unwanted tweets in
both timelines (13 percent for both the chronological and engagement
timelines). However, when we restrict to only political tweets, 22 percent of
tweets in the engagement timeline are unwanted while only 16 percent of tweets
in the chronological timeline are unwanted.

Figure 15: Distribution of readers’ stated preference in the chronological (C) and engagement-based (E) timelines across all tweets (left figure) and political tweets only (right figure) Answer |  % Chronological |  % Engagement
---|---|---
No (-1) |  13.38 |  13.29
Indifferent (0) |  24.03 |  21.94
Yes (1) |  62.59 |  64.78
Table 32: Reader stated preference Answer |  % Chronological |  % Engagement
---|---|---
No (-1) |  16.21 |  21.96
Indifferent (0) |  24.02 |  22.37
Yes (1) |  59.77 |  55.68
Table 33: Reader stated preference after restricting to political tweets only

####  S4.4 Outcomes by tweet rank

Here, we explore how the outcomes measured change based on the position of the
tweet in each timeline (Figures 16-18). A priori, we expect that the outcomes
in the chronological timeline have no correlation with tweet position since
the chronological timeline is simply ordered by time. However, it would be
reasonable to hypothesize that higher-ranked tweets in the engagement-based
timeline would exhibit differences from lower-ranked tweets from the timeline.
That being said, we did not find any statistically significant differences
between the high-ranked tweets and low-ranked tweets, even in the engagement-
based timeline. However, it is important to note that we only consider the top
10 tweets in the timeline. It could be the case that if we considered more
tweets, we would find larger differences between high-ranked tweets and low-
ranked tweets.

Figure 16: The mean of the author and reader’s emotions broken down by the
rank of the tweet in the timeline. The figures show outcomes both for tweets
overall and specifically for political tweets. There is no statistically
significant difference between high-ranked tweets and low-ranked tweets,
although there appears to be a trend in which higher-ranked tweets are less
likely to express negative emotions. The 95 percent confidence intervals are
generated through bootstrapping. Figure 17: The mean of the readers’ explicit
preference as a function of the rank of the tweet in the timeline. The figures
show outcomes both for tweets overall and specifically for political tweets.
There is no statistically significant difference between high-ranked tweets
and low-ranked tweets. The 95 percent confidence intervals are generated
through bootstrapping. Figure 18: Out-group animosity, in-group perception,
out-group perception, and partisanship broken down by the rank of the tweet in
timeline. There is no statistically significant difference between high-ranked
tweets and low-ranked tweets. The 95 percent confidence intervals are
generated through bootstrapping.

####  S4.5 Effects of Engagement Timeline with Varied Tweet Threshold

We only survey users about the first 10 tweets in their engagement-based
timeline and in their chronological timeline. Thus, when computing the average
treatment effect (ATE) of Twitter’s engagement-based timeline, we only
consider the first 10 tweets in both. To evaluate the robustness of our
results to this choice of threshold, here we also calculate the ATE across a
range of thresholds, spanning from 5 to 10 tweets. Across these thresholds,
our findings were consistently quite robust. This consistency aligns with our
observations in Section S4.4, where no significant disparities were identified
between high and low-ranked tweets concerning the outcomes we evaluated.
Nonetheless, potential differences could emerge further down the timeline
ranking. As future work, it would be interesting to execute a broader
robustness analysis that encompasses thresholds beyond the initial 10 tweets.

Figure 19: The ATE, based on different tweet thresholds, for the author and
readers’ emotions. The top two plots show these outcomes for all tweets, and
the bottom plots restrict to only political tweets. The effect sizes are in
the original units and are not standardized. Figure 20: The ATE, based on
different tweet thresholds, for partisanship, out-group animosity, in-group
perception, and out-group perception. The effect sizes are in the original
units and are not standardized. Figure 21: The ATE, based on different tweet
thresholds, for the readers’ explicit preference for the tweet. The effect
sizes are in the original units and are not standardized.

####  S4.6 Effects of engagement timeline with GPT-4 labels

Figure 22: A comparison of the average treatment effects for tweet-based outcomes, using both the labels given by our participants and labels given by GPT-4. For this comparison, only tweets that GPT-4 provided a validly formatted response to were used to estimate the average treatment effects (for both the ATEs calculated with the human labels and the ATEs calculated with the GPT-4 labels). Outcome | Standardized | Unstandardized | Chronological | Engagement | $p$-value
---|---|---|---|---|---
Effect | Effect | Mean | Mean
Emotional effects (all tweets)
Author Angry | 0.5490 | 0.1710 | 0.2495 | 0.4200 | 0.0002
Author Sad | 0.2302 | 0.0508 | 0.2285 | 0.2845 | 0.0002
Author Anxious | 0.2021 | 0.0283 | 0.1067 | 0.1372 | 0.0004
Author Happy | -0.1158 | -0.0670 | 1.3921 | 1.3303 | 0.0012
Emotional effects (political tweets only)
Author Angry | 1.4653 | 0.4565 | 1.2030 | 1.6045 | 0.0002
Author Sad | 0.1294 | 0.0285 | 0.5129 | 0.5338 | 0.4938
Author Anxious | 0.1668 | 0.0233 | 0.2060 | 0.2169 | 0.4408
Author Happy | -0.2244 | -0.1299 | 0.6099 | 0.4841 | 0.0066
Political effects
Partisanship | 0.1909 | 0.0283 | 0.0950 | 0.1264 | 0.0002
Out-group Animosity | 0.2548 | 0.0311 | 0.0627 | 0.0922 | 0.0002
Table 34: Average treatment effects on tweet-based outcomes using GPT-4
judgments.

In our main analysis, all outcomes are measured through survey questions given
to readers; this includes both outcomes that explicitly ask about the reader’s
perspective (i.e., the reader’s emotions, in-group perceptions, out-group
perceptions, and stated preferences) as well as outcomes that pertain to the
tweet itself (the author’s emotions, the ideological leaning of the tweet, and
whether the tweet contains out-group animosity). Here, we elicit labels for
the tweet-based outcomes from GPT-4 (?) and report effects based on these
machine learning labels (Table 34 and Figure 22).

All the results based on GPT-4 judgments are notably similar to the results
based on reader judgments. Even under the GPT-4 labels, the algorithm is found
to amplify content that is more emotional (especially angry), partisan, and
likely to contain out-group animosity. The main difference in results is that
GPT-4 judges the political tweets chosen by the engagement-based algorithm to
be even angrier than how humans judged the tweets to be.

#####  S4.6.1 Prompts and method

We had GPT-4 (gpt-4-1106-preview in the OpenAI API) label the 28,301 unique
tweets shown to users in our study. We asked GPT-4 to label the author’s
emotions in the tweet (along the four dimensions of angry, sad, happiness, and
anxiety), whether the tweet was political, and if so, to label the ideological
leaning of the tweet and whether the tweet had out-group animosity. All
questions were phrased in the same way as the questions asked to our study
participants.

Out of the 28,301 unique tweets that we submitted to GPT-4 to label, it
returned validly formatted responses for 24,998 tweets. To estimate the
average treatment effects displayed in Figure 22, we used only the tweets with
valid GPT-4 responses (for both the ATEs based on human labels and those based
on GPT-4 labels). This was done to ensure that any observed difference in ATEs
between the human and GPT-4 labels was not influenced by using different sets
of tweets for each. After filtering to tweets with valid GPT-4 responses, the
ATEs were computed through the procedure described in Section S1.3, simply
replacing human labels with GPT-4 labels when relevant.

Below is the prompt given to GPT-4 when a tweet is not a quote tweet or a
reply to another tweet.

⬇

Consider the given tweet by @{author_main}: ‘{main_tweet_text}‘

1. How is @{author_main} feeling in their tweet?

- Angry

- Anxious

- Happy

- Sad

2. Is @{author_main}’s tweet about a political or social issue?

Answer NULL to the subsequent questions if your response to the previous

question is "No".

3. How does @{author_main}’s tweet lean politically?

4. Is @{author_main}’s tweet expressing anger, frustration, or hostility

towards a person or group on the Left?

5. Is @{author_main}’s tweet expressing anger, frustration, or hostility

towards a person or group on the Right?

Respond with a JSON, of the format:

{{

"author_emotions": {{

"angry": [VALUE],

"anxious": [VALUE],

"happy": [VALUE],

"sad": [VALUE]

}},

"is_political": [VALUE],

"political_leaning": [VALUE],

"animosity_left": [VALUE],

"animosity_right": [VALUE]

}}

where [VALUE] is an integer key from:

1. {{

"Not at all": 0,

"Slightly": 1,

"Somewhat": 2,

"Moderately": 3,

"Extremely": 4

}}

2. {{"No": 0, "Yes": 1}}

3. {{

"Far Left": -2,

"Left": -1,

"Moderate": 0,

"Right": 1,

"Far Right": 2,

NULL

}}

4. {{"No": 0, "Yes": 1, NULL}}

5. {{"No": 0, "Yes": 1, NULL}}

Below is the prompt used when a tweet is a quote tweet or a reply to another
tweet. Like our participants, GPT-4 is given both tweets for context and asked
to label both of them. In the following prompt, the variable other_tweet_type
is either equal to “quote tweet of” or “reply to.”

⬇

Consider the following tweets, where @{author_main}’s tweet is a

{other_tweet_type} @{author_other}’s tweet:

@{author_main}: ‘{main_tweet_text}‘

@{author_other}: ‘{other_tweet_text}‘

Note that there are two tweets.

We will first be asking you questions about @{author_main}’s tweet.

You can use @{author_other}’s tweet for context, but answer the following

questions while focusing on @{author_main}’s tweet.

1. How is @{author_main} feeling in their tweet?

- Angry

- Anxious

- Happy

- Sad

2. Is @{author_main}’s tweet about a political or social issue?

Answer NULL to the subsequent questions if your response to the previous

question is "No".

3. How does @{author_main}’s tweet lean politically?

4. Is @{author_main}’s tweet expressing anger, frustration, or hostility

towards a person or group on the Left?

5. Is @{author_main}’s tweet expressing anger, frustration, or hostility

towards a person or group on the Right?

Next, we will be asking you questions about @{author_other}’s tweet.

6. How is @{author_other} feeling in their tweet?

- Angry

- Anxious

- Happy

- Sad

7. Is @{author_other}’s tweet about a political or social issue?

Answer NULL to the subsequent questions if your response to the previous

question is "No".

8. How does @{author_other}’s tweet lean politically?

9. Is @{author_other}’s tweet expressing anger, frustration, or hostility

towards a person or group on the Left?

10. Is @{author_other}’s tweet expressing anger, frustration, or hostility

towards a person or group on the Right?

Respond with a JSON, of the format:

{{

"@{author_main}’s tweet": {{

"author_emotions": {{

"angry": [VALUE],

"anxious": [VALUE],

"happy": [VALUE],

"sad": [VALUE]

}},

"is_political": [VALUE],

"political_leaning": [VALUE],

"animosity_left": [VALUE],

"animosity_right": [VALUE]

}},

"@{author_other}’s tweet": {{

"author_emotions": {{

"angry": [VALUE],

"anxious": [VALUE],

"happy": [VALUE],

"sad": [VALUE]

}},

"is_political": [VALUE],

"political_leaning": [VALUE],

"animosity_left": [VALUE],

"animosity_right": [VALUE]

}}

}}

where [VALUE] is an integer key from:

1. {{

"Not at all": 0,

"Slightly": 1,

"Somewhat": 2,

"Moderately": 3,

"Extremely": 4

}}

2. {{"No": 0, "Yes": 1}}

3. {{

"Far Left": -2,

"Left": -1,

"Moderate": 0,

"Right": 1,

"Far Right": 2,

NULL

}}

4. {{"No": 0, "Yes": 1, NULL}}

5. {{"No": 0, "Yes": 1, NULL}}

6. {{

"Not at all": 0,

"Slightly": 1,

"Somewhat": 2,

"Moderately": 3,

"Extremely": 4

}}

7. {{"No": 0, "Yes": 1}}

8. {{

"Far Left": -2,

"Left": -1,

"Moderate": 0,

"Right": 1,

"Far Right": 2,

NULL

}}

9. {{"No": 0, "Yes": 1, NULL}}

10. {{"No": 0, "Yes": 1, NULL}}

####  S4.7 Heterogeneous effects

Here, we explore how the effects shown in Figure 1 change when restricting our
analysis to subpopulations of users, among demographic lines (or using other
of our survey questions). We subset users the demographic subgroups described
in SM section S2, and additionally by two survey questions (the users’ self-
reported main reason for using Twitter and main category of content seen
during the user study):

  * •

User political leaning (“Political Leaning”)

  * •

User political party (“Political Party”)

  * •

User age group (“Age”)

  * •

User gender (“Gender”)

  * •

User race (“Race”)

  * •

User ethicity (“Ethnicity”)

  * •

User education level (“Education Level”)

  * •

User annual household income (“Household Income”)

  * •

User primary reason for using Twitter (“Why Twitter”)

  * •

User category of content seen (“Primary Category of Content”)

For all questions we consider here that allow multiple answers (race,
ethnicity, category of content), we double-count participants as being present
in each subpopulation that they selected.

The phrasings of all questions, and answer formats can be found in SM section
S5. For easier reference we include below that of the non-demographic survey
questions.

We ask participants about the primary reason they use Twitter as follows:
“What would you say is the main reason you use Twitter?” The options to select
from are: “A way to stay informed,” “Entertainment,” “Keeping me connected to
other people,” “It’s useful for my job or school,” “Let’s me see different
points of view,” “A way to express my opinions.”

We ask participants about the content shown to them as follows: “What were the
tweets we showed you today predominantly about? Select a maximum of two.” The
options to select from are “News,” “Politics,” “Work,” “Entertainment,”
“Hobbies.”

We report these effects by first displaying them for each outcome (Section
S4.7.1) and then for each demographic attribute or survey question response
(Section S4.7.2).

#####  S4.7.1 Outcome effects for each subgroup, grouped by outcome

Figure 23: Conditional average treatment effect (CATE) across subgroups for
the outcome variable ‘author angry.’ The blue line shows the average treatment
effect (ATE).

Figure 24: Conditional average treatment effect (CATE) across subgroups for
the outcome variable ‘author sad.’ The blue line shows the average treatment
effect (ATE).

Figure 25: Conditional average treatment effect (CATE) across subgroups for
the outcome variable ‘author anxious.’ The blue line shows the average
treatment effect (ATE).

Figure 26: Conditional average treatment effect (CATE) across subgroups for
the outcome variable ‘author happy.’ The blue line shows the average treatment
effect (ATE).

Figure 27: Conditional average treatment effect (CATE) across subgroups for
the outcome variable ‘reader angry (overall)’. The blue line shows the average
treatment effect (ATE).

Figure 28: Conditional average treatment effect (CATE) across subgroups for
the outcome variable ‘reader sad (overall).’ The blue line shows the average
treatment effect (ATE).

Figure 29: Conditional average treatment effect (CATE) across subgroups for
the outcome variable ‘reader anxious (overall).’ The blue line shows the
average treatment effect (ATE).

Figure 30: Conditional average treatment effect (CATE) across subgroups for
the outcome variable ‘reader happy (overall).’ The blue line shows the average
treatment effect (ATE).

Figure 31: Conditional average treatment effect (CATE) across subgroups for
the outcome variable ‘author angry (political).’ The blue line shows the
average treatment effect (ATE).

Figure 32: Conditional average treatment effect (CATE) across subgroups for
the outcome variable ‘author sad (political).’ The blue line shows the average
treatment effect (ATE).

Figure 33: Conditional average treatment effect (CATE) across subgroups for
the outcome variable ‘author anxious (political).’ The blue line shows the
average treatment effect (ATE).

Figure 34: Conditional average treatment effect (CATE) across subgroups for
the outcome variable ‘author happy (political).’ The blue line shows the
average treatment effect (ATE).

Figure 35: Conditional average treatment effect (CATE) across subgroups for
the outcome variable ‘reader angry (political).’ The blue line shows the
average treatment effect (ATE).

Figure 36: Conditional average treatment effect (CATE) across subgroups for
the outcome variable ‘reader sad (political).’ The blue line shows the average
treatment effect (ATE).

Figure 37: Conditional average treatment effect (CATE) across subgroups for
the outcome variable ‘reader anxious (political).’ The blue line shows the
average treatment effect (ATE).

Figure 38: Conditional average treatment effect (CATE) across subgroups for
the outcome variable ‘reader happy (political).’ The blue line shows the
average treatment effect (ATE).

Figure 39: Conditional average treatment effect (CATE) for the outcome
variable ‘partisanship.’ The blue line shows the average treatment effect
(ATE).

Figure 40: Conditional average treatment effect (CATE) for the outcome
variable ‘out-group animosity.’ The blue line shows the average treatment
effect (ATE).

Figure 41: Conditional average treatment effect (CATE) for the outcome
variable ‘in-group perception.’ The blue line shows the average treatment
effect (ATE).

Figure 42: Conditional average treatment effect (CATE) for the outcome
variable ‘out-group perception.’ The blue line shows the average treatment
effect (ATE).

Figure 43: Conditional average treatment effect (CATE) across subgroups for
the outcome variable ‘reader preference (overall).’ The blue line shows the
average treatment effect (ATE).

Figure 44: Conditional average treatment effect (CATE) across subgroups for
the outcome variable ‘reader preference (political).’ The blue line shows the
average treatment effect (ATE).

#####  S4.7.2 Outcome effects for each subgroup, grouped by subgroup

Figure 45: Conditional average treatment effect (CATE) for all outcomes when
conditioned on different races. The blue line shows the average treatment
effect (ATE). Figure 46: Conditional average treatment effect (CATE) for all
outcomes when conditioned on different ethnicities. The blue line shows the
average treatment effect (ATE). Figure 47: Conditional average treatment
effect (CATE) for all outcomes when conditioned on different genders. The blue
line shows the average treatment effect (ATE). Figure 48: Conditional average
treatment effect (CATE) for all outcomes when conditioned on different
ideological political leanings. The blue line shows the average treatment
effect (ATE). Figure 49: Conditional average treatment effect (CATE) for all
outcomes when conditioned on different ideological political parties. The blue
line shows the average treatment effect (ATE). Figure 50: Conditional average
treatment effect (CATE) for all outcomes when conditioned on different
education levels. The blue line shows the average treatment effect (ATE).
Figure 51: Conditional average treatment effect (CATE) for all outcomes when
conditioned on different main reasons for using Twitter. The blue line shows
the average treatment effect (ATE). We ask participants about the main reason
they use Twitter as follows: “What would you say is the main reason you use
Twitter?” (SM section S5). The options to select from are: “A way to stay
informed,” “Entertainment” “Keeping me connected to other people,” “It’s
useful for my job or school,” “Let’s me see different points of view,” “A way
to express my opinions”. Figure 52: Conditional average treatment effect
(CATE) for all outcomes when conditioned on different age groups. The blue
line shows the average treatment effect (ATE). Figure 53: Conditional average
treatment effect (CATE) for all outcomes when conditioned on different annual
household income levels. The blue line shows the average treatment effect
(ATE). Figure 54: Conditional average treatment effect (CATE) for all outcomes
when conditioned on different types of primary content category shown. The
blue line shows the average treatment effect (ATE). We ask participants about
the content shown to them as follows: “What were the tweets we showed you
today predominantly about? Select a maximum of two” (SM section S5). The
options to select from are “News,” “Politics,” “Work,” “Entertainment,”
“Hobbies.”

####  S4.8 Effects of stated preference timeline

Outcome | Standardized | Unstandardized | Chron. | Eng. | $p$-value
---|---|---|---|---|---
Effect | Effect | Mean | Mean
Emotional effects (all tweets)
Author Angry | 0.110 | 0.044 | 0.352 | 0.402 | 0.0002
Author Sad | 0.070 | 0.024 | 0.293 | 0.326 | 0.0008
Author Anxious | 0.082 | 0.035 | 0.391 | 0.427 | 0.0002
Author Happy | 0.189 | 0.150 | 1.307 | 1.455 | 0.0002
Reader Angry | 0.095 | 0.038 | 0.306 | 0.350 | 0.0002
Reader Sad | 0.056 | 0.021 | 0.316 | 0.344 | 0.0018
Reader Anxious | 0.082 | 0.035 | 0.346 | 0.384 | 0.0002
Reader Happy | 0.439 | 0.314 | 0.941 | 1.249 | 0.0002
Emotional effects (political tweets only)
Author Angry | 0.205 | 0.081 | 1.084 | 1.173 | 0.0040
Author Sad | 0.170 | 0.059 | 0.675 | 0.731 | 0.0090
Author Anxious | 0.043 | 0.018 | 0.804 | 0.821 | 0.3762
Author Happy | 0.062 | 0.049 | 0.523 | 0.589 | 0.0336
Reader Angry | 0.142 | 0.057 | 1.014 | 1.087 | 0.0240
Reader Sad | 0.088 | 0.033 | 0.814 | 0.858 | 0.1718
Reader Anxious | 0.058 | 0.025 | 0.839 | 0.872 | 0.2546
Reader Happy | 0.111 | 0.079 | 0.450 | 0.548 | 0.0002
Political effects
Partisanship | 0.042 | 0.009 | 0.151 | 0.163 | 0.0342
Out-group Animosity | 0.045 | 0.006 | 0.085 | 0.093 | 0.0242
In-group Perception (all users) | 0.094 | 0.017 | 0.060 | 0.077 | 0.0002
Out-group Perception (all users) | -0.099 | -0.021 | -0.108 | -0.132 | 0.0002
In-group Perception (left users) | 0.083 | 0.015 | 0.056 | 0.071 | 0.0002
Out-group Perception (left users) | -0.100 | -0.020 | -0.097 | -0.120 | 0.0002
In-group Perception (right users) | 0.111 | 0.023 | 0.072 | 0.100 | 0.0002
Out-group Perception (right users) | -0.093 | -0.024 | -0.145 | -0.175 | 0.0018
Reader Preference
Reader Pref (all tweets) | 1.105 | 0.396 | 0.507 | 0.892 | 0.0002
Reader Pref (political tweets) | 0.788 | 0.283 | 0.581 | 0.854 | 0.0002
Table 35: The effects of our alternative, exploratory ranking of content based on users’ stated preferences (SP). The table shows the average treatment effects (standardized and unstandardized), $p$-values, and $p$-values for all outcomes. All statistics are calculated as described in Section S1.3 except we replace the engagement-based timeline with the SP timeline as the treatment of interest. Outcome | Standardized | Unstandardized | Chron. | Eng. | $p$-value
---|---|---|---|---|---
Effect | Effect | Mean | Mean
Emotional effects (all tweets)
Author Angry | -0.058 | -0.023 | 0.352 | 0.338 | 0.0138
Author Sad | -0.011 | -0.004 | 0.293 | 0.299 | 0.6243
Author Anxious | 0.022 | 0.009 | 0.391 | 0.402 | 0.2250
Author Happy | 0.242 | 0.192 | 1.307 | 1.497 | 0.0002
Reader Angry | -0.036 | -0.015 | 0.306 | 0.298 | 0.0750
Reader Sad | -0.015 | -0.006 | 0.316 | 0.319 | 0.4522
Reader Anxious | 0.025 | 0.011 | 0.346 | 0.360 | 0.1466
Reader Happy | 0.490 | 0.350 | 0.941 | 1.283 | 0.0002
Emotional effects (political tweets only)
Author Angry | -0.275 | -0.109 | 1.036 | 0.938 | 0.0010
Author Sad | 0.015 | 0.005 | 0.653 | 0.657 | 0.8191
Author Anxious | -0.052 | -0.022 | 0.790 | 0.761 | 0.3370
Author Happy | 0.145 | 0.115 | 0.536 | 0.668 | 0.0002
Reader Angry | -0.158 | -0.063 | 0.991 | 0.942 | 0.0300
Reader Sad | -0.002 | -0.001 | 0.800 | 0.810 | 0.9695
Reader Anxious | 0.011 | 0.005 | 0.837 | 0.841 | 0.8471
Reader Happy | 0.195 | 0.139 | 0.471 | 0.628 | 0.0002
Political effects
Partisanship | -0.166 | -0.035 | 0.151 | 0.119 | 0.0002
Out-group Animosity | -0.252 | -0.033 | 0.085 | 0.054 | 0.0002
In-group Perception (all users) | -0.004 | -0.001 | 0.060 | 0.059 | 0.8345
Out-group Perception (all users) | 0.046 | 0.010 | -0.108 | -0.101 | 0.0180
In-group Perception (left users) | -0.021 | -0.004 | 0.056 | 0.053 | 0.3364
Out-group Perception (left users) | 0.053 | 0.010 | -0.097 | -0.091 | 0.0170
In-group Perception (right users) | 0.035 | 0.007 | 0.072 | 0.077 | 0.2524
Out-group Perception (right users) | 0.034 | 0.009 | -0.145 | -0.137 | 0.3616
Reader Preference
Reader Pref (all tweets) | 1.117 | 0.400 | 0.507 | 0.896 | 0.0002
Reader Pref (political tweets) | 0.811 | 0.291 | 0.585 | 0.870 | 0.0002
Table 36: The effects of the SP-OA timeline, the timeline that ranks by users’
stated preference and uses the presence of out-group animosity to break ties.
The table shows the average treatment effects (standardized and
unstandardized), $p$-values, and $p$-values for all outcomes. All statistics
are calculated as described in Section S1.3 except we replace the engagement-
based timeline with the SP-OA timeline as the treatment of interest.

####  S4.9 Effects of SP-OA timeline

Ranking by stated preferences (SP) reduced the amount of partisan animosity,
relative to the engagement-based timeline. However, this primarily occurred
through a reduction in animosity towards the reader’s in-group, rather than
animosity towards the reader’s out-group. In other words, there was an
asymmetry in which readers tended to be tolerant of animosity towards their
out-group but not their own in-group. This asymmetry caused a slight echo
chamber effect, in which the SP timeline had the highest proportion of in-
group content and the lowest proportion of out-group content (relative to both
the engagement and chronological timeline). Is it possible to satisfy users’
stated preferences without amplifying echo chambers? To investigate this, we
considered a variant of the SP timeline that used the presence of out-group
animosity to break ties between tweets. We call this new timeline the SP-OA
timeline.

We constructed the SP-OA timeline by adding down-ranking for out-group
animosity to the SP timeline. In particular, in the SP timeline, we scored the
approximately twenty unique tweets for each user by the users’ stated
preference for the tweet: 1 = “Yes,” 0 = “Indifferent,” -1 = “No.” In the SP-
OA timeline, if a tweet was labeled as having out-group animosity, then we
bumped its score down by $0.5$ points. The SP-OA timeline for each user
consists of their top ten tweets as ranked by this modified score. Note that
even if a tweet has out-group animosity, if the user stated that they valued
the tweet, it will always be ranked higher than a tweet that they are
indifferent to or do not value. Since the presence of out-group animosity is
effectively only used to break ties among tweets with the same stated
preference, the SP-OA timeline will satisfy users’ preferences to the same
extent that the regular SP timeline does.

Table 36 shows results for the SP-OA timeline, and Figure 56 compares the
effects of all three timelines: the engagement-based timeline, the SP
timeline, and the SP-OA timeline. The SP-OA yielded the lowest instances of
angry, partisan, and out-group hostile content when stacked against the
chronological, engagement, and SP timelines. Furthermore, the SP-OA timeline
was the most effective at reducing animosity towards the reader’s _out-group_
(Figure 55). In the engagement timeline and SP timeline, respectively 34
percent and 33 percent of political tweets contained animosity towards the
users’ out-group. In contrast, the SP-OA timeline halved this proportion: only
17 percent contained animosity towards the users’ out-group.

In conclusion, the SP-OA timeline has high satisfaction of users’ stated
preferences, mitigates the amplification of divisive content, and does so
without reinforcing echo chambers.

Figure 55: On the left, the graph illustrates the distribution of political
tweets in each timeline, categorized by whether they align with the reader’s
in-group, out-group, or are moderate. Meanwhile, the right graph delineates
the proportion of political tweets that express out-group animosity, broken
down by whether they target the reader’s in-group or out-group. Compared to
the engagement and chronological timeline, the stated preference (SP) timeline
reduces animosity, but only by reducing animosity towards the reader’s _in-
group_. In contrast, the SP-OA timeline reduces animosity towards both the
reader’s in-group and out-group. Figure 56: The average treatment effect for
all pre-registered outcomes along with their 95% Bootstrap confidence
intervals (unadjusted for multiple testing). The effects of three different
timelines are shown: (1) Twitter’s own engagement-based timeline, (2) our
exploratory timeline that ranks based on users’ stated preferences (SP), (3) a
variant that tie-breaks based on the presence of out-group animosity. The
effect sizes for both timelines are relative to the reverse-chronological
timeline (the zero line). Average treatment effects are standardized using the
standard deviation of outcomes in the chronological timeline (see Section S1.3
for details).

###  Supplementary Materials S5 Survey questionnaires

See pages - of qualtrics_survey.pdf