# Memory in humans and deep language models: Linking hypotheses for model
augmentation

Omri Raccah
Intel Labs
New York University
<EMAIL_ADDRESS>
Phoebe Chen
New York University
<EMAIL_ADDRESS>
Ted L. Willke
Intel Labs
<EMAIL_ADDRESS>David Poeppel
New York University
<EMAIL_ADDRESS>
Vy A. Vo
Intel Labs
<EMAIL_ADDRESS>

###### Abstract

The computational complexity of the self-attention mechanism in Transformer
models significantly limits their ability to generalize over long temporal
durations. Memory-augmentation, or the explicit storing of past information in
external memory for subsequent predictions, has become a constructive avenue
for mitigating this limitation. We argue that memory-augmented Transformers
can benefit substantially from considering insights from the memory literature
in humans. We detail an approach to integrating evidence from the human memory
system through the specification of cross-domain linking hypotheses. We then
provide an empirical demonstration to evaluate the use of surprisal as a
linking hypothesis, and further identify the limitations of this approach to
inform future research.

##  1 Introduction

Transformer model architectures [1] have become an indispensable tool in
several domains, such as natural language processing [2], image processing [3]
and reinforcement learning [4, 5]. A widely acknowledged scaling limitation of
the self-attention mechanism is that its computational complexity scales
quadratically with the size of the attention window. This limits the model’s
ability to capture long-range dependencies in data such as books, scientific
articles, or code. Several efficient Transformers have been proposed to
address this principal limitation [6]. A subset of these focus on augmenting
the network with an external memory, which we henceforth refer to as memory-
augmented Transformers [7, 8, 9, 10]. Notably, knowledge in artificial neural
networks (ANNs) is thought to be implicitly stored in the parameters of a pre-
trained model, requiring ever-larger networks to store more facts [11, 12]. In
memory-augmented Transformers, however, information is more explicitly stored
in an external memory and retrieved when making predictions. This may increase
the capacity of the network to represent knowledge about the world, in
addition to helping it capture information over long temporal durations.
Unlike temporal convolutions, which require a pre-specified width, external
memories can be stored for arbitrary durations and retrieved when relevant.

This property is also true of human memory, which demonstrates the remarkable
ability to generalize over an immense amount of information in written
documents and over events in one’s life. The rich literature on memory in
psychology and neuroscience presents ample opportunities for augmenting ANNs
with biologically-inspired memory. Here, we aim to lay some groundwork for
understanding memory across fields, and describe some practical considerations
for effectively integrating findings from cognitive neuroscience into
Transformers, with a particular focus on language models (LMs). Finally, we
provide an empirical demonstration of evaluating a memory augmentation
strategy for GPT-2 [13] using human behavioral data and identify its
limitations and strengths to inform future research.

##  2 Considerations for biologically-inspired memory augmentation

###  2.1 Memory-augmentation from a cognitive lens

We argue that specifying appropriate linking hypotheses across domains will
not only facilitate novel biologically-inspired approaches, but will also
provide a way to empirically evaluate different hypotheses. In cognitive
neuroscience, a linking hypothesis is a proposition for a formal mapping
between a neurobiological state and a psychological state [14, 15], such as
the firing of a single neuron leading to a visual percept. A central aim of
biologically-inspired AI is to formulate linking hypotheses between a
component in an AI system and a well-defined aspect of cognition. Strong
linking hypotheses should lead to a formal and quantifiable mapping between a
representation in an AI system and some neurobiological/psychological data, as
has been demonstrated in some cases for computer vision [16, 17, 18] and
natural language [19, 20, 21, 22]. These linking hypotheses must be specified
at the correct level of analysis [15, 23], e.g., a modification of the
equations to perform similarity search on a database in a retrieval-augmented
system should map to research on the biological mechanisms of memory
retrieval. In our view, proper accounts would be best derived through
decomposing the problem into computational subroutines appropriate for
comparison across domains. Many AI systems already assume a linking hypothesis
between ANNs and human cognition without explicitly stating them as hypotheses
or evaluating them. Here, we briefly explore some of these hypotheses in
memory-augmented Transformers and propose possible mappings to findings in the
human literature.

We divide memory-augmented Transformers into two general types. A _static_
memory stores information in a corpus of fixed size and content (e.g., a
Wikipedia knowledge base), which it learns to retrieve from during training
[24, 25, 9]. The contents of a static memory do not get modified, although
they can be encoded in different formats such as raw text or embeddings [25,
26]. _Dynamic_ memory mechanisms store new information as inputs that are
being processed by the model. Training the network involves learning both
storage and retrieval policies. For example, new information may be remembered
or forgotten on the basis of input properties or model activations.
Furthermore, inputs may be transformed in some manner (e.g., through
compression) before being stored in the external memory [7]. Both static and
dynamic memory-augmented Transformers have shown significant improvements over
non-augmented models when making predictions over long texts [7, 8, 27, 28].

These augmentation strategies do not map cleanly to the types of memory
commonly delineated in cognitive theories of human memory [29]. That said,
classical memory taxonomies are often the source of AI inspiration, with
papers citing work on short- vs. long-term memory or episodic memory [30, 31,
10, 11]. In our view, a static memory could be like human semantic memory if
it uses a knowledge base, or it could be a fairly direct analog of episodic
memory if it stores previously seen examples [24]. Instead, our proposed
division focuses on the subprocesses thought to be involved in human long-term
memory: encoding, consolidation, and retrieval [32]. Different strategies for
memory augmentation will therefore pursue different implementations of each
subprocess, and can draw direct inspiration from studies of that specific
subprocess. Current work on memory-augmented Transformers has already proposed
separate mechanisms for each subprocess, although there is often no direct
link to human data. For example, there is a growing literature on retrieval
augmentation [8, 9, 24, 25, 33, 34, 35] that proposes similarity search as the
retrieval mechanism. Other work has proposed specific encoding policies which
determine what to store and forget, either by exploiting the attention weights
[7] or learning which memories to forget [36].

###  2.2 Incorporating insights from human memory via policy modifications

Here we discuss some findings from the human memory literature to demonstrate
how they may be used to inform policy modifications in memory-augmented
Transformers. Lexical properties (e.g., written-frequency, word length,
animacy, etc.) serve as strong predictors of subsequent memory for individual
words and lists [37, 38, 39]. Furthermore, humans have been shown to have the
remarkable ability to remember whether they have seen an image from up to
10,000 images after only a single exposure [40]. The properties that determine
the memorability of an image are thought to be multifaceted, including high-
level properties such as emotional valence [41] and overall semantic meaning
[42, 43]. If some property is directly computable from the inputs, it can be
efficiently used as a biologically-plausible _encoding policy_ in memory-
augmented models. Recent work in cognitive neuroscience has also been focused
on uncovering the process by which humans segment continuous experience into
composite events in memory, known as event segmentation [44, 45, 46, 47]. This
area of research can be leveraged to inform _storage policies_ , which
delineate how sequential information with ordered constituents is structured
or formatted in memory. Lastly, _retrieval policies_ , or the manner by which
information is read from an existing memory store, can take practical
influence from human memory. For example, items that share a temporal or
semantic context during encoding are retrieved sequentially with relation to
one another [48, 49]. These examples provide theoretically and empirically
motivated hypotheses for memory-augmentation. Next, we demonstrate the
evaluation of a specific linking hypothesis.

##  3 Evaluating a candidate linking hypothesis for memory augmentation

#### Surprisal

The loss function of an LM estimates the negative log likelihood of an
upcoming word given its context. In information theoretic terms, this is known
as surprisal. Some have proposed that next-word prediction is a fundamental
computational process that occurs during human language processing [50, 51,
52], and have shown evidence that LM-estimated surprise predicts behavioral
[53, 54, 55] and neural data [51]. Surprise (or unsigned prediction error) is
also theorized to play a critical role in memory and learning, and
experimental evidence supports this notion [56, 57, 58]. Word surprisal in
particular may predict human memory during natural language comprehension [59,
60]. Since surprise is a readily available quantity in LMs, we test its
feasibility as a linking hypothesis by examining human behavioral data in a
memory experiment. If model-based surprisal can predict human memory, it could
be a practical and effective memory encoding policy for augmented models.

#### Dataset of human recall behavior

We used a public dataset collected by Michelmann et al. from two groups of
participants [61]. The first group ("story-exposure"; N = 50) listened to a
naturally spoken story containing 965 words. Then participants completed a
cloze task [62] similar to an autoregressive LM objective, in which they were
given 10 words from the story and asked to guess the final word (continuing
for every word in the story). The second group ("no exposure"; N = 50)
completed the same cloze task but had no exposure to the story. The memory
effect is the difference in performance across groups. For a full account of
the methods, see Appendix A.

Figure 1: Behavioral results. (A) Cloze performance increases as a function of
story-exposure across individual words. Black lines indicate the mean. (B)
Histogram of memory improvement across words (signed difference between the
story-exposure and no-exposure groups.)

For each word tested in the story, cosine similarity was computed between the
GloVe embeddings [63, 61] for the responded word and the correct answer, and
averaged across participants. In contrast to a binary scoring approach
(correct vs. incorrect), this allows partial credit to be assigned for
semantically similar responses to the correct answer (Appendix A). Replicating
the findings in Michelmann et al. [61], we found that the story-exposure group
significantly outperformed the no-exposure group in guessing the correct words
($p<0.001$; one-tailed test; Figure 1).

###  3.1 Model-based word surprisal is related to human memory for spoken
narratives

We next tested the effect of word surprisal on cloze performance. We used
GPT-2 to estimate surprisal for each of the 1033 story tokens and combined
sub-tokens for each word. We found that word surprisal shows a robust inverse
correlation in both the no-exposure ($R^{2}=0.61$; $p<0.001$) and the story-
exposure group ($R^{2}=0.55$; $p<0.001$; Figure 2A). This indicates that
surprising words predict lower performance in the cloze task, regardless of
prior experience with the story.

We find that GPT-2 estimated surprise is positively correlated with the effect
of memory on cloze task performance, i.e., the signed difference between the
story-exposure and no-exposure groups ($R^{2}=0.17$, $p<0.001$), shown in
Figure 2B. This effect was consistent for larger models with better
perplexity, with only marginal differences in the overall correlation (GPT-2
medium: $R^{2}=0.13$, $p<0.001$; large: $R^{2}=0.13$, $p<0.001$; XL:
$R^{2}=0.11$, $p<0.001$).

However, it is possible that other properties that co-vary with surprise
explain this memory effect. In particular, we evaluate how word frequency and
distinctiveness [64] affect the relationship between LM-based surprise and the
memory effect (Appendix Figure B.1). We fit these features along with word
surprisal as predictors in a multiple regression model to evaluate their
individual impact on memory performance. As expected, we found that the
overall model significantly predicts memory performance, with a higher $R^{2}$
than word surprisal alone ($R^{2}=.24$, $p<0.001$). Each normalized $\beta$
coefficient show a significant contribution in predicting memory
(distinctiveness $\beta=0.011$, $p<0.001$; word surprisal $\beta=0.012$,
$p<0.001$; written-frequency $\beta=-0.02$, $p<0.001$, Appendix Figure B.2).
These findings suggest that word surprisal in GPT-2 predicts human memory
performance for narratives, indicating its candidacy as an encoding policy.

Figure 2: Predicting human behavior with GPT-2 surprise. (A) Model-based
surprisal strongly predicts cloze performance, regardless of story exposure.
(B) Surprisal is significantly correlated with memory performance for
individual words in a spoken story.

###  3.2 Testing the ability of attention weights to predict human memory

Prior work has proposed that attention weights could serve as a memory
encoding mechanism [7]. Extensive research on the role of attention in human
memory [65, 66, 67] suggests this may be a feasible linking hypothesis,
although see [68]. In our dataset, we tested whether memory performance was
correlated with attention weights from several of the 12 layers, and found
either no relation or an extremely weak one for layer 11 (Appendix Figure
B.3).

##  4 Discussion and Future Directions

In this article, we detailed an approach for augmenting LMs on the basis of
the human memory literature. We also provided an empirical demonstration for
validating candidate mechanisms through a principled comparison with human
behavioral data. In particular, we tested the hypothesis that word surprisal
in GPT-2 would predict human memory performance in a narratives dataset [61],
a prediction borne from the human memory literature more broadly [56, 57, 58,
59, 60]. We found that surprisal significantly correlated with human memory in
this task, indicating its viability as a candidate encoding policy for future
architectures. For example, surprisal could determine whether some context or
word is written to an external memory. A similar encoding policy has been
attempted in lifelong learning setups with mixed success [69, 70], suggesting
that understanding the interaction of surprisal with other variables affecting
memory may improve the policy [71].

Several other limitations should be identified in our empirical approach.
First, the story that was used only consisted of 1033 tokens, which only
exceeds the input size of GPT-2 by a small margin. Future work would benefit
from using texts which significantly exceed 1024 tokens to better address
temporal generalization limits in Transformer models. In addition, evaluating
model performance on naturalistic free-recall data would provide a more
ecologically valid benchmark for comparison [72]. Future work should also
consider a more comprehensive evaluation of surprisal-related measures which
can be generated using Transformer outputs or parameter states. In this work,
we also define two types of memory-augmented Transformers, static and dynamic,
which have direct consequences for successful modification. Research in this
area would benefit by continuing to develop a taxonomy of model types which
integrates explicit memory mechanisms to facilitate further progress.

## References

  * [1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is All you Need.  In Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc., 2017.
  * [2] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.  Huggingface’s transformers: State-of-the-art natural language processing.  arXiv preprint arXiv:1910.03771, 2019.
  * [3] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah.  Transformers in Vision: A Survey.  ACM Computing Surveys, 54(10s):200:1–200:41, September 2022.
  * [4] Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel.  A Survey of Reinforcement Learning Informed by Natural Language.  In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19). arXiv, June 2019\.
  * [5] Luckeciano C. Melo.  Transformers are Meta-Reinforcement Learners.  In Proceedings of the 39th International Conference on Machine Learning. arXiv, June 2022.
  * [6] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler.  Efficient Transformers: A Survey.  September 2020.
  * [7] Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap.  Compressive Transformers for Long-Range Sequence Modelling.  In International Conference on Learning Representations, September 2019.
  * [8] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang.  REALM: Retrieval-Augmented Language Model Pre-Training.  In Proceedings of the 37 Th International Conference on Machine Learning, page 10, Vienna, Austria, 2020.
  * [9] Angela Fan, Claire Gardent, Chloé Braud, and Antoine Bordes.  Augmenting Transformers with KNN-Based Composite Memory for Dialog.  Transactions of the Association for Computational Linguistics, 9:82–99, March 2021.
  * [10] Dani Yogatama, Cyprien de Masson d’Autume, and Lingpeng Kong.  Adaptive Semiparametric Language Models.  arXiv:2102.02557 [cs], 9:362–373, February 2021.
  * [11] Aida Nematzadeh, Sebastian Ruder, and Dani Yogatama.  On Memory in Human and Artificial Language Processing Systems.  In "Bridging AI and Cognitive Science" at International Conference on Learning Representations, 2020.
  * [12] Cory Stephenson, Suchismita Padhy, Abhinav Ganesh, Yue Hui, Hanlin Tang, and SueYeon Chung.  On the geometry of generalization and memorization in deep neural networks.  In International Conference on Learning Representations, 2021.
  * [13] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language Models are Unsupervised Multitask Learners.  page 24.
  * [14] Davida Y Teller.  Linking Propositions.  Vision Research, 24(10):1233–1246, 1984.
  * [15] David Poeppel.  The maps problem and the mapping problem: Two challenges for a cognitive neuroscience of speech and language.  Cognitive Neuropsychology, 29(1-2):34–55, March 2012.
  * [16] Radoslaw Martin Cichy, Aditya Khosla, Dimitrios Pantazis, Antonio Torralba, and Aude Oliva.  Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence.  Scientific Reports, 6(1):27755, June 2016.  Number: 1 Publisher: Nature Publishing Group.
  * [17] Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Rishi Rajalingham, Ha Hong, Najib J. Majaj, Elias B. Issa, Pouya Bashivan, Jonathan Prescott-Roy, Kailyn Schmidt, Aran Nayebi, Daniel Bear, Daniel L. K. Yamins, and James J. DiCarlo.  Brain-like object recognition with high-performing shallow recurrent ANNs.  In Proceedings of the 33rd International Conference on Neural Information Processing Systems, number 1147, pages 12805–12816. Curran Associates Inc., Red Hook, NY, USA, December 2019.
  * [18] Daniel LK Yamins and James J. DiCarlo.  Using goal-driven deep learning models to understand sensory cortex.  Nature neuroscience, 19(3):356–365, 2016.  Publisher: Nature Publishing Group.
  * [19] Micha Heilbron, Kristijan Armeni, Jan-Mathijs Schoffelen, Peter Hagoort, and Floris P. de Lange.  A hierarchy of linguistic predictions during natural language comprehension.  Proceedings of the National Academy of Sciences, 119(32):e2201968119, August 2022.  Publisher: Proceedings of the National Academy of Sciences.
  * [20] Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko.  The neural architecture of language: Integrative modeling converges on predictive processing.  Proceedings of the National Academy of Sciences, 118(45):e2105646118, November 2021.  Publisher: Proceedings of the National Academy of Sciences.
  * [21] Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger Levy.  On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior, June 2020.  arXiv:2006.01912 [cs].
  * [22] Leila Wehbe, Brian Murphy, Partha Talukdar, Alona Fyshe, Aaditya Ramdas, and Tom Mitchell.  Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses.  PloS one, 9(11):e112575, 2014.  Publisher: Public Library of Science San Francisco, USA.
  * [23] David Marr.  Vision: A Computational Investigation into the Human Representation and Processing of Visual Information.  W.H. Freeman and Company, 1982.
  * [24] Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis.  Generalization through Memorization: Nearest Neighbor Language Models.  arXiv:1911.00172 [cs], February 2020.
  * [25] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela.  Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.  In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474. Curran Associates, Inc., 2020.
  * [26] Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, and Tom Kwiatkowski.  Entities as Experts: Sparse Memory Access with Entity Supervision.  arXiv:2004.07202 [cs], October 2020.
  * [27] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov.  Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context.  arXiv:1901.02860 [cs, stat], January 2019.
  * [28] Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, and Christian Szegedy.  Memorizing Transformers.  arXiv:2203.08913 [cs], March 2022.
  * [29] Endel Tulving.  Episodic and semantic memory.  In Organization of Memory, pages xiii, 423–xiii, 423. Academic Press, Oxford, England, 1972.
  * [30] Alex Graves, Greg Wayne, and Ivo Danihelka.  Neural Turing Machines.  arXiv:1410.5401 [cs], October 2014.
  * [31] Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, Adrià Puigdomènech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil Blunsom, Koray Kavukcuoglu, and Demis Hassabis.  Hybrid computing using a neural network with dynamic external memory.  Nature, 538(7626):471–476, October 2016.
  * [32] Dale Purves, George J. Augustine, David Fitzpatrick, William C. Hall, James O. McNamara, and S. Mark Williams, editors.  Neuroscience, 3rd ed. Sinauer Associates, Sunderland, MA, US, 2004.
  * [33] Andreas Blattmann, Robin Rombach, Kaan Oktay, and Björn Ommer.  Retrieval-Augmented Diffusion Models.  arXiv:2204.11824, April 2022.
  * [34] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre.  Improving Language Models by Retrieving from Trillions of Tokens.  In Proceedings of the 39th International Conference on Machine Learning, pages 2206–2240. PMLR, June 2022.
  * [35] Oron Ashual, Shelly Sheynin, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, and Yaniv Taigman.  KNN-Diffusion: Image Generation via Large-Scale Retrieval.  arXiv:2204.02849, April 2022.
  * [36] Sainbayar Sukhbaatar, Da Ju, Spencer Poff, Stephen Roller, Arthur Szlam, Jason Weston, and Angela Fan.  Not All Memories are Created Equal: Learning to Forget by Expiring.  In Proceedings of the 38th International Conference on Machine Learning, pages 9902–9912. PMLR, July 2021.
  * [37] C. Hulme, S. Roodenrys, R. Schweickert, G. D. Brown, M. Martin, and G. Stuart.  Word-frequency effects on short-term memory tasks: evidence for a redintegration process in immediate serial recall.  Journal of Experimental Psychology. Learning, Memory, and Cognition, 23(5):1217–1232, September 1997.
  * [38] Earl Y. Popp and Michael J. Serra.  The animacy advantage for free-recall performance is not attributable to greater mental arousal.  Memory (Hove, England), 26(1):89–95, January 2018.
  * [39] Ada Aka, Tung D. Phan, and Michael J. Kahana.  Predicting recall of words and lists.  Journal of Experimental Psychology: Learning, Memory, and Cognition, 47:765–784, 2021.  Place: US Publisher: American Psychological Association.
  * [40] Lionel Standing.  Learning 10000 pictures.  Quarterly Journal of Experimental Psychology, 25(2):207–222, May 1973.
  * [41] Aditya Khosla, Akhil S. Raju, Antonio Torralba, and Aude Oliva.  Understanding and Predicting Image Memorability at a Large Scale.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2390–2398, 2015.
  * [42] Wilma A. Bainbridge.  Memorability: How what we see influences what we remember.  In Psychology of Learning and Motivation, volume 70, pages 1–27. Elsevier, 2019.
  * [43] Nicole C. Rust and Vahid Mehrpour.  Understanding image memorability.  Trends in Cognitive Sciences, 24(7):557–568, July 2020.
  * [44] Jeffrey M Zacks and Barbara Tversky.  Event structure in perception and conception.  Psychological bulletin, 127(1):3, 2001.
  * [45] Zhilin Wang, Anna Jafarpour, and Maarten Sap.  Uncovering Surprising Event Boundaries in Narratives.  In Proceedings of the 4th Workshop of Narrative Understanding (WNU2022), pages 1–12, Seattle, United States, July 2022\. Association for Computational Linguistics.
  * [46] Omri Raccah, Keith B Doelling, Lila Davachi, and David Poeppel.  Acoustic features drive event segmentation in speech.  Journal of Experimental Psychology: Learning, Memory, and Cognition, 2022.
  * [47] Manoj Kumar, Ariel Goldstein, Sebastian Michelmann, Jeffrey M Zacks, Uri Hasson, and Kenneth A Norman.  Bayesian surprise predicts human event segmentation in story listening.  PsyArXiv, Sep 2022.
  * [48] Marc W Howard and Michael J Kahana.  A distributed representation of temporal context.  Journal of mathematical psychology, 46(3):269–299, 2002.
  * [49] Jeremy R Manning and Michael J Kahana.  Interpreting semantic clustering effects in free recall.  Memory, 20(5):511–517, 2012.
  * [50] Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko.  The neural architecture of language: Integrative modeling converges on predictive processing.  Proceedings of the National Academy of Sciences, 118(45):e2105646118, November 2021.
  * [51] Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A. Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, Aren Jansen, Harshvardhan Gazula, Gina Choe, Aditi Rao, Catherine Kim, Colton Casto, Lora Fanda, Werner Doyle, Daniel Friedman, Patricia Dugan, Lucia Melloni, Roi Reichart, Sasha Devore, Adeen Flinker, Liat Hasenfratz, Omer Levy, Avinatan Hassidim, Michael Brenner, Yossi Matias, Kenneth A. Norman, Orrin Devinsky, and Uri Hasson.  Shared computational principles for language processing in humans and deep language models.  Nature Neuroscience, 25(3):369–380, March 2022.
  * [52] Cory Shain, Idan Asher Blank, Marten van Schijndel, William Schuler, and Evelina Fedorenko.  fMRI reveals language-specific predictive coding during naturalistic sentence comprehension.  Neuropsychologia, 138:107307, February 2020.
  * [53] Adam Goodkind and Klinton Bicknell.  Predictive power of word surprisal for reading times is a linear function of language model quality.  In Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018), pages 10–18, Salt Lake City, Utah, January 2018. Association for Computational Linguistics.
  * [54] Yiding Hao, Simon Mendelsohn, Rachel Sterneck, Randi Martinez, and Robert Frank.  Probabilistic predictions of people perusing: Evaluating metrics of language model performance for psycholinguistic modeling.  CoRR, abs/2009.03954, 2020.
  * [55] Cassandra L. Jacobs and Arya D. McCarthy.  The human unlikeness of neural language models in next-word prediction.  In Proceedings of the The Fourth Widening Natural Language Processing Workshop, page 115, Seattle, USA, July 2020. Association for Computational Linguistics.
  * [56] Alyssa H. Sinclair and Morgan D. Barense.  Surprise and destabilize: Prediction error influences episodic memory reconsolidation.  Learning & Memory, 25(8):369–381, August 2018.
  * [57] Meadhbh I. Foster and Mark T. Keane.  The Role of Surprise in Learning: Different Surprising Outcomes Affect Memorability Differentially.  Topics in Cognitive Science, 11(1):75–87, 2019.
  * [58] James W. Antony, Thomas H. Hartshorne, Ken Pomeroy, Todd M. Gureckis, Uri Hasson, Samuel D. McDougle, and Kenneth A. Norman.  Behavioral, Physiological, and Neural Signatures of Surprise during Naturalistic Sports Viewing.  Neuron, November 2020.
  * [59] Richard Futrell, Edward Gibson, and Roger P. Levy.  Lossy-Context Surprisal: An Information-Theoretic Model of Memory Effects in Sentence Processing.  Cognitive Science, 44(3):e12814, 2020.
  * [60] Katja I. Haeuser and Jutta Kray.  Effects of prediction error on episodic memory retrieval: Evidence from sentence reading and word recognition.  Language, Cognition and Neuroscience, 0(0):1–17, May 2021.
  * [61] Sebastian Michelmann, Amy R. Price, Bobbi Aubrey, Camilla K. Strauss, Werner K. Doyle, Daniel Friedman, Patricia C. Dugan, Orrin Devinsky, Sasha Devore, Adeen Flinker, Uri Hasson, and Kenneth A. Norman.  Moment-by-moment tracking of naturalistic learning and its underlying hippocampo-cortical interactions.  Nature Communications, 12(1):5394, September 2021.
  * [62] Wilson L. Taylor.  “Cloze Procedure”: A New Tool for Measuring Readability.  Journalism Quarterly, 30(4):415–433, September 1953.
  * [63] Jeffrey Pennington, Richard Socher, and Christopher D. Manning.  Glove: Global vectors for word representation.  In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, 2014.
  * [64] Greta Tuckute, Kyle Mahowald, Phillip Isola, Aude Oliva, Edward Gibson, and Evelina Fedorenko.  Intrinsically memorable words have unique associations with their meanings.  PsyArXiv, February 2018.
  * [65] Marvin M Chun and Nicholas B Turk-Browne.  Interactions between attention and memory.  Current opinion in neurobiology, 17(2):177–184, 2007.
  * [66] Mariam Aly and Nicholas B Turk-Browne.  Attention promotes episodic encoding by stabilizing hippocampal representations.  Proceedings of the National Academy of Sciences, 113(4):E420–E429, 2016.
  * [67] Mariam Aly and Nicholas B Turk-Browne.  How hippocampal memory shapes, and is shaped by, attention.  In The hippocampus from cells to systems, pages 369–403. Springer, 2017.
  * [68] Grace W. Lindsay.  Attention in Psychology, Neuroscience, and Machine Learning.  Frontiers in Computational Neuroscience, 14, 2020.
  * [69] Tiago Ramalho and Marta Garnelo.  Adaptive posterior learning: Few-shot learning with a surprise-based memory module.  In International Conference on Learning Representations, page 14, 2019.
  * [70] Cyprien de Masson d’Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama.  Episodic Memory in Lifelong Language Learning.  arXiv:1906.01076 [cs, stat], June 2019.
  * [71] Nina Rouhani, Kenneth A. Norman, and Yael Niv.  Dissociable effects of surprising rewards on learning and memory.  Journal of Experimental Psychology: Learning, Memory, and Cognition, 44(9):1430–1443, September 2018.
  * [72] Liberty S. Hamilton and Alexander G. Huth.  The revolution will not be controlled: Natural stimuli in speech neuroscience.  Language, Cognition and Neuroscience, 35(5):573–582, June 2020\.
  * [73] Lynn J. Lohnas and Michael J. Kahana.  Parametric effects of word frequency effect in memory for mixed frequency lists.  Journal of experimental psychology. Learning, memory, and cognition, 39(6):1943–1946, November 2013.
  * [74] David A. Balota, Melvin J. Yap, Michael J. Cortese, Keith A. Hutchison, Brett Kessler, Bjorn Loftis, James H. Neely, Douglas L. Nelson, Greg B. Simpson, and Rebecca Treiman.  The English Lexicon Project.  Behavior Research Methods, 39(3):445–459, August 2007.
  * [75] Jeffrey Pennington, Richard Socher, and Christopher Manning.  GloVe: Global Vectors for Word Representation.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar, 2014. Association for Computational Linguistics.

##  Appendix A Supplemental Materials and Methods

#### Dataset

To evaluate model parameters with respect to human memory performance, we used
a previously published behavioral dataset [61] that was made publicly
available through a Creative Commons 4.0 license. This dataset contains data
collected online in which an experimental group of participants listened to a
spoken story before performing a behavioral task. All participants provided
informed consent in accordance with the local Institutional Review Board.

The behavioral experiment was run online (Amazon’s Mechanical MTurk) across
two groups of volunteer participants [61]. In total, 100 participants were
collected. We used anonymized data from these participants in the current
study (“replication” dataset in [61], which is the only shared dataset that
provides participant-level results).

#### Story materials

Participants listened to a humorous story which is 7 minutes and 30 seconds in
duration and contained 965 words (recorded live as part of the “The Moth”
storytelling event in New York City; “Pieman” by Jim O’ Grady). The only
offensive content in the story is the use of occasional swear words. The
transcript for this story was provided as part of the public dataset release.

#### Experimental procedures

All participants took part in a cloze task [62], which provided participants
with text from the story and asked them to guess the next word. The cloze task
is widely used in the psychological literature to estimate the probability of
the next word given some context, making this behavioral design particularly
suitable for comparison with probabilistic predictions computed from language
models. In the current paradigm, participants saw 10 words from the
transcribed story and asked to guess the word that followed (continuing for
every word in the story). The task starts with the third word in the story,
limiting the context for words in the beginning of the story [61]. In order to
capture memory for the story, participants were split into two experimental
groups. In the first group (N = 50), participants listened to the story once
before performing the cloze task (“story-exposure group”), while the second
group (N = 50) did not listen to the story before performing the cloze task
(“no-exposure group”). The signed difference in performance between the story-
exposure group and the no-exposure group represented the improvement in
performance as a function of recalling the story from memory. The paradigm and
experimental procedures are described in greater detail in [61].

#### Behavioral data analysis

As acquired from [61], cloze task performance was computed using GloVe vector
embeddings (pre-trained on Common Crawl data [63]). Specifically,
participants’ word predictions were scored by computing cosine-similarity
between the vector embeddings for the predicted word and the correct word. The
average semantic similarity among participants for a particular word
represents the group-level performance for that word. We compared these word
similarity scores across the story-exposure and no-exposure groups to evaluate
differences in performance as a function of story recollection. Notably, this
approach provides a more continuous measure of performance as opposed to a
binary dependent measure (correct or incorrect) as used in the main text of
Michelmann et al. ([61]; but see Michelmann Supplementary Method 2 and
Supplementary Note 3). In other words, this measure of performance can be
thought to provide a more sensitive measure than binary scoring, in that some
credit can be assigned in the case that a participant predicts a semantically
similar, but not identical, word to the correct answer.

#### Word surprisal as negative-log likelihood

We computed word surprisal for each word in the story using GPT-2 [13], with
the GPT-2 tokenizer and model from Huggingface [2]. For each of the 1033 sub-
word tokens in the story, we included all previous tokens as preceding context
(up to the maximum 1024). This ensured that we maximized contextual
information for each token. We then take the cross entropy loss for the last
token.

The surprisal for each word in the story is represented by the negative log
likelihood generated by the loss function:

$S_{model}\ (x_{i}\ )=-logP_{model}\ (x_{i}\ |x_{j<i})$

To compute the overall negative log likelihood of a word, we simply sum the
negative log likelihood values of the component sub-tokens. As such, a higher
value represents a more surprising word given its context according to the
model. We refer to the inverse log likelihood at the word-level as word
surprisal.

#### Factoring in word frequency and distinctiveness

Past research has shown that both word frequency [73] and distinctiveness [64]
have a significant impact on word retention. We sought to understand to what
extent these factors along with word surprisal predict memory performance in
the current dataset. We acquired the word written-frequency for each word in
the story using the Hyperspace Analogue to Language (HAL) frequency norms
[74]. Word-distinctiveness, or how few unique word associations a word
possesses, was computed using GloVe word embeddings [75]. In particular, we
computed the cosine-similarity between each word in our story and all of the
other words in the story. Taking one minus the average of these similarity
values provided us with a story-level distinctiveness value for each word,
such that higher values represent more distinctive words. We fit word written-
frequency and distinctiveness along with word surprisal as predictors in a
general linear model (GLM) to predict memory performance. Notably, words for
which written-frequency was not available [74], or for which we were not able
to compute distinctiveness, were excluded from this analysis (19 words
excluded in total)

#### Statistical testing

Throughout this work, statistical analysis was applied using nonparametric
permutation tests. Across comparisons, we implemented 10,000 permutations to
ensure a reliable estimate of the null distribution. Significance was
evaluated at $p<0.05$. Note, we indicate the use of a one-tailed test when an
effect is evaluated in a specific direction, otherwise a two-tailed statistic
is reported.

##  Appendix B Supplemental Results

Figure B.1: Relationships across predictors and memory effect (cosine
similarity) in multiple regression analysis. Figure B.2: Normalized beta
coefficients for predictors in multiple regression analysis. Error bars
represent 95% bootstrapped confidence intervals (10,000 bootstrap iterations).
Figure B.3: Attention weights for 11th layer in GPT-2 versus memory effect
($R^{2}=005$; $p=0.036$). We did not find a significant effect in the 1st,
6th, and 12th layers ($p>0.05$).


# What Twitter Data Tell Us about the Future?

Alina Landowska  SWPS University of Social Sciences and HumanitiesWarsawPoland
<EMAIL_ADDRESS>0000-0002-7966-8243 ,  Marek Robak  Cardinal Stefan
Wyszyński University in WarsawWarsawPoland<EMAIL_ADDRESS>0000-0001-8331-2891 and  Maciej Skórski  University of WarsawWarsawPoland
<EMAIL_ADDRESS>0000-0003-2997-7539

(2023; 10 June 2023)

###### Abstract.

Anticipation is a fundamental human cognitive ability that involves thinking
about and living towards the future. While language markers reflect
anticipatory thinking, research on anticipation from the perspective of
natural language processing is limited. This study aims to investigate the
futures projected by futurists on Twitter and explore the impact of language
cues on anticipatory thinking among social media users. We address the
research questions of what futures Twitter’s futurists anticipate and share,
and how these anticipated futures can be modeled from social data. To
investigate this, we review related works on anticipation, discuss the
influence of language markers and prestigious individuals on anticipatory
thinking, and present a taxonomy system categorizing futures into "present
futures" and "future present." This research presents a compiled dataset of
over 1 million publicly shared tweets by future influencers and develops a
scalable NLP pipeline using SOTA models. The study identifies 15 topics from
the LDA approach and 100 distinct topics from the BERTopic approach within the
futurists’ tweets. These findings contribute to the research on topic
modelling and provide insights into the futures anticipated by Twitter’s
futurists. The research demonstrates the futurists’ language cues signals
futures-in-the-making that enhance social media users to anticipate their own
scenarios and respond to them in present. The fully open-sourced dataset,
interactive analysis, and reproducable source code are available for further
exploration.

Futures, futurists, anticipated futures, Twitter, topic modelling, language
markers, social data, natural language processing

††copyright: acmcopyright††journalyear: 2023††copyright:
rightsretained††conference: ACM International Conference on Information
Technology for Social Good; September 6–8, 2023; Lisbon, Portugal††booktitle:
ACM International Conference on Information Technology for Social Good (GoodIT
’23), September 6–8, 2023, Lisbon, Portugal††doi:
10.1145/3582515.3609573††isbn: 979-8-4007-0116-0/23/09††ccs: Networks Social
media networks††ccs: General and reference Empirical studies††ccs: Mathematics
of computing Exploratory data analysis

##  1\. Introduction

###  1.1. Background

_What is anticipation about?_ Anticipation is the human cognitive ability of
thinking about and living towards the future. It is a natural part of our
lives and may be found in almost any situation. Anticipatory thinking and
behaving are necessary to manage a complex and unforeseeable future (Poli,
2009). A function of the human brain as “anticipation machine” is to prepare
people for future events with a whole range of probabilities (from low to
high), not simply predict what might happen (Gilbert and Wilson, 2007). Most
of the time, people are engaged in an effort to figure out how to predict
particular occurrences. In the face of complexity and uncertainty generated by
an ever-changing world, anticipation plays a crucial role in managing the
challenges and opportunities. Understanding how individuals anticipate the
future is vital for effective future management.

Language markers reflect anticipatory thinking (Bennett, 1976). However,
research on anticipation from the perspective of natural language processing
has yet to be conducted. In this study, we aim to investigate the futures
projected by futurists on Twitter, assuming that social media users anticipate
their futures based on language cues provided by these futurists.
Specifically, we seek to answer the following research questions: can we refer
to these specific questions?  RQ.1 What futures do Twitter’s futurists
anticipate and share? RQ.2 How can anticipated futures be modelled from social
data? We discuss the development of anticipation research through different
concepts, the features of anticipatory thinking, and approaches to
categorizing futures. By reviewing related works, we establish a foundation
for our investigation into Twitter’s futurists and their projected futures.

Our research aims to shed light on the futures anticipated by Twitter’s
futures influencers and explore the impact of language cues on anticipatory
thinking among social media users. Through the analysis of social data,
specifically futurist’s tweets, and employing the LDA and BERT models, our aim
is to deepen our comprehension of the anticipated futures that emerge within
the digital realm. This study contributes to the growing body of research on
anticipatory rhetoric and its implications for anticipation thinking.

Section 2.1 discusses the influence of language markers and prestigious
individuals, like futurists, on others’ anticipatory thinking, providing the
foundation for our research. In Section 2.2, we present a taxonomy system that
categorizes futures based on their relation to the past and present, setting
the stage for understanding the futures projected by Twitter’s futurists.
Section 3 describes the data acquisition and pre-processing (3.1), modelling
(3.2), and inference (3.3). In Section 4, we present and discuss our results
obtained from the LDA and the BERT modelling.

###  1.2. Contribution

Our technical contributions are:

  * •

Future Influencers Dataset we contribute a compiled dataset of more than 1M
tweets shared publicly by Future Influencers. We believe that this interesting
dataset will be useful for other research projects.

  * •

Open-Sourced & Post-API analysis, including source code, data and several
interactive presentations, is available as the Open Science Framework
repository (Skórski et al., 2023). Given limitations of Twitter’s Academic
License, as well as general concerns around data availability in the Post-API
world (Tromble, 2021) we decided to scrape the publicly available data, and
demonstrated that this is both accurate and efficient.

  * •

Scalable NLP pipeline with SOTA models: we leverage established topic
extraction methods and implement an efficient pipeline, tweaked to Twitter
data. The code is available in the repository (Skórski et al., 2023).

##  2\. Related Works

In this section, we discuss the foundations of anticipation (Section 2.1) and
group related works into concepts contributing to the features of anticipatory
thinking (Section 2.2).

###  2.1. Anticipatory cues in language

_Why do people anticipate?_ Human’s ability to combine knowledge, experiences,
stories, or concepts and formulate certain conclusions for the futures allows
us to deal with uncertainty and ambiguity (Klein and Snowden, 2011). Knowing
if the future event will occur (i.e., the hazard rate, HR) and when it will
happen (i.e., time estimation) are two key factors determining this
uncertainty (Grabenhorst et al., 2021). Anticipation helps people manage the
unknown future. On one side, it is a forward-thinking mindset (Poli, 2019). On
the other, it is the application of the former’s result to action (Poli,
2019). Research shows that there is a human tendency to anticipate positive
future events over negative ones, e.g., (Bennett, 1976; Grabenhorst et al.,
2019).

_How are people attracted to anticipated futures?_ Humans’ attention is
primarily attracted by the mental projection of a scenario onto a possible
future (Moore and Hoffman, 2011). While attention is focused on certain events
(or information), ones are intentionally neglected; when sensory cues (e.g.,
visual cues) are signalled, then human anticipation is activated (Grabenhorst
et al., 2019). Attracting attention with content plays a crucial role on
social media platforms. The content, containing some anticipated futures,
might particularly attract the attention of the followers. By paraphrasing
Rosen’s model (Rosen, 1972), it can be said that a number of followers might
anticipate the future on the basis of the input information (i.e., social
media content in a form of posts/tweets/videos about the future state of,
e.g., future events, the economy, health system, etc.) gained from some
prestigious individual.

_Who is signalling cues that activate anticipatory thinking?_ The Foggian
approach states that persuasive technologies, e.g., social media platforms
such as Twitter) aim at changing people’s attitudes, which in turn modify
behaviours (Fogg, 2003; Holmes, 2016). One of Twitter’s persuasive elements is
influencers using their privileged position; when someone is prestigious, it
means the person is honoured and esteemed, and people tend to listen to them
(Henrich and Gil-White, 2001). Prestigious status is given to a person who
shares knowledge, assets, skills, and/or other resources with other people
(Boyd and Richerson, 2005; Henrich and Gil-White, 2001; Patton, 2000; Pinker,
1997; Tooby and Cosmides, 1996). Because people are "default infocopiers",
they attempt to learn directly from prestigious individuals rather than
“reinvent the wheel” which leads to significant cost savings (Henrich and
Boyd, 1998; Henrich and Gil-White, 2001). Adopting this approach, we recognize
’future influencers’, who at the same time are prestigious individual and
futurists, as those who impact others’ anticipatory thinking by tweeting
anticipatory cues in language markers. As research proved, the credibility and
trustworthiness of such a future influencer determine the success of the
content spread, e.g., (Jin and Phua, 2014; Spry et al., 2011).

_How do language markers serve as carriers for the sharing of anticipated
futures?_ The development of language made possible, among other things, the
social transfer of a significant amount of information that is impossible to
deduce based just on auditory and visual senses e.g., (Henrich and Gil-White,
2001). On social media platforms, such as Twitter, people with a prestige
status share the information with others (Acerbi, 2019, 2016). Many different
types of information are shared, e.g., (Acerbi, 2022; Berger, 2013; Gloor et
al., 2019)). Language markers in the form of written words are recognized as
visual cues that reflect anticipatory thinking (Bennett, 1976). Tweets attract
others attention, especially like-minded others (Mosleh et al., 2021), and
this is their primary function. Further, they might contain anticipatory cues
coded in language for social media users, which might guide their own
anticipation about ’what’ and ’when’ (Momennejad and Haynes, 2012).

###  2.2. Futures Taxonomy

_What are the types of futures?_ For the purpose of this research, we
recognize two main categories of the future: a. ‘present futures’ (Luhmann,
1982) are ‘pre-given futures’ rooted in the past, i.e., lived ones (Adam and
Groves, 2011); and b. ‘future present’ (Luhmann, 1982) are ‘futures in the
making’ that are possibly latent, growing, and changing ones, i.e., living
ones (Adam and Groves, 2011). Following Poli‘s (Poli, 2017, 2014) taxonomy,
the present future refers to the concrete (practical, presentational,
explicit) and it is the contextual, embodied, and embedded future (Adam and
Groves, 2007); and the future present refers to the abstract (symbolic,
representational, implicit) future that is the “decontextualised future
emptied of content, which is open to exploration and exploitation,
calculation, and control” (Adam and Groves, 2007; Poli, 2014, 2017).

Present futures are linear continuations of the past in the present (Poli,
2017), which shape the future by present means (Miller, 2007). Shaping by
imagination, planning, projection, and production is a need of the present
that looks for economic or scientific forecasts (Adam and Groves, 2007). These
concrete-like futures are closed-type futures that can be calculated. Hence,
present futures look as one’s viewpoint _versus_ another’s viewpoint (Luhmann,
1982). The value of the present future is calculated against its alternatives
and traded as a commodity; On the basis of its expected returns, one future
outcome may be traded for another (Adam and Groves, 2007), and the future with
the highest value is the one with the largest profit. Because of the
optimistic or pessimistic character of their transformed or reinterpreted
visions, the present futures serve as utopian projections aiming at evoking
hope or fear (Luhmann, 1982). Despite that, they seem to be the most realistic
ones because they are common sense-based from the present perspective.

Future presents are latent, but they can be recognized and foreseen (Adam and
Groves, 2011) thus impacting the present by entering into it and being used in
the present. These futures are conceptualized, isolated from their contexts,
and technologically prejudiced (Luhmann, 1982), which makes them limitless
possibilities, leading to high uncertainty (Adam and Groves, 2007; Beckert,
2013; Poli, 2014, 2017). These abstract futures are open-type futures with
abstract values that are freely traded on the assumption that they can be
computed at any location and at any time, and it may be used for any situation
(Adam and Groves, 2007). Like fantasies, future presents are unpredictable and
evoke fictional expectations based on the “as if” rule (Beckert, 2013).
However, they emerge from abstract systems as a result of scientific modelling
based on past data (e.g., forecasting extrapolation) (Giddens, 1991), and they
are quantitative by nature, so they might be ranked and compared to one
another (Poli, 2017). Future presents are anticipatory in their character
because of their ‘use-of-the-future’ orientation (Poli, 2017).

##  3\. Research Methods

In this section, we describe in detail our workflow. There are three main
stages: data collection, topic modelling (followed by validation feedback),
and inference (classification with visualization). The process is illustrated
on Figure 1.

Figure 1. Our workflow: from data acquisition, through topic modelling with
validation feedback, until topics inference.

###  3.1. Data Acquisition

The data set was obtained using the scraping library snsscrape
(JustAnotherArchivist, 2023), and Tweets were gathered between the 1sth of
January 2021, and 31th of March 2023. There are 1,2 milions tweets in the
’future’ data set. The data set comprises tweets posted by ’future
influencers’ that were carefully selected by human experts. The final list
included about 250 futurists who were expected to make references to their
anticipated future scenarios in their tweets.

Here our contribution is the successful use of the scrapping API
(JustAnotherArchivist, 2023) at large scale, demonstrating that a fully open-
source data-mining on Twitter is possible in the Post-API world.

###  3.2. Topic Modelling

Following prior work on extracting topics from Twitter data (Bogdanowicz and
Guan, 2022; Meddeb and Romdhane, 2022; Egger and Yu, 2022), we trained two
state-of-the-art models: the Latent Dirichlet Allocation model (LDA) provided
by the Gensim library (Rek and Sojka, 2010) (we used the multi-core
implementation), and the BERTopic model implemented in the BERTopic library,
with submodels (embeddings, dimensionality reduction, clustering, tokenizing
and class-weighting) chosen to support batch training, as detailed in Figure
2. Note that we couldn’t use some more accurate algorithms (such as Uniform
Manifold Approximation and Projection for dimensionality reduction) if their
implementations did not support online training.

all-MiniLM-L6-v2sentence embeddingPCAdimensionality
reductionKMeansclusteringCountVectorizertokenizingc-TF-IDFweighting schema
Figure 2. The architecture of the BERTopic pipeline used in this work.

###  3.3. Inference

Hyperparameters of both models were optimized by considering the coherence
measure and expert judgmentoptimization packages like (Terragni et al., 2021).
The heaviest part of the BERT model - the transformer layer - was optimized by
precomputing sentence transformers on GPU. In turn, the heaviest part of the
LDA model - text preprocessing - was optimized by parallelizing. Both models
were trained on batches (online) to ensure scalability, so that the memory
consumption was constant with respect to the data size. The LDA model was
trained on multiple CPU cores.

##  4\. Results and Discussion

In this section, we present what we have accomplished so far in terms of
futures’ topic modelling by using the LDA and the BERT approaches, as well as
discuss these findings in terms of futures as described in Section 4, trying
to recognize futures that are anticipated by ’future influencers’. Due to the
space constraints, we refer readers to the repository (Skórski et al., 2023)
for details and interactive content.

###  4.1. Futures from the LDA Topic Modelling

By applying the LDA method and performing additional validation with the
established coherence score developed by Röder et al. (Röder et al., 2015)
(see Figure 3 and the repository for more details), 15 topics were identified
in the data set. The boundaries of topics are not always clearly defined, so
topics overlap with each other. For instance, a topic dedicated to the future
of AI, which includes i.a., machine learning, learning, data science, big
data, coding, and analytics, overlaps with a topic dedicated to the future of
technology, which includes i.a., cybersecurity, metaverse, technology, IoT,
behaviour, leadership, management, and startups. Because the tweets are very
short text utterances (a regular tweet is limited to 280 characters) and there
might be too many topics within the analysed dataset, it clearly becomes
unsuitable for the LDA. Overall, the model achieved a decent (in its class)
coherence score of more than 50%.

(a) LDA topics. The cluster in red is the "future of work" topic, with
keywords such as "#remotework", "culture", "office", "conversation".

(b) Validation with the $C_{V}$ coherence score suggests $n=15$ as the optimal
number of topics.

Figure 3. Validation of the trained LDA model. Figure 4. Topics dynamic in
time.

As we see on Figure 4 all topic contributions remain consistent in time. It
suggests that the themes or subjects covered by the topics continue to be
relevant and maintain their significance without evident fluctuations or
shifts in prominence. This consistency implies that the topics consistently
capture and represent the main content on future within the given dataset over
time.

###  4.2. Futures from the BERT Topic Modelling

The sentence transformers applied in BERTopic allow for the extraction of
embeddings that capture the semantic meaning of sentences. This enables the
clustering of documents based on not only lexical but also semantic
similarities. In turn, the c-TF-IDF algorithm, used to generate dense
clusters, enables us to visualize a readily understandable list of topics
while retaining significant or defining words in the topic description. Hence,
the implementation of the BERTopic enabled the extraction of significantly
more extensive and detailed topics than the LDA. The results revealed
discussions, which were not identified in the LDA, for instance anticipated
trends in robotics or expectation on COVID-19 research progress. Finally, the
model proposed a total of about 100 hierarchically organized clusters of good
coherence (65%), that can be further merged to about 20 well-separated topics
(albeit with a loss in coherence). Among the merged topics are: future of AI,
automation, digital transformation, future of work, expected changes in
management culture, privacy concerns around social media, These potential
futures are conceptualized and detached from their specific contexts.
Consequently, they represent boundless possibilities and must force social
media users reading them to anticipate their own futures, leading to a state
of high uncertainty. For instance, the future of automation is expected to
generate approximately 133 million new job opportunities, despite an estimated
loss of 75 million jobs. However, the overall employment rate is expected to
continue its upward trajectory, so people reading this might anticipate a
skill shift and, in consequence, a need for upskilling or even reskilling.

(a) The hierarchy of topics identified by the BERTopic model.

(b) The inter-topic distance map (before topics reduction).

Figure 5. Results from BERTopic.

###  4.3. Limitations

While our study provides valuable insights, it is important to acknowledge its
limitations. Firstly, the analysis was confined to a specific subset of
Twitter users (namely, influencers), which may slightly restrict the
generalizability of our findings. Therefore, for future research we plan to
investigate the generalizability of our findings to broader and more
representative future influencer populations, specifically including those who
were not included in the analysis. Another limitation is that not all tweets
attributed to future influencers necessarily reflect their own future
anticipations. Some of these tweets could have been retweeted from other users
or may contain quotations.

Additionally, the study was limited to tweets published by future influencers.
It would be worthwhile to extend the research to include user comments and
examine the sentiment associated with those comments—essentially, exploring
how users perceive the future suggested by future influencers.

##  5\. Conclusion

This study examines the topics referring to the futures introduced by
futurists on the widely-used social media platform, Twitter. Twitter was
selected as the platform of choice due to its global reach and its ability to
empower influencers, including futurists, to express their viewpoints on
upcoming events, technologies, and processes that they anticipate, and
communicate through tweets. In this paper, the LDA and the BERTopic models,
were used to identify the most frequently discussed topics in the dataset of
’future influencers’ tweets, and the results were compared between both
models. Firstly, our process began with a thorough data cleansing phase aimed
at eliminating any irrelevant elements. Subsequently, we employed the LDA and
the BERTopic on the pre-processed data. In comparison to the LDA, the BERTopic
is more flexible and gives more significant and varied results.

It is evident that the futures anticipated by future influencers and shared by
them in tweets are the future present types that are not fixed entities;
instead they represent dynamic and evolving states that undergo constant
shaping and transformation, e.g. the future of digital twins holds latent
potential that is yet to be fully realized - there may be untapped
capabilities, unexplored use cases, and advancements that can transform how
digital twins are utilized and integrated into various industries and domains.
These ’futures in the making’ are continuously developing, and subject to
change, indicating their vibrant and alive nature. Because they are not
predetermined, they are susceptible to modifications. In other words, future
presents discussed by future influencers on Twitter are conceptualized
suggestions that might change if certain conditions change, but they possess
significant influence in compelling others to anticipate various possibilities
and take immediate action to prepare for them.

As we have shown, utilizing social data extracted from Twitter allows for the
modelling of futures presented to the audience through tweets, which holds
significance for several reasons. Firstly, patterns and trends can be
identified, which can potentially offer predictive insights (e.g., market
trends, emerging opportunities). Secondly, by anticipating future directions,
people can proactively adapt and prepare for upcoming changes. Furthermore,
analysing the futures presented in tweets and comments (not investigated here)
will help in gaining insights into how the audience perceives and interprets
these future scenarios (the audience’s sentiment to certain futures). In the
future, our focus will be directed towards enhancing and refining the
modelling methods. Furthermore, we aim to extend the BERT model capabilities
to encompass the prediction of emerging opportunities. However, we are aware
that the success of using BERT for this depends on the quality and
representativeness of the training data, as well as the ability of the model
to capture and generalize patterns effectively.

## References

  * (1)
  * Acerbi (2016) Alberto Acerbi. 2016\.  A Cultural Evolution Approach to Digital Media.  _Frontiers in Human Neuroscience_ 10 (Dec. 2016), 1–12.  https://www.frontiersin.org/article/10.3389/fnhum.2016.00636
  * Acerbi (2019) Alberto Acerbi. 2019\.  _Cultural Evolution in the Digital Age_.  Oxford University Press, Oxford.
  * Acerbi (2022) Alberto Acerbi. 2022\.  From Storytelling to Facebook.  _Human Nature_ 33 (2022), 132–144.  https://doi.org/10.1007/s12110-022-09423-1
  * Adam and Groves (2007) Barbara Adam and Chris Groves (Eds.). 2007.  _Future matters. Action, Knowledge, Ethic_.  Brill, Leiden & Boston.
  * Adam and Groves (2011) Barbara Adam and Chris Groves. 2011.  Futures Tended: Care and Future-Oriented Responsibility.  _Bulletin of Science, Technology & Society_ 31, 1 (Feb. 2011), 17–27.  https://doi.org/10.1177/0270467610391237 Publisher: SAGE Publications Inc.
  * Beckert (2013) Jens Beckert. 2013\.  Capitalism as a System of Expectations: Toward a Sociological Microfoundation of Political Economy.  _Politics & Society_ 41, 3 (Sept. 2013), 323–350.  https://doi.org/10.1177/0032329213493750 Publisher: SAGE Publications Inc.
  * Bennett (1976) John W. Bennett. 1976\.  Anticipation, Adaptation, and the Concept of Culture in Anthropology.  _Science_ 192, 4242 (1976), 847–853.  http://www.jstor.org/stable/1742130 Publisher: American Association for the Advancement of Science.
  * Berger (2013) Jonah Berger. 2013\.  _Contagious: why things catch on_.  Simon & Schuster, New York.  https://books.google.pl/books?id=J2l7pgwTiW4C&printsec=frontcover&hl=pl&source=gbs_ge_summary_r&cad=0#v=onepage&q&f=false
  * Bogdanowicz and Guan (2022) Alexander Bogdanowicz and ChengHe Guan. 2022.  Dynamic topic modeling of twitter data during the COVID-19 pandemic.  _PLOS ONE_ 17, 5 (May 2022), e0268669.  https://doi.org/10.1371/journal.pone.0268669
  * Boyd and Richerson (2005) Robert Boyd and Peter Richerson. 2005.  _The Origin and the evolution of culture_.  Oxford University Press, Oxford.  https://ia801207.us.archive.org/16/items/TheAgeOfManipulationWilsonBryanKey/The%20Origin%20And%20Evolution%20Of%20Cultures%20by%20Robert%20Boyd.pdf
  * Egger and Yu (2022) Roman Egger and Joanne Yu. 2022.  A Topic Modeling Comparison Between LDA, NMF, Top2Vec, and BERTopic to Demystify Twitter Posts.  _Frontiers in Sociology_ 7 (May 2022), 886498.  https://doi.org/10.3389/fsoc.2022.886498
  * Fogg (2003) B.J. Fogg. 2003\.  _Persuasive Technology: Using Computers to Change What We Think and Do_.  Association for Computing Machinery, New York, NY, USA.  https://doi.org/10.1145/764008.763957
  * Giddens (1991) Anthony Giddens. 1991\.  _Modernity and self identity: Self and society in the late modern age_.  Stanford University Press, Stanford, CA.
  * Gilbert and Wilson (2007) Daniel Gilbert and Timothy Wilson. 2007.  Prospection: Experiencing the Future.  _Science (New York, N.Y.)_ 317 (Oct. 2007), 1351–4.  https://doi.org/10.1126/science.1144161
  * Gloor et al. (2019) Peter A. Gloor, Andrea Fronzetti Colladon, Joao Marcos de Oliveira, Paola Rovelli, Manuel Galbier, and Manfred Vogel. 2019.  Identifying Tribes on Twitter Through Shared Context.  In _Collaborative Innovation Networks_ (1 ed.), Yang Song, Francesca Grippa, Peter A. Gloor, and João Leitão (Eds.). Springer Cham, Electronic, 91–111.  https://doi.org/10.1007/978-3-030-17238-1
  * Grabenhorst et al. (2021) Matthias Grabenhorst, Laurence T. Maloney, David Poeppel, and Georgios Michalareas. 2021.  Two sources of uncertainty independently modulate temporal expectancy.  _Proceedings of the National Academy of Sciences_ 118, 16 (April 2021), e2019342118.  https://doi.org/10.1073/pnas.2019342118 Publisher: Proceedings of the National Academy of Sciences.
  * Grabenhorst et al. (2019) Matthias Grabenhorst, Georgios Michalareas, Laurence T. Maloney, and David Poeppel. 2019\.  The anticipation of events in time.  _Nature Communications_ 10, 1 (Dec. 2019), 5802\.  https://doi.org/10.1038/s41467-019-13849-0
  * Henrich and Boyd (1998) Joe Henrich and Robert Boyd. 1998.  The evolution of conformist transmission and the emergence of between-group differences.  _Evolution and Human Behavior_ 19, 4 (1998), 215–241.  https://doi.org/10.1016/S1090-5138(98)00018-X Place: Netherlands Publisher: Elsevier Science.
  * Henrich and Gil-White (2001) Joseph Henrich and Francisco J. Gil-White. 2001.  The evolution of prestige: Freely conferred deference as a mechanism for enhancing the benefits of cultural transmission.  _Evolution and Human Behavior_ 22, 3 (2001), 165–196.  https://doi.org/10.1016/S1090-5138(00)00071-4 Place: Netherlands Publisher: Elsevier Science.
  * Holmes (2016) Steve Holmes. 2016\.  Ethos, Hexis, and the Case for Persuasive Technologies.  _Enculturation: A Journal of Rhetoric, Writing, and Culture_ 23 (Nov. 2016), –.  https://www.enculturation.net/ethos-hexis-and-the-case-for-persuasive-technologies
  * Jin and Phua (2014) Seung-A Annie Jin and Joe Phua. 2014.  Following Celebrities’ Tweets About Brands: The Impact of Twitter-Based Electronic Word-of-Mouth on Consumers’ Source Credibility Perception, Buying Intention, and Social Identification With Celebrities.  _Journal of Advertising_ 43, 2 (April 2014), 181–195.  https://doi.org/10.1080/00913367.2013.827606 Publisher: Routledge.
  * JustAnotherArchivist (2023) JustAnotherArchivist. 2023\.  snscrape.  https://github.com/JustAnotherArchivist/snscrape.git
  * Klein and Snowden (2011) Gary Klein and David Snowden. 2011.  Anticipatory Thinking.  In _Informed by Knowledge: Expert Performance in Complex Situations_ , Kathleen L. Mosier and Ute M. Fischer (Eds.). Psychology Press Taylor & Francis Group, New York.  https://doi.org/10.4324/9780203847985.
  * Luhmann (1982) Niklas Luhmann. 1982\.  _The differentiation of society_.  Columbia University Press, New York.
  * Meddeb and Romdhane (2022) Amna Meddeb and Lotfi Ben Romdhane. 2022.  Using Topic Modeling and Word Embedding for Topic Extraction in Twitter.  _Procedia Computer Science_ 207 (Jan. 2022), 790–799.  https://doi.org/10.1016/j.procs.2022.09.134
  * Miller (2007) Riel Miller. 2007\.  Futures Literacy: A Hybrid Strategic Scenario Method.  _Futures_ 39, 4 (2007), 341–362.
  * Momennejad and Haynes (2012) Ida Momennejad and John-Dylan Haynes. 2012.  Human anterior prefrontal cortex encodes the ‘what’ and ‘when’ of future intentions.  _NeuroImage_ 61, 1 (May 2012), 139–148.  https://doi.org/10.1016/j.neuroimage.2012.02.079
  * Moore and Hoffman (2011) David T. Moore and Robert R. Hoffman. 2011.  Data-Frame Theory of Sensemaking as a Best Model for Intelligence.  _American Intelligence Journal_ 29, 2 (2011), 145–158.  http://www.jstor.org/stable/26201963 Publisher: National Military Intelligence Association.
  * Mosleh et al. (2021) Mohsen Mosleh, Gordon Pennycook, Antonio A. Arechar, and David G. Rand. 2021.  Cognitive reflection correlates with behavior on Twitter.  _Nature Communications_ 12, 1 (Feb. 2021), 921\.  https://doi.org/10.1038/s41467-020-20043-0
  * Patton (2000) John Q. Patton. 2000\.  Reciprocal Altruism and Warfare: A Case from the Ecuadorian Amazon.  In _Adaptation and Human Behavior. An Anthropological Perspective_ , Lee Cronk, Napoleon Chagnon, and William Irons (Eds.). Routledge, New York, 417–436.
  * Pinker (1997) Steven Pinker. 1997\.  _How the mind works_.  W.W. Norton & Co., New York.
  * Poli (2009) Robert Poli. 2009\.  The Complexity of Anticipation.  _Balkan Journal of Philosophy_ 1, 1 (2009), 19–29.  https://www.pdcnet.org/C1257CEF0046A0DE/file/628BD364BCFB58C6C1257CF90071DBB8/$FILE/bjp_2009_0001_0001_0019_0029.pdf
  * Poli (2014) Roberto Poli. 2014\.  Anticipation: A New Thread for the Human and Social Sciences?  _PromotingCADMUS: Leadership in Thought that Leads to Action_ 2, 3 (2014), 23–36.
  * Poli (2017) Roberto Poli. 2017\.  _Introduction to Anticipation Studies_. Vol. 1.  Springer International Publishing AG, Electronic.  https://doi.org/10.1007/978-3-319-63023-6
  * Poli (2019) Roberto Poli (Ed.). 2019\.  _Handbook of Anticipation: Theoretical and Applied Aspects of the Use of Future in Decision Making_.  Springer International Publishing, Cham.  https://doi.org/10.1007/978-3-319-91554-8
  * Rek and Sojka (2010) R Rehr Uv Rek and P Sojka. 2010.  Software Framework for Topic Modelling with Large Corpora.  https://doi.org/10.13140/2.1.2393.1847 Publisher: Unpublished.
  * Rosen (1972) Robert Rosen (Ed.). 1972\.  _Foundations of Mathematical Biology: Subcellular Systems_ (1 ed.).  Academic Press, New York.  doi.org/10.1016/B978-0-12-597201-7.50006-1
  * Röder et al. (2015) Michael Röder, Andreas Both, and Alexander Hinneburg. 2015\.  Exploring the Space of Topic Coherence Measures. In _Proceedings of the Eighth ACM International Conference on Web Search and Data Mining_. ACM, Shanghai China, 399–408.  https://doi.org/10.1145/2684822.2685324
  * Skórski et al. (2023) Maciej Skórski, Alina Landowska, and Marek Robak. 2023\.  What Twitter Data Tell Us about the Future?  https://osf.io/z925y/
  * Spry et al. (2011) Amanda Spry, Ravi Pappu, and T. Bettina Cornwell. 2011\.  Celebrity endorsement, brand credibility and brand equity.  _European Journal of Marketing_ 45, 6 (Jan. 2011), 882–909.  https://doi.org/10.1108/03090561111119958 Publisher: Emerald Group Publishing Limited.
  * Terragni et al. (2021) Silvia Terragni, Elisabetta Fersini, Bruno Giovanni Galuzzi, Pietro Tropeano, and Antonio Candelieri. 2021.  OCTIS: Comparing and Optimizing Topic models is Simple!. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations_. Association for Computational Linguistics, Online, 263–270.  https://doi.org/10.18653/v1/2021.eacl-demos.31
  * Tooby and Cosmides (1996) John Tooby and Leda Cosmides. 1996.  Friendship and the banker’s paradox: Other pathways to the evolution of adaptations for altruism.  _Evolution of Social behaviour Patterns in Primates and Man. Proceedings of the British Academy_ 88, 297 (1996), 119–144–vi.
  * Tromble (2021) Rebekah Tromble. 2021\.  Where Have All the Data Gone? A Critical Reflection on Academic Digital Research in the Post-API Age.  _Social Media + Society_ 7, 1 (Jan. 2021), 205630512198892\.  https://doi.org/10.1177/2056305121988929