Predicting Pandemic Health Decisions and Outcomes Through Social Media and LLMs]Leveraging Prompt-Based Large Language Models: Predicting Pandemic Health Decisions and Outcomes Through Social Media Language

Xiaohan Ding

Department of Computer Science, Virginia Tech

Buse Carik

Department of Computer Science, Virginia Tech

Uma Sushmitha Gunturi

Department of Computer Science, Virginia Tech

Valerie Reyna

Human Neuroscience Institute, Cornell University
New York

Eugenia H. Rho

Department of Computer Science, Virginia Tech

We introduce a multi-step reasoning framework using prompt-based LLMs to examine the relationship between social media language patterns and trends in national health outcomes. Grounded in fuzzy-trace theory, which emphasizes the importance of “gists” of causal coherence in effective health communication, we introduce Role-Based Incremental Coaching (RBIC), a prompt-based LLM framework, to identify gists at-scale. Using RBIC, we systematically extract gists from subreddit discussions opposing COVID-19 health measures (Study 1). We then track how these gists evolve across key events (Study 2) and assess their influence on online engagement (Study 3). Finally, we investigate how the volume of gists is associated with national health trends like vaccine uptake and hospitalizations (Study 4). Our work is the first to empirically link social media linguistic patterns to real-world public health trends, highlighting the potential of prompt-based LLMs in identifying critical online discussion patterns that can form the basis of public health communication strategies.

<concept_desc>Do Not Use This Code, Generate the Correct Terms for Your Paper</concept_desc>
<concept_desc>Do Not Use This Code, Generate the Correct Terms for Your Paper</concept_desc>
<concept_desc>Do Not Use This Code, Generate the Correct Terms for Your Paper</concept_desc>
<concept_desc>Do Not Use This Code, Generate the Correct Terms for Your Paper</concept_desc>

[500]Human-centered computing Empirical studies in HCI
[300]Social Media/Online Communities
Natural language processing Large Language Models
[100]Quantitative Methods

Our research empirically links social media conversations to national health outcomes related to COVID-19. We introduce Role-Based Incremental Coaching (RBIC), a Large Language Model (LLM) prompting framework. (A) Collecting Reddit datasets focused on communities known for opposing COVID-19 health practices. (B) Guided by Fuzzy-Trace Theory, we introduce a novel LLM framework called Role-Based Incremental Coaching (RBIC) to extract cause-effect pairs and formulate coherent gists capturing causal relations in texts. (C) Granger-causality tests and data analytics reveal the impact of these gists on community engagement and national health outcomes such as vaccination uptake and hospitalization rates.

§ INTRODUCTION

During the COVID-19 pandemic, social media was at the center of proliferating mass antipathy and distrust towards government health policies and recommendations <cit.>. Millions took to social media to oppose federal and state health practices, criticize medical professionals, or organize anti-vaccine and mask-wearing rallies <cit.>. The viral growth of such online conversations fueled animosity and extremist views that encouraged people to resist public health guidelines <cit.>. Disregarding public health practices, such as wearing masks, maintaining social distance, and getting vaccinated resulted in significant societal costs. Between November and December of 2021 alone, over 692,000 preventable hospitalizations were reported among unvaccinated patients, leading to a staggering $13.8 billion <cit.>. Soaring COVID-19 infection cases put a massive burden on healthcare systems, depleting medical resources and contributing to severe employee burn-outs and shortages of healthcare workers <cit.>. Meanwhile, COVID-19 conspiracies and hyper-partisan news on social media led to nationwide protests, obstruction of medical facilities <cit.>, and even fatal assaults of employees requesting customers to wear masks <cit.>.

According to fuzzy-trace theory (FTT), texts that clearly establish cause-and-effect relationships facilitate humans extraction of gist mental representations, helping people understand and remember information better than texts without any causal coherence <cit.>. This aligns with previous studies in decision sciences, which have shown that causal coherence of gists in texts plays a crucial role in how individuals perceive risks and make health-related decisions <cit.>. Throughout the pandemic, social media conversations refuting COVID-19 public health practices based on mis-/disinformation and identity politics continued to obscure people’s knowledge of safe health practices, making well-informed health decisions extremely difficult <cit.>. Using evidence-based theories like FTT allows us to create psychologically descriptive models that transform language into analyzable units shown to predict human behavior <cit.>.

In this paper, we leverage the capabilities of prompt-based Large Language Models (LLMs) to delve into the nuanced language patterns in social media discussions opposing COVID-19 public health practices through a theory-driven approach using fuzzy-trace theory. By leveraging prompt-based LLMs, we dissect the language around resistance to COVID-19 health practices through the lens of FTT and its central concept of gist. Specifically, we examine how causal language patterns or gists manifest across social media communities that denounce pandemic health practices, contribute to trends in people's health decisions, and by extension, impact national health outcomes. We divide our work into four main studies to address the following research questions:

* RQ1. How can we efficiently predict gists across social media discourse at-scale? (Study 1)

* RQ2. What kind of gists characterize how and why people oppose COVID-19 public health practices? How do these gists evolve over time across key events? (Study 2)

* RQ3. Do gist patterns significantly predict patterns in online engagement across users in banned subreddits that oppose COVID-19 health practices? (Study 3)

* RQ4. Do gist patterns significantly predict trends in national health outcomes? (Study 4)

We answer RQ1 by leveraging LLMs and their prompt-based capabilities to identify gists in social media conversations at-scale (Study 1). We do so by developing a novel prompting framework that detects and extracts cause-effect pairs in sentences from a corpus of online discussions collected from banned Reddit communities known for opposing public COVID-19 health practices. Study 1 allows us to identify the causal language (cause-effect pairs that form gists) that underlie how people argue against COVID-19 health practices on social media. We answer RQ2 by clustering sentence embeddings of gists (sentences with causal relations identified from Study 1) to identify the most salient gist clusters, and demonstrate how they evolve across key events (Study 2). Finally, we answer RQs 3 and 4 by using Granger-causality to test whether causal discourse (gists) on social media can significantly predict online engagement patterns (Study 3), and trends in national health decisions and outcomes in the U.S. (Study 4).

Contributions: This work's intellectual merits are methodological and theoretical. The computational techniques introduced in this work enable efficient and scaled prediction of gists on social media, and thus can be used to better identify and understand underlying mental representations that motivate health decisions and attitudes towards public health practices (Study 1). The clustering and evolution of gists in Study 2 identify the most salient themes associated with how people causally argue against pandemic health practices online. Patterns in gist volumes across cluster topics fluctuate closely with topically-related high-profile events, including federal health announcements, congressional policies, and remarks by a country’s leader. Study 3 empirically confirms how gist volumes significantly drive subreddit engagement patterns (upvotes and comments), providing implications for how causal language may play a role in monitoring conversations in content-moderation practices of controversial online health communities. Finally, gist patterns within subreddits that support anti-pandemic health practices were significantly interrelated with nationwide trends in important health decisions and outcomes (Study 4). To the best of our knowledge, our research is the first to empirically establish Granger causality between linguistic patterns in social media discussions about COVID-19 health measures and real-world trends in public health outcomes. Our work entails the following contributions:

* The task of accurately predicting causal language patterns and generating coherent gists (causal statements) is a complex challenge <cit.>. We overcome this by introducing a multi-step prompting framework: Role-Based Incremental Coaching (RBIC). RBIC is a prompting mechanism that allows efficient prediction of gists across social media conversations at-scale. RBIC integrates role-based cognition with effective learning in sub-tasks to enhance the model's overall understanding of a given task prior to generating a final output. We overcome prior challenges in detecting subtle and complex expressions of semantic causality in noisy text by leveraging RBIC. By doing so, this work advances state-of-the-art approaches in detecting gists at-scale, yielding a novel, psychologically relevant, and efficient technique for identifying and examining bottom-line meanings in massive amounts of textual data.

* We demonstrate the novel application of prompt-based LLMs in advancing computational social science (CSS) methods in Human-Computer Interaction (HCI) research. Generic Natural Language Processing (NLP) models and LLMs typically lack multi-step reasoning capabilities <cit.>. This limitation makes it difficult to apply such models in performing nuanced and complex text analyses in CSS research <cit.>.  By applying RBIC, we overcome this limitation and demonstrate the versatility and effectiveness of prompt-based LLMs in identifying and synthesizing nuanced linguistic patterns. In so doing, we contribute to broadening the potential application of prompt-based LLMs for theory-driven textual analysis in CSS research in the HCI domain.

* Our research enhances the analytical depth and scope of insights into the causal discourse surrounding people’s opposition to public health practices on social media. We identify the most salient gist clusters that embody the core topics at the center of how and why people oppose public health practices throughout COVID-19, from May 2020 to October 2021. We use sentence embeddings and clustering to provide a characterization of how the volume of gists across each topic fluctuates in relation to key events associated with the core topics embodied by the gist clusters. By doing so, we capture how causal online discourse surrounding anti-COVID-19 health practices evolves over time across real-world events. Such insights can, in turn, inform timely public health communication strategies and interventions that account for ongoing current events <cit.>.

* Finally, we address the question of whether and how social media language patterns in the form of gists influence nationwide trends in vaccinations, COVID-19 cases, and hospitalization in the U.S., providing new evidence around how important health decisions and national health outcomes are impacted by causal linguistic signatures across social media health discussions—an important link that has not been empirically established at-scale in prior research.

§ RELATED WORK

§.§ Understanding the Impact of Social Media Language Patterns on Health Decisions and Outcomes

The COVID-19 pandemic has ignited an unprecedented increase in social media discourse on health decisions and practices <cit.>, spurring a wave of computational social science research <cit.> aimed at understanding this phenomenon in the field of HCI <cit.> and CSCW <cit.>. Using text mining and computational linguistics, researchers have analyzed pandemic-related social media discourse through the lens of mental health <cit.>,  political views <cit.>, attitudes towards vaccines <cit.>, misinformation <cit.>, and perceptions of health policies and government institutions <cit.>. Such studies have uncovered key insights on how language patterns reflect people’s beliefs <cit.>, sentiments <cit.>, and emotional well-being <cit.> during Covid-19. For example, researchers have examined collective shifts in the public mood in response to the evolving pandemic news cycles by analyzing the daily sentiment of tweets <cit.>. Similarly, others have analyzed social media posts containing a subset of depression-indicative n-grams to track the fluctuation in mental health of social media users over the course of the pandemic <cit.>.

While such studies have made valuable contributions to understanding the role of language patterns in health-related discourse on social media <cit.>, there remains an opportunity to explore their impact on real-world health decisions and outcomes. To the best of our knowledge, there has been a lack of research that examines how social media discussion patterns surrounding health practices can predict patterns in health decisions and outcomes in the real world. Our research aims to fill this gap. Some emerging research, such as the study by Nyawa et al. (2022), has started to explore this link by applying computational linguistics to categorize individuals as either vaccine-accepting or vaccine-hesitant based on their online language patterns <cit.>. Yet, the majority of empirical studies examining the impact of social media discourse on real-world behavior thus far have leaned heavily on survey-based methods <cit.>. These surveys often depend on self-reported metrics about social media use and health behaviors, thereby offering only a limited perspective on the complex relationship between social media discourse patterns and actual health decisions. This limitation underscores the existing challenges in understanding how health-related discussions on the internet translate into or shape real-world outcomes and decisions <cit.>.  Our research aims to address this challenge by investigating how language patterns in social media conversations can serve as predictive markers for understanding real-world trends in people’s health decisions and outcomes during the pandemic.

§.§ Understanding Health Discourse Through the Lens of Fuzzy-Trace Theory and Its Core Concept of Gist

Scholars have used fuzzy-trace theory (FTT) as a theoretical lens to explore risk perceptions and decisions underlying health practices and discussions in various contexts, including vaccines <cit.>, cancer <cit.>, HIV-AIDS <cit.> and the prescription of antibiotics <cit.>. These studies support FTT’s core tenet that gists are stronger and more effective forms of communication than verbatim representations in the sense that they are (a) better remembered and (b) more likely to influence decisions <cit.>.

For example, a study comparing articles on vaccines posted on Facebook showed that those containing gists (e.g., bottom-line meaning) are shared 2.4 times more often on average than articles with verbatim details (e.g., statistics) <cit.>. Having a story or images did not add unique variance to predictions once gist was accounted for. The study’s results show that communications about vaccines are more widespread when they express a clear gist explaining the bottom-line meaning of the statistics rather than just the data themselves. Likewise, scholars have also used FTT as a theoretical framework to examine people's behavior across diverse contexts, such as law, medicine, public health, systems engineering, and HCI <cit.>.
For example, in HCI, researchers have used FTT to examine people's behavior in online social tagging <cit.> and to improve speech-to-text interface design through gist-based communication <cit.>. Others have used FTT in designing a web-based intelligent tutoring system for communicating the genetic risk of breast cancer through gists <cit.>.
Overall, FTT’s theoretical breadth and empirical support as a cognitive explanation of how people process and communicate information related to health decisions makes FTT a well-suited theoretical lens to examine resistance towards public health practices in our research.

Further, gists that causally link some event, actor, or outcome tend to facilitate more effective uptake of information than those that are less causally coherent <cit.>. In fact, causal coherence is one of the most important semantic aspects of gists that make gist-based communications effective <cit.>. For example, in a study analyzing 9,845 vaccine-related tweets, researchers discovered that tweets containing explicitly causal gists (e.g., "vaccines cause autism") were far more likely to be retweeted and to go viral. This was in contrast to tweets that suggested a link between vaccines and autism but emphasized details and lacked a meaningful causal connection <cit.>. Simply, information with stronger causal structure produces more meaningful gists in people, who then are more likely to remember, apply, and share that information <cit.>. Fuzzy-trace theory draws on psycholinguistic research on mental representations of narratives that underlies both human memory models and computational models in which causal connections are a central feature of common gists <cit.>.
Hence, we focus on causal gists, or gists that contain a cause-effect relation. From hereon, we refer to causal gists as gists.

§.§ Challenges in Predicting Semantic Causality in Online Health Discourse

Extracting cause-effect relations in text is one of the many open challenges in NLP research that has seen significant breakthroughs in recent years through the development of generative Large Language Models <cit.>. However, computational social science research has yet to take advantage of these advancements <cit.>, particularly in examining gists related to health practices. For example, scholars have used topic modeling, such as Latent Dirichlet Allocation (LDA) <cit.> to identify gists in vaccine hesitancy <cit.>. While useful, these methods do not enable granular detection of gists at the sentence or phrase level. For instance, LDA only allows the detection of gists at the corpus level, where each identified topic across the entire dataset is treated as a proxy identification of one gist. Recent scholarship in medical informatics has examined health-related attitudes in social media by extracting causality through machine learning approaches with rule-based dependency parsing and named entity recognition <cit.>. While such approaches are an improvement, they can only detect intra-sentential (within a single sentence) and not inter-sentential causality where cause and effect lie in different sentences (e.g., God made us to breathe naturally. I won’t be forced to wear masks.). More recently, transformer models such as InferBERT and CausalBERT, specifically designed for extracting causal relationships, have yielded more promising results <cit.>. However, the token limit of these models significantly reduces performance when dealing with longer texts <cit.>. Additionally, like humans, these models struggle to discern subtle forms of semantic causality in noisy or incoherent data. Our research aims to not only identify causality in text, but also generate coherent gists based on the identified cause-effect pairs. To achieve this, we address prior limitations by leveraging recent advancements in pretrained LLMs and their prompt-based approaches to develop a novel prompting framework to systematically predict gists <cit.>.

§ STUDY 1: PREDICTING GISTS IN SOCIAL MEDIA CONVERSATIONS AT-SCALE

As a first step to analyzing how causal language patterns on social media impact health decisions and outcomes, we leverage the power of prompt-based LLMs in Study 1. Specifically, we develop and apply a multi-step prompting framework called Role-Based Incremental Coaching (RBIC) to efficiently predict gists across social media discourse at-scale. Role-Based Incremental Coaching is a prompting framework (Fig. <ref>) built with few-shot demonstrations using GPT-4, which consists of two primary prompting techniques: Role-Based Knowledge Generation and Incremental Coaching. Combined together, RBIC allows the model to  (1) learn its role for a given task by generating role-specific knowledge as a task-performing agent and (2) perform a series of small sub-tasks to refine its understanding and quality of the final output by incrementally building upon the sub-task responses. RBIC allows us to systematically identify the presence of semantic causality in a given post, and generate causally coherent gists across large volumes of textual corpora at-scale.

Illustration of the Role-Based Incremental Coaching (RBIC) prompting framework: RBIC incorporates role-based cognition and sub-task training to improve the model's comprehension of a specific task before generating the final output.

§.§ Data

We collected all publicly available posts from 20 anti-COVID-19 subreddits that were banned for denouncing COVID-19 public health practices. These subreddits were chosen based on their community size, as well as the significant media attention they received from major news outlets <cit.>, and their virality among American social media users <cit.>. We obtained all posts and corresponding metadata (comments, post id, timestamp, up/down-vote ratio, etc.) for each of these subreddits using Pushift API. This resulted in a total of 79,680 posts spanning from May 2020 to October 2021 from the following subreddits: conspiracy_commons, CoronavirusCirclejerk, CoronavirusFOS, Coronavirus_Rights, COVID19, covid19_testimonials, covidvaccinateduncut, VaxKampf, DebateVaccines, FauciForPrison, ivermectin, lockdownskepticism, NoNewNormal, trueantivaccination, vaccinelonghaulers, VAERSreports, Wuhan_Flu, CovidIsAFraud, COVID19Origin, churchofcovid.

§.§ Method: Role-Based Incremental Coaching (RBIC)

Role-Based Knowledge Generation. Drawing inspiration from prior NLP research that leverages multi-step reasoning capabilities in LLMs <cit.>, we developed Role-Based Knowledge Generation as the initial grounding component of our prompting framework. Before producing a final response from LLMs, asking LLMs to generate potentially useful information about a given task improves the final response <cit.>. For example, as shown in an open online course "Learn Prompting" [https://learnprompting.org/], when prompted with "Which country is larger, Congo or South Africa?", GPT-3  answers incorrectly. However, when the model is prompted to "Generate some knowledge about the sizes of South Africa and Congo", before answering the final question, the model uses the output to the intermediate prompt ("South Africa [has] an area of...") to generate the correct answer: Congo is larger than South Africa. We leverage this prompting intuition in Role-Based Knowledge Generation to enhance the model's understanding of its role as a task-performing agent. By doing so, the model can achieve better performance by accessing potentially relevant contextual information, as shown in prompts, P1 and P2 (Fig. <ref>). The corresponding outputs to P1 and P2 - O1 and O2, respectively -  are then integrated with a task-specific prompt (P3) in the following step. The role-based knowledge outputs (O1 and O2) allow the model to perform tasks more accurately given its enhanced understanding of its specific role for achieving the task.

Incremental Coaching. Inspired by Chain of Thought (CoT) <cit.>, Incremental Coaching is a technique within the Role-Based Incremental Coaching (RBIC) framework that involves breaking down a complex task into smaller, manageable sub-tasks as shown in P3-P5 in Fig. <ref>. The role-based agent is coached through a series of sub-tasks in a step-by-step manner, with each sub-task building upon the previous one. To implement Incremental Coaching effectively within RBIC, it is necessary to follow a logical sequence of sub-task prompts that allows the model to build understanding and confidence in performing the final task by generating incremental outputs (O3-O4). By breaking down the final task into a series of incremental sub-tasks, the role-based agent can gradually improve its comprehension of the final task to deliver a more accurate final response.

Role-Based Incremental Coaching (RBIC)

[1]
User Input ($P$), Role-Based Agent (RBA)
Knowledge Base (KB) Final Task Output ($F$)

Comment: Role-Based Knowledge Generation
Input: User Input $P$
Output: Intermediate knowledge $K$
$K \leftarrow \text{RBA.GenerateKnowledge}(P)$
Update Knowledge Base (KB) with $K$

Comment: Incremental Coaching
Input: Final Complex Task $T$, Sequence of Sub-Tasks: $T = \{S_1, S_2, \ldots, S_n\}$
Output: Incremental outputs $O_i = \{O_1, O_2, \ldots, O_n\}$ and Final Task Output $F$
$i = 1$ $n$
$O_i \leftarrow \text{RBA.Coach}(S_i, \text{KB})$
Update KB with $O_i$
$F \leftarrow \text{RBA.FinalOutput}(KB,T)$

Application of RBIC. Here, we demonstrate the algorithmic conceptualization of the RBIC prompting framework in the context of generating gists. The essence of the Role-Based Incremental Coaching (RBIC) framework lies in its two core algorithmic components: Role-Based Knowledge Generation and Incremental Coaching, as shown in Algorithm 1. The RBIC algorithm requires the following inputs:

* User Input \(P\): The RBIC is initialized by the user input. For example, in our study, we operationalized user input as $P = (P1, P2, P3, P4A \text{ or } P4B, P5)$, as shown in Fig.<ref>.

* Role-Based Agent: Essentially, this can be any prompt-based LLM. For our study, we used GPT-4 as our Role-Based Agent.

Next, the RBIC algorithm will generate the following output:

* Knowledge Base (KB): The first phase of the RBIC algorithm, denoted as Role-Based Knowledge Generation, is symbolized by the function \( \text{RBA.GenerateKnowledge}(P) \). In this step, the Role-Based Agent (in our case, we use GPT-4, but this can be substituted with any prompt-based LLMs) is prompted with a user input \( P \) to elicit relevant background knowledge \( K \). This knowledge forms the basis for task execution and is stored in an initial Knowledge Base (KB).

\begin{equation}
K \leftarrow \text{RBA.GenerateKnowledge}(P)
\end{equation}

Here, \( K \) represents the knowledge generated, and \( P \) represents the user input posed by the user. \( \leftarrow \) signifies the assignment of generated knowledge \( K \) to the Knowledge Base (KB), thus creating a dynamic knowledge architecture that adapts over time. For instance, in our study, \( K \) comprised of O1 and O2 (as shown in the upper right of Fig. <ref>), which collectively formed our Knowledge Base (KB).

* Final Task Output (F): The subsequent phase, known as Incremental Coaching, is predicated on a sequence of sub-tasks \(\{S_1, S_2, \ldots, S_n\} \) and their corresponding outputs:

\( \{O_1, O_2, \ldots, O_n\} \).

\begin{equation}
O_i \leftarrow \text{RBA.Coach}(S_i, \text{KB})
\end{equation}

In this phase, each sub-task \( S_i \) leverages the updated Knowledge Base (KB) to produce an output \( O_i \). \( O_i \) is then used to update the KB, thus iteratively coaching the model through a series of sub-tasks in a step-by-step manner. Breaking down the final Complex Task \(T\) into simpler sub-tasks \( S_i \) allows the model to incrementally build up the necessary knowledge and skills to tackle the final task. Therefore, this incremental knowledge building across sub-tasks enables the model to better understand and perform the final Complex Task \(T\). In our case, our Complex Task ($T$) generates a "gist" based on the cause-effect pairs. The individual sub-tasks that contribute to this complex task are labeled as P3, P4A, P4B and P5 (Fig. <ref>). The algorithm proceeds sequentially, producing intermediate outputs O3, O4, and ultimately culminating in O5, which is the gist generated from the cause and effect pairs identified in sub-task P4A.

When applied to predicting gists in social media conversations, RBIC instructs the model to understand the concept of cause-effect relations as a task-performing agent. The model then incrementally performs sub-tasks to recognize and extract cause-effect pairs, and finally generates a concise gist that captures the essence of the identified causal relationship.

Sample results from applying the RBIC method for extracting cause-effect relationships and generating gists from Reddit posts discussing health mandates between May 2020 and October 2021, including the post content

customcolor2 No.     Reddit Post     Label     Cause     Effect     Gist
customcolor3 1   99.995% of children survive cv infection, why are they pushing so hard to have kids take an experimental vaccine?    Yes     they are pushing so hard to have kids take an experimental vaccine     99.995% of children survive cv infection     Despite the high survival rate of children from cv infection, there is a push to have kids take an experimental vaccine.

customcolor42   Had my Pfizer jab last Wed and have felt like death since.     Yes     Had my Pfizer jab     have felt like death since     The cause of feeling like death is having the Pfizer jab last Wednesday.

customcolor33   Imagine pointing and laughing at a single father of 3 who's now jobless and has to take care of 3 kids with no income all because he didn't want to wear a face diaper or take the experimental gene modification.     Yes     he didn't want to wear a face diaper or take the experimental gene modification     he's now jobless and has to take care of 3 kids with no income     The man's refusal to wear a face mask or take the experimental gene modification led to him losing his job and being unable to provide for his three children, resulting in financial hardship and increased responsibility for him.

customcolor44   LA Fitness cancelled my membership against my will today because I refused to wear a mask.     Yes     I refused to wear a mask     LA Fitness cancelled my membership     The cause of LA Fitness cancelling the membership was the refusal of the person to wear a mask, which led to the effect of the membership being canceled against their will.

customcolor35   I’ve been thinking a lot about COVID data that’s been circulating and want to share some thoughts. I think it’s essential to remember that COVID data is not beyond skepticism, because what counts as a case varies.     Yes     what counts as a case varies     COVID data is not beyond skepticism     The variation in what is considered a COVID case has led to skepticism about the accuracy and reliability of COVID data.

customcolor46   I stumbled upon some news. Governor Wolf has a false positive, won’t admit it because it would be admitting the tests are unreliable. And do you think it’s possible that politicians might hide their own false positive results to maintain confidence in the testing system?     Yes     false positive, won’t admit it     it would be admitting the tests are unreliable     Governor Wolf won’t admit a false positive because it would undermine COVID-19 test reliability, potentially affecting public health and safety measures.

§.§.§ Human Evaluation.

To assess the effectiveness of RBIC's application in predicting gists in our data, we conducted a human evaluation of the RBIC-generated outputs. We recruited 6 human evaluators to evaluate the presence of causal coherence (O3), cause-effect pairs (O4), and gists (O5) for each Reddit post based on the following criteria:

* Accuracy (classification): Is there a cause-effect relationship in the post (1/0; Yes/No)?

* Relevance (extraction): How well does the cause-effect pair capture the primary causal relationship in the post (1-5; not well at all, slightly well, moderately well, very well, extremely well)?

* Conciseness (generation): How well does the gist concisely summarize the cause-effect relationship in the post (1-5; not well at all, slightly well, moderately well, very well, extremely well)?

To mitigate error propagation, the evaluation was designed as a sequential process but with checks for accuracy. First, evaluators focused on `Accuracy', verifying the presence of a cause-effect relationship. Second, `Relevance' was examined to ensure the identified cause-effect pairs accurately reflected the post's main causal relationship. The final and third evaluation stage, `Conciseness', was only evaluated in posts that had already met the `Accuracy' and `Relevance' criteria. This approach minimized propagation of errors from earlier stages.

The accuracy criteria assesses the model's performance in identifying the presence of a causal relationship in a post. Relevance evaluates the model's ability to correctly extract the cause and effect phrases that are most salient to the core message of the post's content. Conciseness assesses the model's generative performance in concisely synthesizing a coherent gist based on the identified cause and effect phrases. In total, each of the 6 annotators evaluated 3,100 posts that were randomly selected from the entire dataset. For each criteria, each post received three evaluation scores from three annotators. The evaluators' assessment of the model's performance across the three criteria were generally high based on inter-rater agreement scores using Fleiss kappa ($k$) <cit.>: accuracy ($k$ = $0.892$); relevance (mean = $4.3$, $k$ = $0.839$); conciseness (mean = $4.5$, $k$ = $0.864$).

§.§ Result

Table <ref> presents the results of RBIC's application, demonstrating the effectiveness of our prompting framework in predicting gists at-scale. We identified a total of 6,861 gists in our data. As shown, RBIC cannot only detect semantic causality (O3), but also extract verbatim phrases corresponding to the main cause-effect pairs (O4), and generate coherent gists (O5) based on the identified pairs. In the first example, RBIC detects sentences where causality is implied with nuance, as well as those that are more explicitly stated. Although most of the gists accurately capture the semantic essence of the causal relationship, some are more eloquent than others. For instance, the gists in examples 2 and 4 use sentence inversions, beginning with "the cause of", while others are more semantically fluid. We also performed a comparison using fine-tuned language models (BERT, RoBERTa and XLNet), as detailed in the appendix (Table <ref>), which showed that RBIC outperformed the baseline models in extracting cause-effect pairs (O4) by 26.6% in F1-score when comparing RBIC to the best-performing baseline model (RoBERTa with 0.814 F1-score).

§ STUDY 2: HOW GISTS EVOLVE OVER TIME

Given the rapidly evolving public health discussions on social media, it is crucial to examine how they evolve over time <cit.>. This enables a better understanding of shifts in public opinion and emerging concerns across contentious debates around health practices like vaccinations, mask-wearing, and social-distancing <cit.>. Hence in Study 2, we build upon our Study 1 findings to address: What kind of gists characterize how and why people oppose COVID-19 public health practices? How do these gists evolve over time across key events?  To answer these questions, we extract sentence embeddings from each of the gists identified from Study 1, and cluster the embeddings to identify distinct gist clusters that characterize the core topics at the center of how people argue against COVID-19 health practices.

§.§ Method

§.§.§ Extracting Sentence Embeddings from Gists.

To identify the most salient topics across the causal language (gists) surrounding the social media discourse against public health practices, we use Sentence-BERT (S-BERT) to extract semantically rich representations of the gists identified in Study 1. S-BERT is a transformer-based model designed to produce contextualized sentence embeddings, which are particularly valuable in clustering texts  <cit.>. After preprocessing the gists with standard text cleaning operations (lowercasing, removal of special characters,  tokenization), we implemented S-BERT using the SentenceTransformer to extract embeddings from our gists. The S-BERT model comprises 12 hidden layers, with each layer producing an output representation of $1 (\mathcal{N}) \times 768 (\mathcal{M})$ dimensions. To obtain high-quality embeddings, we extracted the output representations from each of the last three hidden layers of the model (layers 10-12), and computed their means. By doing so, we are able to capture and generate semantically rich representation of each gist as high dimensional vectors.

§.§.§ Clustering of Sentence Embeddings.

After obtaining the sentence embeddings, we applied Principal Component Analysis (PCA) <cit.> to reduce the dimensionality of the embeddings prior to the clustering step. This was done to better visualize the language embeddings in a lower-dimensional space and to facilitate a more effective interpretation of the embedding results. We selected PCA as our method, given its frequent use and proven effectiveness in reducing dimensionality, especially for language embeddings <cit.>. We used k-means <cit.> for clustering, as it is especially reliable for clustering semantic word representations <cit.>. The k-means algorithm iteratively assigns each embedding to a cluster with the closest centroid, and updates the centroid by calculating the mean of the embeddings assigned to the cluster <cit.>. This process continues until the centroids stabilize. To enhance the reliability and robustness of our clustering approach, we incorporated sentence embeddings of posts that did not contain any gists, following Samosir's study <cit.>. This step allows us to assess the quality of sentence embeddings by verifying that embeddings from sentences that do not contain gists cluster apart from embeddings derived from gists. Finally, we used the elbow method <cit.> to determine the optimal number of clusters by calculating the sum of squared errors (SSE) in ascending order of cluster numbers until additional clusters resulted in diminishing returns <cit.>.

§.§.§ Verifying Gist Clusters.

The first author initially identified the primary themes of each cluster through categorization, screening, and summarization of 200 randomly selected gists from each cluster. Next, we recruited 6 annotators to manually evaluate and verify five primary gist clusters, as shown in Table <ref>. Annotators manually evaluated the clustering results by iteratively examining and discussing the themes across 200 randomly selected gists belonging to each cluster (1/0). See annotation agreement in Table <ref>. The verification process also includes two additional steps: (1) refining cluster descriptions such that they were thematically salient and representative of the core ideas and topics embodied by the gists in each cluster and (2) examining the sentences in the non-gist cluster that did not include any gists (non-gist cluster is C6).

Inter-rater reliability scores for human evaluation of topic clustering results. Fleiss' kappa coefficient was calculated to assess agreement between six annotators judging whether each gist was correctly assigned to one of the five clusters listed.

customcolor2 Cluster     IRR value (Fleiss $k$)

C1. Implications of Vaccine Policies, Efficacy, Side-Effects     0.926
C2. Controversies Related to Masks-Wearing Practices      0.894
C3. Impact of Lockdown       0.884
C4. Societal and Economic (Macro) Impact of COVID-19     0.902
C5. Conspiracy Theories, Domestic Politics, Foreign Countries     0.821
C6. Lack of distinct causal relationship or coherent gists     0.980

§.§ Result

A representative sample of gists from each cluster is presented in Table <ref>, illustrating the core topics that characterize the opposition discourse surrounding pandemic health practices. In Fig. <ref>,  (top panel), we visualize the evolution of our gist clusters across four time points (May 2020 - October 2021).

Representative examples are shown for each cluster from C1 to C5, highlighting the main ideas identified in the health mandate debate on Reddit. Cluster 6 is not included, as it lacks a distinct causal relationship or coherent gist.

customcolor2 Cluster           Gist
customcolor3 53.0cmC1. Implications of Vaccine Policies, Efficiency, Side-Effects

* The implementation of a vaccine mandate has resulted in people losing their jobs.

* The use of experimental COVID vaccines is causing an increase in COVID deaths.

* The vaccine was ineffective against new variants, which led to the death of 7,000 people who received the spike protein mRNA jab, including little kids. This suggests that the vaccine was administered for no reason, as it failed to provide protection against the new variants.

customcolor4 53.0cmC2. Controversies Related to Masks-Wearing Practices

* If a person refuses to wear a mask at a business for medical reasons, the business may deny them services.

* The lifting of mask mandates for vaccinated individuals has caused the proliferation of a deadly biohazard, which could lead to the CDC and other agencies being charged with involuntary manslaughter.

* Wearing masks prevents people from seeing each other's faces, which leads to difficulties in understanding and building trust with others.

customcolor3 63.0cmC3. Impact of Lockdown

* The lockdowns have caused tourism-dependent islands in Thailand to suffer from a lack of income, leading to a situation where they have been on food aid for over a year.

* The lockdowns caused a loved one to almost commit suicide, highlighting the negative impact of lockdowns on mental health.

* The prolonged lockdown imposed by Cuomo for six months has resulted in the inability of the speaker to pay their bills.

customcolor4 53.0cmC4. Societal and Economic (Macro) Impact of COVID-19

* The outbreak of COVID-19 has caused people to struggle with their livelihood, leading to financial difficulties and economic instability.

* The COVID-19 pandemic has caused the biggest drop in US life expectancy since the second world war.

* The COVID-19 shutdowns have resulted in 1 in 5 churches facing permanent closure within 18 months due to the financial strain caused by the pandemic.

customcolor3 83.0cmC5. Conspiracy Theories, Domestic Politics, Foreign Countries

* People refuse to share a table or work with certain people because they see "certain people" as sub-human because of their vaccination status.

* The sentence suggests that if COVID-19 was intentionally released, it would lead to a major benefit for China and billionaires. The implication is that the cause of COVID-19's intentional release would be to bring about this benefit for these parties.

* The lack of information on the epidemic from people on whether they think something is safe or not is preventing the speaker from being able to debate with their conspiracy theory friends.

The upper portion of the illustration displays the progression of clusters across four-month periods. The line graph illustrates the month-by-month evolution of the number of posts containing gists, representing the central themes discussed on Reddit concerning health mandates. The graph highlights specific dates when each topic was most prominently discussed and presents relevant news events related to COVID-19 and health mandates during those periods.

The visualization reveals interesting relations between the clusters. For instance,
cluster 4, which embodies gists discussing the impact of COVID-19 on the economy and society at-large  wraps around
cluster 3 gists related to the impact of lockdown policies. This spatial proximity suggests that causal discussions on the broader consequences of the pandemic are closely intertwined with gist-based conversations on the impact of lockdown measures, shedding light on the interconnectedness of how people talk about these two topics in a causal manner.  Similarly, clusters 1 and 2 are not only close in proximity, but also similar, in terms of position and shape: both clusters are diagonally positioned from top left to the bottom right, and run parallel to each other. Given that both clusters represent gists concerning specific health practices (vaccinations, mask-wearing), it is likely that these topics may share similarities in the causal manner in which people talk about the effectiveness of such health practices. Cluster 5, which represents gists related to conspiracy theories, domestic politics, and foreign countries appears to lack a clear boundary and is spatially dispersed compared to other clusters. This could be due to the fact that cluster 5 encompasses multiple topics, as indicated by its description, in contrast to other clusters that are more uniformly focused on specific health practices,  government measures, or particular aspects of the pandemic's impact. The lower inter-rater reliability agreement for cluster 5 (Fleiss $k$ = 0.821) further supports the notion that it is a heterogeneous cluster consisting of various topics compared to other clusters.

The bottom half of Fig. <ref> demonstrates that peak volumes of gists within each cluster align closely with key events related to the respective topics embodied by those clusters. To identify these key events, we relied on reports from major health organizations including the Centers for Disease Control and Prevention (CDC), World Health Organization (WHO), and United Nations (UN) for announcements related to public health interventions like lockdowns and vaccine rollouts <cit.>. News reports from these organizations were widely recognized as authoritative information sources across the global community. Hence, we used such announcements and reports from these sources that highlighted key pandemic events, public health announcements, and significant milestones across the timeline of COVID-19.

In addition to these organization reports, we also analyzed articles related to COVID-19 published by major news outlets, such as AP News, Reuters, CNN, Fox News, Wall Street Journal, New York Times, and NPR. We then identified highly mediatized events by using the number of shares and article comments. This process also entailed iterative discussions among all the authors to ensure a comprehensive and balanced selection of events. Our approach aimed to minimize biases by incorporating a diverse range of sources and validating the significance of events through multiple indicators such as media coverage intensity and public engagement.

For example, cluster 1 peak occurs in November 2020, coinciding with the country's initial phase of vaccine distributions to healthcare workers and high-risk groups <cit.>. Similarly, cluster 2 gists (mask-wearing) peaks in June 2021, the same month in which the federal mask mandate is lifted <cit.>. Cluster 3 gists, which relate to the impact of lockdowns, peak in May 2020, by which approximately 4.2 billion or  54% of the world's population was under lockdown <cit.>. In December 2020, the U.S. Congress passed a bill to distribute $90 billion in stimulus checks to households, as nearly 30 million American adults reported food and income insecurity in the same month <cit.>. These events temporally coincide with the peak in cluster 4 gists, which concerns the socioeconomic consequences of the pandemic. Finally, cluster 5 gists, which are related to conspiracy theories, politics, and foreign countries, reached their peak volume in August 2020, around the time when President Trump retweeted a popular online conspiracy theory <cit.> and referred to the "China virus" in his White House briefing <cit.>. Our findings imply that trends in gist volumes are linked with real-world events.

§ STUDY 3: HOW SOCIAL MEDIA GIST PATTERNS INFLUENCE ONLINE ENGAGEMENT BEHAVIOR

Delineating key semantic patterns (e.g., gists) that drive online behavior can help gain insight into how social media language impacts the dissemination of health information online. This, in turn, can better inform public communication strategies for time-sensitive health interventions.  Hence, in Study 3, we use Granger-causality to examine the extent to which gist patterns influence online engagement, such as up-voting and commenting in subreddit communities that oppose COVID-19 health practices.

§.§ Hypothesis Testing with Granger Causality

Granger causality determines whether a time series $\mathcal{X}$ is meaningful in forecasting another time series $\mathcal{Y}$ <cit.>. For two aligned time series $\mathcal{X}$ and $\mathcal{Y}$, it can be said that $\mathcal{X}$ Granger-causes $\mathcal{Y}$ if past values $\mathcal{X}_{t-l} \in \mathcal{X}$ lead to better predictions of the current $\mathcal{Y}_t \in \mathcal{Y}$ than do the past values $\mathcal{Y}_{t-l} \in \mathcal{Y}$ alone, where $t$ is the time point and $l$ is the lag time or the time interval unit in which changes in $\mathcal{X}$ are observed in $\mathcal{Y}$. Lag time $l$ in Granger causality refers to the delay between a change in one time series potentially causing a change in another, indicating the time it takes for the effect to be observed. We used Granger-causality to test hypotheses 1-2, as shown below, as well as the reversed variations of H1 and H2 (H1R and H2R) where $i$ ranges from $1$ to $5$.

arc=1mm, auto outer arc,

H1. The daily volume of gists in cluster $i$ significantly Granger-causes the upvote ratio of Reddit posts containing gists in cluster $i$.

H2. The daily volume of gists in cluster $i$ significantly Granger-causes the number of comments associated with Reddit posts containing gists in cluster $i$.

§.§ Method and Analysis

First, we constructed the time series data,  $\mathcal{T}_G$ for each cluster, where $\mathcal{T}_{G_i}$ represents the daily number of gists in cluster i spanning from May 2020 to October 2021. We then created two more temporally corresponding time series data,   $\mathcal{T}_{U}$ and $\mathcal{T}_{C}$, which represent the daily upvote ratio and the daily comment count for each Reddit post containing gists from cluster $i$, respectively. We conducted a total of 20 Granger causality tests (5 clusters $\times$ 4 hypotheses - H1, H2, H1R, H2R), using time lags ranging from 1 to 14 days.  To ensure that the value of the time series was not merely a function of time, we conducted the Augmented Dickey-Fuller (ADF) test <cit.> using the serial difference method to achieve stationarity with ADF test values exceeding the 5% threshold.

Granger causality test results analyzing the relationships between the daily volume of gists in Clusters (C1-C5) and online engagement behavior - upvote ratios (UR),  number of comments (NC) -  across Reddit discussions.

customcolor2 Cluster     Behavior     Hypotheses     Lag (days)     $F$-value     $p$-value

upvote ratio (UR)     C1 $\not\to\mathcal{}$ UR     4     4.228     0.022

23cmC1. Vaccination Implications     upvote ratio (UR)     UR $\not\to\mathcal{}$ C1     3     1.739     0.277

number of comments (NC)     C1 $\not\to\mathcal{}$ NC     4     3.090     0.032

number of comments (NC)     NC $\not\to\mathcal{}$ C1     7     1.107     0.445

upvote ratio (UR)     C2 $\not\to\mathcal{}$ UR     6     5.818     0.012

23.4cmC2. Controversies and Policies Related to Masks     number of comments (NC)     C2 $\not\to\mathcal{}$ NC     7     6.007     0.014

upvote ratio (UR)     UR $\not\to\mathcal{}$ C2     6     4.738     0.022

number of comments (NC)     NC $\not\to\mathcal{}$ C2     9     0.771     0.463

upvote ratio (UR)     C3 $\not\to\mathcal{}$ UR     2     0.712     0.545

22.9cmC3. Impact of Lockdown     upvote ratio (UR)     UR $\not\to\mathcal{}$ C3     7     3.410     0.033

number of comments (NC)     C3 $\not\to\mathcal{}$ NC     3     0.715     0.496

number of comments (NC)     NC $\not\to\mathcal{}$ C3     8     6.121     0.016

upvote ratio (UR)     C4 $\not\to\mathcal{}$ UR     3     6.485     0.011

22.9cmC4. Societal and Economic (Macro) Impact of Covid-19     upvote ratio (UR)     UR $\not\to\mathcal{}$ C4     7     2.011     0.092

number of comments (NC)     C4 $\not\to\mathcal{}$ NC     2     7.912     0.002

number of comments (NC)     NC $\not\to\mathcal{}$ C4     4     0.815     0.413

upvote ratio (UR)     C5 $\not\to\mathcal{}$ UR     5     1.441     0.512

22.9cmC5. Social Issues, Health, and Personal Experiences     upvote ratio (UR)     UR $\not\to\mathcal{}$ C5     2     1.715     0.289

number of comments (NC)     C5 $\not\to\mathcal{}$ NC     4     2.412     0.089

number of comments (NC)     NC $\not\to\mathcal{}$ C5     4     1.135     0.530

§.§ Results

Table <ref> shows significant Granger causal results ($p < 0.05$). Gists across certain topics are significantly predictive of up-voting and commenting patterns, and vice-versa, in banned subreddits that oppose pandemic health practices. Specifically, the daily volume of gists significantly forecasts up-voting and commenting behavior across the topic of vaccines (cluster 1), mask-wearing (cluster 2),  and macro-impacts of the pandemic (cluster 4) with significant lag lengths ranging from 2-7 days. These results align with prior research highlighting the linguistic power of gists in spreading online information. The reverse (H1R and H2R) is true for gists discussing the impact of lockdowns (cluster 3): up-voting and commenting behavior both significantly forecast fluctuations in the volume of lockdown related gists.

§.§.§ Bidirectional Causality:
Notably for cluster 2,  which pertains to controversies and policies related to masks-wearing, we observe an interesting feedback loop between gist volumes and commenting behavior. As the volume of gists related to mask-wearing practices increases, corresponding online engagement around posts containing such gists, also increases in the form of up-votes. This behavior, in turn, further influences the volume of gists that are topically related to mask-wearing practices. In other words, there is a mutually reinforcing effect between causal language and online behavior in the context of mask-related discussions.

§ STUDY 4: HOW SOCIAL MEDIA GIST PATTERNS INFLUENCE NATIONWIDE TRENDS IN HEALTH OUTCOMES

In Study 4, we address the question of whether and how social media language patterns in the form of gists influence health decisions and outcomes in the U.S. We follow Study 3's application of Granger causality to examine the relationship between gists patterns and important health decisions and outcomes related to COVID-19 in America. Considering the extensive attention the subreddits we analyzed received from the American public and the media <cit.>, we focus on U.S. health outcomes.

§.§ COVID-19 Data on Health Outcomes

We used the following data from Our World in Data [https://github.com/owid/covid-19-data/tree/master/public/data], a trusted source for COVID-19 health data for our analysis:

* Number of Vaccinations (NV): the total number of COVID-19 vaccine doses administered on a given day.

* General Hospitalization (GH): the number of individuals hospitalized due to COVID-19 on a given day.

* ICU Hospitalization (ICU): the number of patients with COVID-19 who are in the ICU on a given day.

* Total Daily COVID-19 Cases (TC): the total number of confirmed COVID-19 cases, including probable cases.

* New Daily COVID-19 Cases (NC): the number of newly confirmed COVID-19 cases, including probable cases.

§.§ Hypothesis Testing with Granger Causality

Following Study 3, we Granger-test the relationship between the daily volume of gists and patterns in people's health decisions (vaccinations) and national health outcomes (General/ ICU Hospitalization, Total/ New Daily COVID-19 Cases) through H3 and its reversed variation (H3R):

arc=1mm, auto outer arc,
H3. The daily frequency of gists (Cluster $i$) significantly Granger-causes people's health decisions and/or national health outcomes, where $i$ ranges from $1$ to $5$.

H3R. People's health decisions and/or national health outcomes significantly Granger-causes the daily frequency of gists (Cluster $i$), where $i$ ranges from $1$ to $5$.

We created five time series data,  $\mathcal{T}_{NV}$, $\mathcal{T}_{GH}$, $\mathcal{T}_{ICU}$, $\mathcal{T}_{TC}$, $\mathcal{T}_{NC}$ , corresponding to the five health outcome data described above. We temporally align our data with the time frame for Studies 1-3. We performed 25 Granger causality tests (5 clusters $\times$ 5 health outcome data) with a range of lag times from 1 to 14 days. We conducted ADF tests using the serial difference method to ensure statistical robustness.

§.§ Results

Table <ref> show shows significant Granger-causal results with corresponding lag lengths ($p < 0.05$). We summarize our findings below.

Result of Granger causality test for relationships between Reddit discussion clusters (C1-C4) and health outcomes dataset. Cluster 5 is not included due to the absence of significant Granger causality findings. Notes: See Appendix A for complete statistical results (Table <ref> and <ref>).
customcolor2 Cluster     Health Outcomes (HO)     Hypotheses     Lag     $F$-val     $p$-val     Direction of Impact

23cmC1. Vaccination Implications      Number of Vaccinations     C1 $\not\to\mathcal{}$ NV     4    4.778    0.027

Number of Vaccinations     NV $\not\to\mathcal{}$ C1     14     7.771     0.007     -2*bidirectional

23.4cmC2. Controversies and Policies Related to Masks     Total Daily Covid Cases     C2 $\not\to\mathcal{}$ TC     5     9.395     0.005

New Daily Covid Cases     C2 $\not\to\mathcal{}$ NC     5     11.829    0.004     -2*gists impacts HO

22.9cmC3. Impact of Lockdown     ICU Hospitalization     ICU $\not\to\mathcal{}$ C3     14    4.000     0.039

General Hospitalization     GH $\not\to\mathcal{}$ C3     9    8.822    0.006    -2*HO impacts gists

23.9cmC4. Societal and Economic (Macro) Impact of COVID-19     Total Daily Covid Cases     TC $\not\to\mathcal{}$ C4     9    3.663    0.031

New Daily Covid Cases     NC $\not\to\mathcal{}$ C4     9    3.092    0.032     -2*HO impacts gists

Causal Talk Around Vaccines and National Vaccination Trends are Bidirectional. Our results demonstrate bidirectional causality between causal discourse patterns related to vaccines and the number of vaccinations administered in the U.S. The daily volume of
cluster 1 gists, which consists of causal arguments related to vaccine regulations, efficacy, and side effects, is predictive of vaccination patterns across the U.S., and vice-versa. However, there is a difference in the lag lengths between H3 and H3R. It takes 4 days for gist patterns to influence vaccine adoptions (H3), while it takes two weeks for vaccination trends to shape how people talk about vaccine-related topics in a causal manner (H3R) across COVID-19 subreddits known for vaccine skepticism. In addition to a more significant Granger-causal relationship, we also observe a higher Pearson correlation for H3 ($r = 0.413$, $p = 0.005$) compared to H3R ($r = 0.105$, $p = 0.028$), indicating that national vaccination patterns have a greater impact on shaping vaccine-related causal language on social media than the other way around. There are two possible explanations: first, as more people get vaccinated, online discussions on the experiences and potential side effects of vaccines may become more prevalent - leading people to talk in a causal manner about the side effects of vaccines (e.g., "Had my Pfizer jab last Wed and have felt like death since").  Another possible explanation is that the increasing vaccination requirements by corporations and governments as a condition for work or travel (and therefore, nationwide uptick in vaccinations) during the pandemic may have compelled vaccine-skeptics to argue more vehemently against vaccines <cit.>. Previous research has shown that vaccine skeptics are susceptible to confirmation bias, as are most individuals, such that initial beliefs lead to polarization <cit.>.  That is, vaccine skeptics are likely to seek out and discuss information about vaccines that confirms pre-existing beliefs when presented with opposing information or situated in contexts that challenge their views. Our findings align with this research, suggesting that as national vaccination uptake increases, vaccine skeptics might increasingly argue against vaccines in a causal manner (e.g., "If you take the vaccine, it's probably because you're unhealthy."), as commonly expressed in posts that contain cluster 1 gists.

Causal Talk Around Mask-Wearing Practices Significantly Predicts Trends in COVID-19 Cases. Our Granger-causal results show that national health outcomes, such as the total and new daily COVID-19 cases can be significantly predicted by the volume of mask-related gists (cluster 2) with a lag of 5 days.  The mask mandate was one of the most controversial health practices that impacted people of all ages and occupations during the pandemic <cit.>. Parents were polarized over school mask requirements to the extent of resorting to violence <cit.>. Employees who asked customers to wear masks were physically assaulted <cit.>. Although people initially adhered to wearing masks, more individuals started to protest mask mandates both on and offline, citing physical distress ("If having healthy lungs is important for COVID, why would we wear masks that reduce lung function?" ) or invasion of personal rights: "They will call you a `coward' or `scared' for not wanting an intrusive mask over your face (for no reason)", as exemplified by posts containing cluster 2 gists in our data. Over time, the proliferation of anti-mask views, followed by extreme resistance as demonstrated by violent altercations and wide-scale protests across the nation, may have led people to abandon mask-wearing practices <cit.>, which in turn may have led to an increase in COVID-19 cases within a relatively short time-frame of 5 days, as indicated in our results.

Rising Hospitalization Trends Prompt Causal Talk on Lockdown Impact.
Our findings show that nationwide trends in the number of patients hospitalized in both general and intensive care units significantly prompt more gists discussing the impact of lockdowns with a lag of 9 and 14 days, respectively (Table <ref>). Nationwide lockdowns were implemented to curb steep rises in COVID-19 cases and hospitalization rates. In fact, some posts containing cluster 3 gists often explicitly link lockdowns with hospitalizations: "The main reason for implementing restrictions or lockdowns was to prevent ICUs from overflowing." Despite its necessity and intended benefit as a public health measure, studies have shown that lockdowns significantly contributed to social isolation, decrease in mental health, and rise in domestic violence across the U.S. <cit.>. As the lockdown continued to amplify challenges and problems in people's lives, rising hospitalization trends across the country may have heightened people's fear and distress, leading to more intensified and causal online discourse on the lockdown's impact on everyday life. Such sentiments are clearly expressed across posts containing cluster 3 gists: "People are literally starting to go hungry because of lockdown restrictions"; "The implementation of lockdowns has resulted in more harm than good".

Rising Trends in COVID-19 Cases Prompt Causal Talk on the Pandemic's Macro-Level Impact.
Nationwide trends in COVID-19 cases significantly Granger-causes the volume of gists discussing the pandemic's impact on society at large, with a lag of 9 days for both total and new cases. In other words, increasing trends in COVID-19 cases seem to nudge people to talk casually about the macro-level consequences of COVID-19. COVID-19 presented major economic and social setbacks that impacted all aspects of society. Some of these concerns were expressed across posts containing cluster 4 gists that linked the pandemic with economic crises ("The pandemic caused one of the largest economic crises, which in turn led to one of the largest poverty and hunger crises"), decreased life expectancy ("The COVID-19 pandemic has caused the biggest drop in US life expectancy since the second world war"), potentially oppressive public health measures ("The cause of the next deadly pandemic will lead to the implementation of authoritarian prevention measures"), and even racism ("The fact that Covid19 affects people of color more than whites is the cause of the conclusion that Covid19 is racist"). With COVID-19 cases rising and situations continuing to remain unpredictable, people may have become more anxious and distressed about the long-term effects on society. Consequently, this may have led individuals to discuss the pandemic's impact in a causal manner on social media, as they try to make sense of its far-reaching consequences on society <cit.>.

§ DISCUSSION

In summary, our findings underscore RBIC's effectiveness in efficiently predicting social media gists at scale (Study 1), thereby enriching our insight into the underlying mental constructs that shape people's health decisions and attitudes towards public health practices. In Study 2, we cluster and track the evolution of such gists, revealing key themes in online arguments against pandemic health practices. These gist volumes closely align with significant topical events, such as health announcements, policy changes, and leadership statements. In Study 3, we empirically demonstrate how gist volumes significantly drive subreddit engagement patterns (up-votes and comments). Finally, Study 4 reveals the interplay between gist patterns in anti-Covid-19 subreddits and nationwide health trends. We discuss the implications of these findings below.

§.§ Harnessing Large Language Models in
Computational Social Science (CSS) Research in HCI

Prompt-based LLMs are increasingly used in the CHI community <cit.>, primarily contributing to the development of applications like chatbots <cit.> and tools for co-writing <cit.>, virtual simulations <cit.>, story-telling <cit.>, and visualization enhancement <cit.>. Such studies have primarily focused on using LLMs as production tools <cit.> rather than tools for analysis. More recently, computational social scientists in HCI have used prompt-based LLMs for text analyses <cit.>. However, there remain several challenges for using LLMs in nuanced examination of social media discourse.
First, traditional NLP models and commonly used LLMs in CSS research often lack reasoning capabilities <cit.>. For instance, LLMs like BERT-based models, which are extensively used in HCI research that analyze large volumes of social media data <cit.>, are typically fine-tuned for specific discrete downstream tasks (e.g., classification). While these pretrained language models have shown promise in performing discrete analyses, some emerging HCI research <cit.> demonstrate the additional value of prompting LLMs to perform multi-step reasoning for a more comprehensive analysis. Building on these prior insights, RBIC aims to enable a more nuanced analysis of social media discourse by leveraging the multi-reasoning capabilities of large language models. To this end, RBIC operates by performing multiple, step-by-step interrelated sub-tasks (question-answering, classification, extraction, generation) prior to generating its final output. This incremental coaching mechanism enhances the model's overall understanding and performance of the final task, allowing us to analyze social media discourse with a more comprehensive and nuanced approach.

Second, LLM development paradigms often incentivize researchers to optimize model performance using established evaluation datasets <cit.>. While valuable for comparing an LLM's performance with other models, this approach may not result in high performance when applied to new, unseen, in-the-wild datasets <cit.> or with tasks that are slightly different from those that the model was evaluated on <cit.>. As a result, this may limit the potential application of such LLMs for analyzing intricate, heterogeneous in-the-wild data, such as unstructured social media conversations. The role-based cognition component of RBIC addresses this limitation by allowing researchers to define and customize the role of any prompt-based LLM to perform a complex and nuanced language task. By introducing and applying RBIC in the analysis of social media conversations, we demonstrate the versatility and effectiveness of prompt-based LLMs in identifying and synthesizing nuanced linguistic patterns, thus broadening the potential application of prompt-based LLMs for theory-driven textual analysis in CSS research in the HCI domain.

§.§ Leveraging Causal Language Patterns in Online Content Moderation Practices

Our results show that the volume of gists across certain topics are significantly predictive of up-voting and commenting patterns, and vice-versa, in banned subreddits that oppose pandemic health practices. For example, daily gist volumes significantly predict up-voting and commenting behavior across topics related to vaccines, masks, and the pandemic's impacts, highlighting the linguistic power of gists in spreading online information as demonstrated in prior literature <cit.>. Similarly, our findings show that increasing trends in vaccine adoptions in the U.S. are strongly predictive of the growing volumes of vaccine-related gists in subreddits whose members are generally skeptical of vaccines. While a nationwide rise in vaccine uptake is certainly beneficial, such conditions may present challenging contexts that may reinforce vaccine skeptics to become further entrenched in their views. Vaccine opponents exposed to situations that contradict their perceptions are especially vulnerable to confirmation biases <cit.>, which may lead to an increased tendency to express their anti-vaccine sentiments in online communities in a causal manner, as implied by our findings.

These insights underscore the critical role of understanding and monitoring causal language patterns in public health discourse, particularly within online spaces. Current content moderation practices that rely on language models traditionally focus on flagging hate speech or monitoring specific keywords <cit.>. However, our research suggests that monitoring causal language patterns can be a valuable addition to these content moderation practices, especially in controversial online communities where people exchange and learn health information. By leveraging nuanced insights from gists across various health topics, content moderation can become more effective in identifying and managing discussions that may contribute to the spread of online health misinformation or resistance to public health guidelines.

§.§.§ Design Implications for Moderation Dashboard:
Prior studies have shown that the design of a social media platform plays an important role in promoting transparency in content moderation <cit.>. Moderators often fail to articulate what aspect of the content prompted moderation or why such moderation was necessary <cit.>. The approach taken in our study can be built on to effectively inform users about the consequences of their posting behavior, and which aspects of their posts can potentially lead to negative outcomes. The results can also inform design strategies that  platforms can undertake to assist moderators in communicating such information to users.

Understanding and identifying causality can be difficult for humans as causality may be expressed implicitly and across sentences or intersententially <cit.>. Currently, there is no automated mechanism for moderators to systematically identify and understand the impact of causal language across online discussions. A design feature in the moderation dashboard, such as the one shown in Fig. <ref> (Appendix D), serves as an illustrative example of how RBIC may address this gap.  For example, when a moderator clicks on a button called ‘Enable Gist Detection (RBIC)’, an RBIC-powered extension can automatically scan posts, highlight the cause-and-effect pairs, and identify the overarching gists within the posts. This functionality may also allow moderators to see a list of top gists across community discussions in descending order of gist volumes, and an option to organize these gists based on engagement metrics, including the upvote ratio and comment volume. Additionally, the system may be designed such that the moderator may be able to drill down into posts that pertain to each of these top gists, in which the system can highlight the relevant text spans that pertain to the  cause and effect in each post.

§.§.§ Improving Moderation and Community Guidelines
Identifying posts that do not contain moderator-specified keywords (e.g., profanity) or those that exclude explicit causal language, can still violate community norms or include misleading information in subtle ways <cit.>. Traditional keyword-based filters fall short in identifying such content <cit.>. This can lead to difficulties in setting specific rules for moderation practices, explaining moderation-decisions, or adapting community guidelines during critical times, such as a global pandemic. With RBIC-powered gist detection, moderators can scale the searching of such posts to identify those that reflect common and theoretically predicted disconnects between the public and public health experts. This mechanism can potentially enable moderators to use concrete examples to better explain moderation decisions, as well as improve community guidelines to explain how posts that contain implicit causal narratives may impact people's knowledge and decisions around safe health practices, as shown in our work.

§.§ Broader Implications for Understanding Engagement Patterns Across Online Communities and Offline Health Outcomes During Public Health Crises

Our work shows that capturing psychologically important language patterns across social media, in the form of gists, can be useful in predicting human behavior and, consequently, health outcomes. In Study 2, we demonstrate that fluctuations in the volume of gists can significantly predict online engagement patterns, specifically in terms of up-vote ratio patterns (H1) and the volume of comments (H2). This has important implications for researchers studying user behaviors in online communities <cit.>. Researchers have shown that the virality of online content is often influenced by a positivity bias in engagement metrics <cit.>, such as up-votes and comments: posts receiving higher engagement are more visible and thus have a greater likelihood of going viral <cit.>. This tendency can exacerbate the spread of misinformation, especially during public health crises <cit.>. Posts challenging pandemic health practices are often laden with misleading information <cit.>, and online posts embedded with gists are more likely to attract more user engagement compared to those without gists <cit.>. H1 and H2 results demonstrate that such user engagement patterns are predictable through gist volumes, thus highlighting the potential of using RBIC for gist analysis to track and understand the dynamics of how health-related content, especially during pandemics, resonates with and influences online user engagement. This insight is crucial for developing strategies to combat misinformation and guide public health communication effectively.

Furthermore, HCI research in crisis informatics has contributed to advancing public health monitoring systems by developing tools that track public health outcomes, online engagement patterns, or health-related topics on social media <cit.>. Some of these tools that monitor online conversations extract various linguistic aspects from social media discourse, such as sentiment <cit.> and topical keywords  <cit.>. While these advancements have been valuable in providing descriptive insights, most do not go the full distance in linking such linguistic patterns to real-world health decisions and outcomes <cit.>. Our work addresses this gap by demonstrating how RBIC can be leveraged to better connect online conversation patterns to offline health outcome trends. Study 4 results show that online causal talk related to controversial health practices, such as face-masks, are significantly predictive of total and new daily COVID-19 cases across the U.S. Likewise, our findings show that the uncertainty arising from deteriorating trends in national health outcomes may prompt people to increasingly engage online in causal discussions on the pandemic's influence on their lives and society as a whole. For example, nationwide COVID-19 cases and hospitalization patterns significantly drive up the volume of gist-based conversations concerning the pandemic's impact on society, economy, and individuals under lockdown. These findings imply that integrating gist-based language patterns into public health monitoring systems can hold promise for gaining valuable insights into the cognition that underlies skepticism and resistance to public health practices and, by extension, their impact on real-world health outcomes. Integrating RBIC-powered gist detection and real-time analysis of national health indicators into tools can potentially enhance public health agencies' ability to understand and respond to critical health challenges in relation to people's online behavior.

§ CONCLUSION & LIMITATIONS

This research synthesizes LLM techniques with theoretical perspectives from cognitive and social psychology to advance the knowledge of health decisions and outcomes in the context of the most recent pandemic. Our work is the first to systematically identify and characterize how causal language patterns surrounding anti-pandemic health practices on social media are significantly predictive of national health outcomes. These findings carry crucial implications for public health communication and policy interventions. By recognizing the influential role of causal language patterns across social media in shaping national health outcomes, public health efforts and online moderation practices can be tailored to address and mitigate the impact of social media conversations that adversely affect public health consequences.

Our study has a limitation in our data source: it concentrates on Reddit posts and omits comments. This exclusion is primarily due to certain months of comment data being either restricted or deleted in compliance with Reddit's policies by Archive administrators. While this focus allows for an in-depth analysis of original posts, it may not capture the full discourse, including diverse viewpoints and nuanced discussions that often take place in the comments section. Consequently, our findings may offer a limited perspective on the topic under study. Future work might consider alternate ways to capture community discourse, such as through interviews or surveys, to complement the data from Reddit posts. Furthermore, as datasets from the future expand, integrating machine learning models that are capable of detecting subtle changes in discourse over time and adjust to extensive datasets may offer a dynamic view of how gists evolve. This method has the potential to uncover patterns and trends that may not be immediately obvious when using a traditional unsupervised clustering approach.

In summary, we built an LLM-based model to identify psychologically influential mental representations–gists–from social media posts, demonstrated the links between these gists and public health events, and verified associations with user engagement and national health trends, with implications for HCI design and the promotion of public health.

#1     #1#1#1     #1     #1     #1     #1#1          #1#1

[Ahmed and Rasul(2022)]
authorpersonSaifuddin Ahmed and personMuhammad Ehab Rasul. year2022.
Social Media News Use and COVID-19 Misinformation Engagement: Survey Study.
journalJournal of Medical Internet Research  volume24 (year2022).

[Al-Ramahi et al(2020)]
authorpersonM. Al-Ramahi, personA. Elnoshokaty, personO. El-Gayar, personTareq Nasralah, and personA. Wahbeh. year2020.
Public Discourse Against Masks in the COVID-19 Era: Infodemiology Study of Twitter Data.
journalJMIR Public Health and Surveillance  volume7 (year2020).

[Aldous et al(2019)]
authorpersonKholoud Khalil Aldous, personJisun An, and personBernard Jim Jansen. year2019.
View, Like, Comment, Post: Analyzing User Engagement by Topic at 4 Levels across 5 Social Media Platforms for 53 News Organizations. In booktitleInternational Conference on Web and Social Media.

[Ali et al(2022)]
authorpersonWajid Ali, personWanli Zuo, personRahman Ali, personGohar Rahman, personXianglin Zuo, and personInam Ullah. year2022.
Towards Improving Causality Mining using BERT with Multi-level Feature Networks.
journalKSII Transactions on Internet & Information Systems volume16, number10 (year2022), pages3230–3255.

[Azarpanah et al(2021)]
authorpersonHossein Azarpanah, personMohsen Farhadloo, personRustam M. Vahidov, and personLouise Pilote. year2021.
Vaccine hesitancy: evidence from an adverse events following immunization database, and the role of cognitive biases.
journalBMC Public Health  volume21 (year2021).

authorpersonDavid W. Baker. year2020.
Trust in Health Care in the Time of COVID-19.
journalJAMA volume324, number23 (year2020), pages2373–2375.

[Bavel et al(2020)]
authorpersonJay J. Van Bavel, personKatherine Baicker, personPaulo S. Boggio, and personet al. year2020.
Using social and behavioural science to support COVID-19 pandemic response.
journalNature Human Behaviour volume4, number5 (year2020), pages460–471.
Number: 5 Publisher: Nature Publishing Group.

authorpersonAbha Bhattarai. year2020.
Retail workers are being pulled into the latest culture war: Getting customers to wear masks.
journalThe Washington Post (year2020).

[Biester et al(2021)]
authorpersonLaura Biester, personKatie Matton, personJanarthanan Rajendran, personEmily Mower Provost, and personRada Mihalcea. year2021.
Understanding the Impact of COVID-19 on Online Mental Health Forums.
journalACM Trans. Manage. Inf. Syst. volume12, number4, Article articleno31 (datesep year2021), numpages28 pages.

[Blair et al(2021)]
authorpersonJohnna Blair, personChi-Yang Hsu, and personet al. year2021.
Using Tweets to Assess Mental Well-Being of Essential Workers During the COVID-19 Pandemic. In booktitleExtended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (seriesCHI EA '21). publisherAssociation for Computing Machinery, addressNew York, NY, USA, Article articleno236, numpages6 pages.

[Blei et al(2003)]
authorpersonDavid M. Blei, personAndrew Y. Ng, personMichael I. Jordan, and personJohn Lafferty. year2003.
Latent Dirichlet Allocation.
journalJournal of Machine Learning Research volume3, number4 (year2003), pages993–1022.

authorpersonJames S. Brady. year2020.
titleRemarks by President Trump in Press Briefing August 19, 2020.

The White House, National Archives.

[Bro and K. Smilde(2014)]
authorpersonRasmus Bro and personAge K. Smilde. year2014.
Principal component analysis.
journalAnalytical Methods volume6, number9 (year2014), pages2812–2831.

[Broniatowski and Reyna(2018)]
authorpersonDavid Broniatowski and personValerie F. Reyna. year2018.
Causal Gist Predicts Spread of Tweets about Vaccination. In booktitle40th Annual Meeting of the Society for Medical Decision Making. publisherSMDM.

[Broniatowski et al(2016)]
authorpersonDavid Andre Broniatowski, personKaren M. Hilyard, and personMark Dredze. year2016.
Effective Vaccine Communication during the Disneyland Measles Outbreak.
journalVaccine volume34, number28 (year2016), pages3225–3228.

[Caldeira et al(2023)]
authorpersonClara Caldeira, personCleidson R.B. de Souza, personLetícia Machado, personMarcelo Perin, and personPernille Bjørn. year2023.
Crisis Readiness: Revisiting the Distance Framework During the COVID-19 Pandemic.
volume32, number2 (year2023), pages237–273.

[Calvillo et al(2020)]
authorpersonDustin P. Calvillo, personBryan J. Ross, personRyan J. B. Garcia, personThomas J. Smelter, and personAbraham M. Rutchick. year2020.
Political Ideology Predicts Perceptions of the Threat of COVID-19 (and Susceptibility to Fake News About It).
journalSocial Psychological and Personality Science volume11, number8 (year2020), pages1119–1128.

[Cannon et al(2021)]
authorpersonClare E. B. Cannon, personRegardt Ferreira, personFrederick Buttell, and personJennifer First. year2021.
COVID-19, Intimate Partner Violence, and Communication Ecologies.
journalAmerican Behavioral Scientist volume65, number7 (year2021), pages992–1013.

[Caselli and Vossen(2017)]
authorpersonTommaso Caselli and personPiek Vossen. year2017.
The Event StoryLine Corpus: A New Benchmark for Causal and Temporal Relation Extraction. In booktitleProceedings of the Events and Stories in the News Workshop, editorpersonTommaso Caselli, personBen Miller, personMarieke van Erp, personPiek Vossen, personMartha Palmer, personEduard Hovy, personTeruko Mitamura, and personDavid Caswell (Eds.). publisherAssociation for Computational Linguistics, addressVancouver, Canada, pages77–86.

[Chen et al(2023)]
authorpersonJiangjie Chen, personWei Shi, personZiquan Fu, personSijie Cheng, personLei Li, and personYanghua Xiao. year2023.
Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge. In booktitleProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). publisherAssociation for Computational Linguistics, addressToronto, Canada, pages9890–9908.

[Cheung and Lai(1995)]
authorpersonYin-Wong Cheung and personKon S. Lai. year1995.
Lag Order and Critical Values of the Augmented Dickey–Fuller Test.
journalJournal of Business & Economic Statistics volume13, number3 (dateJuly year1995), pages277–280.

[Chung et al(2022a)]
authorpersonHyung Won Chung, personLe Hou, personS. Longpre, and personet al. year2022a.
Scaling Instruction-Finetuned Language Models.
journalArXiv  volumeabs/2210.11416 (year2022).

[Chung et al(2022b)]
authorpersonJohn Joon Young Chung, personWooseok Kim, personKang Min Yoo, personHwaran Lee, personEytan Adar, and personMinsuk Chang. year2022b.
TaleBrush: Sketching Stories with Generative Pretrained Language Models. In booktitleProceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (seriesCHI '22). publisherAssociation for Computing Machinery, addressNew York, NY, USA, Article articleno209, numpages19 pages.

[Corbin et al(2015)]
authorpersonJonathan C. Corbin, personValerie F. Reyna, personRebecca B. Weldon, and personCharles J. Brainerd. year2015.
How Reasoning, Judgment, and Decision Making are Colored by Gist-based Intuition: A Fuzzy-Trace Theory Approach.
journalJournal of Applied Research in Memory and Cognition volume4, number4 (year2015), pages344–355.

[DeDominicis et al(2020)]
authorpersonKali DeDominicis, personAlison M. Buttenheim, personAmanda C. Howa, personPaul L. Delamater, personDaniel A. Salmon, personSaad B. Omer, and personNicola P. Klein. year2020.
Shouting at each other into the void: A linguistic network analysis of vaccine hesitance and support in online discourse regarding California law SB277.
journalSocial science & medicine  volume266 (year2020), pages113216.

[Demuyakor et al(2021)]
authorpersonJohn Demuyakor, personIsaac Newton Nyatuame, and personS. Obiri. year2021.
Unmasking COVID-19 Vaccine “Infodemic” in the Social Media.
journalOnline Journal of Communication and Media Technologies (year2021).

[Ding et al(2019)]
authorpersonKeyan Ding, personRonggang Wang, and personShiqi Wang. year2019.
Social Media Popularity Prediction: A Multiple Feature Fusion Approach with Deep Neural Networks. In booktitleProceedings of the 27th ACM International Conference on Multimedia (Nice, France) (seriesMM '19). publisherAssociation for Computing Machinery, addressNew York, NY, USA, pages2682–2686.

[Ding et al(2023)]
authorpersonXiaohan Ding, personMichael A. Horning, and personEugenia Ha Rim Rho. year2023.
Same Words, Different Meanings: Semantic Polarization in Broadcast Media Language Forecasts Polarity in Online Public Discourse.
journalProceedings of the International AAAI Conference on Web and Social Media (year2023).

[Doan et al(2019)]
authorpersonSon Doan, personElly W. Yang, personSameer S. Tilak, personPeter W. Li, personDaniel S. Zisook, and personManabu Torii. year2019.
Extracting health-related causality from twitter messages using natural language processing.
journalBMC Medical Informatics and Decision Making volume19, number3 (year2019), pages79.

[Fang and Zhu(2023)]
authorpersonAnna Fang and personHaiyi Zhu. year2023.
Measuring the Stigmatizing Effects of a Highly Publicized Event on Online Mental Health Discourse. In booktitleProceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (seriesCHI '23). publisherAssociation for Computing Machinery, addressNew York, NY, USA, Article articleno482, numpages18 pages.

authorpersonPaige M. Farrenkopf. year2022.
The Cost of Ignoring Vaccines.
journalThe Yale Journal of Biology and Medicine volume95, number2 (year2022), pages265–269.

authorpersonJoseph L. Fleiss. year1981.
booktitleStatistical Methods for Rates and Proportions (edition2nd ed.).
publisherJohn Wiley & Sons, addressNew York.

[Gagneux-Brunon et al(2022)]
authorpersonAmandine Gagneux-Brunon, personElisabeth Botelho-Nevers, personMarion Bonneton, personPatrick Peretti-Watel, personPierre Verger, personOdile Launay, and personJeremy K. Ward. year2022.
Public opinion on a mandatory COVID-19 vaccination policy in France: a cross-sectional survey.
journalClinical Microbiology and Infection volume28, number3 (year2022), pages433–439.

[Gilardi et al(2023)]
authorpersonFabrizio Gilardi, personMeysam Alizadeh, and personMaël Kubli. year2023.
ChatGPT outperforms crowd workers for text-annotation tasks.
journalProceedings of the National Academy of Sciences of the United States of America  volume120 (year2023).

authorpersonC. W. J. Granger. year1980.
Testing for causality: A personal viewpoint.
journalJournal of Economic Dynamics and Control  volume2 (dateJan. year1980), pages329–352.

authorpersonJordan Grunawalt. year2021.
The Villain Unmasked: COVID-19 and the Necropolitics of the Anti-Mask Movement.
journalDisability Studies Quarterly volume41, number3 (dateSep. year2021), pages8343.

authorpersonCheng Guo. year2020.
Identity and User Behavior in Online Communities. In booktitleCompanion Proceedings of the 2020 ACM International Conference on Supporting Group Work (Sanibel Island, Florida, USA) (seriesGROUP '20). publisherAssociation for Computing Machinery, addressNew York, NY, USA, pages35–38.

[Hamilton and et al.(2018)]
authorpersonWilliam L. Hamilton and personet al. year2018.
titleDiachronic Word Embeddings Reveal Statistical Laws of Semantic Change.

[Harrigian and Dredze(2022)]
authorpersonKeith Harrigian and personMark Dredze. year2022.
The Problem of Semantic Shift in Longitudinal Monitoring of Social Media: A Case Study on Mental Health During the COVID-19 Pandemic. In booktitleProceedings of the 14th ACM Web Science Conference 2022 (Barcelona, Spain) (seriesWebSci '22). publisherAssociation for Computing Machinery, addressNew York, NY, USA, pages208–218.

[Hosseini et al(2019)]
authorpersonPedram Hosseini, personMona Diab, and personDavid A. Broniatowski. year2019.
Does Causal Coherence Predict Online Spread of Social Media?. In booktitleSocial, Cultural, and Behavioral Modeling, editorpersonRobert Thomson, personHalil Bisgin, personChristopher Dancy, and personAyaz Hyder (Eds.). publisherSpringer International Publishing, addressCham, pages184–193.

[Hussain and Sheikh(2021)]
authorpersonAmir Hussain and personAziz Sheikh. year2021.
Opportunities for Artificial Intelligence–Enabled Social Media Analysis of Public Attitudes Toward Covid-19 Vaccines.
journalCatalyst non-issue content volume2, number1 (year2021).

authorpersonIEA. year2020.
titleGlobal Energy Review 2020.
International Energy Agency.

authorpersonMike Isaac. year2020.
titleReddit, acting against hate speech, bans “The_donald” subreddit.

authorpersonRishi Iyengar. year2021.
titleReddit takes action against groups spreading Covid misinformation | CNN business.

[Jhaver et al(2019)]
authorpersonShagun Jhaver, personIris Birman, personEric Gilbert, and personAmy Bruckman. year2019.
Human-Machine Collaboration for Content Regulation: The Case of Reddit Automoderator.
journalACM Transactions on Computer-Human Interaction (TOCHI)  volume26 (year2019), pages1 – 35.

[Jo et al(2023)]
authorpersonEunkyung Jo, personDaniel A. Epstein, personHyunhoon Jung, and personYoung-Ho Kim. year2023.
Understanding the Benefits and Challenges of Deploying Conversational AI Leveraging Large Language Models for Public Health Intervention. In booktitleProceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (seriesCHI '23). publisherAssociation for Computing Machinery, addressNew York, NY, USA, Article articleno18, numpages16 pages.

[Juneja et al(2020)]
authorpersonPrerna Juneja, personDeepika Rama Subramanian, and personTanushree Mitra. year2020.
Through the Looking Glass: Study of Transparency in Reddit's Moderation Practices.
volume4, numberGROUP, Article articleno17 (datejan year2020), numpages35 pages.

[Kaplan et al(2020)]
authorpersonSheila Kaplan, personKatherine J. Wu, and personKatie Thomas. year2020.
C.D.C. Tells States How to Prepare for Covid-19 Vaccine by Early November.
journalThe New York Times (year2020).

[Karusala and Anderson(2022)]
authorpersonNaveena Karusala and personRichard Anderson. year2022.
Towards Conviviality in NavigatingHealth Information on Social Media. In booktitleProceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (seriesCHI '22). publisherAssociation for Computing Machinery, addressNew York, NY, USA, Article articleno43, numpages14 pages.

[Khetan et al(2021)]
authorpersonVivek Khetan, personRoshni Ramnani, personMayuresh Anand, personShubhashis Sengupta, and personet al. year2021.
titleCausal BERT : Language models for causality detection between events expressed in text.

authorpersonJiyoun Kim. year2020.
The Meaning of Numbers: Effect of Social Media Engagement Metrics in Risk Communication.
journalCommunication Studies  volume72 (year2020), pages195 – 213.

[Klein et al(2017)]
authorpersonEili Y. Klein, personElena M. Martinez, personLarissa May, and personet al. year2017.
Categorical Risk Perception Drives Variability in Antibiotic Prescribing in the Emergency Department: A Mixed Methods Observational Study.
journalJournal of General Internal Medicine volume32, number10 (year2017), pages1083–1089.

[Kretchy et al(2021)]
authorpersonIrene A. Kretchy, personMichelle Asiedu-Danso, and personJames-Paul Kretchy. year2021.
Medication management and adherence during the COVID-19 pandemic: Perspectives and experiences from low-and middle-income countries.
journalResearch in Social and Administrative Pharmacy volume17, number1 (year2021), pages2023–2026.

[Kumar and Chinnalagu(2020)]
authorpersonD. Ashok Kumar and personAnandan Chinnalagu. year2020.
Sentiment and Emotion in Social Media COVID-19 Conversations: SAB-LSTM Approach. In booktitle2020 9th International Conference System Modeling and Advancement in Research Trends (SMART). publisherIEEE, addressMoradabad, India, pages463–467.

[Lee et al(2023)]
authorpersonJiyoung Lee, personJihyang Choi, and personRebecca K. Britt. year2023.
Social Media as Risk-Attenuation and Misinformation-Amplification Station: How Social Media Interaction Affects Misperceptions about COVID-19.
journalHealth Communication volume38, number6 (year2023), pages1232–1242.

PMID: 34753361.

[Lee et al(2022)]
authorpersonMina Lee, personPercy Liang, and personQian Yang. year2022.
CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities. In booktitleProceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (seriesCHI '22). publisherAssociation for Computing Machinery, addressNew York, NY, USA, Article articleno388, numpages19 pages.

[Liederholm et al(2000)]
authorpersonTracy Liederholm, personMichelle Gaddy Everson, personPaul van den Broek, personMaureen Mischinski, personAlex Crittenden, and personJay Samuels. year2000.
Effects of causal text revisions on more-and less-skilled readers' comprehension of easy and difficult texts.
journalCognition and Instruction (year2000), pages525–556.

[Liu et al(2022)]
authorpersonJiacheng Liu, personAlisa Liu, personXiming Lu, personSean Welleck, personPeter West, personRonan Le Bras, personYejin Choi, and personHannaneh Hajishirzi. year2022.
Generated Knowledge Prompting for Commonsense Reasoning. In booktitleProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Dublin, Ireland). publisherAssociation for Computational Linguistics, addressDublin, Ireland, pages3154–3169.

[Lu et al(2015)]
authorpersonYafeng Lu, personXia Hu, personFeng Wang, personShamanth Kumar, personHuan Liu, and personRoss Maciejewski. year2015.
Visualizing Social Media Sentiment in Disaster Scenarios. In booktitleProceedings of the 24th International Conference on World Wide Web (Florence, Italy) (seriesWWW '15 Companion). publisherAssociation for Computing Machinery, addressNew York, NY, USA, pages1211–1215.

[Malik et al(2023)]
authorpersonMuhammad Shahid Iqbal Malik, personTahir Imran, and personJamjoom Mona Mamdouh. year2023.
How to detect propaganda from social media? Exploitation of semantic and fine-tuned language models.
journalPeerJ Computer Science  volume9 (year2023).

[Marti and Broniatowski(2020)]
authorpersonDeniz Marti and personDavid A. Broniatowski. year2020.
Does gist drive NASA experts’ design decisions?
journalSystems Engineering volume23, number4 (year2020), pages460–479.

_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/sys.21538.

[Martin and Vanderslott(2022)]
authorpersonSam Martin and personSamantha Vanderslott. year2022.
“Any idea how fast ‘It’s just a mask!’ can turn into ‘It’s just a vaccine!’”: From mask mandates to vaccine mandates during the COVID-19 pandemic.
journalVaccine volume40, number51 (year2022), pages7488–7499.

[Martinelli and Zaina(2021)]
authorpersonSuéllen R. Martinelli and personLuciana A. M. Zaina. year2021.
Learning HCI from a Virtual Flipped Classroom: Improving the Students' Experience in Times of COVID-19. In booktitleProceedings of the XX Brazilian Symposium on Human Factors in Computing Systems (Virtual Event, Brazil) (seriesIHC '21). publisherAssociation for Computing Machinery, addressNew York, NY, USA, Article articleno34, numpages11 pages.

[Marutho et al(2018)]
authorpersonDhendra Marutho, personSunarna Hendra Handaka, personEkaprana Wijaya, and personMuljono. year2018.
The Determination of Cluster Number at k-Mean Using Elbow Method and Purity Evaluation on Headline News. In booktitle2018 International Seminar on Application for Technology of Information and Communication. publisherIEEE, addressSemarang, Indonesia, pages533–538.

[Mehra et al(2023)]
authorpersonBrinda Mehra, personKejia Shen, personHen Chen Yen, and personCan Liu. year2023.
Gist and Verbatim: Understanding Speech to Inform New Interfaces for Verbal Text Composition. In booktitleProceedings of the 5th International Conference on Conversational User Interfaces (Eindhoven, Netherlands) (seriesCUI '23). publisherAssociation for Computing Machinery, addressNew York, NY, USA, Article articleno15, numpages11 pages.

[Meppelink et al(2019)]
authorpersonCorine S. Meppelink, personEdith G. Smit, personMarieke L. Fransen, and personNicola Diviani. year2019.
“I was Right about Vaccination”: Confirmation Bias and Health Literacy in Online Health Information Seeking.
journalJournal of Health Communication volume24, number2 (year2019), pages129–140.
1081-0730, 1087-0415

[Mihăilă and Ananiadou(2014)]
authorpersonClaudiu Mihăilă and personSophia Ananiadou. year2014.
Semi-supervised learning of causal relations in biomedical scientific discourse.
journalBioMedical Engineering OnLine volume13, number2 (year2014), pagesS1.

[Mirza and Tonelli(2016)]
authorpersonParamita Mirza and personSara Tonelli. year2016.
CATENA: CAusal and TEmporal relation extraction from NAtural language texts. In booktitleProceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, editorpersonYuji Matsumoto and personRashmi Prasad (Eds.). publisherThe COLING 2016 Organizing Committee, addressOsaka, Japan, pages64–75.

[Na et al(2010)]
authorpersonShi Na, personLiu Xumin, and personGuan Yong. year2010.
Research on k-means Clustering Algorithm: An Improved k-means Clustering Algorithm. In booktitle2010 Third International Symposium on Intelligent Information Technology and Security Informatics. publisherIEEE, addressGeorgia, USA, pages63–67.

[Nainggolan et al(2019)]
authorpersonRena Nainggolan, personResianta Perangin-angin, personEmma Simarmata, and personAstuti Feriani Tarigan. year2019.
titleImproved the performance of the K-means cluster using the sum of squared error (SSE) optimized by using the Elbow method.
, numpages012015 pages.

[Nyawa et al(2022)]
authorpersonSerge Nyawa, personDieudonné Tchuente, and personSamuel Fosso-Wamba. year2022.
COVID-19 vaccine hesitancy: a social media analysis using deep learning.
journalAnnals of Operations Research (year2022), pages1–39.

[Park et al(2021)]
authorpersonChan Young Park, personJulia Mendelsohn, personKarthik Radhakrishnan, personKinjal Jain, personTushar Kanakagiri, personDavid Jurgens, and personYulia Tsvetkov. year2021.
Detecting Community Sensitive Norm Violations in Online Conversations. In booktitleConference on Empirical Methods in Natural Language Processing.

[Park et al(2023)]
authorpersonJinkyung Park, personRahul Dev Ellezhuthil, personJoseph Isaac, personChristoph Mergerson, personLauren Feldman, and personVivek Singh. year2023.
Misinformation Detection Algorithms and Fairness across Political Ideologies: The Impact of Article Level Labeling. In booktitleProceedings of the 15th ACM Web Science Conference 2023 (Austin, TX, USA) (seriesWebSci '23). publisherAssociation for Computing Machinery, addressNew York, NY, USA, pages107–116.

[Park et al(2022)]
authorpersonSeungeun Park, personBetty Bekemeier, personAbraham Flaxman, and personMelinda Schultz. year2022.
Impact of data visualization on decision-making and its implications for public health practice: a systematic literature review.
journalInformatics for Health and Social Care volume47, number2 (year2022), pages175–193.

PMID: 34582297.

[Pavalanathan and De Choudhury(2015)]
authorpersonUmashanthi Pavalanathan and personMunmun De Choudhury. year2015.
Identity Management and Mental Health Discourse in Social Media. In booktitleProceedings of the 24th International Conference on World Wide Web (Florence, Italy) (seriesWWW '15 Companion). publisherAssociation for Computing Machinery, addressNew York, NY, USA, pages315–321.

[Petridis et al(2023)]
authorpersonSavvas Petridis, personNicholas Diakopoulos, personKevin Crowston, personMark Hansen, personKeren Henderson, personStan Jastrzebski, personJeffrey V Nickerson, and personLydia B Chilton. year2023.
AngleKindling: Supporting Journalistic Angle Ideation with Large Language Models. In booktitleProceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (seriesCHI '23). publisherAssociation for Computing Machinery, addressNew York, NY, USA, Article articleno225, numpages16 pages.

[Puri et al(2021)]
authorpersonManish Puri, personZachary Dau, and personAparna S. Varde. year2021.
COVID and Social Media: Analysis of COVID-19 and Social Media Trends for Smart Living and Healthcare.
journalSIGWEB Newsl. volume2021, numberAutumn, Article articleno5 (datedec year2021), numpages20 pages.

[Puri et al(2020)]
authorpersonNeha Puri, personEric A. Coomes, personHourmazd Haghbayan, and personKeith Gunaratne. year2020.
Social media and vaccine hesitancy: new updates for the era of COVID-19 and globalized infectious diseases.
journalHuman Vaccines & Immunotherapeutics volume16, number11 (year2020), pages2586–2593.

PMID: 32693678.

[Qorib et al(2023)]
authorpersonMiftahul Qorib, personTimothy Oladunni, personMax Denis, personEsther Ososanya, and personPaul Cotae. year2023.
Covid-19 vaccine hesitancy: Text mining, sentiment analysis and machine learning on COVID-19 vaccination Twitter dataset.
journalExpert Systems with Applications  volume212 (year2023), pages118715.

[Rashmi Prasad and Joshi(2019)]
authorpersonAlan Lee Rashmi Prasad, Bonnie Webber and personAravind Joshi. year2019.
Penn Discourse Treebank Version 3.0.

[Raza et al(2018)]
authorpersonAgha Ali Raza, personBilal Saleem, personShan Randhawa, personZain Tariq, personAwais Athar, personUmar Saif, and personRoni Rosenfeld. year2018.
Baang: A Viral Speech-Based Social Platform for Under-Connected Populations. In booktitleProceedings of the 2018 CHI Conference on Human Factors in Computing Systems (<conf-loc>, <city>Montreal QC</city>, <country>Canada</country>, </conf-loc>) (seriesCHI '18). publisherAssociation for Computing Machinery, addressNew York, NY, USA, pages1–12.

[Reimers and Gurevych(2019)]
authorpersonNils Reimers and personIryna Gurevych. year2019.
titleSentence-BERT: Sentence Embeddings using Siamese BERT-Networks.

authorpersonValerie F. Reyna. year2012a.
A new intuitionism: Meaning, memory, and development in Fuzzy-Trace Theory.
journalJudgment and Decision Making volume7, number3 (year2012), pages332–359.

authorpersonValerie F. Reyna. year2012b.
Risk perception and communication in vaccination decisions: a fuzzy-trace theory approach.
journalVaccine  volume30 25 (year2012), pages3790–7.

authorpersonValerie F. Reyna. year2021.
A scientific theory of gist communication and misinformation resistance, with implications for health, education, and policy.
journalProceedings of the National Academy of Sciences volume118, number15 (year2021), pagese1912441117.

[Reyna and Brainerd(2023)]
authorpersonValerie F Reyna and personCharles J Brainerd. year2023.
Numeracy, gist, literal thinking and the value of nothing in decision making.
journalNature Reviews Psychology (year2023), pages1–19.

[Reyna et al(2016)]
authorpersonValerie F. Reyna, personJonathan C. Corbin, personRebecca B. Weldon, and personCharles J. Brainerd. year2016.
How Fuzzy-Trace Theory Predicts True and False Memories for Words, Sentences, and Narratives.
journalJournal of Applied Research in Memory and Cognition volume5, number1 (year2016), pages1–9.

[Reyna et al(2022)]
authorpersonValerie F. Reyna, personSarah Edelson, personBridget Hayes, and personDavid Garavito. year2022.
Supporting Health and Medical Decision Making: Findings and Insights from Fuzzy-Trace Theory.
journalMedical Decision Making volume42, number6 (year2022), pages741–754.

Publisher: SAGE Publications Inc STM.

[Reyna and Kiernan(1994)]
authorpersonValerie F. Reyna and personBarbara J Kiernan. year1994.
Development of Gist versus Verbatim Memory in Sentence Recognition: Effects of Lexical Familiarity, Semantic Content, Encoding Instructions, and Retention Interval.
journalDevelopmental Psychology  volume30 (year1994), pages178–191.

[Rheault and Cochrane(2020)]
authorpersonLudovic Rheault and personChristopher Cochrane. year2020.
Word Embeddings for the Analysis of Ideological Placement in Parliamentary Corpora.
journalPolitical Analysis volume28, number1 (year2020), pages112–133.

[Rho and Mazmanian(2020)]
authorpersonEugenia Ha Rim Rho and personMelissa Mazmanian. year2020.
Fostering Civil Discourse Online: Linguistic Behavior in Comments of #MeToo Articles Across Political Perspectives.

[Samosir et al(2022)]
authorpersonFeliks Victor Parningotan Samosir, personHapnes Toba, and personMewati Ayub. year2022.
BESKlus : BERT Extractive Summarization with K-Means Clustering in Scientific Paper.
journalJurnal Teknik Informatika dan Sistem Informasi volume8, number1 (year2022), pages202–217.

[Seitlinger and Ley(2012)]
authorpersonPaul Seitlinger and personTobias Ley. year2012.
Implicit Imitation in Social Tagging: Familiarity and Semantic Reconstruction. In booktitleProceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (seriesCHI '12). publisherAssociation for Computing Machinery, addressNew York, NY, USA, pages1631–1640.

authorpersonDeepa Shivaram. year2021.
The Topic Of Masks In Schools Is Polarizing Some Parents To The Point Of Violence.

[Silva et al(2020)]
authorpersonMirela Silva, personFabrício Ceschin, personPrakash Shrestha, personChristopher Brant, personJuliana Fernandes, personCatia S. Silva, personAndr'e Gr'egio, personDaniela Oliveira, and personLuiz H. F. Giovanini. year2020.
Predicting Misinformation and Engagement in COVID-19 Twitter Discourse in the First Months of the Outbreak.
journalArXiv  volumeabs/2012.02164 (year2020).

[Singh et al(2022)]
authorpersonIshika Singh, personValts Blukis, personArsalan Mousavian, personAnkit Goyal, personDanfei Xu, personJonathan Tremblay, personDieter Fox, personJesse Thomason, and personAnimesh Garg. year2022.
ProgPrompt: Generating Situated Robot Task Plans using Large Language Models.
journal2023 IEEE International Conference on Robotics and Automation (ICRA) (year2022), pages11523–11530.

[Song et al(2019)]
authorpersonEun Young Song, personHoe-Ryeon Choi, and personHong Chul Lee. year2019.
A Study on Efficient Training Method for Named Entity Recognition Model with Word Embedding Applied to PCA.
journalJournal of the Korean Institute of Industrial Engineers (year2019).

[St-Amant et al(2022)]
authorpersonOona St-Amant, personJ. Anneke Rummens, personHenry Parada, and personKarline Wilson-Mitchell. year2022.
The COVID-19 Mask.
journalAns. Advances in Nursing Science volume45, number2 (year2022).

[Su et al(2020)]
authorpersonQi Su, personClara M. Wan, personXiaoqian Liu, and personChu-Ren Huang. year2020.
Motivations, Methods and Metrics of Misinformation Detection: An NLP Perspective.

authorpersonPeter Sullivan. year2020.
titleTrump retweets conspiracy theory questioning COVID-19 death toll.

The Hill.

[Thakur et al(2021)]
authorpersonNandan Thakur, personNils Reimers, personJohannes Daxenberger, and personIryna Gurevych. year2021.
Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks. In booktitleProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages296–310.

authorpersonPetter Törnberg. year2023.
ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning.
journalArXiv  volumeabs/2304.06588 (year2023).

[Trabasso and van den Broek(1985)]
authorpersonTom Trabasso and personPaul van den Broek. year1985.
Causal thinking and the representation of narrative events.
journalJournal of Memory and Language  volume24 (year1985), pages612–630.

[Trajkova et al(2020)]
authorpersonMilka Trajkova, personA’aeshah Alhakamy, personFrancesco Cafaro, personSanika Vedak, personRashmi Mallappa, and personSreekanth R. Kankara. year2020.
Exploring Casual COVID-19 Data Visualizations on Twitter: Topics and Challenges.
volume7, number3 (year2020), pages35.

Number: 3 Publisher: Multidisciplinary Digital Publishing Institute.

[Valdez et al(2020)]
authorpersonDanny Valdez, personMarijn ten Thij, personKrishna Bathina, personLauren A Rutter, and personJohan Bollen. year2020.
Social Media Insights Into US Mental Health During the COVID-19 Pandemic: Longitudinal Analysis of Twitter Data.
journalJ Med Internet Res volume22, number12 (date14 Dec year2020), pagese21418.

authorpersonDeirdre Walsh. year2020.
Congress Passes $900 Billion Coronavirus Relief Bill, Ending Months-Long Stalemate.
journalNPR (year2020).

authorpersonMischa Wanek-Libman. year2021.
titleCDC to revise mask requirements concerning outdoor areas of transportation systems.

[Wang et al(2023b)]
authorpersonGuanzhi Wang, personYuqi Xie, personYunfan Jiang, personAjay Mandlekar, personChaowei Xiao, personYuke Zhu, personLinxi (Jim) Fan, and personAnima Anandkumar. year2023b.
Voyager: An Open-Ended Embodied Agent with Large Language Models.
journalArXiv  volumeabs/2305.16291 (year2023).

[Wang et al(2021a)]
authorpersonHanyin Wang, personYikuan Li, personMeghan Hutch, personAndrew Naidech, and personYuan Luo. year2021a.
Using Tweets to Understand How COVID-19–Related Health Beliefs Are Affected in the Age of Social Media: Twitter Data Analysis Study.
journalJ Med Internet Res volume23, number2 (date22 Feb year2021), pagese26302.

[Wang et al(2023a)]
authorpersonSitong Wang, personSavvas Petridis, personTaeahn Kwon, personXiaojuan Ma, and personLydia B Chilton. year2023a.
PopBlends: Strategies for Conceptual Blending with Large Language Models. In booktitleProceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (seriesCHI '23). publisherAssociation for Computing Machinery, addressNew York, NY, USA, Article articleno435, numpages19 pages.

[Wang et al(2021b)]
authorpersonXingqiao Wang, personXiaowei Xu, personWeida Tong, personRuth Roberts, and personet al. year2021b.
InferBERT: A Transformer-Based Causal Inference Framework for Enhancing Pharmacovigilance.
journalFrontiers in Artificial Intelligence (year2021).

[Wei et al(2023)]
authorpersonJason Wei, personXuezhi Wang, personDale Schuurmans, personMaarten Bosma, personBrian Ichter, personFei Xia, personEd Chi, personQuoc Le, and personDenny Zhou. year2023.
titleChain-of-Thought Prompting Elicits Reasoning in Large Language Models.

[Weinzierl and Harabagiu(2022)]
authorpersonMaxwell Weinzierl and personSanda Harabagiu. year2022.
Identifying the Adoption or Rejection of Misinformation Targeting COVID-19 Vaccines in Twitter Discourse. In booktitleProceedings of the ACM Web Conference 2022 (New York, NY, USA) (seriesWWW '22). publisherAssociation for Computing Machinery, pages3196–3205.

[Wilhelms et al(2015)]
authorpersonEvan A. Wilhelms, personValerie F. Reyna, personPriscila Brust-Renck, and personet al. year2015.
Gist Representations and Communication of Risks about HIV-AIDS: A Fuzzy-Trace Theory Approach.
journalCurrent HIV research volume13, number5 (year2015), pages399–407.

[Wolfe et al(2015)]
authorpersonChristopher R. Wolfe, personValerie F. Reyna, personColin L. Widmer, and personet al. year2015.
Efficacy of a web-based intelligent tutoring system for communicating genetic risk of breast cancer: a fuzzy-trace theory approach.
journalMedical Decision Making: An International Journal of the Society for Medical Decision Making volume35, number1 (year2015), pages46–59.

[Wu et al(2022b)]
authorpersonTongshuang Wu, personMichael Terry, and personCarrie Jun Cai. year2022b.
AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts. In booktitleProceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (seriesCHI '22). publisherAssociation for Computing Machinery, addressNew York, NY, USA, Article articleno385, numpages22 pages.

[Wu et al(2022a)]
authorpersonTongshuang Sherry Wu, personEllen Jiang, personAaron Donsbach, personJeff Gray, personAlejandra Molina, personMichael Terry, and personCarrie J. Cai. year2022a.
PromptChainer: Chaining Large Language Model Prompts through Visual Programming.
journalCHI Conference on Human Factors in Computing Systems Extended Abstracts (year2022).

[Wu and Shen(2022)]
authorpersonYi Wu and personFei Shen. year2022.
Exploring the impacts of media use and media trust on health behaviors during the COVID-19 pandemic in China.
journalJournal of Health Psychology volume27, number6 (year2022), pages1445–1461.

PMID: 33646827.

[Xu and Guo(2018)]
authorpersonZhan Xu and personHao Guo. year2018.
Using Text Mining to Compare Online Pro- and Anti-Vaccine Headlines: Word Usage, Sentiments, and Online Popularity.
journalCommunication Studies volume69, number1 (year2018), pages103–122.

[Yang et al(2022)]
authorpersonJie Yang, personSoyeon Caren Han, and personJosiah Poon. year2022.
A survey on extraction of causal relations from natural language text.
journalKnowledge and Information Systems volume64, number5 (year2022), pages1161–1186.

[Yin et al(2023)]
authorpersonBaosheng Yin, personMeishu Zhao, personLu Guo, and personLiqi Qiao. year2023.
Sentence-BERT and k-means Based Clustering Technology for Scientific and Technical Literature. In booktitle2023 15th International Conference on Computer Research and Development (ICCRD). pages15–20.

[Zhang et al(2021)]
authorpersonRenwen Zhang, personNatalya N. Bazarova, and personMadhu Reddy. year2021.
Distress Disclosure across Social Media Platforms during the COVID-19 Pandemic: Untangling the Effects of Platforms, Affordances, and Audiences. In booktitleProceedings of the 2021 CHI Conference on Human Factors in Computing Systems (New York, NY, USA) (seriesCHI '21). publisherAssociation for Computing Machinery, pages1–15.

[Ziems et al(2023)]
authorpersonCaleb Ziems, personWilliam Held, personOmar Shaikh, personJiaao Chen, personZhehao Zhang, and personDiyi Yang. year2023.
titleCan Large Language Models Transform Computational Social Science?

[arxiv]2305.03514 [cs]

[Zottoli et al(2023)]
authorpersonTina M. Zottoli, personRebecca K. Helm, personVanessa A. Edkins, and personMichael T. Bixter. year2023.
Developing a model of guilty plea decision-making: Fuzzy-trace theory, gist, and categorical boundaries.
journalLaw and Human Behavior volume47, number3 (year2023), pages403–421.

Place: US Publisher: Educational Publishing Foundation.

§ STUDY 1 MODEL PERFORMANCE COMPARISON RESULTS

In our study, we trained several baseline models on our human-annotated Reddit dataset to evaluate their performance in cause-effect pair extraction. Table <ref> summarizes the performance metrics of these models.

Model performance on cause-effect pair extraction from our manually labeled Reddit dataset.

customcolor2 Model     Precision     Recall     F1-score     Accuracy

BERT      0.709     0.708     0.707     0.711

RoBERTa      0.826     0.829     0.814     0.846
XLNet      0.804     0.819     0.811     0.821
RBIC ()       0.934     0.949     0.941     0.940
RBIC ()       0.972     0.951     0.961     0.977

§ STUDY 4 STATISTICAL RESULT

Result of Granger causality test for relationships between gist clusters (C1, C2 and C3) and health outcomes dataset.
customcolor2 Cluster     Health Outcomes     Hypotheses     Lag     $F$-val     $p$-val     Direction of Impact

103cmC1. Vaccination Implications
NV     C1 $\not\to\mathcal{}$ NV     4    4.778    0.027     gists impacts NV

NV     NV $\not\to\mathcal{}$ C1     14     7.771     0.007     NV impacts gists

GH     C1 $\not\to\mathcal{}$ GH     11    1.765    0.271     -

GH     GH $\not\to\mathcal{}$ C1     5     1.015     0.410     -

ICU     C1 $\not\to\mathcal{}$ ICU     6     1.118     0.402     -

ICU     ICU $\not\to\mathcal{}$ C1     3     1.818     0.242     -

TC     C1 $\not\to\mathcal{}$ TC     11    2.495     0.089     -

TC     TC $\not\to\mathcal{}$ C1     11     2.960     0.072     -

NC     C1 $\not\to\mathcal{}$ NC     4    2.986    0.068     -

NC     NC $\not\to\mathcal{}$ C1     9     0.749     0.488     -

103.4cmC2. Controversies and Policies Related to Masks

NV     C2 $\not\to\mathcal{}$ NV     17    0.654    0.5124      -

NV     NV $\not\to\mathcal{}$ C2     7     0.745     0.4888     -

GH     C2 $\not\to\mathcal{}$ GH     3    1.110    0.411     -

GH     GH $\not\to\mathcal{}$ C2     5    2.210     0.091     -

ICU     C2 $\not\to\mathcal{}$ ICU     4    2.282    0.090     -

ICU     ICU $\not\to\mathcal{}$ C2     14     1.416     0.343     -

TC     C2 $\not\to\mathcal{}$ TC     5     9.395     0.005     gists impacts TC

TC     TC $\not\to\mathcal{}$ C2     8     1.101     0.414     -

NC     C2 $\not\to\mathcal{}$ NC     5     11.829    0.004     gists impacts NC

NC     NC $\not\to\mathcal{}$ C2     7     1.181     0.409     -

102.9cmC3. Impact of Lockdown

NV     C3 $\not\to\mathcal{}$ NV     4    2.437    0.084     -

NV     NV $\not\to\mathcal{}$ C3     4     1.101     0.413     -

GH     C3 $\not\to\mathcal{}$ GH     12    1.118    0.410     -

GH     GH $\not\to\mathcal{}$ C3     9    8.822    0.006     GH impacts gists

ICU     C3 $\not\to\mathcal{}$ ICU     7    0.715    0.494     -

ICU     ICU $\not\to\mathcal{}$ C3     14    4.000     0.039     ICU impacts gists

TC     C3 $\not\to\mathcal{}$ TC     4     1.135     0.409     -

TC     TC $\not\to\mathcal{}$ C3     11     0.796     0.470     -

NC     C3 $\not\to\mathcal{}$ NC     5     2.357     0.088     -

NC     NC $\not\to\mathcal{}$ C3     1     1.178     0.382     -

Result of Granger causality test for relationships between gist clusters (C4 and C5) and health outcomes dataset
customcolor2 Cluster     Health Outcomes     Hypotheses     Lag     $F$-val     $p$-val     Direction of Impact

103.9cmC4. Societal and Economic (Macro) Impact of COVID-19

NV     C4 $\not\to\mathcal{}$ NV     3    2.579    0.072     -

NV     NV $\not\to\mathcal{}$ C4     3     2.014     0.091     -

GH     C4 $\not\to\mathcal{}$ GH     9    1.787    0.261     -

GH     GH $\not\to\mathcal{}$ C4     4    2.072    0.080     -

ICU     C4 $\not\to\mathcal{}$ ICU     15    2.618    0.066     -

ICU     ICU $\not\to\mathcal{}$ C4     1    2.071     0.081     -

TC     C4 $\not\to\mathcal{}$ TC     1     1.780     0.263     -

TC     TC $\not\to\mathcal{}$ C4     9    3.663    0.031     TC impacts gists

NC     C4 $\not\to\mathcal{}$ NC     4     1.100    0.412     -

NC     NC $\not\to\mathcal{}$ C4     9    3.092    0.032     NC impacts gists

103.9cmC5. Social Issues, Health, and Personal Experiences

NV     C5 $\not\to\mathcal{}$ NV     7    2.385    0.085     -

NV     NV $\not\to\mathcal{}$ C5     4     1.101     0.415     -

GH     C5 $\not\to\mathcal{}$ GH     12    0.786    0.471     -

GH     GH $\not\to\mathcal{}$ C5     14    0.799    0.464     -

ICU     C5 $\not\to\mathcal{}$ ICU     1    1.119    0.399     -

ICU     ICU $\not\to\mathcal{}$ C5     9    2.282     0.090     -

TC     C5 $\not\to\mathcal{}$ TC     5     2.001     0.107     -

TC     TC $\not\to\mathcal{}$ C5     2    1.137    0.389     -

NC     C5 $\not\to\mathcal{}$ NC     2     2.427    0.079     -

NC     NC $\not\to\mathcal{}$ C5     1    1.710    0.212     -

§ COST AND TIME OF RUNNING RBIC

Comparative Analysis of Cost and Time Efficiency for RBIC Framework Using GPT-4 and GPT-3.5 Models. The table provides a detailed breakdown of the estimated costs and processing time required for analyzing 79,680 posts using two different OpenAI LLMs.

customcolor2  Model     Cost per Token (Input)     Cost per Token (Output)      Cost per Post     Total Cost      Total Time(hrs)

GPT-4          $0.03 / 1K tokens           $0.06 / 1K tokens             $0.0078                         $621.50                199.20

GPT-3.5        $0.0010 / 1K tokens         $0.0020 / 1K tokens          $0.00039                          $31.07                154.93

Cost estimation is important for researchers considering the RBIC framework in their studies, especially with large datasets. Applying the GPT-4 pricing model of $0.06 per 1,000 tokens (to the combined total of prompts, input posts, and outputs), the total cost for using RBIC on our dataset of 79,680 posts amounted to $621.51, translating to roughly $0.0078 per post. The total duration of running RBIC on GPT-4 was 199.2 hours or 8 seconds per post, as shown in Table 9. While we did not use GPT-3.5 in this study, we show the estimated cost and running time based on GPT-3.5's pricing at the time of this study. While we used GPT-4, which can be expensive for large datasets, the RBIC framework is adaptable to other open-source large language models that may be more cost-effective.

§ DESIGN FEATURE IN THE MODERATION DASHBOARD

Design Implication: Illustration of moderator’s view of a submitted post or comment.