# Connecting the Dots in Social Media Data - Harnessing Causal Reasoning using
Knowledge Graph Enhanced LLM’s

Rahul Ravi  Dept. of Computer Science
University of Calgary
Calgary, Canada
<EMAIL_ADDRESS>Gouri Ginde  Dept. of Electrical and Software
Engineering
University of Calgary
Calgary, Canada
<EMAIL_ADDRESS>Jon Rokne  Dept. of Computer Science
University of Calgary
Calgary, Canada
<EMAIL_ADDRESS>
###### Abstract

X (formerly Twitter) provides rich source of first hand experience details and
information reporting frequently used by lay people. Thus, as social media
platforms grow, understanding the underlying reasons behind events and
statements becomes crucial for businesses, policymakers, and researchers. This
research explores the integration of knowledge graphs (KGs) with large
language models (LLMs) to perform causal analysis of tweets dataset. The LLM
aided analysis techniques often lack depth in uncovering the causes driving
observed effects. By leveraging KGs and LLMs, which encode rich semantic
relationships and temporal information, this study aims to uncover the complex
interplay of factors influencing causal dynamics and compare the results with
GPT-3.5 Turbo. We employ a Retrieval-Augmented Generation (RAG) model,
utilizing a KG stored in Neo4j, to retrieve relevant context for causal
reasoning. Our approach demonstrates that the KG-enhanced LLM RAG can provide
an edge over the baseline LLM (GPT-3.5 Turbo) model as the source corpus
increases in size. Our findings highlight the advantages of combining KGs with
LLMs for improved interpretability and actionable insights, facilitating
informed decision-making across various domains. On average across the three
metrics, we observed an improvement in the responses by about 11.86%.

###### Index Terms:

Index Terms: Knowledge Graphs (KG), Natural Language Processing (NLP), Causal
Analysis, Large Language Models (LLM), Relation Extraction, Retrieval-
Augmented Generation (RAG), Embeddings.

##  I Introduction

The pervasive influence of social media platforms has revolutionized
communication, enabling individuals to express opinions, share experiences,
and engage in discourse on a global scale. With the exponential growth of
social media usage, understanding the underlying reasons behind events and
statements has become paramount for businesses, policymakers, and researchers
alike. With the rise of LLMs, such as Generative Pre-trained Transformer (GPT)
and Bidirectional Encoder Representations from Transformers (BERT), state-of-
the-art language processing techniques have shown great success in capturing
contextual and temporal information. LLMs such as GPT-3.5 Turbo provide deep,
context-aware language understanding that is critical for deciphering the
subtle nuances of language and time-based progression used in social media
discourse. However, these techniques often provide surface-level insights and
lack the depth to uncover the root causes driving observed effects
[jin2023cladder].

Knowledge graphs, which function on the graph theory fundamentals, store data
in the form of nodes and links and offer a structured representation of
domain-specific knowledge capturing rich semantic relationships between
entities [kejriwal2021knowledge]. They also provide semantic frameworks for
organizing domain-specific information that can be queried using the graph
structure [dieudonat2020exploring] [peng2023knowledge]. The KG for this
specific study incorporates an additional dimension in it’s representation.
Temporal information plays a critical role in understanding causal
relationships, particularly in the context of social media data such as
tweets. The sequence in which events occur and the time intervals between them
can provide essential insights into the causation behind observed effects.
Previous studies have demonstrated the importance of integrating temporal
information into causal reasoning frameworks. For example, in
[hong2017event2vec], the authors discuss the challenges and methodologies for
incorporating time into causal analysis. They emphasize the need for precise
timestamping and chronological ordering of events to accurately infer
causation

Thus, in this research, we explore the effectiveness of integrating knowledge
graphs (KGs) and large language models (LLMs) in performing causal analysis of
text-based social media data such as tweets from X (formerly Twitter) platform
[pan2024unifying]. We hypothesize that these two approaches (LLM and KG) can
enable a effective detection of causes and their underlying effects, providing
a richer understanding of the dynamics in social media interactions
[dieudonat2020exploring]. Thus, the goal of this study is to explore and
compare the hybrid method, a combination of KG and LLM, to enhance the
interpretability and actionable insights derived from causal analysis by
delving deeper into the underlying drivers of events and statements in time
series tweets. Hence, we propose Retrieval-Augmented Generation (RAG) as a key
component, in our methodology, that binds KG and LLM together as shown in Fig
1. RAG combines retrieval mechanisms with generative models to enhance the
generation process by providing relevant contextual information. Specifically
[ding2024survey], RAG involves using a retriever model to fetch pertinent
documents or knowledge snippets and a generator model to produce responses
conditioned on both the query and the retrieved information. This approach
ensures that the model generates contextually accurate and informative
outputs, leveraging the vast knowledge encapsulated in KGs and the
sophisticated language understanding of LLMs.

RAG offers a significant advantage over using an LLM alone. While traditional
approaches might directly query an LLM without fine-tuning, they are limited
by the scope and diversity of the data used during training. RAG, on the other
hand, dynamically incorporates external knowledge at inference time, allowing
for a more flexible and contextually enriched response generation
[gao2023retrieval]. This dynamic integration of knowledge makes RAG
particularly well-suited for tasks requiring up-to-date information and
complex reasoning, such as causal analysis in social media data. A graph DB is
efficient in storage as well as querying for fast results so it is an
efficient platform to perform context extraction [lewis2020retrieval].

The paper is structured as follows: Section III details various elements of
this study. Section IV provides details about the datasets used in this study,
and Section V explains the methodology of this study followed by Section VI
showcasing results further. Finally Section VIII lists the limitations of the
study and Section VII elaborates related work finally concluding with IX
future works and conclusion.

We compare GPT model with our hybrid approach of KG-enhanced LLM RAG for
causal analysis on the COVID-19 and XYZ dataset.

Figure 1: Integrated Representation of KG and LLM through RAG

##  II Motivating example

During the COVID-19 pandemic, we noticed a recurring theme in tweets where
people frequently mentioned an increased appreciation for simple things in
life. This observation sparked our curiosity, and we attempted to manually
understand the reasons behind this sentiment. However, this process proved to
be extremely challenging due to the scattered and unstructured nature of the
relevant tweets.

To answer a question like, ”What caused people to appreciate simple things
around them during the COVID-19 pandemic?” we had to sift through countless
tweets. These tweets were not posted sequentially nor were they directly
related to each other. The information was dispersed across different parts of
the dataset, making it difficult to piece together a comprehensive answer
manually.

After spending significant effort manually gathering and interpreting
scattered pieces of information over the course of a time period, we were able
to derive an answer. We found that people’s appreciation for simple things was
influenced by various interconnected factors. The lockdown measures led to
isolation, which in turn heightened people’s awareness of their immediate
surroundings. The restriction of activities forced individuals to spend more
time at home, allowing them to notice and value the simple things that were
always around them. This combination of factors provided a sense of comfort
and appreciation during challenging times.

This manual process highlighted the need for a more efficient solution. The
difficulty in manually analyzing the vast and scattered data from social media
underscored the necessity of developing a method that could automatically
integrate and analyze this information to uncover deeper insights. Thus, we
aimed to understand and interpret the complex dynamics of social media
interactions represented on a temporal graph and automate this difficult task
further.

##  III Background

###  III-A Knowledge Graphs

Knowledge Graphs provide semantic frameworks for organizing domain-specific
information that can be queried using the graph structure [peng2023knowledge].
In this research, KGs are used to query for semantic information as found in
textual data stored as KGs, facilitating causal analysis of social media data.
A few of the benefits of using KG are as follows:
Semantic Enrichment- When textual data is processed and stored in a KG, it
enriches the textual data with structured semantic information, facilitating a
deeper understanding of the underlying context and relationships between
textual information items. This enrichment is essential for discerning the
underlying drivers of events and statements, enabling stakeholders to identify
causal relationships more effectively.
Contextual Insights- The contextual information encoded within KGs, such as
entity relationships and attributes, can be leveraged to gain insights into
the broader context surrounding discussions on social media platforms. This
contextual understanding enables efficient interpretations of causal dynamics
and aids in identifying the root causes driving observed effects.
Multi-dimensional Analysis- KGs enable a multi-dimensional analysis of social
media data by incorporating diverse sources of information, including domain-
specific knowledge, user profiles, and social network structures. This multi-
dimensional perspective enhances the robustness of causal analysis, allowing
for a comprehensive exploration of the factors influencing observed events and
statements.

###  III-B Graph Database

Neo4j is a leading graph database platform that provides a highly efficient
and scalable environment for storing and querying knowledge graphs
[tjortjisgraph]. One of the key advantages of using a graph database like
Neo4j is its ability to represent and manage complex relationships within
large text corpora in a structured manner. It allows for the inclusion of
temporal information as properties on the edges, which is essential for
understanding the temporal dynamics of causal relationships. By storing
timestamps on the relational edges, the graph can accurately represent the
chronological order of events and their causal links. This enables more
sophisticated queries that take into account the sequence and timing of
events, leading to deeper insights and more accurate causal inferences. A few
of the advantages that important for our study are as follows:
Dynamic Schema - Unlike relational databases, Neo4j allows for a flexible
schema that can evolve as new types of nodes and relationships are added,
making it ideal for dynamic and heterogeneous data like social media content.
Efficient Querying - Neo4j’s query language, Cypher, is designed for efficient
traversal and querying of graph structures, enabling rapid extraction of
relevant semantic information from large datasets.
Real-time Updates - Neo4j supports real-time updates and querying, allowing
for dynamic integration of new information as it becomes available, which is
crucial for timely analysis of social media data.
Semantic Enrichment - By leveraging the graph structure, Neo4j enhances the
capture of semantic context of textual data, facilitating more accurate and
meaningful causal analysis.

###  III-C Large Language Models

The advent of Large Language Models (LLMs), such as GPT and BERT, has
revolutionized natural language understanding tasks [minaee2024large]. LLMs
possess remarkable capabilities in capturing intricate semantic nuances within
textual data, thereby offering unparalleled opportunities for performing
reasoning and inference tasks. LLMs are trained on vast corpora of textual
data, enabling them to develop a deep understanding of language structures and
variations. This extensive training allows LLMs to generate predictions (text
generative LLMs such as GPT) or encodings (encoders such as BERT, RoBERTa)
that reflect the underlying semantic properties of the text. LLM frameworks
serve multiple purposes:
Contextual Understanding -  LLMs strongly interpret local context withing
sentences, which is crucial when analyzing texts as dynamic and varied as
social media posts. This contextual understanding aids in understanding
accurate meaning of words within the context and their intensities.
Semantic Richness -  By employing LLMs, we can extract rich semantic encodings
from the text, which capture deeper levels of meaning that go beyond basic
classification tasks. These encodings provide a nuanced view of the combined
text corpus, revealing underlying themes and emotional currents.
Integration with Knowledge Graphs - This paves way for enhanced causal
analysis. The contextual understanding from LLMs can be ustilised on a limited
scope of context derived from knowledge graphs, enhancing the capability of
the graphs to model complex relationships and interactions within the data.
This integration allows for a more structured exploration of how different
effects and causal drivers are interconnected.

#### GPT-3.5 Turbo

GPT-3.5 Turbo is an advanced and sophisticated version of the GPT models by
OpenAI used as the baseline LLM in this study. It is particularly adept in
comprehension of queries and context, resulting in more accurate and
contextually relevant responses [zeidisynergizing]. It can handle longer and
complex contexts which is especially necessary in our case (complete text
corpus for the baseline model). It is easily accessible through integration
with OpenAI’s API and is known to produce high quality responses, such as
causal reasoning. In the subsequent sections of this paper, we detail the
procedure of how the LLMs are employed, the process of relation extraction,
and how they are used to perform prompt engineering with appropriate contexts
derived from the KG and thus inferring causality.

###  III-D Embedding and Encoding Techniques

Dimensionality reduction is the process of reducing the high level text
representations into a 2D vector space that provide richer meaning to the
data, otherwise known as embedding. This research performs 2 kinds of
embeddings in order to combine KG and LLM for optimised context extraction.

  * •

The KG is embedded using Node2Vec method that captures semantic structural
meaning among sentences. It is performed on nodes and edges.

  * •

The query to infer cause is encoded with LLM based Bert-uncased model along
with the nodes in the graph to identify highly similar sentences contexually,
which is then expanded into the source context by clustering semantically
close sentences to it obtained from the graph embedding.

####  III-D1 Node2Vec

Node2Vec is a powerful technique for generating node embeddings by simulating
random walks on the graph and applying the Skip-Gram model. It belongs to the
family of unsupervised learning algorithms and is able to capture semantic
similarities between nodes based on their structural context within the graph
[wang2021kg2vec] [grover2016node2vec].

The following steps are performed for node embeddings in the context of
knowledge graphs:

  * •

Node2Vec Walks - Random walks are simulated on the graph, generating sequences
of nodes based on their connections. This process captures the graph’s
structural and semantic properties. By incorporating temporal information into
the edges, the random walk process is modified to respect the chronological
order of events. This means that the algorithm can prioritize or constrain
transitions to follow the temporal flow of events, leading to walks that
better capture the temporal dynamics of the graph.

  * •

Learning Node Embeddings - The Skip-Gram model, a variant of Word2Vec, is
applied to the sequences of nodes generated from the random walks. This
results in dense, low-dimensional vector representations (embeddings) for each
node in the graph.

  * •

Aggregating Node Embeddings - The embeddings for nodes are aggregated which
can then be used for various downstream tasks such as node classification,
clustering, and link prediction that are also closely representative of the
real-world temporal sequences and dependencies in the data.

Node2Vec embeddings were selected for this work due to their ability to
capture semantic and temporal relationships between nodes based on their
structural context. This capability is essential for understanding the nuanced
connections in social media data, facilitating the identification of
underlying factors driving events and statements. Leveraging Node2Vec
embeddings enables efficient transfer learning, harnessing the structural
knowledge encoded in the graph.Their interpretability, efficiency, and
scalability make them well-suited for large-scale causal analysis tasks,
empowering stakeholders with deeper insights and informed decision-making
capabilities.

####  III-D2 BERT-Uncased

BERT (Bidirectional Encoder Representations from Transformers) is a
transformer based model used for text comprehension tasks. The uncased
encoding technique works by first converting all input text to lowercase and
then tokenizing it using a WordPiece tokenizer. The tokenized input is then
transformed into input encodings by summing token, segment, and position
encodings. These encodings are fed into a transformer model with multiple
layers of bidirectional self-attention, which captures contextual
relationships between words in both directions. The output from the
transformer layers provides encodings for each token. The process involves:

  * •

Lowercasing and Tokenization: Convert text to lowercase and tokenize it into
subword units.

  * •

Encoding Representation: Convert tokens to input encodings by summing token,
segment, and position encodings.

  * •

Transformer Layers: Pass the encodings through transformer layers to capture
context.

  * •

Extracting Encodings: Obtain final encodings for each token, using them as
needed for various NLP tasks.

Figure 2: Combining Embeddings and Encodings during RAG to Identify Context

=

###  III-E Synergized KG and Large Language Models through RAG

In this research, KGs are used in conjunction with LLMs to construct the graph
(enhancing KG), obtain context (integration - middleman) and use it for prompt
engineering (enhancing LLM).

LLMs have been used in conjunction with KGs in the past especially, GPT
models. We leverage the integration of GPT with KGs to enhance the querying
process, especially given that our KG contain correlation information rather
than direct causation. While traditional language models like GPT excel in
understanding and generating context-aware responses, they often lack the
structured knowledge needed to fully grasp the complex relationships present
in social media data as explained here, [chan2024exploring]. By combining GPT
with KGs, we can dynamically retrieve and incorporate contextually relevant
information, thereby enriching the model’s responses with structured
correlation data from the KG. This integration allows us to effectively
identify and articulate the intricate web of correlations between events and
statements, improving our ability to interpret and respond to queries with
greater accuracy and depth. Our research takes this concept a step further by
enhancing the KGs with embeddings before using GPT-3.5 Turbo to make
inferences.

The process begins with a relation extraction LLM, which is fine-tuned to
identify entities and relationships from text and generate triples. These
triples are then used to construct a KG. Once the KG is constructed, Node2Vec
embeddings are generated for the nodes, capturing the semantic relationships
between entities.

The next step involves integrating these graph embeddings with LLMs for causal
reasoning (Fig. 2). The LLM is used with Retrieval-Augmented Generation (RAG),
where relevant context from the KG is retrieved and provided to the LLM to
generate contextually enriched responses. This combined approach leverages the
structured knowledge in KGs and the deep language understanding of LLMs,
enhancing the model’s ability to identify and analyze causal relationships.
BERT-Uncased is used to identify relveant context which is expanded based on
graph structure.

Finally, the retrieved context is used by the GPT-3.5 Turbo to reason causes
behind user defined queries.

By using KGs and LLMs together, this research aims to improve the accuracy and
depth of causal analysis, providing a more comprehensive understanding of the
factors influencing social media data. The combined approach demonstrates
superior performance compared to using LLMs or KGs alone, as it benefits from
the strengths of both techniques. The comparative analysis in this research
highlights the efficiency and effectiveness of this integrated method in
uncovering the causal factors driving events and statements on social media
platforms.

##  IV Dataset

We obtained the COVID19 Tweets dataset from Kaggle [gabriel_preda_2020].
Dataset had thirteen features/fields however, we used the ”text” that contains
all the tweets along with ”date” that contains timestamps for each tweet.
Other related data for future work might include ”user_location” (geography
based analysis of causes) and ”date” (temporal parameter that directly affects
causation). This is one of the highly sought after dataset with more than 22k
downloads to date, mainly for data cleaning and learning purposes as stated on
the website [gabriel_preda_2020].

The tweets were gathered between 02/28/2020 and 07/23/2020 with a coverage
across the world using Twitter API with the hashtag, ”#covid19”. It’s size is
68.71MB and contains over 35k rows. On average each tweet has around 17 words
after pre-processing.

##  V Methodology

###  V-A Research Question

We have formulated a key research question focused on evaluating the
effectiveness of combining large language models (LLMs) with knowledge graph
(KG) embeddings for causal analysis:

Does the integration of KG and LLM through RAG enhance causal reasoning
capabilities on social media data as opposed to using LLMs without context?

Explanation: This research question aims to investigate whether integrating
knowledge graph embeddings with LLMs enhances the context for querying and
accuracy of causal reasoning in social media data. By combining the structured
knowledge captured in KGs with the deep contextual understanding provided by
LLMs, we hypothesize that this integrated approach will yield superior results
in identifying and reasoning about causation. The expected inputs for this
question include raw social media text data, knowledge graph embeddings, and
LLMs (without fine-tuning) for encoding and prompt engineering. The expected
output or solution would involve demonstrating improved performance in causal
reasoning tasks when using the combined approach, as compared to using LLMs
without the integration of KG embeddings, in other words without specific
contexts to query on.

Input: Text data $T$, Relation extraction model $R$, Node2Vec model $N$, BERT
Uncased Encoder $B$, LLM $M$, Graph database $DB$

Output: Causal inferences

0: Text data $T$, Relation extraction model $R$, Node2Vec model $N$, LLM $M$,
Graph database $DB$

10: Causal inferences

1: $T_{preprocessed}\leftarrow\text{Preprocess}(T)$

2: $E,Rel\leftarrow\text{ExtractEntitiesAndRelations}(T_{preprocessed},R)$

23: $G\leftarrow\text{ConstructKnowledgeGraph}(E,Rel)$

4: if GraphDBExists(DB) then

5: for each node, edge in $G$ do

6: AddNodeToGraphDB(node, DB)

7: AddEdgeToGraphDB(edge, DB)

8: end for

9: else

10: InitializeGraphDB(G, DB)

311: end if

412: $E_{embeddings}\leftarrow\text{GenerateEmbeddings}(E,N)$

13: for each $query$ in $queries$ do

14: $query\\_encoding\leftarrow\text{GenerateEncoding}(query,B)$

15:
$similarities\leftarrow\text{CalculateSimilarities}(query\\_encoding,E_{embeddings})$

16: $TopK\leftarrow\text{SelectTopKSubgraph}(G,similarities,5)$

517: $context\leftarrow\text{RetrieveContext}(TopK)$

18: Using Retrieved Context (RAG)

19: $Answer\\_RAG\leftarrow\text{GenerateAnswer}(query,context,M)$

620: StoreAnswer(Answer_RAG)

21: Using Entire Corpus

22: $Answer\\_Corpus\leftarrow\text{GenerateAnswer}(query,T_{preprocessed},M)$

23: StoreAnswer(Answer_Corpus)

724: end for

25:
$F1_{RAG}\leftarrow\text{EvaluateModel}(Answers\\_RAG,\text{TestingDataset})$

26:
$F1_{Corpus}\leftarrow\text{EvaluateModel}(Answers\\_Corpus,\text{TestingDataset})$

827: $\Delta F1\leftarrow F1_{RAG}-F1_{Corpus}$

28: for each $query$ in $queries$ do

29:
$Best\\_Answer\leftarrow\text{SelectBestAnswer}(Answer\\_RAG,Answer\\_Corpus)$

30: StoreBestAnswer(Best_Answer)

931: end for

32: $CausalInferences\leftarrow\text{InferCauses}(Best\\_Answers)$

33: return $CausalInferences$

Algorithm 1 Pseudo Code for Causal Inference with Knowledge Graph Enhanced LLM

The research methodology combines causal analysis with temporal knowledge
graph (KG) construction to comprehensively understand the underlying reasons
behind events and statements in social media data. The process involves
several stages: data preprocessing, entity and relationship extraction,
knowledge graph construction, graph embeddings generation, and evaluating the
performance of the integrated approach. Additionally, we utilize a Retrieval-
Augmented Generation (RAG) process to enhance causal inference. The
methodology entails the following steps:

###  V-B Study design

Figure 3: Overview of study design

Pre-processing: Figure 3 shows the overall architecture of our study. First
step ❶ is data pre-processing which is common to both baseline and our
proposed method. In addition to tweets’ text, we also extract timestamps from
tweets and ensure their accuracy by handling different time zones, correcting
any inconsistencies, and standardizing the format. The tweets strings are
highly unstructured, ambiguous and full of noise (special characters, acronyms
and URL’s), Thus we first utilize a dictionary to expand contractions of words
often used in social media slangs with their appropriate representation to
replace them. For example, ”aren’t”: ”are not / am not”, ”can’t’ve”: ”cannot
have”, etc. We also remove special characters, emojis, URLs and Hashtags.

####  V-B1 Baseline

####  V-B2 KG-LLM

The proposed KG-LLM is a three-step process: Relation extraction,
Embedding/Encoding, and Context Retrieval and QA, as illustrated in Figure 3.
Each step leverages specific technologies chosen for their capabilities to
enhance the overall process.

Relation extraction: In step ❷, the LLaMa3 model [towardsDS], specifically the
fine-tuned variant from Hugging Face (solanaO/llama3-8b-sft-qlora-re), is used
for relation extraction, identifying triplets from the pre-processed tweets
(step ❸). LLaMa3 [wadhwa2023revisiting], known for its advanced natural
language understanding, enhances traditional relation extraction methods by
leveraging supervised fine-tuning from LLaMa3-70B. These extracted triplets
are then stored in as a KG within a graph database, specifically Neo4j (step
❹). Neo4j is employed due to its powerful querying capabilities and efficiency
in handling dynamic data updates, making it an ideal choice for managing the
complex and evolving data from social media.

Entity extraction involves identifying and extracting meaningful entities
(words) from the preprocessed text. A relation extraction model
(solanaO/llama3-8b-sft-qlora-re [towardsDS]) is employed to extract entities
and their relationships from text data in the form of triples. The process
begins by loading the pre-trained model with a PEFT adapter, ensuring
efficient memory usage. The tokenizer is then used to create prompts from the
preprocessed text. The model generates outputs based on these prompts,
extracting entities and their relationships, and forming the foundational
triples for the knowledge graph.

Knowledge Graph Construction Once entities and relationships are extracted, a
knowledge graph is constructed. In this graph: Nodes represent entities (e.g.,
words, phrases) and Edges represent relationships between entities based on
the triples generated by the relation extraction model.

Integrating with Graph DB The knowledge graph is stored in the Neo4j graph
database (DB), which allows for efficient querying and dynamic updates as new
sentences are processed. New nodes and edges from real time tweets are merged
on to the database. Their timestamps are added are edge properties.

Embedding and Encoding: In the second step, the DB is queried for KG and it is
embedded using Node2Vec. This method is chosen for its effectiveness in
capturing both the local and global structural information of the graph,
creating high-quality vector representations. These embeddings are stored in
Neo4j, which supports efficient retrieval of contextually relevant data,
crucial for uncovering causal relationships. Neo4j’s graph database structure
allows for dynamic updates and seamless integration with Node2Vec embeddings,
ensuring that the graph remains up-to-date with the latest data. When a user
inputs a query related to causal analysis, the query is encoded using an LLM
(BERT-Uncased) to create a vector representation. This is mapped onto the
correcponding node2vec embeddings that are highly similar and their proximity
to highly similar embeddings in the node2vec space provide the most relevant
context.

Context Retrieval and QA: The final step involves retrieving the most relevant
contexts from the Knowledge Graph to engineer prompts, a process known as
prompt engineering. These prompts are fed into a GPT-3.5 Turbo model, chosen
for its strong performance in understanding and generating natural language
text, to generate a response. The response, enriched with relevant context
from the Knowledge Graph, is provided to the user, ensuring a detailed and
comprehensive answer.

Graph Embeddings Generation The objective is to maximize the probability of
observing a node’s neighbors given its embedding through Node2vec algorithm,
as:

$\max\sum_{u\in V}\sum_{v\in N(u)}\log\Pr(v|u)$

where $\Pr(v|u)$ is the probability of node $v$ being a neighbor of node $u$,
computed using the softmax function:

$\Pr(v|u)=\frac{\exp(\mathbf{z}_{v}^{\top}\mathbf{z}_{u})}{\sum_{w\in
V}\exp(\mathbf{z}_{w}^{\top}\mathbf{z}_{u})}$

Here, $\mathbf{z}_{u}$ and $\mathbf{z}_{v}$ are the embeddings of nodes $u$
and $v$, respectively. The learned embeddings are stored in the graph database
for efficient retrieval and analysis. In a dynamic graph where new events
continuously occur, by running periodically while incorporating the latest
temporal data, we ensure that the embeddings stay up-to-date with the evolving
graph. They are then used to calculate similarity scores and retrieve the most
relevant subgraphs, which provide the context for causal reasoning tasks.

Retrieval-Augmented Generation (RAG) This is the step where the source context
is identified for prompt engineering for causal reasoning. It involves:
Encoding the Query: The input query is first embedded using the BERT-Uncased
LLM encoding model.
Retrieving Relevant Context: BERT-Uncased LLM is used to encode the KG to
identify top 25 contextually similar set of nodes, edges with the encoded
query which are then used to retrieve semantically similar neighbouring sets
based on Node2Vec. Both are validated for threshold of 0.35 using Cosine
similarity scores, which has been decided with respect to the volume of data
being handled in the KG and the size of the context window to provide the LLM
for querying. This has a direct implication to the amount of knowledge that is
passed on to the LLM to consider for reasoning causes hence, after a series of
trials and inspections of the contexts with various thresholds, this number
proved ideal with the right amount of context that could potentially provide
true cause. These subgraphs are mapped onto the preprocessed sentences, thus
retrieving the top ranked sentences (contextually and semantically most
similar to the input statement) that serves as the source context for
prompting with LLM. By including temporal constraints, the retrieved context
respects the chronological order of events .
Generating Responses: The retrieved context is used by the GPT-3.5 Turbo LLM
to generate responses with an effective prompt instructing the model to infer
cause for the input query.

By incorporating relevant contextual information dynamically retrieved from
the knowledge graph, RAG improves the model’s ability to understand and reason
about causal relationships in social media data, providing richer and more
informative responses.

####  V-B3 Evaluation and Comparative Analysis

The performance of the integrated approach is evaluated and compared with a
baseline approach using LLMs without KG embeddings. The baseline LLM processes
the entire corpus without specific context from the knowledge graph,
generating responses based solely on the raw text data. The evaluation process
involves manually designed queries from the tweets and correct causes created
for validation. The goal is to determine which approach yields the closest
truth or most relevant cause statements to the right answers based on specific
metrics. The LLM is not fine-tuned for this specific task; instead, it
utilizes the pre-trained model to generate responses based on the provided
context. The evaluation process is conducted as follows:

  * •

Manual Query Design: A set of queries is manually designed to reflect typical
causal questions that might arise in social media data analysis. Eg: ”What led
to people wearing masks incorrectly during the COVID-19 pandemic?”

  * •

Manual Cause Creation: Correct causes for each query are manually created to
serve as the ground truth for evaluation. Eg: ”The misinformation about
COVID-19 and mask usage caused people to wear masks incorrectly.”

  * •

Response Generation: Responses are generated using two approaches:

    1. 1.

LLM with RAG Context: The LLM generates responses using the context retrieved
from the knowledge graph via the RAG process.

    2. 2.

LLM without Context: The baseline LLM generates responses using the entire
text corpus without additional context.

  * •

Comparison with Ground Truth: The generated responses are compared with the
manually created correct causes using BLEU, Jaccard Similarity and BERT-
Uncased LLM encoding similarity scores to evaluate the effectiveness of the
responses.

#### Why the Metrics are Appropriate

The chosen metrics are appropriate for validating our results because:

  * •

BLEU Score: (BiLingual Evaluation Understudy) It evaluates the precision of
the generated responses by measuring how many words or phrases in the
generated text match the reference causes. This is crucial for assessing
whether the model can accurately generate causes that are close to the ground
truth. It calculates overall similarity between the reference text and the
generated text. It’s a value between 0 and 1 with 1 being the highest
similarity.

  * •

Jaccard Similarity: It measures similarity between finite sample sets, defined
as the size of the intersection divided by the size of the union of the sample
sets. Higher scores indicate greater overlap between the sets of tokens in the
generated and reference texts. Since it records presence/absence of tokens, it
is effective to use in our case that needs checks for presence of causal
keywords.

  * •

Cosine Similarity on BERT-Uncased Encodings: The LLM encoder captures
contextual meanings within data and it helps our case since we are not looking
at the exact structure formation but the information that the sentences are
conveying. It’s a value between 0 and 1 with 1 being the highest similarity.

By comparing the performance of the LLM with RAG context and the baseline LLM
without context, this evaluation aims to demonstrate whether enhanced causal
reasoning is achieved through the integration of knowledge graphs and the RAG
process. The approach that yields responses most similar to the ground truth
causes, as indicated by higher metric scores, is considered more effective in
identifying and reasoning about causal relationships in social media data.

##  VI Results

###  VI-A Quantitaive evaluation

We present a comparative analysis of the baseline model and the RAG with KG
model. The baseline model processes the entire text corpus without additional
context from the knowledge graph, while the RAG with KG model utilizes
relevant contextual information dynamically retrieved from the knowledge
graph.

Figure 4: Box Plot showing metrics between the Baseline and Proposed Models

####  VI-A1 Baseline Model vs. KG RAG with LLM Model

The baseline model generates responses based solely on the raw text data,
leading to variations in responses for the same set of queries when run
multiple times. This is a characteristic of generative AI models like LLMs,
which can produce different outputs depending on the input context and
randomness inherent in the generation process, unless fine tuned
comprehensively for datasets or most efficient prompt engineering. In
contrast, the RAG with KG model enhances the generation process by providing
specific context derived from the knowledge graph, resulting in more
consistent and accurate responses.

####  VI-A2 Interpreting the Metric Values

The 3 average metric scores for both models observed over multiple queries are
presented in Table I. The RAG with KG model outperforms the baseline model
across all metrics, indicating its superior performance in generating
responses that are both accurate and comprehensive.

TABLE I: Evaluation Metrics for Baseline GPT-3.5 Turbo Model and RAG with KG Model Metric | Baseline Model | RAG with KG
---|---|---
BLEU | 0.42168 | 0.48357
Jaccard Similarity | 0.47733 | 0.57528
Cosine Sim with LLM Encoder | 0.9220 | 0.92563

The higher BLEU scores for the RAG with KG model suggest that incorporating
relevant context from the knowledge graph significantly enhances the model’s
ability to generate responses that closely match the ground truth causes. The
Jaccard similarity scores indicate that the proposed model has a better
overlap of tokens generated between the response and the true cause,
suggesting it is generating more lexically similar texts. And higher Cosine
similarity scores with BERT indicates the model produces more semantically
similar responses than the baseline. This demonstrates the importance of
providing semantic and local contextual information, which the RAG with KG
model achieves by combining text from multiple parts of the knowledge graph.

####  VI-A3 Analysis of Box Plot

Based on the box plots (Fig. 4) and the associated analysis, we can infer the
following about the performance of the KG RAG and LLM models:

####  VI-A4 BLEU Scores Plots

  * •

KG-RAG-LLM Model: The KG RAG model shows a wider range of BLEU scores, with
some instances achieving very high scores. This suggests that the model has
the capability to produce highly relevant and accurate text in some cases. The
higher variability, indicated by the wide interquartile range and the position
of the average score (orange dot) above the median, suggests that while the
model can perform very well, it is not very consistent with high-performance
across all instances.

  * •

Baseline LLM Model: The BLEU scores for the LLM model are a little more
tightly clustered, indicating more consistent performance. However, the lower
average and median scores suggest that, on average, the LLM model does not
achieve the same high level of relevance and accuracy as the KG RAG model. The
consistency is beneficial for tasks requiring stable performance but may lack
the peak performance seen in the KG RAG model.

####  VI-A5 Jaccard Similarity Plots

  * •

KG-RAG-LLM Model: The Jaccard Similarity scores for the KG RAG model are
higher, with a wider range of values. The box plot shows a median around
0.6471, indicating that at least half of the values are above this point. The
interquartile range (IQR), which is the height of the box, is relatively
large, suggesting some variability in the similarity scores which are still
above the scores from the baseline model.

  * •

Baseline LLM Model: The Jaccard Similarity scores for the LLM model are lower
compared to the KG RAG model. The box plot shows a median around 0.4667, which
is lower than the KG RAG model.

####  VI-A6 Cosine Similarity Plots

  * •

KG-RAG-LLM Model: The BERT Encoding Similarity scores for the KG RAG model are
consistently high. The box plot shows a median close to 0.9327, indicating
high semantic similarity for at least half of the values. The IQR is small,
suggesting that the scores are tightly clustered around the median. This too
has a bit more variability with wider range though but higher on average.

  * •

Baseline LLM Model: The Jaccard Similarity scores for the LLM model are lower
compared to the KG RAG model. The box plot shows a median around 0.4667, which
is lower than the KG RAG model. It is consistent but doesn’t peak in
performance as the former.

####  VI-A7 Overall Performance Inference

The KG RAG model demonstrates superior average performance and the capability
to produce highly relevant and accurate text, as indicated by higher metric
scores. This makes it particularly beneficial for analyzing social media data,
where context and relevance are crucial. The variability in its performance
highlights its potential to excel in many instances, suggesting that
integrating knowledge graphs with retrieval-augmented generation enhances the
model’s ability to generate high-quality text. For social media data analysis,
where the context of posts and the connections between them can vary widely,
the KG RAG model’s ability to leverage structured knowledge and retrieve
relevant information dynamically is a significant advantage. This approach
allows the model to better understand and generate text that aligns closely
with the nuances of social media content, leading to more accurate sentiment
analysis, trend detection, and contextual understanding. In contrast, while
the LLM model provides more consistent performance, its generally lower scores
indicate less relevance and accuracy in text generation. The absence of high
outliers further suggests that the LLM model struggles to reach the peak
performance seen with the KG RAG model. This can be a limitation when dealing
with the dynamic and context-rich nature of social media data, where capturing
the full meaning and implications of posts is essential. But the higher
variations in the scores suggests there is room for improvement specifically
in the way KG is constructed. Right now it only conveys correlation
information and due to the controlled testing environment being finite and
small, a larger data environment could yield better results. In contrast, it
would mean a larger overhead for the baseline models which can potentially
affect the results along with longer processing periods. Having a graph DB
helps with dynamic real time social media updates and faster querying.
Therefore, using a KG with RAG offers significant advantages for social media
data analysis. It enhances the model’s ability to generate contextually
relevant and accurate information, improving the quality of insights derived
from social media content. This combined approach leverages the strengths of
both knowledge graphs and retrieval-augmented generation, resulting in better
overall performance.

###  VI-B Sample result for demonstration

Query: ”What caused people to appreciate simple things around them during the
COVID-19 pandemic?”

#### Retrieved Context from KG

From the Knowledge Graph (KG), the derived context would include triplets
like:

  * •

(COVID-19, caused, appreciation for simple things)

  * •

(Lockdown measures, led to, isolation)

  * •

(Isolation, increased, awareness of immediate surroundings)

  * •

(Pandemic, restricted, activities)

  * •

(Activities restricted, led to, more time at home)

The graph information corresponds to the following preprocessed tweets from
the text corpus in chronological order that forms the context,

  * •

”the one gift covid19 has given me is an appreciation for the simple things
that were always around me.”

  * •

”people felt isolated due to lockdown measures, which led to a heightened
awareness of their immediate surroundings.”

  * •

”during the pandemic, many activities were restricted, causing individuals to
spend more time at home.”

####  VI-B1 Response from LLM (GPT-3.5 Turbo) with KG-RAG Context

Response:

> ”People appreciated simple things around them during the COVID-19 pandemic
> due to several interconnected reasons. The lockdown measures caused
> isolation, which heightened awareness of their immediate surroundings. With
> many activities restricted, individuals spent more time at home. This
> combination of factors led people to notice and value the simple things that
> were always around them, providing a sense of comfort and appreciation
> during challenging times.”

####  VI-B2 Response from LLM (GPT-3.5 Turbo) without KG Context

Response:

> ”People appreciated simple things during the COVID-19 pandemic because they
> spent more time at home and had fewer social interactions.”

####  VI-B3 Explanation

With KG-RAG Context:

  * •

The response is enriched with specific reasons derived from the knowledge
graph, providing a comprehensive explanation that ties together multiple
factors contributing to the appreciation of simple things.

  * •

The derived context from the KG includes detailed information about isolation
due to lockdown measures, restricted activities, and more time spent at home,
resulting in a more reasonable answer.

Without KG Context:

  * •

The response is more generic and lacks the depth and detail provided by the
context from the KG.

  * •

It does not connect multiple factors and provides a simpler, less informative
explanation.

####  VI-B4 Comparison

  * •

With KG-RAG Context: The response consistently provides a detailed and
contextually enriched explanation that combines multiple factors, resulting in
a comprehensive understanding of the query.

  * •

Without KG Context: The responses vary each time and are generally less
detailed and specific. They do not provide a holistic view and may miss
important contextual connections.

By integrating the KG with the LLM using the RAG process, the system
effectively reduces the variability in responses and enhances the detail and
relevance of the information provided, resulting in more accurate and
insightful answers.

###  VI-C Qualitative evaluation

TABLE II: Selective tweets identified through KG embeddings form the context for the KG+LLM model, whereas the LLM model considers the entire text corpus Question | Summary from LLM | Tweets | Summary from KG+LLM
---|---|---|---
What measures are being taken to support patients recovering from COVID-19? |  Measures are being taken to support patients recovering from COVID-19 due to the impacts of the pandemic and the need to address the health and well-being of individuals affected by the virus. |  | [2020-07-25 12:02:41] Rajasthan government today
---
started plasma bank at sawai man singh hospital
in jaipur for treatment of covid 19.
[2020-07-25 12:25:24] Mom is in icu due to covid
just want prayer from you and everyone who
is listening you covid19.
[2020-07-25 12:26:59] Praying for good health
and recovery form covid19.
Efforts to enhance treatment, such as the setup of a plasma bank, prayer
support, and sharing of updates on ICU patients indicate a focus on improving
recovery outcomes for COVID-19 patients.
How did the COVID-19 pandemic affect grocery shopping behaviors and store operations? |  Fear of shortages and uncertainty due to the COVID-19 pandemic led to panic buying and stockpiling of essential items, such as food, toilet paper, and sanitizing products, in anticipation of potential disruptions in supply chains and daily routines. |  | [2020-03-16]: Coronavirus Australia: Woolworths
---
to give elderly, disabled dedicated shopping hours
amid COVID-19 outbreak
[2020-03-16]: As news of the region’s first confirmed
COVID-19 case came out of Sullivan
County last week, people flocked to area
stores to purchase cleaning supplies,
hand sanitizer, food, toilet paper and other goods.
[2020-03-16]: Due to the Covid-19 situation,
we have increased demand for all food products…
The wait time may be longer for all online orders,
particularly beef share and freezer packs.
[2020-03-16]: All month there hasn’t been crowding
in the supermarkets or restaurants, however
reducing all the hours and closing the
malls means everyone is now using the same
entrance and dependent on a single supermarket
The COVID-19 pandemic led to panic buying, resulting in empty shelves and
increased demand for online shopping. Grocery stores implemented measures like
dedicating shopping hours for the elderly and encouraging online orders. The
pandemic caused shortages of essential items like toilet paper and hand
sanitizer, and store employees faced increased pressure to keep shelves
stocked.

##  VII Related Work

The burgeoning interest in leveraging computational methods for dynamic large
text data such as social media has led to significant advancements in both
data representation and analysis techniques. At the forefront, Knowledge
Graphs (KGs) and Large Language Models (LLMs) have proven instrumental.
Advanced Graph Techniques and NER in Social Media [dieudonat2020exploring]
explore the combination of contextual word encodings and KG embeddings,
supporting our goal of enhancing sentiment analysis through integrated
representations. [liu2011recognizing] and [liu2013named] detail the importance
of effective Named Entity Recognition (NER) in social media texts, which
informs our preprocessing steps, ensuring accurate entity recognition before
sentiment analysis. [haradizadehtweeki] introduce Tweeki, an entity linking
system that bridges unstructured tweets with structured knowledge bases,
enhancing our capability to map and analyze entities within social media data.
Additionally, the article from [towardsDS] implements a fine tuning of a
LLAMA3 model to perform relation extraction (triples) achieving high
performance. The 8B model is one of the best GenerativeAI models till date
with upgraded tokenizer and context window for efficient training on data.
Applications and Methodological Insights[shamrat2021sentiment] demonstrate the
application of NLP and machine learning in analyzing public health sentiments
on Twitter, paralleling our domain of analysis. [dessi2021generating] and
[peng2023knowledge] provide advanced techniques for generating and utilizing
KGs in large datasets typical of social media environments, enriching our
methodology. [lu2020vgcn] propose VGCN-BERT, which combines BERT with Graph
Convolutional Networks to enhance text classification tasks, providing
insights into integrating local and global information for better analysis.
Causation from Social Media through Time Series Events The paper
[kayesh2019event] explores the integration of temporal information into causal
reasoning frameworks. They emphasize the importance of capturing the temporal
sequence of events to accurately infer causation. The methodologies presented
incorporate time-based features into causal analysis, demonstrating
significant improvements in identifying causal relationships.
Causal Knowledge Graphs and Explainability [tan2023constructing],
[jaimini2022causalkg], and [gopalakrishnan2023text] delve into constructing
causal KGs from varied data sources and enhancing explainability through
causal reasoning. These studies provide a framework for our work in creating
KGs that not only predict relations and similarities but also causation,
offering actionable insights into the factors driving information on social
media.
Causality with Large Language Models [kiciman2023causal] and [jin2023can]
explore the capabilities of LLMs in causal reasoning. [loureiro2022timelms]
introduce TimeLMs, which are diachronic language models that adapt to the
evolving linguistic landscape of social media, enhancing our ability to
capture and analyze the nuances of cause-effect relationship over time.
Retrieval-Augmented Generation (RAG) Process The RAG process plays a pivotal
role in our methodology by using LLMs and KGs to rank similarities between
input text and a list of sentences. This approach provides the top 5 ranked
sentences containing causal information, thus reducing the context for further
LLM-based cause extraction. [pan2024unifying] propose frameworks for unifying
LLMs and KGs, which we build upon to enhance predictive power and semantic
richness.
Combining LLMs and KGs The integration of Knowledge Graphs (KGs) with Large
Language Models (LLMs) signifies a transformative shift in computational
methods. [pan2024unifying] outline a comprehensive roadmap for merging LLMs
and KGs, which we build upon. This unification harnesses the predictive power
of LLMs and the rich semantic structures of KGs, overcoming limitations found
in each model when used in isolation. Our research leverages these frameworks
to construct an ideal pipeline that better suits our goal. Through the
integration of these pioneering studies, our research leverages cutting-edge
technologies and methodologies to advance the field of causal reasoning. It
combines the concepts of NER / relation extraction with LLMs (better prospect
in case of unstructured, ambiguous social media data), KG constructions and
databases, Embedding techniques, RAG and prompt engineering.

##  VIII Limitations

Dependence on Data Quality: Social media data, particularly tweets, are often
noisy, ambiguous, and unstructured. Preprocessing steps like tokenization,
lemmatization, and stopword removal help, but may not fully eliminate
inaccuracies. By exploring advanced preprocessing techniques, including more
sophisticated noise reduction algorithms and robust entity recognition models
such as a combination of LLM models with word embeddings, to better handle the
unstructured nature of social media data.
Scalability Issues: While Neo4j is efficient, handling extremely large
datasets, especially with real-time updates, can strain resources and affect
performance especially when the dynamic updates are too much in volume in
short spans of time. This research considered a finite volume of data over a
period of time with accommodation to integrate new tweets. Considering
distributed graph databases or graph processing frameworks like Apache
TinkerPop or DGraph that offer better scalability and performance for large-
scale data.
Model Integration Challenges: Integrating various models and technologies
(e.g., Node2Vec, BERT-Uncased, GPT-3.5 Turbo) can be challenging due to
differences in data formats, embedding spaces, domains and processing
requirements. By developing a more streamlined data pipeline with consistent
data formats and unified embedding spaces. And investigating the use of
middleware or APIs that can facilitate smoother integration.
Limited Causation Information: Current KG construction focuses on correlation
rather than causation. This can limit the depth of causal analysis and may
hinder the efficiency of perceived insights. One option would be to perform KG
construction to explicitly capture causation information. Explore methods like
causal inference algorithms (e.g., Granger causality, structural causal
models) to better identify and represent causal relationships.
Evaluation Metrics Metrics like BLEU, Jaccard Similarity, and Cosine
Similarity provide quantitative assessments and have been chosen specifically
for the type of data needed to be evaluated but may not fully capture the
qualitative aspects of causal reasoning. By incorporating human evaluation for
a robust qualitative assessment of causal inferences. Computational Resources:
The combination of LLMs, graph embeddings, and real-time querying demands
significant computational resources, potentially limiting accessibility and
scalability. Optimizing the model and query processing pipeline for efficiency
and automations with cloud-based solutions for scalable and cost-effective
resource management.

##  IX Conclusion and Future work

In this research we demonstratethe effectiveness of integrating Knowledge
Graphs (KGs) with Large Language Models (LLMs) for causal analysis of social
media data. The combined approach significantly enhances the depth and
accuracy of understanding the factors driving social media interactions. The
Retrieval-Augmented Generation (RAG) method used in our framework improves the
quality of causal inferences, as validated by higher similarity scores
compared to the baseline model. This integration leverages the strengths of
both KGs and LLMs, providing detailed and contextually enriched insights that
are crucial for informed decision-making. The added layer of temporal
information further ensures that only temporally plausible relationships are
considered, and it enriches the context for queries by capturing the dynamic
evolution of events over time. Overall, the integration of KGs with LLMs
offers a powerful methodology for analyzing the complex dynamics of social
media data, yielding more precise and actionable insights. In the future, we
aim to refine the construction of the knowledge graph (KG) further by
including causation information along with correlation to ensure better
retrieval of context for the LLM query. This enhancement will provide a more
robust understanding of causal relationships in the data. Additionally, we
will explore alternative embedding techniques such as GraphSAGE or GAT (Graph
Attention Networks) to capture richer semantic information.